From harshscience777 at gmail.com Thu Jun 3 12:07:41 2021 From: harshscience777 at gmail.com (harsh_google lastname) Date: Thu, 3 Jun 2021 17:37:41 +0530 Subject: [Beowulf] Theoretical peak performance of DGX A100 Message-ID: I am calculating the theoretical peak (FP64) performance of the Nvidia DGX A100 system. Now, A100 datasheet lists FP64 performance to be 9.7 TFLOPS. Two AMD 7742 CPUs will give 128 cores x 2.25 GHz base clock x 16 FP64 ops / cycle = 4.6 TFLOPS. This gives a total of 82.2 TFLOPS per DGX-A100. Here is my problem. For any system with DGX A100 on top500.org, numbers just don't add up. For eg: Selene has 560 DGX boxes, but its theoretical peak is listed as 79.2 PFLOPS, whereas I expect it should be 46 PFLOPS (ie 82.2 TFLOPS x560). The same is true for any other DGX based system listed on top500. What am I missing here? Thanks! Harsh Hemani -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.bederian at unc.edu.ar Thu Jun 3 12:20:17 2021 From: carlos.bederian at unc.edu.ar (=?UTF-8?Q?Carlos_Bederi=C3=A1n?=) Date: Thu, 3 Jun 2021 09:20:17 -0300 Subject: [Beowulf] Theoretical peak performance of DGX A100 In-Reply-To: References: Message-ID: A100 does 19.5 FP64 TFLOPS using tensor cores. On Thu, Jun 3, 2021 at 9:08 AM harsh_google lastname < harshscience777 at gmail.com> wrote: > I am calculating the theoretical peak (FP64) performance of the Nvidia DGX > A100 system. > > Now, A100 datasheet lists FP64 performance to be 9.7 TFLOPS. > Two AMD 7742 CPUs will give 128 cores x 2.25 GHz base clock x 16 FP64 ops > / cycle = 4.6 TFLOPS. > This gives a total of 82.2 TFLOPS per DGX-A100. > > Here is my problem. For any system with DGX A100 on top500.org, numbers > just don't add up. For eg: Selene has 560 DGX boxes, but its theoretical > peak is listed as 79.2 PFLOPS, whereas I expect it should be 46 PFLOPS (ie > 82.2 TFLOPS x560). The same is true for any other DGX based system listed > on top500. What am I missing here? > > Thanks! > > Harsh Hemani > _______________________________________________ > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing > To change your subscription (digest mode or unsubscribe) visit > https://beowulf.org/cgi-bin/mailman/listinfo/beowulf > -------------- next part -------------- An HTML attachment was scrubbed... URL: From harshscience777 at gmail.com Thu Jun 3 12:22:30 2021 From: harshscience777 at gmail.com (harsh_google lastname) Date: Thu, 3 Jun 2021 17:52:30 +0530 Subject: [Beowulf] Theoretical peak performance of DGX A100 In-Reply-To: References: Message-ID: But that wouls bring the theoretical performance to 160 TFLOPS per box, which also doesn't match! On Thu, Jun 3, 2021, 5:50 PM Carlos Bederi?n wrote: > A100 does 19.5 FP64 TFLOPS using tensor cores. > > On Thu, Jun 3, 2021 at 9:08 AM harsh_google lastname < > harshscience777 at gmail.com> wrote: > >> I am calculating the theoretical peak (FP64) performance of the Nvidia >> DGX A100 system. >> >> Now, A100 datasheet lists FP64 performance to be 9.7 TFLOPS. >> Two AMD 7742 CPUs will give 128 cores x 2.25 GHz base clock x 16 FP64 ops >> / cycle = 4.6 TFLOPS. >> This gives a total of 82.2 TFLOPS per DGX-A100. >> >> Here is my problem. For any system with DGX A100 on top500.org, numbers >> just don't add up. For eg: Selene has 560 DGX boxes, but its theoretical >> peak is listed as 79.2 PFLOPS, whereas I expect it should be 46 PFLOPS (ie >> 82.2 TFLOPS x560). The same is true for any other DGX based system listed >> on top500. What am I missing here? >> >> Thanks! >> >> Harsh Hemani >> _______________________________________________ >> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >> To change your subscription (digest mode or unsubscribe) visit >> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.bederian at unc.edu.ar Thu Jun 3 12:55:36 2021 From: carlos.bederian at unc.edu.ar (=?UTF-8?Q?Carlos_Bederi=C3=A1n?=) Date: Thu, 3 Jun 2021 09:55:36 -0300 Subject: [Beowulf] Theoretical peak performance of DGX A100 In-Reply-To: References: Message-ID: The Top500 has been listing wrong Rpeak values for most clusters for many years now, so I wouldn't dwell on it... Take a Skylake-based cluster like Frontera. Its listed Rpeak is 38,745.9 TFLOPS = 8008 nodes * 56 cores * 32 ops/cycle * 2.7GHz. But 2.7GHz is the regular base frequency, and to do 32 ops/cycle you need to use AVX-512. All-core AVX-512 frequencies for a Xeon 8280 are 1.8GHz base and 2.4GHz turbo, so the Rpeak is off by 12-33%. On Thu, Jun 3, 2021 at 9:22 AM harsh_google lastname < harshscience777 at gmail.com> wrote: > But that wouls bring the theoretical performance to 160 TFLOPS per box, > which also doesn't match! > > On Thu, Jun 3, 2021, 5:50 PM Carlos Bederi?n > wrote: > >> A100 does 19.5 FP64 TFLOPS using tensor cores. >> >> On Thu, Jun 3, 2021 at 9:08 AM harsh_google lastname < >> harshscience777 at gmail.com> wrote: >> >>> I am calculating the theoretical peak (FP64) performance of the Nvidia >>> DGX A100 system. >>> >>> Now, A100 datasheet lists FP64 performance to be 9.7 TFLOPS. >>> Two AMD 7742 CPUs will give 128 cores x 2.25 GHz base clock x 16 FP64 >>> ops / cycle = 4.6 TFLOPS. >>> This gives a total of 82.2 TFLOPS per DGX-A100. >>> >>> Here is my problem. For any system with DGX A100 on top500.org, numbers >>> just don't add up. For eg: Selene has 560 DGX boxes, but its theoretical >>> peak is listed as 79.2 PFLOPS, whereas I expect it should be 46 PFLOPS (ie >>> 82.2 TFLOPS x560). The same is true for any other DGX based system listed >>> on top500. What am I missing here? >>> >>> Thanks! >>> >>> Harsh Hemani >>> _______________________________________________ >>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing >>> To change your subscription (digest mode or unsubscribe) visit >>> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From harshscience777 at gmail.com Thu Jun 3 13:29:50 2021 From: harshscience777 at gmail.com (harsh_google lastname) Date: Thu, 3 Jun 2021 18:59:50 +0530 Subject: [Beowulf] Theoretical peak performance of DGX A100 In-Reply-To: References: Message-ID: Cool, thanks! On Thu, Jun 3, 2021, 6:25 PM Carlos Bederi?n wrote: > The Top500 has been listing wrong Rpeak values for most clusters for many > years now, so I wouldn't dwell on it... > > Take a Skylake-based cluster like Frontera. Its listed Rpeak is 38,745.9 > TFLOPS = 8008 nodes * 56 cores * 32 ops/cycle * 2.7GHz. > But 2.7GHz is the regular base frequency, and to do 32 ops/cycle you need > to use AVX-512. All-core AVX-512 frequencies for a Xeon 8280 are 1.8GHz > base and 2.4GHz turbo, so the Rpeak is off by 12-33%. > > On Thu, Jun 3, 2021 at 9:22 AM harsh_google lastname < > harshscience777 at gmail.com> wrote: > >> But that wouls bring the theoretical performance to 160 TFLOPS per box, >> which also doesn't match! >> >> On Thu, Jun 3, 2021, 5:50 PM Carlos Bederi?n >> wrote: >> >>> A100 does 19.5 FP64 TFLOPS using tensor cores. >>> >>> On Thu, Jun 3, 2021 at 9:08 AM harsh_google lastname < >>> harshscience777 at gmail.com> wrote: >>> >>>> I am calculating the theoretical peak (FP64) performance of the Nvidia >>>> DGX A100 system. >>>> >>>> Now, A100 datasheet lists FP64 performance to be 9.7 TFLOPS. >>>> Two AMD 7742 CPUs will give 128 cores x 2.25 GHz base clock x 16 FP64 >>>> ops / cycle = 4.6 TFLOPS. >>>> This gives a total of 82.2 TFLOPS per DGX-A100. >>>> >>>> Here is my problem. For any system with DGX A100 on top500.org, >>>> numbers just don't add up. For eg: Selene has 560 DGX boxes, but its >>>> theoretical peak is listed as 79.2 PFLOPS, whereas I expect it should be 46 >>>> PFLOPS (ie 82.2 TFLOPS x560). The same is true for any other DGX based >>>> system listed on top500. What am I missing here? >>>> >>>> Thanks! >>>> >>>> Harsh Hemani >>>> _______________________________________________ >>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin >>>> Computing >>>> To change your subscription (digest mode or unsubscribe) visit >>>> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mdidomenico4 at gmail.com Mon Jun 14 16:38:50 2021 From: mdidomenico4 at gmail.com (Michael Di Domenico) Date: Mon, 14 Jun 2021 12:38:50 -0400 Subject: [Beowulf] odd vlan issue Message-ID: i got roped into troubleshooting an odd network issue. we have a mix of cisco (mostly nexus) gear spread over our facility. on one particular vlan it's operating as if it's a hub instead of switch. has anyone seen this before? i bounced around the net, my issue appears to be "unicast flooding", but the chief correction seems to be adjusting the ARP/CAM timeouts. but that hasn't working. i'm sure this is a loaded question given none of you can see the network, but i'm grasping at straws to come up with a reason why this is happening. what's weird is we have two dozen other vlans, which aren't affected. i cannot locate a difference in the config on the switches. any thoughts are appreciated at this point