[Beowulf] 10G networking?

[Douglas Eadline]

Mark,

I have been pleasantly surprised by the performance of Chelsio.
My tests are very small scale (no switch) and the hardware I'm
testing is for my Limulus desk side cluster.

I ran netpipe TCP and got latencies of 9us back-to-back (SFP+)
Hopping through a two port adapter I got 18 us.

I then tried openMX. On omx_perf I got the following latencies,

 0 hops 5.503 us (Head to n0)
 1 hops 10.438 us (Head to n1)
 2 hops 15.829 us (Head to n2)

The network is a simple linear connection. looks like this

  head --- n0 --- n1 --- n2

I ran HPL and the results went from 395.2 GFLOPS with 1 GbE to
567.4 GFLOPS with 10 GbE. I saw similar improvements in NAS latency
sensitie codes. The white paper with the data is being prepared
right now.

These test were for Chelsio T420-SO-CR (low cost stripped down
2 port NICS) I just received a 4-port 10 GbE card that I'm going to
test with both the Chelsio T520-SO-CR (2 port) and the Mellanox MCX311A
(1 port) The idea is 4-ports on the head node connected to each of
three compute nodes, with the remaining port open for expansion.

For my purposes, small low cost desk-side HPC, 10-GbE is
an excellent option.

--
Doug

> Hi all,
> I'd appreciate any comments about the state of 10G as a reasonable
> cluster network.  Have you done any recent work on 10G performance?
>
> https://lwn.net/Articles/629155/
>
> shows some excellent evidence-based work on kernel paths, but it
> seems focused on bulk/saturation, rather than single-packet latency.
> (not that 10G MPI wouldn't be saturated in a lot of cases vs FDR!)
>
> I also read some comments recently about improvements that particularly
> benefit certain Intel adapters - anyone have comments on Intel vs
> Broadcom (vs Mellanox, I suppose).
>
> It seems like 10gT is on the cusp of real volume-type prices: I saw a
> quote today for a major-vendor 24pt switch for something like $140/port
> after educational discounts.  I know 10gT is somewhat higher latency
> than other PHYs - or is it worth sticking to SFP+?  Any comments on
> latency properties of various switches?  Any nice results of recent
> hardware and OpenMX?
>
> much appreciated!
> -mark hahn
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
> --
> Mailscanner: Clean
>


-- 
Doug

-- 
Mailscanner: Clean