[Beowulf] Q: IB message rate & large core counts (per node)?

[Gilad Shainer]

When you look on low level, marketing driven benchmarks, you should be careful. Mellanox latest message rate numbers with ConnectX-2 more than doubled versus the old cards, and are for real message rate - separate messages on the wire. The competitor numbers are with using message coalescing, so it is not real separate messages on the wire, or not really message rate.

Gilad



-----Original Message-----
From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Michael Di Domenico
Sent: Friday, February 19, 2010 12:49 PM
To: beowulf at beowulf.org
Subject: Re: [Beowulf] Q: IB message rate & large core counts (per node)?

the folks on the linux-rdma mailing list can probably share some
slides with you about app load over different cards.  if you dont get
a response, i can drop a few names of people who definitely have the
info, but i dont want to do it at large on the list

The last set of slides i can (thinking way back when i was still with
qlogic) recall, yes the ipath cards could do 30m mesg/sec whereas the
mlnx cards were half to a two thirds lower.  this does have an affect
on core count traffic, but only under certain application loads

mlnx and qlogic made a trade off with their card designs, the qlogic
cards have a real high msg/rate, but the rdma bandwidth performance
can suffer in certain cases, where as mlnx has a higher rdma bandwidth
but a lower mesg rate

getting the balance right is a mastery of art.  and the tipping point
slides easily and with every application

it's been a while since i was at qlogic and have forgotten a lot of
the sales mumbo...



On Fri, Feb 19, 2010 at 1:25 PM, Brian Dobbins <bdobbins at gmail.com> wrote:
>
> Hi guys,
>
>   I'm beginning to look into configurations for a new cluster and with the
> AMD 12-core and Intel 8-core chips 'here' (or coming soonish), I'm curious
> if anyone has any data on the effects of the messaging rate of the IB
> cards.  With a 4-socket node having between 32 and 48 cores, lots of
> computing can get done fast, possibly stressing the network.
>
>   I know Qlogic has made a big deal about the InfiniPath adapter's extremely
> good message rate in the past... is this still an important issue?  How do
> the latest Mellanox adapters compare?  (Qlogic documents a ~30M messages
> processsed per second rate on its QLE7342, but I didn't see a number on the
> Mellanox ConnectX-2... and more to the point, do people see this effecting
> them?)
>
>   On a similar note, does a dual-port card provide an increase in on-card
> processing, or 'just' another link?  (The increased bandwidth is certainly
> nice, even in a flat switched network, I'm sure!)
>
>   I'm primarily concerned with weather and climate models here - WRF, CAM,
> CCSM, etc., and clearly the communication rate will depend to a large degree
> on the resolutions used, but any information, even 'gut instincts' people
> have are welcome.  The more info the merrier.
>
>   Thanks very much,
>   - Brian
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
>

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf