[Beowulf] building Infiniband 4x cluster questions
diep at xs4all.nl
Mon Nov 7 12:09:46 PST 2011
It seems the latency of DDR infiniband to do a blocked read from
remote memory (RDMA) is between that of SDR and quadrics, with
quadrics being a lot faster.
If i click there i get to a MarkLehrer.pdf
It claims a RDMA read has latency of 1.91 us.
However i'll have to see that in my own benchmark first before i
believe it when we hammer with many different processes
at that card at the same time. You get problems like switch latencies
and other nasty stuff then.
This is a presentation slide and i need something that works in reality.
HP 4X DDR InfiniBand Mezzanine HCA 410533-B21 SFF-8470
they're $75 but just 2 of them available on ebay. The next 'ddr' one is
QLE7104 QLOGIC INFINIBAND 8X DDR SINGLE PORT HBA
So that's a qlogic one, $108 just 3 of them available, but we already
get at a dangerous price level.
Remember i want well over a million reads getting done a second and i
didn't count the pollution by writes even yet.
HP 4X DDR InfiniBand Mezzanine HCA - 2 Ports 448262-B21
They're $121 and again just 2 available.
This seems a problem with infiniband on ebay. Even if you search 16
cards, you can each time buy 2 or so max.
As if sometimes a scientist takes 2 back home and puts 'em on ebay.
No big 'old' masses get posted there.
The first one to offer 10, that's http://www.ebay.com/itm/HP-
That 's at $192.11 a piece.
It seems DDR infiniband still isn't in my pricerange Prentice.
The QM500-B's from quadrics go for between $30 and $50 however.
On Nov 7, 2011, at 7:16 PM, Prentice Bisbal wrote:
> Don't forget that between SDR and QDR, there is DDR. If SDR is too
> slow, and QDR is too expensive, DDR might be just right.
> On 11/07/2011 11:58 AM, Vincent Diepeveen wrote:
>> hi Prentice,
>> I had noticed the diff between SDR up to QDR,
>> the SDR cards are affordable, the QDR isn't.
>> The SDR's are all $50-$75 on ebay now. The QDR's i didn't find cheap
>> prices in that pricerange yet.
>> If i would want to build a network that's low latency and had a
>> of $800 or so a node of course i would
>> build a dolphin SCI network, as that's probably the fastest latency
>> card sold for a $675 or so a piece.
>> I do not really see a rival latency wise to Dolphin there. I bet most
>> manufacturers selling clusters don't use
>> it as they can make $100 more profit or so selling other networking
>> stuff, and universities usually swallow that.
>> So price total dominates the network. As it seems now infiniband
>> 4x is
>> not going to offer enough performance.
>> The one-way pingpong latencies over a switch that i see of it, are
>> very convincing. I see remote writes to RAM
>> are like nearly 10 microseconds for 4x infiniband and that card is
>> only one affordable.
>> The old QM400's i have here are one-way pingpong 2.1 us or so, and
>> QM500-B's are plentyful on the net (of course big disadvantage: needs
>> which are a 1.3 us or so there and have SHMEM. Not seeing a cheap
>> switch for the QM500's though nor cables.
>> You see price really dominates everything here. Small cheap nodes you
>> cannot build if the port price, thanks to expensive network card,
>> more than doubles.
>> Power is not the real concern for now - if a factory already burns a
>> couple of hundreds of megawatts, a small cluster somewhere on the
>> attick eating
>> a few kilowatts is not really a problem :)
>> On Nov 7, 2011, at 5:36 PM, Prentice Bisbal wrote:
>>> On 11/06/2011 06:01 PM, Vincent Diepeveen wrote:
>>>> There is a lot of infiniband 4x stuff on ebay now.
>>> Do you mean 4x, or QDR? They refer to different parts of the IB
>>> architecture. 4x refers to the number of lanes for the data to
>>> down and QDR refers to the data signalling rate.
>>> It's probably irrelevant for this conversation, but if you are just
>>> learning about IB, It's good to understand that difference.
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
>>> To change your subscription (digest mode or unsubscribe) visit
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
> To change your subscription (digest mode or unsubscribe) visit
More information about the Beowulf