[Beowulf] 512 nodes Myrinet cluster Challanges

[Joe Landman]

Mark Hahn wrote:
>> IF the link aggregation algorithm sends data over the line which IS 
>> connected to the IPMI card, then you CAN talk to it.  IF the algorithm 
>> sends data over the line which is NOT connected to the IPMI card, then you 
>> can't talk with it.
> 
> so in the case where path = (srcmac ^ dstmac) % links, for only 
> two links, it seems like you could set up two IPMI-monitor-master
> machines (with the right differences in their macs) and have 
> full connectivity.  in fact, the bonding would lead a node's BMC to 
> DHCP from the master that it could reach, so you could even assign them to
> two separate subnets to stay sane... 

On the iWill MB's we used for a cluster recently, we could contact the 
embedded IPMI off of either port.  The issue was  ...

> 
> hah.  sometimes I wonder whether clusters should use straight 
> eth-level addressing, rather than dealing with LACP/arp/etc oddities.

... arp.  The motherboard OS goes down and no longer services arp 
requests.  So you need to arp -s a whole lotta machines in order to have 
the connectivity.  Luckily we automated this, but still it was annoying.

Joe

> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615