<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On Oct 26, 2009, at 10:55 AM, <a href="mailto:beowulf-request@beowulf.org">beowulf-request@beowulf.org</a> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; ">Message: 6<br>Date: Mon, 26 Oct 2009 10:50:26 -0500<br>From: Rahul Nabar <<a href="mailto:rpnabar@gmail.com">rpnabar@gmail.com</a>><br>Subject: Re: [Beowulf] any creative ways to crash Linux?: does a<br><span class="Apple-tab-span" style="white-space: pre; "> </span>shared NIC<span class="Apple-converted-space"> </span><span class="Apple-tab-span" style="white-space: pre; "> </span>IMPI always remain responsive?<br>To: Bogdan Costescu <<a href="mailto:bcostescu@gmail.com">bcostescu@gmail.com</a>><br>Cc: Beowulf Mailing List <<a href="mailto:beowulf@beowulf.org">beowulf@beowulf.org</a>><br>Message-ID:<br><span class="Apple-tab-span" style="white-space: pre; "> </span><<a href="mailto:c4d69730910260850w5daf7de0ue26340adf8589da1@mail.gmail.com">c4d69730910260850w5daf7de0ue26340adf8589da1@mail.gmail.com</a>><br>Content-Type: text/plain; charset=ISO-8859-1<br><br>On Mon, Oct 26, 2009 at 8:11 AM, Bogdan Costescu <<a href="mailto:bcostescu@gmail.com">bcostescu@gmail.com</a>> wrote:<br><blockquote type="cite">On Sat, Oct 24, 2009 at 11:13 PM, Rahul Nabar <<a href="mailto:rpnabar@gmail.com">rpnabar@gmail.com</a>> wrote:<br></blockquote><blockquote type="cite"><blockquote type="cite">What surprised me was that even if I take down my eth interface with a<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">ifdown the IPMI still works. How does it do that ?<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">The IPMI traffic is IP (UDP) based and by inspecting the IP header one<br></blockquote><blockquote type="cite">can make a difference between packets with the same MAC and different<br></blockquote><blockquote type="cite">IPs.<br></blockquote><br>Actually, the MAC is different too. I have one NIC but it responds to<br>two MACs. I guess one is transparent to the OS and the other is<br>handled by the BMC.<br></span></blockquote>Correct. In some blades they used to share the mac, but I don't think anyone does that anymore. The BMC MAC/IP is hidden and functional regardless of the OS state. IPMI drivers can talk to the chip through the OS if need be by starting the appropriate service or kernel modules, but that's usually only fun for configuring the card, since you'll use the Network interface in most situations. </div><div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><br><br><br><blockquote type="cite">taken down, it's the Linux networking stack that doesn't see any<br></blockquote><blockquote type="cite">packet coming in, however the BMC's network stack will still be<br></blockquote><blockquote type="cite">active. That's the whole point of the BMC being a separate entity from<br></blockquote><blockquote type="cite">the main system, so that its functionality remains undisturbed when<br></blockquote><blockquote type="cite">something bad happens to the main system.<br></blockquote><br>I see. So I assume the BMC's network stack is something that's<br>hardware or firmware implemented. It's funny that in spite of this the<br>IPMI gets hung sometimes (like Gerry says in his reply). I guess I can<br>just attribute that to bad firmware coding in the BMC.<br></span></blockquote>"A Rich feature set" includes these issues :)</div><div><br><blockquote type="cite"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><br><br><blockquote type="cite"><blockquote type="cite">Another mysterious observation was this: Whenever I took eth down via the OS there is a latent period when the IPMI stops<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">responding but then somehow it magically resurrects itself and starts working again.<br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Without claiming that this is the best explanation: it's possible that<br></blockquote><blockquote type="cite">the Linux driver talks to the hardware and takes down the link at the<br></blockquote><blockquote type="cite">physical level. The BMC driver then detects this and brings the link<br></blockquote><blockquote type="cite">back up so that it can continue to receive the IPMI packets.<br></blockquote><br>You are probably right. THe explanation sounds reasonable to me. A<br>similar observation is for accessing the BIOS as well. The BMC stack<br>is not responsive right from the power-up. It does become responsive<br>for a bit but then the system drags it down (maybe when the BIOS hands<br>over to PXE). If I manage to "ipmitool sol activate" within this<br>correct window then I am able to see the BIOS. But that's pretty much<br>trial and error.<br></span></blockquote><div>You will probably also notice the BMC only brings the link up at 100Mb but the OS brings it up to 1Gb. Switches can add some lag here too, if Spanning tree is enabled. Turning off Spanning Tree or turning on "Port Fast" will help. Otherwise there is a period of up to about 40 seconds that the link is "up" but the switch hasn't started passing traffic (as it checks to make sure there's no ethernet loop). This has caused many Cluster Deployments hours of head banging.</div></div><br><div>Cheers!</div><div>Greg</div><div><br></div></body></html>