From walid.shaari at gmail.com Sat Dec 2 02:45:00 2006 From: walid.shaari at gmail.com (Walid) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] non-proprietary IPMI card? In-Reply-To: References: <456CE9A3.1040103@cse.ucdavis.edu> <456D32CD.2030000@streamline-computing.com> Message-ID: On 11/29/06, Mark Hahn wrote: > >> The Supermicro IPMI cards can share the eth0 port with the main gigabit > >> channel. The cards are powered up and on the network whenever power is > >> applied to the chassis. There is a bridge somewhere on the motherboard > >> which bridges between eth0 and the IPMI interface. > > I would guess that the phy is just stolen by the IPMI, since sharing > the actual eth controller (registers, etc) would probably be too touchy. > > > Wouldn't that sharing introduce a limitation, we have seen that we are > > not able to manage nodes remotley when the node has a kernel panic for > > example. > > no - the IPMI wouldn't interact with the host OS, so if the latter is > paniced (or turned off), it doesn't matter. It does not interact with the OS, but it shares the same network buffer in the NIC, when this gets Full, you can not manage it any more, and apperantly it does get filled up during crashes. regards Walid. From csamuel at vpac.org Sun Dec 3 15:15:06 2006 From: csamuel at vpac.org (Chris Samuel) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] More technical information and spec of beowulf In-Reply-To: References: Message-ID: <200612041015.06679.csamuel@vpac.org> On Thursday 30 November 2006 18:19, reza bakhshi wrote: > How can i find some more detailed technical information on Beowulf > software infrastructure? Hopefully these will help.. http://www.phy.duke.edu/~rgb/Beowulf/beowulf_book/beowulf_book/ http://en.wikipedia.org/wiki/Beowulf_(computing) http://clustermonkey.net/ Good luck! Chris -- Christopher Samuel - (03)9925 4751 - VPAC Deputy Systems Manager Victorian Partnership for Advanced Computing http://www.vpac.org/ Bldg 91, 110 Victoria Street, Carlton South, VIC 3053, Australia -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20061204/65dfeab8/attachment.bin From ballen at gravity.phys.uwm.edu Mon Dec 4 03:48:55 2006 From: ballen at gravity.phys.uwm.edu (Bruce Allen) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks Message-ID: Dear Beowulf list, For my next cluster room, I am hoping to use 'liquid cooled' racks make by Knurr (CoolTherm, http://www.thermalmanagement.de/). The scale is 67 racks x 7.5kW heat removal per rack (36 U usable per rack). Does anyone on the list have experiences with these or similar racks (you can reply to me privately if you don't want to share your experiences publicly)? I am told that there are lots of installations in Europe, especially in Germany, but that the only US sites are at Penn State, somewhere in the Chicago area, and U. of Vermont, and that these US installations are fairly small-scale. The racks have front and back doors that close, and contains fans in the back which circulate air through a heat exchanger located in the bottom of the rack. The heat exchanger transfers the heat into chilled water. The advantages of this are that it is quiet, and that you don't need vertical height for underfloor ducting or overhead hot air removal. Disadvantages are cost, potential difficulty of working within the rack. and loss of one rack of cooling capacity if you open both the front and back doors. (In this case I have access to 'building' funds that can not be used to buy more cpus, so the cost issue is not important.) I would be very interested to hear about people's experiences with these racks (or similar ones from another manufacturer) in an HPC environment. Cheers, Bruce From twm at tcg-hsv.com Mon Dec 4 07:15:34 2006 From: twm at tcg-hsv.com (Tim Moore) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off In-Reply-To: <16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> References: <45578E91.1080407@tcg-hsv.com> <16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> Message-ID: <45743B96.10003@tcg-hsv.com> Update to node drop-off: I wrote a few weeks ago to ask about node drop-off. A quick note...I had a cluster run for 3 years without failure and I upgraded the Opteron 240 CPUs to 250s. The upgrade required a BIOS upgrade and while I was at it, upgraded the OS and security. Some readers provided good suggestions for diagnosis. As it turned out, of the 16 CPU batch...two were flawed. No success was derived from replacing power supplies, HDD, resetting memory and the cooling solution. The CPU flaw only manifested itself (at first) after several hours of CPU usage. With each failure, the time duration shortened before the next failure and by the time I figured it out was down to about 2 minutes. The AMD engineer with whom I talked was amazed that such CPUs made it beyond quality control. He also suggested that the vendor may have inadvertently mixed returned (previously fetermined to be flawed processors) with the new ones and sent them out (again) as new. Just for future reference...is there an easy way to determine if a CPU is flawed with 2 weeks of down time and extensive hair extraction???? Tim -------------- next part -------------- A non-text attachment was scrubbed... Name: twm.vcf Type: text/x-vcard Size: 336 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20061204/c1a5452d/twm.vcf From ladd at che.ufl.edu Mon Dec 4 07:35:29 2006 From: ladd at che.ufl.edu (Tony Ladd) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off In-Reply-To: <45743B96.10003@tcg-hsv.com> Message-ID: <023501c717b9$d425e3b0$656ce30a@ladd02> Tim Our university HPC cluster had similar problems with dual-core opterons 275's. They had about 20 bad ones out of a batch of 400. The nodes would run OK for a while and then die. It took many months to track down the source of the problem-AMD gave the same lame excuse-bad QA. Tony -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Tim Moore Sent: Monday, December 04, 2006 10:16 AM To: beowulf@beowulf.org Subject: Re: [Beowulf] Node Drop-Off Update to node drop-off: I wrote a few weeks ago to ask about node drop-off. A quick note...I had a cluster run for 3 years without failure and I upgraded the Opteron 240 CPUs to 250s. The upgrade required a BIOS upgrade and while I was at it, upgraded the OS and security. Some readers provided good suggestions for diagnosis. As it turned out, of the 16 CPU batch...two were flawed. No success was derived from replacing power supplies, HDD, resetting memory and the cooling solution. The CPU flaw only manifested itself (at first) after several hours of CPU usage. With each failure, the time duration shortened before the next failure and by the time I figured it out was down to about 2 minutes. The AMD engineer with whom I talked was amazed that such CPUs made it beyond quality control. He also suggested that the vendor may have inadvertently mixed returned (previously fetermined to be flawed processors) with the new ones and sent them out (again) as new. Just for future reference...is there an easy way to determine if a CPU is flawed with 2 weeks of down time and extensive hair extraction???? Tim From gerry.creager at tamu.edu Mon Dec 4 06:17:01 2006 From: gerry.creager at tamu.edu (Gerry Creager N5JXS) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: References: Message-ID: <45742DDD.3090407@tamu.edu> No experience yet but our new IBM P575 installation will use these or something similar. We don't have sufficient residual forced air cooling to deal with 640 P5+ CPUs in our campus data center and our plans for dedicated HPC space on campus are still embryonic. I'll relate our experiences. WRT noise load, the "cost" of another Leibert is trivial compared to the case fans we'll still have to endure on the new system. I've heard numbers in excess of 87 dBa for this system, necessitating hearing protection now for anyone entering the data center space. gerry Bruce Allen wrote: > Dear Beowulf list, > > For my next cluster room, I am hoping to use 'liquid cooled' racks make > by Knurr (CoolTherm, http://www.thermalmanagement.de/). The scale is 67 > racks x 7.5kW heat removal per rack (36 U usable per rack). > > Does anyone on the list have experiences with these or similar racks > (you can reply to me privately if you don't want to share your > experiences publicly)? I am told that there are lots of installations > in Europe, especially in Germany, but that the only US sites are at Penn > State, somewhere in the Chicago area, and U. of Vermont, and that these > US installations are fairly small-scale. > > The racks have front and back doors that close, and contains fans in the > back which circulate air through a heat exchanger located in the bottom > of the rack. The heat exchanger transfers the heat into chilled water. > > The advantages of this are that it is quiet, and that you don't need > vertical height for underfloor ducting or overhead hot air removal. > Disadvantages are cost, potential difficulty of working within the rack. > and loss of one rack of cooling capacity if you open both the front and > back doors. > > (In this case I have access to 'building' funds that can not be used to > buy more cpus, so the cost issue is not important.) > > I would be very interested to hear about people's experiences with these > racks (or similar ones from another manufacturer) in an HPC environment. > > Cheers, > Bruce > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf -- Gerry Creager -- gerry.creager@tamu.edu Texas Mesonet -- AATLT, Texas A&M University Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983 Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843 From diep at xs4all.nl Mon Dec 4 08:57:50 2006 From: diep at xs4all.nl (Vincent Diepeveen) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off References: <45578E91.1080407@tcg-hsv.com><16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> <45743B96.10003@tcg-hsv.com> Message-ID: <002901c717c5$578bbbc0$9600000a@gourmandises> To let nodes fail quickly, go find BIG primes with latest GMP nonstop at the opterons. After a few days of nonstop calculation bad nodes should get total hung, when you hit worst case path. most use prime95 for such stuff, but that thing is using exclusively "intel" assembly SIMD code and the cpu's seem quite well tested/fixed for SIMD. Seems to me that a mix of integer multiplication and modulo there is worst case path of those opteron chips. It is of course also possible it hits a bug in the linux kernel, because my own code doesn't get hung at all when running at 4 cores, whereas GMP did do exactly that. My code currently works fastest at windows, as it doesn't have inline assembly for linux yet. I wouldn't rule out that linux kernel simply has bugs there. The testing of those kernels is total amateuristic. Vincent ----- Original Message ----- From: "Tim Moore" To: Sent: Monday, December 04, 2006 4:15 PM Subject: Re: [Beowulf] Node Drop-Off > Update to node drop-off: > > I wrote a few weeks ago to ask about node drop-off. A quick note...I > had a cluster run for 3 years without failure and I upgraded the Opteron > 240 CPUs to 250s. The upgrade required a BIOS upgrade and while I was > at it, upgraded the OS and security. Some readers provided good > suggestions for diagnosis. As it turned out, of the 16 CPU batch...two > were flawed. No success was derived from replacing power supplies, HDD, > resetting memory and the cooling solution. The CPU flaw only manifested > itself (at first) after several hours of CPU usage. With each failure, > the time duration shortened before the next failure and by the time I > figured it out was down to about 2 minutes. > > The AMD engineer with whom I talked was amazed that such CPUs made it > beyond quality control. He also suggested that the vendor may have > inadvertently mixed returned (previously fetermined to be flawed > processors) with the new ones and sent them out (again) as new. > > Just for future reference...is there an easy way to determine if a CPU > is flawed with 2 weeks of down time and extensive hair extraction???? > > Tim > > -------------------------------------------------------------------------------- > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > From hahn at physics.mcmaster.ca Mon Dec 4 09:57:26 2006 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: References: Message-ID: > For my next cluster room, I am hoping to use 'liquid cooled' racks make by > Knurr (CoolTherm, http://www.thermalmanagement.de/). The scale is 67 racks x > 7.5kW heat removal per rack (36 U usable per rack). 7.5 KW/rack isn't much; are you designing low-power nodes? > The racks have front and back doors that close, and contains fans in the back > which circulate air through a heat exchanger located in the bottom of the > rack. The heat exchanger transfers the heat into chilled water. APC has something similar with the cooling on the side. > The advantages of this are that it is quiet, and that you don't need vertical > height for underfloor ducting or overhead hot air removal. Disadvantages are > cost, potential difficulty of working within the rack. and loss of one rack > of cooling capacity if you open both the front and back doors. I'm pretty skeptical of the sealed-pod approach, since it seems to multiply the number of parts, create access issues, doesn't seem to actually save on space, etc. I've also been burned by cold water cooling, so to speak (assuming you don't have your own, well-controlled CW plant.) I would definitely consider a normal big-chillers approach _with_ back-of-rack CW boosters (heat exchangers). and I'd definitely consider creative layouts of racks and chillers (for instance, I'm not crazy about the hot/cold-aisle approach - something W-shaped would be better for my ~50-rack machineroom. or even just lining all the hot racks up against the chillers along one wall.) > (In this case I have access to 'building' funds that can not be used to buy > more cpus, so the cost issue is not important.) are you sure you can't just be clever about laying out normal front-to-back racks? possibly with back-of-rack heat-exchangers? regards, mark hahn. From James.P.Lux at jpl.nasa.gov Mon Dec 4 10:09:09 2006 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off In-Reply-To: <45743B96.10003@tcg-hsv.com> References: <45578E91.1080407@tcg-hsv.com> <16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> <45743B96.10003@tcg-hsv.com> Message-ID: <6.2.3.4.2.20061204100214.032b7c30@mail.jpl.nasa.gov> At 07:15 AM 12/4/2006, Tim Moore wrote: >Update to node drop-off: > > >The AMD engineer with whom I talked was amazed that such CPUs made >it beyond quality control. He also suggested that the vendor may >have inadvertently mixed returned (previously fetermined to be >flawed processors) with the new ones and sent them out (again) as new. > >Just for future reference...is there an easy way to determine if a >CPU is flawed with 2 weeks of down time and extensive hair extraction???? This has been a persistent problem in the industry for decades. It's not unheard of for wholesalers to get batches of processors that have been remarked somewhere along the line. They'll run at the speed at room temp, but not over the entire temperature range, or, maybe with some bus loading conditions. Obviously, the mfrs hate it when this occurs, and so, they've been doing things like making the speed grade readable from some built in ROM or as a bond wire option. And, as much as the mfrs hate it, there's always the "fell of the back of the truck on the way to the disposal company" problem.. parts that don't pass the mfr inspection are supposed to be destroyed, but sometimes aren't. Processors are a high dollar item for something quite compact, they're sort of commodity (at least as far as the end user is concerned), so they're ripe for all the fiddles that have been used on such items for millenia. Hey, didn't Archimedes get famous for devising some sort of test along those lines? Jim From rgb at phy.duke.edu Mon Dec 4 10:59:17 2006 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off In-Reply-To: <6.2.3.4.2.20061204100214.032b7c30@mail.jpl.nasa.gov> References: <45578E91.1080407@tcg-hsv.com> <16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> <45743B96.10003@tcg-hsv.com> <6.2.3.4.2.20061204100214.032b7c30@mail.jpl.nasa.gov> Message-ID: On Mon, 4 Dec 2006, Jim Lux wrote: > Processors are a high dollar item for something quite compact, they're sort > of commodity (at least as far as the end user is concerned), so they're ripe > for all the fiddles that have been used on such items for millenia. Hey, > didn't Archimedes get famous for devising some sort of test along those > lines? Eureka! So he did! Are you suggesting that he take his processors into the bath with him so they can talk to his rubber duckies:-) or that he run naked through the streets in front of the main AMD office in protest? The latter would probably work better than the former at least if >>I<< were doing the running. Any toplevel AMD exec would do anything to crank up quality control rather than have to endure the sight of an old fat bald guy running around naked out front screaming about defective processors...;-) rgb -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From i.kozin at dl.ac.uk Mon Dec 4 11:26:21 2006 From: i.kozin at dl.ac.uk (Kozin, I (Igor)) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] MEW17 webcast Message-ID: If things go as planned we will be webcasting our 17th Machine Evaluation Workshop tomorrow 5th December and 6th December. As usual the presentations will be made available from our web site but hopefully this time with recorded video stream as well. Igor I. Kozin (i.kozin at dl.ac.uk) CCLRC Daresbury Laboratory, WA4 4AD, UK skype: in_kozin tel: +44 (0) 1925 603308 http://www.cse.clrc.ac.uk/disco From ballen at gravity.phys.uwm.edu Mon Dec 4 11:29:04 2006 From: ballen at gravity.phys.uwm.edu (Bruce Allen) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: References: Message-ID: Hi Mark, >> For my next cluster room, I am hoping to use 'liquid cooled' racks make by >> Knurr (CoolTherm, http://www.thermalmanagement.de/). The scale is 67 racks >> x 7.5kW heat removal per rack (36 U usable per rack). > > 7.5 KW/rack isn't much; are you designing low-power nodes? Yup -- I guess so. For example our current cluster nodes are dual core Opteron 175s (Supermicro H8SSL-i motherboards). They cost about $1200 for 2 x 2.2 GHz, with 1GB per core, and use 180 W under load (per 1U). I will have enough rack space (67 x 34 U, per 500kW power and cooling) that if I want to use nodes that dissipate 300 W per node then I will simply limit myself to 25 nodes per rack. >> The racks have front and back doors that close, and contains fans in the >> back which circulate air through a heat exchanger located in the bottom of >> the rack. The heat exchanger transfers the heat into chilled water. > > APC has something similar with the cooling on the side. Can you give a positive or negative opinion about either the Knurr or APC racks? Have you used them yourself in a system? >> The advantages of this are that it is quiet, and that you don't need >> vertical height for underfloor ducting or overhead hot air removal. >> Disadvantages are cost, potential difficulty of working within the rack. >> and loss of one rack of cooling capacity if you open both the front and >> back doors. > > I'm pretty skeptical of the sealed-pod approach, since it seems to > multiply the number of parts, create access issues, doesn't seem to > actually save on space, etc. It does save on vertical space, and it reduces noise to the level where you can work comfortably in the room. > I've also been burned by cold water cooling, so to speak (assuming you > don't have your own, well-controlled CW plant.) I'll have my own self-contained plant, designed by German engineers. They seem to know their stuff. > I would definitely consider a normal big-chillers approach _with_ > back-of-rack CW boosters (heat exchangers). Can you provide a URL or recommendation for these back-of-rack CW boosters? > and I'd definitely consider creative layouts of racks and chillers (for > instance, I'm not crazy about the hot/cold-aisle approach - something > W-shaped would be better for my ~50-rack machineroom. or even just > lining all the hot racks up against the chillers along one wall.) > >> (In this case I have access to 'building' funds that can not be used to buy >> more cpus, so the cost issue is not important.) > > are you sure you can't just be clever about laying out normal front-to-back > racks? possibly with back-of-rack heat-exchangers? I'd like a URL for the back-of-rack heat exchangers, please! (PS: though that approach still sounds noisy!) Cheers, Bruce From jmdavis1 at vcu.edu Mon Dec 4 11:55:41 2006 From: jmdavis1 at vcu.edu (Mike Davis) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: References: Message-ID: <45747D3D.4040308@vcu.edu> Bruce, I am investigating the APC Infrastructure and Liebert XDH systems for a new small (1000 sqft). The benefits seem to outweigh the potential downsides. They include relatively easy n+1 capacity, tightly coupled cooling for efficiency, and containment to prevent air mixing. Virginia Tech has been using the Liebert XDV units as supplemental cooling for System X since it was installed. I believe that other top 100 installations are planned that will use the XDH. Infrastuxture seems to be good on a price for performance basis because the units are modular and generic. Mike Davis From hahn at physics.mcmaster.ca Mon Dec 4 12:28:38 2006 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: References: Message-ID: >> 7.5 KW/rack isn't much; are you designing low-power nodes? > > Yup -- I guess so. For example our current cluster nodes are dual core > Opteron 175s (Supermicro H8SSL-i motherboards). They cost about $1200 for 2 > x 2.2 GHz, with 1GB per core, and use 180 W under load (per 1U). nice. for loosely-coupled workloads, that's a smart design, though I'm curious: did you consider and reject something from the Core2 world? (it's excellent in-cache FP throughput would also match loose-coupled.) >> APC has something similar with the cooling on the side. > > Can you give a positive or negative opinion about either the Knurr or APC > racks? Have you used them yourself in a system? I'm afraid not. if your heart is set on this approach, my main suggestion is to carefully consider the capacity and quality of your CW supply. for instance, if your CW loop is run by campus people and is shared with human-space cooling, you're probably in trouble. >> I'm pretty skeptical of the sealed-pod approach, since it seems to multiply >> the number of parts, create access issues, doesn't seem to actually save on >> space, etc. > > It does save on vertical space, and it reduces noise to the level where you > can work comfortably in the room. well, I can't speak to your room's limitations, but the Knurr stuff does appear to consume some vertical space on its own. I can't tell whether it's also significantly deeper (to allow for hot and cold airhandling internale plenums (plena?) my experience with system noise is that it's very dependent on the systems themselves. the chillers are not particularly noisy, but some systems are 80dB all the time; others are 65-70 when cool and 85 when warm. >> I would definitely consider a normal big-chillers approach _with_ >> back-of-rack CW boosters (heat exchangers). > > Can you provide a URL or recommendation for these back-of-rack CW boosters? um, well, SGI, HP and IBM all seem to source them from somewhere, not sure. I think it makes more sense than the liebert XDR approach (over-the-top boosters). > (PS: though that approach still sounds noisy!) why? the main noise source is the fans in the server, not the chillers. and I think the back-door coolers are normally passive. the "eServer Rear Door Heat Exchanger" appears to be. HP's thing: http://h18004.www1.hp.com/products/servers/proliantstorage/racks/mcs/index.html is like a rack with a half-rack of fans+heat-ex next to it. APC has a similar half-rack slice that can be slotted anywhere (it just cools back-to-front). http://www-03.ibm.com/press/us/en/pressrelease/20343.wss --- if you need a quiet room, well OK, but it sounds more like putting computers in office space, rather than a machineroom. (I don't find there's much reason to be in machinerooms any more.) I hate to sound like an ass, but I'm pretty skeptical of the vertical-space argument as well, since you lose some vertical space inside the rack with Knurr, and normal open-concept rooms _don't_ actually consume much vertical space. for instance, consider a room with three normal liebert downdraft units, a 16" underfloor plenum and all the racks arranged in a row with their hot side facing the chillers. that would work awesomely, and would probably work with a total of 8 ft. I'd worry more about optimizing the layout of the racks, inside and out (for instance, eth and power breakers consume 4u out of my compute racks; further, I have 11 racks of incredibly low-dissipation quadrics switches, which should really be taken out of the airflow, since they're <2KW/rack.) From James.P.Lux at jpl.nasa.gov Mon Dec 4 14:16:48 2006 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off In-Reply-To: References: <45578E91.1080407@tcg-hsv.com> <16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> <45743B96.10003@tcg-hsv.com> <6.2.3.4.2.20061204100214.032b7c30@mail.jpl.nasa.gov> Message-ID: <6.2.3.4.2.20061204141006.03296550@mail.jpl.nasa.gov> At 10:59 AM 12/4/2006, Robert G. Brown wrote: >On Mon, 4 Dec 2006, Jim Lux wrote: > >>Processors are a high dollar item for something quite compact, >>they're sort of commodity (at least as far as the end user is >>concerned), so they're ripe for all the fiddles that have been used >>on such items for millenia. Hey, didn't Archimedes get famous for >>devising some sort of test along those lines? > >Eureka! So he did! > >Are you suggesting that he take his processors into the bath with him so >they can talk to his rubber duckies:-) No.. you have that confused with the liquid cooling thread. >or that he run naked through the >streets in front of the main AMD office in protest? I suppose so.. I hadn't thought about that aspect of Archimedes's "work". I was more thinking of the detection of fraudulent jewelry. >The latter would probably work better than the former at least if >>I<< >were doing the running. Any toplevel AMD exec would do anything to >crank up quality control rather than have to endure the sight of an old >fat bald guy running around naked out front screaming about defective >processors...;-) They're in Silcon Valley in Northern California, aren't they. While it might not be as "counter culture" as downtown San Francisco a few miles away, or as mellow as Marin county a few miles farther north or Santa Cruz to the west, you might not get as much attention as you might think. They might just think that you're just another one of those eccentric cluster monkeys (the "naked ape" revisited).. At least the weather is usually temperate enough that you wouldn't die instantly of hypothermia or sun stroke. James Lux, P.E. Spacecraft Radio Frequency Subsystems Group Flight Communications Systems Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 From rgb at phy.duke.edu Mon Dec 4 16:01:52 2006 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off In-Reply-To: <6.2.3.4.2.20061204141006.03296550@mail.jpl.nasa.gov> References: <45578E91.1080407@tcg-hsv.com> <16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> <45743B96.10003@tcg-hsv.com> <6.2.3.4.2.20061204100214.032b7c30@mail.jpl.nasa.gov> <6.2.3.4.2.20061204141006.03296550@mail.jpl.nasa.gov> Message-ID: On Mon, 4 Dec 2006, Jim Lux wrote: > At 10:59 AM 12/4/2006, Robert G. Brown wrote: >> On Mon, 4 Dec 2006, Jim Lux wrote: >> >>> Processors are a high dollar item for something quite compact, they're >>> sort of commodity (at least as far as the end user is concerned), so >>> they're ripe for all the fiddles that have been used on such items for >>> millenia. Hey, didn't Archimedes get famous for devising some sort of >>> test along those lines? >> >> Eureka! So he did! >> >> Are you suggesting that he take his processors into the bath with him so >> they can talk to his rubber duckies:-) > > No.. you have that confused with the liquid cooling thread. Good thinking! Another benefit! He can measure their density and cool them at the same time! > They're in Silcon Valley in Northern California, aren't they. While it might > not be as "counter culture" as downtown San Francisco a few miles away, or as > mellow as Marin county a few miles farther north or Santa Cruz to the west, > you might not get as much attention as you might think. They might just > think that you're just another one of those eccentric cluster monkeys (the > "naked ape" revisited).. At least the weather is usually temperate enough > that you wouldn't die instantly of hypothermia or sun stroke. Oh yeah, forgot about that. There are plenty of old bald fat naked guys on the streets and beaches of CA, especially near SV. Well, that does it. Out of ideas. Guess one has to fall back on the old standbys, pulling one's hair out by the roots, cursing, drinking heavily, sacrificing chickens, running 48 hours of large-prime-number generation, or (my personal favorite) buying from a tier 1 vendor who also sells you 3 years of onsite service so you pick up a phone and say in delicate tones "Gee, these three nodes crash as soon as they start to actually work -- fix them. Now." This is really the basic difference between tier 1 and tier 2. You can save short term money with the latter, but have to do things like just plain throw out hardware -- after sweating over it for a long time, nagging your tier 2 vendor, getting angry, losing a lot of productivity and time. For some projects that works -- for others it doesn't. Note that I'm not asserting any sort of TCO bullshit advantage to one or the other -- as long as you spend some of the money you save on throwaway and replacement you can minimize losses as long as you aren't TOO unlucky you can do fine with tier 2, but you do need to recognize up front the trade-offs of the decision and deal with the additional hassles with a sigh if not a smile...;-) rgb > > > James Lux, P.E. > Spacecraft Radio Frequency Subsystems Group > Flight Communications Systems Section > Jet Propulsion Laboratory, Mail Stop 161-213 > 4800 Oak Grove Drive > Pasadena CA 91109 > tel: (818)354-2075 > fax: (818)393-6875 > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From hahn at physics.mcmaster.ca Mon Dec 4 16:59:03 2006 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off In-Reply-To: References: <45578E91.1080407@tcg-hsv.com> <16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> <45743B96.10003@tcg-hsv.com> <6.2.3.4.2.20061204100214.032b7c30@mail.jpl.nasa.gov> Message-ID: > were doing the running. Any toplevel AMD exec would do anything to > crank up quality control rather than have to endure the sight of an old AMD and our system vendor (HP) did a good job replacing >1500 opteron 252's in my cluster this year. this was the result of a "test escape", which did eventually show up publicly on amd.com. it's important to choose your vendor carefully, for reasons like this. regards, mark hahn. (not all of (old,fat,bald), and living in a no-streak-in-december climate) From jlb17 at duke.edu Tue Dec 5 04:03:22 2006 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off In-Reply-To: References: <45578E91.1080407@tcg-hsv.com> <16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> <45743B96.10003@tcg-hsv.com> <6.2.3.4.2.20061204100214.032b7c30@mail.jpl.nasa.gov> <6.2.3.4.2.20061204141006.03296550@mail.jpl.nasa.gov> Message-ID: On Mon, 4 Dec 2006 at 7:01pm, Robert G. Brown wrote > This is really the basic difference between tier 1 and tier 2. You can > save short term money with the latter, but have to do things like just > plain throw out hardware -- after sweating over it for a long time, > nagging your tier 2 vendor, getting angry, losing a lot of productivity > and time. For some projects that works -- for others it doesn't. I think you're painting with too broad a brush there. I find the Tier 2 I buy from to be *far* more helpful and proactive than the dominant Tier 1 here on campus. I can relate a number of stories about them finding spare parts for systems long out of warranty, upgrading components when it's the quicker rather than the cheaper way to fix a problem, replacing shipper damaged systems within days, etc. In short, the sort of personal, helpful service I've never seen from a Tier 1. And it's not like I'm a major customer, either -- they supply some *big* operations. Sure, there are many less than good Tier 1s out there, so caveat beowulfer. But you can some who, IMHO, outperform the big boys considerably, and not just on price. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University From amjad11 at gmail.com Mon Dec 4 02:30:01 2006 From: amjad11 at gmail.com (amjad ali) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] More technical information and spec of beowulf In-Reply-To: <200612041015.06679.csamuel@vpac.org> References: <200612041015.06679.csamuel@vpac.org> Message-ID: <428810f20612040230s16d1f9b4m9284e8c129105bcf@mail.gmail.com> > > How can i find some more detailed technical information on Beowulf > > software infrastructure? > > Hopefully these will help.. > > http://www.phy.duke.edu/~rgb/Beowulf/beowulf_book/beowulf_book/ > > http://en.wikipedia.org/wiki/Beowulf_(computing) > > http://clustermonkey.net/ > > Good luck! > Chris Might start with the following as well: () http://cs.boisestate.edu/~amit/research/beowulf/beowulf-setup.pdf () Building a Beowulf System Jan Lindheim, Caltech () "A survey of open source cluster management systems" by StoneLion at http://www.linux.com/ () http://oscar.openclustergroup.org () Docs availaible at the website of ASPEN SYSTEMS () "How to Build a Beowulf Linux Cluster" at teh website of The Mississippi Center for Supercomputing Research () "Configuring a Beowulf Cluster" by Forrest Hoffman, Extreme Linux regards, AMJAD ALI, BZU, Multan. From merc4krugger at gmail.com Mon Dec 4 03:54:13 2006 From: merc4krugger at gmail.com (Krugger) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] A suitable motherboard for newbie In-Reply-To: <428810f20611280325j44e500cdm7c9e91c6abd57899@mail.gmail.com> References: <1bef2ce30611260352r33e8d1aia809f75280e1ca07@mail.gmail.com> <428810f20611280325j44e500cdm7c9e91c6abd57899@mail.gmail.com> Message-ID: I would like to contribute that going quad core is a really bad idea at the moment, as linux support for the new boards is not very good at the moment. Even some Core Duo 2 ready boards have trouble, either it is the SATA controller or some other new stuff that they put in it. So before you buy check your hardware against known hardware problems in google: like some Tyan models that happen to have erratic behavior and need BIOS upgrades or SATA controller that don't work or RAID boards that break under heavy load. This is especially true if your are buying in a small amount and don't have a maintainance contract with a supplier. Now to get the most out of your money you should be really considering how much you get for your money. For example going rack mounted is only an option if you intent to expand and have a cooled room. Another thing is not to buy the latest tecnology available but go for the reliable and cheaper computers. Why? Because you get more computers which in the end will get you more overall computing power. For example for 20.000 dolares you can get almost twice as many opterons if you had dropped the dual core option, which means twice the total memory, which allows you to run bigger simulations. Basically you get the same number of tasks, 5(machines) * 2(cpus) * 2(cores) = 10(machines) * 2(cpus), but you get more memory 20Gb RAM compared to 40Gb. This considering if you had chosen the top of the line dual core opteron, which costed three times the cost of top of the line cpu without dual core, which really isn't considered top of the line. From eric-shook at uiowa.edu Mon Dec 4 07:13:38 2006 From: eric-shook at uiowa.edu (Eric Shook) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: References: Message-ID: <45743B22.5090908@uiowa.edu> Hi Bruce, Our University is also looking into these racks. We have also looked at other vendors with similar liquid cooling and something called "Spraycool" technology (limited in deployment) among others. I would also be interested in the information you collect on or off the list and would be willing to share some information. Thanks, Eric -- Eric Shook (319) 335-6714 Technical Lead, Systems and Operations - GROW http://grow.uiowa.edu Bruce Allen wrote: > Dear Beowulf list, > > For my next cluster room, I am hoping to use 'liquid cooled' racks make > by Knurr (CoolTherm, http://www.thermalmanagement.de/). The scale is 67 > racks x 7.5kW heat removal per rack (36 U usable per rack). > > Does anyone on the list have experiences with these or similar racks > (you can reply to me privately if you don't want to share your > experiences publicly)? I am told that there are lots of installations > in Europe, especially in Germany, but that the only US sites are at Penn > State, somewhere in the Chicago area, and U. of Vermont, and that these > US installations are fairly small-scale. > > The racks have front and back doors that close, and contains fans in the > back which circulate air through a heat exchanger located in the bottom > of the rack. The heat exchanger transfers the heat into chilled water. > > The advantages of this are that it is quiet, and that you don't need > vertical height for underfloor ducting or overhead hot air removal. > Disadvantages are cost, potential difficulty of working within the rack. > and loss of one rack of cooling capacity if you open both the front and > back doors. > > (In this case I have access to 'building' funds that can not be used to > buy more cpus, so the cost issue is not important.) > > I would be very interested to hear about people's experiences with these > racks (or similar ones from another manufacturer) in an HPC environment. > > Cheers, > Bruce > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From ruhollah.mb at gmail.com Mon Dec 4 23:34:14 2006 From: ruhollah.mb at gmail.com (Ruhollah Moussavi Baygi ) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] SATA II Message-ID: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> Hi All at Beowulf ! There are some questions about implementation of a Beowulf cluster: 1-Regarding OS, is "Fedora Core 64bit" a good option for AMD Athlon 64 X2 4200+? 2- Is SATA II HDD compatible with Fedora Core 64bit? 3- Concerning RAM, is "2 GB 800 MHz DDR2" sufficient? Any other suggestion? Thanks in Advance -- Best, Ruhollah Moussavi B. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20061205/6bdcbe1b/attachment.html From ruhollah.mb at gmail.com Sun Dec 3 08:13:57 2006 From: ruhollah.mb at gmail.com (Ruhollah Moussavi Baygi ) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] More technical information and spec of beowulf In-Reply-To: References: Message-ID: <1bef2ce30612030813j1d8c2cb6r9c9c49f8db617683@mail.gmail.com> Hi Reza! What do u mean exactly by "Beowulf software infrastructure" ? u know Beowulf is just a manner of connecting commonplace computers (nodes) together to make a robust system. Then your computational job will be divided among nodes. As far as I know, there is no especial software for this framework. All you need is a message passing interface (MPI) which makes it possible for nodes to pass messages together while running job. There are a lot of scientific packages (and mainly open source) which have been developed in such a way that you can run in parallel, like GROMACS, DLPOLY, AMBER, WIEN2K, and many others. All these packages (at least their computational parts), have been compiled by FORTRAN or C/C++ .This is due to the capability of these two compilers which can be run in parallel and work with MPI. However, there are some engineering and finite element software which are absolutely commercial and capable of being run in parallel, among them are FLUENT, CFX, and so on. I hope this could be helpful for you, Ruhollah Moussavi B. On 11/30/06, reza bakhshi wrote: > > Hi there, > How can i find some more detailed technical information on Beowulf > software infrastructure? > Thank you :) > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > -- Best, Ruhollah Moussavi B. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20061203/a17a6d55/attachment.html From jlb17 at duke.edu Tue Dec 5 06:01:53 2006 From: jlb17 at duke.edu (Joshua Baker-LePain) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] Node Drop-Off In-Reply-To: References: <45578E91.1080407@tcg-hsv.com> <16aa0e180611121340u1ad13c2ch892808b653e7abb@mail.gmail.com> <45743B96.10003@tcg-hsv.com> <6.2.3.4.2.20061204100214.032b7c30@mail.jpl.nasa.gov> <6.2.3.4.2.20061204141006.03296550@mail.jpl.nasa.gov> Message-ID: On Tue, 5 Dec 2006 at 7:03am, Joshua Baker-LePain wrote > Sure, there are many less than good Tier 1s out there, so caveat ^^ *sigh* That should read 'Tier 2s', of course. That'll teach me to post before coffee. > beowulfer. But you can some who, IMHO, outperform the big boys > considerably, and not just on price. > > -- Joshua Baker-LePain Department of Biomedical Engineering Duke University From cap at nsc.liu.se Tue Dec 5 08:48:42 2006 From: cap at nsc.liu.se (Peter Kjellstrom) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] SATA II In-Reply-To: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> Message-ID: <200612051748.46469.cap@nsc.liu.se> On Tuesday 05 December 2006 08:34, Ruhollah Moussavi Baygi wrote: > Hi All at Beowulf ! > > There are some questions about implementation of a Beowulf cluster: > > 1-Regarding OS, is "Fedora Core 64bit" a good option for AMD Athlon 64 X2 > 4200+? You'll have to upgrade to a later fedora core after a while when support runs out, but pretty much a matter of taste. (technically it'll be as good as any) > 2- Is SATA II HDD compatible with Fedora Core 64bit? The sata and/or raid controller will be a lot more critical to your linux experience than any choice of HDD. > 3- Concerning RAM, is "2 GB 800 MHz DDR2" sufficient? That depends close to 100% on what you'll run on it. /Peter -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20061205/6a4c6709/attachment.bin From amacater at galactic.demon.co.uk Tue Dec 5 08:49:43 2006 From: amacater at galactic.demon.co.uk (Andrew M.A. Cater) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <45743B22.5090908@uiowa.edu> References: <45743B22.5090908@uiowa.edu> Message-ID: <20061205164943.GA2722@galactic.demon.co.uk> On Mon, Dec 04, 2006 at 09:13:38AM -0600, Eric Shook wrote: > Hi Bruce, > > Our University is also looking into these racks. We have also looked at > other vendors with similar liquid cooling and something called > "Spraycool" technology (limited in deployment) among others. I would > also be interested in the information you collect on or off the list and > would be willing to share some information. > The only things I know that used spraycool technology were big machines like Crays and ?? Thinking Machines ?? which dunked circuit boards in freon and sprayed the liquid to keep it moving. Surely they can't have revived that :) AndyC From hahn at physics.mcmaster.ca Tue Dec 5 09:07:36 2006 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] SATA II In-Reply-To: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> Message-ID: > 1-Regarding OS, is "Fedora Core 64bit" a good option for AMD Athlon 64 X2 > 4200+? sure. distros are just desktop decoration, and anything recent will perform equally well. you do probably want 64b, but that's not rare. > 2- Is SATA II HDD compatible with Fedora Core 64bit? disks don't have compatibility - controllers do. so it depends on your motherboard choice. but I haven't seen any builtin controllers that don't work well. > 3- Concerning RAM, is "2 GB 800 MHz DDR2" sufficient? any general answer is wrong. 2G is a huge waste of money for some applications, and not nearly enough for others. you'll pay a noticable premium for ddr2/800 as opposed to ddr2/667, though, and might not notice the difference (again, depending on your workload). there is only one general correlations I'll draw: many, many loosely-coupled and/or serial jobs have tiny memory footprints (so 2GB is overkill, and since the cache is effective, higher bandwidth is wasted). it's hard to say anything useful about tighter-than-loose parallel jobs, since memory size/intensiveness varies a lot - moreso than for loose/serial, at least in my experience... regards, mark hahn. From rbw at ahpcrc.org Tue Dec 5 10:11:07 2006 From: rbw at ahpcrc.org (Richard Walsh) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <20061205164943.GA2722@galactic.demon.co.uk> References: <45743B22.5090908@uiowa.edu> <20061205164943.GA2722@galactic.demon.co.uk> Message-ID: <4575B63B.5080709@ahpcrc.org> Andrew M.A. Cater wrote: > On Mon, Dec 04, 2006 at 09:13:38AM -0600, Eric Shook wrote: > >> Hi Bruce, >> >> Our University is also looking into these racks. We have also looked at >> other vendors with similar liquid cooling and something called >> "Spraycool" technology (limited in deployment) among others. I would >> also be interested in the information you collect on or off the list and >> would be willing to share some information. >> >> > The only things I know that used spraycool technology were big machines > like Crays and ?? Thinking Machines ?? which dunked circuit boards in > freon and sprayed the liquid to keep it moving. Surely they can't have > revived that :) > > AndyC > All, In case there is interest ... A "spray-cool" technology is the approach taken currently in the Cray X1. The liquid is a high-heat capacity fluoro-carbon, fairly inert, very expensive, but nasty if it starts to burn. One of the key innovations on the Cray X1 is that the circuits are "on the ceiling" so to speak and sprayed from below. The fluid is gravity collected and cycled up again. A "spray-cooled" rack would seem to require hermetic enclosure at some level if the same liquid were used. rbw -- Richard B. Walsh "The world is given to me only once, not one existing and one perceived. The subject and object are but one." Erwin Schroedinger Project Manager Network Computing Services, Inc. Army High Performance Computing Research Center (AHPCRC) rbw@ahpcrc.org | 612.337.3467 ----------------------------------------------------------------------- This message (including any attachments) may contain proprietary or privileged information, the use and disclosure of which is legally restricted. If you have received this message in error please notify the sender by reply message, do not otherwise distribute it, and delete this message, with all of its contents, from your files. ----------------------------------------------------------------------- From amjad11 at gmail.com Tue Dec 5 06:27:41 2006 From: amjad11 at gmail.com (amjad ali) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] More technical information and spec of beowulf In-Reply-To: <1bef2ce30612030813j1d8c2cb6r9c9c49f8db617683@mail.gmail.com> References: <1bef2ce30612030813j1d8c2cb6r9c9c49f8db617683@mail.gmail.com> Message-ID: <428810f20612050627v629d284fy84a86eeb459ce331@mail.gmail.com> Moussavi, you should also have a look at the links given in the email from Chris Samuel and Me (for better undestanding). Specially the following, "A survey of open source cluster management systems" by StoneLion at http://www.linux.com/ http://oscar.openclustergroup.org CBeST from PSSC labs. MOAB Scali-Manage Scyld Software. regards, AMJAD ALI. From ajitm at cdac.in Tue Dec 5 08:46:18 2006 From: ajitm at cdac.in (ajit mote) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] mpi cleanup of tasks Message-ID: <1165337178.27963.10.camel@sysadm.stp.cdac.ernet.in> Hi, I was trying to write a script that cleans up orphaned mpi tasks.If u can actually mail me a mpi-cleanup script (and install instructions) that cleans all orphaned mpi processors ,it will save me a lot of trouble writing the script. And I will be indebted to you for life. Thanking you. From amacater at galactic.demon.co.uk Tue Dec 5 12:24:31 2006 From: amacater at galactic.demon.co.uk (Andrew M.A. Cater) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] SATA II In-Reply-To: References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> Message-ID: <20061205202431.GA3520@galactic.demon.co.uk> On Tue, Dec 05, 2006 at 12:07:36PM -0500, Mark Hahn wrote: > >1-Regarding OS, is "Fedora Core 64bit" a good option for AMD Athlon 64 X2 > >4200+? > > sure. distros are just desktop decoration, and anything recent will > perform equally well. you do probably want 64b, but that's not rare. > Think carefully about what it is you are looking to do. Fedora is relatively "bleeding edge" and has a very short lifetime - typically a year or 18 months of support. If you are looking to build a large cluster, you may want to evaluate something with a longer lifecycle / more stability. I will always recommend Debian for this: there is a huge amount of software ready packaged but, more importantly, the minimal install really is minimal if that is what you want. Debian Etch for AMD64 is stable enough for me: it is the current testing release but should be released as stable Real Soon Now (it was expected yesterday - it may be released as stable by the end of December). > >2- Is SATA II HDD compatible with Fedora Core 64bit? > > disks don't have compatibility - controllers do. so it depends on your > motherboard choice. but I haven't seen any builtin controllers that > don't work well. > How long is a piece of string? As Mark says, it depends on the motherboard controller. Brand new boards may need brand new kernels to support the chipset. > >3- Concerning RAM, is "2 GB 800 MHz DDR2" sufficient? > > any general answer is wrong. 2G is a huge waste of money for some > applications, and not nearly enough for others. you'll pay a noticable > premium for ddr2/800 as opposed to ddr2/667, though, and might not > notice the difference (again, depending on your workload). > Is interconnect speed a limiting factor or is memory access speed most important? Build a toy Beowulf for evaluation from four machines and run a subset of your work on it. Where are you sourcing your memory from and what premium are you paying if you want to take the cluster down to improve memory on the nodes later? Power,environment, cooling and heat wise, memory is cheaper and more reliable than spinning hard disks. > there is only one general correlations I'll draw: many, many > loosely-coupled and/or serial jobs have tiny memory footprints (so 2GB is > overkill, and since > the cache is effective, higher bandwidth is wasted). > > it's hard to say anything useful about tighter-than-loose parallel jobs, > since memory size/intensiveness varies a lot - moreso than for loose/serial, > at least in my experience... > > regards, mark hahn. > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf From john.hearns at streamline-computing.com Tue Dec 5 14:45:13 2006 From: john.hearns at streamline-computing.com (John Hearns) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] SATA II In-Reply-To: <20061205202431.GA3520@galactic.demon.co.uk> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <20061205202431.GA3520@galactic.demon.co.uk> Message-ID: <4575F679.8040908@streamline-computing.com> Andrew M.A. Cater wrote: > On Tue, Dec 05, 2006 at 12:07:36PM -0500, Mark Hahn wrote: >>> 1-Regarding OS, is "Fedora Core 64bit" a good option for AMD Athlon 64 X2 >>> 4200+? >> sure. distros are just desktop decoration, and anything recent will >> perform equally well. you do probably want 64b, but that's not rare. >> > Think carefully about what it is you are looking to do. Fedora is > relatively "bleeding edge" and has a very short lifetime - typically a > year or 18 months of support. If you are looking to build a large I agree with what Andrew says. I would give some very serious consideration to SuSE Linux - which has a reputation for better support for 64-bit processors and the latest SATA/SAS controllers. There is the free professional version or SLES. On the Redhat side, give consideration to Redhat Enterprise. Or one of the Redhat-alike recompiles. I'm hinting at Scientific Linux here. Yes, SL 4 may not have the very latest leading edge kernel, but it does have a long suport lifetime. From greg.lindahl at qlogic.com Tue Dec 5 15:34:49 2006 From: greg.lindahl at qlogic.com (Greg Lindahl) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <4575B63B.5080709@ahpcrc.org> References: <45743B22.5090908@uiowa.edu> <20061205164943.GA2722@galactic.demon.co.uk> <4575B63B.5080709@ahpcrc.org> Message-ID: <20061205233449.GA5628@greglaptop> On Tue, Dec 05, 2006 at 12:11:07PM -0600, Richard Walsh wrote: > One of the key innovations on the Cray X1 is that the circuits > are "on the ceiling" > so to speak and sprayed from below. The fluid is gravity collected and > cycled up again. This technology predates the X1 by a while. -- g From rbw at ahpcrc.org Tue Dec 5 15:45:27 2006 From: rbw at ahpcrc.org (Richard Walsh) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <20061205233449.GA5628@greglaptop> References: <45743B22.5090908@uiowa.edu> <20061205164943.GA2722@galactic.demon.co.uk> <4575B63B.5080709@ahpcrc.org> <20061205233449.GA5628@greglaptop> Message-ID: <45760497.3030101@ahpcrc.org> Greg Lindahl wrote: > On Tue, Dec 05, 2006 at 12:11:07PM -0600, Richard Walsh wrote: > > >> One of the key innovations on the Cray X1 is that the circuits >> are "on the ceiling" >> so to speak and sprayed from below. The fluid is gravity collected and >> cycled up again. >> > > This technology predates the X1 by a while. > Mmm ... I did not know that ... in a reasonably successful commercial product (i.e. an innovation, rather than a mere invention)? What was/is/were/are the product(s)? Perhaps outside of HPC ... rbw > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > > -- Richard B. Walsh "The world is given to me only once, not one existing and one perceived. The subject and object are but one." Erwin Schroedinger Project Manager Network Computing Services, Inc. Army High Performance Computing Research Center (AHPCRC) rbw@ahpcrc.org | 612.337.3467 ----------------------------------------------------------------------- This message (including any attachments) may contain proprietary or privileged information, the use and disclosure of which is legally restricted. If you have received this message in error please notify the sender by reply message, do not otherwise distribute it, and delete this message, with all of its contents, from your files. ----------------------------------------------------------------------- From eric-shook at uiowa.edu Tue Dec 5 13:09:51 2006 From: eric-shook at uiowa.edu (Eric Shook) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <20061205164943.GA2722@galactic.demon.co.uk> References: <45743B22.5090908@uiowa.edu> <20061205164943.GA2722@galactic.demon.co.uk> Message-ID: <4575E01F.8020309@uiowa.edu> It has been revived. I was at their booth at SC06 asking about their technology. The website is http://www.spraycool.com. They offer it for a small set of Tier1 systems (or others for a cost I'm sure) If anyone knows anything I would be interested in feedback as I am interested. Thanks, Eric Andrew M.A. Cater wrote: > On Mon, Dec 04, 2006 at 09:13:38AM -0600, Eric Shook wrote: >> Hi Bruce, >> >> Our University is also looking into these racks. We have also looked at >> other vendors with similar liquid cooling and something called >> "Spraycool" technology (limited in deployment) among others. I would >> also be interested in the information you collect on or off the list and >> would be willing to share some information. >> > The only things I know that used spraycool technology were big machines > like Crays and ?? Thinking Machines ?? which dunked circuit boards in > freon and sprayed the liquid to keep it moving. Surely they can't have > revived that :) > > AndyC > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Eric Shook (319) 335-6714 Technical Lead, Systems and Operations - GROW http://grow.uiowa.edu From James.P.Lux at jpl.nasa.gov Tue Dec 5 19:23:05 2006 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <45760497.3030101@ahpcrc.org> References: <45743B22.5090908@uiowa.edu> <20061205164943.GA2722@galactic.demon.co.uk> <4575B63B.5080709@ahpcrc.org> <20061205233449.GA5628@greglaptop> <45760497.3030101@ahpcrc.org> Message-ID: <6.2.3.4.2.20061205191702.032bbe30@mail.jpl.nasa.gov> At 03:45 PM 12/5/2006, Richard Walsh wrote: >Greg Lindahl wrote: >>On Tue, Dec 05, 2006 at 12:11:07PM -0600, Richard Walsh wrote: >> >> >>>One of the key innovations on the Cray X1 is that the circuits are >>>"on the ceiling" >>>so to speak and sprayed from below. The fluid is gravity >>>collected and cycled up again. >>> >> >>This technology predates the X1 by a while. >> > Mmm ... I did not know that ... in a reasonably successful > commercial product (i.e. an innovation, rather > than a mere invention)? What was/is/were/are the product(s)? >Perhaps outside of HPC ... This has been around for quite a while (decades at least). It was in a Fluorinert brochure back in the mid 80s that I recall. There's also versions with ebullient (boiling) cooling. There might even be high power vacuum tubes cooled this way, although I think they either tend to use a cooling jacket or a boiler, as opposed to spraying). And, of course, it's been mentioned on this list, several years ago, at least, as a potential solution for cooling a sealed box portable cluster that uses off the shelf mobos designed for convection cooling) As far as spray cooling of things goes, that's been around since Newcomen invented his steam engine, predating James Watt. Watt's big advance was realizing that the temperature changes in the engine cycle didn't all have to occur in the same place (i.e. you could keep the cylinder hot, and condense the steam somewhere else). Spray type heat exchangers are also fairly standard devices (a notable example being the "swamp cooler" familiar to those in dry desert climates) So, a decent idea, been around for a while, all it takes to make it a good *business* idea is the convergence of a need and a decent execution that is reliable and works. > James Lux, P.E. Spacecraft Radio Frequency Subsystems Group Flight Communications Systems Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 From daniel.kidger at clearspeed.com Wed Dec 6 04:20:35 2006 From: daniel.kidger at clearspeed.com (Daniel Kidger) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <4575E01F.8020309@uiowa.edu> Message-ID: <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> In spite of many of the slides saying "ISR Propiatory and Confidential" I did find this presentation on the web: http://www.vita.com/cool/pres-2004/1430-tilton.pdf see slide 11 for a photo of what they are doing. Slide 18 implies there early market is for Defence systems - I guess getting rid of hot air on a submarine is a bit tricky? I am at a UK HPC Conference today (so is Greg L for that matter) One of the speakers said he was evaluating Spraycool to retrofit to his existing cluster. So if these guys spray *downwards* on the chip - what is the risk of a blocked tube causing the Flurinert to catch fire? Daniel Dr. Daniel Kidger, Technical Consultant, ClearSpeed Technology plc, Bristol, UK E: daniel.kidger@clearspeed.com T: +44 117 317 2030 M: +44 7738 458742 "Write a wise saying and your name will live forever." - Anonymous. -----Original Message----- From: beowulf-bounces@beowulf.org [mailto:beowulf-bounces@beowulf.org] On Behalf Of Eric Shook Sent: 05 December 2006 21:10 To: Andrew M.A. Cater Cc: beowulf@beowulf.org Subject: Re: [Beowulf] 'liquid cooled' racks It has been revived. I was at their booth at SC06 asking about their technology. The website is http://www.spraycool.com. They offer it for a small set of Tier1 systems (or others for a cost I'm sure) If anyone knows anything I would be interested in feedback as I am interested. Thanks, Eric Andrew M.A. Cater wrote: > On Mon, Dec 04, 2006 at 09:13:38AM -0600, Eric Shook wrote: >> Hi Bruce, >> >> Our University is also looking into these racks. We have also looked at >> other vendors with similar liquid cooling and something called >> "Spraycool" technology (limited in deployment) among others. I would >> also be interested in the information you collect on or off the list and >> would be willing to share some information. >> > The only things I know that used spraycool technology were big machines > like Crays and ?? Thinking Machines ?? which dunked circuit boards in > freon and sprayed the liquid to keep it moving. Surely they can't have > revived that :) > > AndyC > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf -- Eric Shook (319) 335-6714 Technical Lead, Systems and Operations - GROW http://grow.uiowa.edu _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf From cap at nsc.liu.se Wed Dec 6 04:39:41 2006 From: cap at nsc.liu.se (Peter Kjellstrom) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] SATA II In-Reply-To: <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> Message-ID: <200612061339.45032.cap@nsc.liu.se> On Wednesday 06 December 2006 07:11, Ruhollah Moussavi Baygi wrote: > Thanks Peter, > But do you mean that SATA is not a suitable choice for a beowulf cluster? huh? Both me and Mark relied almost identically. Pointing out that the controller (motherboard SATA or add-on raid-controller) is what you have to be careful about, SATA or SATAII drives makes very little difference from a compatibility point of view. That said, SATA is very suitable for a beowulf cluster. /Peter ... > > > 2- Is SATA II HDD compatible with Fedora Core 64bit? > > > > The sata and/or raid controller will be a lot more critical to your linux > > experience than any choice of HDD. -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://www.scyld.com/pipermail/beowulf/attachments/20061206/9ad35122/attachment.bin From landman at scalableinformatics.com Wed Dec 6 04:59:19 2006 From: landman at scalableinformatics.com (Joe Landman) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> References: <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> Message-ID: <4576BEA7.5080008@scalableinformatics.com> Daniel Kidger wrote: > Slide 18 implies there early market is for Defence systems - I guess getting > rid of hot air on a submarine is a bit tricky? Hmmm... I would suggest that the important aspect is the lower noise level associated with heat removal on a sub. A bunch of 40mm fans whining away would make for rather easy tracking of subs... Considering that they are immersed in a huge heat bath of reasonable temperature, the important part of this would be how to couple the heat source effectively to this heat bath, also without lots of noise. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com phone: +1 734 786 8423 fax : +1 734 786 8452 cell : +1 734 612 4615 From ashley at quadrics.com Wed Dec 6 05:05:18 2006 From: ashley at quadrics.com (Ashley Pittman) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> References: <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> Message-ID: <1165410318.2246.20.camel@localhost.localdomain> On Wed, 2006-12-06 at 12:20 +0000, Daniel Kidger wrote: > Slide 18 implies there early market is for Defence systems - I guess getting > rid of hot air on a submarine is a bit tricky? I shouldn't imagine it's to tricky although it probably depends on where you park it, I can't imagine hot liquid would be any different however. Ashley, From greg.lindahl at qlogic.com Tue Dec 5 05:49:46 2006 From: greg.lindahl at qlogic.com (Greg Lindahl) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> References: <4575E01F.8020309@uiowa.edu> <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> Message-ID: <20061205134946.GA1871@greglaptop.dl.ac.uk> On Wed, Dec 06, 2006 at 12:20:35PM -0000, Daniel Kidger wrote: > So if these guys spray *downwards* on the chip - what is the risk of a blocked > tube causing the Flurinert to catch fire? They don't use the same Flurinert that does nasty things when it catches on fire. -- greg From rbw at ahpcrc.org Wed Dec 6 06:07:41 2006 From: rbw at ahpcrc.org (Richard Walsh) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <6.2.3.4.2.20061205191702.032bbe30@mail.jpl.nasa.gov> References: <45743B22.5090908@uiowa.edu> <20061205164943.GA2722@galactic.demon.co.uk> <4575B63B.5080709@ahpcrc.org> <20061205233449.GA5628@greglaptop> <45760497.3030101@ahpcrc.org> <6.2.3.4.2.20061205191702.032bbe30@mail.jpl.nasa.gov> Message-ID: <4576CEAD.2000007@ahpcrc.org> Jim Lux wrote: > At 03:45 PM 12/5/2006, Richard Walsh wrote: >> Greg Lindahl wrote: >>> On Tue, Dec 05, 2006 at 12:11:07PM -0600, Richard Walsh wrote: >>>> One of the key innovations on the Cray X1 is that the circuits are >>>> "on the ceiling" >>>> so to speak and sprayed from below. The fluid is gravity collected >>>> and cycled up again. >>> This technology predates the X1 by a while. >>> >> Mmm ... I did not know that ... in a reasonably successful >> commercial product (i.e. an innovation, rather >> than a mere invention)? What was/is/were/are the product(s)? >> Perhaps outside of HPC ... > > This has been around for quite a while (decades at least). It was in > a Fluorinert brochure back in the mid 80s that I recall. There's also > versions with ebullient (boiling) cooling. There might even be high > power vacuum tubes cooled this way, although I think they either tend > to use a cooling jacket or a boiler, as opposed to spraying). > This is drifting away from the useful, but I was asking what other successful HPC (or computing generally) product has used a spray-cool, gravity collection system the like Cray X1. I am not saying there isn't one, just asking someone to tell what it is. And as a side note, to me innovation implies successful commercial application ... I understand that evaporation is a cooling process ... ;-) ... rbw -- Richard B. Walsh "The world is given to me only once, not one existing and one perceived. The subject and object are but one." Erwin Schroedinger Project Manager Network Computing Services, Inc. Army High Performance Computing Research Center (AHPCRC) rbw@ahpcrc.org | 612.337.3467 ----------------------------------------------------------------------- This message (including any attachments) may contain proprietary or privileged information, the use and disclosure of which is legally restricted. If you have received this message in error please notify the sender by reply message, do not otherwise distribute it, and delete this message, with all of its contents, from your files. ----------------------------------------------------------------------- From James.P.Lux at jpl.nasa.gov Wed Dec 6 06:23:42 2006 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> References: <4575E01F.8020309@uiowa.edu> <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> Message-ID: <6.2.3.4.2.20061206061702.032cd340@mail.jpl.nasa.gov> At 04:20 AM 12/6/2006, Daniel Kidger wrote: >In spite of many of the slides saying "ISR Propiatory and Confidential" Hah.. they've published it on the web, so it's not proprietary and confidential any more. Furthermore, if they really want trade secret protection, they've got to be a bit more careful about how they mark their stuff, or someone who DID steal their stuff could use the following defense: "How was I to know that this really was proprietary, they mark their stuff any old way, and lots of published information is marked as proprietary, even when it isn't, so the markings have no meaning." been there, done that, sat in the depositions. >I did find this presentation on the web: > >http://www.vita.com/cool/pres-2004/1430-tilton.pdf > >see slide 11 for a photo of what they are doing. > >Slide 18 implies there early market is for Defence systems - I guess getting >rid of hot air on a submarine is a bit tricky? And for land and portable apps. Spray cooling in one form or another has been around for quite a while. It's really only useful when you can't afford to design a conduction cooled system (i.e. you absolutely, positively have to use some COTS widget, available in no other form, in sealed box). It's a lot of complexity (pumps, sprayers, fluids, orientation sensistivity) compared to simple things like blowing cold air over the device. >I am at a UK HPC Conference today (so is Greg L for that matter) >One of the speakers said he was evaluating Spraycool to retrofit to his >existing cluster. > >So if these guys spray *downwards* on the chip - what is the risk of a blocked >tube causing the Flurinert to catch fire? Fluorinert is, as the name implies, inert. It doesn't burn. Actually, it's pretty amazing stuff. If you remember the photo from decades ago of the mouse breathing under the surface of a liquid.. that was Flourinert. It IS quite pricey. Back in the 80s it was in the hundred dollars/gallon range. James Lux, P.E. Spacecraft Radio Frequency Subsystems Group Flight Communications Systems Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 From rgb at phy.duke.edu Wed Dec 6 06:46:42 2006 From: rgb at phy.duke.edu (Robert G. Brown) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <1165410318.2246.20.camel@localhost.localdomain> References: <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> <1165410318.2246.20.camel@localhost.localdomain> Message-ID: On Wed, 6 Dec 2006, Ashley Pittman wrote: > On Wed, 2006-12-06 at 12:20 +0000, Daniel Kidger wrote: >> Slide 18 implies there early market is for Defence systems - I guess getting >> rid of hot air on a submarine is a bit tricky? > > I shouldn't imagine it's to tricky although it probably depends on where > you park it, I can't imagine hot liquid would be any different however. Right, the waste heat from cluster nodes is a tiny bit of redirected energy and waste heat from the nuclear reactor. Submarines have always had to deal with thermal regulation in both directions as they generate 2nd-law based waste heat in many locations (including human bodies) and sail through waters that range from tepid to warm in the tropics to quite cold in the arctic (and generally quite cool at cruising depths wherever). It's not like all the OTHER electronics on a submarine don't generate heat. rgb > > Ashley, > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > -- Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rbw at ahpcrc.org Wed Dec 6 07:50:55 2006 From: rbw at ahpcrc.org (Richard Walsh) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <6.2.3.4.2.20061206061702.032cd340@mail.jpl.nasa.gov> References: <4575E01F.8020309@uiowa.edu> <041601c71930$eeeea3d0$7680a8c0@win.clearspeed.com> <6.2.3.4.2.20061206061702.032cd340@mail.jpl.nasa.gov> Message-ID: <4576E6DF.9020701@ahpcrc.org> Jim Lux wrote: > At 04:20 AM 12/6/2006, Daniel Kidger wrote: >> I am at a UK HPC Conference today (so is Greg L for that matter) >> One of the speakers said he was evaluating Spraycool to retrofit to his >> existing cluster. >> >> So if these guys spray *downwards* on the chip - what is the risk of >> a blocked >> tube causing the Flurinert to catch fire? > > Fluorinert is, as the name implies, inert. It doesn't burn. > Actually, it's pretty amazing stuff. If you remember the photo from > decades ago of the mouse breathing under the surface of a liquid.. > that was Flourinert. Non-combustable, but not exactly inert. At temperatures greater than 200 degrees C it begins to decompose yielding an HF aerosol among other things. I believe there was an incident with the old Cray T90 that required a computer room evacuation, but perhaps that was an urban myth. rbw -- Richard B. Walsh "The world is given to me only once, not one existing and one perceived. The subject and object are but one." Erwin Schroedinger Project Manager Network Computing Services, Inc. Army High Performance Computing Research Center (AHPCRC) rbw@ahpcrc.org | 612.337.3467 ----------------------------------------------------------------------- This message (including any attachments) may contain proprietary or privileged information, the use and disclosure of which is legally restricted. If you have received this message in error please notify the sender by reply message, do not otherwise distribute it, and delete this message, with all of its contents, from your files. ----------------------------------------------------------------------- From gdjacobs at gmail.com Wed Dec 6 08:30:56 2006 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] A suitable motherboard for newbie In-Reply-To: References: <1bef2ce30611260352r33e8d1aia809f75280e1ca07@mail.gmail.com> <428810f20611280325j44e500cdm7c9e91c6abd57899@mail.gmail.com> Message-ID: <4576F040.90803@gmail.com> Krugger wrote: > I would like to contribute that going quad core is a really bad idea > at the moment, as linux support for the new boards is not very good at > the moment. Even some Core Duo 2 ready boards have trouble, either it > is the SATA controller or some other new stuff that they put in it. So > before you buy check your hardware against known hardware problems in > google: like some Tyan models that happen to have erratic behavior and > need BIOS upgrades or SATA controller that don't work or RAID boards > that break under heavy load. This is especially true if your are > buying in a small amount and don't have a maintainance contract with a > supplier. > > Now to get the most out of your money you should be really considering > how much you get for your money. For example going rack mounted is > only an option if you intent to expand and have a cooled room. > > Another thing is not to buy the latest tecnology available but go for > the reliable and cheaper computers. Why? Because you get more > computers which in the end will get you more overall computing power. > For example for 20.000 dolares you can get almost twice as many > opterons if you had dropped the dual core option, which means twice > the total memory, which allows you to run bigger simulations. > Basically you get the same number of tasks, 5(machines) * 2(cpus) * > 2(cores) = 10(machines) * 2(cpus), but you get more memory 20Gb RAM > compared to 40Gb. This considering if you had chosen the top of the > line dual core opteron, which costed three times the cost of top of > the line cpu without dual core, which really isn't considered top of > the line. For such a low density application, you might consider dual core Athlon X2 cpus. Couple these with an entry level enthusiast motherboard (possibly an MSI k9nu?), ECC ram, and possibly a high performance GigE nic depending on what's on the motherboard. Intel and Broadcom based nics are nice because they're fast, well supported by standard Linux kernel drivers, and work with low-latency stacks like gamma. Opteron carries a hefty price premium, and is only necessary when you want to run 2+ socket motherboards. -- Geoffrey D. Jacobs Go to the Chinese Restaurant, Order the Special From bill at cse.ucdavis.edu Wed Dec 6 12:01:36 2006 From: bill at cse.ucdavis.edu (Bill Broadley) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] A suitable motherboard for newbie In-Reply-To: <4576F040.90803@gmail.com> References: <1bef2ce30611260352r33e8d1aia809f75280e1ca07@mail.gmail.com> <428810f20611280325j44e500cdm7c9e91c6abd57899@mail.gmail.com> <4576F040.90803@gmail.com> Message-ID: <457721A0.7000207@cse.ucdavis.edu> > For such a low density application, you might consider dual core Athlon > X2 cpus. Couple these with an entry level enthusiast motherboard > (possibly an MSI k9nu?), ECC ram, and possibly a high performance GigE > nic depending on what's on the motherboard. Intel and Broadcom based > nics are nice because they're fast, well supported by standard Linux > kernel drivers, and work with low-latency stacks like gamma. > > Opteron carries a hefty price premium, and is only necessary when you > want to run 2+ socket motherboards. Speaking of which, Mark recently pointed out to me that the new AMD Quad FX chips (a pair of dual cores) are basically half price opterons. From what limited info I have it looks like they don't support registered memory, but otherwise use the same socket, same cache, etc. Seems ideal for many HPC uses (except those which require a large number of dimms per cpu). Has anyone played with a pair? Is the "Quad FX" socket actually different? Or does it just depend on non-registered dimms being plugged into the dimm slots? Pricing at: http://www.amd.com/us-en/Processors/ProductInformation/0,,30_118_609,00.html?redir=CPT301 From fant at pobox.com Wed Dec 6 15:01:07 2006 From: fant at pobox.com (Andrew D. Fant) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] LSF and Fluent in a beowulf environment Message-ID: <45774BB3.8060102@pobox.com> If I may interrupt for a minute with an application-oriented question... If there is anyone out there who is running Fluent (CFD) in parallel on a shared-access Linux cluster who would be willing to answer a couple questions for me in email, I would really appreciate hearing from you. Part of the problem involves LSF interactions, so it would be nice if you have seen this, but at this point, I would take any advice I am offered. Thanks, Andy -- Andrew Fant | And when the night is cloudy | This space to let Molecular Geek | There is still a light |---------------------- fant@pobox.com | That shines on me | Disclaimer: I don't Boston, MA | Shine until tomorrow, Let it be | even speak for myself From James.P.Lux at jpl.nasa.gov Wed Dec 6 19:47:17 2006 From: James.P.Lux at jpl.nasa.gov (Jim Lux) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] 'liquid cooled' racks In-Reply-To: <4576CEAD.2000007@ahpcrc.org> References: <45743B22.5090908@uiowa.edu> <20061205164943.GA2722@galactic.demon.co.uk> <4575B63B.5080709@ahpcrc.org> <20061205233449.GA5628@greglaptop> <45760497.3030101@ahpcrc.org> <6.2.3.4.2.20061205191702.032bbe30@mail.jpl.nasa.gov> <4576CEAD.2000007@ahpcrc.org> Message-ID: <6.2.3.4.2.20061206194125.02f79b58@mail.jpl.nasa.gov> At 06:07 AM 12/6/2006, Richard Walsh wrote: >Jim Lux wrote: >>At 03:45 PM 12/5/2006, Richard Walsh wrote: >>>Greg Lindahl wrote: >>>>On Tue, Dec 05, 2006 at 12:11:07PM -0600, Richard Walsh wrote: >>>>>One of the key innovations on the Cray X1 is that the circuits >>>>>are "on the ceiling" >>>>>so to speak and sprayed from below. The fluid is gravity >>>>>collected and cycled up again. >>>>This technology predates the X1 by a while. >>> Mmm ... I did not know that ... in a reasonably successful >>> commercial product (i.e. an innovation, rather >>> than a mere invention)? What was/is/were/are the product(s)? >>> Perhaps outside of HPC ... >> >>This has been around for quite a while (decades at least). It was >>in a Fluorinert brochure back in the mid 80s that I >>recall. There's also versions with ebullient (boiling) >>cooling. There might even be high power vacuum tubes cooled this >>way, although I think they either tend to use a cooling jacket or a >>boiler, as opposed to spraying). >This is drifting away from the useful, but I was asking what other >successful HPC (or computing generally) >product has used a spray-cool, gravity collection system the like >Cray X1. I am not saying there isn't one, just >asking someone to tell what it is. How do you define "successful"? Paying the salaries of the developers? Making a profit for the company using it? Working without breaking? >And as a side note, to me innovation implies successful commercial >application ... I understand that evaporation >is a cooling process ... ;-) ... Lots of successful innovations aren't very commercially viable (in the sense of providing a return to the shareholders). The Mars Rovers are fairly successful, contain lots of innovation, but there's not much commercial value in the science data returned from them. There are "wick/evaporation" cooling systems around too.. some "heat pipes" work by this principle. In fact, most passive heat pipes work by some sort of phase change (evaporate where it's hot, condense where it's cool) scheme. >rbw > >-- James Lux, P.E. Spacecraft Radio Frequency Subsystems Group Flight Communications Systems Section Jet Propulsion Laboratory, Mail Stop 161-213 4800 Oak Grove Drive Pasadena CA 91109 tel: (818)354-2075 fax: (818)393-6875 From ruhollah.mb at gmail.com Tue Dec 5 22:11:32 2006 From: ruhollah.mb at gmail.com (Ruhollah Moussavi Baygi ) Date: Sat May 10 01:05:33 2008 Subject: [Beowulf] SATA II In-Reply-To: <200612051748.46469.cap@nsc.liu.se> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> Message-ID: <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> Thanks Peter, But do you mean that SATA is not a suitable choice for a beowulf cluster? On 12/5/06, Peter Kjellstrom wrote: > > On Tuesday 05 December 2006 08:34, Ruhollah Moussavi Baygi wrote: > > Hi All at Beowulf ! > > > > There are some questions about implementation of a Beowulf cluster: > > > > 1-Regarding OS, is "Fedora Core 64bit" a good option for AMD Athlon 64 > X2 > > 4200+? > > You'll have to upgrade to a later fedora core after a while when support > runs > out, but pretty much a matter of taste. (technically it'll be as good as > any) > > > 2- Is SATA II HDD compatible with Fedora Core 64bit? > > The sata and/or raid controller will be a lot more critical to your linux > experience than any choice of HDD. > > > 3- Concerning RAM, is "2 GB 800 MHz DDR2" sufficient? > > That depends close to 100% on what you'll run on it. > > /Peter > > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit > http://www.beowulf.org/mailman/listinfo/beowulf > > > > -- Best, Ruhollah Moussavi B. Computational Physical Sciences Research Laboratory, Department of NanoScience, IPM -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20061206/f4cd85d0/attachment.html From ruhollah.mb at gmail.com Wed Dec 6 00:58:49 2006 From: ruhollah.mb at gmail.com (Ruhollah Moussavi Baygi ) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] More technical information and spec of beowulf In-Reply-To: References: <1bef2ce30612030813j1d8c2cb6r9c9c49f8db617683@mail.gmail.com> Message-ID: <1bef2ce30612060058x21c80f91qd8722465ce4edc6e@mail.gmail.com> Hi Reza, Following may be helpful. Please anyone at Beowulf that thinks his/her notes can help, participate in this subject: MPI is nothing, but a library of some functions that can be called by a code (written in FORTRAN or C) to pass information between CPUs. On the other hand, Beowlf is nothing but a network of commonplace computers with this library installed on all fo them. Truthfully speaking, i have not a thorough experience in adjusting MPI configurations, but, as far as i know there are no special configurations, and just installing of MPI is enough. As you told, Beowulf is just this. However, after implementation you can use it depending on your applications. There are 2 possibilities: 1-you (or your staff) yourself write parallel program (FORTRAN/C) and then compile it on Beowulf cluster, or, 2- you use prepared packages like GROMACS, DLPOLY, ESPRESSO, ... which mainly are ready with their source codes. On 12/5/06, reza bakhshi wrote: > > Hello Mr. Mousavi, > Thank you for your notes, but what I have read about Beowulf told me that > we need for example some job scheduling mechanism, distributed programming > (like PVM, MPI, ...). by software infra, i meant in an implemented Beowulf > cluster one just needs pvm, mpich2, condor, ... and that's all? > or we need more configurations applied on pvm, condor, and any other > utility to gain a full beowulf. > I hope these cleared my problem :) > Can you (or anyone) help more? > Thank you again, > > --Reza Bakhshi > > -----Original Message----- > From: "Ruhollah Moussavi Baygi " > To: "reza bakhshi" > Cc: beowulf@beowulf.org > Date: Sun, 3 Dec 2006 19:43:57 +0330 > Subject: Re: [Beowulf] More technical information and spec of beowulf > > > Hi Reza! > > What do u mean exactly by "Beowulf software infrastructure" ? > > > > u know Beowulf is just a manner of connecting commonplace computers > > (nodes) > > together to make a robust system. Then your computational job will be > > divided among nodes. As far as I know, there is no especial software > > for > > this framework. All you need is a message passing interface (MPI) which > > makes it possible for nodes to pass messages together while running > > job. > > > > There are a lot of scientific packages (and mainly open source) which > > have > > been developed in such a way that you can run in parallel, like > > GROMACS, > > DLPOLY, AMBER, WIEN2K, and many others. All these packages (at least > > their > > computational parts), have been compiled by FORTRAN or C/C++ .This is > > due to > > the capability of these two compilers which can be run in parallel and > > work > > with MPI. > > > > However, there are some engineering and finite element software which > > are > > absolutely commercial and capable of being run in parallel, among them > > are > > FLUENT, CFX, and so on. > > > > I hope this could be helpful for you, > > > > Ruhollah Moussavi B. > > On 11/30/06, reza bakhshi wrote: > > > > > > Hi there, > > > How can i find some more detailed technical information on Beowulf > > > software infrastructure? > > > Thank you :) > > > > > > > > > _______________________________________________ > > > Beowulf mailing list, Beowulf@beowulf.org > > > To change your subscription (digest mode or unsubscribe) visit > > > http://www.beowulf.org/mailman/listinfo/beowulf > > > > > > > > > > > -- > > Best, > > Ruhollah Moussavi B. > > > > > -- Best, Ruhollah Moussavi Baygi Computational Physical Sciences Research Laboratory, Department of NanoScience, IPM -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20061206/0a75ea36/attachment.html From ruhollah.mb at gmail.com Wed Dec 6 01:03:03 2006 From: ruhollah.mb at gmail.com (Ruhollah Moussavi Baygi ) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] OSCAR? Message-ID: <1bef2ce30612060103x7460435dqbe1cc145e6179617@mail.gmail.com> Hi all at Boewulf, Does anyone have any experience in using OSCAR? Does anyone recommend/oppose using this suite for a small or large Beowulf cluster (AMD dual core 64bit)? -- Best, Ruhollah Moussavi Baygi Computational Physical Sciences Research Laboratory, Department of NanoScience, IPM -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.scyld.com/pipermail/beowulf/attachments/20061206/2e936092/attachment.html From turuncu at be.itu.edu.tr Thu Dec 7 12:18:52 2006 From: turuncu at be.itu.edu.tr (turuncu@be.itu.edu.tr) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] Re: LSF and Fluent in a beowulf environment Message-ID: <49255.85.102.189.28.1165522732.squirrel@www.be.itu.edu.tr> Hi, I will write a simple lsf script that fix the lsf and fluent integration. you can use the following script to submit job to lsf, >> Begin script (don't add this line to script) #!/usr/bin/ksh #BSUB -a fluent # lsf fluent integration paramter #BSUB -J FLUENT # job name #BSUB -o %J.out # LSF out file #BSUB -e %J.err # LSF error file #BSUB -n 12 # number of process (must be same with fluent -t parameter) #BSUB -q trccsq # queue name # ------------------------- # erase rm -rf host.file # get empty node names np=`echo $LSB_HOSTS` # generate new machine file for i in $np do echo $i >> host.file done # ------------------------- fluent -lsf -pnmpi -cnf=./host.file 2ddp -i fluent.jou -t12 -p -g> fluent.out >> End script (don't add this line) To use the script you have to create fluent journal (fluent.jou) and data files. I hope this helps to you. ps: -t12 and lsf -n parameter must be point same number of cpu. Best regards, Ufuk Utku Turuncoglu Istanbul Technical University HPC Lab From gdjacobs at gmail.com Thu Dec 7 22:30:47 2006 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] More technical information and spec of beowulf In-Reply-To: <1bef2ce30612060058x21c80f91qd8722465ce4edc6e@mail.gmail.com> References: <1bef2ce30612030813j1d8c2cb6r9c9c49f8db617683@mail.gmail.com> <1bef2ce30612060058x21c80f91qd8722465ce4edc6e@mail.gmail.com> Message-ID: <45790697.8040609@gmail.com> Nothing says you must use MPI. PVM was around on clusters years and years ago, and many people still prefer it. Some people are using OpenMosix and shared mem on small clusters. A software maniac might code a parallel application in raw tcp/ip. I knew a guy who built a parallel Mandelbrot appliance using raw IPX on DOS in high school. Beowulf is nothing more than using networked commodity-based hardware to collectively solve large, parallel computational problems. -- Geoffrey D. Jacobs From gdjacobs at gmail.com Thu Dec 7 22:34:28 2006 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II In-Reply-To: <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> Message-ID: <45790774.8040400@gmail.com> Ruhollah Moussavi Baygi wrote: > Thanks Peter, > But do you mean that SATA is not a suitable choice for a beowulf cluster? SATA is fine. You just have to be choosy about the SATA/SAS controller, and be mindful of reliability issues with desktop drives. -- Geoffrey D. Jacobs From hahn at physics.mcmaster.ca Fri Dec 8 06:29:06 2006 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II In-Reply-To: <45790774.8040400@gmail.com> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> <45790774.8040400@gmail.com> Message-ID: >> Thanks Peter, >> But do you mean that SATA is not a suitable choice for a beowulf cluster? > SATA is fine. You just have to be choosy about the SATA/SAS controller, it's interesting that SAS advertising has obscured the fact that SAS is just a further development of SCSI, and not interchangable with SATA. for instance, no SATA controller will support any SAS disk, and any SAS setup uses a form of encapsulation to communicate with the foreign SATA protocol. SAS disks follow the traditional price formula of SCSI disks (at least 4x more than non-boutique disks), and I suspect the rest of SAS infrastructure will be in line with that. > and be mindful of reliability issues with desktop drives. I would claim that this is basically irrelevant for beowulf. for small clusters (say, < 100 nodes), you'll be hitting a negligable number of failures per year. for larger clusters, you can't afford any non-ephemeral install on the disks anyway - reboot-with-reimage should only take a couple minutes more than a "normal" reboot. and if you take the no-install (NFS root) approach (which I strongly recommend) the status of a node-local disks can be just a minor node property to be handled by the scheduler. by all means, buy only 5-year warranty mass-market drives, since there's no longer any premium vs 3 or even 1-year drives. the failure rate I've seen over the past couple years has been quite low - probably around .1-.5% AFR (failures/disk-year). (that's ignoring infant mortality, of course, and a reasonably cooled operating environment; expect higher rates if your supply chain involves piles of un-padded disks sitting on on some shop's counter/shelf/display-case ;) From gdjacobs at gmail.com Fri Dec 8 12:01:51 2006 From: gdjacobs at gmail.com (Geoff Jacobs) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II In-Reply-To: References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> <45790774.8040400@gmail.com> Message-ID: <4579C4AF.5080607@gmail.com> Mark Hahn wrote: >>> Thanks Peter, >>> But do you mean that SATA is not a suitable choice for a beowulf >>> cluster? >> SATA is fine. You just have to be choosy about the SATA/SAS controller, > > it's interesting that SAS advertising has obscured the fact that SAS is > just a further development of SCSI, and not interchangable > with SATA. for instance, no SATA controller will support any SAS disk, > and any SAS setup uses a form of encapsulation to communicate with > the foreign SATA protocol. SAS disks follow the traditional price > formula of SCSI disks (at least 4x more than non-boutique disks), > and I suspect the rest of SAS infrastructure will be in line with that. Yes, SAS encapsulates SATA, but not vice-versa. The ability to use a hardware raid SAS controller with large numbers of inexpensive SATA drives is very attractive. I was also trying to be thorough. >> and be mindful of reliability issues with desktop drives. > > I would claim that this is basically irrelevant for beowulf. > for small clusters (say, < 100 nodes), you'll be hitting a negligable > number of failures per year. for larger clusters, you can't afford > any non-ephemeral install on the disks anyway - reboot-with-reimage > should only take a couple minutes more than a "normal" reboot. > and if you take the no-install (NFS root) approach (which I strongly > recommend) the status of a node-local disks can be just a minor node > property to be handled by the scheduler. PXE/NFS is absolutely the slickest way to go, but any service nodes should have some guarantee of reliability. In my experience, disks (along with power supplies) are two of the most common points of failure. -- Geoffrey D. Jacobs Go to the Chinese Restaurant, Order the Special From mwill at penguincomputing.com Fri Dec 8 13:14:31 2006 From: mwill at penguincomputing.com (Michael Will) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <4579C4AF.5080607@gmail.com> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> <45790774.8040400@gmail.com> <4579C4AF.5080607@gmail.com> Message-ID: <4579D5B7.7090608@jellyfish.highlyscyld.com> Geoff Jacobs wrote: > Mark Hahn wrote: > >> it's interesting that SAS advertising has obscured the fact that SAS is >> just a further development of SCSI, and not interchangable >> with SATA. for instance, no SATA controller will support any SAS disk, >> and any SAS setup uses a form of encapsulation to communicate with >> the foreign SATA protocol. SAS disks follow the traditional price >> formula of SCSI disks (at least 4x more than non-boutique disks), >> and I suspect the rest of SAS infrastructure will be in line with that. >> > Yes, SAS encapsulates SATA, but not vice-versa. The ability to use a > hardware raid SAS controller with large numbers of inexpensive SATA > drives is very attractive. I was also trying to be thorough. > > >>> and be mindful of reliability issues with desktop drives. >>> >> I would claim that this is basically irrelevant for beowulf. >> for small clusters (say, < 100 nodes), you'll be hitting a negligable >> number of failures per year. for larger clusters, you can't afford >> any non-ephemeral install on the disks anyway - reboot-with-reimage >> should only take a couple minutes more than a "normal" reboot. >> and if you take the no-install (NFS root) approach (which I strongly >> recommend) the status of a node-local disks can be just a minor node >> property to be handled by the scheduler. >> > PXE/NFS is absolutely the slickest way to go, but any service nodes > should have some guarantee of reliability. In my experience, disks > (along with power supplies) are two of the most common points of failure Most of the clusters we configure for our customers use diskless compute nodes to minimize compute node failure for precisely the reason you mentioned unless either the application can benefit from additional local scratchspace (i.e. software raid0 over four sata drives allows to read/write large datastreams at 280MB/s in a 1U server with 3TB of disk space on each compute node), or because they need to sometimes run jobs that require more virtual memory than they can afford to put in physically -> local swapspace. We find that customers don't typically want to pay for the premium for redundant power supplies+pdus+cabling for the compute nodes through, that's something that is typically requested for head nodes and NFS servers. Also we find that NFS-offloading on the NFS-server with the rapidfile card helps avoid scalability issues where the NFS server bogs down under massively parallel requests from say 128 cores in a 32 compute node dual cpu dual core cluster. The rapidfile card is a pci-x card with two fibre channel ports + two gige ports + nfs/cifs offloading processor on the same card. Since most bulk data transfer is redirected from fibre channel to gige nfs clients without passing through the NFS server cpu+ram itself, the nfs servers cpu load is not becoming the bottleneck, we find it's rather the amount of spindles before saturating the two gige ports. We configure clusters for our customers with Scyld Beowulf which does not nfs-mount root but rather just nfs-mounts the home directories because of its particular lightweight compute node model, (PXE booting into RAM) and so does not run into the typical nfs-root scalability issues. Michael Michael Will SE Technical Lead / Penguin Computing / www.penguincomputing.com From harsha at zresearch.com Thu Dec 7 21:03:24 2006 From: harsha at zresearch.com (Harshavardhana) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] Fluent Running on LSF env Message-ID: <4573.220.227.64.170.1165554204.squirrel@zresearch.com> Hi Andrew, i have some experience in doing fluent runs over the LSF env. You can hit me with some questions, may be i can answer some. Regards -Harshavardhana -- Harshavardhana "Software gets slower faster as Hardware gets faster" From eugen at leitl.org Fri Dec 8 01:04:03 2006 From: eugen at leitl.org (Eugen Leitl) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II In-Reply-To: <45790774.8040400@gmail.com> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> <45790774.8040400@gmail.com> Message-ID: <20061208090403.GR6974@leitl.org> On Fri, Dec 08, 2006 at 12:34:28AM -0600, Geoff Jacobs wrote: > SATA is fine. You just have to be choosy about the SATA/SAS controller, > and be mindful of reliability issues with desktop drives. Some SATA is more reliable than others. Caviar RE2 drives claim enterprise-level reliability, only at a slight price premium in comparison to e.g. WD Raptor series. -- Eugen* Leitl leitl http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE From steve_heaton at iinet.net.au Fri Dec 8 18:05:04 2006 From: steve_heaton at iinet.net.au (Steve Heaton) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] Beowulf analogy for a classroom Message-ID: <457A19D0.3070605@iinet.net.au> G'day all I'll skip the background as to 'why' but I've recently been working on a way to explain the Beowulf concept to a classroom of school kids. No computers required. I think I've come up with a useful analogy/experiment that might work. I've posted it here on the off chance that someone else might want to give it a try if they get pressed into such a situation. First, some 'newsgroup preempters'. This is designed for school kids not your typical Beowulf list reader. Sorry but no cars or harnessing of chickens involved ;) No mentions of heat dissipation or switching fabrics. This is a non-technical post that someone might find handy at some point. So, here's the idea. Generate a list of, say, 100 random numbers (int). Make two copies. Break the 3rd copy into 10 lists of 10 numbers each. You need 12 students. They're going to be your 'nodes' :) Ask the teacher which student is the best at mental arithmetic. They're going to be the 'standalone' node. You then pick 11 other students and they'll be the Beowulf. Of those select 1 as the 'master' node. Give the two long (100 number) lists to the 'standalone node' student and the teacher. Give the 10 number lists to the master node. Then you say 'Go!' and start the stopwatch. The 'standalone node' and the teacher start summing their lists. The 'master' node student hands out their lists to the other 'nodes' at their desks and they start summing. As each 'node' student finishes, they walk their summed result back to the 'master' node. The master node student sums the returning results. When the 'standalone' node, 'master' node and teacher have finished their sums, record the time on the board. Now I'm sure you can see there's a huge amount of variables and things that could go wrong (as in any classroom demo). However, hopefully the overall result goes something like the teacher (representing a singe powerful node) finished first. Then the Beowulf group of students, then the 'standalone' node. I think this structure could be quite useful and there's some good 'learning outcomes'. eg: The Beowulf group was 10 times bigger but why wasn't it ten times faster? (Moving the results around = comms time, coordination etc) Why was the teacher faster? ((Hopefully) sharper math skills and no comms overhead) ... Image a Beowulf cluster of 'teachers'! ... Imagine the results if the cluster were (lower grade) students!? What if there were 20 of them instead of 10? (Leads to a discussion of Amdahl's Law etc etc) Obviously you can change the number of 'nodes' and difficulty of the addition to match the student abilities. Anyway, that's the idea. I'd love to hear if anyone ever has cause to use it :) Cheers Stevo From gmpc at sanger.ac.uk Sat Dec 9 01:11:07 2006 From: gmpc at sanger.ac.uk (Guy Coates) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <4579D5B7.7090608@jellyfish.highlyscyld.com> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> <45790774.8040400@gmail.com> <4579C4AF.5080607@gmail.com> <4579D5B7.7090608@jellyfish.highlyscyld.com> Message-ID: <457A7DAB.8070403@sanger.ac.uk> > We configure clusters for our customers with Scyld Beowulf which does > not nfs-mount > root but rather just nfs-mounts the home directories because of its > particular lightweight > compute node model, (PXE booting into RAM) and so does not run into the > typical > nfs-root scalability issues. > > Michael > > Michael Will At what node count does the nfs-root model start to break down? Does anyone have any rough numbers with the number of clients you can support with a generic linux NFS server vs a dedicated NAS filer? Cheers, Guy -- Dr. Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK Tel: +44 (0)1223 834244 x 6925 Fax: +44 (0)1223 496802 From landman at scalableinformatics.com Sat Dec 9 05:58:55 2006 From: landman at scalableinformatics.com (Joe Landman) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <457A7DAB.8070403@sanger.ac.uk> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> <45790774.8040400@gmail.com> <4579C4AF.5080607@gmail.com> <4579D5B7.7090608@jellyfish.highlyscyld.com> <457A7DAB.8070403@sanger.ac.uk> Message-ID: <457AC11F.9060609@scalableinformatics.com> Guy Coates wrote: > > At what node count does the nfs-root model start to break down? Does anyone > have any rough numbers with the number of clients you can support with a generic > linux NFS server vs a dedicated NAS filer? If you use warewulf or the new perceus variant, it creates a ram disk which is populated upon boot. Thats one of the larger transients. Then you nfs mount applications, and home directories. I haven't looked at Scyld for a while, but I seem to remember them doing something like this. If you have this operational in this manner, apart from home/scratch directory file service, you should be able to handle a few hundred nodes without too much pain, though you will want to beef up the design a bit. Scyld requires a meatier head node as I remember due to its launch model. Even better if you can load balance your NFS servers and set them up to mirror each other, or use a hardware unit like Panasas, and so on. Joe > > Cheers, > > Guy > -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com phone: +1 734 786 8423 fax : +1 734 786 8452 or +1 866 888 3112 cell : +1 734 612 4615 From john.hearns at streamline-computing.com Sat Dec 9 07:42:58 2006 From: john.hearns at streamline-computing.com (John Hearns) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <457AC11F.9060609@scalableinformatics.com> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> <45790774.8040400@gmail.com> <4579C4AF.5080607@gmail.com> <4579D5B7.7090608@jellyfish.highlyscyld.com> <457A7DAB.8070403@sanger.ac.uk> <457AC11F.9060609@scalableinformatics.com> Message-ID: <457AD982.9090405@streamline-computing.com> Joe Landman wrote: > > Guy Coates wrote: > >> At what node count does the nfs-root model start to break down? Does anyone >> have any rough numbers with the number of clients you can support with a generic >> linux NFS server vs a dedicated NAS filer? > > If you use warewulf or the new perceus variant, it creates a ram disk > which is populated upon boot. Thats one of the larger transients. Then > you nfs mount applications, and home directories. I haven't looked at > Scyld for a while, but I seem to remember them doing something like this. > I've booted up and run a 130 node cluster in ramdisk mode, with applications shared over NFS. I ran HPL benchmarks for the client across all nodes in the cluster in this configuration. That's 130 Sun galaxy 4200s, the head node being a Galaxy 4200 also in this case. I agree with what Joe says about a few hundred nodes being the time you would start to look closer at this approach. > > Even better if you can load balance your NFS servers and set them up to > mirror each other, or use a hardware unit like Panasas, and so on. Second that. -- John Hearns Senior HPC Engineer Streamline Computing, The Innovation Centre, Warwick Technology Park, Gallows Hill, Warwick CV34 6UW Office: 01926 623130 Mobile: 07841 231235 From buccaneer at rocketmail.com Sat Dec 9 08:38:52 2006 From: buccaneer at rocketmail.com (Buccaneer for Hire.) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes Message-ID: <20061209163852.31426.qmail@web30604.mail.mud.yahoo.com> [snip] > I agree with what Joe says about a few hundred nodes being the time you > would start to look closer at this approach. I have started to explore the possibility of using this technology because I would really like to see us with the ability to change OSs and OS Personalities as needed. The question I have is with 2000+ compute nodes what kind of infrastructure do I need to support this? ____________________________________________________________________________________ Do you Yahoo!? Everyone is raving about the all-new Yahoo! Mail beta. http://new.mail.yahoo.com From eric-shook at uiowa.edu Sat Dec 9 11:27:45 2006 From: eric-shook at uiowa.edu (Eric Shook) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <20061209163852.31426.qmail@web30604.mail.mud.yahoo.com> References: <20061209163852.31426.qmail@web30604.mail.mud.yahoo.com> Message-ID: <457B0E31.8090205@uiowa.edu> Not to diverge this conversation, but has anyone had any experience using this pxe boot / nfs model with a rhel variant? I have been wanting to do a nfs root or ramdisk model for some-time but our software stack requires a rhel base so Scyld and Perceus most likely will not work (although I am still looking into both of them to make sure) Thanks for any help, Eric Shook Buccaneer for Hire. wrote: > [snip] > > >> I agree with what Joe says about a few hundred nodes being the time you >> would start to look closer at this approach. >> > > I have started to explore the possibility of using this technology because I would really like to see us with the ability to change OSs and OS Personalities as needed. The question I have is with 2000+ compute nodes what kind of infrastructure do I need to support this? > > > > > > > > > > ____________________________________________________________________________________ > Do you Yahoo!? > Everyone is raving about the all-new Yahoo! Mail beta. > http://new.mail.yahoo.com > > _______________________________________________ > Beowulf mailing list, Beowulf@beowulf.org > To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf > From john.hearns at streamline-computing.com Sat Dec 9 11:35:59 2006 From: john.hearns at streamline-computing.com (John Hearns) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <20061209163852.31426.qmail@web30604.mail.mud.yahoo.com> References: <20061209163852.31426.qmail@web30604.mail.mud.yahoo.com> Message-ID: <457B101F.7030402@streamline-computing.com> Buccaneer for Hire. wrote: > [snip] > >> I agree with what Joe says about a few hundred nodes being the time you >> would start to look closer at this approach. > > I have started to explore the possibility of using this technology because I would really like to see us with the ability to change OSs and OS Personalities as needed. The question I have is with 2000+ compute nodes what kind of infrastructure do I need to support this? > With 2000+ nodes you should definitely look at remote power control, and remote serial console access. Also you might think of separate install servers for each (say) 500 machines. Mirror them up to each other of course. Its unlikely that you would ever reboot 2000 machines at once, but think ahead to (say) quick power on following a power cut. I would hazard that any DHCP/PXE type install server would struggle with 2000 requests (yes- you arrange the power switching and/or reboots to stagger at N second intervals). From buccaneer at rocketmail.com Sat Dec 9 12:03:27 2006 From: buccaneer at rocketmail.com (Buccaneer for Hire.) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes Message-ID: <20061209200327.65078.qmail@web30610.mail.mud.yahoo.com> Thank you for writing... > With 2000+ nodes you should definitely look at remote power control, and > remote serial console access. Have it already in place with remote monitoring as well. > Also you might think of separate install servers for each (say) 500 > machines. Mirror them up to each other of course. We currently have 5 kickstart servers (one web server), the kickstart file is dynamically altered to reflect the assigned server. > Its unlikely that you would ever reboot 2000 machines at once, but think > ahead to (say) quick power on following a power cut. We have had to do that during a couple of Hurricanes last year and power outages. We actually have complete startup and shutdown procedures that are well tested now. > I would hazard that any DHCP/PXE type install server would struggle with > 2000 requests (yes- you arrange the power switching and/or reboots to > stagger at N second intervals). There are a few modifications you have to make to increase the number of bootps before it fails. So now to figure out my next step. I will need local space for logs and data/temp data files. ____________________________________________________________________________________ Any questions? Get answers on any topic at www.Answers.yahoo.com. Try it now. From laytonjb at charter.net Sat Dec 9 13:04:34 2006 From: laytonjb at charter.net (Jeffrey B. Layton) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <457B0E31.8090205@uiowa.edu> References: <20061209163852.31426.qmail@web30604.mail.mud.yahoo.com> <457B0E31.8090205@uiowa.edu> Message-ID: <457B24E2.20208@charter.net> Eric Shook wrote: > Not to diverge this conversation, but has anyone had any experience > using this pxe boot / nfs model with a rhel variant? I have been > wanting to do a nfs root or ramdisk model for some-time but our > software stack requires a rhel base so Scyld and Perceus most likely > will not work (although I am still looking into both of them to make sure) Warewulf and Perceus will work withe something like CentOS. Jeff From landman at scalableinformatics.com Sat Dec 9 13:56:29 2006 From: landman at scalableinformatics.com (Joe Landman) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <20061209200327.65078.qmail@web30610.mail.mud.yahoo.com> References: <20061209200327.65078.qmail@web30610.mail.mud.yahoo.com> Message-ID: <457B310D.7060201@scalableinformatics.com> Buccaneer for Hire. wrote: >> I would hazard that any DHCP/PXE type install server would struggle >> with 2000 requests (yes- you arrange the power switching and/or >> reboots to stagger at N second intervals). fwiw: we use dnsmasq to serve dhcp and handle pxe booting. It does a marvelous job of both, and is far easier to configure (e.g. it is less fussy) than dhcpd. > There are a few modifications you have to make to increase the number > of bootps before it fails. Likely with dhcpd, not sure how many dnsmasq can handle, but we have done 36 at a time to do system checking. No problems with it. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman@scalableinformatics.com web : http://www.scalableinformatics.com phone: +1 734 786 8423 fax : +1 734 786 8452 or +1 866 888 3112 cell : +1 734 612 4615 From hahn at physics.mcmaster.ca Sat Dec 9 14:39:25 2006 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <457A7DAB.8070403@sanger.ac.uk> References: <1bef2ce30612042334j3cfa6f00ta0a7c3f90afb1842@mail.gmail.com> <200612051748.46469.cap@nsc.liu.se> <1bef2ce30612052211v7f16033bh61f53b11fee26100@mail.gmail.com> <45790774.8040400@gmail.com> <4579C4AF.5080607@gmail.com> <4579D5B7.7090608@jellyfish.highlyscyld.com> <457A7DAB.8070403@sanger.ac.uk> Message-ID: >> particular lightweight >> compute node model, (PXE booting into RAM) and so does not run into the >> typical >> nfs-root scalability issues. I'm not sure I know what those would be. do you mean that the kernel code for nfs-root has inappropriate timeouts or lacked effective retries? > At what node count does the nfs-root model start to break down? Does anyone > have any rough numbers with the number of clients you can support with a generic > linux NFS server vs a dedicated NAS filer? I think the answer depends mostly on your config. for instance, if you have a typical distro's incredibly baroque /etc/rc.d tree, then you'll be generating tons of traffic even though NFS caches quite well. but for HPC clustering, most of that is completely spurious - often a clusters nodes are all identical, so no extensive configurability is necessary in modules, daemons, etc. if there are scalability issues, they depend on saturating your NFS server with traffic, but you control that amount. on a somewhat neglected cluster I have, kernel+initrd amount to 3779277 bytes, which seems quite high. but probably limits the cluster to 10-ish nodes/second booting (it has 100 nodes, but I've never timed the boot). once a node has the kernel+initrd, it reads some other files via NFS, but nothing much (syslog binary+config, same for portmap, and sshd). to me, the tradeoff is transmitting via TFTP vs NFS. I would strongly suspect that the latter is more efficient and robust, so would prefer to minimize the kernel+initrd size. for what it's worth, I tcpdumped a node booting just now: 11428451 bytes in 14236 packets (40.8 seconds). that's a 2.6 kernel, myrinet support, syslog, ssh, queuing system written in perl, and home and scratch mounts. with some effort, that could probably be 5MB or so. it's also clear that separate servers could handle subsets of the traffic in a large cluster (separate dhcp/tftp/syslog, separate servers for nfs root vs other) regards, mark hahn. From hahn at physics.mcmaster.ca Sat Dec 9 15:44:25 2006 From: hahn at physics.mcmaster.ca (Mark Hahn) Date: Sat May 10 01:05:34 2008 Subject: [Beowulf] SATA II - PXE+NFS - diskless compute nodes In-Reply-To: <20061209200327.65078.qmail@web30610.mail.mud.yahoo.com> References: <20061209200327.65078.qmail@web30610.mail.mud.yahoo.com> Message-ID: >> I would hazard that any DHCP/PXE type install server would struggle with >> 2000 requests a single server (implying 1 gb nic?) might have trouble with the tftp part, but I don't see why you couldn't scale up by splitting the tftp part off to multiple servers. I'd expect a single DHCP (no TFTP) would be plenty in all cases. 100 tftp clients per server would probably be pretty safe. I personally like the id