From brian@chpc.utah.edu Mon, 4 Jan 1999 13:38:17 -0500 Date: Mon, 4 Jan 1999 13:38:17 -0500 From: Brian Haymore brian@chpc.utah.edu Subject: IFCONFIG reports maxed out transmitted and recieved packets I just noteced that my ifconfig has hit the limit on transmitted and recieved packets. eth2 Link encap:Ethernet HWaddr 00:C0:F0:40:09:1F inet addr:10.110.10.254 Bcast:10.110.10.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2147483647 errors:4 dropped:4 overruns:0 frame:4 TX packets:2147483647 errors:235 dropped:0 overruns:1 carrier:234 collisions:0 Interrupt:14 Base address:0xb400 What I am worried about is that this could cause a problem. Can anyone set my nerves at rest on this? IF it is a problem how have people solved this? It worries me because this will happen twice a month at the rate we are going. Thanks. -- Brian Haymore - Center for High Performance Computing, University of Utah Email: brian@chpc.utah.edu Fax: (801) 585-5366 Phone: (801) 585-1755 "I can please only one person a day. Today is not your day. Tomorrow isn't looking good either." --Anonymous From lombao@arrakis.es Tue, 5 Jan 1999 04:19:28 -0500 Date: Tue, 5 Jan 1999 04:19:28 -0500 From: Cesar Lombao lombao@arrakis.es Subject: Nerd Hello from Spain: I would like know some things about beowulf and clusters under linux, but FAQ's I saw don't have the answers I search for. If exists any paper or faq that contains the information, please, tell me where I can find it. First: BeoWulf can run and improve the performance of soft not compiled with PVM or other library of distributed computing? If apache (by example) launch several process, These process run on the same machine or beowulf can balance the process over other nodes? With 486 100 CPU and Ehternet 10Mbps can be reached some advantage? or, really, 10Mbps Ethernet is too poor for Beowulf? I know the web of www.beowulf.org, but is there any other important page about linux and distributed computing? Thanks in adavance and sorry for my english Saludos Cesar Lombao From enano@ceu.fi.udc.es Tue, 5 Jan 1999 07:10:11 -0500 Date: Tue, 5 Jan 1999 07:10:11 -0500 From: Miguel Barreiro Paz enano@ceu.fi.udc.es Subject: Nerd Hi Cesar, and nice to hear from you again :) > First: > BeoWulf can run and improve the performance of soft not compiled with > PVM or other library of distributed computing? Black magic is not implemented yet under Linux, I'm afraid. > If apache (by example) launch several process, These process run on the > same machine or beowulf can balance the process over other nodes? Round-robin DNS and / or dynamic load balancing via an inverse proxy are probably the best approaches for a webserver that large that can't be hosted on a single machine. > With 486 100 CPU and Ehternet 10Mbps can be reached some advantage? or, > really, 10Mbps Ethernet is too poor for Beowulf? As usual, depends a lot on the application. For IPC-intensive tasks, it's unlikely to be useful except for testing purposes. If you are really seeking performance, a massive number of 486 nodes is not usually the way to go, giving today pricings... Regards, Miguel From ferguson.joseph@nesc.epa.gov Tue, 5 Jan 1999 11:07:25 -0500 Date: Tue, 5 Jan 1999 11:07:25 -0500 From: Joe FergusonBoth types of receipt ferguson.joseph@nesc.epa.gov Subject: FP on AMD K6 Does anyone have performance info comparing floating point performance on AMD K6 processors with comparably-clocked P-IIs? How about Cyrix? Thanks, -- Joe Ferguson ferguson.joseph@nesc.epa.gov Lockheed Martin Corp. NESC Support, EPA-RTP -- NC Voice: (919)-541-3716 From lxm@arch.cs.pku.edu.cn Tue, 5 Jan 1999 21:55:20 -0500 Date: Tue, 5 Jan 1999 21:55:20 -0500 From: Li Xiaoming lxm@arch.cs.pku.edu.cn Subject: No subject Hi, there, I wonder if any one knows some Linux based parallel file systems that support stipping a file across multiple nodes in a Beowulf environment. Thanks ! Regards, Li Xiaoming, Professor Dept of Computer Sci. & Tech, Peking University, Beijing 100871, China From jakob@ostenfeld.dk Wed, 6 Jan 1999 00:53:22 -0500 Date: Wed, 6 Jan 1999 00:53:22 -0500 From: jakob@ostenfeld.dk jakob@ostenfeld.dk Subject: your mail On Wed, Jan 06, 1999 at 11:01:13AM +0800, Li Xiaoming wrote: > > Hi, there, > > I wonder if any one knows some Linux based parallel file systems that > support stipping a file across multiple nodes in a Beowulf environment. > > Thanks ! Actually one could try to do software RAID over the new Network Block Device found in the 2.[12].x kernels. This way you could do any of the normal RAID levels, eg. optimize for stability, speed, or whatever. Does anyone know whether this would be feasible/possible ? I would like to try this myself, just as soon as I get some more machines and disks :) ................................................................ : jakob@ostenfeld.dtu.dk : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: From rbross@parl.ces.clemson.edu Wed, 6 Jan 1999 01:14:59 -0500 Date: Wed, 6 Jan 1999 01:14:59 -0500 From: Rob Ross rbross@parl.ces.clemson.edu Subject: your mail Li, PVFS does exactly this. See http://ece.clemson.edu/parl/pvfs/index.html for more information. Rob Ross Parallel Architecture Research Laboratory, Clemson University mailto:rbross@parl.eng.clemson.edu On Wed, 6 Jan 1999, Li Xiaoming wrote: > Hi, there, > > I wonder if any one knows some Linux based parallel file systems that > support stipping a file across multiple nodes in a Beowulf environment. > > Thanks ! > > Regards, > Li Xiaoming, Professor > Dept of Computer Sci. & Tech, Peking University, Beijing 100871, China From camm@enhanced.com Wed, 6 Jan 1999 09:15:35 -0500 Date: Wed, 6 Jan 1999 09:15:35 -0500 From: Camm Maguire camm@enhanced.com Subject: your mail What about Coda? Rob Ross writes: > Li, > > PVFS does exactly this. See http://ece.clemson.edu/parl/pvfs/index.html > for more information. > > Rob Ross > Parallel Architecture Research Laboratory, Clemson University > mailto:rbross@parl.eng.clemson.edu > > On Wed, 6 Jan 1999, Li Xiaoming wrote: > > > Hi, there, > > > > I wonder if any one knows some Linux based parallel file systems that > > support stipping a file across multiple nodes in a Beowulf environment. > > > > Thanks ! > > > > Regards, > > Li Xiaoming, Professor > > Dept of Computer Sci. & Tech, Peking University, Beijing 100871, China From kragen@pobox.com Wed, 6 Jan 1999 11:32:08 -0500 Date: Wed, 6 Jan 1999 11:32:08 -0500 From: Kragen Sitaker kragen@pobox.com Subject: Nerd On Tue, 5 Jan 1999, Cesar Lombao wrote: > I would like know some things about beowulf and clusters under linux, > but FAQ's I saw don't have the answers I search for. This is a real problem; someone asks this question every month. You should ask the FAQ maintainers to add this answer. > First: > BeoWulf can run and improve the performance of soft not compiled with > PVM or other library of distributed computing? No. Run, yes; improve performance, no. > If apache (by example) launch several process, These process run on the > same machine or beowulf can balance the process over other nodes? No. You want MOSIX. mosix.cs.huji.ac.il. Available for Linux soon. > With 486 100 CPU and Ehternet 10Mbps can be reached some advantage? or, > really, 10Mbps Ethernet is too poor for Beowulf? Depends on your problem. If you're cracking crypto keys, a 2400bps modem is probably fast enough. If you're trying to simulate car crashes, you will probably want better bandwidth. > I know the web of www.beowulf.org, but is there any other important page > about linux and distributed computing? Dunno. -- Kragen Sitaker [around 1998-12-23], it is amazing to watch fear and loathing and greed at play with the more speculative Internet stocks. To call this a tulip craze would be a vast understatement. -- Adam Rifkin, From rauch@inf.ethz.ch Wed, 6 Jan 1999 12:17:46 -0500 Date: Wed, 6 Jan 1999 12:17:46 -0500 From: Felix Rauch rauch@inf.ethz.ch Subject: your mail On Wed, 6 Jan 1999 jakob@ostenfeld.dk wrote: > On Wed, Jan 06, 1999 at 11:01:13AM +0800, Li Xiaoming wrote: > > I wonder if any one knows some Linux based parallel file systems that > > support stipping a file across multiple nodes in a Beowulf environment. > > Actually one could try to do software RAID over the new Network Block Device > found in the 2.[12].x kernels. > > This way you could do any of the normal RAID levels, eg. optimize for stability, > speed, or whatever. > > Does anyone know whether this would be feasible/possible ? I'd say this allows only for one machine to access the distributed files, because RAID is not used to share disks (usually, ONE machine controlls MANY disks, but not MANY machines control MANY disks). But I actually never tried this myself. We have some ongoing student work here in the very early stage of a parallel distributed serverless filesystem over myrinet. We have nothing working yet... - Felix -- Felix Rauch | Email: rauch@inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H15 | Phone: ++41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: ++41 1 632 1307 From Christopher.Bohn@afit.af.mil Wed, 6 Jan 1999 13:19:39 -0500 Date: Wed, 6 Jan 1999 13:19:39 -0500 From: Bohn, Christopher A Christopher.Bohn@afit.af.mil Subject: your mail > I'd say this allows only for one machine to access the distributed > files, because RAID is not used to share disks (usually, ONE machine > controlls MANY disks, but not MANY machines control MANY disks). But I > actually never tried this myself. Methinks the Berkeley NOW project (http://now.cs.berkeley.edu/) did / is doing some work on using the disk drives on the many workstations as a RAID, which each machine could then access. Take care, cb *-*-*-*-*-*-*-* Capt Christopher A. Bohn Graduate Student, Electrical (digital) Engineering Air Force Institute of Technology Phone (937)255-3636 (DSN 785) AFIT/EN638 Lab x4606 Voicemail x6638 2950 P St, Box 4638 email Christopher.Bohn@afit.af.mil Wright-Patterson AFB OH 45433-7765 EngrBohn@aol.com http://members.aol.com/EngrBohn/ *-*-*-*-*-*-*-* > -----Original Message----- > From: Felix Rauch [SMTP:rauch@inf.ethz.ch] > Sent: Wednesday, January 06, 1999 12:18 PM > To: jakob@ostenfeld.dk > Cc: Li Xiaoming; beowulf@beowulf.gsfc.nasa.gov > Subject: Re: your mail > > On Wed, 6 Jan 1999 jakob@ostenfeld.dk wrote: > > On Wed, Jan 06, 1999 at 11:01:13AM +0800, Li Xiaoming wrote: > > > I wonder if any one knows some Linux based parallel file systems that > > > support stipping a file across multiple nodes in a Beowulf > environment. > > > > Actually one could try to do software RAID over the new Network Block > Device > > found in the 2.[12].x kernels. > > > > This way you could do any of the normal RAID levels, eg. optimize for > stability, > > speed, or whatever. > > > > Does anyone know whether this would be feasible/possible ? > > I'd say this allows only for one machine to access the distributed > files, because RAID is not used to share disks (usually, ONE machine > controlls MANY disks, but not MANY machines control MANY disks). But I > actually never tried this myself. > > We have some ongoing student work here in the very early stage of a > parallel distributed serverless filesystem over myrinet. We have > nothing working yet... > > - Felix > -- > Felix Rauch | Email: rauch@inf.ethz.ch > Institute for Computer Systems | Homepage: > http://www.cs.inf.ethz.ch/~rauch/ > ETH Zentrum / RZ H15 | Phone: ++41 1 632 7489 > CH - 8092 Zuerich / Switzerland | Fax: ++41 1 632 1307 From rbross@parl.ces.clemson.edu Wed, 6 Jan 1999 13:44:45 -0500 Date: Wed, 6 Jan 1999 13:44:45 -0500 From: Rob Ross rbross@parl.ces.clemson.edu Subject: Beowulf, file systems, nbd (was Re: your mail) The NBD (Network Block Device) support in 2.1-2.2 could be quite interesting as a building block for a parallel file system, but significant additional software will be needed to provide a file system on the distributed resources and to handle consistency issues (so multiple clients could access simultaneously). Definitely a neat tool to play with... Rob Ross Parallel Architecture Research Lab, Clemson University rbross@parl.ces.clemson.edu On Wed, 6 Jan 1999, Felix Rauch wrote: > On Wed, 6 Jan 1999 jakob@ostenfeld.dk wrote: > > On Wed, Jan 06, 1999 at 11:01:13AM +0800, Li Xiaoming wrote: > > > I wonder if any one knows some Linux based parallel file systems that > > > support stipping a file across multiple nodes in a Beowulf environment. > > > > Actually one could try to do software RAID over the new Network Block Device > > found in the 2.[12].x kernels. > > > > This way you could do any of the normal RAID levels, eg. optimize for stability, > > speed, or whatever. > > > > Does anyone know whether this would be feasible/possible ? > > I'd say this allows only for one machine to access the distributed > files, because RAID is not used to share disks (usually, ONE machine > controlls MANY disks, but not MANY machines control MANY disks). But I > actually never tried this myself. > > We have some ongoing student work here in the very early stage of a > parallel distributed serverless filesystem over myrinet. We have > nothing working yet... From kpm@ids.net Wed, 6 Jan 1999 22:46:13 -0500 Date: Wed, 6 Jan 1999 22:46:13 -0500 From: Kevin McAloon kpm@ids.net Subject: your mail On Wed, 06 Jan 1999, Felix Rauch wrote: > >I'd say this allows only for one machine to access the distributed >files, because RAID is not used to share disks (usually, ONE machine >controlls MANY disks, but not MANY machines control MANY disks). But I >actually never tried this myself. > Actually that depends on the disk controller. If you are using a multi-port controller then subsequent port assignments can be for other hosts to have access to the disk array simultaneously in a clustered environment. I believe Connelly Data Systems in Cambridge, MA, USA has a product which can do this. I looked at their stuff a couple of years ago for an application but it was not for LINUX, alas Solaris. mac where do you want to go next year - support open source systems >From the MacroHard Federation - Rhode Island member From paciucci@pcainf1.ing.uniroma1.it Thu, 7 Jan 1999 04:47:15 -0500 Date: Thu, 7 Jan 1999 04:47:15 -0500 From: Gabriele Paciucci paciucci@pcainf1.ing.uniroma1.it Subject: FP on AMD K6 www.cpureview.com ? -->see the article section Gabriele Paciucci Linux User Group Roma - Pluto LiMe98 - Pluto Meeting 1998: IO C'ERO !!! On Tue, 5 Jan 1999, Joe FergusonBoth types of receipt wrote: > Does anyone have performance info comparing floating point performance > on AMD K6 processors > with comparably-clocked P-IIs? How about Cyrix? > > Thanks, > > -- > Joe Ferguson ferguson.joseph@nesc.epa.gov > Lockheed Martin Corp. > NESC Support, EPA-RTP -- NC Voice: (919)-541-3716 > > > From alain.coetmeur@icdc.caissedesdepots.fr Thu, 7 Jan 1999 04:56:16 -0500 Date: Thu, 7 Jan 1999 04:56:16 -0500 From: Coetmeur, Alain alain.coetmeur@icdc.caissedesdepots.fr Subject: your mail > PVFS does exactly this. See > http://ece.clemson.edu/parl/pvfs/index.html this system seems very interesting, but is there a mapping on MPI-IO. otherwise are there parallel file system on beowulf that are useable through MPI-IO ? another point. when doing RAID0 (no parity) stripping across disks the failure of one disk ruins all the file system. The technics used often is to use redundent (eg: RAID5) RAID filesystem on each node, as individual volume to contain stripes. is it possible with a beowulf. it would seems possible to use a RAID5 or mirroring driver to hold each stripe volume of the PVFS. > On Wed, 6 Jan 1999, Li Xiaoming wrote: > > I wonder if any one knows some Linux based parallel file > systems that > > support stipping a file across multiple nodes in a Beowulf > environment. -- Alain Coetmeur, Informatique-CDC DTA mailto:alain.coetmeur@icdc.CaisseDesDepots.fr From mkh100@york.ac.uk Thu, 7 Jan 1999 05:14:59 -0500 Date: Thu, 7 Jan 1999 05:14:59 -0500 From: Michael Hodgson mkh100@york.ac.uk Subject: FP on AMD K6 > Does anyone have performance info comparing floating point performance > on AMD K6 processors > with comparably-clocked P-IIs? How about Cyrix? > Whilst I'm not sure about K6-2 (100MHz bus etc.) the origional K6 was no match for a PII for our particular application (CHARMm), which I believe to be fpu intensive. If you can be more specific, eg. is it single or double precision that you are interested in then places like Toms harware page have some benchmarks. For the myoglobin 1000 steps of MD benchmark CHARMM K6 233 - 4.49 hrs PPro 210 - 1.98 hrs 300MHz PII - 1.48 hrs The other thing to consider is that certainly with the origional K6 it was not possible to put multiple processors on the same MB, as they did not support SMP. Dependant upon your application you might want to look at Celeron cpu's Particuarly the later ones with 128k full speed cache. They are: 1) cheap ( a 300a is around 60 ukp in the UK) 2) (overclockable if you are into that kind of thing) 3) and providing your application doesn't require oodles of cache, just as quick as a PII. For the same charmm benchmark. Celeron 300a - 1.49 hrs Celeron 300a (oc to 450MHz) - 0.99 hrs (compare this to an R10k time of 1.23hrs :) If you are not into the overclocking thing as seems to be the case with many readers of this list (lets not start that discussion again :) then the 400MHz chip has just been released and should give good performance without doing anything to it. Intel plan to release the Celeron in 100MHz FSB format some time soon. NOTE: Celerons like the origional K6 do not support SMP (well not without a LOT of fiddling), and so if it's multiple processors/board you are after then it still has to be PII's. Hope there may be something useful to someone in this rambling... -Michael From alain.coetmeur@icdc.caissedesdepots.fr Thu, 7 Jan 1999 12:16:44 -0500 Date: Thu, 7 Jan 1999 12:16:44 -0500 From: Coetmeur, Alain alain.coetmeur@icdc.caissedesdepots.fr Subject: CORBA, MPI, and Portland Group Compilers on a Beowulf To integrate (on our intranet, with java or MS GUI) our MPI application running on a beowulf cluster, we plan to use a Linux CORBA. however we plan also to use the Portland Group C++ compiler for performance. Who have experience in using a Corba with PGCC. Do we need to use a free CORBA with sources, or is there a commercial version compatible. Is PGCC really incompatible with gcc in C++ (I hope they are compatible in C)? does anybody had advice, experience or info? -- Alain Coetmeur, Informatique-CDC DTA mailto:alain.coetmeur@icdc.CaisseDesDepots.fr From jakob@ostenfeld.dk Thu, 7 Jan 1999 15:24:11 -0500 Date: Thu, 7 Jan 1999 15:24:11 -0500 From: jakob@ostenfeld.dk jakob@ostenfeld.dk Subject: CORBA, MPI, and Portland Group Compilers on a Beowulf On Thu, Jan 07, 1999 at 06:18:23PM +0100, Coetmeur, Alain wrote: ... > > however we plan also to use the Portland Group C++ > compiler for performance. > Who have experience in using a Corba with PGCC. > > Do we need to use a free CORBA with sources, > or is there a commercial version compatible. There are several ones out there. Mico (search and you will find) is C++. The Gnome project spawned a C implementation of Corba, ORBit, which I think is developed with performance in mind, unlike Mico. > Is PGCC really incompatible with gcc in C++ > (I hope they are compatible in C)? For clarity: PGCC = Pentium GCC EGCS = Experimental GCC PGroupCC = Portland Group CC PGroupCC is a C++ compiler. Eg. it can compile anything I've ever seen in ANSI C++. GCC cannot. GCC is not yet a full C++ compiler. However, both EGCS and PGCC versions 1.1.0 and above seem to also be fully ANSI C++ compatible. I tested the GCC, EGCS, PGCC, and PGroupCC compilers a while ago. Only plain GCC failed the C++ compliance test. Although I still need to redo the performance test, I believe performance of the PGroup compiler is comparable to EGCS or PGCC. EGCS and PGCC where usually very close, with PGCC only a little faster. PGroup was extremely slow, but that was because I overlooked a poorly documented compiler switch. With the right switches PGroupCC seems to generate code comparable to EGCS or PGCC. Another option for you is to use the PGCC or PGroupCC compiler and the KAI C++ to C translator. If you find that either PGroupCC or PGCC are not fully C++ compatible after all, KAI C++ most probably is. Links for the above: www.kai.com, www.pgroup.com, gcc.ml.org, egcs.cygnus.com ................................................................ : jakob@ostenfeld.dtu.dk : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: From fortuned@cuug.ab.ca Thu, 7 Jan 1999 18:20:37 -0500 Date: Thu, 7 Jan 1999 18:20:37 -0500 From: Doug Fortune fortuned@cuug.ab.ca Subject: more on Celeron SMP chip modification procedure Doug Fortune wrote: > Michael Hodgson wrote: > > > > Does anyone have performance info comparing floating point performance > > > on AMD K6 processors > > > with comparably-clocked P-IIs? How about Cyrix? > > > > > > For the myoglobin 1000 steps of MD benchmark CHARMM > > > > K6 233 - 4.49 hrs > > PPro 210 - 1.98 hrs > > 300MHz PII - 1.48 hrs > > > Dependant upon your application you might want to look at Celeron cpu's > > Particuarly the later ones with 128k full speed cache. > > They are: > > 1) cheap ( a 300a is around 60 ukp in the UK) > > 2) (overclockable if you are into that kind of thing) > > 3) and providing your application doesn't require oodles of cache, just as > > quick as a PII. > > > > For the same charmm benchmark. > > Celeron 300a - 1.49 hrs > > Celeron 300a (oc to 450MHz) - 0.99 hrs (compare this to an R10k time of > > 1.23hrs :) > > > > NOTE: Celerons like the origional K6 do not support SMP (well not without > > a LOT of fiddling), and so if it's multiple processors/board you are after > > then it still has to be PII's. > > Of course, there is the Celeron SMP modification, that variously allows you > to add a second cheap Celeron to your existing PII (dual motherboard of > course), > or go for two cheap Celerons off the bat. Duals said to be perfectly > reliable. > Many run much much faster. > > <> > > The original dual SMP Celeron modification article is at > > > > http://kikumaru.w-w.ne.jp/pc/celeron/index_e.html > > > > above abbreviated somewhat at: > >http://www.cpu-central.com/dualceleron/index-dc.html > > > > One of the guys on Linux-SMP is into overclocking his dual Celerons, > > you can see his story at > > > > http://www.psychosis.com/doa/ From colin@ddns.org Fri, 8 Jan 1999 03:17:51 -0500 Date: Fri, 8 Jan 1999 03:17:51 -0500 From: Colin Gan colin@ddns.org Subject: Scanning Accoustical Microscope on Beowulf cluster Hi, does anyone out there know or have any experience to set up Beowulf on Scanning Accoustical Microscope running volume rendering softwares on 3D images ? What will be the required softwares and hardwares ? Thanks in advance. -- Colin Gan Webworks Pte Ltd colin@webworks.com.sg 103A, Geylang Road Research Singapore 389212 http://www.webworks.com.sg Tel: (+65) 741-9526 Sales: sales@webworks.com.sg Fax: (+65) 749-3806 Tech : tech@webworks.com.sg Pgr: 9284-4823 Info : info@webworks.com.sg From makinb@ctc.com Fri, 8 Jan 1999 07:35:07 -0500 Date: Fri, 8 Jan 1999 07:35:07 -0500 From: Brian Makin makinb@ctc.com Subject: Scanning Accoustical Microscope on Beowulf cluster I do not know what a Snanning Accoustical Microscope is buy I do Virtual Reality and Volume Rendering. If you would like to explain your problem to me better I would be glad to render(pun intended) what assistance I could. Brian N. Makin makinb@ctc.com Colin Gan wrote: > Hi, does anyone out there know or have any experience to set up Beowulf on > Scanning Accoustical Microscope running volume rendering softwares on 3D > images ? What will be the required softwares and hardwares ? Thanks in > advance. > > -- > Colin Gan Webworks Pte Ltd > colin@webworks.com.sg 103A, Geylang Road > Research Singapore 389212 > > http://www.webworks.com.sg Tel: (+65) 741-9526 > Sales: sales@webworks.com.sg Fax: (+65) 749-3806 > Tech : tech@webworks.com.sg Pgr: 9284-4823 > Info : info@webworks.com.sg From cworley@altatech.com Fri, 8 Jan 1999 11:59:38 -0500 Date: Fri, 8 Jan 1999 11:59:38 -0500 From: Chris Worley cworley@altatech.com Subject: rack-mount cases Jeffrey Mark Siskind wrote: > > I'm looking to configure a 20 node (40 CPU) rack-mount compute-server cluster. > Ideally, it would fit in one rack. Ideally, it would be based on 2U cases. My > proposed configuration would be: Altatech ( http://www.altatech.com ) integrates Linux based clusters for both rack-mount (6U CPCI) and commodity (standard ATX motherboards in drawers bundled in 8 node clusters) enclosures. Would you be interested in that? I've not seen motherboards based on the BX chipset in a 2U format. The only motherboards I've seen that could fit in such a small area are the Corel netwinder boards based on the DEC strong-arm chip. There was a picture in last months Linux Journal of a ten-node hand-held home-brew beowulf cluster built by one of the Corel staff, shown at the Atlanta Linux Showcase. These chips don't do native floating point, but these motherboards have two built in 10/100 ethernet chipsets. Video, sound, etc are all built-in too. The whole cluster was powered by one ATX power supply. See: http://www.corelcomputer.com/products/linux/netwinder_dm.htm for a description (it doesn't show the Beowulf cluster, just the pre-packaged unit). > > - 440BX motherboard (100MHz front bus) > - dual 450MHz Pentium II with heat sink fans > - 512M ECC SDRAM (using 128M DIMMs) > - 100Mbit/s ethernet > - IBM 10G 5400rpm IDE UltraDMA disk drive > - 2U chassis > - 300W power supply > - no floppy, no CDROM, no SCSI card, no sound card, no video card Our systems do all of the above, except, they are not 2U (the commodity based cluster allows use of all PCI slots on a standard ATX motherboard). Also, we have one floppy per CPU. The floppy is an integral component, since it is part of the cloning system and allows disk-based and diskless booting. > > Ideally, the 3.5" disk drives would be mounted in caddys that fit in 5.25" > front-accessible bays so that I could easily swap them. I don't need hot-swap > capability. Just that I plan to install the OS by plugging the drives into a > caddy on another machine and doing a disk-to-disk copy. We have seemless cloning built into our systems. There is no need for disk swapping. > Can anybody give me feedback on any of the above vendors? We've (Alta Technology) been integrating Linux based clusters for over 1 year. Our largest systems are 128 processor (64 dual PII's) system (commodity enclosure) at Los Alamos and a 32 node AXP system (rack mount enclosure) in Japan. > > 2. I'm told that PCI cards can't fit in 2U cases. So I need a motherboard with > at least builtin 100Mb/sec ethernet. I'm know of motherboards that come with > various combinations of onboard ethernet, SCSI, video, and sound. I don't need > SCSI, video, and sound. Are there dual 440BX motherboards that have at least 4 > DIMM slots that have ethernet but no SCSI, video, and sound? What motherboard > do people recommend for a configuration like this? Our AXP systems have built-in ethernet, but they're 6U. Our commodity clusters support standard PCI boards, all slots available. > > 5. I plan to run without video. Which motherboards have BIOSes that can be > configured to run without video? I expect that from time to time (hopefully > infrequently) I will need to boot a node with a video card to change BIOS > settings. Ideally, the 2U case would be on sliders so that I could slide it > out and install a video card temporarily to change settings? Do any 2U cases > support this? If not, could I remove the 2U case from the rack, put it on a > table, open the top, and install a video card to change settings? Our commodity boards use the standard non-serial bios (these are meant to be commodity systems, so we have to use what's generally available), this is setup during our integration. We daisy-chain serial ports, which allows "back door" (terminal server style) access to any node to watch and control the boot process from the point when lilo starts. In our standard offering, only the root node has a video card. We also setup the system for root logins through the serial ports on all nodes. And, we setup the system so all boot I/O (including manual FSCK after a catastrophe) can be accomplished through the serial port. AXP boards do use a serial bios (milo), so the bios settings can be controlled via serial port access. In all configurations (rack and commodity), individual motherboards are on sliders. On the commodity systems, each motherboard has its own standard ATX power supply (in the same drawer as the motherboard), so can be "hot swapped" without bringing other nodes down. Hope this info helps, Chris From dfs@cacr.caltech.edu Fri, 8 Jan 1999 13:05:17 -0500 Date: Fri, 8 Jan 1999 13:05:17 -0500 From: Daniel F. Savarese dfs@cacr.caltech.edu Subject: CORBA, MPI, and Portland Group Compilers on a Beowulf >Do we need to use a free CORBA with sources, >or is there a commercial version compatible. I can't comment on Portland's C++, not having used it, the latest EGCS appears to finally be ANSI C++ compliant, incorporating namespaces, RTTI, and proper exception support. As for an open source ORB, I would recommend OmniORB, produced by Olivetti & Oracale Research. You can download it from http://www.orl.co.uk/omniORB/omniORB.html. It appears to be by far the fastest implementation. The only reason the GNOME project shunned it was because OmniORB does not support a C language binding (only C++) and they were dead set on avoiding C++ for reasons beyond my ken. Anyway, there are lots of ORBs to choose from. A decent starting point for finding more info on CORBA on Linux is http://linas.org/linux/corba.html daniel From joelja@darkwing.uoregon.edu Fri, 8 Jan 1999 13:51:54 -0500 Date: Fri, 8 Jan 1999 13:51:54 -0500 From: Joel Jaeggli joelja@darkwing.uoregon.edu Subject: rack-mount cases > > > > I'm looking to configure a 20 node (40 CPU) rack-mount compute-server cluster. > > Ideally, it would fit in one rack. Ideally, it would be based on 2U cases. My > > proposed configuration would be: about the smallest bx board your going to be able to get would be the fic kb-6120. http://www.fic.com.tw/motherboards/KB-6120/KB-6120_intro.htm the nlx form factor doesn't lend itself dual boards. > > > > - 440BX motherboard (100MHz front bus) > > - dual 450MHz Pentium II with heat sink fans > > - 512M ECC SDRAM (using 128M DIMMs) > > - 100Mbit/s ethernet > > - IBM 10G 5400rpm IDE UltraDMA disk drive > > - 2U chassis > > - 300W power supply > > - no floppy, no CDROM, no SCSI card, no sound card, no video card A kind of wierd hack for a 2u dual p2 exists which uses a full size atx board, see telenet systems website. http://www.tesys.com/enclosures/rackmount_telepro_201.shtml which only gives you two pci slots the case with power supply is like $350. I really have my doubts as to how well a really large dual atx motherboards with built in ethernet like the tyan s1836dluan > > > > 2. I'm told that PCI cards can't fit in 2U cases. So I need a motherboard with > > at least builtin 100Mb/sec ethernet. I'm know of motherboards that come with > > various combinations of onboard ethernet, SCSI, video, and sound. I don't need > > SCSI, video, and sound. Are there dual 440BX motherboards that have at least 4 > > DIMM slots that have ethernet but no SCSI, video, and sound? What motherboard > > do people recommend for a configuration like this? they do if you put them sideways (in the fashion of nlx motherboard designs, risers of indiviual pci slots do exist so that you can mount one or two cards sideways on a standard atx board. keep in mind than many pc bioses won't allow the system to boot without a video card so you need to assure yourself that one you pick can't assuming you don't actually puta video card in the system. > > > > > 5. I plan to run without video. Which motherboards have BIOSes that can be > > configured to run without video? I expect that from time to time (hopefully > > infrequently) I will need to boot a node with a video card to change BIOS > > settings. Ideally, the 2U case would be on sliders so that I could slide it > > out and install a video card temporarily to change settings? Do any 2U cases > > support this? If not, could I remove the 2U case from the rack, put it on a > > table, open the top, and install a video card to change settings? joelja -------------------------------------------------------------------------- Joel Jaeggli joelja@darkwing.uoregon.edu Academic User Services consult@gladstone.uoregon.edu PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E -------------------------------------------------------------------------- It is clear that the arm of criticism cannot replace the criticism of arms. Karl Marx -- Introduction to the critique of Hegel's Philosophy of the right, 1843. From brian@chpc.utah.edu Fri, 8 Jan 1999 15:23:49 -0500 Date: Fri, 8 Jan 1999 15:23:49 -0500 From: Brian Haymore brian@chpc.utah.edu Subject: rack-mount cases Chris Worley wrote: > > I've not seen motherboards based on the BX chipset in a 2U format. > > The only motherboards I've seen that could fit in such a small area are > the Corel netwinder boards based on the DEC strong-arm chip. There was > a picture in last months Linux Journal of a ten-node hand-held home-brew > beowulf cluster built by one of the Corel staff, shown at the Atlanta > Linux Showcase. These chips don't do native floating point, but these > motherboards have two built in 10/100 ethernet chipsets. Video, sound, > etc are all built-in too. The whole cluster was powered by one ATX power > supply. See: >From what I understand you can put an ASUS P2B or P2B-D motherboard in a 2U case. Unless I'm mistaken you can get some cards that plug into the pci slots that let you get up to two PCI cards horizontaly in a 2U case. -- Brian Haymore - Center for High Performance Computing, University of Utah Email: brian@chpc.utah.edu Fax: (801) 585-5366 Phone: (801) 585-1755 "I can please only one person a day. Today is not your day. Tomorrow isn't looking good either." --Anonymous From Ryan@carrera.com Fri, 8 Jan 1999 15:38:46 -0500 Date: Fri, 8 Jan 1999 15:38:46 -0500 From: Ryan Dunn Ryan@carrera.com Subject: 2U Rack-mount cases -- For Pent II Carrera Computers builds a 2U dual Pentium II system. You can even have SCSI, ENET and video, if you like. www.carrera.com Ryan Dunn From lucas@mercurio.econnect.com.br Fri, 8 Jan 1999 15:56:16 -0500 Date: Fri, 8 Jan 1999 15:56:16 -0500 From: Lucas do R. B. Brasilino da Silva lucas@mercurio.econnect.com.br Subject: FP on AMD K6 Hi Joe: K6, in overall performance, is faster than PII, but it FP unit is quite slow. Look at: http://www.gabrieltorres.com regards Lucas On Tue, 5 Jan 1999, Joe FergusonBoth types of receipt wrote: > Does anyone have performance info comparing floating point performance > on AMD K6 processors > with comparably-clocked P-IIs? How about Cyrix? > > Thanks, > > -- > Joe Ferguson ferguson.joseph@nesc.epa.gov > Lockheed Martin Corp. > NESC Support, EPA-RTP -- NC Voice: (919)-541-3716 > > > From diegert@cs.sandia.gov Fri, 8 Jan 1999 17:47:37 -0500 Date: Fri, 8 Jan 1999 17:47:37 -0500 From: Carl Diegert diegert@cs.sandia.gov Subject: 2U Rack-mount cases -- For Pent II > Subject: 2U Rack-mount cases -- For Pent II > Carrera Computers builds a 2U dual Pentium II > system. You can even have > SCSI, ENET and video, if you like. HP makes a 2xPII in 2U, too. No slides on this box, however. Check out the Compaq 1850R is you want to go 3U! The 1850R includes 100BT Ethernet, two channels of Ultra SCSI, an integrated display adapter, really _nice_ box and slides, etc., etc. We used 72 of these 1850R's to set the terabyte sort record last year, with 2 ea. 400 MHz P-II's in each, along with a second pair of SCSI controllers on a PCI card. From nav@pop.jaring.my Fri, 8 Jan 1999 21:02:22 -0500 Date: Fri, 8 Jan 1999 21:02:22 -0500 From: Khalid nav@pop.jaring.my Subject: Virtual Reality and Volume Rendering software on Beowulf cluster Dear Brian, Can you please inform me what software is available for Virtual Reality and Volume Rendering for a Beowulf cluster. Thanks in advance. Best regards. Khalid. From deadline@plogic.com Sun, 10 Jan 1999 10:35:57 -0500 Date: Sun, 10 Jan 1999 10:35:57 -0500 From: Douglas Eadline deadline@plogic.com Subject: MPICH stalls I can run the the nas parallel benchmarks using LAM all day without a single problem. However, If I recompile using MPICH, I find that the performance is less (as expected) and every once and a while the program just stalls - it sometimes finishes, but often it just dies. I know this was talked about before. I have investigated this over the weekend and found it happens on a number of systems: kernel: 2.0.35 SMP and UP three different NICs (INtel, 21143, 21140) with both fort77 and g77 The systems are Supermicro MBs P6DBE and P6DLE 266 PIIs to 400 Mhz. Has anyone seen anything like this? We use LAM almost exclusively, but I think it is good to have MPICH working. Also it seems the MPICH web page is down. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From deadline@plogic.com Mon, 11 Jan 1999 07:31:51 -0500 Date: Mon, 11 Jan 1999 07:31:51 -0500 From: Douglas Eadline deadline@plogic.com Subject: MPICH stalls On Sun, 10 Jan 1999, Douglas Eadline wrote: I also sent a copy of this to the MPICh group, here was their respoonse to the stalling problem: "We believe that this is caused by the LINUX TCP implementation chosing to close a connection when there is a lot of network traffic. This is acceptable TCP behavior - the defintion of "reliable" in TCP just isn't what you might think. To fix this, we'll need to recover from closed connections. We plan to make this fix at the same time that we redo the socket code in general, which should be later this year." Is there anyone who knows about the TCP implemetation that can commenton this? (Alan) Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From makinb@ctc.com Mon, 11 Jan 1999 11:55:40 -0500 Date: Mon, 11 Jan 1999 11:55:40 -0500 From: Brian Makin makinb@ctc.com Subject: Scanning Accoustical Microscope on Beowulf cluster Well, off the top of my head I do not know of any parallel volume rendering libraries. (although this would be a cool program, need lots of bandwidth, but I digress) There is a library called VTK (Visualization tool kit) That you could use to read in a volumetric dataset and do manipulations on it. ( I think it is at www.kitware.com ) If you have a SGI for a visualization node you could also run OpenGL volumizer. I am going to look through some things and see If I can find a free parallel volume rendering library. Brian N. Makin makinb@ctc.com Colin Gan wrote: > Hi, does anyone out there know or have any experience to set up Beowulf on > Scanning Accoustical Microscope running volume rendering softwares on 3D > images ? What will be the required softwares and hardwares ? Thanks in > advance. > > -- > Colin Gan Webworks Pte Ltd > colin@webworks.com.sg 103A, Geylang Road > Research Singapore 389212 > > http://www.webworks.com.sg Tel: (+65) 741-9526 > Sales: sales@webworks.com.sg Fax: (+65) 749-3806 > Tech : tech@webworks.com.sg Pgr: 9284-4823 > Info : info@webworks.com.sg From cec@ee.duke.edu Mon, 11 Jan 1999 16:44:43 -0500 Date: Mon, 11 Jan 1999 16:44:43 -0500 From: Christopher Cramer cec@ee.duke.edu Subject: MPI vs PVM Not to start any religious arguments, but can anyone point me to a benchmark suite that will compare MPI (any implementation) and PVM performance on a given system? Thanks in advance. -Chris ----------------------------------------------------------------------- Chris Cramer, PhD - "The world is my country, all mankind are my cec@ee.duke.edu - brethren, and to do good is my religion." -Thomas Paine From dpx@acl.lanl.gov Mon, 11 Jan 1999 18:50:40 -0500 Date: Mon, 11 Jan 1999 18:50:40 -0500 From: Dean Prichard dpx@acl.lanl.gov Subject: MPICH stalls We see the same thing, very reproducable w/ the more communication intensive NAS benchmarks like "CG". When in hangs you can look at the socket w/ netstat -eo and you will most likely find some that are retrying: peak04> netstat -eo ... tcp 0 28732 peak04.acl.lanl.go:1039 peak05.acl.lanl.go:1042 ESTABLISHED dpx on (94.83/9) peak05> netstat -eo tcp 27544 28992 peak05.acl.lanl.go:1042 peak04.acl.lanl.go:1039 ESTABLISHED dpx on (47.81/9) On Sun, 10 Jan 1999, Douglas Eadline wrote: > Date: Sun, 10 Jan 1999 10:34:33 -0500 (EST) > From: Douglas Eadline > To: beowulf > Subject: MPICH stalls > > > I can run the the nas parallel benchmarks using LAM all day without > a single problem. However, If I recompile using MPICH, I find that > the performance is less (as expected) and every once and a while > the program just stalls - it sometimes finishes, but often it just dies. > > I know this was talked about before. I have investigated this over the > weekend and found it happens on a number of systems: > > kernel: 2.0.35 SMP and UP > three different NICs (INtel, 21143, 21140) > with both fort77 and g77 > The systems are Supermicro MBs P6DBE and P6DLE 266 PIIs to 400 Mhz. > > Has anyone seen anything like this? > > We use LAM almost exclusively, but I think it is good to have > MPICH working. > > > Also it seems the MPICH web page is down. > > > Doug > ------------------------------------------------------------------- > Paralogic, Inc. | PEAK | Voice:+610.861.6960 > 115 Research Drive | PARALLEL | Fax:+610.861.8247 > Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com > ------------------------------------------------------------------- > From rgb@phy.duke.edu Tue, 12 Jan 1999 08:28:12 -0500 Date: Tue, 12 Jan 1999 08:28:12 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: MPI vs PVM On Mon, 11 Jan 1999, Christopher Cramer wrote: > > Not to start any religious arguments, but can anyone point me to a > benchmark suite that will compare MPI (any implementation) and PVM > performance on a given system? I'm not sure that it has exactly what you want, but there is a very nice white paper on the netlib PVM site that compares PVM and MPI both philosophically and historically. I cannot recall for certain but it may also contain benchmarks. However, I don't think that there are strong performance reasons for or against either one -- I think that it is more a question of what kind of parallel program you are writing and whether it is to be run on a heterogeneous or homogeneous or dedicated environment. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From mdw@cs.berkeley.edu Tue, 12 Jan 1999 12:16:27 -0500 Date: Tue, 12 Jan 1999 12:16:27 -0500 From: Matt Welsh mdw@cs.berkeley.edu Subject: O'Reilly Linux Conference Call for Presenters Hello, I'm the program chair for the O'Reilly Linux Conference (August 21-24, 1999 in Monterey, California). We're looking for talks and tutorials from Linux developers and enthusiasts to present at this conference, and I'd really like to encourage folks on these mailing lists to contribute. All of the information is at http://conferences.oreilly.com or by e-mail at linuxextracts@oreilly.com. You can e-mail me if you have any questions or need other details. Thanks much! Matt Welsh, mdw@cs.berkeley.edu University of California, Berkeley +1 510 643 7566 (voice) --- Announcement and Call for Invited Talks Linux Conference August 21-24, 1999 Monterey, California http://conferences.oreilly.com Sponsored by O'Reilly & Associates Co-sponsored by Linux International (Confirm with maddog) Invited Talks Committee Matt Welsh, Chair, University of California, Berkeley Jon "Maddog" Hall, Linux International Andy Oram, O'Reilly & Associates Greg Hankins, Georgia Tech Russ Nelson, Crynwr Software Erik Troan, Red Hat Software Overview The O'Reilly Linux Conference will be held August 21-24, 1999 at the Monterey Conference Center in Monterey, California. There will be two days of tutorials followed by a two-day, multi-track conference including sessions for submitted invited presentations on practical and experimental uses of Linux; daily Q and A sessions with leading Linux developers, and evening breakaway sessions for special interest groups. Practical Presentations, Talks, and Panels This is not a traditional solicitation for academic papers. We seek presenters for talks and panels that demonstrate the diversity and strength of Linux. In the practical spirit of Linux, this means not just showing the clever and interesting ways you use Linux, but how your experience and code can help others. We're interested in large stories, small stories, silly hacks, case studies from the trenches ("Introducing Linux in an NT Shop"), philosophical perspectives ("Can Linux replace NT?") and even more traditional computer science pieces ("Distributed Computing with Linux"). We welcome presentations on every aspect of Linux, from new applications to case studies of Linux at work to panels. If you have a use for Linux that saves time, money, and headaches for your and your organization, we would like to hear about it. In short, we encourage submissions that highlight Linux's features and benefits. Some suggestions for talks follow --- but are not limited to these topics. If you've got an idea that will benefit the Linux community, please let us know. * Kernel development and device drivers * Networking and communications * Databases, data mining, and storage management * System and network administration * Programming environments (C, C++, Java, Perl, etc.) * Graphical User Interface toolkits (X11, GTK, KDE, GNOME, ...) * Ports to non-Intel architectures (SPARC, Alpha, PowerPC, ...) * World Wide Web and Internet applications * High-performance and parallel computing (Beowulf, Extreme Linux, ...) * Experiences with Linux: Using Linux for WWW, databases, large installations, enterprise applications, etc. * Philosophical musings on the future and role of Linux What, How, and Where to Submit Speakers should submit an abstract (250 words) and a detailed outline of their talk. The abstract and outline should describe what your talk will be about, and be specific about problems, solutions, and conclusions. Both the abstract and outline will be used together to evaluate talks. All submissions will be held in confidence. Talks that do not include both an abstract and an outline will not be considered. Important deadlines: Submissions: February 15, 1999 Acceptances: February 22, 1999 Camera-ready presentations: June 30, 1999 Each submission must include: 1. An initial page with the: --complete title of the presentation --name and affiliation of a speaker who will be the primary contact --that person's complete contact information including phone, fax, email, postal address, --The names of all other speakers with their affiliations and email addresses B. An abstract as detailed above C. A detailed outline Abstracts should be sent to linuxabstracts@oreilly.com. Email inquiries should be sent to linuxextracts@oreilly.com. Registration Information Complete conference and registration information will be available in mid-April. Keep checking the conference web site for the latest information: http://conferences.oreilly.com/ About O'Reilly & Associates Sponsor of Geekfest, the OpenSource Conference, O'Reilly & Associates is the leading publisher of books for UNIX, X, the Internet, and other open systems, as well as a pioneer in on-line publishing. We also publish the leading web server for Windows NT and Windows 95, and are defining new ways to develop and sell software on and over the Internet. For more information about O'Reilly & Associates, visit our web site: http://www.oreilly.com/. O'Reilly & Associates 101 Morris Street Sebastopol, California 95472 707/829-0515 800/998-9938 From jozef@fatra.ph.hunter.cuny.edu Tue, 12 Jan 1999 13:47:45 -0500 Date: Tue, 12 Jan 1999 13:47:45 -0500 From: Jozef Skvarcek jozef@fatra.ph.hunter.cuny.edu Subject: Hardware questions Hi, I am going to setup very small cluster for our department and I would like to clarify few problems before we start ordering hardware. Perhaps someone can help... 1. Will a machine without keyboard, mouse and video card boot? Let's assume it has Asus P2B (P2L resp.) motherboard. If the positive answer depends on the MB or BIOS can you suggest type? 2. If we change PII 400MHz (100MHz bus) to Celeron 400MHz (66MHz bus) on the worker nodes, will the performace drop `dramatically' i.e. more than if changing PII 400 to PII 350 (100MHz bus)? 3. The cluster nodes will be hooked to a fast ethernet switch (I am thinking about SMC EZSwitch 100, 8ports since I was able to found the most comprehensive info about it) that will be connected to our network (10BT), obviously, I do not want to flood the cluster with other then cluster's traffic. My understanding is that the switch will not propagate such packets. Am I right or do I need to put the cluster behind firewall? Thank you, Jozef Skvarcek jskvarce@shiva.hunter.cuny.edu From Christopher.Bohn@afit.af.mil Tue, 12 Jan 1999 14:54:45 -0500 Date: Tue, 12 Jan 1999 14:54:45 -0500 From: Bohn, Christopher A Christopher.Bohn@afit.af.mil Subject: Hardware questions Good day, > -----Original Message----- > From: Jozef Skvarcek [SMTP:jozef@fatra.ph.hunter.cuny.edu] > Sent: Tuesday, January 12, 1999 1:52 PM > To: beowulf@cesdis1.gsfc.nasa.gov > Subject: Hardware questions > [...] > 1. Will a machine without keyboard, mouse and video card boot? > Let's assume it has Asus P2B (P2L resp.) motherboard. If the positive > answer depends on the MB or BIOS can you suggest type? [Bohn, Christopher A.] I know from personal experience that late-model PCs will (but my old 8088 machine wouldn't). The nodes in our cluster have successfully booted without a keyboard or mouse plugged-in. We didn't try removing the video controller, though. > 2. If we change PII 400MHz (100MHz bus) to Celeron 400MHz (66MHz bus) > on the worker nodes, will the performace drop `dramatically' i.e. > more than if changing PII 400 to PII 350 (100MHz bus)? [Bohn, Christopher A.] "It depends on your application." That said, I would expect that, yes, you would see worse performance with the 400MHz Celeron than a 350MHz PII. The difference in processor speed would most likely be dominated by loss in memory performance. The 100MHz bus, obviously, is 50% faster than the 66MHz bus. So unless your problem fits entirely within cache, that's going to be a sever hamper on the 14% faster processor core in the 400MHz Celeron over the 350MHz PII. And, the L2 cache on the Celeron is only 128KB, as opposed to 512KB on the PII. That said, IIRC, the Celeron's L2 cache is clocked at full-speed instead of half-speed. > 3. The cluster nodes will be hooked to a fast ethernet switch (I am > thinking about SMC EZSwitch 100, 8ports since I was able to found > the most comprehensive info about it) that will be connected to our > network (10BT), obviously, I do not want to flood the cluster with > other then cluster's traffic. My understanding is that the switch > will not propagate such packets. Am I right or do I need to put > the cluster behind firewall? [Bohn, Christopher A.] Dunno, but I doubt the switch would pass along any packets not directed to its subnet (assuming you configure it to "know" its subnet). Of course, if the switch even has to look at the packets from the outside world, that takes up some of the switch's aggregate capacity. Our solution was to have one of our front-ends connected to the outside world and to the switch, rather than connecting the switch to the outside world. Of course, this was also necessary since our connection to the outside world was 10B2, and neither our switch, nor the hub that preceded it, have a coax port. Hope that helps. Take care, cb *-*-*-*-*-*-*-* Capt Christopher A. Bohn Graduate Student, Electrical (digital) Engineering Air Force Institute of Technology Phone (937)255-3636 (DSN 785) AFIT/EN638 Lab x4606 Voicemail x6638 2950 P St, Box 4638 email Christopher.Bohn@afit.af.mil Wright-Patterson AFB OH 45433-7765 EngrBohn@aol.com http://members.aol.com/EngrBohn/ *-*-*-*-*-*-*-* > From fortuned@cuug.ab.ca Tue, 12 Jan 1999 15:12:04 -0500 Date: Tue, 12 Jan 1999 15:12:04 -0500 From: Doug Fortune fortuned@cuug.ab.ca Subject: Hardware questions / 'very small networks' Jozef Skvarcek wrote: > I am going to setup very small cluster for our department and I would > like to clarify few problems before we start ordering hardware. > 3. The cluster nodes will be hooked to a fast ethernet switch (I am > thinking about SMC EZSwitch 100, 8ports. The networking might depend somewhat on the size of 'very small', and if no further expansion is envisioned. For myself, 'very small' means five dual PII boxes. ie One Master and Four Slaves (ten total cpu's) My networking is to consist of one Adaptec/Cogent 4 port 100 Mbit ethernet in the Master, and one 100Mbit card in each of the 4 Slaves. Thus the peak transfer rate to & from the Master is ~800 Mbits/sec, (theoretical) using duplex mode, four cross-over cables, no expensive switches, and no collisions (vs perhaps ~200 Mbits/sec max with a switch, reduced by any ethernet collisions). The cost is likely to be less than a switch, but delivering higher performance. For 'very small' Beowulfs, this could be extended to perhaps 4 quad port cards in the Master (for a total of 16 machines, or 8 channel-bonded at double the rate). Doug Fortune Calgary From joelja@darkwing.uoregon.edu Tue, 12 Jan 1999 16:58:17 -0500 Date: Tue, 12 Jan 1999 16:58:17 -0500 From: Joel Jaeggli joelja@darkwing.uoregon.edu Subject: Hardware questions On Tue, 12 Jan 1999, Jozef Skvarcek wrote: > Hi, > > I am going to setup very small cluster for our department and I would > like to clarify few problems before we start ordering hardware. > Perhaps someone can help... > > 1. Will a machine without keyboard, mouse and video card boot? > Let's assume it has Asus P2B (P2L resp.) motherboard. If the positive > answer depends on the MB or BIOS can you suggest type? Most current board can be made to boot without kayboard and mouse. I'm not directly familiar with pc boards with serial bioses (ie that can boot without video card). another thing to note is that in my understanding not all bx or lx boards support a 6x clock double (6*66=396) so if you chose celeron 366 or 400 it may limit you choice of boards to one that does, such as the abit bm6 which has the socket 370 instead of socket 1: http://www.abit.com.tw/html/bm6.htm because even standard bx boards like abit bh6 top out at a 5.5x clock multiple. since celerons and pII's are clokc locked to the appropiate speed getting the right motherboard is a real issue with the new celerons... joelja > 2. If we change PII 400MHz (100MHz bus) to Celeron 400MHz (66MHz bus) > on the worker nodes, will the performace drop `dramatically' i.e. > more than if changing PII 400 to PII 350 (100MHz bus)? depends on how memory bound you're application is, your stream benchmark scores (per host) will probably be 55-70% of what it would be on 100mhz fsb. again that might or might not be an issue with your particular situation.. http://www.cs.virginia.edu/stream/ > 3. The cluster nodes will be hooked to a fast ethernet switch (I am > thinking about SMC EZSwitch 100, 8ports since I was able to found > the most comprehensive info about it) that will be connected to our > network (10BT), obviously, I do not want to flood the cluster with > other then cluster's traffic. My understanding is that the switch > will not propagate such packets. Am I right or do I need to put > the cluster behind firewall? correct it wouldn't propigate, however in my experience (and our cluster) people typically have one node with two ethernet cards one connected to the outside world, the other connected to the switch and no packet forwarding so that they have a disconnected private network for all the nodes. that allows you to do things you wouldn't otherwise do on your regular network like use rsh, set-uid read/write nfs etc for the compastive security of a network which has no route to the outside world. joelja > Thank you, > > Jozef Skvarcek > jskvarce@shiva.hunter.cuny.edu > -------------------------------------------------------------------------- Joel Jaeggli joelja@darkwing.uoregon.edu Academic User Services consult@gladstone.uoregon.edu PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E -------------------------------------------------------------------------- It is clear that the arm of criticism cannot replace the criticism of arms. Karl Marx -- Introduction to the critique of Hegel's Philosophy of the right, 1843. From andy@meshplc.co.uk Tue, 12 Jan 1999 19:28:48 -0500 Date: Tue, 12 Jan 1999 19:28:48 -0500 From: Andy andy@meshplc.co.uk Subject: Hardware questions Very few motherboards that I have seen will boot without some video device or other. I have several Asus P2B based systems, and for those without the need for monitors, I just used old video cards (ISA 256K Trident, S3 PCI 1MB etc). It will boot quite happily without a keyboard though, as long as the bios is set to ignore k/b errors (this is true of almost all motherboards from 486 onward) Andy MESH Computers Webmaster andy@meshplc.co.uk From rich@freebsd.org Tue, 12 Jan 1999 19:40:57 -0500 Date: Tue, 12 Jan 1999 19:40:57 -0500 From: Murphey rich@freebsd.org Subject: Hardware questions >Date: Tue, 12 Jan 1999 13:51:52 -0500 (EST) >From: Jozef Skvarcek > >Hi, > >I am going to setup very small cluster for our department and I would >like to clarify few problems before we start ordering hardware. >Perhaps someone can help... > >1. Will a machine without keyboard, mouse and video card boot? >Let's assume it has Asus P2B (P2L resp.) motherboard. If the positive >answer depends on the MB or BIOS can you suggest type? I havne't tried the P2B, but the B2B-DS will indeed boot without keyboard, mouse and video card. However, I don't know whether there's a way to setup the bios without a vido card. I used a video card till the bios, etc., was configured. The rest of the setup can be handled by a serial console I believe. >2. If we change PII 400MHz (100MHz bus) to Celeron 400MHz (66MHz bus) >on the worker nodes, will the performace drop `dramatically' i.e. >more than if changing PII 400 to PII 350 (100MHz bus)? It makes a difference if the application's working set is small enough to fit in cache on the PII but not on the Celeron. If you've got a very tiny or a huge working set it might not make as much of a difference. >3. The cluster nodes will be hooked to a fast ethernet switch (I am >thinking about SMC EZSwitch 100, 8ports since I was able to found >the most comprehensive info about it) that will be connected to our >network (10BT), obviously, I do not want to flood the cluster with >other then cluster's traffic. My understanding is that the switch >will not propagate such packets. Am I right or do I need to put >the cluster behind firewall? You could either configure the switch as a router or use one of the nodes as a router. Either would cut down on the traffic through the switch. It would also help elliminate collisions due to propagation of broadcast packets if I'm not mistaken. Rich Murphey >Thank you, > >Jozef Skvarcek >jskvarce@shiva.hunter.cuny.edu > > > From qobi@research.nj.nec.com Tue, 12 Jan 1999 19:44:17 -0500 Date: Tue, 12 Jan 1999 19:44:17 -0500 From: Jeffrey Mark Siskind qobi@research.nj.nec.com Subject: TreadMarks 1. Does anybody know if TreadMarks runs under Linux? 2. If so, how does one obtain a copy? The Web page http://www.cs.rice.edu/~willy/TreadMarks/overview.html says that universities and non-profit institutions should contact tmk@cs.rice.edu. I sent email to that address and it bounced. And the Web page says that commercial institutions should contact ParallelTools L.L.C. 281-398-7035. But the like is stale and the phone number doesn't work. Jeff (http://www.neci.nj.nec.com/homepages/qobi) From deadline@plogic.com Tue, 12 Jan 1999 19:53:02 -0500 Date: Tue, 12 Jan 1999 19:53:02 -0500 From: Douglas Eadline deadline@plogic.com Subject: Hardware questions / 'very small networks' On Tue, 12 Jan 1999, Doug Fortune wrote: > For myself, 'very small' means five dual PII boxes. > ie One Master and Four Slaves (ten total cpu's) > > My networking is to consist of one Adaptec/Cogent 4 port 100 Mbit > ethernet in the Master, and one 100Mbit card in each of the 4 Slaves. > > Thus the peak transfer rate to & from the Master is ~800 Mbits/sec, > (theoretical) using duplex mode, four cross-over cables, no expensive > switches, and no collisions (vs perhaps ~200 Mbits/sec max with a > switch, > reduced by any ethernet collisions). > > The cost is likely to be less than a switch, but delivering higher > performance. If you have a master/slave algorithm. If the slaves need to talk to each other, you have to hop through the master and take a big performance hit. There are quite a few algorithms that require data exchanges. This arrangement would work quite well for the data flow model used in our BERT Fortran conversion tool. We also support other models that exchange data. What model is best? It all depends on the application, the CPU, compiler, and the communications network. Not an easy question to answer. > > For 'very small' Beowulfs, this could be extended to perhaps 4 quad > port cards in the Master (for a total of 16 machines, or 8 > channel-bonded at > double the rate). Can you really fill all 16 pipes at the same time? I know using U-net it has been done for 4, but I'm not sure if your could run 16 interfaces flat out through tcp/ip. You may have slaves waiting. Has anyone tried this? Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From jhpark@nurapt.kaist.ac.kr Tue, 12 Jan 1999 19:57:43 -0500 Date: Tue, 12 Jan 1999 19:57:43 -0500 From: Jeong Hwan Park jhpark@nurapt.kaist.ac.kr Subject: Hardware questions > > 1. Will a machine without keyboard, mouse and video card boot? > Let's assume it has Asus P2B (P2L resp.) motherboard. If the positive > answer depends on the MB or BIOS can you suggest type? ASUS p2b can do work without keyboard, mouse, video card. I think Asus p2b is best choice. From elaine@bann.com Tue, 12 Jan 1999 20:10:19 -0500 Date: Tue, 12 Jan 1999 20:10:19 -0500 From: Elaine Bann elaine@bann.com Subject: HOw do I get off this mailing list? I would really like to get off this mailing list. How do I go about doing that? -elaine From efinch@cais.com Tue, 12 Jan 1999 21:35:52 -0500 Date: Tue, 12 Jan 1999 21:35:52 -0500 From: Ed Finch efinch@cais.com Subject: [Fwd: Please help: DHCP, Red Hat 5.2 and Kickstart] Greetings! I'm trying to use Red Hat 5.2's Kickstart process to build my Beowulf cluster. DHCP seems to negotiate sucesssfully, but I can't an alternate kickstart filename passed to the client. In fact, the client always looks for 10.0.0.200:/kickstart /etc/dhcpd.conf looks like this: default-lease-time 300; option subnet-mask 255.255.255.0; option broadcast-address 10.0.0.255; option routers 10.0.0.200; option domain-name-servers 10.0.0.200; option domain-name "hitc.com"; filename "/redhat-5.2/cdrom/dhcp.kickstart"; subnet 10.0.0.0 netmask 255.255.255.0 { range 10.0.0.1 10.0.0.100; } and /redhat-5.2/cdrom/dhcp.kickstart looks like: leng en network --bootproto dhcp nfs --server 10.0.0.200 --dir /redhat-5.2 keyboard us zerombr yes clearpart --all part /boot --size 10 part swap --size 127 part / --size 1 --grow timezone US/Eastern rootpw --iscrypted XwOmhs6zTSKok lilo %packages @ Base @ Networked Workstation @ C Development @ Development Libraries @ C++ Development Beowulf is waiting - please help! :-) Best regards, Ed Q: Why do PCs have a reset button on the front? A: Because they are expected to run Microsoft operating systems. From efinch@cais.com Tue, 12 Jan 1999 21:36:39 -0500 Date: Tue, 12 Jan 1999 21:36:39 -0500 From: Ed Finch efinch@cais.com Subject: [Fwd: Exciting New Casino] -------- Original Message -------- Received: from smtp.interlog.com (root@smtp.interlog.com [207.34.202.37])by brap-1.cais.net (8.9.1/8.9.1) with ESMTP id VAA11703for ; Tue, 12 Jan 1999 21:30:26 -0500 (EST) From: mikey9811@yahoo.com Received: from yahoo.com (209-20-45-154.dialin.interlog.com [209.20.45.154])by smtp.interlog.com (8.9.1/8.9.1) with SMTP id VAA27176;Tue, 12 Jan 1999 21:29:41 -0500 (EST) Date: Tue, 12 Jan 1999 21:36:39 -0500 Message-Id: <199901130229.VAA27176@smtp.interlog.com> To: mikey9811@yahoo.com Subject: Exciting New Casino X-Mozilla-Status: 8001 X-Mozilla-Status2: 00000000 X-UIDL: 6fb334517077bed450f02663584cf11e Dear Sir or Madam, Please excuse this unsolicted email, but we thought you like to know that one of the nets premiere on-line casinos is giving away $50.00 to every customer who signs up now. Visit http://www.worldcasino.mk.ua for all the details. Enjoy great casino games, including BlackJack, Roulette, Craps, Slots, Video Poker and more and get $50.00 just for trying it out. Thank for bearing with us. Please replay with subject "remove" to be removed from this mailing list From efinch@cais.com Tue, 12 Jan 1999 21:36:15 -0500 Date: Tue, 12 Jan 1999 21:36:15 -0500 From: Ed Finch efinch@cais.com Subject: [Fwd: Re: Please help: DHCP, Red Hat 5.2 and Kickstart] Ed Finch (efinch@yellow.vais.net) wrote: : Greetings! : I'm trying to use Red Hat 5.2's Kickstart process to build : my Beowulf cluster. DHCP seems to negotiate sucesssfully, but : I can't an alternate kickstart filename passed to the client. : In fact, the client always looks for 10.0.0.200:/kickstart Oh yeah: I'm using these DHCP rpms from the 5.2 set, dhcpcd-0.70-2 dhcp-2.0b1pl6-2 and the filesystem is exported: ls -l /net/beowulf/redhat-5.2/cdrom/dhcp.kickstart -rw-r--r-- 1 root root 337 Jan 8 19:06 dhcp.kickstart -- Ed Q: Why do PCs have a reset button on the front? A: Because they are expected to run Microsoft operating systems. From rgb@phy.duke.edu Wed, 13 Jan 1999 00:00:28 -0500 Date: Wed, 13 Jan 1999 00:00:28 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: Hardware questions On Tue, 12 Jan 1999, Jozef Skvarcek wrote: > Hi, > > I am going to setup very small cluster for our department and I would > like to clarify few problems before we start ordering hardware. > Perhaps someone can help... > > 1. Will a machine without keyboard, mouse and video card boot? > Let's assume it has Asus P2B (P2L resp.) motherboard. If the positive > answer depends on the MB or BIOS can you suggest type? Without keyboard and mouse, certainly. I've never tried to boot one without a video card because (at maybe $30) they are so cheap and useful, but I understand that it is possible. Perhaps somebody else can verify this. > 2. If we change PII 400MHz (100MHz bus) to Celeron 400MHz (66MHz bus) > on the worker nodes, will the performace drop `dramatically' i.e. > more than if changing PII 400 to PII 350 (100MHz bus)? The Celeron is a PII with smaller cache. It makes up for the smaller cache to some extent by virtue of the cache being faster. Although in principle the memory bus should matter, for many programs you won't see any difference. I'd even say "most" programs, or "nearly all" programs, but on this list I know that there are people for whom it matters very much and they'll jump on me;-). In my programs (the only ones I really care about:-) I see very little besides CPU clock speed scaling from 200 MHz Pentium Pros running EDO memory through to 450 MHz PII's running registered PC100 SDRAM, and my 300 MHz Celeron system benchmarks out to with a hair of my 300 MHz PIIs. If your application is KNOWN to be memory I/O bound is the right size for cache size/stride to be right for a real PII but wrong for a Celeron, it will matter. The safest thing to do (if you can afford it) is to get one of each and benchmark YOUR code on it. You can probably get by just fine with the Celeron and save a lot of money. If not, you can either pull the Celeron and replace it with a PII (and write off the cost to development) or use the system for your beowulf's "head". Or something. I'm assuming that you're getting a real 400 MHz Celeron, btw, although reportedly the Celeron 300 overclocks to 400+ MHz with nearly 100% reliability. The "nearly 100%" is OK for games but worrisome for calculations containing maybe 10^18 floating point operations, to me at least. > 3. The cluster nodes will be hooked to a fast ethernet switch (I am > thinking about SMC EZSwitch 100, 8ports since I was able to found > the most comprehensive info about it) that will be connected to our > network (10BT), obviously, I do not want to flood the cluster with > other then cluster's traffic. My understanding is that the switch > will not propagate such packets. Am I right or do I need to put > the cluster behind firewall? The switch will not propagate the internal traffic between beowulf nodes and the rest of the network - a switch only forwards packets to a known host to the port on which the known host resides and allows symmetric communications between all nodes. If this concerns you, though, there are other solutions and topologies available. For example, I wouldn't call it a "firewall" but it is very easy to put an extra NIC in the system you want to call your "head" node, attach it to your department LAN, and use the switch only "inside" between nodes with no direct link to the LAN at all. This approach conserves IP numbers -- you can give all the internal hosts IP numbers from 192.168.x.x, for example, which is a block of numbers intended for use in "private internal networks" that are guaranteed by a RFC not to pass a router, and access nodes by logging into the head node first. Or, of course, you can give the nodes real IP numbers from your space and make the head into a real (but small) router so that you can login to each node from anywhere on your LAN (while internode traffic remains strictly local). You can also skip the switch altogether and put multiple NICs in each host and connect them in a hypercube, and there are undoubtedly still more possibilities. Some will make more sense than others (either economically or in terms of performance) -- only after the task to be run on the system is well understood will a given topology be "obviously" the best price performer. It's hard to go wrong with a simple switched network, though -- I'm not trying to talk you out of a switch if you can afford it. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From masud@googgun.com Wed, 13 Jan 1999 00:45:53 -0500 Date: Wed, 13 Jan 1999 00:45:53 -0500 From: Ahmed Masud masud@googgun.com Subject: HOw do I get off this mailing list? On Tue, 12 Jan 1999, Elaine Bann wrote: > I would really like to get off this mailing list. How do I go about doing > that? > -elaine Send email to majordomo@cesdis1.gsfc.nasa.gov to get help on how to get off the mailing list. You have to send a message to majordomo with the correct command (i believe it's "unsubscribe") and do it from the email address that you actually subscribed to the list from. Hope this helps Ahmed From daniel.pfenniger@obs.unige.ch Wed, 13 Jan 1999 03:37:03 -0500 Date: Wed, 13 Jan 1999 03:37:03 -0500 From: PFENNIGER Daniel daniel.pfenniger@obs.unige.ch Subject: Hardware questions Andy wrote: > > Very few motherboards that I have seen will boot without some video device > or other. I have several Asus P2B based systems, and for those without the > need for monitors, I just used old video cards (ISA 256K Trident, S3 PCI > 1MB etc). > > It will boot quite happily without a keyboard though, as long as the bios > is set to ignore k/b errors (this is true of almost all motherboards from > 486 onward) With an installed Linux on an Asus P2B board one can also remove the video card, beside the keyboard and the mouse. The system reboots correctly (I tried). It can then be reached by the network. Of course in case of problem it is more convenient to have a video screen, but much could be done through a serial console (an option in the Linux kernel install). Sparing on cheap video cards was however considered not worth for our cluster. Daniel Pfenniger From mark@rrsg.ee.uct.ac.za Wed, 13 Jan 1999 05:14:38 -0500 Date: Wed, 13 Jan 1999 05:14:38 -0500 From: Mark Gebhardt mark@rrsg.ee.uct.ac.za Subject: PVM and SMP/non-SMP Linux] This is a multi-part message in MIME format. --------------0BA20AABAB41085E42C7C424 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi all I am running PVM3.4.beta7 on Redhat 5.2 Linux. The hardware I am using consists of 7 single-CPU PIIs and one dual-processor PII (this machine has Linux recompiled for SMP). If I run PVM apps on the dual processor machine by itself, I find that both processors get used, but if I include another machine in the VM, the CPU utilisation (via top) on the parallel machine is approx. the same as on the single procesor machine. Does anyone know why this happens and if there is any way of using the dual-processor machine more fully? Thanks Mark -- Mark Gebhardt Radar Remote Sensing Group University of Cape Town South Africa tel: +27 21 6503756 email: mark@rrsg.ee.uct.ac.za www: http://rrsg.ee.uct.ac.za --------------0BA20AABAB41085E42C7C424 Content-Type: message/rfc822 Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-ID: <3699DDD3.5ED5A6EA@rrsg.ee.uct.ac.za> Date: Wed, 13 Jan 1999 05:14:38 -0500 From: Mark Gebhardt Organization: RRSG X-Mailer: Mozilla 4.07 [en] (X11; I; Linux 2.0.36 i686) MIME-Version: 1.0 Newsgroups: comp.parallel.pvm Subject: PVM and SMP/non-SMP Linux Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi all I am running PVM3.4.beta7 on Redhat 5.2 Linux. The hardware I am using consists of 7 single-CPU PIIs and one dual-processor PII (this machine has Linux recompiled for SMP). If I run PVM apps on the dual processor machine by itself, I find that both processors get used, but if I include another machine in the VM, the CPU utilisation (via top) on the parallel machine is approx. the same as on the single procesor machine. Does anyone know why this happens and if there is any way of using the dual-processor machine more fully? Thanks Mark --------------0BA20AABAB41085E42C7C424-- From mprinkey@champion.org Wed, 13 Jan 1999 05:44:19 -0500 Date: Wed, 13 Jan 1999 05:44:19 -0500 From: Michael Prinkey mprinkey@champion.org Subject: Hardware questions I can't add anything to further answer these questions, but they do raise another issue. Several months ago, I think one of the BIOS vendors (Award?) was toying with the idea of serial support. Does any current x86 BIOS include serial support? I just set up several headless Alpha systems using the serial console to AlphaBIOS, and I was quite impressed with the ability to change low-level settings without dragging a monitor and keyboard to each system. Cheers, Mike Prinkey CFD Group West Virginia University ---------- > From: Jozef Skvarcek > To: beowulf@cesdis1.gsfc.nasa.gov > Subject: Hardware questions > Date: Tuesday, January 12, 1999 1:51 PM > > Hi, > > I am going to setup very small cluster for our department and I would > like to clarify few problems before we start ordering hardware. > Perhaps someone can help... > > 1. Will a machine without keyboard, mouse and video card boot? > Let's assume it has Asus P2B (P2L resp.) motherboard. If the positive > answer depends on the MB or BIOS can you suggest type? From whately@cos.ufrj.br Wed, 13 Jan 1999 07:23:07 -0500 Date: Wed, 13 Jan 1999 07:23:07 -0500 From: Lauro Whately whately@cos.ufrj.br Subject: TreadMarks Yes, TreadMarks runs under linux. We've been using it with linux on intel and ppc machines. We bought one license from Rice some years ago. I think you should try the people in the university anyway. -- Lauro Whately Parallel Computing Lab. / COPPE Federal University of Rio de Janeiro Brazil ===================================== "A distributed system is one on which I cannot get any work done, because a machine I have never heard of has crashed." -- Leslie Lamport From mkh100@york.ac.uk Wed, 13 Jan 1999 08:24:31 -0500 Date: Wed, 13 Jan 1999 08:24:31 -0500 From: Michael Hodgson mkh100@york.ac.uk Subject: Hardware questions > > > 2. If we change PII 400MHz (100MHz bus) to Celeron 400MHz (66MHz bus) > > on the worker nodes, will the performace drop `dramatically' i.e. > > more than if changing PII 400 to PII 350 (100MHz bus)? > [Bohn, Christopher A.] "It depends on your application." That said, I > would expect that, yes, you would see worse performance with the 400MHz > Celeron than a 350MHz PII. The difference in processor speed would most > likely be dominated by loss in memory performance. The 100MHz bus, > obviously, is 50% faster than the 66MHz bus. So unless your problem fits > entirely within cache, that's going to be a sever hamper on the 14% faster > processor core in the 400MHz Celeron over the 350MHz PII. And, the L2 cache > on the Celeron is only 128KB, as opposed to 512KB on the PII. That said, > IIRC, the Celeron's L2 cache is clocked at full-speed instead of half-speed. > All true and well made points. But why not go for the best of both worlds? The Celeron core is the same as that of a PII and is perfectly happy at 100Mhz. Personally I'd recomend buying 300a's and running them at 4.5x100 (450MHz they do this without getting any hotter btw). You then gain all the advantages of the faster front side bus, but you also gain in the amaisingly cheap price of these CPU's. -Michael From efinch@cais.com Wed, 13 Jan 1999 09:20:58 -0500 Date: Wed, 13 Jan 1999 09:20:58 -0500 From: Ed Finch efinch@cais.com Subject: [Fwd: Exciting New Casino] Ed Finch spammed Beowulf with: > > Dear Sir or Madam, > Please excuse this unsolicted email, but we thought you like to know > that one of the nets premiere on-line casinos is giving away $50.00 to > every customer who signs up now. Visit http://www.worldcasino.mk.ua > for all the details. Enjoy great casino games, including BlackJack, > Roulette, Craps, Slots, Video Poker and more and get $50.00 just for > trying it out. Ugh. Please excuse my fat-fingering. I meant to send this to my local "abuse" account. Best regards, Ed -- Q: Why do PCs have a reset button on the front? A: Because they are expected to run Microsoft operating systems. From rgb@phy.duke.edu Wed, 13 Jan 1999 10:01:51 -0500 Date: Wed, 13 Jan 1999 10:01:51 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: Hardware questions On Wed, 13 Jan 1999, PFENNIGER Daniel wrote: > Andy wrote: > > > > Very few motherboards that I have seen will boot without some video device > > or other. I have several Asus P2B based systems, and for those without the > > need for monitors, I just used old video cards (ISA 256K Trident, S3 PCI > > 1MB etc). > > > > It will boot quite happily without a keyboard though, as long as the bios > > is set to ignore k/b errors (this is true of almost all motherboards from > > 486 onward) > > With an installed Linux on an Asus P2B board one can also remove the video card, > beside the keyboard and the mouse. The system reboots correctly (I tried). > It can then be reached by the network. Of course in case of problem it is > more convenient to have a video screen, but much could be done through a serial > console > (an option in the Linux kernel install). Sparing on cheap video cards was > however > considered not worth for our cluster. This is still good to know. I have diskless boot stuff setup that can very reliably bring a system up to where I have network login capability and scripts to complete a diskful install over the net, and in a beowulf (where the systems are really intended to be headless a priori) not having a video card can save a PCI slot (allowing more network connections) and some money (allowing more network connections at constant cost). A video card is indeed useful for debugging hardware/driver problems, but if one has a large collection of identical hardware, one can probably put a graphics card in just one to get the kernel configuration debugged, and then remove it and install the rest totally stripped. I actually have onboard SVGA on a lot of the systems I've tinkered with diskless, so I've never had occasion to test this, but on my next round of major purchases I'll definitely test a prototype videoless. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From hahn@coffee.psychology.mcmaster.ca Wed, 13 Jan 1999 10:32:42 -0500 Date: Wed, 13 Jan 1999 10:32:42 -0500 From: Mark Hahn hahn@coffee.psychology.mcmaster.ca Subject: Hardware questions > at 4.5x100 (450MHz they do this without getting any hotter btw). You then > gain all the advantages of the faster front side bus, but you also gain in > the amaisingly cheap price of these CPU's. and will never know whether your processor computes correctly. consider the untestability of just FMUL: 128 bits of operand... overclocking is strictly for low-loss applications. I doubt that many Beowulfers consider their work low-loss... From Doug.Hughes@Eng.Auburn.EDU Wed, 13 Jan 1999 11:02:05 -0500 Date: Wed, 13 Jan 1999 11:02:05 -0500 From: Doug Hughes Doug.Hughes@Eng.Auburn.EDU Subject: redhat, kernel, alpha We're recently acquired 3 Microway Alphas on loan as part of a plan to establish a 40 node beowulf cluster for physics research under an NSF grant. They came equipped with redhat 5.2 with 2.0.35. I notice that the beowulf home page has RPM source for kernel-2.0.30-3.src.rpm. So, it looks like I have a few possible options 1) downgrade the kernel to 2.0.30-3 2) find out if the above kernel patches will work with 2.0.35 (doubted, but possible I suppose if the mods are relatively minor) 3) find out if some kind soul has diffs or patches for 2.0.35 kernel and would care to make them available. Doug Hughes Engineering Network Services doug@eng.auburn.edu Auburn University From allanb@sparta.apmaths.uwo.ca Wed, 13 Jan 1999 11:31:44 -0500 Date: Wed, 13 Jan 1999 11:31:44 -0500 From: Allan B MacIsaac allanb@sparta.apmaths.uwo.ca Subject: redhat, kernel, alpha On Wed, 13 Jan 1999, Doug Hughes wrote: > > We're recently acquired 3 Microway Alphas on loan as part of a plan > to establish a 40 node beowulf cluster for physics research under > an NSF grant. They came equipped with redhat 5.2 with 2.0.35. > I notice that the beowulf home page has RPM source for kernel-2.0.30-3.src.rpm. > So, it looks like I have a few possible options > 1) downgrade the kernel to 2.0.30-3 > 2) find out if the above kernel patches will work with 2.0.35 (doubted, but > possible I suppose if the mods are relatively minor) > 3) find out if some kind soul has diffs or patches for 2.0.35 kernel and > would care to make them available. > > Doug Hughes Engineering Network Services > doug@eng.auburn.edu Auburn University > Grab the development kernel v2.2pre(6 or 7). Generally, they're just as stable on the alpha as the stable branch of the kernel, faster, and no "Alpha" patches. At the moment we're running Linux dp1.deeppurple.cluster 2.2.0-pre4-ac4 #1 Just my two cents (3 cents Canadian), A.B. A.B. -- Dr. Allan B. MacIsaac There once was an old man from Esser, Dept. of Applied Math Whose knowledge grew lesser and lesser. University of Western Ontario It at last grew so small, London, Ontario, Canada That he knew nothing at all, N6A 5B7 And now he's a College Professor. From deadline@plogic.com Wed, 13 Jan 1999 11:42:01 -0500 Date: Wed, 13 Jan 1999 11:42:01 -0500 From: Douglas Eadline deadline@plogic.com Subject: redhat, kernel, alpha On Wed, 13 Jan 1999, Doug Hughes wrote: > > We're recently acquired 3 Microway Alphas on loan as part of a plan > to establish a 40 node beowulf cluster for physics research under > an NSF grant. They came equipped with redhat 5.2 with 2.0.35. > I notice that the beowulf home page has RPM source for kernel-2.0.30-3.src.rpm. > So, it looks like I have a few possible options > 1) downgrade the kernel to 2.0.30-3 > 2) find out if the above kernel patches will work with 2.0.35 (doubted, but > possible I suppose if the mods are relatively minor) > 3) find out if some kind soul has diffs or patches for 2.0.35 kernel and > would care to make them available. You can use 2.0.35. There is nothing special about the Beowulf kernel other than the channel bonding patch (and some other minor things). Download PVM or LAM-MPI and you are in business. If you want to do channel bonding, let me know I have dev.c for 2.0.35 that supports channel bonding. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From lindahl@cs.virginia.edu Wed, 13 Jan 1999 12:06:25 -0500 Date: Wed, 13 Jan 1999 12:06:25 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: redhat, kernel, alpha > We're recently acquired 3 Microway Alphas on loan as part of a plan > to establish a 40 node beowulf cluster for physics research under > an NSF grant. They came equipped with redhat 5.2 with 2.0.35. > I notice that the beowulf home page has RPM source for kernel-2.0.30-3.src.rpm. > So, it looks like I have a few possible options 4) Ignore the kernel patches, which just provide a global PID space, and/or address bugs which were long fixed and aren't relelvant for single-cpu alphas anyway. If you're running IDE drives, you'll need to use 2.2. Be sure to get onto axplist@redhat.com and get up to speed on Alpha Linux issues. -- greg From mkh100@york.ac.uk Wed, 13 Jan 1999 13:03:42 -0500 Date: Wed, 13 Jan 1999 13:03:42 -0500 From: Michael Hodgson mkh100@york.ac.uk Subject: Hardware questions On Wed, 13 Jan 1999, Mark Hahn wrote: > > at 4.5x100 (450MHz they do this without getting any hotter btw). You then > > gain all the advantages of the faster front side bus, but you also gain in > > the amaisingly cheap price of these CPU's. > > and will never know whether your processor computes correctly. consider > the untestability of just FMUL: 128 bits of operand... > > overclocking is strictly for low-loss applications. I doubt that many > Beowulfers consider their work low-loss... > > All I can say is that it is well worth testing your particular application to see if this is indeed a problem. Under Charmm for example we have been running with overclocked chips for over a year now. I am often forced to repeat runs, benchmarks etc. and I can honestly say that there has never, not once, been any deviation in the energy, or any other feature of the output file aside from the date and the elapsed time... If an application can run over 4 overclocked processers for 40days of continual number crunching and result in no deviation, then I would say that the processor is infact making NO more errors than a normal run. It may be that for some applications this in not the case... But give it a go... Run a test case 100 times and see if you get the same answer every time... If you do chances are the overclocking has not made any differance. Gamers often go for as much overclocking as possible and this is why overclocking has a bad name.. Be moderate in your overclocking and you'll never know... The chips do come out of the same factory afterall :) I would put good money on you finding the same core in some Xeon processors as you find in a Celeron. If overclocking doesn't work then why is it that intel feel the need to clock-lock (and soon bus-lock) their processors? Once I would have agreed with Mark, but just occasionally a CPU comes along that really is too good to be true. The Pentium 166MMX (same as a 200) was one, the PPro 166 was one and now there is the celeron... Overclocking anything else just doesn't give you the same gain, but these processors fly. I do not consider my work low-loss, but neither do I consider the chance to get work done in half the time a trivial issue. To those people out there who do not have oodles of cash to throw at there Beocub, I would encourage you to do a bit of investigation for yourself, rather than listening to the scare-mongering that companies like intel would love to propergate.... There are sadly those people that are just to eager to listen and instead throw good money away. -Michael If any one has any comments on these issues then I would suggest that they are taken up personally rather than futher cluttering the list, as it really is not on subject... From lindahl@cs.virginia.edu Wed, 13 Jan 1999 14:04:11 -0500 Date: Wed, 13 Jan 1999 14:04:11 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: Hardware questions > All I can say is that it is well worth testing your particular application > to see if this is indeed a problem. Do we have to repeat this religious war every week? How do you _know_ that the run you publish doesn't have a problem, even if you repeat it and get the same answer? > If an application can run over 4 overclocked processers for 40days of > continual number crunching and result in no deviation, then I would say > that the processor is infact making NO more errors than a normal run. I would disagree, and I wouldn't put my name on that paper, nor fly in that airplane. > If any one has any comments on these issues then I would suggest that they > are taken up personally rather than futher cluttering the list, as it > really is not on subject... Just like last time it came up, and the time before, and the time before, and the time before, and... -- g From kragen@pobox.com Wed, 13 Jan 1999 14:31:49 -0500 Date: Wed, 13 Jan 1999 14:31:49 -0500 From: Kragen Sitaker kragen@pobox.com Subject: Hardware questions On Wed, 13 Jan 1999, Michael Hodgson wrote: > All I can say is that it is well worth testing your particular application > to see if this is indeed a problem. Under Charmm for example we have been > running with overclocked chips for over a year now. I am often forced to > repeat runs, benchmarks etc. and I can honestly say that there has never, > not once, been any deviation in the energy, or any other feature of the > output file aside from the date and the elapsed time... > > If an application can run over 4 overclocked processers for 40days of > continual number crunching and result in no deviation, then I would say > that the processor is infact making NO more errors than a normal run. Possibly. Or possibly it's making the same errors every time. Computer hardware failures are often systematic and very input-dependent. > It may be that for some applications this in not the case... But give it a > go... Run a test case 100 times and see if you get the same answer every > time... If you do chances are the overclocking has not made any > differance. Even assuming that getting the same answer 100 times in a row means that that answer is correct, it may still be the case that your overclocking will cause errors for programs other than your test case. > Gamers often go for as much overclocking as possible and this > is why overclocking has a bad name.. Be moderate in your overclocking and > you'll never know... The chips do come out of the same factory afterall > :) And for some of them, the reason they are rated for lower clock speeds is that they don't function reliably at higher clock speeds. > I would put good money on you finding the same core in some Xeon > processors as you find in a Celeron. If overclocking doesn't work then > why is it that intel feel the need to clock-lock (and soon bus-lock) their > processors? My guess is that it's because other companies were reselling Intel chips rebadged for higher clock rates, resulting in people getting incorrect answers and blaming Intel for shipping faulty chips. > If any one has any comments on these issues then I would suggest that they > are taken up personally rather than futher cluttering the list, as it > really is not on subject... Of course -- and your post was, right? :) -- Kragen Sitaker Computers are the tools of the devil. It is as simple as that. There is no monotheism strong enough that it cannot be shaken by Unix or any Microsoft product. The devil is real. He lives inside C programs. -- philg@mit.edu From deadline@plogic.com Wed, 13 Jan 1999 14:47:26 -0500 Date: Wed, 13 Jan 1999 14:47:26 -0500 From: Douglas Eadline deadline@plogic.com Subject: Hardware questions On Wed, 13 Jan 1999, Greg Lindahl wrote: > > All I can say is that it is well worth testing your particular application > > to see if this is indeed a problem. > > Do we have to repeat this religious war every week? Greg is right. Please let's not repeat this weekly (or so it seems) overclocking "discussion". You build your own bed and you sleep in it. Good night. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From rgb@phy.duke.edu Wed, 13 Jan 1999 15:13:07 -0500 Date: Wed, 13 Jan 1999 15:13:07 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: [Fwd: Exciting New Casino] On Wed, 13 Jan 1999, Ed Finch wrote: > Ed Finch spammed Beowulf with: > > > > Dear Sir or Madam, > > Please excuse this unsolicted email, but we thought you like to know > > that one of the nets premiere on-line casinos is giving away $50.00 to > > every customer who signs up now. Visit http://www.worldcasino.mk.ua > > for all the details. Enjoy great casino games, including BlackJack, > > Roulette, Craps, Slots, Video Poker and more and get $50.00 just for > > trying it out. > > Ugh. Please excuse my fat-fingering. I meant to send this to my > local "abuse" account. Awww, I was excited by the offer. It was right up there with the live nude girls offer we had a few months ago. I confess, though (knowing you to be a sane and responsible list member) that I was wondering -- what's the connection? Are they running the website on a beowulf? Has Ed figured out how to beat BlackJack online with his beowulf? Is Ed suggesting that if we all get "$50 for free" from this site we can buy a hell of a lot of beer at our next meeting? Gotta be a connection here somewhere... ;-) rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb@phy.duke.edu Wed, 13 Jan 1999 16:00:50 -0500 Date: Wed, 13 Jan 1999 16:00:50 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: Hardware questions On Wed, 13 Jan 1999, Michael Hodgson wrote: > On Wed, 13 Jan 1999, Mark Hahn wrote: .. > > and will never know whether your processor computes correctly. consider > > the untestability of just FMUL: 128 bits of operand... > > > > overclocking is strictly for low-loss applications. I doubt that many > > Beowulfers consider their work low-loss... > > > > > > All I can say is that it is well worth testing your particular application > to see if this is indeed a problem. Under Charmm for example we have been > running with overclocked chips for over a year now. I am often forced to ... > propergate.... There are sadly those people that are just to eager to > listen and instead throw good money away. Old discussion, long since in the list archives. Nobody ever convinces anybody and it wastes bandwidth (I know, I'm a major sinner in this regard in the past myself:-). However, I've seen the light and been converted to a new state of tolerance and ecumenicality. That is, I've resolved to simply ignore the issue (after this one last time). I would ASK that advocates of overclocking at least put a routine YMMV-type disclaimer in; all too often it is presented as a "sure thing" when there is clear evidence that it isn't -- people certainly have gotten burned in the past and will in the future (until Intel arranges their chips so they cannot be overclocked in the next few months). People do YMMV for everything else, don't see why overclocking should be an obsession (excuse me, I meant exception;-). In exchange for this minor concession, perhaps the rest of us can ALL ignore the occurrence of the word "overclocking" in a note without doing the hornet thing or cranking up the laser blaster. If somebody is making truly egregious claims in an open letter to a neophyte, I suppose one could always respond to their claims directly to said neophyte offline and avoid the usual jihad. Om Shanti, rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From jozef@fatra.ph.hunter.cuny.edu Wed, 13 Jan 1999 17:18:08 -0500 Date: Wed, 13 Jan 1999 17:18:08 -0500 From: Jozef Skvarcek jozef@fatra.ph.hunter.cuny.edu Subject: Hardware questions, One more Thank you to all who responded to my previous questions. I think I will send back some conclusions but first I would like to ask: 1. Is it worth to get ECC memory? I don't remember seeing any problem (crashed computer, application, garbled data...) which could be related to erroreous memory using a few different types on non-brand non-ECC DRAM/EDO/SDRAM. Also, I guess the ECC RAM is slower since it probably employs an error checking algorithm. Jozef Skvarcek jskvarce@shiva.hunter.cuny.edu From Drake.Diedrich@anu.edu.au Wed, 13 Jan 1999 18:41:38 -0500 Date: Wed, 13 Jan 1999 18:41:38 -0500 From: Drake Diedrich Drake.Diedrich@anu.edu.au Subject: PVM and SMP/non-SMP Linux] On Wed, Jan 13, 1999 at 12:15:31PM +0200, Mark Gebhardt wrote: > Hi all > > I am running PVM3.4.beta7 on Redhat 5.2 Linux. The hardware I am using > consists of 7 single-CPU PIIs and one dual-processor PII (this machine > has Linux recompiled for SMP). > > If I run PVM apps on the dual processor machine by itself, I find that > both processors get used, but if I include another machine in the VM, > the CPU utilisation (via top) on the parallel machine is approx. the > same as on the single procesor machine. > > Does anyone know why this happens and if there is any way of using the > dual-processor machine more fully? > > Thanks > > Mark Your program needs to start two processes on the dual CPU machine. PVM doesn't report number of processors to your program, so either you need to supply this information or your program needs to get it elsewhere. One possibility would be to hide this in the SPEED parameter (sp=1,2,...) of the hostfile. If the program is pvmpov, use "+N pvm_hosts=host1,host2,host3,host3,..." with your dual CPU machine listed twice. Running two processes on every machine is also possible (+NT2) but consumes more memory and has a higher context switching overhead. -Drake From kerr@wizard.net Wed, 13 Jan 1999 19:04:24 -0500 Date: Wed, 13 Jan 1999 19:04:24 -0500 From: Kerr kerr@wizard.net Subject: Hardware questions, One more On Wed, 13 Jan 1999, Jozef Skvarcek wrote: > Thank you to all who responded to my previous questions. > I think I will send back some conclusions but first I would like > to ask: > > 1. Is it worth to get ECC memory? I don't remember seeing any problem > (crashed computer, application, garbled data...) which could be related > to erroreous memory using a few different types on non-brand non-ECC > DRAM/EDO/SDRAM. Also, I guess the ECC RAM is slower since it probably > employs an error checking algorithm. This depends on your comfort level at having possibly strange results. Bad memory *does* exist, and the chances of you having bad memory increases the more that you have. If 99.9% of DIMM's are good, your 16-box cluster has a 1 in 32 chance of having a bad DIMM (2 per box). That's pretty high, if you ask me. And you may never find out -- until publish bogus results and get laughed at by the scientific community. :) OTOH, it sure is pricy. Shani From mike@wire.net.au Thu, 14 Jan 1999 06:13:59 -0500 Date: Thu, 14 Jan 1999 06:13:59 -0500 From: Michael Sexton mike@wire.net.au Subject: Oracle 8.0.5 on a beowulf cluster Hi, I'm very new to beowulf but have had quite a bit of experience with oracle ! Im wondering if anyone has set up Oracle on a beowulf cluster ? Has it been a success or failure ? Any architecture do's or don'ts or hints ? Any input would be appreciated. Mike From deadline@plogic.com Thu, 14 Jan 1999 07:22:36 -0500 Date: Thu, 14 Jan 1999 07:22:36 -0500 From: Douglas Eadline deadline@plogic.com Subject: Oracle 8.0.5 on a beowulf cluster On Thu, 14 Jan 1999, Michael Sexton wrote: > Hi, > > I'm very new to beowulf but have had quite a bit of experience with oracle ! > > Im wondering if anyone has set up Oracle on a beowulf cluster ? If you are implying has Oracle released a parallel version for Linux clusters, I think the answer is no. Otherwise, you can run Oracle on one node. > Has it been a success or failure ? > Any architecture do's or don'ts or hints ? > > Any input would be appreciated. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From ferguson.joseph@nesc.epa.gov Thu, 14 Jan 1999 07:39:25 -0500 Date: Thu, 14 Jan 1999 07:39:25 -0500 From: Joe Ferguson ferguson.joseph@nesc.epa.gov Subject: Hardware questions, One more Jozef Skvarcek wrote: > > Thank you to all who responded to my previous questions. > I think I will send back some conclusions but first I would like > to ask: > > 1. Is it worth to get ECC memory? I don't remember seeing any problem > (crashed computer, application, garbled data...) which could be related > to erroreous memory using a few different types on non-brand non-ECC > DRAM/EDO/SDRAM. Also, I guess the ECC RAM is slower since it probably > employs an error checking algorithm. > > Jozef Skvarcek > jskvarce@shiva.hunter.cuny.edu RE ECC RAM: I am looking at a price list from a vendor and find, for "PC100 SDRAM (100 MHz)" the following" 168 pin SDRAM 256 MEG 32X64 $513 168 pin SDRAM ECC 256 MEG 32X64 $536 It seems to be a modest Delta$ for the value received. Most of the chip sets whose specs I have examined support ECC, including the od LX, the BX, and the NX. I sorely miss the current non-availability of the old HX set for the Pentiums, since it had ECC. In general, the specs I've seen show that using ECC costs nothing in performance GIVEN THAT YOU ARE USING A SYSTEM THAT CAN DO IT. The only time you take a hit is when an error is detected; then the question becomes whether you want to know about it. If it's a single-bit error in the 64-bit memory word, it's corrected on the fly so your answer is right. If it's a multiple-bit error, it can't be corrected and there is a processor trap (what happens then is up to the OS). As for me, I'll pay a little more for the ECC RAM and sleep better. RE OVERCLOCKING: Historically, vendors of semiconductors have tested to screen their products for speed, temperature range, etc., often putting special circuits in the IC package which are intended to represent the performance envelope they want to claim. These tests generally are NOT extensive (the price would be much higher if they were) What we'd really like to know is the parameter limits Intel uses in grading the chips. Some vendors have been reported to just do the refined tests on only those components that receive the higher grading stamp and can be sold at the higher price. Thus it may mean that lower graded parts simply haven't been tested, not that they failed the tests. I would think that being tested on your problem against your tolerances would be as valid a screening criterion as what the manufacturer might use. Do you have the time and patience to do it? -- Joe Ferguson ferguson.joseph@nesc.epa.gov Lockheed Martin Corp. NESC Support, EPA-RTP -- NC Voice: (919)-541-3716 From gerry@cs.tamu.edu Thu, 14 Jan 1999 08:01:21 -0500 Date: Thu, 14 Jan 1999 08:01:21 -0500 From: Gerry Creager gerry@cs.tamu.edu Subject: Hardware questions, One more Kerr wrote: > > On Wed, 13 Jan 1999, Jozef Skvarcek wrote: > > > Thank you to all who responded to my previous questions. > > I think I will send back some conclusions but first I would like > > to ask: > > > > 1. Is it worth to get ECC memory? I don't remember seeing any problem > > (crashed computer, application, garbled data...) which could be related > > to erroreous memory using a few different types on non-brand non-ECC > > DRAM/EDO/SDRAM. Also, I guess the ECC RAM is slower since it probably > > employs an error checking algorithm. > > This depends on your comfort level at having possibly strange results. > Bad memory *does* exist, and the chances of you having bad memory Bad memory surely DOES exist: I just got bitten with some in one of my stand-alone boxes. I had to do a shutdown for some facilities maintenance and when we powered back up, it was dead... the box never made it through the boot process. Memory failure wasthe diagnosis, and a rapid repair was effected. For all the world, the failure mode, to me, looked like processor or mother board! Gerry Creager Mapping Sciences Lab Texas A&M University From kpm@ids.net Thu, 14 Jan 1999 08:36:33 -0500 Date: Thu, 14 Jan 1999 08:36:33 -0500 From: Kevin McAloon kpm@ids.net Subject: MOLECULAR MODELING APPLICATION Hello, Are any active list members working with molecular modeling? Rendering and recognition stuff. I'm hunting applications down. So if anyone can assist it would be greatly appreciated. Thanks. mac -- Where do you want to go next year - support open source systems >From the MacroHard Federation - Rhode Island member From hahn@coffee.psychology.mcmaster.ca Thu, 14 Jan 1999 12:16:47 -0500 Date: Thu, 14 Jan 1999 12:16:47 -0500 From: Mark Hahn hahn@coffee.psychology.mcmaster.ca Subject: Hardware questions, One more > I am looking at a price list from a vendor and find, for "PC100 SDRAM > (100 MHz)" the following" > 168 pin SDRAM 256 MEG 32X64 $513 > 168 pin SDRAM ECC 256 MEG 32X64 $536 > It seems to be a modest Delta$ for the value received. strange prices. ECC always does reflect the physical premium of containing 9/8 as much ram. 536 is a decent price, but 513 is too much: should be no more than 455 (see http://www.thechipmerchant.com/complete.htm). > In general, the specs I've seen show that using ECC costs nothing in > performance GIVEN THAT YOU ARE USING A SYSTEM THAT CAN DO IT. The only no, it always costs at least one cycle, on all Intel chipsets, if you turn on ECC. parity is free, though, and since errors are so rare, might be the better choice sometimes. I think the cycle overhead is pipelined, though, so you go from something like a 7111 burst to 8111. you should decide about ECC based on how much ram you have, how hard you use it (those are the two main factors determining how often you'll see and error) and how much an error will cost. regards, mark hahn. From fortuned@cuug.ab.ca Thu, 14 Jan 1999 14:53:47 -0500 Date: Thu, 14 Jan 1999 14:53:47 -0500 From: Doug Fortune fortuned@cuug.ab.ca Subject: MOLECULAR MODELING APPLICATION Kevin McAloon wrote: > Hello, > Are any active list members working with molecular modeling? Rendering > and recognition stuff. I'm hunting applications down. So if anyone can assist > it would be greatly appreciated. Thanks. One you might check out is PMD (Parallel Molecular Dynamics)which can run PVM. http://tincan.bioc.columbia.edu/pmd Doug Fortune Calgary From josip@icase.edu Thu, 14 Jan 1999 15:40:14 -0500 Date: Thu, 14 Jan 1999 15:40:14 -0500 From: Josip Loncaric josip@icase.edu Subject: g77 problems We just installed egcs-1.1.1 and g77 is still missing the ability to accept intrinsic functions like MAX() in PARAMETER statements. As a result, standard LAPACK cannot be compiled unmodified because its timing routines contain constructs such as: INTRINSIC MAX PARAMETER ( LWORK = MAX( MAXN*( 4*MAXN+2 ), $ 2*LDAMAX+1+3*MAXN+2*MAXN*LG2MXN+3*MAXN**2 ) ) Moreover, LAPACK tests of complex number routines dump core, unless the compiler switch -fno-emulate-complex is added. This problem was seen with the g77 -O3 -funroll-all-loops -fomit-frame-pointer -m486 -malign-double compiler command. Although this g77 can produce code which runs several times faster than the same code compiled with "fort77 -O", at least fort77 compiles LAPACK without any problems. Josip -- Dr. Josip Loncaric, Senior Staff Scientist mailto:josip@icase.edu ICASE, Mail Stop 403 http://www.icase.edu/~josip/ NASA Langley Research Center mailto:j.loncaric@larc.nasa.gov Hampton, VA 23681-2199, USA Tel. +1 757 864-2192 Fax +1 757 864-6134 From smulder@netscape.net Thu, 14 Jan 1999 16:09:33 -0500 Date: Thu, 14 Jan 1999 16:09:33 -0500 From: Sam Mulder smulder@netscape.net Subject: Introduction Hello, I've been using Linux now for under a year but I've had a lot of experience with computers in general (TI99/4a,DOS,Windows,etc). I'm trying to get my own Beowulf cluster started for experimentation and testing. The biggest problem I'm having is lack of documentation. I've found a few sources online claiming to be tutorials on how to set up a Beowulf but they are insufficient. They generally just list the files used on the authors particular cluster without any explanation. I'm interested in gaining a thorough understanding of the Beowulf system and distributed computing in general and I understand that I will need to look at the source code to do so but I'd really like to be able to get something up and working before I have to take that leap. Does anyone know of better resources? Are there any good introductory books in progress from O'reilly or someone else? Is there any hope that this technology will become more accessible? I'm confident that I will be able to figure the system out but I'd also like to have something to recommend to friends looking to do the same. I do have some good books about distributed computing but nothing very introductory. Thanks for your help, Sam Mulder sam@sleepless.to (preferred) smulder@netscape.net (used for lists) smulder@ttu.edu ____________________________________________________________________ More than just email--Get your FREE Netscape WebMail account today at http://home.netscape.com/netcenter/mail From mad96@hamp.hampshire.edu Thu, 14 Jan 1999 21:05:22 -0500 Date: Thu, 14 Jan 1999 21:05:22 -0500 From: Michael Dartt mad96@hamp.hampshire.edu Subject: Introduction--I second that motion Greetings everyone, I, too, am an experienced computer user/programmer who is planning on setting up a cluster for my college. I've also found the Web documentation rather sparse for my needs: most of the pages I've seen that give any instructions do so only for homogeneous systems. I'm planning on building a heterogenous system, and I'm still uncertain about a few issues. If anyone has any tips or pointers towards useful docs, I'd be *very* grateful. Basically, I'd like the cluster to function like one big computer, to the extent that that's possible. I.e. if I've got 10 80MB hard drives, I'd like to be able to access them as one 800MB drive (at least as far as the kernel, accounts, and the like go). Ideally, I'd also like to use the processors in the same way, but I've been told that's only possible if the applications have been explicitly written to take advantage of multiple processors. Finally, I'd like to implement some sort of redundancy to guard against faulty components. Is any of this possible? Or is a cluster basically a bunch of individual computers, only providing an advantage for multiprocessor applications? Thank you for your help. Mike From phro@segfault.ind.WPI.EDU Thu, 14 Jan 1999 22:07:36 -0500 Date: Thu, 14 Jan 1999 22:07:36 -0500 From: Jeffrey Moyer phro@segfault.ind.WPI.EDU Subject: Introduction ==> Regarding Introduction; Sam Mulder adds: smulder> Hello, I've been using Linux now for under a year but I've had a smulder> lot of experience with computers in general smulder> (TI99/4a,DOS,Windows,etc). I'm trying to get my own Beowulf smulder> cluster started for experimentation and testing. The biggest smulder> problem I'm having is lack of documentation. I've found a few smulder> sources online claiming to be tutorials on how to set up a Beowulf smulder> but they are insufficient. They generally just list the files smulder> used on the authors particular cluster without any explanation. Try this HOWTO http://www.sci.usq.edu.au/staff/jacek/beowulf/HOWTO/ smulder> I'm interested in gaining a thorough understanding of the Beowulf smulder> system and distributed computing in general and I understand that smulder> I will need to look at the source code to do so but I'd really No, you really don't need to look at the source code. What you need to do is read the above HOWTO. It has a good definition of Beowulf, and some other beginner info that you should find interesting. smulder> like to be able to get something up and working before I have to smulder> take that leap. Does anyone know of better resources? Are there smulder> any good introductory books in progress from O'reilly or someone smulder> else? Is there any hope that this technology will become more smulder> accessible? I'm confident that I will be able to figure the More accessible? I won't write a dissertation on this, but I think you should rephrase that question. Open source is quite accessible. smulder> system out but I'd also like to have something to recommend to smulder> friends looking to do the same. I do have some good books about smulder> distributed computing but nothing very introductory. A good book to start with is "High Performance Computing," published by O'Reilly and Assoc. Unfortuantely (for me), most of the examples are in fortran. *shrug* Anyway, it talks about architectures, some of the evolution thereof, and good explaination on why that evolution occurred. Then covers such topics as compiler optimizations, parallelizing code, and then PVM and MPI. A very good compilation. Now, on the whole, you will find setting up a Beowulf consists of installing linux on a bunch of machines, isolating these machine on a private network, perhaps installing a kernel patch or two for channel bonding and/or global process ID's, installing 3rd party software (if you will) like PVM and MPI. On the whole, a cluster is of no use to someone who doesn't have parallel applications. :) Though, I am sure you will use it as a tool to learn parallel programming, yes? Hope this is the sort of thing you were looking for. -Jeff From deadline@plogic.com Thu, 14 Jan 1999 22:33:02 -0500 Date: Thu, 14 Jan 1999 22:33:02 -0500 From: Douglas Eadline deadline@plogic.com Subject: Introduction--I second that motion On Thu, 14 Jan 1999, Michael Dartt wrote: > Greetings everyone, > > I, too, am an experienced computer user/programmer who is planning on > setting up a cluster for my college. I've also found the Web > documentation rather sparse for my needs: most of the pages I've seen that > give any instructions do so only for homogeneous systems. I'm planning on > building a heterogenous system, and I'm still uncertain about a few > issues. If anyone has any tips or pointers towards useful docs, I'd be > *very* grateful. > > Basically, I'd like the cluster to function like one big computer, to the > extent that that's possible. I.e. if I've got 10 80MB hard drives, I'd > like to be able to access them as one 800MB drive (at least as far as the > kernel, accounts, and the like go). Ideally, I'd also like to use the > processors in the same way, but I've been told that's only possible if the > applications have been explicitly written to take advantage of multiple > processors. Finally, I'd like to implement some sort of redundancy to > guard against faulty components. > > Is any of this possible? Or is a cluster basically a bunch of individual > computers, only providing an advantage for multiprocessor applications? Sure it is possible. But it does not exist - if it did most of us would be out of business. Unfortunateley there is no free lunch. The howto presents some of the issues that you must contend with, but there are many more. I do not have single referenece handyy. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From jwright@panam.edu Thu, 14 Jan 1999 23:15:58 -0500 Date: Thu, 14 Jan 1999 23:15:58 -0500 From: Jeff/Carol Wright jwright@panam.edu Subject: desperate This is a multi-part message in MIME format. ------=_NextPart_000_0049_01BE400B.C930F780 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I am setting up a small Beowulf as a master's project and am completely = new to networking in general. I have gotten all the nodes to "ping" now = except one, which just sits and tries with no response -- it is as if it = is cutoff from the rest of the network. I fixed the plug-n-play problem = with the 3Com 3c509b and all the rest were detected and set up properly = and communicate with each other. It is just the one and I have swapped = the cards and connection all around to make sure that it is not faulty = hardware. The ornery one is configured exactly the same as the others. = They are all the same machine, same mb, same card with a slightly = different version number on the bios. Anyone have a clue?? I am getting = desperate to get this thing operational and move on to the actual = software project. Any ideas?? Jeff Wright ------=_NextPart_000_0049_01BE400B.C930F780 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
I am setting up a small Beowulf as a = master's=20 project and am completely new to networking in general.  I have = gotten all=20 the nodes to "ping" now except one, which just sits and tries = with no=20 response -- it is as if it is cutoff from the rest of the network.  = I fixed=20 the plug-n-play problem with the 3Com 3c509b and all the rest were = detected and=20 set up properly and communicate with each other.  It is just the = one and I=20 have swapped the cards and connection all around to make sure that it is = not=20 faulty hardware.  The ornery one is configured exactly the same as = the=20 others.  They are all the same machine, same mb, same card with a = slightly=20 different version number on the bios.  Anyone have a clue?? I am = getting=20 desperate to get this thing operational and move on to the actual = software=20 project.  Any ideas??
Jeff = Wright
------=_NextPart_000_0049_01BE400B.C930F780-- From jozef@fatra.ph.hunter.cuny.edu Fri, 15 Jan 1999 00:07:53 -0500 Date: Fri, 15 Jan 1999 00:07:53 -0500 From: Jozef Skvarcek jozef@fatra.ph.hunter.cuny.edu Subject: Introduction Hello, On 14 Jan 1999, Sam Mulder wrote: > Hello, I've been using Linux now for under a year but I've had a lot of > experience with computers in general (TI99/4a,DOS,Windows,etc). I'm trying to > get my own Beowulf cluster started for experimentation and testing. The > biggest problem I'm having is lack of documentation. I've found a few sources > online claiming to be tutorials on how to set up a Beowulf but they are > insufficient. They generally just list the files used on the authors > particular cluster without any explanation. I'm interested in gaining a I felt similarly a few days ago when I was looking for the "instalation instructions", one usualy expect when reading about the beowulf clusters for the first time that some special software should be installed. The truth really is, according my current uderstanding, that a cluster of computers is called to be of `beowulf' kind if it consists of ordinary, mass produced hardware and employing Linux (I guess) as OS. That is, there is no special beowulf software to install which would make from your cluster the `beowulf'. > thorough understanding of the Beowulf system and distributed computing in > general and I understand that I will need to look at the source code to do so > but I'd really like to be able to get something up and working before I have > to take that leap. Does anyone know of better resources? Are there any good > introductory books in progress from O'reilly or someone else? Is there any > hope that this technology will become more accessible? I'm confident that I > will be able to figure the system out but I'd also like to have something to > recommend to friends looking to do the same. I do have some good books about > distributed computing but nothing very introductory. I am in similar situation, this time I am figuring out the proper hardware for our first small virtual machine. Maybe it is good idea just to grab some software used for parallel programing and administering such cluster. The two most widely used, as I know, are the PVM and MPI. The PVM seems to be more popular and easier to program with. Also, you can download great manual for it in PostScript or HTML. In order to start work with PVM you don't need much, extreme minimum is one machine running as master and slave. That's like what I am doing, I installed PVM ver 3.4 beta 7 on two Pentium machines with RedHat 5.1 (kernel 2.0.35) and Debian 2.0 (2.0.35). It compiled without a problem, then I tried the example applications that came with the source, without problem again... The only book I have read so far was `High Performance Computing' by O'Reilly (2nd edition), I didn't like it much, however, it provided terminology and itroduction into a few useful concepts. I hope this helps, if interested e-mail me privately I will send you the web links, Jozef Skvarcek jskvarce@shiva.hunter.cuny.edu From phro@segfault.ind.WPI.EDU Fri, 15 Jan 1999 00:54:29 -0500 Date: Fri, 15 Jan 1999 00:54:29 -0500 From: Jeffrey Moyer phro@segfault.ind.WPI.EDU Subject: Introduction--I second that motion ==> Regarding Re: Introduction--I second that motion; Michael Dartt adds: mad96> Greetings everyone, I, too, am an experienced computer mad96> user/programmer who is planning on setting up a cluster for my mad96> college. I've also found the Web documentation rather sparse for my mad96> needs: most of the pages I've seen that give any instructions do so mad96> only for homogeneous systems. I'm planning on building a mad96> heterogenous system, and I'm still uncertain about a few issues. If mad96> anyone has any tips or pointers towards useful docs, I'd be *very* mad96> grateful. See previous post. ick. I wasn't going to tackle this one tonight, but here goes... mad96> Basically, I'd like the cluster to function like one big computer, mad96> to the extent that that's possible. I.e. if I've got 10 80MB hard mad96> drives, I'd like to be able to access them as one 800MB drive (at mad96> least as far as the kernel, accounts, and the like go). Ideally, I do believe this was the intent of nfs. The nfs implementation for linux is certainly not the fastest in the world, however. This is also not so transparent as I am sure you would like. By this, I mean only that you cannot have a file larger than 80MB, if your largest hdd is 80MB. I think I heard about work being done that would allow for a 'cluster file system,' but I can't remember where I saw it. In short, no, I don't think this exists right now. mad96> I'd also like to use the processors in the same way, but I've been mad96> told that's only possible if the applications have been explicitly mad96> written to take advantage of multiple processors. Finally, I'd like Well, you certainly won't have your program accessing registers on another cpu. :) However, you may want to look into MOSIX. These are kernel patches which allow for transparent process migration and memory ushering. http://www.cnds.jhu.edu/mirrors/mosix/ mad96> to implement some sort of redundancy to guard against faulty mad96> components. This can be done with software that I have not yet written. Not necessarily as accurate as one would hope, but it will guard against at least a machine falling off of the face of the planet. That project is slated to finish in early May. I'll be sure to post my first release. mad96> Is any of this possible? Or is a cluster basically a bunch of mad96> individual computers, only providing an advantage for multiprocessor mad96> applications? A beowulf is a PoPCs (Pile of PCs). A beowulf is a type of cluster. It certainly is not the definition of 'cluster.' Other clusters (commercial ones) support some of the features you ask about. But those darned things cost a great deal of money, and are targetted toward rather large businesses who really need someone to blame for the system going down and costing them 3 billion dollars a minute. You get the idea. mad96> Thank you for your help. Your welcome. Again, see my previous post. -Jeff From kodym@mit.jyu.fi Fri, 15 Jan 1999 05:01:40 -0500 Date: Fri, 15 Jan 1999 05:01:40 -0500 From: Petr Ladislav Kodym kodym@mit.jyu.fi Subject: Introduction Hi, >The >biggest problem I'm having is lack of documentation. I've found a few sources >online claiming to be tutorials on how to set up a Beowulf but they are >insufficient. If you are not in a hurry, you might try to check when the book How to Build a Beowulf : A Guide to the Implementation and Application of PC Clusters (Scientific and Engineering Computation Series) by Thomas L. Sterling, Donald J. Becker, John S. Salmon http://www.amazon.com/exec/obidos/ASIN/026269218X/qid%3D916394248/002-5752500-7008031 becomes available. Petr ---------------------------------------------------------------------- Nobody >>is<< perfect. But who wants to be nobody ??? From kerr@wizard.net Fri, 15 Jan 1999 10:12:47 -0500 Date: Fri, 15 Jan 1999 10:12:47 -0500 From: Kerr kerr@wizard.net Subject: Introduction--I second that motion On Thu, 14 Jan 1999, Michael Dartt wrote: > Basically, I'd like the cluster to function like one big computer, to the > extent that that's possible. I.e. if I've got 10 80MB hard drives, I'd > like to be able to access them as one 800MB drive (at least as far as the > kernel, accounts, and the like go). Ideally, I'd also like to use the > processors in the same way, but I've been told that's only possible if the > applications have been explicitly written to take advantage of multiple > processors. Finally, I'd like to implement some sort of redundancy to > guard against faulty components. You should probably check out some of the Linux High Availability stuff. You can review the HOWTO here: http://metalab.unc.edu/pub/Linux/ALPHA/linux-ha/High-Availability-HOWTO.html Enjoy. Shani From tbaer@columbus.rr.com Fri, 15 Jan 1999 11:06:09 -0500 Date: Fri, 15 Jan 1999 11:06:09 -0500 From: Troy A. Baer tbaer@columbus.rr.com Subject: SGI/Cray "modules" for Linux? We (some folks at the Ohio Supercomputer Center) are putting together a Beowulf cluster that is eventually intended to be one of our production machines. As such, we're setting up batch queueing systems, multiple version of compilers, and some of the other fun stuff that goes with a production supercomputer. The problem we've run into is that there are often namespace collisions between similar products or different versions of the same product. For instance, both the LSF and PBS batch systems have commands named "qstat". SGI has a solution for this problem on their IRIX and Unicos systems called "modules". The modules system is an shell-independent system for dynamically modifying a user's environment. For those who haven't seen it, it's really quite slick; it lets you have multiple version of compilers or whatever lying around, and users can pick which one they want to use by simply running "module load compiler-version" instead of manually hacking PATH and MANPATH. At a presentation some of the LANL folks gave at a CUG Origin 2000 workshop last year, it was mentioned that someone at LANL has implemented a modules clone/workalike for Linux on one of LANL's Beowulf clusters. I've searched LANL's web site, but I can't find any mention of it. Has anybody seen this code or know if it's publically available? --Troy -- Troy Baer, MS(AAE) I may have wasted all those years; tbaer@columbus.rr.com They're not worth their time in tears. http://home.columbus.rr.com/tbaer/ I may have spent too long in darkness, Linux: hex, bugs, and rock'n'roll. In the warmth of my fears. --Dream Theater From lindahl@cs.virginia.edu Fri, 15 Jan 1999 11:31:54 -0500 Date: Fri, 15 Jan 1999 11:31:54 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: SGI/Cray "modules" for Linux? > SGI has a solution for this problem on their IRIX and Unicos systems > called "modules". The modules system is an shell-independent system > for dynamically modifying a user's environment. For those who haven't > seen it, it's really quite slick; it lets you have multiple version of > compilers or whatever lying around, and users can pick which one they > want to use by simply running "module load compiler-version" instead > of manually hacking PATH and MANPATH. This is trivial to implement using a script. If the user wants LSF, put LSF earlier in the PATH and MANPATH. -- g From tbaer@columbus.rr.com Fri, 15 Jan 1999 11:58:27 -0500 Date: Fri, 15 Jan 1999 11:58:27 -0500 From: Troy A. Baer tbaer@columbus.rr.com Subject: SGI/Cray "modules" for Linux? >>>>> "GL" == Greg Lindahl writes: >> SGI has a solution for this problem on their IRIX and Unicos systems >> called "modules". The modules system is an shell-independent system >> for dynamically modifying a user's environment. For those who haven't >> seen it, it's really quite slick; it lets you have multiple version of >> compilers or whatever lying around, and users can pick which one they >> want to use by simply running "module load compiler-version" instead >> of manually hacking PATH and MANPATH. GL> This is trivial to implement using a script. If the user wants LSF, GL> put LSF earlier in the PATH and MANPATH. OK, that's easy enough to do for a couple packages, but it doesn't scale well. It doesn't let you easily change which version of a package you're without either making PATH ungodly long or logging out and logging back in. It doesn't track what packages your environment is currently set up for. The SGI modules system does all that. The point is, I'm pretty sure somebody at LANL's already written this, so I'd prefer to use their code over reinventing it myself. I'm just trying to figure out if it's publically available. --Troy -- Troy Baer, MS(AAE) I may have wasted all those years; tbaer@columbus.rr.com They're not worth their time in tears. http://home.columbus.rr.com/tbaer/ I may have spent too long in darkness, Linux: hex, bugs, and rock'n'roll. In the warmth of my fears. --Dream Theater From lindahl@cs.virginia.edu Fri, 15 Jan 1999 12:07:44 -0500 Date: Fri, 15 Jan 1999 12:07:44 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: SGI/Cray "modules" for Linux? > GL> This is trivial to implement using a script. If the user wants LSF, > GL> put LSF earlier in the PATH and MANPATH. > > OK, that's easy enough to do for a couple packages, but it doesn't > scale well. What doesn't scale about it? > It doesn't let you easily change which version of a > package you're without either making PATH ungodly long or logging out > and logging back in. Logging in and out is not required. The usual trick for modifying your environment applies: alias module eval `module.backend` > It doesn't track what packages your environment > is currently set up for. Your script can do that, using a file, say $HOME/.modulerc. > The point is, I'm pretty sure somebody at LANL's already written this, > so I'd prefer to use their code over reinventing it myself. I'm just > trying to figure out if it's publically available. Sure. It would be nice for this to be available. I use a less flexible scheme now to make things like mpich and legion available to my users -- they have to eval a setup script before using the software. My point was just that this is pretty easy to do, not that you shouldn't look for existing software. -- g From berman@fndaub.fnal.gov Fri, 15 Jan 1999 12:45:24 -0500 Date: Fri, 15 Jan 1999 12:45:24 -0500 From: Eileen Berman berman@fndaub.fnal.gov Subject: SGI/Cray "modules" for Linux? From berman@fndaub.fnal.gov Fri, 15 Jan 1999 12:51:38 -0500 Date: Fri, 15 Jan 1999 12:51:38 -0500 From: Eileen Berman berman@fndaub.fnal.gov Subject: SGI/Cray "modules" for Linux? (Sorry, operator error on the last attempt). Hi. At Fermilab we have been using a software package for the last 7 or more years that does what I think you are talking about. This package is called UPS (Unix Product Support) and it runs on Linux boxes as well as many flavors of Unix. It is written in C and is part of the general Fermilab package management strategy (oh my gosh I sound like a manager). It can, however, be used completely separate from these other software packages. It is freely available. In brief, when using UPS, each software package also contains an ASCII file describing what needs to be done in order to make the product usable by the user. These instructions can be as simple as 'please add the following dir to PATH' to very complicated instructions indeed. What needs to happen when a user logs in, is that his login script file will source a file that sets up his environment to use UPS. This UPS script file does not change from login to login. Then when he needs access to say a specific compiler, he says - %setup the_specific_compiler 'setup' is a UPS defined alias that will do the following - o read the ASCII file with the instructions for the_specific_compiler product o create a script file to alter the users environment o source the script file. The user now has access to the specific compiler. There will be an environment variable defined that records which version, flavor (OS version) of the_specific_compiler product is being used. Of course, there is an 'unsetup' command which will clean up the environment. UPS has many more features which may be used if desired. For more information see the following document, and in particular section 1.2, the intro page. http://www.fnal.gov/docs/products/ups/ReferenceManual/html/upsv4rev1_0/upsv4.html I have reproduced a little of the document here. It is a reference manual, though, and as such is quite long. UPS (UNIX Product Support) is a software support toolkit developed here at Fermilab for the management and access of software products on local systems by the system administrators and users. It was also designed to facilitate the product distribution and configuration management tasks of the product providers. The three principal user benefits are: a uniform interface for accessing all products on a UNIX system via the setup command unified and coordinated support of in-house and vendor-supplied software across all the supported UNIX operating systems the capacity for running multiple concurrent versions of products on the same system, with a standard, simple version-selection mechanism for the end user A UPS installation has two parts: one or more databases; a UPS database is a directory which functions as a central repository of information about products and contains pointers to products. Collectively the metadata files and directories for all the products installed under UPS make up the database. a set of procedures/programs to manipulate and view the database(s) -------------------- Why has Fermilab developed and implemented the particular product support methodology that it has, namely UPS ? The principal reason is to provide self-contained products. Each release of a self-contained product can be installed and accessed independently of other releases. There are two main advantages to this, which are especially important for applications such as real-time data acquisition: Multiple versions of a software product can be made available concurrently. This is useful in many situations, especially when some products depend on particular versions of others for compiling or running. Multiple concurrent versions also makes possible the second advantage: You can back out of a new software release completely and assuredly if something goes wrong, and immediately start up a previous tried-and-true release. The UPS methodology also provides tracking. A glance at the UPS database tells you what version of a product you're running; you don't need to keep track of it elsewhere or risk forgetting. You can also easily tell if different machines are running the same version of a product. eileen berman (berman@fnal.gov) From muno@aem.umn.edu Fri, 15 Jan 1999 14:04:07 -0500 Date: Fri, 15 Jan 1999 14:04:07 -0500 From: Ray Muno muno@aem.umn.edu Subject: SGI/Cray "modules" for Linux? Out of the ether Troy A. Baer writes: > > We (some folks at the Ohio Supercomputer Center) are putting together > a Beowulf cluster that is eventually intended to be one of our > production machines. As such, we're setting up batch queueing > systems, multiple version of compilers, and some of the other fun > stuff that goes with a production supercomputer. The problem we've > run into is that there are often namespace collisions between similar > products or different versions of the same product. For instance, > both the LSF and PBS batch systems have commands named "qstat". > > SGI has a solution for this problem on their IRIX and Unicos systems > called "modules". The modules system is an shell-independent system > for dynamically modifying a user's environment. For those who haven't > seen it, it's really quite slick; it lets you have multiple version of > compilers or whatever lying around, and users can pick which one they > want to use by simply running "module load compiler-version" instead > of manually hacking PATH and MANPATH. > > At a presentation some of the LANL folks gave at a CUG Origin 2000 > workshop last year, it was mentioned that someone at LANL has > implemented a modules clone/workalike for Linux on one of LANL's > Beowulf clusters. I've searched LANL's web site, but I can't find any > mention of it. Has anybody seen this code or know if it's publically > available? > > --Troy > -- > Troy Baer, MS(AAE) I may have wasted all those years; > tbaer@columbus.rr.com They're not worth their time in tears. > http://home.columbus.rr.com/tbaer/ I may have spent too long in darkness, > Linux: hex, bugs, and rock'n'roll. In the warmth of my fears. --Dream Theater > I wouldn't really call this SGI's solution, I believe it came out of Cray long before SGI bought them. It was nice that SGI did the work to get modules running because it was a hard platform to get it working on. We use modules to manage environments on all of our Unix platforms, SGI, Solaris, SunOS and Linux. See http://www.modules.org/ -- ============================================================================= Ray Muno http://www.aem.umn.edu/people/staff/muno University of Minnesota e-mail: muno@aem.umn.edu Aerospace Engineering and Mechanics Phone: (612) 625-9531 110 Union St. S.E. FAX: (612) 626-1558 Minneapolis, Mn 55455 ============================================================================= From djholm@fndaub.fnal.gov Fri, 15 Jan 1999 14:20:38 -0500 Date: Fri, 15 Jan 1999 14:20:38 -0500 From: Don Holmgren djholm@fndaub.fnal.gov Subject: SGI/Cray "modules" for Linux? Some brief information about the Fermilab product (UPS), in case there's interest: - UPS seems to be very similar to the description given of the SGI/Cray tools ("modules"), in that environment variables and various paths (PATH, LD_LIBRARY_PATH, etc...) are manipulated. - It's very much "industrial strength" and was developed for development on heterogeneous clusters (in the sense of multiple operating systems). It is not a trivial tool to install and learn; however, it is very flexible and powerful. - It is actively used and maintained. It's been in use for over 7 years. Essentially all major programming projects in the lab (including collaborators at their home institutions) use it. - It is freely available "cut" for many different operating system versions, including source code. There is an RPM available. - I've personally used it for four years on IRIX, OSF/1, SunOS, and Linux, and would be very unhappy without it. On some of our larger projects, a given software package may depend upon dozens of subordinate packages (including libraries and compilers). When I "setup" a particular version of a package, all products on which that package depends are also setup (i.e., added to $PATH and so forth). Whereas RPM is appropriate for installing and removing packages while tracking dependencies, UPS is appropriate for maintaining an environment containing multiple versions of interdependent products (and it can be used for installing/removing as well). Like RPM it has a rich set of command line tools for querying and modifying the underlying database. - It's part of the Fermilab Software Tools Program ("Fermitools"), which is an effort to provide the internet community with Fermi-developed software packages which may have a general value to other application domains (i.e., non-high energy physics). A good example of a software product from Fermitools is NEdit. For a description of the Fermitools program and the software available, see http://www.fnal.gov/fermitools/ Don Holmgren Fermilab From dmcombs@groton.org Fri, 15 Jan 1999 14:34:15 -0500 Date: Fri, 15 Jan 1999 14:34:15 -0500 From: Mike Combs dmcombs@groton.org Subject: Token Ring Hello All, I was wondering if anyone could point me in the right direction for finding TokenRing information for Linux and Linux Beowulf systems. I Know what your gonad say, Why on Earth are you using TokenRing? Well about 20 SMC token ISA cards and about 15 #com 3c339's and a Chipcom 48 port Switch was given to me for free, ( along with about 20 486's of various MHz). Thanks in advance, and have a good Day. dmcombs@groton.org mike_combs@groton.org dmcombs From andy@meshplc.co.uk Fri, 15 Jan 1999 14:37:13 -0500 Date: Fri, 15 Jan 1999 14:37:13 -0500 From: Andy andy@meshplc.co.uk Subject: SGI/Cray "modules" for Linux? I have also experienced this system on SGI machines, and beowulf aside, I would be interested in a linux version of it, as it would certainly sort out some problems with compilers on some of my machines. Andy MESH Computers Webmaster webmaster@meshplc.co.uk From PJSchube@mail.delcoelect.com Fri, 15 Jan 1999 14:42:30 -0500 Date: Fri, 15 Jan 1999 14:42:30 -0500 From: Peter J Schubert PJSchube@mail.delcoelect.com Subject: Unused Cycles or Cluster? Can embarrasingly parallel applications be run on a corporate LAN? My bosses don't want a white elephant, and when I say "Beowulf Cluster", they hear pachyderm. Can I send jobs to desktops running Win95, and have them return a file or stdout.txt? What sort of software is needed on each machine? How would it affect other users? Thanks, Peter Schubert Delphi Delco Electronics Systems From Christopher.Bohn@afit.af.mil Fri, 15 Jan 1999 15:33:40 -0500 Date: Fri, 15 Jan 1999 15:33:40 -0500 From: Bohn, Christopher A Christopher.Bohn@afit.af.mil Subject: Unused Cycles or Cluster? Good day, > -----Original Message----- > From: Peter J Schubert [SMTP:PJSchube@mail.delcoelect.com] > Sent: Friday, January 15, 1999 2:33 PM > To: beowulf@beowulf.gsfc.nasa.gov > Subject: Unused Cycles or Cluster? > > Can embarrasingly parallel applications be run on a > corporate LAN? [Bohn, Christopher A.] Yes, they can. Even non-embarassingly parallel applications. I sharpened my parallel-processing teeth on a private cluster of six Sun UltraSPARC workstations and on a network of 33 general-use Sun SPARCs. Just expect some variability in your runtimes do to the interactive-user load. And don't send too many jobs out at once -- a good way to tick off the interactive users is to make their Tetris run too slow. > My bosses don't want a white elephant, > and when I say "Beowulf Cluster", they hear pachyderm. > Can I send jobs to desktops running Win95, and have them > return a file or stdout.txt? [Bohn, Christopher A.] Might be hard with Win95. I think you'll do better with WinNT or a UNIX variant. > What sort of software is needed on each machine? [Bohn, Christopher A.] A message passing library, such as MPI. For WinNT, there's MPI Software Technology's MPI-Pro and a product by PaTENT. (sorry, I don't have URLs handy). For UNIX & Linux, there's MPICH available from http://www.mcs.anl.gov/mpi, and there's LAM (URL?); there's also PVM. > How would it affect other users? [Bohn, Christopher A.] As long as you limit yourself to one process per workstation, the interactive users shouldn't notice. > Thanks, > > Peter Schubert > Delphi Delco Electronics Systems > [Bohn, Christopher A.] Take care, cb *-*-*-*-*-*-*-* Capt Christopher A. Bohn Graduate Student, Electrical (digital) Engineering Air Force Institute of Technology Phone (937)255-3636 (DSN 785) AFIT/EN638 Lab x4606 Voicemail x6638 2950 P St, Box 4638 email Christopher.Bohn@afit.af.mil Wright-Patterson AFB OH 45433-7765 EngrBohn@aol.com http://members.aol.com/EngrBohn/ *-*-*-*-*-*-*-* From kodym@mit.jyu.fi Fri, 15 Jan 1999 16:42:32 -0500 Date: Fri, 15 Jan 1999 16:42:32 -0500 From: Petr Ladislav Kodym kodym@mit.jyu.fi Subject: Introduction Hi, >The >biggest problem I'm having is lack of documentation. I've found a few sources >online claiming to be tutorials on how to set up a Beowulf but they are >insufficient. If you are not in a hurry, you might try to check when the book How to Build a Beowulf : A Guide to the Implementation and Application of PC Clusters (Scientific and Engineering Computation Series) by Thomas L. Sterling, Donald J. Becker, John S. Salmon http://www.amazon.com/exec/obidos/ASIN/026269218X/qid%3D916394248/002-5752500-7008031 becomes available. Petr ---------------------------------------------------------------------- Nobody >>is<< perfect. But who wants to be nobody ??? From tbaer@columbus.rr.com Fri, 15 Jan 1999 18:03:06 -0500 Date: Fri, 15 Jan 1999 18:03:06 -0500 From: Troy A. Baer tbaer@columbus.rr.com Subject: SGI/Cray "modules" for Linux? >>>>> "GL" == Greg Lindahl writes: >> SGI has a solution for this problem on their IRIX and Unicos systems >> called "modules". The modules system is an shell-independent system >> for dynamically modifying a user's environment. For those who haven't >> seen it, it's really quite slick; it lets you have multiple version of >> compilers or whatever lying around, and users can pick which one they >> want to use by simply running "module load compiler-version" instead >> of manually hacking PATH and MANPATH. GL> This is trivial to implement using a script. If the user wants LSF, GL> put LSF earlier in the PATH and MANPATH. OK, that's easy enough to do for a couple packages, but it doesn't scale well. It doesn't let you easily change which version of a package you're without either making PATH ungodly long or logging out and logging back in. It doesn't track what packages your environment is currently set up for. The SGI modules system does all that. The point is, I'm pretty sure somebody at LANL's already written this, so I'd prefer to use their code over reinventing it myself. I'm just trying to figure out if it's publically available. --Troy -- Troy Baer, MS(AAE) I may have wasted all those years; tbaer@columbus.rr.com They're not worth their time in tears. http://home.columbus.rr.com/tbaer/ I may have spent too long in darkness, Linux: hex, bugs, and rock'n'roll. In the warmth of my fears. --Dream Theater From lindahl@cs.virginia.edu Fri, 15 Jan 1999 18:30:42 -0500 Date: Fri, 15 Jan 1999 18:30:42 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: SGI/Cray "modules" for Linux? > GL> This is trivial to implement using a script. If the user wants LSF, > GL> put LSF earlier in the PATH and MANPATH. > > OK, that's easy enough to do for a couple packages, but it doesn't > scale well. What doesn't scale about it? > It doesn't let you easily change which version of a > package you're without either making PATH ungodly long or logging out > and logging back in. Logging in and out is not required. The usual trick for modifying your environment applies: alias module eval `module.backend` > It doesn't track what packages your environment > is currently set up for. Your script can do that, using a file, say $HOME/.modulerc. > The point is, I'm pretty sure somebody at LANL's already written this, > so I'd prefer to use their code over reinventing it myself. I'm just > trying to figure out if it's publically available. Sure. It would be nice for this to be available. I use a less flexible scheme now to make things like mpich and legion available to my users -- they have to eval a setup script before using the software. My point was just that this is pretty easy to do, not that you shouldn't look for existing software. -- g From lindahl@cs.virginia.edu Fri, 15 Jan 1999 18:30:40 -0500 Date: Fri, 15 Jan 1999 18:30:40 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: SGI/Cray "modules" for Linux? > GL> This is trivial to implement using a script. If the user wants LSF, > GL> put LSF earlier in the PATH and MANPATH. > > OK, that's easy enough to do for a couple packages, but it doesn't > scale well. What doesn't scale about it? > It doesn't let you easily change which version of a > package you're without either making PATH ungodly long or logging out > and logging back in. Logging in and out is not required. The usual trick for modifying your environment applies: alias module eval `module.backend` > It doesn't track what packages your environment > is currently set up for. Your script can do that, using a file, say $HOME/.modulerc. > The point is, I'm pretty sure somebody at LANL's already written this, > so I'd prefer to use their code over reinventing it myself. I'm just > trying to figure out if it's publically available. Sure. It would be nice for this to be available. I use a less flexible scheme now to make things like mpich and legion available to my users -- they have to eval a setup script before using the software. My point was just that this is pretty easy to do, not that you shouldn't look for existing software. -- g From tbaer@columbus.rr.com Fri, 15 Jan 1999 18:31:21 -0500 Date: Fri, 15 Jan 1999 18:31:21 -0500 From: Troy A. Baer tbaer@columbus.rr.com Subject: SGI/Cray "modules" for Linux? >>>>> "GL" == Greg Lindahl writes: >> SGI has a solution for this problem on their IRIX and Unicos systems >> called "modules". The modules system is an shell-independent system >> for dynamically modifying a user's environment. For those who haven't >> seen it, it's really quite slick; it lets you have multiple version of >> compilers or whatever lying around, and users can pick which one they >> want to use by simply running "module load compiler-version" instead >> of manually hacking PATH and MANPATH. GL> This is trivial to implement using a script. If the user wants LSF, GL> put LSF earlier in the PATH and MANPATH. OK, that's easy enough to do for a couple packages, but it doesn't scale well. It doesn't let you easily change which version of a package you're without either making PATH ungodly long or logging out and logging back in. It doesn't track what packages your environment is currently set up for. The SGI modules system does all that. The point is, I'm pretty sure somebody at LANL's already written this, so I'd prefer to use their code over reinventing it myself. I'm just trying to figure out if it's publically available. --Troy -- Troy Baer, MS(AAE) I may have wasted all those years; tbaer@columbus.rr.com They're not worth their time in tears. http://home.columbus.rr.com/tbaer/ I may have spent too long in darkness, Linux: hex, bugs, and rock'n'roll. In the warmth of my fears. --Dream Theater From kerr@wizard.net Fri, 15 Jan 1999 19:12:55 -0500 Date: Fri, 15 Jan 1999 19:12:55 -0500 From: Kerr kerr@wizard.net Subject: Introduction--I second that motion On Thu, 14 Jan 1999, Michael Dartt wrote: > Basically, I'd like the cluster to function like one big computer, to the > extent that that's possible. I.e. if I've got 10 80MB hard drives, I'd > like to be able to access them as one 800MB drive (at least as far as the > kernel, accounts, and the like go). Ideally, I'd also like to use the > processors in the same way, but I've been told that's only possible if the > applications have been explicitly written to take advantage of multiple > processors. Finally, I'd like to implement some sort of redundancy to > guard against faulty components. You should probably check out some of the Linux High Availability stuff. You can review the HOWTO here: http://metalab.unc.edu/pub/Linux/ALPHA/linux-ha/High-Availability-HOWTO.html Enjoy. Shani From tbaer@columbus.rr.com Fri, 15 Jan 1999 19:32:33 -0500 Date: Fri, 15 Jan 1999 19:32:33 -0500 From: Troy A. Baer tbaer@columbus.rr.com Subject: SGI/Cray "modules" for Linux? We (some folks at the Ohio Supercomputer Center) are putting together a Beowulf cluster that is eventually intended to be one of our production machines. As such, we're setting up batch queueing systems, multiple version of compilers, and some of the other fun stuff that goes with a production supercomputer. The problem we've run into is that there are often namespace collisions between similar products or different versions of the same product. For instance, both the LSF and PBS batch systems have commands named "qstat". SGI has a solution for this problem on their IRIX and Unicos systems called "modules". The modules system is an shell-independent system for dynamically modifying a user's environment. For those who haven't seen it, it's really quite slick; it lets you have multiple version of compilers or whatever lying around, and users can pick which one they want to use by simply running "module load compiler-version" instead of manually hacking PATH and MANPATH. At a presentation some of the LANL folks gave at a CUG Origin 2000 workshop last year, it was mentioned that someone at LANL has implemented a modules clone/workalike for Linux on one of LANL's Beowulf clusters. I've searched LANL's web site, but I can't find any mention of it. Has anybody seen this code or know if it's publically available? --Troy -- Troy Baer, MS(AAE) I may have wasted all those years; tbaer@columbus.rr.com They're not worth their time in tears. http://home.columbus.rr.com/tbaer/ I may have spent too long in darkness, Linux: hex, bugs, and rock'n'roll. In the warmth of my fears. --Dream Theater From tbaer@columbus.rr.com Fri, 15 Jan 1999 19:31:58 -0500 Date: Fri, 15 Jan 1999 19:31:58 -0500 From: Troy A. Baer tbaer@columbus.rr.com Subject: SGI/Cray "modules" for Linux? We (some folks at the Ohio Supercomputer Center) are putting together a Beowulf cluster that is eventually intended to be one of our production machines. As such, we're setting up batch queueing systems, multiple version of compilers, and some of the other fun stuff that goes with a production supercomputer. The problem we've run into is that there are often namespace collisions between similar products or different versions of the same product. For instance, both the LSF and PBS batch systems have commands named "qstat". SGI has a solution for this problem on their IRIX and Unicos systems called "modules". The modules system is an shell-independent system for dynamically modifying a user's environment. For those who haven't seen it, it's really quite slick; it lets you have multiple version of compilers or whatever lying around, and users can pick which one they want to use by simply running "module load compiler-version" instead of manually hacking PATH and MANPATH. At a presentation some of the LANL folks gave at a CUG Origin 2000 workshop last year, it was mentioned that someone at LANL has implemented a modules clone/workalike for Linux on one of LANL's Beowulf clusters. I've searched LANL's web site, but I can't find any mention of it. Has anybody seen this code or know if it's publically available? --Troy -- Troy Baer, MS(AAE) I may have wasted all those years; tbaer@columbus.rr.com They're not worth their time in tears. http://home.columbus.rr.com/tbaer/ I may have spent too long in darkness, Linux: hex, bugs, and rock'n'roll. In the warmth of my fears. --Dream Theater From lindahl@cs.virginia.edu Fri, 15 Jan 1999 19:41:48 -0500 Date: Fri, 15 Jan 1999 19:41:48 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: SGI/Cray "modules" for Linux? > SGI has a solution for this problem on their IRIX and Unicos systems > called "modules". The modules system is an shell-independent system > for dynamically modifying a user's environment. For those who haven't > seen it, it's really quite slick; it lets you have multiple version of > compilers or whatever lying around, and users can pick which one they > want to use by simply running "module load compiler-version" instead > of manually hacking PATH and MANPATH. This is trivial to implement using a script. If the user wants LSF, put LSF earlier in the PATH and MANPATH. -- g From lindahl@cs.virginia.edu Fri, 15 Jan 1999 19:42:08 -0500 Date: Fri, 15 Jan 1999 19:42:08 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: SGI/Cray "modules" for Linux? > SGI has a solution for this problem on their IRIX and Unicos systems > called "modules". The modules system is an shell-independent system > for dynamically modifying a user's environment. For those who haven't > seen it, it's really quite slick; it lets you have multiple version of > compilers or whatever lying around, and users can pick which one they > want to use by simply running "module load compiler-version" instead > of manually hacking PATH and MANPATH. This is trivial to implement using a script. If the user wants LSF, put LSF earlier in the PATH and MANPATH. -- g From eugene.leitl@lrz.uni-muenchen.de Sat, 16 Jan 1999 15:26:31 -0500 Date: Sat, 16 Jan 1999 15:26:31 -0500 From: Eugene Leitl eugene.leitl@lrz.uni-muenchen.de Subject: MOLECULAR MODELING APPLICATION Doug Fortune writes: > One you might check out is PMD (Parallel Molecular Dynamics)which can run PVM. > > http://tincan.bioc.columbia.edu/pmd I would suggest hunting around http://sal.kachinatech.com specifically the chemistry/biochemistry subareas. Personally I use EGO, which can both use MPI and PVM. I intend to port it to U-Net. Generally, most modern MD codes are parallel. Regards, Eugene From jei@zor.hut.fi Sat, 16 Jan 1999 18:59:02 -0500 Date: Sat, 16 Jan 1999 18:59:02 -0500 From: Jukka E Isosaari jei@zor.hut.fi Subject: Freedom CPU project Hi! I thought some of you might be interested in following/ participating in this 'open' CPU project. Sounds neat: http://f-cpu.tux.org/ Here's their FAQ-page: ++ J ------------------------------------ http://f-cpu.tux.org/faq.php3 A GNU/GPLed 64-bit CPU architecture Philosophy Q1: What does the F in F-CPU stand for? A: It stands for Freedom, which is the original name of the architecture, or Free, in the GNU/GPL sense. The F does not stand for free in a monetary sense. You will have to pay for the F1 chip, just as you have to pay nowadays for a copy of a GNU/Linux distribution on CD-ROMs. Of course, you're free to take the design and masks to your favorite fab and have a few batches manufactured for your own use. Q2: Why not call it an O-CPU (where O stands for Open)? A: There are some fundamental philosophical differences between the Open Source movement and the original Free Software movement. We abide by the latter, hence the F. The fact that a piece of code is labeled Open Source doesn't mean that your freedom to use it, understand it and improve upon it is guaranteed. Further discussion of these matters can be found here. Architecture Q1: Why did you choose a memory-to-memory architecture? Why not a register-to-register architecture like all other RISC processors? A: Basically, for performance: we wanted to improve on RISC, so we had to look for a different solution. One of the critical performance bottlenecks in modern Operating Systems is the context switch latency, basically caused by saving and restoring register sets. The F architecture provides near-zero context switch latencies. However, there are many RISC-like ideas in the F architecture. We are working on a White Paper which discusses this issue in detail. A preliminary version can be found here. Q2: Why an external FPU? A: This is implementation specific and depends entirely on how many gates we can fit in a single 122 mm2 die. The F2 could have up to 4 FPUs on chip, since it will be manufactured using 0.18 micron technology. The architecture only states that up to four FPUs can be used in parallel with the main CPU. Q3: Why don't you support SMP? A: SMP is an Intel-proprietary implementation of Symmetric Multi-Processing. In our case, it is not even desirable to implement shared-bus multiprocessing, because the F1-CPU already has a very high rate of bus utilization. Adding more processors would inevitably mean creating a bottleneck that would severely limit performance. We opted for a novel implementation of NUMA-style multiprocessing, which allows both shared memory (with cache-coherency) and message-passing. Performance Q1: Wat can we expect in terms of performance from the F1 CPU? A: Merced-killer. :-). Although the F1 CPU will probably run at a lower clock rate compared to the Merced, due to improvements in design (geared to achieving very high levels of parallelism at execution time) and gcc and kernel-specific optimizations, it is expected that integer performance will be comparable to or better than the Merced. FPU performance will basically depend on the number of FPUs plugged on the CPU bus. With two FPUs the F1 will probably provide better performance than a Merced. With four FPUs it should leave the Merced in the dust. Beowulf F supercomputer implementations should provide superior performance, because of the exceptional bandwidth and low latencies of our NUMA-style multiprocessing mechanism. Yeah, we know that sounds a little bit ambitious... Compatibility Q1: Will the F1 run my DOS software? A: Maybe. We are planning to build in an x86 emulator in the F1 BIOS. But we only want to boot the CPU on standard x86 motherboards. We doubt the F1 will support sophisticated DOS games and other x86-specific packages. Q2: Is the F1 CPU Windows (98, NT) compatible? A: No. It will not run WINE either. It should however run Windows emulators that include x86 CPU emulators such as Twin, as well as Windows itself under whole-PC emulators such as Bochs. In either case however you will need to run another operating system, such as GNU/Linux, and emulation will likely be fairly slow. And of course there is the slight possibility that Microsoft will port Windows NT to the F1, but I wouldn't hold my breath. ;-) Q3: Will I be able to plug the F1 in a standard Socket 7, Super 7 or Slot 1 motherboard? A: In principle, yes. This is one of its design parameters. Your motherboard will have to support the 2.2V rating of the F1 CPU, and there may be other requirements/limitations. But in principle there is enough bus bandwidth in a 100 MHz Super 7 or Slot 1 motherboard to support a high-performance CPU like the F1. 66MHz Socket 7 motherboards will be supported too, but their performance will be somewhat limited. Q4: What OS kernels will the F-CPU support? A: Linux will be ported first, and MkLinux will perhaps be ported too later on. A port of Linux will be developed simultaneously with the F1 CPU development. We will first port gcc to the F architecture. Basically the F1 will run all the software available for a standard GNU/Linux distribution. Features Q1: What kind and size of caches will the F1 CPU have? A: Internally, the F1 will have Harvard-style separate caches for instructions and data. We are planning on 64KB 4-way set associative L1 caches (one each for data and instructions). We also have small L0 caches for instructions (512 bytes) and data/virtual registers (8KB). Cost/Price/Purchasing Q1: How much will the F1 CPU cost? A: Our estimates are that an F1 CPU will cost approximately US$100. See our detailed cost estimate if you don't believe us! From deadline@plogic.com Sat, 16 Jan 1999 19:54:08 -0500 Date: Sat, 16 Jan 1999 19:54:08 -0500 From: Douglas Eadline deadline@plogic.com Subject: Freedom CPU project I do not know that much about CPU design, but the web page lists four people. Quite franly this seems a bit thin. If this effort does produce a fast CPU it will undoubtedly be be quite an impressive feat. I would love to see it succeed, but how can four people build a CPU when the commercail world is finding it more and more expensive to build and manufacture CPUs. Which is, by the way, why the number of CPU families has been getting smaller. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From kragen@pobox.com Sat, 16 Jan 1999 20:50:47 -0500 Date: Sat, 16 Jan 1999 20:50:47 -0500 From: Kragen kragen@pobox.com Subject: Freedom CPU project On Sat, 16 Jan 1999, Douglas Eadline wrote: > . . . how can four people build a CPU . . . They probably can design a decent CPU. Reportedly the i21 is comparable in speed to recent Pentia, and with a design team of one person, Chuck Moore. CPU design is getting easier to do and requiring fewer people. I'm interested to hear how they're going to get access to 0.18-micron fabs, and where they're getting the funding for fab runs. (btw, I designed a memory-to-memory CPU architecture last week, and have built a simulator and run a few programs on it. I suspect I would be able to build one out of relays if I had to. But it'd be slower than any CPU I've ever used, including the ones I used in 1980-1985.) I am of the opinion that projects like this should not generate hype until they have produced products, simply because they so often fail to produce products. -- Kragen Sitaker I don't do .INI, .BAT, .DLL or .SYS files. I don't assign apps to files. I don't configure peripherals or networks before using them. I have a computer to do all that. I have a Macintosh, not a hobby. -- Fritz Anderson From deadline@plogic.com Sat, 16 Jan 1999 21:25:58 -0500 Date: Sat, 16 Jan 1999 21:25:58 -0500 From: Douglas Eadline deadline@plogic.com Subject: Freedom CPU project On Sat, 16 Jan 1999, Kragen wrote: > On Sat, 16 Jan 1999, Douglas Eadline wrote: > > . . . how can four people build a CPU . . . > > They probably can design a decent CPU. Reportedly the i21 is comparable > in speed to recent Pentia, and with a design team of one person, > Chuck Moore. CPU design is getting easier to do and requiring fewer > people. I'm not trying to be negative, but why does it take HP and Intel hundreds of engineers to make the merced, when these four guys think they can do it their spare time. Given that HP and Intel have legacy issues that constrain the design, it still bogles my mind to think that four people (or even 40) could build something that will "blow away the merced". > > I'm interested to hear how they're going to get access to 0.18-micron > fabs, and where they're getting the funding for fab runs. Yes, as I understand it fabs are expensive and controlled by the "CPU cartels". > > (btw, I designed a memory-to-memory CPU architecture last week, and have > built a simulator and run a few programs on it. I suspect I would be > able to build one out of relays if I had to. But it'd be slower than > any CPU I've ever used, including the ones I used in 1980-1985.) Are there not other issues of getting the design working in silicon? Like testing or bug identification? As I understand it, Intel is always fixing bugs etc. (stepping) Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From eugene.leitl@lrz.uni-muenchen.de Sun, 17 Jan 1999 15:09:03 -0500 Date: Sun, 17 Jan 1999 15:09:03 -0500 From: Eugene Leitl eugene.leitl@lrz.uni-muenchen.de Subject: Freedom CPU project Kragen writes: > On Sat, 16 Jan 1999, Douglas Eadline wrote: > > . . . how can four people build a CPU . . . > > They probably can design a decent CPU. Reportedly the i21 is comparable > in speed to recent Pentia, and with a design team of one person, > Chuck Moore. CPU design is getting easier to do and requiring fewer > people. I am afraid that the i21 MIPS are paper MIPS. The silicon is still insufficiently debugged for production. To harness its full power, you would have to write in machine language, or code in Forth. i21 has no FPU, and its integer ALU is 21 bit broad, and doesn't do multiply in hardware. Chuck had to write his own simulator to be able to push a given fab process beyond its advertized limits, which is not exactly easy (e.g. he had to localize the hot spots since certain opcode combinations didn't run because of thermal problems, and to address that by doubling the area of certain critical transistors). To me, this approach doesn't at all sound like doodling paper CPUs and scribbling little VHDL code bits. I admire Chuck's way of doing hardware design, however, unless backed up by major players he could found himself in the role of late Seymour Cray: great designs, yet ignored by the mainstream. You could do a bit with open source hardware if FPGA areas on CPUs would become commonplace, however, here you would be outperformed by GA design methods. Regards, Eugene From eugene.leitl@lrz.uni-muenchen.de Sun, 17 Jan 1999 15:07:53 -0500 Date: Sun, 17 Jan 1999 15:07:53 -0500 From: Eugene Leitl eugene.leitl@lrz.uni-muenchen.de Subject: Unused Cycles or Cluster? Peter J Schubert writes: > Can embarrasingly parallel applications be run on a > corporate LAN? My bosses don't want a white elephant, > and when I say "Beowulf Cluster", they hear pachyderm. > Can I send jobs to desktops running Win95, and have them > return a file or stdout.txt? > What sort of software is needed on each machine? > How would it affect other users? Since facing a similiar problem, I have been thinking about solutions. Apart from using a Beowulf as a giant file server (10x10 GByte=100 GByte with PVFS and samba) and using telnet, I intend to install an X server (there are at least 2 decent commercial products out there) on each Win95 machine. One of the Beo nodes is running a visualization front end (e.g. VMD), harvesting information from several computational nodes (e.g. running NAMD2), and casting the X visuals via the network to the Winblows boxen. Custom solutions could involve cgi-bin scripts for web front ends to applications, and maybe some Java/Python for remote control and visualization. Regards, Eugene From smic@wire.net.au Sun, 17 Jan 1999 18:36:23 -0500 Date: Sun, 17 Jan 1999 18:36:23 -0500 From: Steven Micallef smic@wire.net.au Subject: Setting up a Beowulf Cluster I am interested in setting up a Beowulf cluster, but am unsure of the specifics of setting one up. Can anyone direct me to documentation containing step-by-step instructions or detailed documentation on setting one up? Steven Micallef World Wire Unix Systems Administrator From jozef@fatra.ph.hunter.cuny.edu Mon, 18 Jan 1999 00:38:15 -0500 Date: Mon, 18 Jan 1999 00:38:15 -0500 From: Jozef Skvarcek jozef@fatra.ph.hunter.cuny.edu Subject: Summary: Hardware questions Hello, Thank you very much for the suggestions, I have received a lot of valuable responses so that I decided to write a short summary. It might be of some help for someone. Below are my questions and your assorted answers in no particular order: 1. Will a machine without keyboard, mouse and video card boot? Let's assume it has Asus P2B (P2L resp.) motherboard. If the positive answer depends on the MB or BIOS can you suggest type? - Asus P2B will allow to boot without keyboard and video indeed - however, it's good to keep cheap video card in place - any machine (PC) will boot without mouse, of course 2. If we change PII 400MHz (100MHz bus) to Celeron 400MHz (66MHz bus) on the worker nodes, will the performace drop `dramatically' i.e. more than if changing PII 400 to PII 350 (100MHz bus)? - "depends on application" seems to be the right answer, I didn't get any quantitave comparison, but it is expected to get similar performance for "cpu intensive" application since the bus speed should not play role in the case. - PII 300MHz (66MHz bus) and Celeron 300MHz do perform almost alike for "most" applications. - some advised to get Celeron (300) and overclock it (400-450), others opposed the idea because of fear of errors, hardware failure (looked like the list's regulars had a fight about the topic in the past, I do not want to start new one) - special care must be taken when picking a motherboard for Celeron 400/450 since many boards do not support 400MHz = 6 * 66MHz bus... 3. The cluster nodes will be hooked to a fast ethernet switch (I am thinking about SMC EZSwitch 100, 8ports since I was able to found the most comprehensive info about it) that will be connected to our network (10BT), obviously, I do not want to flood the cluster with other then cluster's traffic. My understanding is that the switch will not propagate such packets. Am I right or do I need to put the cluster behind firewall? - the switch will forward only packets destined for the computers plugged into its ports, however, it will propagate broadcasts. That's problem on our network plagued by Windoze machines. - it is better to install extra NIC on one node and then use it as a front end of the cluster. - some use port-to-port links instead of a switch, it is cheaper and faster solution (in the case when having fixed-low number of nodes and the application doesn't require communication among random nodes, I think) 4. Is it worth to get ECC memory? I don't remember seeing any problem (crashed computer, application, garbled data...) which could be related to erroreous memory using a few different types on non-brand non-ECC DRAM/EDO/SDRAM. Also, I guess the ECC RAM is slower since it probably employs an error checking algorithm. - it's better to have the ECC. There is many more memory chips in a typical cluster than in single computer, and single bad one can damage all the results. The probability of memory fault increases with lower memory space and with higher number of accesses. - the memory errors do occur, their symptoms not always allow correct identification of the cause Thank you once again, Jozef Skvarcek jskvarce@shiva.hunter.cuny.edu From alain.coetmeur@icdc.caissedesdepots.fr Mon, 18 Jan 1999 04:24:05 -0500 Date: Mon, 18 Jan 1999 04:24:05 -0500 From: Coetmeur, Alain alain.coetmeur@icdc.caissedesdepots.fr Subject: Unused Cycles or Cluster? -- Alain Coetmeur, Informatique-CDC DTA mailto:alain.coetmeur@icdc.CaisseDesDepots.fr > > What sort of software is needed on each machine? > [Bohn, Christopher A.] A message passing library, such as > MPI. For WinNT, > there's MPI Software Technology's MPI-Pro and a product by > PaTENT. (sorry, > I don't have URLs handy). see www.genias.de PatentMPI is the commercial/supported version of WMPI which is freware at: http://dsg.dei.uc.pt/w32mpi/intro.html it works well, and can use shared memory instead of sockets. moreover they can works with Unix MPICH/P4 as documented, so you can make Linux/NT clusters/now work together. I don't know anyway if Linux Beowulf MPI-CH is using p4 transport or something else. >For UNIX & Linux, there's MPICH > available from > http://www.mcs.anl.gov/mpi, and there's LAM (URL?); there's also PVM. > > > How would it affect other users? > [Bohn, Christopher A.] As long as you limit yourself to one > process per > workstation, the interactive users shouldn't notice. > on NT I've found an evident trick to run without the usen notice it (except on the CPU meter) I know this is evident, but this work far better that nice on Unix: change the thread priority to LOWEST SetThreadPriority( GetCurrentThread(), THREAD_PRIORITY_LOWEST); or even IDLE but then you pass after the screensaver which mean you have a few percent of the CPU time)... the machine really response as if you were not running and if the user work you get less that a few % of the CPU. beware of the Word Grammatical corrector (at least french Grammatik I use) that run in normal priority and such all CPU it is a pity Novell (Grammatik) and Microsoft(Word) don't know how to lower their priority and avoid slowing your machine... just using the below normal priority would make the machine run as light when checking (yet if you use lowes priority you get quite nothing anyway)... but that is not the point... the point is that if you control your priority all get as you want... From ssassinak@hotmail.com Mon, 18 Jan 1999 11:12:52 -0500 Date: Mon, 18 Jan 1999 11:12:52 -0500 From: Kim Moorman ssassinak@hotmail.com Subject: Call for Articles - Crossroads Magazine Crossroads, the Association for Computing Machinery Student Magazine Linux (Fall 1999) DUE DATE: March 2, 1999 SUBMISSION ADDRESS: xrds-submit@acm.org INFORMATION: crossroads@acm.org http://www.acm.org/crossroads/ The Crossroads editorial staff invites authors to submit articles dealing with topics drawn from several areas pertaining to Linux. The following partial list of topics is provided to give prospective authors ideas for articles and is by no means exhaustive; other relevant topics will be considered. -History and future of Linux -Interaction between Linux and other operating systems; Interaction between Linux and various windowing systems -Software development issues and projects; Compatibility and portability issues; Linux system administration -Linux and the Internet -Legal issues surrounding Linux and licensing -Productivity software and Linux -Linux Multimedia Development (e.g. 3D graphics rendering etc) Articles should include a basic description of the kinds of problems being worked on, the state of the art of research, the state of the art of commercial applications, open problems, or future research/commercial development trends. Interviews with researchers; reviews of related books, software, videos, or conferences; and opinion columns on related issues are also welcome. We especially encourage both undergraduate and graduate students to submit articles. However, articles written or coauthored by professionals will also be considered. Crossroads articles should be written for a broad audience. They should be easily understandable by someone who has had only the most basic computer science instruction, and yet still be interesting to the advanced computer enthusiast. Articles longer than 6000 words will generally not be considered for publication. Feature articles should be between 1500 and 6000 words; reviews should be between 800 and 2000 words; and opinion columns should be between 800 and 3000 words. Articles should be written in a magazine style rather than a research paper style. In consideration of our diverse readership, authors should try to use language that is inclusive of people regardless of their gender, race, religion, nationality, or field of study. Additional writing guidelines and submission information are available online at the Crossroads web site http://www.acm.org/crossroads/doc/information/writing.html. Crossroads is published both online and in print. We have a print circulation of about 15,000. All back issues are available for free on our website. Authors that have an article printed in Crossroads can receive complementary copies of the issue they were published in. All submissions should be formatted in HTML or plain text format and emailed to xrds-submit@acm.org. Please include your submission in the body of your message: DO NOT include it as an attachment. If you have any images or graphics, put them somewhere on your own website and use a full URL reference to them inside the article (example use img src= http://www.myhome.edu/me/pic1.gif). Submissions are due March 2, 1999. They will be reviewed shortly thereafter and authors of accepted submissions will be notified within two to three weeks of the deadline. For detailed submission guidelines, see http://www.acm.org/crossroads/doc/information/writing.html Prospective authors are invited to send email to the editors of Crossroads at crossroads@acm.org indicating their intention to submit an article. In this way we can keep everyone informed of any changes in deadlines or formats and make sure we have a good variety of articles. General questions should also be sent to the Crossroads editors. ______________________________________________________ Get Your Private, Free Email at http://www.hotmail.com From rich@freebsd.org Mon, 18 Jan 1999 12:51:00 -0500 Date: Mon, 18 Jan 1999 12:51:00 -0500 From: Murphey rich@freebsd.org Subject: Summary: Hardware questions >Date: Mon, 18 Jan 1999 00:43:26 -0500 (EST) >From: Jozef Skvarcek > >Hello, > >Thank you very much for the suggestions, I have received a lot of >valuable responses so that I decided to write a short summary. >It might be of some help for someone. >Below are my questions and your assorted answers in no particular order: > >1. Will a machine without keyboard, mouse and video card boot? >Let's assume it has Asus P2B (P2L resp.) motherboard. If the positive >answer depends on the MB or BIOS can you suggest type? > >- Asus P2B will allow to boot without keyboard and video indeed >- however, it's good to keep cheap video card in place Are there situations where $50 more RAM would be more benefit than a video card? That's what I'd realisticly expect to spend for a 'cheap' card if you limit yourself to the supliers of cluster hardware. I may be biased, but I'm tempted to choose more RAM and no video (or even hard disk) becuase CPU and RAM have a greater impact on performance in general. To go one step further, hard disks fail more often than anything else, which makes a diskless configuration that much easier to manage anyway. Unless you are working on large out-of-core problems, doubling your RAM instead of buying a disk reduces chance of swapping being an issue. >- any machine (PC) will boot without mouse, of course > >2. If we change PII 400MHz (100MHz bus) to Celeron 400MHz (66MHz bus) >on the worker nodes, will the performace drop `dramatically' i.e. >more than if changing PII 400 to PII 350 (100MHz bus)? > >- "depends on application" seems to be the right answer, > I didn't get any quantitave comparison, but it is expected to get > similar performance for "cpu intensive" application since the bus > speed should not play role in the case. >- PII 300MHz (66MHz bus) and Celeron 300MHz do perform almost alike > for "most" applications. >- some advised to get Celeron (300) and overclock it (400-450), > others opposed the idea because of fear of errors, hardware failure > (looked like the list's regulars had a fight about the topic in > the past, I do not want to start new one) >- special care must be taken when picking a motherboard for Celeron > 400/450 since many boards do not support 400MHz = 6 * 66MHz bus... > >3. The cluster nodes will be hooked to a fast ethernet switch (I am >thinking about SMC EZSwitch 100, 8ports since I was able to found >the most comprehensive info about it) that will be connected to our >network (10BT), obviously, I do not want to flood the cluster with >other then cluster's traffic. My understanding is that the switch >will not propagate such packets. Am I right or do I need to put >the cluster behind firewall? > >- the switch will forward only packets destined for the computers > plugged into its ports, however, it will propagate broadcasts. > That's problem on our network plagued by Windoze machines. >- it is better to install extra NIC on one node and then use it > as a front end of the cluster. >- some use port-to-port links instead of a switch, it is cheaper > and faster solution (in the case when having fixed-low number > of nodes and the application doesn't require communication among > random nodes, I think) 1) If you use direct connections you'll have to create routing tables. That's extra work and there isn't canned software to do this for you as far as I know. 2) The latency and the CPU usage will both increase if you route packets through compute nodes. Robert Brown showed some the performance impact of on idle v.s. busy CPUs on network performance. Rich Murphey >4. Is it worth to get ECC memory? I don't remember seeing any problem >(crashed computer, application, garbled data...) which could be related >to erroreous memory using a few different types on non-brand non-ECC >DRAM/EDO/SDRAM. Also, I guess the ECC RAM is slower since it probably >employs an error checking algorithm. > >- it's better to have the ECC. There is many more memory chips in > a typical cluster than in single computer, and single bad one can > damage all the results. The probability of memory fault increases > with lower memory space and with higher number of accesses. >- the memory errors do occur, their symptoms not always allow correct > identification of the cause > >Thank you once again, > >Jozef Skvarcek >jskvarce@shiva.hunter.cuny.edu > > > > From jozef@fatra.ph.hunter.cuny.edu Mon, 18 Jan 1999 13:19:06 -0500 Date: Mon, 18 Jan 1999 13:19:06 -0500 From: Jozef Skvarcek jozef@fatra.ph.hunter.cuny.edu Subject: Summary: Hardware questions On Mon, 18 Jan 1999, Murphey wrote: > >- Asus P2B will allow to boot without keyboard and video indeed > >- however, it's good to keep cheap video card in place > > Are there situations where $50 more RAM would be more > benefit than a video card? That's what I'd realisticly > expect to spend for a 'cheap' card if you limit > yourself to the supliers of cluster hardware. - $50 is not so much towards RAM (128MB ECC costs approx. $250) unless you have many such nodes - I plan to reuse leftover, obsolete cards anyway > greater impact on performance in general. To go one > step further, hard disks fail more often than anything > else, which makes a diskless configuration that much > easier to manage anyway. Unless you are working on > large out-of-core problems, doubling your RAM instead > of buying a disk reduces chance of swapping being an > issue. I understand, however, I plan to get a smaller, cheap Ultra ATA HD for a slave node. It makes the computer more versatile, we are planning to set up a sort of "dynamic" cluster where the nodes will be taken off and/or added in. If you want to get rid of swaping then just don't mount the swap partition. > >- some use port-to-port links instead of a switch, it is cheaper > > and faster solution (in the case when having fixed-low number > > of nodes and the application doesn't require communication among > > random nodes, I think) > > 1) If you use direct connections you'll have to create > routing tables. That's extra work and there isn't > canned software to do this for you as far as I know. You are right, that's why I believe it is a good solution for a cluster with fixed nodes and with a defined communication pattern. On the other hand, you save money not buying a switch. > 2) The latency and the CPU usage will both increase if > you route packets through compute nodes. Robert Brown > showed some the performance impact of on idle v.s. busy > CPUs on network performance. Have fun, Jozef Skvarcek jskvarce@shiva.hunter.cuny.edu From dpx@acl.lanl.gov Mon, 18 Jan 1999 17:01:25 -0500 Date: Mon, 18 Jan 1999 17:01:25 -0500 From: Dean Prichard dpx@acl.lanl.gov Subject: SGI/Cray "modules" for Linux? Hi, What we did at LANL is a perl script that does just what Greg suggested, that is module is an alias for: 'eval `${MODULESHOME}/bin/modulecmd.pl \!*`' this alias is set when you source the file $(MODULEHOME)/init/csh the modulecmd.pl is a script that parses a bunch of little shell scripts (one for each package) in the $(MODULESHOME)/modulefiles directory. you can grab our scripts and such from: http://www.acl.lanl.gov/~dpx/modules-0.3.tgz but i would first take a look at the module project: http://www.modules.org/ -dp ================================ Dean Prichard Advanced Computing Laboratory Los Alamos National Laboratory On Fri, 15 Jan 1999, Greg Lindahl wrote: > Date: Fri, 15 Jan 1999 12:07:26 -0500 (EST) > From: Greg Lindahl > To: tbaer@columbus.rr.com > Cc: lindahl@cs.virginia.edu, beowulf@beowulf.gsfc.nasa.gov, extreme-linux@acl.lanl.gov > Subject: Re: SGI/Cray "modules" for Linux? > > > GL> This is trivial to implement using a script. If the user wants LSF, > > GL> put LSF earlier in the PATH and MANPATH. > > > > OK, that's easy enough to do for a couple packages, but it doesn't > > scale well. > > What doesn't scale about it? > > > It doesn't let you easily change which version of a > > package you're without either making PATH ungodly long or logging out > > and logging back in. > > Logging in and out is not required. The usual trick for modifying your > environment applies: > > alias module eval `module.backend` > > > It doesn't track what packages your environment > > is currently set up for. > > Your script can do that, using a file, say $HOME/.modulerc. > > > The point is, I'm pretty sure somebody at LANL's already written this, > > so I'd prefer to use their code over reinventing it myself. I'm just > > trying to figure out if it's publically available. > > Sure. It would be nice for this to be available. I use a less flexible > scheme now to make things like mpich and legion available to my users > -- they have to eval a setup script before using the software. My > point was just that this is pretty easy to do, not that you shouldn't > look for existing software. > > -- g > From neil@causality.com Mon, 18 Jan 1999 17:40:45 -0500 Date: Mon, 18 Jan 1999 17:40:45 -0500 From: Neil A. Carson neil@causality.com Subject: Freedom CPU project Unfortunately, these days, people think just because they add 'free|open|gpl' to a project, stick up a web site and announce it on a geek news site, they can do it. A serious case of 'get real' is needed. Linux is very much a one-off. From mdw@cs.berkeley.edu Mon, 18 Jan 1999 18:01:42 -0500 Date: Mon, 18 Jan 1999 18:01:42 -0500 From: Matt Welsh mdw@cs.berkeley.edu Subject: Extreme Linux/Beowulf Magazine Article? Hi folks, I'm helping to put together articles for a new magazine on Linux which is aiming for launch at Linux Expo in March. One of the regular columns is on "Extreme Linux" and I'm looking for someone to write a short (about 2000 words) article on an Extreme Linux/Beowulf topic for this first issue. This can be on just about any Extreme Linux topic: Your own personal experience with a Linux cluster, an overview of fast network interfaces, a performance analysis of various MPI layers, a plug for your own research project, you name it: as technical or non-technical as you see fit. Any takers? If you'd like to write an article please let me know and we'll talk more! Thanks, Matt Welsh From sean@ntr.net Mon, 18 Jan 1999 23:06:14 -0500 Date: Mon, 18 Jan 1999 23:06:14 -0500 From: Sean McPherson sean@ntr.net Subject: Freedom CPU project -----BEGIN PGP SIGNED MESSAGE----- Geez, I hope the Apache people, the sendmail people, the BIND people (or just paul vixie), the FreeBSD people, or anyone involved in a lot of the code for dozens of other apps hears that so they can quit! Sean McPherson sean@ntr.net Systems Engineer Information Systems On Sat, 16 Jan 1999, Neil A. Carson wrote: > Unfortunately, these days, people think just because they add > 'free|open|gpl' to a project, stick up a web site and announce it on a > geek news site, they can do it. A serious case of 'get real' is needed. > Linux is very much a one-off. > Sean McPherson sean@ntr.net Systems Engineer Information Systems -----BEGIN PGP SIGNATURE----- Version: 2.6.3a Charset: noconv iQCVAwUBNqQEsXhe7DGjGXp9AQENYQP9HLKWr2mzUm2Jd8QbPe5RDDIYXPO6HfhA Qty8/iivbD9fqNomPyyzhz6gDTQkxpsPhRCZU+egfBi0UO4vRacTiFfiTRrFykQW Y4UmP7hGEbURkhcq4DMfTLV+QH8eIn78uvIBaAQECASxgP2z976mkC7WqTkA5Od4 2SJ7WpftcJY= =QtV2 -----END PGP SIGNATURE----- From levin@openprojects.net Tue, 19 Jan 1999 00:42:21 -0500 Date: Tue, 19 Jan 1999 00:42:21 -0500 From: Rob Levin levin@openprojects.net Subject: Freedom CPU project On Sat, 16 Jan 1999, Neil A. Carson wrote: > Unfortunately, these days, people think just because they add > 'free|open|gpl' to a project, stick up a web site and announce it on a > geek news site, they can do it. A serious case of 'get real' is needed. > Linux is very much a one-off. Certainly, it could be argued that Beowulf is the same sort of thing. In fact, if you make a list of all the one-off's, there might be some sort of trend operating there. If someone announces a project, what is it to you? If it's quite meaningless, ignore it and move on. Snide comments don't change the character of whatever is going on. They're noise. Rob L. From mario@regulus.pcs.usp.br Tue, 19 Jan 1999 13:37:17 -0500 Date: Tue, 19 Jan 1999 13:37:17 -0500 From: Mario Donato Marino mario@regulus.pcs.usp.br Subject: TreadMarks Hello! Does someone have some SPLASH II or any other PUBLIC DOMAIN benchmark ported to TreadMarks ? Thanks in advance, Mario From kragen@pobox.com Tue, 19 Jan 1999 15:07:35 -0500 Date: Tue, 19 Jan 1999 15:07:35 -0500 From: Kragen Sitaker kragen@pobox.com Subject: RAID5 on NBD (answer) Several people in the recent past have asked about doing redundant fileservice over the network -- PVFS, Coda, NFS, whatever. Here's a post from today on the Coda mailing list from a guy who said he backported the nbd (network block device) and is doing RAID5 over his network now. It also suggests that Coda is not really what you want to be running your cluster on at the moment. :) -- Kragen Sitaker Computers are the tools of the devil. It is as simple as that. There is no monotheism strong enough that it cannot be shaken by Unix or any Microsoft product. The devil is real. He lives inside C programs. -- philg@mit.edu ---------- Forwarded message ---------- Date: Tue, 19 Jan 1999 15:07:35 -0500 From: "Peter T. Breuer" To: codalist@TELEMANN.coda.cs.cmu.edu Cc: "Peter T. Breuer" Subject: Re: please help Resent-Date: Tue, 19 Jan 1999 14:01:35 -0500 Resent-From: codalist@TELEMANN.coda.cs.cmu.edu "A month of sundays ago Brian Bartholomew wrote:" > > I have the perception that coda is hard to build and install. > Actually, I have the perception that it's a complete and total > gelatinous mess, which every so often gets scraped up out of students' I agree, but don't want to be hit by the gelatin from the makers! Last time I said this was 6 months ago, when it took me a lot of effort to fix enough to compile, and then I never really got it working anyway. It took 100MB of disk space compiled then. A month or so ago when I tried again I was able to compile it with minimal intervention, and get it going exactly as per instructions - a big improvement. In fact, that there were instructions was a big improvement. > If you improve the release engineering so that I can get coda running > with one ftp, one tar, one configure, one make, and one shell script > to start the daemons -- I will do so! Simple as that. And so will a > lot of other people, I think. Yep - and in particular I would like people like me (well, I like me, anyway :-) who mail in a set of patches that they had to use to at least get an acknowledge back. I never heard anything. I could probably remake the patches. The problem probably is that the developers are working on 5.0 and releasing 4.6, and sorry, I'm hardly likely to be interested in 5.0 while you're developing it and you're hardly likely to be interested in 4.6 while you're not! > > Hopefully you can write some testing software for us > > If you can deliver the filesystem in a nice package, I'll be happy to > break it for you. Please supply two scripts: One script that resets Agreed. > >From what little I see on the announcement list, you're quite firmly a > Cathedral. This is not from an autocratic style, but simply because > you're the only people that understand the design and the internals Yep - I can't make head or tail of it. It being so monolithic drove me to backport the nbd driver and now I have raid5 working nicely over the net. I don't want the disconnected facilities of coda. I just want redundancy. If that comes via failover, well and good, but I only needed 1000 lines of code to do it via redundancy. > well enough to do anything. First attract a bunch of bug reporters > with an easy-to-install product. Then some of them will slowly wade > into the code, after they've absorbed its behavior from their bug > reporting. Yep. It's quite impervious to examination at the mo. When I looked at it first the names of the demons weren't the same as those used in the docs - I understand why, and that's just a forinstance for why it is/was impossible to get up and running then. I see claims that people have installed it, and I maybe believe that the rpms's helped (hint: take the rpms, examine their spec, then undo the redhat-isms, and compile to taste: gives you a workable product) for people running exactly the same system as the rpms were written on. But that's not interesting to the rest of us. Peter ptb2it.uc3m.es From ralfs@heineken.chemie.uni-dortmund.de Wed, 20 Jan 1999 03:57:29 -0500 Date: Wed, 20 Jan 1999 03:57:29 -0500 From: Ralf Schmelter ralfs@heineken.chemie.uni-dortmund.de Subject: Performance problem with mpich Hello! I'm working on a parallel MD-code and encountered the following problem: The performance of the MPI_Sendrecv() command seems very poor. The code looks like this (nodes are the number of nodes and my_number is the number of the node): int send_to = (my_number + 1) % nodes; int get_from = (my_number - 1 + nodes) % nodes; MPI_Status status; for (int j = 1; j < nodes; ++j) { MPI_Sendrecv(x + send_to * size, size, MPI_DOUBLE, send_to, 0, x + my_number * size, size, MPI_DOUBLE, get_from, 0, MPI_COMM_WORLD, &status); send_to = (send_to + 1) % nodes; get_from = (get_from - 1 + nodes) % nodes; } If I'm using 2 nodes I'll get a throughput of about 5 MBytes/s. If I'm using 3 or more nodes, the performance drops to 3 MBytes/s and is stable regardless of the number of nodes. I'm using a NIC (Level one FNC-0100TX), which uses a DEC 21140 and the driver from Donald Becker (v0.89H) in full-duplex mode (options=16). The Nodes consists of PII-400 MHz (on a BX-Board) and are connected by a FastEthernet switch (also in full-duplex mode) from Allied Telesyn (AT-8124). The version of mpich is 1.1.1. Netperf calculates a troughput of 94 MBits/s and a normal MPI_Send() has a troughput of 9 MBytes/s. If I try to simulate the MPI_Sendrecv() with a MPI_Isend / MPI_Irecv - pair the performance is as low as with the MPI_Sendrecv (in fact it is allmost the same). Does anyone have any idea how to improve the performance of this code? Ralf Schmelter P.S.: In the caclulation of the troughput of the MPI_Sendrecv() function I used size as the number of transfered doubles, not 2 * size (because I'm using the full-duplex mode) -- Ralf Schmelter Dep. of Physical Chemistry University of Dortmund e-mail: ralfs@heineken.chemie.uni-dortmund.de From deadline@plogic.com Wed, 20 Jan 1999 09:19:40 -0500 Date: Wed, 20 Jan 1999 09:19:40 -0500 From: Douglas Eadline deadline@plogic.com Subject: Performance problem with mpich On Wed, 20 Jan 1999, Ralf Schmelter wrote: > Hello! > > I'm working on a parallel MD-code and encountered the following problem: > The performance of the MPI_Sendrecv() command seems very poor. The code > looks like this (nodes are the number of nodes and my_number is the > number of the node): > > int send_to = (my_number + 1) % nodes; > int get_from = (my_number - 1 + nodes) % nodes; > MPI_Status status; > > for (int j = 1; j < nodes; ++j) > { > MPI_Sendrecv(x + send_to * size, size, MPI_DOUBLE, send_to, 0, > x + my_number * size, size, MPI_DOUBLE, get_from, 0, > MPI_COMM_WORLD, &status); > send_to = (send_to + 1) % nodes; > get_from = (get_from - 1 + nodes) % nodes; > } > > If I'm using 2 nodes I'll get a throughput of about 5 MBytes/s. If I'm > using 3 or more nodes, the performance drops to 3 MBytes/s and is stable > regardless of the number of nodes. > I'm using a NIC (Level one FNC-0100TX), which uses a DEC 21140 and the > driver from Donald Becker (v0.89H) in full-duplex mode (options=16). > The Nodes consists of PII-400 MHz (on a BX-Board) and are connected by a > FastEthernet switch (also in full-duplex mode) from Allied Telesyn > (AT-8124). The version of mpich is 1.1.1. > Netperf calculates a troughput of 94 MBits/s and a normal MPI_Send() has a > troughput of 9 MBytes/s. If I try to simulate the MPI_Sendrecv() with a > MPI_Isend / MPI_Irecv - pair the performance is as low as with the > MPI_Sendrecv (in fact it is allmost the same). > > Does anyone have any idea how to improve the performance of this code? In my opinion, MPICH is not stable on LINUX. There seems to be some type of problem with the tcp. I suggest using LAM-MPI http://www.mpi.nd.edu/lam/ It works quite well and is faster than MPICH (using -c2c mode). Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From mack@nesc.epa.gov Wed, 20 Jan 1999 12:41:11 -0500 Date: Wed, 20 Jan 1999 12:41:11 -0500 From: Joseph Mack mack@nesc.epa.gov Subject: networking data, global /proc space I'm about to build my 2nd beowulf. This one will be a high end machine (if it's funded). I notice that nearly all machines are switched fast ethernet. The very high end machines are Myrinet. I've looked at the webpages of all the ones I;ve found but not seen much comparison between the two and at what point you would want to change over. Presumably you buy what you can afford. Is any comparison available? I notice few people are using Gbit ethernet and no-one is using SCI - any reason for this? Is there any point in having a global /proc space in (say) a Linux beowulf. This would allow local calls to find which machines/disks/network cards were busy. I assume this would be easier to use than say SMTP. Thank you Joe -- Joseph Mack, Senior Systems Engineer, Lockheed Martin National Environmental Supercomputer Center (NESC), mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA From lindahl@cs.virginia.edu Wed, 20 Jan 1999 13:38:19 -0500 Date: Wed, 20 Jan 1999 13:38:19 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: networking data, global /proc space > I notice that nearly all machines are switched > fast ethernet. The very high end machines are Myrinet. I've looked > at the webpages of all the ones I;ve found but not seen much > comparison between the two and at what point you would want > to change over. Presumably you buy what you can afford. Is any > comparison available? If you know what application that you want to run, then you should know if it runs adequately using fast ethernet. If the answer is "no", then you should consider Myrinet. > I notice few people are using Gbit ethernet and no-one is using SCI > - any reason for this? Gigabit cards are expensive, the switches are small and expensive, and today's drivers are slow. With tomorrow's cheaper, bigger switches and user-level drivers, this may change. Ditto for SCI. > Is there any point in having a global /proc space in (say) a Linux > beowulf. This would allow local calls to find which > machines/disks/network > cards were busy. I assume this would be easier to use than say SMTP. The pain of making such a beast outweighs the benefit, in my mind. Most folks use a queuing system to dedicate nodes to jobs, or they dedicate the whole machine to one job at a time. Then you don't care who's busy. -- g From lindahl@cs.virginia.edu Wed, 20 Jan 1999 13:46:04 -0500 Date: Wed, 20 Jan 1999 13:46:04 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: Performance problem with mpich > I'm working on a parallel MD-code and encountered the following problem: > The performance of the MPI_Sendrecv() command seems very poor. The code > looks like this (nodes are the number of nodes and my_number is the > number of the node): Your code implements an "all to all" sending pattern, except you have invented your own algorithm, which is done in a style appropriate for a Cray T3E, but is pretty much guaranteed to not work optimally on a cluster. You are usually best off doing all your sending before you do any of your receiving, or even calling MPI_AllToAll... -- greg From Christopher.Bohn@afit.af.mil Wed, 20 Jan 1999 14:35:28 -0500 Date: Wed, 20 Jan 1999 14:35:28 -0500 From: Bohn, Christopher A Christopher.Bohn@afit.af.mil Subject: networking data, global /proc space > From: Greg Lindahl [SMTP:lindahl@cs.virginia.edu] > Sent: Wednesday, January 20, 1999 1:38 PM > > I notice that nearly all machines are switched > > fast ethernet. The very high end machines are Myrinet. I've looked [...] > > I notice few people are using Gbit ethernet and no-one is using SCI > > - any reason for this? > > Gigabit cards are expensive, the switches are small and expensive, and > today's drivers are slow. With tomorrow's cheaper, bigger switches and > user-level drivers, this may change. Ditto for SCI. > [Bohn, Christopher A.] Also, IIRC, for x86-based systems (vice Alpha-based systems), the TCP & UDP throughput is too limited to support beyond 150-200Mbps, which is fine for Fast Ethernet & channel-bonded Fast Ethernet. But it would not take advantage of Gbit Ethernet. With the dedicated communications processor on the Myrinet NIC, though, you can get around this with "fast messages," "bulldog messages," and the like. Take care, cb *-*-*-*-*-*-*-* Capt Christopher A. Bohn Graduate Student, Electrical (digital) Engineering Air Force Institute of Technology Phone (937)255-3636 (DSN 785) AFIT/EN638 Lab x4606 Voicemail x6638 2950 P St, Box 4638 email Christopher.Bohn@afit.af.mil Wright-Patterson AFB OH 45433-7765 EngrBohn@aol.com http://members.aol.com/EngrBohn/ *-*-*-*-*-*-*-* > From chrism@brahma.ticam.utexas.edu Wed, 20 Jan 1999 15:01:41 -0500 Date: Wed, 20 Jan 1999 15:01:41 -0500 From: chris mccraw chrism@brahma.ticam.utexas.edu Subject: networking data, global /proc space >If you know what application that you want to run, then you should >know if it runs adequately using fast ethernet. If the answer is "no", >then you should consider Myrinet. agreed. our particular application set was up against a network bottleneck with fast ethernet and 16 nodes, so we moved to myrinet for the 64 node cluster. (because, unfortunately, communication needs grow with the numbers of workers in our applications) >> I notice few people are using Gbit ethernet and no-one is using SCI >> - any reason for this? > >Gigabit cards are expensive, the switches are small and expensive, and >today's drivers are slow. With tomorrow's cheaper, bigger switches and >user-level drivers, this may change. Ditto for SCI. altho i have no personal experience with the company, a fellow from foundry networks (www.foundrynet.com) called just yesterday to tell me that he had a 64port gigabit switch in the ~$100k range. i neglected to ask for details because, well, we already bought myrinet and i was kind of busy. if anyone talks to them and actually specs a product, i'd love to hear about it... no comment on the speed of gigabit ethernet drivers, because i have no experience there. -- chris mccraw | network operations | TICAM | university of texas at austin From deadline@plogic.com Wed, 20 Jan 1999 15:15:33 -0500 Date: Wed, 20 Jan 1999 15:15:33 -0500 From: Douglas Eadline deadline@plogic.com Subject: Some News THIS JUST RELEASED: Because of a minor technical problem release of Windows 2000 is delayed to first quarter of 1901. Sorry, I could not resist. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From walt@parl.ces.clemson.edu Wed, 20 Jan 1999 15:22:51 -0500 Date: Wed, 20 Jan 1999 15:22:51 -0500 From: Walter B. Ligon III walt@parl.ces.clemson.edu Subject: networking data, global /proc space -------- > I'm about to build my 2nd beowulf. This one will be a high end > machine (if it's funded). > > I notice that nearly all machines are switched > fast ethernet. The very high end machines are Myrinet. I've looked > at the webpages of all the ones I;ve found but not seen much > comparison between the two and at what point you would want > to change over. Presumably you buy what you can afford. Is any > comparison available? NO, but someone needs to do one. We are installing Myrinet, maybe we'll do one in the near future. > > I notice few people are using Gbit ethernet and no-one is using SCI > - any reason for this? > > Is there any point in having a global /proc space in (say) a Linux > beowulf. This would allow local calls to find which > machines/disks/network > cards were busy. I assume this would be easier to use than say SMTP. What does a global /proc space have to do with delivery of email? Maybe I'm missing something. Otherwise, a global pid space is a good thing by our way of thinking. We are looking at different techniques for doing that and we'd be interested in any discussion of that. Walt -- Dr. Walter B. Ligon III Associate Professor ECE Department Clemson University From mack@nesc.epa.gov Wed, 20 Jan 1999 15:59:17 -0500 Date: Wed, 20 Jan 1999 15:59:17 -0500 From: Joseph Mack mack@nesc.epa.gov Subject: networking data, global /proc space "Walter B. Ligon III" wrote: > > -------- > > > > Is there any point in having a global /proc space in (say) a Linux > > beowulf. This would allow local calls to find which > > machines/disks/network > > cards were busy. I assume this would be easier to use than say SMTP. > ack sorry SNMP Joe > What does a global /proc space have to do with delivery of email? Maybe -- Joseph Mack, Senior Systems Engineer, Lockheed Martin National Environmental Supercomputer Center (NESC), mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA From glamm@ece.umn.edu Wed, 20 Jan 1999 16:08:23 -0500 Date: Wed, 20 Jan 1999 16:08:23 -0500 From: Bob Glamm glamm@ece.umn.edu Subject: Performance problem with mpich (fwd) > I'm working on a parallel MD-code and encountered the following problem: > The performance of the MPI_Sendrecv() command seems very poor. The code > looks like this (nodes are the number of nodes and my_number is the > number of the node): Your code implements an "all to all" sending pattern, except you have invented your own algorithm, which is done in a style appropriate for a Cray T3E, but is pretty much guaranteed to not work optimally on a cluster. You are usually best off doing all your sending before you do any of your receiving, or even calling MPI_AllToAll... Unless things have changed, the MPICH implementation of MPI_AllToAll() isn't very good either (it uses a naive linear point-to-point communication for *every* node). Someone really needs to rewrite the reference implementation to do hierarchical collective operations. -Bob From deadline@plogic.com Wed, 20 Jan 1999 16:29:06 -0500 Date: Wed, 20 Jan 1999 16:29:06 -0500 From: Douglas Eadline deadline@plogic.com Subject: networking data, global /proc space On Wed, 20 Jan 1999, Joseph Mack wrote: > I'm about to build my 2nd beowulf. This one will be a high end > machine (if it's funded). > > I notice that nearly all machines are switched > fast ethernet. The very high end machines are Myrinet. I've looked > at the webpages of all the ones I;ve found but not seen much > comparison between the two and at what point you would want > to change over. Presumably you buy what you can afford. Is any > comparison available? As it has been stated, it all depends on the application. usually you want to maximize your performance for a set dollar amount. This involves testing and benchmarking (even using our BERT tool to get estimates on various platforms) In general, with Myrinet you are trading CPUs for bandwidth, so if your application can use FE, then you usually can afford more nodes. We do need some benchmarks for Myricom vs. FE. > > I notice few people are using Gbit ethernet and no-one is using SCI > - any reason for this? I think the technology is still a little new. GE is used mostly as a backbone technology. Perhaps the copper spec will change this. SCI is interesting, but drivers may be a problem. > > Is there any point in having a global /proc space in (say) a Linux > beowulf. This would allow local calls to find which > machines/disks/network > cards were busy. I assume this would be easier to use than say SMTP. DOug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From lindahl@cs.virginia.edu Wed, 20 Jan 1999 16:30:55 -0500 Date: Wed, 20 Jan 1999 16:30:55 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: networking data, global /proc space > altho i have no personal experience with the company, a fellow from > foundry networks (www.foundrynet.com) called just yesterday to tell > me that he had a 64port gigabit switch in the ~$100k range. That's still way expensive. A gigabit network consists of nic cards, cables, and the switch. Myrinet breaks down to: nic card: $1400/node cables: $150/node switch: $250/node == $1700/node Gigabit ethernet cards are falling to less than $1000 and the fiber optic cables aren't so bad. But that switch is $1500/port. Best I've seen so far for so big, but not there yet. -- g From chrism@brahma.ticam.utexas.edu Wed, 20 Jan 1999 17:05:49 -0500 Date: Wed, 20 Jan 1999 17:05:49 -0500 From: chris mccraw chrism@brahma.ticam.utexas.edu Subject: tcp speed >[Bohn, Christopher A.] Also, IIRC, for x86-based systems (vice Alpha-based >systems), the TCP & UDP throughput is too limited to support beyond >150-200Mbps, which is fine for Fast Ethernet & channel-bonded Fast Ethernet. >But it would not take advantage of Gbit Ethernet. With the dedicated >communications processor on the Myrinet NIC, though, you can get around this >with "fast messages," "bulldog messages," and the like. just fyi, using the myrinet "GM" drivers (http://www.myri.com/GM), we see tcp speeds of ~46MByte/s with +/-1MB/s depending on packet size. we haven't tried using the BIP stuff yet altho we plan to take a look... (http://lhpca.univ-lyon1.fr/) but their published numbers aren't as nice as the ones we got...of course we're using pII-400's instead of ppro-200's.. From newt@hq.nasa.gov Wed, 20 Jan 1999 17:33:24 -0500 Date: Wed, 20 Jan 1999 17:33:24 -0500 From: Daniel Ridge newt@hq.nasa.gov Subject: networking data, global /proc space On Wed, 20 Jan 1999, Greg Lindahl wrote: > That's still way expensive. A gigabit network consists of nic cards, > cables, and the switch. Myrinet breaks down to: > > nic card: $1400/node > cables: $150/node > switch: $250/node == $1700/node > > Gigabit ethernet cards are falling to less than $1000 and the fiber > optic cables aren't so bad. But that switch is $1500/port. Best I've > seen so far for so big, but not there yet. I have ordered several NetGear gigabit ethernet cards for $288 each. That's not even twice the cost of the Myrinet _cable_. -Dan -------------------------------------+--------------------------------- Daniel Ridge | Computer Crime Division | N A S A email: dridge@hq.nasa.gov | Office of Inspector General tel: 202-358-1901 | 300 E Street SW fax: 202-358-3439 | Washington, D.C. 20546 NexTel: 301-440-9153 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From lindahl@cs.virginia.edu Wed, 20 Jan 1999 18:53:04 -0500 Date: Wed, 20 Jan 1999 18:53:04 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: Performance problem with mpich (fwd) > Unless things have changed, the MPICH implementation of > MPI_AllToAll() isn't very good either (it uses a naive linear > point-to-point communication for *every* node). Someone really > needs to rewrite the reference implementation to do hierarchical > collective operations. Patrick Worley has done the writing -- he has a test suite which he runs to verify that vendor implementations don't use algorithms worse than the obvious ones. All someone needs to do is to incorporate that code into mpich, preferably so that the algorithm can dynamically be picked at run-time, based on benchmarking. http://www.epm.ornl.gov/~worley/ -- g From lindahl@cs.virginia.edu Wed, 20 Jan 1999 18:55:19 -0500 Date: Wed, 20 Jan 1999 18:55:19 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: networking data, global /proc space > I have ordered several NetGear gigabit ethernet cards for $288 each. > That's not even twice the cost of the Myrinet _cable_. Great. Too bad the switch costs more than the entire Myrinet setup. -- g From greg@varesearch.com Wed, 20 Jan 1999 19:12:10 -0500 Date: Wed, 20 Jan 1999 19:12:10 -0500 From: Greg Kucharo greg@varesearch.com Subject: Beowulf/2.2 kernel What amount of the "Beowulf" tweaks and drivers that make the Beowulf kernel seperate from a regular Linux distro are now included in the 2.2 kernel? What exactly makes the Extreme Linux distro different from the standard Redhat? Thanks Greg Kucharo -- _________________________________________________________ Greg Kucharo "Evil works a lot better when it's subtle" VA Research -Joel Robinson-MST3K From rgb@phy.duke.edu Wed, 20 Jan 1999 23:32:50 -0500 Date: Wed, 20 Jan 1999 23:32:50 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: networking data, global /proc space On Wed, 20 Jan 1999, Daniel Ridge wrote: > > On Wed, 20 Jan 1999, Greg Lindahl wrote: > > > That's still way expensive. A gigabit network consists of nic cards, > > cables, and the switch. Myrinet breaks down to: > > > > nic card: $1400/node > > cables: $150/node > > switch: $250/node == $1700/node > > > > Gigabit ethernet cards are falling to less than $1000 and the fiber > > optic cables aren't so bad. But that switch is $1500/port. Best I've > > seen so far for so big, but not there yet. > > I have ordered several NetGear gigabit ethernet cards for $288 each. > That's not even twice the cost of the Myrinet _cable_. While we're on the subject of cheap GbE, there were rumors that UTP copper NIC's were due to hit the market about now which should significantly drop the price of all the components. I couldn't find any the last pass I made with a web search engine, but that was over a month ago. Anybody know anything more concrete? BTW, if one can connect GbE NIC's directly via crossover (as one can with regular FE NIC's) and prices are indeed in the $300 range, one could build small hypercubes with no switch at all with prices that would appear to be quite competitive. If/when switch prices drop, one could break the NIC's out of the cube and convert to more switched nodes. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From tbaer@columbus.rr.com Thu, 21 Jan 1999 00:49:46 -0500 Date: Thu, 21 Jan 1999 00:49:46 -0500 From: Troy A. Baer tbaer@columbus.rr.com Subject: networking data, global /proc space >>>>> "WBL" == Walter B Ligon writes: WBL> -------- >> I'm about to build my 2nd beowulf. This one will be a high end >> machine (if it's funded). >> >> I notice that nearly all machines are switched >> fast ethernet. The very high end machines are Myrinet. I've looked >> at the webpages of all the ones I;ve found but not seen much >> comparison between the two and at what point you would want >> to change over. Presumably you buy what you can afford. Is any >> comparison available? WBL> NO, but someone needs to do one. We are installing Myrinet, maybe we'll do WBL> one in the near future. We're working on it. One of the stated goals of the Beowulf cluster project at OSC is to do a detailed comparison of the currently available communications technologies (switched 100Mbps Ethernet, switched Myrinet, and switched SCI). We hope to have a paper ready for SC99. --Troy -- Troy Baer, MS(AAE) I may have wasted all those years; tbaer@columbus.rr.com They're not worth their time in tears. http://home.columbus.rr.com/tbaer/ I may have spent too long in darkness, Linux: hex, bugs, and rock'n'roll. In the warmth of my fears. --Dream Theater From rauch@inf.ethz.ch Thu, 21 Jan 1999 02:58:59 -0500 Date: Thu, 21 Jan 1999 02:58:59 -0500 From: Felix Rauch rauch@inf.ethz.ch Subject: networking data, global /proc space On Thu, 21 Jan 1999, Troy A. Baer wrote: > We're working on it. One of the stated goals of the Beowulf cluster > project at OSC is to do a detailed comparison of the currently > available communications technologies (switched 100Mbps Ethernet, > switched Myrinet, and switched SCI). We hope to have a paper ready > for SC99. Colleagues here at the institute made a comparison SCI vs. Myrinet (without switches though). I thought you might be interested in the paper: Ch. Kurmann, T. Stricker. A Comparison of two Gigabit SAN/LAN technologies: Scalable Coherent Interface versus Myrinet. Appears in Proceedings of the SCI Europe'98 Conference, EMMSEC'98, 28-30 Sept 1998, Bordeaux, France. http://www.cs.inf.ethz.ch/CoPs/publications/ - Felix -- Felix Rauch | Email: rauch@inf.ethz.ch Institute for Computer Systems | Homepage: http://www.cs.inf.ethz.ch/~rauch/ ETH Zentrum / RZ H15 | Phone: ++41 1 632 7489 CH - 8092 Zuerich / Switzerland | Fax: ++41 1 632 1307 From walt@parl.ces.clemson.edu Thu, 21 Jan 1999 10:26:29 -0500 Date: Thu, 21 Jan 1999 10:26:29 -0500 From: Walter B. Ligon III walt@parl.ces.clemson.edu Subject: networking data, global /proc space -------- Actually, this is a subject I find rather interesting, so I'd like to explore it a little more. Over and over I have heard that the problem with Myrinet is that you can't actually use the 2.5Gbps bandwidth in and out of the NIC. This is certainly true, but not really the point. The beauty of Myrinet is that you can build a large network of virtually any topology with the bandwidth you need. The issue isn't packets in and out of a NIC, but packets between switching devices. The best reason I can think of for using Myrinet is building a LARGE cluster. If you are only using 16 machines, it probably isn't worth it. Having a single Myrinet switch with one node on each port is a huge waste of bandwidth. The power comes in connecting several switches. If a single node can only produce 200Mbps then five nodes on one switch can easily share a single link to another switch, thus it becomes reasonable to expand the network to rather large sizes. This is no surprise since the devices used to implement Myrinet were originally designed for the Intel Paragon MPP which supported up to thousands of processors. Of course the Myrinet versus Gb ethernet is a different issue. Another way to build medium to large clusters is the use 100Mb ethernet switches connected by Gb ethernet links. Still another approach is to use a Gb switch with Gb NICs in the nodes, but put several nodes on each port (again, the nodes can't begin to saturate the link by themselves). Still, Myrinet has the advantage that you can connect the switches using a variety of topologies That give you even more flexibility than using Gb ethernet. Also Myrinet has much lower latency - which is important for some applications. So I suggest that Myrinet has its place - there are things you can only do with Myrinet, and ethernet based technology will not easily replace it. It is only useful if your application needs the maximum performance, and usually for very large clusters. In the mean time, there is a lot of experimentation to be done to understand how different combinations of network technology can work together to provide performance, flexibility, and scalability. The members of this community are the ones to study this and eventually provide guidance on the building of Beowulf clusters. It has long been my contention that Beowulf is not something that is complete, known, and understood, but rather something that is in it infancy. There are many many areas of exploration, improvement, and development that need study before we will have something that lives up to the potential that Beowulf has to offer. This area is one example. While there are many people out there that are putting Beowulf into production today, we must realize that every cluster and every application is an important experience that contributes to the growth of Beowulf. There are simply too many issues where WE DON'T KNOW the "right" way to do things. There is a tendency in a community such as this one to quickly form opinions in an attempt to finalize the knowledge, but as is often the case, the opinions are often premature. So I suggest we try to keep an open mind - I consider it a challenge to find a use for every applicable technology and understand its ramifications. We all must play the role of scientist and keep in mind the level of uncertainty that is present. I sure am glad to hear that a number of you ARE working in this area, and I hope the results of that work are made available and announced on this list. Clearly I have preached long enough, so until next time ... Walt > > From: Greg Lindahl [SMTP:lindahl@cs.virginia.edu] > > Sent: Wednesday, January 20, 1999 1:38 PM > > > I notice that nearly all machines are switched > > > fast ethernet. The very high end machines are Myrinet. I've looked > [...] > > > I notice few people are using Gbit ethernet and no-one is using SCI > > > - any reason for this? > > > > Gigabit cards are expensive, the switches are small and expensive, and > > today's drivers are slow. With tomorrow's cheaper, bigger switches and > > user-level drivers, this may change. Ditto for SCI. > > > [Bohn, Christopher A.] Also, IIRC, for x86-based systems (vice Alpha-based > systems), the TCP & UDP throughput is too limited to support beyond > 150-200Mbps, which is fine for Fast Ethernet & channel-bonded Fast Ethernet. > But it would not take advantage of Gbit Ethernet. With the dedicated > communications processor on the Myrinet NIC, though, you can get around this > with "fast messages," "bulldog messages," and the like. > > Take care, > cb -- Dr. Walter B. Ligon III Associate Professor ECE Department Clemson University From lindahl@cs.virginia.edu Thu, 21 Jan 1999 10:53:39 -0500 Date: Thu, 21 Jan 1999 10:53:39 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: networking data, global /proc space > While we're on the subject of cheap GbE, there were rumors that UTP > copper NIC's were due to hit the market about now which should > significantly drop the price of all the components. I don't think the switch prices are driven by copper vs. fiber. > BTW, if one can connect GbE NIC's directly via crossover (as one can > with regular FE NIC's) and prices are indeed in the $300 range, one > could build small hypercubes with no switch at all with prices that > would appear to be quite competitive. And awful performance. You don't expect nodes to forward packets and compute at the same time, do you? -- g From rgb@phy.duke.edu Thu, 21 Jan 1999 12:46:42 -0500 Date: Thu, 21 Jan 1999 12:46:42 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: networking data, global /proc space On Thu, 21 Jan 1999, Greg Lindahl wrote: > > While we're on the subject of cheap GbE, there were rumors that UTP > > copper NIC's were due to hit the market about now which should > > significantly drop the price of all the components. > > I don't think the switch prices are driven by copper vs. fiber. Maybe not, but fiber has stayed considerably more expensive than copper in just about all networking contexts I've ever seen them both represented in, probably because of sales volume more than anything else (you can charge less if you sell lots, and copper is cheap and plentiful and easy to install and manage and hence common). It's strange to me because intuitively I'd expect the opposite -- technically I'd expect light to be easier to generate and modulate with bandwidth to spare. Why do you say that GbE is now or more properly will be different? > > BTW, if one can connect GbE NIC's directly via crossover (as one can > > with regular FE NIC's) and prices are indeed in the $300 range, one > > could build small hypercubes with no switch at all with prices that > > would appear to be quite competitive. > > And awful performance. You don't expect nodes to forward packets and > compute at the same time, do you? In an SMP system I might. One CPU to handle the routing and networking, the other to do the computing, since wasting a CPU is irrelevant in an IPC-bound problem. There are also problem/cube topologies that would minimize or eliminate routing, for example a tetrahedron would permit static routes betwee all four nodes with 3 NICs, and a 3-cube with opposing corners connected would provide 16 direct connections (the cube edges and the corner connections) and would require one hop for the 12 remaining face-diagonal pairs. This would require (static) routing for only 3/7 of the node pairs, and in the event that all nodes have to share their data with all nodes anyway, a store-and-forward pattern (A sends to B, B sends its own AND A's data to C) would effectively eliminate almost all the routing overhead, leaving only the partial sharing of the bandwidth. Since GbE NICs are bound to use DMA (send it and forget it) there should be considerable parallelism in actual data transmission or reception and calculation, once a packet is set up to send or receive. It would be a pain to program, but it would be cheap and might work well for some problems. I think that this would be an interesting and fun thing to experiment with and it sounds (from earlier responses) like some people may be playing with it now. But sure, my real hope is that GbE over copper historically recapitulates FE over copper and that NIC and copper switch prices plunge (regardless of what they are now) as volumes swell and comnpetition kicks in and technology marches on. I know that you like Myrinet and for good reason (it is indeed technically excellent, from what I have read), but I'm holding out for GbE because I see it becoming the true COTS gigabit network in a couple of years regardless of technical merit and price matters to me. I'm tentatively plannig to build a small hypercube just as a relatively cheap experiment with GbE, but I certainly plan to recycle the NICs into a switched configuration as soon as it becomes cheap if I do. If NIC prices don't come down fast enough, I may do the experiment with $30 FE NICs (even easier to recycle when I'm done, and I can hope for at least approximate scaling). rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From sean@ntr.net Thu, 21 Jan 1999 12:50:54 -0500 Date: Thu, 21 Jan 1999 12:50:54 -0500 From: Sean McPherson sean@ntr.net Subject: Switch Boxes Wow. Every time I think I have a list of all the parts I'm gonna need for a beowulf, I find something else. Specs never stay the same, do they? :-) I now have to expect to connect the beowulf cluster into a switching box to let me keep from having keyboards, mice, and monitor cables strung all over the place. I know, just ignore them. That's not an option for me for reasons far too lengthy to explain here. This switch box will need to emulate a keyboard and mouse attached all the time, so I can't use a $10 keyboard/switch box from CompUSA, much as I'd like to. If I also have to plug an SGI and a NT box into this switcher as frontend machines (don't ask, it ain't pretty) I can't have them locking up when the keyboard and mouse goes away (as NT is apt to do), or just not seeing them when I switch back. So, can anyone point me at what they've used? Maybe not in your beowulf, but in your data centers, etc? I've seen KVM, but I know they cost a bundle, and do silly things I don't need like adding sound support :-) I may go with a KVM switch if I have to, but the big thing I need is to be able to plug into PC's running Linux and NT (and SGI would be nice but not necessarily required), and to emulate the kb/mouse all the time. Thanks in advance, (as I pray one day I get to actually put this sucker together!), Sean McPherson sean@ntr.net Systems Engineer Information Systems From lindahl@cs.virginia.edu Thu, 21 Jan 1999 12:58:00 -0500 Date: Thu, 21 Jan 1999 12:58:00 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: networking data, global /proc space > > I don't think the switch prices are driven by copper vs. fiber. > > Maybe not, but fiber has stayed considerably more expensive than copper Um, that's only the cost of the piece that attaches the device to other devices. When you have a 64-port switch costing $100,000, the extra you're paying for fiber verses copper is small. The cost is switching all those gigabits, which is the same for both copper and fiber. > But sure, my real hope is that GbE over copper > historically recapitulates FE over copper and that NIC and copper switch > prices plunge (regardless of what they are now) as volumes swell and > comnpetition kicks in and technology marches on. Everyone knows prices will plunge; that has little to do with copper verses fiber. > I know that you like > Myrinet and for good reason (it is indeed technically excellent, from > what I have read), but I'm holding out for GbE because I see it becoming > the true COTS gigabit network in a couple of years regardless of > technical merit and price matters to me. My crystal ball doesn't go out a couple of years. I'm building clusters now. If you want to experiment with GbE, then perhaps you should work on the driver issues, as they are a killer today. > If NIC > prices don't come down fast enough, I may do the experiment with $30 FE > NICs (even easier to recycle when I'm done, and I can hope for at least > approximate scaling). I don't think so. Experiment with just a couple of machines with a couple of GbE NICs; I bet that you'll quickly see that relaying packets is right out with TCP/IP, and nobody's even begun working on a user-level driver that would do routing. -- g From rgb@phy.duke.edu Thu, 21 Jan 1999 14:00:43 -0500 Date: Thu, 21 Jan 1999 14:00:43 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: networking data, global /proc space On Thu, 21 Jan 1999, Greg Lindahl wrote: > > > I don't think the switch prices are driven by copper vs. fiber. > > > > Maybe not, but fiber has stayed considerably more expensive than copper > > Um, that's only the cost of the piece that attaches the device to > other devices. When you have a 64-port switch costing $100,000, the > extra you're paying for fiber verses copper is small. The cost is > switching all those gigabits, which is the same for both copper and > fiber. I'll buy that. I understand that you are saying that right now it is an issue of marginal cost, and although fiber transceivers are more costly the switch is more costly still. In the case of FE, the switching technology has gotten (much) cheaper than the transceivers, so the marginal cost of fiber has remained relatively high. Of course, when we bought our first FE switch a few years ago, the situation was much the same as it is now for GbE, with a relatively small marginal cost differential, but the marginal cost differential grew and grew. > > > But sure, my real hope is that GbE over copper > > historically recapitulates FE over copper and that NIC and copper switch > > prices plunge (regardless of what they are now) as volumes swell and > > comnpetition kicks in and technology marches on. > > Everyone knows prices will plunge; that has little to do with copper > verses fiber. It does in one economic, not technical, respect. Lots of places have Cat5 UTP copper installed to run the local FE. Relatively few have fiber installed is considerable overkill for FE. There has therefore been a rather large marginal cost barrier to using the fiber-only GbE transceivers that have mostly been available everywhere but a relatively few sites (first one has to install all that fiber!). Fiber equals small market, mostly resource rich and able to pay high marginal costs. Copper equals large, commodity market, resource constrained (which is why they have copper in the first place) and averse to high margin sales. A lot of folks have simply been waiting for GbE over copper to "arrive", and the manufacturers know that. It is hardly a coincidence that the price points of the two copper NICs I've heard of in this thread define the range $300-500 while the fiber NICs I've found on the web tended more to the $900-1200 range. The pricing reflects an expectation of a much larger market almost immediately. > > I know that you like > > Myrinet and for good reason (it is indeed technically excellent, from > > what I have read), but I'm holding out for GbE because I see it becoming > > the true COTS gigabit network in a couple of years regardless of > > technical merit and price matters to me. > > My crystal ball doesn't go out a couple of years. I'm building > clusters now. If you want to experiment with GbE, then perhaps you > should work on the driver issues, as they are a killer today. Sigh. I'm writing grant proposals now that will fund what I do in two or three years. So are lots of people. One therefore has to crank up a crystal ball or place a blind-squirrel bet, one way or another. Now, do I estimate $3000 per node (your original estimate for a GbE switched connection) as the cost by the end of this year (when we MIGHT get the first installment of the money)? Given that NICs are clearly available for $300 now, do I estimate $1800/node ($300/NIC + your estimate of $1500 for a switched port)? Do I assume that I'll use Myrinet, and estimate what was it, $1700/node or thereabouts? Do I assume that GbE prices will come down to $700-1000/node almost immediately (much as fast ethernet prices did) as soon as it comes out on copper because of the much larger volume and the consequent competition between NIC and switch makers? My inclination is to assume the latter -- as I really believe that this is what will happen. > I don't think so. Experiment with just a couple of machines with a > couple of GbE NICs; I bet that you'll quickly see that relaying > packets is right out with TCP/IP, and nobody's even begun working on a > user-level driver that would do routing. And then there are, as you note, the driver issues. Do I assume that they'll be "fixed" for GbE by the time I get the money? Do I get a couple of GbE NICs as you suggest (reasonable to do at $300/each), crosslink them, and experience firsthand the driver issues and see if they look manageable? To I throw up my hands in disgust and go with Myrinet and find myself totally out of it by year three when every desktop in the department has nice, efficient, cheap, switched GbE connections and I'm still forced to maintain a Myrinet cluster. This is all really interesting information and discussion in and of itself. And of course, I hope to do just what you suggest although I probably would have gotten and may still get more than just a pair of GbE NICs -- I'd need at least four NICs and three systems to be able to test relaying. Still, I cannot possibly do this before the grant proposal deadline. What I would hope for on this list would be that somebody has already done this and can post the results. Barring that, at least a discussion like this one exposes the key issues to keep in mind: drivers and latency, routing, switch and NIC cost, and Walter's remarks on the flexibility of Myrinet for really large collections of nodes. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From lindahl@cs.virginia.edu Thu, 21 Jan 1999 14:21:26 -0500 Date: Thu, 21 Jan 1999 14:21:26 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: networking data, global /proc space > > Everyone knows prices will plunge; that has little to do with copper > > verses fiber. > > It does in one economic, not technical, respect. Lots of places have > Cat5 UTP copper installed to run the local FE. Few people build big clusters on existing wiring. If you want to prognosticate about the overall market, you can read about it in your favorite PC magazine... > Sigh. I'm writing grant proposals now that will fund what I do in two > or three years. So am I. Myricom tells me they plan on hitting $800/node by the end of this year. I think it's a fairly safe bet that in 2-3 years I will be able to get a usable gigabit with a big switch for <$500/node. I don't know if that means I'll be buying Myrinet or GbE in 2-3 years. The kinds of grant proposals I work with don't require that I actually buy what I predicted; I get to buy what's right at the moment I buy. > To I throw up my hands in disgust and go with > Myrinet and find myself totally out of it by year three when every > desktop in the department has nice, efficient, cheap, switched GbE > connections and I'm still forced to maintain a Myrinet cluster. Huh? Where'd you make that up from? In that case, you could discard your old stuff and buy new. That's what we'll all do with Fast Ethernet. That's what I do with old CPUs and machines, too. Will Alpha die in 2-3 years? I don't care; I'll be throwing them away in 2-3 years anyway. I don't know why you care about desktop bandwidth, anyway. At my site, desktops run Windoze and MS Office, and all real work is done on the cluster. I don't need gigabit bandwidth between the desktop and the cluster. -- g From rgb@phy.duke.edu Thu, 21 Jan 1999 14:50:01 -0500 Date: Thu, 21 Jan 1999 14:50:01 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: Switch Boxes On Thu, 21 Jan 1999, Sean McPherson wrote: > Wow. > > Every time I think I have a list of all the parts I'm gonna need for a > beowulf, I find something else. Specs never stay the same, do they? :-) > > I now have to expect to connect the beowulf cluster into a switching box > to let me keep from having keyboards, mice, and monitor cables strung all > over the place. I know, just ignore them. That's not an option for me for > reasons far too lengthy to explain here. This switch box will need to > emulate a keyboard and mouse attached all the time, so I can't use a $10 > keyboard/switch box from CompUSA, much as I'd like to. If I also have to > plug an SGI and a NT box into this switcher as frontend machines (don't > ask, it ain't pretty) I can't have them locking up when the keyboard and > mouse goes away (as NT is apt to do), or just not seeing them when I > switch back. > > So, can anyone point me at what they've used? Maybe not in your beowulf, > but in your data centers, etc? I've seen KVM, but I know they cost a > bundle, and do silly things I don't need like adding sound support :-) I > may go with a KVM switch if I have to, but the big thing I need is to be > able to plug into PC's running Linux and NT (and SGI would be nice but not > necessarily required), and to emulate the kb/mouse all the time. > > Thanks in advance, (as I pray one day I get to actually put this sucker > together!), > > Sean McPherson > sean@ntr.net > Systems Engineer > Information Systems We have a very nice one called a Raritan Master Console. I think I linked their website to our brahma page or my linux page: http://www.phy.duke.edu/brahma http://www.phy.duke.edu/~rgb/linux.html They aren't cheap, but they are much cheaper than popping a M,K&M on every box and they are quite convenient. I believe the Master Console stacks and supports up to 64 nodes. All you need is a dirt cheap S3 card in each host and there you go, ready to run anything from a boring VGA console to a real live X display on any host you choose, keyboard selectable. I think that there are several alternatives now available, by the way. Check out Data Warehouse (or was it their Network Warehouse subsidiary) -- that is where I first found the Raritan but I think that I saw more there last catalog I browsed. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb@phy.duke.edu Thu, 21 Jan 1999 16:01:48 -0500 Date: Thu, 21 Jan 1999 16:01:48 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: networking data, global /proc space On Thu, 21 Jan 1999, Greg Lindahl wrote: > So am I. Myricom tells me they plan on hitting $800/node by the end of > this year. I think it's a fairly safe bet that in 2-3 years I will be > able to get a usable gigabit with a big switch for <$500/node. I don't > know if that means I'll be buying Myrinet or GbE in 2-3 years. The > kinds of grant proposals I work with don't require that I actually buy > what I predicted; I get to buy what's right at the moment I buy. Sure, this is the usual way it works, after all. The problem is in finding a "reasonable" price range prediction that will cover your needs at the time. If Myricom is planning on hitting $800/node, it is also a safe bet that GbE will do at least as well, just to compete, and this is about what I expected for gigabit networks come fall anyway. It is very useful information either way. > > To I throw up my hands in disgust and go with > > Myrinet and find myself totally out of it by year three when every > > desktop in the department has nice, efficient, cheap, switched GbE > > connections and I'm still forced to maintain a Myrinet cluster. > > Huh? Where'd you make that up from? In that case, you could discard > your old stuff and buy new. That's what we'll all do with Fast > Ethernet. That's what I do with old CPUs and machines, too. Will Alpha > die in 2-3 years? I don't care; I'll be throwing them away in 2-3 > years anyway. > > I don't know why you care about desktop bandwidth, anyway. At my site, > desktops run Windoze and MS Office, and all real work is done on the > cluster. I don't need gigabit bandwidth between the desktop and the > cluster. We have organizations of clearly different scale and structure and (probably) goals is all. To start with, nearly all our desktops run linux and are >>part<< of our cluster (places where people run the coarser grained parallel stuff, but part of the cluster nonetheless). We don't do Windows...;-) Our setup has no real boundaries between our "beowulf" and our desktops because we are not huge and have to be very, very efficient. We maintain very close to 100% duty cycle (all CPUs always working on a calculation) on both our dedicated beowulf core and our extended desktop cluster, with some calculations going across the boundaries and others staying home, which is possible because all the systems are on the same FE switch. This is a really nice and flexible and cost/benefit superefficient schema -- I can freely move systems out of the 'wulf core and onto the desktop and replace them with something newer/better/faster (or not) without losing the older CPUs or reconfiguring or reengineering anything. I literally just slap a monitor on the system and put it on a desk and turn it on. Same mounts, same local installation, same switch. If a visitor comes in, we can provide a system on short notice without really losing anything, provided only that we can find a suitable break point in ongoing calculations in which to turn the system off and move it. We are looking for money to scale up the "dedicated" wulfish section considerably over the next few years (we are rather CPU starved and likely to get more so given our various research interests and projects), but we have infrastructure limitations to work that motivate a reasonable effort to preserve this desireable organization-symmetric cluster schema. Appropriately cooled and powered space in the physics building, for example, is a relatively valuable commodity -- popping 64 or more nodes into a single room would totally overstress our infrastructure and major changes in infrastructure are likely to cost way more than we can hope to get budgeted. By "distributing" our operation throughout the building on existing wiring and chillers I can take a lot of this pressure off at zero marginal cost (put 16 nodes in the original computer room, for example, and spread the rest around the building a few to a room) -- if I can run the fast network on our existing wire (we have a minimum of 3 pairs per room as we were lavish when installing it a few years ago). GbE is "supposed" to run 100 meters on Cat5 UTP. It remains to be seen (by me in person) if it will, but if it does it would give it a strong edge here over Myrinet, which has a much more restrictive radius and would necessitate a localized and asymmetric installation. Even with all this motivating GbE on copper (if possible) as an infrastructure- and rollover-replacement-preserving schema, though, I am very seriously considering doing a Myrinet at least for year one (when I can still get all the new systems into our existing server room and hence physically contiguous, if I move out all of our older systems to desktops). Perhaps "throw my hands up in disgust" is excessive, but you can see that I'd "rather" do GbE on copper if at all possible for what I think are sound reasons. Either way, if Myrinet or GbE are either one down to $800/node by next winter, it would be within my tentative budget and I'd probably go for it (GbE by preference IF the drivers work acceptably by then) and if neither of them get down that far I'd probably really act disgusted and just do FE, possibly channel bonded for another six months as a throwaway network (at $30/NIC on existing switch ports) and save my networking budget to try again later. Before I decide (next fall, IF we get the grant), I very much want to see GbE in action (or get reliable reports on this list) BECAUSE I'll believe that it works the way I want/expect it to when I can see it do so. I'm willing to accept that Myrinet works pretty well by now from your and many other people's reports, but GbE is still terra incognita for many of us and has been too expensive for me to "just get a little one" to play with -- until now, maybe. Hopefully this explains why I was concerned that I'd have to "throw away Myrinet" or "maintain it after committing to GbE" as I said before -- if I do put in a really different core network, I'll break the existing (and, in my opinion, desirable) flow from the core group of headless systems to headed systems (still in the core group) to true desktop systems not in the core group -- systems will pretty much have to be hardware and software reconfigured to move off the myrinet. If nothing else, this is more work and work that will need to be done fairly often, if my past experience is any measure. I realize that my cluster design isn't the same as yours (and we've already decided that at most the inner core of it can even be called a "beowulf" per se) but I'd argue that it is hard to beat 100% CPU utilization for months at a time (while covering nearly every desktop in the department with a damn nice system) in terms of cost efficiency, and with limited resources it is this, not network speed or beowulfic purity, that I have worked very hard to optimize so far. We certainly are getting a LOT of work done with all this, and hope to get even more work done as we scale up quietly and within our existing infrastructure base. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From jamesw@giga-net.com Thu, 21 Jan 1999 16:16:10 -0500 Date: Thu, 21 Jan 1999 16:16:10 -0500 From: James Wellnitz jamesw@giga-net.com Subject: Switch Boxes Here we use kb/mouse/monitor switches from a company called Cybex. I'm told by the IS guy in the next cube that they cost $187 for a 4 port switch. We use them on NT boxes (we have 7 or so of these switches right now) and they work fine. You can even cascade them (I haven't tried it though) and switch machines from the keyboard. The model I'm holding seems to be #10040. Regards, Jamie Wellnitz From llonergan@hpti.com Fri, 22 Jan 1999 08:54:51 -0500 Date: Fri, 22 Jan 1999 08:54:51 -0500 From: Luke Lonergan llonergan@hpti.com Subject: Switch Boxes I'm using the Belkin switches and they seem to work very well. They emulate a keyboard signal at all times so that the machines can boot OK and they daisy chain. You can switch from machine to machine from the keyboard. They have 2, 4 and 6 port models. I bought the 4 port model for about $180 each. Remember that for every daisy chain instance, subtract one available port - e.g. to connect 16 machines with 4 port switches, you need 5 of them. Luke From alain.coetmeur@icdc.caissedesdepots.fr Fri, 22 Jan 1999 13:41:31 -0500 Date: Fri, 22 Jan 1999 13:41:31 -0500 From: Coetmeur, Alain alain.coetmeur@icdc.caissedesdepots.fr Subject: networking data, global /proc space -- Alain Coetmeur, Informatique-CDC DTA mailto:alain.coetmeur@icdc.CaisseDesDepots.fr > > I notice that nearly all machines are switched > fast ethernet. The very high end machines are Myrinet. I've looked > at the webpages of all the ones I;ve found but not seen much > comparison between the two and at what point you would want > to change over. Presumably you buy what you can afford. Is any > comparison available? > > I notice few people are using Gbit ethernet and no-one is using SCI > - any reason for this? strange. moreover SCI is an IEEE standard. [SCI] The IEEE 1596 Scalable Coherent Interface Local Area MultiProcessor Users, Developers, and Manufacturers Association. http://www.scizzl.com/ I've looked at the price and they very similar, and Dolphinics (the maker) is quite active, especialy in VIA, with totalview debugger, and even in partnership with NUMA cluster makers. I though that SCI and myrinet where used as much... I've also found a very high performance network standard that is designed to be quite cheap for building // machines. [IEEE1355][HSLink] IEEE Std 1355-1995 Standard for Heterogeneous InterConnect http://grouper.ieee.org/groups/1355/index.html http://www.1355-association.org/ in fact I've read that it find it's success in building high performance ATM/LAN/WAN commuters/routers. the company at www.parsytec.de sells clusters based on this SAN network, as much as SCI or FE. they say it is very efficient, bandwidth, latency and pricing... does anybody have experience in this ? > Is there any point in having a global /proc space in (say) a Linux > beowulf. This would allow local calls to find which > machines/disks/network > cards were busy. I assume this would be easier to use than say SMTP. a beowulf maker company I've meet told me that such extension of linux where very very inefficient. moreover for message passing this is not useful, and for shared memory programming this is not efficient. since SCI (maybe myrinet also) have a memory coherency protocol you can build NUMA machine. but IMHO you should program so you avoid memory migration, thus in a kind of distributed memory model... and in this case Message passing is simpler to understand and debug, yet more bulky. another point for me, maybe I'm not the only one, is that our problem use quite no bandwidth, and fast network is useless. some say that todays new problems (risk pricing, climate, crash test, sensibility analysis, which are often montecarlo) are embarassingly parallel, and that the goal is not to make programs run faster, but to extend the size of the problem. so, if like me, the price of a SCI/Myrinet is the price of a biprocessor PC, my choice is easy. in fact, in which real condition, do you need very high bandwidth ? does anybody have example where communication overide computation on a large computation (hours, days). From mcc@ncc.up.pt Fri, 22 Jan 1999 15:27:04 -0500 Date: Fri, 22 Jan 1999 15:27:04 -0500 From: Manuel Eduardo Correia mcc@ncc.up.pt Subject: Apache Web server Cluster.. Hi, I am looking for a way to configure the apache webserver to serve as a frontend to a series of http servers that run on a Beowulf Linux Machine. I want not only to be able to do load sharing (basically I want the front http server to be able to load balance http requests in the Cluster) but also provide redundancy through replicated resources in the cluster servers. Does anyone have any good idea of how this can be done with apache and Linux ? If someone already did it, could he be please share his wisdom with the list ? Thanks a lot, ------------------------------------------------------------------------ Manuel Eduardo C. D. Correia ------------------------------------------------------------------------ LIACC, Rua do Campo Alegre, 823, 4150 Porto, PORTUGAL Tel: (351-2) 607 8830, Ext: 114, Fax: (351-2) 600 3654, Internet: mcc@ncc.up.pt ------------------------------------------------------------------------ From lindahl@cs.virginia.edu Fri, 22 Jan 1999 15:37:05 -0500 Date: Fri, 22 Jan 1999 15:37:05 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: networking data, global /proc space > strange. moreover SCI is an IEEE standard. Being a standard doesn't necessarily mean cheap, big switches can be built! > another point for me, maybe I'm not the only one, is that > our problem use quite no bandwidth, and fast network > is useless. Then you are right to not buy one. But for people running spectral CFD problems, a fast network is necessary. You don't think that 100% of the people buying big parallem machines like the T3E are idiots, do you? -- g From dhart@indiana.edu Sat, 23 Jan 1999 10:19:03 -0500 Date: Sat, 23 Jan 1999 10:19:03 -0500 From: Dave Hart dhart@indiana.edu Subject: MPICH stalls At 10:34 AM 1/10/1999 -0500, Douglas Eadline wrote: > >I can run the the nas parallel benchmarks using LAM all day without >a single problem. However, If I recompile using MPICH, I find that >the performance is less (as expected) and every once and a while >the program just stalls - it sometimes finishes, but often it just dies. Which compilers do people like? I'm particularly interested in comparisons of GNU, NAG, and Portland Group products, but if there's a good and cheap alternative, that would be extra interesting. -- David Hart http://php.indiana.edu/~dhart Research Computing Support 812-855-2632 University Information Technology Services Indiana University From deadline@plogic.com Sat, 23 Jan 1999 12:28:10 -0500 Date: Sat, 23 Jan 1999 12:28:10 -0500 From: Douglas Eadline deadline@plogic.com Subject: MPICH stalls On Sat, 23 Jan 1999, Dave Hart wrote: > At 10:34 AM 1/10/1999 -0500, Douglas Eadline wrote: > > > >I can run the the nas parallel benchmarks using LAM all day without > >a single problem. However, If I recompile using MPICH, I find that > >the performance is less (as expected) and every once and a while > >the program just stalls - it sometimes finishes, but often it just dies. > > Which compilers do people like? I'm particularly interested in comparisons > of GNU, NAG, and Portland Group products, but if there's a good and cheap > alternative, that would be extra interesting. We use g77 (egcs) and Pgroup. We will bepostin some benchmarks soon. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From doug@Eng.Auburn.EDU Sat, 23 Jan 1999 15:00:12 -0500 Date: Sat, 23 Jan 1999 15:00:12 -0500 From: Doug Hughes doug@Eng.Auburn.EDU Subject: MPICH stalls On Sat, 23 Jan 1999, Douglas Eadline wrote: > On Sat, 23 Jan 1999, Dave Hart wrote: > > > At 10:34 AM 1/10/1999 -0500, Douglas Eadline wrote: > > > > > >I can run the the nas parallel benchmarks using LAM all day without > > >a single problem. However, If I recompile using MPICH, I find that > > >the performance is less (as expected) and every once and a while > > >the program just stalls - it sometimes finishes, but often it just dies. > > > > Which compilers do people like? I'm particularly interested in comparisons > > of GNU, NAG, and Portland Group products, but if there's a good and cheap > > alternative, that would be extra interesting. > > > We use g77 (egcs) and Pgroup. We will bepostin some benchmarks soon. > FYI, g77 generates mediocre code for Alpha Linux. Dec's fortran compiler generates much better code that will run under Alpha/Redhat with coff kernel support. NDS is somewhere in between (from limited testing) ____________________________________________________________________________ Doug Hughes Engineering Network Services System/Net Admin Auburn University doug@eng.auburn.edu From rgb@phy.duke.edu Sat, 23 Jan 1999 15:40:34 -0500 Date: Sat, 23 Jan 1999 15:40:34 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: MPICH stalls On Sat, 23 Jan 1999, Dave Hart wrote: > At 10:34 AM 1/10/1999 -0500, Douglas Eadline wrote: > > > >I can run the the nas parallel benchmarks using LAM all day without > >a single problem. However, If I recompile using MPICH, I find that > >the performance is less (as expected) and every once and a while > >the program just stalls - it sometimes finishes, but often it just dies. > > Which compilers do people like? I'm particularly interested in comparisons > of GNU, NAG, and Portland Group products, but if there's a good and cheap > alternative, that would be extra interesting. We have Gnu (obviously) and Portland here -- Gnu's fortran is a bit broken and Portland's appears to work "fine". gcc works nearly perfectly. I say nearly because I've very, very rarely had some very strange error messages from a compile that caused it to bomb for no obvious/apparent reason, but I've always managed to twiddle things a bit and get it to work after all. We just received a beowulf site license for the Absoft compiler and will be giving it a pretty serious test run very shortly. I'll post back whatever we find on comparable code (probably pvm and/or mpi stuff). When I looked into fortran compilers some time ago, Absoft's was very well thought of. One of the compilers at that time, by the way (NAG? Can't remember for sure) had very strange library licensing requirements that made BINARIES compiled on one system unusable on any other, which made it impossible to buy just one license for a front node on a beowulf. Perhaps they've come to their senses by now. Perhaps not. Something to worry about when you price compare. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From rgb@phy.duke.edu Sun, 24 Jan 1999 00:31:17 -0500 Date: Sun, 24 Jan 1999 00:31:17 -0500 From: Robert G. Brown rgb@phy.duke.edu Subject: Question Dear EL enthusiasts: A question. Would all EL OR beowulf persons who receive this note who are Intel Grant recipients be so good as to respond to me personally? I'm trying to make a head count prior to seeing if Intel might be willing to part with some (fairly nominal) green to help support the EL meeting which looks like it will indeed be run in parallel with this year's linux Expo in Raleigh, NC this May. It would be good if we all participated in and added weight to this request. rgb Robert G. Brown http://www.phy.duke.edu/~rgb/ Duke University Dept. of Physics, Box 90305 Durham, N.C. 27708-0305 Phone: 1-919-660-2567 Fax: 919-660-2525 email:rgb@phy.duke.edu From deadline@plogic.com Sun, 24 Jan 1999 06:56:07 -0500 Date: Sun, 24 Jan 1999 06:56:07 -0500 From: Douglas Eadline deadline@plogic.com Subject: MPICH stalls On Sat, 23 Jan 1999, Doug Hughes wrote: --snip-- > > > > We use g77 (egcs) and Pgroup. We will bepostin some benchmarks soon. > > > FYI, g77 generates mediocre code for Alpha Linux. Dec's fortran > compiler generates much better code that will run under Alpha/Redhat > with coff kernel support. > NDS is somewhere in between (from limited testing) Correct me if I am wrong, but I thought that DEC/FORTRAN binaries running under Linux was a violation of the DEC license (you could only run code under DEC UNIX or something like that) Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From lindahl@cs.virginia.edu Sun, 24 Jan 1999 17:06:01 -0500 Date: Sun, 24 Jan 1999 17:06:01 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: MPICH stalls > > FYI, g77 generates mediocre code for Alpha Linux. Dec's fortran > > compiler generates much better code that will run under Alpha/Redhat > > with coff kernel support. > > NDS is somewhere in between (from limited testing) > Correct me if I am wrong, but I thought that DEC/FORTRAN binaries > running under Linux was a violation of the DEC license (you could only run > code under DEC UNIX or something like that) You are wrong. You can definitely run binaries under Linux as long as you happen to own a Digital Unix license for that cpu. All that we need to do to _legally_ use Digital Fortran as a cross-compiler is for someone to port glibc2 to Digital Unix. There was a glibc1 port, so this isn't that hard. -- g From paul.oertel@citicorp.com Sun, 24 Jan 1999 20:08:26 -0500 Date: Sun, 24 Jan 1999 20:08:26 -0500 From: Paul Oertel paul.oertel@citicorp.com Subject: Beoboard idea I don't see why it would work but then I'm not too strung in the electric wiring department. I've read that you need at least 120 watts (have I got my units right?) for the motherboard and an average 12 watts per each additional device (graphics card, hard drive etc.) For me space is also a problem. I have been thinking if there is some way to combine more than one motherboard to per box and share power supply there would be a significant space savings. Maybe other componenents (mice, keyboard) could be shared as well. So long as the box was properly grounded, I think the box needn't be made of metal to function properly. R. Paul McCarty wrote: > > I've speant weeks looking for info on cases, power supplies, etc. that could > power multiple boards in a single box with this sort of final result in mind. I > don't seen any reason why it wont work (just splice off the main ATX or AT wire > to multiple plugs to each board) as long as the power doesn't exceed the power > supplies capacity. Do you have any idea why this wouldn't work? > > You didn't actually build one did you? ;-) No I didn't. > Thanks for any info. > -Paul -- Paul Oertel ComiConnection: The ultimate guide to DC Comics. http://www.shirokuma.com/comiconnection From hozer@drgw.net Sun, 24 Jan 1999 20:32:08 -0500 Date: Sun, 24 Jan 1999 20:32:08 -0500 From: Troy Benjegerdes hozer@drgw.net Subject: Apache Web server Cluster.. On Fri, 22 Jan 1999, Manuel Eduardo Correia wrote: > > Hi, > > I am looking for a way to configure the apache webserver to serve > as a frontend to a series of http servers that run on a Beowulf Linux > Machine. I want not only to be able to do load sharing (basically I > want the front http server to be able to load balance http requests in > the Cluster) but also provide redundancy through replicated resources > in the cluster servers. Does anyone have any good idea of how this > can be done with apache and Linux ? If someone already did it, could > he be please share his wisdom with the list ? > > Thanks a lot, This might be possible using the CODA filesystem as a backend filesystem providing replicated resources, and having the straight apache daemon running on a bunch of machines with CODA clients. You could have 2 coda servers, with around 4 GB of replicated data on them, and each http server machine can run a coda client with a practical maximum of 300 MB of cache. Load balancing could be done via network address translation of some sort. If you are serious about this, expect to have problems with CODA and submit fixes, but if you realize it's limits, I think this could work very well. see www.coda.cs.cmu.edu for coda src and binaries. -------------------------------------------------------------------------- | Troy Benjegerdes | troy@microux.com | hozer@drgw.net | | Unix is user friendly... You just have to be friendly to it first. | | This message composed with 100% free software. http://www.gnu.org | -------------------------------------------------------------------------- From langzhi@fizik.upm.edu.my Mon, 25 Jan 1999 02:51:52 -0500 Date: Mon, 25 Jan 1999 02:51:52 -0500 From: Eng ler ...(admin) langzhi@fizik.upm.edu.my Subject: Scalapack - HELP !!! I downloaded and installed scalapack from netlib. I saw lots of binaries in TESTING dir, but i don't know how to use them. I want to use scalapack to benchmark my beowulf clusters. Can anyone tell me how to do the benchmark ?? Please ... thanks From rolf@suse.de Mon, 25 Jan 1999 03:02:23 -0500 Date: Mon, 25 Jan 1999 03:02:23 -0500 From: Rolf Haberrecker rolf@suse.de Subject: Apache Web server Cluster.. --0OAP2g/MAC+5xKAE Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable On Fri, Jan 22, 1999 at 08:26:55PM +0000, Manuel Eduardo Correia wrote: >=20 > Hi,=20 >=20 > I am looking for a way to configure the apache webserver to serve > as a frontend to a series of http servers that run on a Beowulf Linux > Machine. I want not only to be able to do load sharing (basically I > want the front http server to be able to load balance http requests in > the Cluster) but also provide redundancy through replicated resources > in the cluster servers. Does anyone have any good idea of how this > can be done with apache and Linux ? If someone already did it, could > he be please share his wisdom with the list ? I've seen a demo version of Netequator once, which did exactly what you are planning to do: http://www.bscsoft.com/ Be aware that this is not free software in whatever definition. --=20 Mit freundlichen Gruessen, Rolf Haberrecker SuSE GmbH, Tel: +49-911-7405330 Mo/Do 13-18.00 Schanzaeckerstr. 10, Fax: +49-911-3206727 90443 Nuernberg, Email: support@suse.de=20 Germany WWW: http://www.suse.de/Support/sdb=20 pub 1024/8C678B65 1998/12/18 Rolf Haberrecker Key fingerprint =3D A4 E7 67 1F 46 5F 61 80 C8 BC 09 76 74 36 21 38 --0OAP2g/MAC+5xKAE Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: 2.6.3i iQCVAwUBNqwk3h5X/7uMZ4tlAQHPGwP/c3e0vBnx0k0nrx0e6RMG/XiUh/XtsXJX 41YL5URIqs8wr3x1U6HNvD0Fo79viUDLt7bhsheyW660c/4IFxYBq0mQAhLVN4NC vvH05tvlwOtf+ox3Qoo1ufwS/gLR4rZI7ap0FR+MpOThePFf/0NHIa71bRUa76Qz lOFvz6unXeA= =nPnR -----END PGP SIGNATURE----- --0OAP2g/MAC+5xKAE-- From mack@nesc.epa.gov Mon, 25 Jan 1999 08:24:30 -0500 Date: Mon, 25 Jan 1999 08:24:30 -0500 From: Joseph Mack mack@nesc.epa.gov Subject: Apache Web server Cluster.. Manuel Eduardo Correia wrote: > > Hi, > > I am looking for a way to configure the apache webserver to serve > as a frontend to a series of http servers that run on a Beowulf Linux > Machine. I am writing a proposal to do this myself. I don't have any solutions to the problems you brought up, but presumably will need to have some good ideas before I start any coding. I won't be starting for at least 2 months (may be longer depending what they want me to do first) and I have to get the OK to do this GPL. I'd be happy to talk to you further about this when I get started. Joe -- Joseph Mack, Senior Systems Engineer, Lockheed Martin National Environmental Supercomputer Center (NESC), mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA From anirudh@bart.ihpca.psu.edu Mon, 25 Jan 1999 09:20:53 -0500 Date: Mon, 25 Jan 1999 09:20:53 -0500 From: Anirudh Modi anirudh@bart.ihpca.psu.edu Subject: Beoboard idea Hi, I thought this might be somewhat relevant to this discussion just in case you guys already did not know: http://www.altatech.com/systems.html http://www.altatech.com/boards.html They make compact motherboards for P-II/Alpha clusters. On Mon, 25 Jan 1999, Paul Oertel wrote: | I don't see why it would work but then I'm not too strung in the | electric wiring department. I've read that you need at least 120 watts | (have I got my units right?) for the motherboard and an average 12 watts | per each additional device (graphics card, hard drive etc.) For me | space is also a problem. I have been thinking if there is some way to | combine more than one motherboard to per box and share power supply | there would be a significant space savings. Maybe other componenents | (mice, keyboard) could be shared as well. So long as the box was | properly grounded, I think the box needn't be made of metal to function | properly. | | R. Paul McCarty wrote: | > | > I've speant weeks looking for info on cases, power supplies, etc. that could | > power multiple boards in a single box with this sort of final result in mind. I | > don't seen any reason why it wont work (just splice off the main ATX or AT wire | > to multiple plugs to each board) as long as the power doesn't exceed the power | > supplies capacity. Do you have any idea why this wouldn't work? | > | > You didn't actually build one did you? ;-) | | No I didn't. | | > Thanks for any info. | > -Paul | | -- | Paul Oertel | ComiConnection: The ultimate guide to DC Comics. | http://www.shirokuma.com/comiconnection | -anirudh [http://bart.aero.psu.edu] --- From alvin@iplink.net Mon, 25 Jan 1999 09:30:44 -0500 Date: Mon, 25 Jan 1999 09:30:44 -0500 From: Alvin Starr alvin@iplink.net Subject: Apache Web server Cluster.. On Sun, 24 Jan 1999, Troy Benjegerdes wrote: > On Fri, 22 Jan 1999, Manuel Eduardo Correia wrote: > > > > > Hi, > > > > I am looking for a way to configure the apache webserver to serve > > as a frontend to a series of http servers that run on a Beowulf Linux > > Machine. I want not only to be able to do load sharing (basically I > > want the front http server to be able to load balance http requests in > > the Cluster) but also provide redundancy through replicated resources > > in the cluster servers. Does anyone have any good idea of how this > > can be done with apache and Linux ? If someone already did it, could > > he be please share his wisdom with the list ? you may also want to take a look at eddie. It is a web cluster managment took kit. Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin@iplink.net || From josip@icase.edu Mon, 25 Jan 1999 09:44:41 -0500 Date: Mon, 25 Jan 1999 09:44:41 -0500 From: Josip Loncaric josip@icase.edu Subject: Scalapack - HELP !!! "Eng ler ...(admin)" wrote: > > I downloaded and installed scalapack from netlib. I saw lots of binaries > in TESTING dir, but i don't know how to use them. I want to use scalapack > to benchmark my beowulf clusters. Can anyone tell me how to do the > benchmark ?? > Please ... > > thanks See the "Installation Guide for ScaLAPACK" http://www.netlib.org/scalapack/scalapack_install.ps and read sections 3.7-3.10 and chapter 4. If performance measurement is your objective, you may have to study these sections carefully and customize the tests. To run the tests, you typically use the mpirun command, e.g. mpirun -np 5 xdlu and then examine the output for anything strange. By the way, ScaLAPACK needs BLACS which is a separate package. I assume that you already have working BLACS. Josip -- Dr. Josip Loncaric, Senior Staff Scientist mailto:josip@icase.edu ICASE, Mail Stop 403 http://www.icase.edu/~josip/ NASA Langley Research Center mailto:j.loncaric@larc.nasa.gov Hampton, VA 23681-2199, USA Tel. +1 757 864-2192 Fax +1 757 864-6134 From alain.coetmeur@icdc.caissedesdepots.fr Mon, 25 Jan 1999 10:21:31 -0500 Date: Mon, 25 Jan 1999 10:21:31 -0500 From: Coetmeur, Alain alain.coetmeur@icdc.caissedesdepots.fr Subject: networking data, global /proc space -- Alain Coetmeur, Informatique-CDC DTA mailto:alain.coetmeur@icdc.CaisseDesDepots.fr > > strange. moreover SCI is an IEEE standard. > Being a standard doesn't necessarily mean cheap, big switches can be > built! you mean that SCI have yet a problem with the cost of switches and that myrinet is more affordable. does anybody know why? > > > another point for me, maybe I'm not the only one, is that > > our problem use quite no bandwidth, and fast network > > is useless. > Then you are right to not buy one. But for people running spectral CFD > problems, a fast network is necessary. You don't think that 100% of > the people buying big parallem machines like the T3E are idiots, do > you? this was a little provocation. If there is only one idiot in a room, I know he is standing in my shoes. I think that spectral CFD is of the family of the large vector problems. in this case communication to computation ratio is quite high. I know users in the scientific domain that critic what they call a hype around scalar MPP and clusters... they say nothing is better that a bunch of good vector processor, with shared memory or at least a very fast interconnexion network. their problem are large linear systems to solve. on the opposite other users, using existing modeling kernels, run thousands of simulation and are fond of their network of workstation busy at night... one question about the success of cluster versus tightly connected parallel machine, is the evolution of the problems solved. some say that phenomenologicals models (tweaked/tuned by experiments) need only a few gigaflops, and will soon run without parallelism. anyway the progress will be in launching many of these simple problems for scenario analysis. in fact this looks like our largest HPC problem. simple computation, but in many scenarios. that is good for clusters future. anyway new problems are comming, either "full physics" (asci spirit) or prophecy (one event, no retry possible : climate, warming, ozone,) that need some teraflops... you nannot tweak a model to look like the experiments, but you have to compute from sound theory. if those problems are solved through highly parallel low communication method (monte carlo, highly local equations) then cluster may take the head. otherwise if the problems get bigger and bigger, but still with a vector style, needing much communication, then tightly connected parallel vector machine (like Fujitsu VPP or vector SMP like C90 or SX) will take the lead. machine like the tera, that in fact behave like a vector SMP efficient on irregular structures, may certainly be the next leaders. there will surely be many types of machine, but the question is if the vast majority of HPC will be done on scalar loose clusters, or on vector/MTA tightly connected supercomputers. I've the unfounded ( tell me i'm wrong) intuition that tightly connected scalar machine like T3x are bad compromise in the near future. From peter@ica3.uni-stuttgart.de Mon, 25 Jan 1999 11:44:58 -0500 Date: Mon, 25 Jan 1999 11:44:58 -0500 From: Peter Bastian peter@ica3.uni-stuttgart.de Subject: What about debugging ? Hi there, I looked through the mailing list archives (5/98) and never found anything about debugging parallel codes. Is there anything like a parallel debugger available freely for use on a Linux cluster? Any pointers would be greatly appreciated! Thanks in advance. --Peter Bastian ICA3, University Stuttgart peter@ica3.uni-stuttgart.de From JesseP@europe.stortek.com Mon, 25 Jan 1999 11:54:50 -0500 Date: Mon, 25 Jan 1999 11:54:50 -0500 From: Jessen, Per JesseP@europe.stortek.com Subject: Beoboard idea > -----Original Message----- > From: Paul Oertel [mailto:paul.oertel@citicorp.com] > Sent: 25 January 1999 12:06 [snip] > I don't see why it would work but then I'm not too strung in the > electric wiring department. I've read that you need at least 120 watts > (have I got my units right?) for the motherboard and an > average 12 watts > per each additional device (graphics card, hard drive etc.) For me > space is also a problem. I have been thinking if there is some way to > combine more than one motherboard to per box and share power supply > there would be a significant space savings. Maybe other componenents > (mice, keyboard) could be shared as well. So long as the box was > properly grounded, I think the box needn't be made of metal > to function properly. There are two ways of looking at this: optimum space, optimum price. Space: if space is at a premium, yes, looking at supplying several motherboards/PCs from one (large) switching supply is certainly possible. There are companies that build e.g. 600W and 800W switchmode supplies, but that's about the biggest I've seen. And for a reasonably size cluster you're probably looking at e.g. 1200W or more. Depends on your config. And in that size you could be looking at a custom build. Which doesn't exactly do wonders for the price. Price: if you've got space, but not the money, I don't think one supply for an entire cluster is optimal. Even standard 600/800W supplies do not 'scale' in price. 3 standard 250W supplies are always cheaper than one 600W supply. As far as determining the size of your supply, you should be able to find your motherboards consumption in your manual, and the same goes for your harddisks. For other cards, you'll probably have to make some educated guessing. Anyway, I'm looking at doing exactly the same thing, and I have been out looking for large power-supplies - so, if you find something that is economically viable when compared to an array of COTS 250W supplies, let me know. Best regards, Per Jessen Software Support Engineer, StorageTek International Software Support mailto://per_jessen@storagetek.com, http://www.storagetek.com From faceprint@faceprint.com Mon, 25 Jan 1999 12:04:10 -0500 Date: Mon, 25 Jan 1999 12:04:10 -0500 From: Nathan Walp faceprint@faceprint.com Subject: Apache Web server Cluster.. I found this on freshmeat the other day....looks like the answer to your prayers... http://ma.us.mirrors.freshmeat.net/appindex/1998/07/09/899961518.html It uses port-forwarding and ipmasq to automagically forward the requests on the main node to a random "worker" node to serve up the file haven't tried it out yet, but it looks pretty good... hope this helps -Nathan From doering@iti.mu-luebeck.de Mon, 25 Jan 1999 12:13:54 -0500 Date: Mon, 25 Jan 1999 12:13:54 -0500 From: Andreas C. Doering doering@iti.mu-luebeck.de Subject: networking data, global /proc space Hello, we have currently three clusters, see http://www.iti.mu-luebeck.de/cluster One is Myrinet and switched fast ethernet (48 dual-nodes) the second is SCI (Dolphin) and the third is HS-Link (1355) with Parsytex Adapter cards. We have an own network interface with a common API to all four and have several medical applications for them which allows a comparison. However this kind of applications does not pose high demands on the network. Therefore the performance difference is more due to different processor power than implementation of routers or network interface. But since the research in this area is still quite fresh here, more demonstrating results can be expected in the near future. > they say it is very efficient, bandwidth, latency and pricing... > does anybody have experience in this ? > > since SCI (maybe myrinet also) have a memory coherency protocol > you can build NUMA machine. but IMHO you should program so > you avoid memory migration, thus in a kind of distributed memory > model... and in this case Message passing is simpler > to understand and debug, yet more bulky. > As far as I know you can not build a NUMA machine across the PCI bus. Thus we need better interfaces to fast networks. Andreas --------------------------------------------------------------- Andreas Doering Medizinische Universitaet zu Luebeck Institut fuer Technische Informatik Ratzeburger Allee 160 D-23538 Luebeck Germany Tel.: +49 451 500-3741 Fax: +49 451 500-3687 Email: doering@iti.mu-luebeck.de ---------------------------------------------------------------- From kragen@pobox.com Mon, 25 Jan 1999 12:51:05 -0500 Date: Mon, 25 Jan 1999 12:51:05 -0500 From: Kragen Sitaker kragen@pobox.com Subject: Apache Web server Cluster.. On Mon, 25 Jan 1999, Joseph Mack wrote: > Manuel Eduardo Correia wrote: > > I am looking for a way to configure the apache webserver to serve > > as a frontend to a series of http servers that run on a Beowulf Linux > > Machine. > > I am writing a proposal to do this myself. Cisco sells a multiplexer box that uses IP masquerading to allow connections to a single world-visible IP address to go to one of several physical machines, thus distributing web load. I can't remember what it's called. Squid might be able to do what you need; see . FastCGI will let you use a single HTTP server (Apache included), but farm out the actual processing to FastCGI processes, which can theoretically be running on several different machines, and it's free, as in open source, too. See www.fastcgi.com. Seems like I remember some kind of inetd replacement that would do something similar, accepting connections from outside and then making connections to one of a pool of servers, then relaying data between the connections. But I can't find it. The traditional way to handle this, by the way, is with round-robin DNS, where each successive DNS query for the web-server's IP address gets a different answer. The guys at NCSA published a paper on this in 1993 or 1994 -- they had to upgrade www.ncsa.uiuc.edu to six servers to handle the load from widespread Mosaic use. (I think home.netscape.com is at least 60 servers now.) The paper is probably still on the Web somewhere. Yet another way to handle this -- useful when the machines you want to redirect requests to are geographically distributed -- is to reply to HTTP requests to the main server, say www.freshmeat.net, with HTTP redirects to a mirror server, say nc.us.mirrors.freshmeat.net. Subsequent links within the mirror server will not put any further load on the main server. Good luck. -- Kragen Sitaker Computers are the tools of the devil. It is as simple as that. There is no monotheism strong enough that it cannot be shaken by Unix or any Microsoft product. The devil is real. He lives inside C programs. -- philg@mit.edu From lindahl@cs.virginia.edu Mon, 25 Jan 1999 13:20:06 -0500 Date: Mon, 25 Jan 1999 13:20:06 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: networking data, global /proc space > However this kind of applications does not pose high demands on the > network. Therefore the performance difference is more due to > different processor power than implementation of routers or network > interface. > But since the research in this area is still quite fresh here, > more demonstrating results can be expected in the near future. Hopefully you will either be able to do micro benchmarks such as the bisection bandwidth, or you can take well-known codes with high network demands and run them on your cluster. -- g From jstan@trendcmhs.org Mon, 25 Jan 1999 13:54:58 -0500 Date: Mon, 25 Jan 1999 13:54:58 -0500 From: Stanley, Jeremy jstan@trendcmhs.org Subject: Beoboard idea This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------ =_NextPart_000_01BE4869.BD721990 Content-Type: text/plain I've been struggling with the space vs. price issue myself. The other problem I've found is rack mountable cases. 4x-5x the price of a standard ATX minitower, easily. It would almost double the price of each node for me to look at anything other than standard cases on shelves. Has anyone else found 2U or even 4U rack ATX cases for under US$300? Also, when looking at one PS per node, I worry both with the cost of excess power consumption and excess ambient heat (requiring heavier HVAC load to cool the console room, thus adding further to the electrical bill). Using 120W PS units would take care of most of the excess electrical consumption (in addition to being cheaper than 250W models), but only half-cures the heat problem... Suggestions? Similar experiences? TIA. -- Jeremy Stanley Trend CMHS IS Admin/Senior Tech http://www.trendcmhs.org The opinions expressed herein do not necessarily represent those of Trend CMHS or Trend Foundation. > ---------- > From: Jessen, Per[SMTP:JesseP@europe.stortek.com] > Sent: Monday, January 25, 1999 11:55 AM > To: 'paul.oertel@citicorp.com' > Cc: 'beowulf'; 'extremelinux' > Subject: RE: Beoboard idea > > > -----Original Message----- > > From: Paul Oertel [mailto:paul.oertel@citicorp.com] > > Sent: 25 January 1999 12:06 > [snip] > > I don't see why it would work but then I'm not too strung in the > > electric wiring department. I've read that you need at least 120 > watts > > (have I got my units right?) for the motherboard and an > > average 12 watts > > per each additional device (graphics card, hard drive etc.) For me > > space is also a problem. I have been thinking if there is some way > to > > combine more than one motherboard to per box and share power supply > > there would be a significant space savings. Maybe other > componenents > > (mice, keyboard) could be shared as well. So long as the box was > > properly grounded, I think the box needn't be made of metal > > to function properly. > > There are two ways of looking at this: optimum space, optimum price. > > Space: if space is at a premium, yes, looking at supplying several > motherboards/PCs from one (large) switching supply is certainly > possible. There are companies that build e.g. 600W and 800W > switchmode > supplies, but that's about the biggest I've seen. And for a > reasonably > size cluster you're probably looking at e.g. 1200W or more. Depends > on your config. And in that size you could be looking at a custom > build. Which doesn't exactly do wonders for the price. > > Price: if you've got space, but not the money, I don't think one > supply > for an entire cluster is optimal. Even standard 600/800W supplies > do not 'scale' in price. 3 standard 250W supplies are always cheaper > than one 600W supply. > > As far as determining the size of your supply, you should be able to > find your motherboards consumption in your manual, and the same goes > for your harddisks. For other cards, you'll probably have to make > some > educated guessing. > > Anyway, I'm looking at doing exactly the same thing, and I have been > out looking for large power-supplies - so, if you find something > that > is economically viable when compared to an array of COTS 250W > supplies, > let me know. > > > Best regards, > Per Jessen > Software Support Engineer, StorageTek International Software Support > mailto://per_jessen@storagetek.com, http://www.storagetek.com > ------ =_NextPart_000_01BE4869.BD721990 Content-Type: application/ms-tnef Content-Transfer-Encoding: base64 eJ8+IgESAQaQCAAEAAAAAAABAAEAAQeQBgAIAAAA5AQAAAAAAADoAAEIgAcAGAAAAElQTS5NaWNy b3NvZnQgTWFpbC5Ob3RlADEIAQWAAwAOAAAAzwcBABkADQAyADoAAQBqAQEggAMADgAAAM8HAQAZ AA0AMgA6AAEAagEBCYABACEAAAA3RkYzMzA3QUE4QjBEMjExOUMyODAwQTBDOTZFRUIwNAAtBwEE gAEAEgAAAFJFOiBCZW9ib2FyZCBpZGVhAMIFAQ2ABAACAAAAAgACAAEDkAYAMA4AACIAAAADACYA AAAAAAMANgAAAAAAHgBwAAEAAAAOAAAAQmVvYm9hcmQgaWRlYQAAAAIBcQABAAAAGwAAAAG+SIoT +3ow8zawqBHSnCgAoMlu6wQAAkfZZAADAC4AAAAAAB4AQhABAAAARQAAADwwMDM5N0FEREE1OTZE MTExQTFFRDAwODA1RkUyQkIyMTc3OUM2OUBlaHFtc2cwMS5ldXJvcGUuc3RvcnRlay5jb20+AAAA AAIBCRABAAAAWgoAAFYKAABFEwAATFpGdRVYkl4DAAoAcmNwZzEyNf4yAP8CBgKkA+QF6wKDAFAT A1QCAGNoCsBzZXT+MgYABsMCgw5QA9UHEwKD1jMQPxFHNBT5TQuAEnAabwKDNQRGAgBwcnFDEdES djU1IFQEkG27C4AUcn0KgAjPCdk7Gz/9DiA4HFMdcQm0HfIdOgmr+xmBHaQ5DlAeNCDhHTMg4C8C gAqBDnELYG4OEDAzgxTgF5ByemRvYwAA2ioSVSACkSOwbCPlCvsLE7IMAWMU0CBJJ3YIZSBiCeEg c3RykHVnZ2wLgGcgA/BYdGggKAAmwHMKsGPhJsB2cy4gGLAN4CbAIwQBClAgbXkSsGxmdyjwGbAo QW8oMQXAGLBvGQJgZW0mhAIQdW5k6SlhIHIA0GspwCvhAZExK0AgY2ESsCjhIDQweC01eCgjKRRv ZmwgYSchAHBkCxEUMFROWCnAC4An8G93BJAsrCBlLWADEHkqMUkFQHp3CGBsLBAHQARgJzAgPSNg dS0SLjswoBJwIG6fBHErwQXAB4AoIG8gGtDabyyQYQVAAHB5KAAnov8qpCgAA5EvJy1TKpAnEShA 9mwmsC2SSC1gNSICICbAZyoAErArxTJVKpAFwGX7JrADoDQ5oCxjL7I3BDQSAyvxKtFVUyQzMDCq PypAQTjwbzCAdyhAXwOgNLInojUBOKJQBfBwFyrRM8IwgEkxUXJyeb8m0CqhJ9gFoDIBM0J4KKCd BBFwMEJAUQCAdW0FMO5pN2EvQUDWYQbQCJACMBYgKEA1ASgbQHF1aTcFECexQ6F2CJE4IFZB+kM0 oWEsEDSBBaAG8EAVfwCABvAmwANgA3AwgCgAdek4UWRkJ6JmCHA11DSQ/ygyKgAFkCdADeAHQCbQ AxB0bCkqMVUAkCexDiAwflc+IivwJ/AEIDFkAZBr/y0yG0AuwjHjLtFI00D0SRn9QbooC4BHo0Ij NIEm4Cei/xJwMKA+YjYjDjBKwQRiOPDyKTCAYnU90jDgQ5AHQPhmLWMIcAeRKDJDoysFdi5UoCpA UydhB5BCInP9PGFTB3ADEArBQOA+YUNRw0EBPGFUSUEuCo8LkREYAXMxNiZBNCAtti1Xd1hdMCpA KkBKBJD7K1A/YFMvMStAP2BcPVn/DjEZsBtALAFDTUhTX10/WwBZj1jVEuBJBfBB+mQZ8S8GYAMA BbEZwDORn1zvWPMU4DGAJIBuaywQgmgCQHA6Ly93ZZDGLidAXoFjbWgo4AWwfmdjfxLgYD9Y11d1 KlRw/zABQcFWQlNhErBlAVthT2H/I2AzsQVAOLBBAgrAMNFXdf8bQGriQ2IoADHwLrNeaDmy3V5k RiviNQBCMS4LRhTiP1jwZ8glPyZDcXsnkDE4gzxAAgBpLTE0NA6wvwzQdNMLWXRQcZADYHRJMb9Z UXb2Z912BQwwdnZGA2G+Onf+dnYMgltBaxFuMIADPiAEkFtTTVRQOjF7s1BAZQhwajBlLiMnMBrh ZWsuBaBtXbd3n3itYmF0ed96600CINUvYHkwgEoAcHUKwD9gnw4wMIAg0IPgSpAxOhmRzEFNfn94 rVRvgL966z4ncaAxgGZgBJB2oGxA1mNPwQWhcH4iJ4SveK18Q2OGz4fcJuAwQDGAZnwnO4iQQOBl 0QeAJ5F17niKP39/MlBqSTGMb3rrdFJFkkBCjmAG4C9yac0OcGFxdXNtMzZ1dxaSrjIWkFrhdmc+ duRPBRD+ZxoCBdBsclUgd0qYcHmUklCIwSBPiRMgW5bPv2SDdnVkxADAAxAwMDqIv/+JxZv/l9h+ ZphwgHQOMIMHIYPUMjowNnF1W3OdAwBwocg/ACNgbicFQP8SsCbAPPA/YCfwMVU/ISyQ/1JiKDED oCaQK2BsAjAwNJD/JzInsU9hKDGaN0kWJ9FEU/sOcHGhdAeAAjAo8CaTG0DvRZISgAVAOJB1bDFr MTUB/ytALWAFQEqhJ9A1AEtQmjfuKBKAJrE/AGdsEVuRSyTzmPFlID8pNAMoMgRgKrL/lJRCcgOR mjevESxgVSBKkf+uDT5iM3NPlkmRDnBE0CihbigJwFDgNXBjBCBMMWR3MIASgSwQZAUQJrESwGP+ LrCgb8E0Mpo3KHQsMQdA/zywLvFUNj7xrwMm4zViPXP/BpAoIkxRLDE8sDRRrhA/YP8wMJo3fjFD QDixBGBMUTYj/ziisUo0gT5iBuAuEEJyN5D3TEJBVCmQcAtQbOaYcLxk/zFkJuAu8pkAAwB0oC1Q Q3HfKHRskETQIrAtkk2C0MOx/yqkfjFBUDiwxtGuShnwKKD/MIBMAMXglKKwoAWgw3XBQ/8xoUth KgCewFTRNJI9klOE78DCrhCz+X1iclLBCcAr4v8JgD7iu6PLJ6zypbLDsQDA/zPhTIISwEmRwmg0 kEggF3B/QiPMdldmaXYqUkxRTEJ0vzFgvTI3QS7gPUk1YXOSQPtqMEIgbUHwKGQwgNWGKRP90g9T KIKSQLwxuVg1ESkB7StQaUHwMIB5B5AwgD1Jb8IEJ6ISsLMSbNKHsUlzuC9QQztBA2E94yhWEf9V ILCgA+G4ADVzwgQsIiigvwAgC3Fs1ypAQVAEEGkrMf8o8NMYxoIAcAiQU4I1AVJg8wMQQpEuZyjw lqBKwUJy/3RhStDfJARi0ofCA+MhUkT9rGInOFEG4KdkSbFVEyaE/6XxcGAUMCwBNBIvAKwRPLDv GhACYODoAJB6LTEKQCcw+yrRrLEnwXIrEeqSPTrkE99KoUrBNCK+wSjwRG1wXoH/rlYqQDdhrLFB k3Sg5DHpkv+pE9sy63KsssjnPUkvAFNA3X2xbdKH47Mo8Fe2kSgQ3yNgB5ClskDgANB0UsFr0d8x YDuyO0QuN9dPUCki2KP/7DImsa9y1gVSYqgjsRM4sP+C4aV2zcQ4osILKkDp0wOgf0NhREDrmCwx 1YMHQCjwRfc6Ai8n5GEv5PTmhdJ4a9Wz55BJgWUnqQL3xCAmcP8vJ1GDAbdMQgdA0/NQtdKH378H 5GPCBFdm0nhBO0Fxsf84QQ5w6/Ev8iexKDPrci7R//BTwgTaQazBN5DDZzJjvZb//gFycCwQ8FPd CkGrT2EPJP2DImwwgOSyKDNDIPoSBcDP/ZvwU7dCR9BzayjhuGL3xiXdggzTJ0ng7JivAzSB389Q TAG88tKHG1B1SYB2oH0sEGccIOGhIrD4H+mQef+9QT7hp/E9SSNgJ6L2FhJH/zVjEeS66e+YUnE9 RunS3rP1QUQtAbctvOEwgPmUDsT/vPI1ZKxi0ocsMXawN2C+QP9JclLBRNAtAzzz4sPJojSBH/5h cbAsYD9gLtFDT1R7byAFGyzShytAr5G7MGt/M8BlsBm8cXWUYK2ha4BnvxWTcXV8QXulcXXKgGbT sP9MQlTwwiB90QAwxWC+cTBi31vAYqCZsWLQ/NBJ/qD3AP/qgLVlLc5n3Zx/nY9lcD5h9l+SAHvS QH2ymbF99TJ//5fYfBA2zzOPZRo17zb/doWvla91d3K1PY19cXAAQUAAAEAAOQBwD2Olk0i+AQMA 8T8JBAAAHgAxQAEAAAAGAAAASlNUQU4AAAADABpAAAAAAB4AMEABAAAABgAAAEpTVEFOAAAAAwAZ QAAAAAADAP0/5AQAAAIBRwABAAAALwAAAGM9VVM7YT0gO3A9VHJlbmRjbWhzO2w9V09SRi05OTAx MjUxODUwNThaLTIyOTYAAAIB+T8BAAAASAAAAAAAAADcp0DIwEIQGrS5CAArL+GCAQAAAAAAAAAv Tz1UUkVORENNSFMvT1U9TUFJTC9DTj1SRUNJUElFTlRTL0NOPUpTVEFOAB4A+D8BAAAADwAAAEpl cmVteSBTdGFubGV5AAAeADhAAQAAAAYAAABKU1RBTgAAAAIB+z8BAAAASAAAAAAAAADcp0DIwEIQ GrS5CAArL+GCAQAAAAAAAAAvTz1UUkVORENNSFMvT1U9TUFJTC9DTj1SRUNJUElFTlRTL0NOPUpT VEFOAB4A+j8BAAAADwAAAEplcmVteSBTdGFubGV5AAAeADlAAQAAAAYAAABKU1RBTgAAAEAABzBw D2Olk0i+AUAACDAQ55Clk0i+AR4APQABAAAABQAAAFJFOiAAAAAAHgAdDgEAAAAOAAAAQmVvYm9h cmQgaWRlYQAAAB4ANRABAAAAPAAAADxEMTdFNzg4Qjk5OTFEMTExOUJFMjAwQTBDOTZFRUIwNDBD MTAyNUB3b3JmLnRyZW5kY21ocy5vcmc+AAsAKQAAAAAACwAjAAAAAAADAAYQbD538QMABxBaCgAA AwAQEAAAAAADABEQAQAAAB4ACBABAAAAZQAAAElWRUJFRU5TVFJVR0dMSU5HV0lUSFRIRVNQQUNF VlNQUklDRUlTU1VFTVlTRUxGVEhFT1RIRVJQUk9CTEVNSVZFRk9VTkRJU1JBQ0tNT1VOVEFCTEVD QVNFUzRYLTVYVEhFUFIAAAAAAgF/AAEAAAA8AAAAPEQxN0U3ODhCOTk5MUQxMTE5QkUyMDBBMEM5 NkVFQjA0MEMxMDI1QHdvcmYudHJlbmRjbWhzLm9yZz4AGuE= ------ =_NextPart_000_01BE4869.BD721990-- From mack@nesc.epa.gov Mon, 25 Jan 1999 13:56:27 -0500 Date: Mon, 25 Jan 1999 13:56:27 -0500 From: Joseph Mack mack@nesc.epa.gov Subject: Apache Web server Cluster.. Nathan Walp wrote: > > I found this on freshmeat the other day....looks like the answer to your > prayers... > Here's a discussion of the problems with rotating DNS (from dejanews) and a solution using reverse proxy http://www.webtechniques.com/features/1998/05/engelschall/engelschall.shtml Looks like Eddie and the Linux Virtual Server Project are the way to go though Joe -- Joseph Mack, Senior Systems Engineer, Lockheed Martin National Environmental Supercomputer Center (NESC), mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA From lindahl@cs.virginia.edu Mon, 25 Jan 1999 14:53:19 -0500 Date: Mon, 25 Jan 1999 14:53:19 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: Beoboard idea > Also, when looking at one PS per node, I worry both with the > cost of excess power consumption and excess ambient heat Last time this was talked about, it was claimed that this wasn't an issue. There *are* vendors who sell reasonably-priced piles of systems on shelves with all the appropriate fans; why not check them out? -- g From hozer@drgw.net Mon, 25 Jan 1999 15:17:15 -0500 Date: Mon, 25 Jan 1999 15:17:15 -0500 From: Troy Benjegerdes hozer@drgw.net Subject: Beoboard idea On Mon, 25 Jan 1999, Paul Oertel wrote: > I don't see why it would work but then I'm not too strung in the > electric wiring department. I've read that you need at least 120 watts > (have I got my units right?) for the motherboard and an average 12 watts > per each additional device (graphics card, hard drive etc.) For me > space is also a problem. I have been thinking if there is some way to > combine more than one motherboard to per box and share power supply > there would be a significant space savings. Maybe other componenents > (mice, keyboard) could be shared as well. So long as the box was > properly grounded, I think the box needn't be made of metal to function > properly. If space and power consumption are of primary importance, Morotola makes a dual 604ev-300Mhz embedded board in an ATX form factor. These boards are rated at 30 watts each, could be stacked at approximately 2 inches apart. This would allow up 4 boards in a mostly standard ATX case with a 250 watt power supply. The boards come with UW SCSI and 100BT ethernet (a tulip 21140 chip) on the board, and the firmware (bios) requires can configured via the serial port. Peak floating point performance is around 80% of a 450mhz Pentium II. If anyone wants more information on these boards, or is interested in getting a hold of a few, please let me know. > > R. Paul McCarty wrote: > > > > I've speant weeks looking for info on cases, power supplies, etc. that could > > power multiple boards in a single box with this sort of final result in mind. I > > don't seen any reason why it wont work (just splice off the main ATX or AT wire > > to multiple plugs to each board) as long as the power doesn't exceed the power > > supplies capacity. Do you have any idea why this wouldn't work? > > > > You didn't actually build one did you? ;-) > > No I didn't. > > > Thanks for any info. > > -Paul > > -- > Paul Oertel > ComiConnection: The ultimate guide to DC Comics. > http://www.shirokuma.com/comiconnection > > -------------------------------------------------------------------------- | Troy Benjegerdes | troy@microux.com | hozer@drgw.net | | Unix is user friendly... You just have to be friendly to it first. | | This message composed with 100% free software. http://www.gnu.org | -------------------------------------------------------------------------- From tmills@total-care.com Mon, 25 Jan 1999 15:59:00 -0500 Date: Mon, 25 Jan 1999 15:59:00 -0500 From: Tyrone Mills tmills@total-care.com Subject: Beoboard idea I could be mistaken but I believe I have (3) 900 - 1200W Power Supplies at home. I am currently in Holland on business so I can't verify the capacity of the Power Supplies at the moment. They are out of 3 Intergraph 8000 Series Workstations. The positive and negative lines coming off these things are the diameter of a nickel or dime (sorry for being so accurate, but I am having to picture them in my mind). I'll be back in Canada in April and can confirm then, or if anyone is really interested in them, I can have my wife read me the decals on the backs of each of them. I am not using any of them and have no plans to. If your looking for some big power supplies, I've got 3. I could possibly get more, I'll check into that. Tyrone Mills -----Original Message----- From: Jessen, Per To: 'paul.oertel@citicorp.com' Cc: 'beowulf' ; 'extremelinux' Date: Mon, 25 Jan 1999 15:59:00 -0500 Subject: RE: Beoboard idea >> -----Original Message----- >> From: Paul Oertel [mailto:paul.oertel@citicorp.com] >> Sent: 25 January 1999 12:06 >[snip] >> I don't see why it would work but then I'm not too strung in the >> electric wiring department. I've read that you need at least 120 watts >> (have I got my units right?) for the motherboard and an >> average 12 watts >> per each additional device (graphics card, hard drive etc.) For me >> space is also a problem. I have been thinking if there is some way to >> combine more than one motherboard to per box and share power supply >> there would be a significant space savings. Maybe other componenents >> (mice, keyboard) could be shared as well. So long as the box was >> properly grounded, I think the box needn't be made of metal >> to function properly. > > There are two ways of looking at this: optimum space, optimum price. > > Space: if space is at a premium, yes, looking at supplying several > motherboards/PCs from one (large) switching supply is certainly > possible. There are companies that build e.g. 600W and 800W switchmode > supplies, but that's about the biggest I've seen. And for a reasonably > size cluster you're probably looking at e.g. 1200W or more. Depends > on your config. And in that size you could be looking at a custom > build. Which doesn't exactly do wonders for the price. > > Price: if you've got space, but not the money, I don't think one supply > for an entire cluster is optimal. Even standard 600/800W supplies > do not 'scale' in price. 3 standard 250W supplies are always cheaper > than one 600W supply. > > As far as determining the size of your supply, you should be able to > find your motherboards consumption in your manual, and the same goes > for your harddisks. For other cards, you'll probably have to make some > educated guessing. > > Anyway, I'm looking at doing exactly the same thing, and I have been > out looking for large power-supplies - so, if you find something that > is economically viable when compared to an array of COTS 250W supplies, > let me know. > > >Best regards, >Per Jessen >Software Support Engineer, StorageTek International Software Support >mailto://per_jessen@storagetek.com, http://www.storagetek.com From aclose72@yahoo.com Mon, 25 Jan 1999 16:06:48 -0500 Date: Mon, 25 Jan 1999 16:06:48 -0500 From: andrew close aclose72@yahoo.com Subject: Beoboard idea do these boards plug into a standard pci bus??? i'm sure i've seen a similar product that looked like it plugged into a standard motherboard, but had it's own cpu's memory and bios built into the card. i wondered how and if these would work.. if so, they would be great for a linux cluster. :) andy >If space and power consumption are of primary >importance, Morotola >makes a >dual 604ev-300Mhz embedded board in an ATX form >factor. These boards are >rated at 30 watts each, could be stacked at >approximately 2 inches >apart. >This would allow up 4 boards in a mostly standard ATX >case with a 250 >watt >power supply. >The boards come with UW SCSI and 100BT ethernet (a >tulip 21140 chip) on >the board, and the firmware (bios) requires can >configured via the >serial >port. Peak floating point performance is around 80% >of a 450mhz Pentium II. _________________________________________________________ DO YOU YAHOO!? Get your free @yahoo.com address at http://mail.yahoo.com From rpmc@troi.cc.rochester.edu Mon, 25 Jan 1999 16:14:20 -0500 Date: Mon, 25 Jan 1999 16:14:20 -0500 From: R. Paul McCarty rpmc@troi.cc.rochester.edu Subject: Beoboard idea The problem I have is the price. I've been trying to see just how cheap a distributed computing cluster I could make, and I doubt this board is competitive with standard intel fare. Last price I came up with was just under $160/computer with harddrive, 300MHz cpu, memory, power, 10/100 network card, etc. I'm also a little skiddish about installing linux on PowerPC hardware. I love the specs on the chips, but I hear very little from people running LinuxPPC, etc. and I'm worried about device support and commodity parts. -Paul andrew close wrote: > > do these boards plug into a standard pci bus??? > i'm sure i've seen a similar product that looked like it plugged into > a standard motherboard, but had it's own cpu's memory and bios built > into the card. i wondered how and if these would work.. if so, they > would be great for a linux cluster. :) > > andy > > >If space and power consumption are of primary >importance, Morotola > >makes a > >dual 604ev-300Mhz embedded board in an ATX form >factor. These boards > are > >rated at 30 watts each, could be stacked at >approximately 2 inches > >apart. > >This would allow up 4 boards in a mostly standard ATX >case with a 250 > >watt > >power supply. > > >The boards come with UW SCSI and 100BT ethernet (a >tulip 21140 chip) > on > >the board, and the firmware (bios) requires can >configured via the > >serial > >port. Peak floating point performance is around 80% >of a 450mhz > Pentium II. > > _________________________________________________________ > DO YOU YAHOO!? > Get your free @yahoo.com address at http://mail.yahoo.com -- R. Paul McCarty / DARS Coordinator / rpmc@troi.cc.rochester.edu / x52059 317 Lattimore Hall, University of Rochester, Rochester, NY 14627 Computers don't make errors; what they do, they do on purpose.-Dale/KOTH From eroman@galaxy.ams.sunysb.edu Mon, 25 Jan 1999 16:39:02 -0500 Date: Mon, 25 Jan 1999 16:39:02 -0500 From: eroman@galaxy.ams.sunysb.edu eroman@galaxy.ams.sunysb.edu Subject: Do Linux 2.1/2.2 clusters work? This is not a stupid question. We've got almost 100 Linux systems here running kernels ranging from 2.1.125 to 2.2.0-pre9. Parallel execution on these systems is at best unreliable and at worst broken. 1/ netpipe (MPI) fails under LAM-c2c with message sizes larger than 63k or so. 2/ Our "premier" application has big time problems with MPI_Allreduce. It just plain crashes after about a hundred uses. mpitask reports a reports tons of messages sitting around in buffers somewhere. 3/ We have to invoke with -nger to even get that far. Otherwise there are "Internal MPI Error - NGER overflow"s about 10 seconds into execution. 4/ MPICH just plain sucks. It works only after a fresh reboot of a system. It seems to hang in a select call. One process sits around select()'ing at a dead socket until we kill it. 5/ Oddly, the NAS benchmarks are about the only thing that I can run. Sometimes. They don't work with -c2c, or -nger. I don't know why. 6/ The NAS NPB programs crash randomly when run under MPICH. Same problem with the select()ing. We're running dual P-II 300 and P-II 400 systems. Everything works _fine_ under kernel 2.0.36. I would like to know if any of you do have a working cluster running 2.1 or 2.2 kernels without any of these mystery problems. If so, how are we different? I want a working 2.2 cluster. -- Eric Roman Department of Applied Mathematics (516)632-8545 SUNY/Stony Brook From Lee.Goodin@cas.honeywell.com Mon, 25 Jan 1999 16:39:52 -0500 Date: Mon, 25 Jan 1999 16:39:52 -0500 From: Goodin, Lee (AZ77) Lee.Goodin@cas.honeywell.com Subject: Check out the cubix boxes! I have been listening to the net for some time know, and have discovered that space is a primary concern. Well just the other day, I had the opportunity to visit http://www.cubix.com and they have up to 8 PC's in one box. These things are probably expensive. One picture shows a wall with 40 boxes and 2 monitors. They claim this system has 320 PC's in it. They also claim the system is for windows NT or Novel OS. I doubt it would have a problem with LINUX though. So if you budgeted $1500 per node perhaps this system would not come in to far over budget. The site did not disclose cost but these boxes look like $20K each to me. Lee Goodin From hozer@drgw.net Mon, 25 Jan 1999 16:58:29 -0500 Date: Mon, 25 Jan 1999 16:58:29 -0500 From: Troy Benjegerdes hozer@drgw.net Subject: Beoboard idea [CC: list clipped] On Mon, 25 Jan 1999, andrew close wrote: > do these boards plug into a standard pci bus??? > i'm sure i've seen a similar product that looked like it plugged into > a standard motherboard, but had it's own cpu's memory and bios built > into the card. i wondered how and if these would work.. if so, they > would be great for a linux cluster. :) > > andy The boards I am referring to are full ATX form factor motherboards, which can accept up to 3 PCI cards (but can't be stacked with PCI cards installed). I believe you are referring to TotalImact PowerPC PCI add-on cards. The problem with the PCI cards is that they don't have an ethernet interface, so communication between boards would be more difficult. (Not impossible, but IMHO, more work than is practical) > >If space and power consumption are of primary >importance, Morotola > >makes a > >dual 604ev-300Mhz embedded board in an ATX form >factor. These boards > are > >rated at 30 watts each, could be stacked at >approximately 2 inches > >apart. > >This would allow up 4 boards in a mostly standard ATX >case with a 250 > >watt > >power supply. > > >The boards come with UW SCSI and 100BT ethernet (a >tulip 21140 chip) > on > >the board, and the firmware (bios) requires can >configured via the > >serial > >port. Peak floating point performance is around 80% >of a 450mhz > Pentium II. > -------------------------------------------------------------------------- | Troy Benjegerdes | troy@microux.com | hozer@drgw.net | | Unix is user friendly... You just have to be friendly to it first. | | This message composed with 100% free software. http://www.gnu.org | -------------------------------------------------------------------------- From chrisaaa@expert.cc.purdue.edu Mon, 25 Jan 1999 17:38:01 -0500 Date: Mon, 25 Jan 1999 17:38:01 -0500 From: Christopher Burnside chrisaaa@expert.cc.purdue.edu Subject: Beoboard idea I'm interested. How much are they? What's the minimum order number? Christopher Burnside Purdue University Junior, Aeronautical & Astronautical Engineering From alvin@iplink.net Mon, 25 Jan 1999 18:02:09 -0500 Date: Mon, 25 Jan 1999 18:02:09 -0500 From: Alvin Starr alvin@iplink.net Subject: Beoboard idea On Mon, 25 Jan 1999, Troy Benjegerdes wrote: > On Mon, 25 Jan 1999, Paul Oertel wrote: > > > I don't see why it would work but then I'm not too strung in the > > electric wiring department. I've read that you need at least 120 watts > > (have I got my units right?) for the motherboard and an average 12 watts > > per each additional device (graphics card, hard drive etc.) For me > > space is also a problem. I have been thinking if there is some way to > > combine more than one motherboard to per box and share power supply > > there would be a significant space savings. Maybe other componenents > > (mice, keyboard) could be shared as well. So long as the box was > > properly grounded, I think the box needn't be made of metal to function > > properly. > > If space and power consumption are of primary importance, Morotola makes a > dual 604ev-300Mhz embedded board in an ATX form factor. These boards are > rated at 30 watts each, could be stacked at approximately 2 inches apart. > This would allow up 4 boards in a mostly standard ATX case with a 250 watt > power supply. > > The boards come with UW SCSI and 100BT ethernet (a tulip 21140 chip) on > the board, and the firmware (bios) requires can configured via the serial > port. Peak floating point performance is around 80% of a 450mhz Pentium > II. > > If anyone wants more information on these boards, or is interested in > getting a hold of a few, please let me know. > > > > > R. Paul McCarty wrote: > > > > > > I've speant weeks looking for info on cases, power supplies, etc. that could > > > power multiple boards in a single box with this sort of final result in mind. I > > > don't seen any reason why it wont work (just splice off the main ATX or AT wire > > > to multiple plugs to each board) as long as the power doesn't exceed the power > > > supplies capacity. Do you have any idea why this wouldn't work? > > > > > > You didn't actually build one did you? ;-) > > > > No I didn't. > > > > > Thanks for any info. > > > -Paul > > > > -- > > Paul Oertel > > ComiConnection: The ultimate guide to DC Comics. > > http://www.shirokuma.com/comiconnection > > > > > > -------------------------------------------------------------------------- > | Troy Benjegerdes | troy@microux.com | hozer@drgw.net | > | Unix is user friendly... You just have to be friendly to it first. | > | This message composed with 100% free software. http://www.gnu.org | > -------------------------------------------------------------------------- > Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin@iplink.net || From deadline@plogic.com Mon, 25 Jan 1999 22:16:00 -0500 Date: Mon, 25 Jan 1999 22:16:00 -0500 From: Douglas Eadline deadline@plogic.com Subject: Do Linux 2.1/2.2 clusters work? On Mon, 25 Jan 1999 eroman@galaxy.ams.sunysb.edu wrote: > > This is not a stupid question. > > We've got almost 100 Linux systems here running kernels ranging from 2.1.125 > to 2.2.0-pre9. Parallel execution on these systems is at best unreliable and > at worst broken. > > 1/ netpipe (MPI) fails under LAM-c2c with message sizes larger than 63k > or so. I think I will try this. Just did. Works Ok (I'm using two dual PII266 LX systems) Latency = 139 us, best rate is 90 Mbps (intel NICS) > > 2/ Our "premier" application has big time problems with MPI_Allreduce. > It just plain crashes after about a hundred uses. mpitask reports a reports > tons of messages sitting around in buffers somewhere. Have you sent this to the LAM mailing list. The LAM guys are very good about responding. > > 3/ We have to invoke with -nger to even get that far. Otherwise there are > "Internal MPI Error - NGER overflow"s about 10 seconds into execution. > > 4/ MPICH just plain sucks. It works only after a fresh reboot of a system. > It seems to hang in a select call. One process sits around select()'ing > at a dead socket until we kill it. There are known problems with MPICH and Linux. > > 5/ Oddly, the NAS benchmarks are about the only thing that I can run. > Sometimes. They don't work with -c2c, or -nger. I don't know why. These are Ok on our machines with -c2c. What kind of machines? > > 6/ The NAS NPB programs crash randomly when run under MPICH. Same problem > with the select()ing. I have never been able to get a reproducible run with MPICH > > We're running dual P-II 300 and P-II 400 systems. Everything works _fine_ > under kernel 2.0.36. You may want to disable shared memory in LAM this may cause some problems - I'm not sure where it is, but the LAM list archive mentions this. > > I would like to know if any of you do have a working cluster running 2.1 > or 2.2 kernels without any of these mystery problems. If so, how are we > different? > I have not done any extenive testing wtih 2.1.x. 2.0.36 seems to work OK in most cases. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From lindahl@cs.virginia.edu Mon, 25 Jan 1999 22:30:19 -0500 Date: Mon, 25 Jan 1999 22:30:19 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: Do Linux 2.1/2.2 clusters work? > We've got almost 100 Linux systems here running kernels ranging from 2.1.125 > to 2.2.0-pre9. Parallel execution on these systems is at best unreliable and > at worst broken. Hm. You'd almost think that a nice and serious "multiple entry into interrupt handler" problem had crept in, or gotten worse, or something along those lines. All of the stuff you cite seems to be very kernel related and bad. I'm no help because all my boxes are still at 2.0.3[56], but I have 65 new Alphas that I want to run with 2.2 in order to get IDE with DMA, so I'll be joining you in misery soon enough. -- greg From lindahl@cs.virginia.edu Tue, 26 Jan 1999 00:28:43 -0500 Date: Tue, 26 Jan 1999 00:28:43 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: Check out the cubix boxes! > I have been listening to the net for some time know, and have discovered > that space is a primary concern. Well just the other day, I had the > opportunity to visit http://www.cubix.com and they have up to 8 PC's in one > box. These things are probably expensive Not only are they more expensive than 2U PC's, they don't use a standard motherboard, so it should be no surprise that they're behind on technology. For example, in October 1998, there was a review published in an NT-related magazine of a system with 4 dual 200 mhz PPro chips. Either these guys just don't have 450 mhz Pentium II boxes or they are setting their prices for marketing reasons and not technical reasons. The other vendors in this space -- Alta Technologies, Paralogic, Aspen Systems -- are all shipping standard form-factor motherboards, so they don't have this problem. And they know what Linux is. > The site did not disclose cost > but these boxes look like $20K each to me. The system in the October 1998 review was 4x2 200 mhz cpus for $38,000. -- g From eroman@galaxy.ams.sunysb.edu Tue, 26 Jan 1999 01:08:18 -0500 Date: Tue, 26 Jan 1999 01:08:18 -0500 From: eroman@galaxy.ams.sunysb.edu eroman@galaxy.ams.sunysb.edu Subject: Do Linux 2.1/2.2 clusters work? > > This is not a stupid question. > > > > We've got almost 100 Linux systems here running kernels ranging from 2.1.125 > > to 2.2.0-pre9. Parallel execution on these systems is at best unreliable and > > at worst broken. > > > > 1/ netpipe (MPI) fails under LAM-c2c with message sizes larger than 63k > > or so. > > I think I will try this. Just did. Works Ok (I'm using two dual PII266 > LX systems) Latency = 139 us, best rate is 90 Mbps (intel NICS) Which kernel? You forgot to mention. Under 2.0.33 I was able to get about 90 Mb/s or so with -c2c. It don't seem to work now. > > 2/ Our "premier" application has big time problems with MPI_Allreduce. > > It just plain crashes after about a hundred uses. mpitask reports a reports > > tons of messages sitting around in buffers somewhere. > > Have you sent this to the LAM mailing list. The LAM guys are very good > about responding. I'm trying to make sure it's not a bug in their code first. If I can isolate it, I'll send a message to the LAM list. (is there an archive handy to browse?) > > 3/ We have to invoke with -nger to even get that far. Otherwise there are > > "Internal MPI Error - NGER overflow"s about 10 seconds into execution. > > > > 4/ MPICH just plain sucks. It works only after a fresh reboot of a system. > > It seems to hang in a select call. One process sits around select()'ing > > at a dead socket until we kill it. > > There are known problems with MPICH and Linux. Do you know what the "known problems" are? I know you were seeing our problem with socket destruction a while ago. Was that under 2.1 or 2.0? I had MPICH working with linux 2.0 a while ago. It seemed like it worked under 2.1 until 2.1.87 or so. I'm not clear on the details. > > 5/ Oddly, the NAS benchmarks are about the only thing that I can run. > > Sometimes. They don't work with -c2c, or -nger. I don't know why. > > These are Ok on our machines with -c2c. What kind of machines? These are SuperMicro P6DLE and DBE machines running at 300 MHz under linux 2.2.0-pre9. I'm going to check the runs with -c2c again. That might be old news. > > 6/ The NAS NPB programs crash randomly when run under MPICH. Same problem > > with the select()ing. > I have never been able to get a reproducible run with MPICH > > > > We're running dual P-II 300 and P-II 400 systems. Everything works _fine_ > > under kernel 2.0.36. > > You may want to disable shared memory in LAM this may cause some > problems - I'm not sure where it is, but the LAM list archive mentions > this. I'll give it a shot. > > I would like to know if any of you do have a working cluster running 2.1 > > or 2.2 kernels without any of these mystery problems. If so, how are we > > different? > > > I have not done any extenive testing wtih 2.1.x. 2.0.36 seems to > work OK in most cases. So let's talk. You're using dual P-II 266's. Do you have NAS Benchmark output files? I'd like to compare notes. I'd like to see how your cluster performs under NAS. We found that scaling was horrible under 2.0 kernels running SMP. 64, 100, 121 processors dropped in performance big-time. Though we still saw some speed up on some class C problems. Generally, the minute we start using 2 processors/node things slow down, real bad. (I don't have the data handy, I'll tell you how bad "real bad" is later.) -- Eric Roman Department of Applied Mathematics (516)632-8545 SUNY/Stony Brook From thorpe@cerco.ups-tlse.fr Tue, 26 Jan 1999 05:01:30 -0500 Date: Tue, 26 Jan 1999 05:01:30 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea Just thought I'd remind you folk of the sa-beowulf idea which is coming along pretty nicely. Chalice Technology (http://www.chaltech.com) are developing a multiprocessor PCI board which (in my humble opinion) would be pretty ideal for building a desktop beowulf system. The system is based on a small CPU module composed of a StrongARM SA-110 microprocessor, 64 Megs of SDRAM and a PCI bridge. The modules work (they've had NetBSD up and running, and Linux shouldn't be a problem) and you can see some nice photos on Chaltechs web-page. The idea now is to put up to 8 of these processor modules on a PCI board and effectively build a beowulf system in which IP messaging via ethernet is replaced by direct processor-to-processor communications over the PCI bus. I hope to get my first boards within a month or so. Each node will be running Linux, so you can program it much as you would any other beowulf system, and a single PC motherboard with four free PCI slots could have up to 32 nodes in it. The only real drawback (as far as I can see) is that the current generation of StrongARMs only does fixed-point arithmatic, but in fact, the recently announced ARM1020T chip will have an optional floating point processor, and (in principle) it shouldn't be too difficult to produce a CPU module using the new chip. For those who are interested, there'a a website that Kragen Sitaker set up at http://www.dnaco.net/~kragen/sa-beowulf/, and a mailing list (send a mail to sa-beowulf-subscribe@kragen.dnaco.net) Cheers Simon __________________ Simon Thorpe Centre de Recherche Cerveau et Cognition 133, route de Narbonne 31062 Toulouse France Tel 33 (0)5 62 17 28 03. Fax 33 (0)5 62 17 28 09 http://www.cerco.ups-tlse.fr/~modele/index.html __________________ From fakel@mail.magelan.ru Tue, 26 Jan 1999 06:28:31 -0500 Date: Tue, 26 Jan 1999 06:28:31 -0500 From: Alikaa fakel@mail.magelan.ru Subject: What programs i need to.. Hi I'm new here so i have a serious question: What programs, applications i need to set up Beowulf with my Slackware! I have heard about something, but is not clear. Could you tell me? Maybe some basic? Thanks Alikaa From alvin@iplink.net Tue, 26 Jan 1999 08:37:42 -0500 Date: Tue, 26 Jan 1999 08:37:42 -0500 From: Alvin Starr alvin@iplink.net Subject: Beoboard idea On Tue, 26 Jan 1999, Simon Thorpe wrote: > Just thought I'd remind you folk of the sa-beowulf idea which is coming > along pretty nicely. Chalice Technology (http://www.chaltech.com) are > developing a multiprocessor PCI board which (in my humble opinion) would be > pretty ideal for building a desktop beowulf system. The system is based on > a small CPU module composed of a StrongARM SA-110 microprocessor, 64 Megs > of SDRAM and a PCI bridge. The modules work (they've had NetBSD up and > running, and Linux shouldn't be a problem) and you can see some nice photos > on Chaltechs web-page. The idea now is to put up to 8 of these processor > modules on a PCI board and effectively build a beowulf system in which IP > messaging via ethernet is replaced by direct processor-to-processor > communications over the PCI bus. I hope to get my first boards within a > month or so. > > Each node will be running Linux, so you can program it much as you would > any other beowulf system, and a single PC motherboard with four free PCI > slots could have up to 32 nodes in it. > > The only real drawback (as far as I can see) is that the current generation > of StrongARMs only does fixed-point arithmatic, but in fact, the recently > announced ARM1020T chip will have an optional floating point processor, and > (in principle) it shouldn't be too difficult to produce a CPU module using > the new chip. > > For those who are interested, there'a a website that Kragen Sitaker set up > at http://www.dnaco.net/~kragen/sa-beowulf/, and a mailing list (send a > mail to sa-beowulf-subscribe@kragen.dnaco.net) This sounds much like a product that corel has. It is about the size of one of the nutshell(O'Reilly) books(How is that for accuracy). For those interested take a look at: http://www.corelcomputer.com/products/index.htm I thought of these earlier but did not mention them on this list because they do not have hardware floating point. But from the point of view of a large number of small computers that will fit in a tiny space these are great and availble. Alvin Starr || voice: (416)585-9971 Interlink Connectivity || fax: (416)585-9974 alvin@iplink.net || From thorpe@cerco.ups-tlse.fr Tue, 26 Jan 1999 09:07:31 -0500 Date: Tue, 26 Jan 1999 09:07:31 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea >On Tue, 26 Jan 1999, Alvin Starr wrote: > >> Just thought I'd remind you folk of the sa-beowulf idea which is coming >> along pretty nicely. Chalice Technology (http://www.chaltech.com) are >> developing a multiprocessor PCI board which (in my humble opinion) would be >> pretty ideal for building a desktop beowulf system. The system is based on >> a small CPU module composed of a StrongARM SA-110 microprocessor, 64 Megs >> of SDRAM and a PCI bridge. Snip > >This sounds much like a product that corel has. It is about the size of >one of the nutshell(O'Reilly) books(How is that for accuracy). > >For those interested take a look at: > http://www.corelcomputer.com/products/index.htm > >I thought of these earlier but did not mention them on this list because >they do not have hardware floating point. But from the point of view of a >large number of small computers that will fit in a tiny space these are >great and availble. Corel's product is certainly an interesting machine. But for beowulf, it's already got loads of things which aren't really necessary in every node. Do you really need graphics, keyboard, hard disk drive and so on in every node? (Actually, I would say that the same criticism applies to almost all current beowulf systems). I can't remember exactly what price Corel are selling the netwinder for - was it $850? Buy 32 of them and you've already spent quite a lot of money (at least if you've got my sort of budget). Chalice have been talking about $3000 for 8 CPUs which is quite a bit cheaper - and that's for the first production run of about 25 boards. Chances are that, with enough interest, prices could come down quite a bit more. But it seems to me that the real advantage is the use of direct PCI based message passing at 132 MBytes/second. In principle, it should be possible for the host machine to load Linux on each of the CPU modules in as little as 3 ms. Actually, you can expect an even higher communications bandwidth than this, because each multiprocessor board will have one (or perhaps two) independent PCI busses. I for one can't wait for this particular bit of vapourware to condense.... ;-) Simon __________________ Simon Thorpe Centre de Recherche Cerveau et Cognition 133, route de Narbonne 31062 Toulouse France Tel 33 (0)5 62 17 28 03. Fax 33 (0)5 62 17 28 09 http://www.cerco.ups-tlse.fr/~modele/index.html __________________ From akc@mip.ou.dk Tue, 26 Jan 1999 09:26:37 -0500 Date: Tue, 26 Jan 1999 09:26:37 -0500 From: Arnold Kirkmand Christensen. akc@mip.ou.dk Subject: Beoboard idea I think SUN makes such a gimmick, or are about to release. akc. On Mon, 25 Jan 1999, andrew close wrote: > do these boards plug into a standard pci bus??? > i'm sure i've seen a similar product that looked like it plugged into > a standard motherboard, but had it's own cpu's memory and bios built > into the card. i wondered how and if these would work.. if so, they > would be great for a linux cluster. :) > > andy > > >If space and power consumption are of primary >importance, Morotola > >makes a > >dual 604ev-300Mhz embedded board in an ATX form >factor. These boards > are > >rated at 30 watts each, could be stacked at >approximately 2 inches > >apart. > >This would allow up 4 boards in a mostly standard ATX >case with a 250 > >watt > >power supply. > > >The boards come with UW SCSI and 100BT ethernet (a >tulip 21140 chip) > on > >the board, and the firmware (bios) requires can >configured via the > >serial > >port. Peak floating point performance is around 80% >of a 450mhz > Pentium II. > > > > > > > > _________________________________________________________ > DO YOU YAHOO!? > Get your free @yahoo.com address at http://mail.yahoo.com > From john.dubberley@nrlssc.navy.mil Tue, 26 Jan 1999 09:33:18 -0500 Date: Tue, 26 Jan 1999 09:33:18 -0500 From: john dubberley john.dubberley@nrlssc.navy.mil Subject: Mobile beowulf Next Wednesday my test beowulf system goes to sea, 4 node Pentium II. Is anyone else working on mobile systems? The plan here is to hash out operating problems this trip and replace with a 16-32 node rack mounted system. Does anyone have suggestions for a tough rack/case for the nodes? They need to be able to stand up to longshoremen and heavy sea states. -john dubberley From camm@enhanced.com Tue, 26 Jan 1999 09:51:30 -0500 Date: Tue, 26 Jan 1999 09:51:30 -0500 From: Camm Maguire camm@enhanced.com Subject: Do Linux 2.1/2.2 clusters work? eroman@galaxy.ams.sunysb.edu writes: > This is not a stupid question. > > We've got almost 100 Linux systems here running kernels ranging from 2.1.125 > to 2.2.0-pre9. Parallel execution on these systems is at best unreliable and > at worst broken. > > 1/ netpipe (MPI) fails under LAM-c2c with message sizes larger than 63k > or so. > We have a 16 node cluster running 2.1.130. We're going to upgrade soon to 2.2.0 now that its released. All seems working on our very limited testing so far. We're using lam 6.1. But I have noticed one thus far inexplicable problem similar to something you note above. When we try to run an MPI_reduce on buffers that are too large, the process hangs. I'm assuming this may have something to do with the NGER hard coded buffer size, but we don't get NGER overflow messages on output. For our code, there is a simple work around, so we have not pursued the matter further thus far. I hope this helps! -- Camm Maguire camm@enhanced.com ========================================================================== "The earth is but one country, and mankind its citizens." -- Baha'u'llah From deadline@plogic.com Tue, 26 Jan 1999 10:16:50 -0500 Date: Tue, 26 Jan 1999 10:16:50 -0500 From: Douglas Eadline deadline@plogic.com Subject: Beoboard idea On Tue, 26 Jan 1999, Simon Thorpe wrote: -snip- > > Corel's product is certainly an interesting machine. But for beowulf, it's > already got loads of things which aren't really necessary in every node. Do > you really need graphics, keyboard, hard disk drive and so on in every > node? (Actually, I would say that the same criticism applies to almost all > current beowulf systems). graphics - well no, cheap PCI video, a real good idea for soving problems. keyboard - unless it is a dual use cluster, no one puts a keyboard or monitor on every system. hardrive - some applications need a local hard drive Puting lots of CPUs in a small space has some merits for some problems, but as a general purpose solution, it tends not to work for many people. > > I can't remember exactly what price Corel are selling the netwinder for - > was it $850? Buy 32 of them and you've already spent quite a lot of money > (at least if you've got my sort of budget). Chalice have been talking about > $3000 for 8 CPUs which is quite a bit cheaper - and that's for the first > production run of about 25 boards. Chances are that, with enough interest, > prices could come down quite a bit more. And what is the limit of CPUs you can put in the machine. This will limit problem size (scalability). > > But it seems to me that the real advantage is the use of direct PCI based > message passing at 132 MBytes/second. In principle, it should be possible > for the host machine to load Linux on each of the CPU modules in as little > as 3 ms. Actually, you can expect an even higher communications bandwidth > than this, because each multiprocessor board will have one (or perhaps two) > independent PCI busses. "In principle" is sometims hard to translate into "In pratice". Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From thorpe@cerco.ups-tlse.fr Tue, 26 Jan 1999 10:45:37 -0500 Date: Tue, 26 Jan 1999 10:45:37 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea >On Tue, 26 Jan 1999, Doug wrote: >> I can't remember exactly what price Corel are selling the netwinder for - >> was it $850? Buy 32 of them and you've already spent quite a lot of money >> (at least if you've got my sort of budget). Chalice have been talking about >> $3000 for 8 CPUs which is quite a bit cheaper - and that's for the first >> production run of about 25 boards. Chances are that, with enough interest, >> prices could come down quite a bit more. > >And what is the limit of CPUs you can put in the machine. This will limit >problem size (scalability). If you're using a standard Pentium II motherboard with four free PCI slots, you should be able to get 32 processors in. There won't be an overheating problem, because the power consumption of the StrongARMs is so low - they hardly get hot at all, even with no heatsink. The plan is to have CPU modules on both sides of the PCI board which really improves the packing density, but still only needs one PCI slot per board. Mind you, what I'd really like to do is get a PCI backplane, put a Single Board Computer in one slot as a host, and then, if my finances allow it, fill up the other slots with multiprocessor boards. You can get PCI backplanes with as many as 17 PCI slots in them - so that would allow for 16 * 8 = 128 CPUs. Could be fun... > > >> But it seems to me that the real advantage is the use of direct PCI based >> message passing at 132 MBytes/second. In principle, it should be possible >> for the host machine to load Linux on each of the CPU modules in as little >> as 3 ms. Actually, you can expect an even higher communications bandwidth >> than this, because each multiprocessor board will have one (or perhaps two) >> independent PCI busses. > >"In principle" is sometims hard to translate into "In pratice". Yeah - I'm praying too... (Actually the way I said it wasn't very clear - what I meant was 3 ms per CPU module, based on a minimal Linux kernel of around 500Kbytes, and a DMA block transfer rate of 132 Mbytes/second. So a PCI backplane with 128 CPUs could be booted in under a second). Simon __________________ Simon Thorpe Centre de Recherche Cerveau et Cognition 133, route de Narbonne 31062 Toulouse France Tel 33 (0)5 62 17 28 03. Fax 33 (0)5 62 17 28 09 http://www.cerco.ups-tlse.fr/~modele/index.html __________________ From joelja@darkwing.uoregon.edu Tue, 26 Jan 1999 11:17:37 -0500 Date: Tue, 26 Jan 1999 11:17:37 -0500 From: Joel Jaeggli joelja@darkwing.uoregon.edu Subject: Beoboard idea On Tue, 26 Jan 1999, Arnold Kirkmand Christensen. wrote: > > > I think SUN makes such a gimmick, or are about to release. > > akc. > > On Mon, 25 Jan 1999, andrew close wrote: > > > do these boards plug into a standard pci bus??? > > i'm sure i've seen a similar product that looked like it plugged into > > a standard motherboard, but had it's own cpu's memory and bios built > > into the card. i wondered how and if these would work.. if so, they > > would be great for a linux cluster. :) what you are refering to is called a Single Board Computer, they're not as small as you might think because they connect ot a passive backplne which contains some combination of pci or isa slots (or pci or isa only) you can get backplanes with twelve or more slots that would support three or four SBC's. SBC's are generally on the verge of terrifyingly expensive because of the level of integration required to get an enitre motherboard minus slots integrated into the space of a full length isa card, still if you need 10 isa slots or 4 computers in one box with 4 pic slots each SBC's are the (pricey) way to go. computer express assembles machines around sbc's. They have pictures on their website at: http://www.computerexpressct.com/ > > andy > > > > >If space and power consumption are of primary >importance, Morotola > > >makes a > > >dual 604ev-300Mhz embedded board in an ATX form >factor. These boards > > are > > >rated at 30 watts each, could be stacked at >approximately 2 inches > > >apart. > > >This would allow up 4 boards in a mostly standard ATX >case with a 250 > > >watt > > >power supply. > > > > >The boards come with UW SCSI and 100BT ethernet (a >tulip 21140 chip) > > on > > >the board, and the firmware (bios) requires can >configured via the > > >serial > > >port. Peak floating point performance is around 80% >of a 450mhz > > Pentium II. > > > > > > > > > > > > > > > > _________________________________________________________ > > DO YOU YAHOO!? > > Get your free @yahoo.com address at http://mail.yahoo.com > > > -------------------------------------------------------------------------- Joel Jaeggli joelja@darkwing.uoregon.edu Academic User Services consult@gladstone.uoregon.edu PGP Key Fingerprint: 1DE9 8FCA 51FB 4195 B42A 9C32 A30D 121E -------------------------------------------------------------------------- It is clear that the arm of criticism cannot replace the criticism of arms. Karl Marx -- Introduction to the critique of Hegel's Philosophy of the right, 1843. From andy@meshplc.co.uk Tue, 26 Jan 1999 11:51:58 -0500 Date: Tue, 26 Jan 1999 11:51:58 -0500 From: Andy andy@meshplc.co.uk Subject: Beoboard idea One topic that keeps cropping up is the need for large (~1000W) PSU's for these assemblies. Someone was saying that getting hold of switched mode PSU's at that level would be very costly, which is true. However, does it have to be switched mode? There are plenty of standard power supplies available at this level with very good tolerances and ripple characteristics, designed primarily for radio transmitters and the like. These should prove amply capable of running a computer setup. As an example, I have a 1KW unit somewhere that was originally used to power a radar console at a RAF air base, and which, although it has suffered some physical damage, has been powering some of my radio gear for several years without incident. These sorts of thing are available via surplus channels, and through radio ralleys and the like. Andy MESH Computers Webmaster webmaster@meshplc.co.uk From mack@nesc.epa.gov Tue, 26 Jan 1999 11:55:28 -0500 Date: Tue, 26 Jan 1999 11:55:28 -0500 From: Joseph Mack mack@nesc.epa.gov Subject: Beoboard idea Alvin Starr wrote: > > On Tue, 26 Jan 1999, Simon Thorpe wrote: > > > > This sounds much like a product that corel has. It is about the size of > one of the nutshell(O'Reilly) books(How is that for accuracy). > > For those interested take a look at: > http://www.corelcomputer.com/products/index.htm > > I thought of these earlier but did not mention them on this list because > they do not have hardware floating point. But from the point of view of a > large number of small computers that will fit in a tiny space these are > great and availble. There's also some problem, which I don't exactly remember, but I think it's something like the Linux kernel being booted from ROM (and hence you can't change it) Joe -- Joseph Mack, Senior Systems Engineer, Lockheed Martin National Environmental Supercomputer Center (NESC), mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA From m.j.s.vandoesburg@student.utwente.nl Tue, 26 Jan 1999 12:35:09 -0500 Date: Tue, 26 Jan 1999 12:35:09 -0500 From: Mark van Doesburg m.j.s.vandoesburg@student.utwente.nl Subject: Beoboard idea Douglas Eadline wrote: And what is the limit of CPUs you can put in the machine. This will limit problem size (scalability). The number of CPU's you can put into the machine is limited by the 4GB PCI memory space. For performance reasons I map all memory of all SA's into the PCI memory space, this limits the number of CPU's to 64 if you use 64MB of RAM per CPU. Obviously the host system needs some of the space as well so i guess the limit will be 56 CPU's. Multiple hosts with 56 SA's can be networked together of course. greetings, Mark. From kragen@pobox.com Tue, 26 Jan 1999 12:41:08 -0500 Date: Tue, 26 Jan 1999 12:41:08 -0500 From: Kragen Sitaker kragen@pobox.com Subject: Beoboard idea On Tue, 26 Jan 1999, Douglas Eadline wrote: > On Tue, 26 Jan 1999, Simon Thorpe wrote: > > I can't remember exactly what price Corel are selling the netwinder for - > > was it $850? Buy 32 of them and you've already spent quite a lot of money > > (at least if you've got my sort of budget). Chalice have been talking about > > $3000 for 8 CPUs which is quite a bit cheaper - and that's for the first > > production run of about 25 boards. Chances are that, with enough interest, > > prices could come down quite a bit more. > > And what is the limit of CPUs you can put in the machine. This will limit > problem size (scalability). Probably 18 CPUs for a normal 3-PCI-slot machine. Possibly 24. > "In principle" is sometims hard to translate into "In pratice". Watch and see. ;) -- Kragen Sitaker Computers are the tools of the devil. It is as simple as that. There is no monotheism strong enough that it cannot be shaken by Unix or any Microsoft product. The devil is real. He lives inside C programs. -- philg@mit.edu From kragen@pobox.com Tue, 26 Jan 1999 12:47:43 -0500 Date: Tue, 26 Jan 1999 12:47:43 -0500 From: Kragen Sitaker kragen@pobox.com Subject: Beoboard idea On Tue, 26 Jan 1999, Simon Thorpe wrote: > If you're using a standard Pentium II motherboard with four free PCI slots, > you should be able to get 32 processors in. I haven't seen a lot of mobos with 4 PCI slots, but I haven't looked in a year or so. And I thought the design had been revised to only have six SAs per PCI card instead of 8? > Mind you, what I'd really like to do is get a PCI backplane, put a Single > Board Computer in one slot as a host, and then, if my finances allow it, > fill up the other slots with multiprocessor boards. You can get PCI > backplanes with as many as 17 PCI slots in them - so that would allow for > 16 * 8 = 128 CPUs. Could be fun... Right. And sebringring.com apparently has plans for much bigger PCI backplanes. Other possibilities include small FE cards, perhaps two or four per PCI card, permitting the SAs to be interconnected with a nice FE switch instead of relying on PCI -- which could get anemic beyond 10 or so CPUs for problems requiring a lot of communication. > So a PCI backplane with 128 CPUs could be booted in under a second). Well, more than likely, it'll take under a second to get Linux to start booting on all those CPUs; it might take half a minute to finish booting. -- Kragen Sitaker Computers are the tools of the devil. It is as simple as that. There is no monotheism strong enough that it cannot be shaken by Unix or any Microsoft product. The devil is real. He lives inside C programs. -- philg@mit.edu From frank@kujawski.org Tue, 26 Jan 1999 12:49:18 -0500 Date: Tue, 26 Jan 1999 12:49:18 -0500 From: Frank Kujawski frank@kujawski.org Subject: Beoboard idea I hope that everybody out fills up thier machine so the proce drops and I can afford to get one that is fully populated. Frank On Tue, 26 Jan 1999, Kragen Sitaker wrote: > On Tue, 26 Jan 1999, Douglas Eadline wrote: > > On Tue, 26 Jan 1999, Simon Thorpe wrote: > > > I can't remember exactly what price Corel are selling the netwinder for - > > > was it $850? Buy 32 of them and you've already spent quite a lot of money > > > (at least if you've got my sort of budget). Chalice have been talking about > > > $3000 for 8 CPUs which is quite a bit cheaper - and that's for the first > > > production run of about 25 boards. Chances are that, with enough interest, > > > prices could come down quite a bit more. > > > > And what is the limit of CPUs you can put in the machine. This will limit > > problem size (scalability). > > Probably 18 CPUs for a normal 3-PCI-slot machine. Possibly 24. > > > "In principle" is sometims hard to translate into "In pratice". > > Watch and see. ;) > > -- > Kragen Sitaker > Computers are the tools of the devil. It is as simple as that. There is no > monotheism strong enough that it cannot be shaken by Unix or any Microsoft > product. The devil is real. He lives inside C programs. -- philg@mit.edu > From xpertz@hotmail.com Tue, 26 Jan 1999 13:02:48 -0500 Date: Tue, 26 Jan 1999 13:02:48 -0500 From: ex pert xpertz@hotmail.com Subject: Beoboard idea I believe that switching power supplies, only use what power they need. In other words if you have a 250w power supply and only 100w of usage, the power supply will only use the 100w not the max of 250w. >From owner-beowulf@beowulf.gsfc.nasa.gov Mon Jan 25 13:37:18 1999 >Received: (from majordomo@localhost) > by beowulf.gsfc.nasa.gov (8.8.7/8.8.7) id NAA15671 > for beowulf-list; Mon, 25 Jan 1999 13:54:58 -0500 >Received: from worf.trendcmhs.org (worf.trendcmhs.org [205.138.62.12]) > by beowulf.gsfc.nasa.gov (8.8.7/8.8.7) with ESMTP id NAA15662 > for ; Mon, 25 Jan 1999 13:54:48 -0500 >Received: by worf.trendcmhs.org with Internet Mail Service (5.0.1457.3) > id ; Mon, 25 Jan 1999 13:51:00 -0500 >Message-ID: >From: "Stanley, Jeremy" >To: "'beowulf'" >Subject: RE: Beoboard idea >Date: Mon, 25 Jan 1999 13:50:58 -0500 >X-Priority: 3 >X-MS-TNEF-Correlator: >MIME-Version: 1.0 >X-Mailer: Internet Mail Service (5.0.1457.3) >Content-Type: multipart/mixed; > boundary="---- =_NextPart_000_01BE4869.BD721990" >Sender: owner-beowulf@beowulf.gsfc.nasa.gov >Precedence: bulk > >I've been struggling with the space vs. price issue myself. The other >problem I've found is rack mountable cases. 4x-5x the price of a >standard ATX minitower, easily. It would almost double the price of >each node for me to look at anything other than standard cases on >shelves. Has anyone else found 2U or even 4U rack ATX cases for under >US$300? Also, when looking at one PS per node, I worry both with the >cost of excess power consumption and excess ambient heat (requiring >heavier HVAC load to cool the console room, thus adding further to the >electrical bill). Using 120W PS units would take care of most of the >excess electrical consumption (in addition to being cheaper than 250W >models), but only half-cures the heat problem... Suggestions? Similar >experiences? TIA. >-- > Jeremy Stanley Trend CMHS >IS Admin/Senior Tech http://www.trendcmhs.org > > The opinions expressed herein do not necessarily >represent those of Trend CMHS or Trend Foundation. > >> ---------- >> From: Jessen, Per[SMTP:JesseP@europe.stortek.com] >> Sent: Monday, January 25, 1999 11:55 AM >> To: 'paul.oertel@citicorp.com' >> Cc: 'beowulf'; 'extremelinux' >> Subject: RE: Beoboard idea >> >> > -----Original Message----- >> > From: Paul Oertel [mailto:paul.oertel@citicorp.com] >> > Sent: 25 January 1999 12:06 >> [snip] >> > I don't see why it would work but then I'm not too strung in the >> > electric wiring department. I've read that you need at least 120 >> watts >> > (have I got my units right?) for the motherboard and an >> > average 12 watts >> > per each additional device (graphics card, hard drive etc.) For me >> > space is also a problem. I have been thinking if there is some way >> to >> > combine more than one motherboard to per box and share power supply >> > there would be a significant space savings. Maybe other >> componenents >> > (mice, keyboard) could be shared as well. So long as the box was >> > properly grounded, I think the box needn't be made of metal >> > to function properly. >> >> There are two ways of looking at this: optimum space, optimum price. >> >> Space: if space is at a premium, yes, looking at supplying several >> motherboards/PCs from one (large) switching supply is certainly >> possible. There are companies that build e.g. 600W and 800W >> switchmode >> supplies, but that's about the biggest I've seen. And for a >> reasonably >> size cluster you're probably looking at e.g. 1200W or more. Depends >> on your config. And in that size you could be looking at a custom >> build. Which doesn't exactly do wonders for the price. >> >> Price: if you've got space, but not the money, I don't think one >> supply >> for an entire cluster is optimal. Even standard 600/800W supplies >> do not 'scale' in price. 3 standard 250W supplies are always cheaper >> than one 600W supply. >> >> As far as determining the size of your supply, you should be able to >> find your motherboards consumption in your manual, and the same goes >> for your harddisks. For other cards, you'll probably have to make >> some >> educated guessing. >> >> Anyway, I'm looking at doing exactly the same thing, and I have been >> out looking for large power-supplies - so, if you find something >> that >> is economically viable when compared to an array of COTS 250W >> supplies, >> let me know. >> >> >> Best regards, >> Per Jessen >> Software Support Engineer, StorageTek International Software Support >> mailto://per_jessen@storagetek.com, http://www.storagetek.com >> > ______________________________________________________ Get Your Private, Free Email at http://www.hotmail.com From andy@meshplc.co.uk Tue, 26 Jan 1999 13:05:34 -0500 Date: Tue, 26 Jan 1999 13:05:34 -0500 From: Andy andy@meshplc.co.uk Subject: Mobile beowulf I wonder if it would be possible to mount a 6-foor rack in gimballs? A bit dangerous in high seas maybe :) Andy MESH Computers Webmaster webmaster@meshplc.co.uk From xpertz@hotmail.com Tue, 26 Jan 1999 13:43:13 -0500 Date: Tue, 26 Jan 1999 13:43:13 -0500 From: ex pert xpertz@hotmail.com Subject: Newbie Please bear with me as I'm new to both linux and beowulf. I have purchased 8 dec alpha's for use in the cluster (21164pc-533 w/ sx mobo). Are there any resources that I should look at? (ie. faq's, step by step instructions, etc...) Also I have red hat linux ver 5.1. I know that 5.2 is out, but is there any reason for me to upgrade? I am really looking forward to supercomputing, and helping to grow this community. Thanks, Joe Kelly ______________________________________________________ Get Your Private, Free Email at http://www.hotmail.com From mack@nesc.epa.gov Tue, 26 Jan 1999 14:30:29 -0500 Date: Tue, 26 Jan 1999 14:30:29 -0500 From: Joseph Mack mack@nesc.epa.gov Subject: networking data, global /proc space "Andreas C. Doering" wrote: > > > As far as I know you can not build a NUMA machine across the > PCI bus. Thus we need better interfaces to fast networks. > I'm trying to track down whether this is true or not. Data General are shipping file servers with quad Xeon processors using SCI to do what looks like NUMA. http://www.dg.com/numaliine/ I can't tell if it's PCI or not. Do you have anymore information? Thanks Joe -- Joseph Mack, Senior Systems Engineer, Lockheed Martin National Environmental Supercomputer Center (NESC), mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA From thorpe@cerco.ups-tlse.fr Tue, 26 Jan 1999 15:27:18 -0500 Date: Tue, 26 Jan 1999 15:27:18 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea >Kragen Wrote: >> If you're using a standard Pentium II motherboard with four free PCI slots, >> you should be able to get 32 processors in. > >I haven't seen a lot of mobos with 4 PCI slots, but I haven't looked in >a year or so. Actually there are some with 5 PCI slots. Mind you, you might well want to keep one slot free for a nice fast ethernet board. Graphics is no problem now that there's a separate AGP graphics slot. >And I thought the design had been revised to only have >six SAs per PCI card instead of 8? Not sure - I suspect that it's really a question of how to replace the 3-way PCI bridge from HiNT (www.hint.com) that was originally planned. Unfortunately, that particular component won't be available for another few months, so there will have to be an alternative interim solution... I think Simtec are working on that one right now.... >And sebringring.com apparently has plans for much bigger PCI >backplanes. Absolutely. Their web page (www.sebringring.com) has me drooling... >> So a PCI backplane with 128 CPUs could be booted in under a second). > >Well, more than likely, it'll take under a second to get Linux to start >booting on all those CPUs; it might take half a minute to finish >booting. You're of course right that it will presumably take some time for the kernel to sort itself out, but the fact is that this will all be happening in parallel, so that the total boot time would still only be the value for a single CPU + 1 second. That doesn't sound too bad ;-) Cheers Simon From thorpe@cerco.ups-tlse.fr Tue, 26 Jan 1999 15:26:22 -0500 Date: Tue, 26 Jan 1999 15:26:22 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea >Mark van Doesburg wrote: >> Douglas Eadline wrote: >> >> And what is the limit of CPUs you can put in the machine. This >> will limit problem size (scalability). > >The number of CPU's you can put into the machine is limited by the 4GB >PCI memory space. For performance reasons I map all memory of all SA's >into the PCI memory space, this limits the number of CPU's to 64 if you >use 64MB of RAM per CPU. Obviously the host system needs some of the >space as well so i guess the limit will be 56 CPU's. Multiple hosts with >56 SA's can be networked together of course. > >greetings, > >Mark. Just in case there are people out there who aren't sure, Mark's the guy who knows how to boot Linux on the StrongARM over a PCI bus, and he knows how to implement IP messaging at the same time. With a bit of luck he will now have received one (and I rather hope 2) of these precious little boards and may be in the process of trying it all out - how's it coming along Mark? Simon From thorpe@cerco.ups-tlse.fr Tue, 26 Jan 1999 15:27:07 -0500 Date: Tue, 26 Jan 1999 15:27:07 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea >Joe Mack wrote >> >> This sounds much like a product that corel has. It is about the size of >> one of the nutshell(O'Reilly) books(How is that for accuracy). >> >> For those interested take a look at: >> http://www.corelcomputer.com/products/index.htm >> >> I thought of these earlier but did not mention them on this list because >> they do not have hardware floating point. But from the point of view of a >> large number of small computers that will fit in a tiny space these are >> great and availble. > >There's also some problem, which I don't exactly remember, >but I think it's something like the Linux kernel being >booted from ROM (and hence you can't change it) > >Joe With the sa-beowulf system, the host will have a copy of the Linux kernel that gets copied to the CPU modules at boot time. No problem whatsover if you want to change the kernel... Simon From dave@loki.sbay.org Tue, 26 Jan 1999 16:01:02 -0500 Date: Tue, 26 Jan 1999 16:01:02 -0500 From: Dave Zarzycki dave@loki.sbay.org Subject: Beoboard idea On Tue, 26 Jan 1999, Kragen Sitaker wrote: > On Tue, 26 Jan 1999, Simon Thorpe wrote: > > If you're using a standard Pentium II motherboard with four free PCI slots, > > you should be able to get 32 processors in. > > I haven't seen a lot of mobos with 4 PCI slots, but I haven't looked in > a year or so. And I thought the design had been revised to only have > six SAs per PCI card instead of 8? Well, while you haven't been motherboard shopping, 4 PCI slot + 1 AGP slot motherboards have become common. 5 PCI slots + 1 AGP can be difficult to find and your local computer reseller. ;-) davez From NewD@esi.com Tue, 26 Jan 1999 17:41:24 -0500 Date: Tue, 26 Jan 1999 17:41:24 -0500 From: Dave New NewD@esi.com Subject: networking data, global /proc space > -----Original Message----- > From: Joseph Mack [SMTP:mack@nesc.epa.gov] > Sent: Tuesday, January 26, 1999 2:29 PM > To: Andreas C. Doering > Cc: beowulf@beowulf.gsfc.nasa.gov > Subject: Re: networking data, global /proc space > > "Andreas C. Doering" wrote: > > > > > > As far as I know you can not build a NUMA machine across the > > PCI bus. Thus we need better interfaces to fast networks. > > > > I'm trying to track down whether this is true or not. Data General > are shipping file servers with quad Xeon processors using SCI > to do what looks like NUMA. > > http://www.dg.com/numaliine/ > > I can't tell if it's PCI or not. Do you have anymore information? [Dave New] IIRC, DG had a special ECL chip that sat on the local bus or their own processor and looked like shared L3 coherent cache amongst the local and remote processors. In the past, a company needed to license and pay royalties to Intel for the privelage of designing anything that sat on a PPro local bus. I believe that that is still the case, although DG might had paid the fees and adopted their design for the Intel bus(es). IBM was also spinning up an ECL interface part, but I'm not sure what processor bus it was meant for. This is essentially what is required to do 'transparent' SCI-based NUMA. A PCI adapter solution doesn't support processor-local cache across the PCI bus, nor virtual circuits, which the Intel processors need to achieve symmetrical Intel MP v1.4 interrupt processing. SCI can be used to transport the MP 1.4-required three-wire APIC protocol across box boundaries. > Thanks Joe > > -- > Joseph Mack, Senior Systems Engineer, Lockheed Martin > National Environmental Supercomputer Center (NESC), > mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA [Dave New] Cheers, -- DaveN =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- =-= Dave New, NewD@elcsci.com | Neutrinos have mass? ESI Vision Products Division | I didn't know they were Catholic... 3980 Ranchero Drive | Ann Arbor, MI 48108 | Opinions expressed are mine. | PGP 2.6 (734) 332-7010 VOX | 08 12 9F AF 5B 3E B2 9B 6F DC 66 5A 41 0B AB 29 (734) 332-7077 FAX From sandy.harris@sympatico.ca Tue, 26 Jan 1999 18:32:01 -0500 Date: Tue, 26 Jan 1999 18:32:01 -0500 From: Sandy Harris sandy.harris@sympatico.ca Subject: Beoboard idea Simon Thorpe wrote: > > >I haven't seen a lot of mobos with 4 PCI slots, but I haven't looked in > >a year or so. Tyan have 9-slot super 7 board, AGP & 4 each for PCI & ISA, though 1's shared. > Actually there are some with 5 PCI slots. Sun have an OEM UltraSPARC board, likely gawdoffal expensive but 400 MHz 64-bit CPU, space for a gig of RAM, on-board ultraSCSI & 100baseT, ATX form factor, six PCI slots. Wonder if it runs UltaPenguin 64-bit Linux. Tyan's dual Xeon board also has onboard SCSI & net and six PCI. Plus I think one each AGP & ISA. -- "The real aim of current [cryptography] policy is to ensure the continued effectiveness of US information warfare assets against individuals, businesses and governments in Europe and elsewhere" Ross Anderson, Cambridge University From m.j.s.vandoesburg@student.utwente.nl Tue, 26 Jan 1999 19:05:50 -0500 Date: Tue, 26 Jan 1999 19:05:50 -0500 From: Mark van Doesburg m.j.s.vandoesburg@student.utwente.nl Subject: Beoboard idea Just in case there are people out there who aren't sure, Mark's the guy who knows how to boot Linux on the StrongARM over a PCI bus, and he knows how to implement IP messaging at the same time. With a bit of luck he will now have received one (and I rather hope 2) of these precious little boards and may be in the process of trying it all out - how's it coming along Mark? No boards yet. If I receive a single board I will be done with it very quickly since the HOST-SA communications already work and the only thing I will have to adapt is the boot ROM. The only thing I really have to test is the SA-SA communications, this is fully implemented and completely untested :-) Mark. From gerry@cs.tamu.edu Tue, 26 Jan 1999 21:59:13 -0500 Date: Tue, 26 Jan 1999 21:59:13 -0500 From: Gerry Creager gerry@cs.tamu.edu Subject: Mobile beowulf Andy wrote: > > I wonder if it would be possible to mount a 6-foor rack in gimballs? > A bit dangerous in high seas maybe :) > > Andy > MESH Computers Webmaster > webmaster@meshplc.co.uk Yes, it is... and there are specific shipboard mounts for things such as this. John's in a pretty good spot to get help on these things. Gerry From thorpe@cerco.ups-tlse.fr Wed, 27 Jan 1999 03:03:30 -0500 Date: Wed, 27 Jan 1999 03:03:30 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea > Douglas Eadline wrote: > > And what is the limit of CPUs you can put in the machine. This > will limit problem size (scalability). > >The number of CPU's you can put into the machine is limited by the 4GB >PCI memory space. For performance reasons I map all memory of all SA's >into the PCI memory space, this limits the number of CPU's to 64 if you >use 64MB of RAM per CPU. Obviously the host system needs some of the >space as well so i guess the limit will be 56 CPU's. Multiple hosts with >56 SA's can be networked together of course. > >greetings, > >Mark. A couple of comments. One is that Simtec will also offer CPU modules with 32 Megs. In this case, you presumably would be able to get 128 modules in the 4 Gigbyte memory space - let's subtract 512 megs for the host, which leaves120 modules. The other is that I believe that by configuring the 21285 PCI Bridge you can declare parts of the memory as local, and other parts as public. If so, it might be possible to have, say, 32 Megs of local memory and 32 Megs available for shared memory per processor. Again, you'd be able to get back up to 120 CPUs... Course, all this is hypothetical - I haven't got the cash. And the question of whether it's worth it depends on how well the thing works.But it's nice to think about it anyway.. Actually, if the Sebring ring idea works, you could imagine a board with 2 sebring ring chips, each driving 4 CPU modules, and that these boards could be linked up in a ring with Gigabytes/second of communications bandwidth. In that case you could really take advantage of those 4 Gigabytes of PCI memory addressing.... :-) Cheers Simon __________________ Simon Thorpe Centre de Recherche Cerveau et Cognition 133, route de Narbonne 31062 Toulouse France Tel 33 (0)5 62 17 28 03. Fax 33 (0)5 62 17 28 09 http://www.cerco.ups-tlse.fr/~modele/index.html __________________ From thorpe@cerco.ups-tlse.fr Wed, 27 Jan 1999 03:23:02 -0500 Date: Wed, 27 Jan 1999 03:23:02 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea >Sandy Harris Wrote: >> >> >I haven't seen a lot of mobos with 4 PCI slots, but I haven't looked in >> >a year or so. > >Tyan have 9-slot super 7 board, AGP & 4 each for PCI & ISA, though 1's >shared. Unfortunately, I don't think the ISA slots will be much use. Can't see Simtec rushing to design a multiprocessor carrier board to plug in there. >> Actually there are some with 5 PCI slots. > >Sun have an OEM UltraSPARC board, likely gawdoffal expensive but 400 MHz >64-bit CPU, space for a gig of RAM, on-board ultraSCSI & 100baseT, ATX >form factor, six PCI slots. Wonder if it runs UltaPenguin 64-bit Linux. > >Tyan's dual Xeon board also has onboard SCSI & net and six PCI. Plus >I think one each AGP & ISA. Don't forget that you can get Alpha-based boards with 6 PCI slots (1 64 bit, and 5 32 bit ones) e.g. http://www.alpha-processor.com/ And if you really want an impressive host, how about the new 21264 based board from Digital (and being used by quite a few OEMs) http://www.digital.com/alphaoem/dsc-pc264dp.html It's got two 500 MHz 21264s, each with 4 Megs of L2 cache and 6 * 64 bit PCI slots. OK, I'll have to convinve Simtec to produce a carrier board with 64 bit PCI, but well..... you never know.... Simon __________________ Simon Thorpe Centre de Recherche Cerveau et Cognition 133, route de Narbonne 31062 Toulouse France Tel 33 (0)5 62 17 28 03. Fax 33 (0)5 62 17 28 09 http://www.cerco.ups-tlse.fr/~modele/index.html __________________ From JesseP@europe.stortek.com Wed, 27 Jan 1999 04:07:53 -0500 Date: Wed, 27 Jan 1999 04:07:53 -0500 From: Jessen, Per JesseP@europe.stortek.com Subject: switchmode vs. linear power-supplies (was: RE: Beoboard idea) > -----Original Message----- > From: ex pert [mailto:xpertz@hotmail.com] > Sent: 27 January 1999 05:02 > To: beowulf@beowulf.gsfc.nasa.gov > Subject: RE: Beoboard idea > > > I believe that switching power supplies, only use what power > they need. > In other words if you have a 250w power supply and only 100w > of usage, > the power supply will only use the 100w not the max of 250w. > With a switchmode supply you get >90% efficiency; i.e. of all the power you feed into it, you get most of it out again - there is little wasted in heat. With a linear supply, the voltage is reduced with a transformer and then regulated. This produce a lot of heat, and is less efficient (<50% I believe). Best regards, Per Jessen Software Support Engineer, StorageTek International Software Support mailto://per_jessen@storagetek.com, http://www.storagetek.com From JesseP@europe.stortek.com Wed, 27 Jan 1999 04:16:47 -0500 Date: Wed, 27 Jan 1999 04:16:47 -0500 From: Jessen, Per JesseP@europe.stortek.com Subject: Beoboard idea > -----Original Message----- > From: Andy [mailto:andy@meshplc.co.uk] > Sent: 27 January 1999 03:54 > To: Alvin Starr; Simon Thorpe [snip] > However, does it have to be switched mode? There are plenty > of standard > power supplies available at this level with very good > tolerances and ripple > characteristics, designed primarily for radio transmitters > and the like. > These should prove amply capable of running a computer setup. Absolutely right. The problem is their efficiency, and the sizes needed. As an example - take 8 dual PPRO motherboards. The one I'm looking at is the P6NDI from AIR. The specs say: Typical Power Supply +5V +/- 5% 17A +12V +/- 10% 800mA -5V +/- 5% 150mA -12V +/- 10% 100mA That times 8 is : 5V@136A, 12V@6.4A, etcetera. And it's the 5V at 136A that's the killer. Power-wise it's nothing, but a linear powersupply that regulates e.g. 10V down to 5V at that current will produce a lot of heat. You could probably heat your house with that :-) Other problems: Weight (+1Kw transformers are BIG) Size (ditto, plus regulation circuitry, mainly transistors, plus heatsinks) > As an example, I have a 1KW unit somewhere that was > originally used to power > a radar console at a RAF air base, and which, although it has > suffered some > physical damage, has been powering some of my radio gear for > several years without incident. Yeah, sounds familiar - I've got the same sort of unit, used to sit in a PDP11. It usually takes 2 to lift it - the PDP11 rack at least had wheels :-(. Best regards, Per Jessen Software Support Engineer, StorageTek International Software Support mailto://per_jessen@storagetek.com, http://www.storagetek.com From m.j.s.vandoesburg@student.utwente.nl Wed, 27 Jan 1999 05:19:17 -0500 Date: Wed, 27 Jan 1999 05:19:17 -0500 From: Mark van Doesburg m.j.s.vandoesburg@student.utwente.nl Subject: Beoboard idea And if you really want an impressive host, how about the new 21264 based board from Digital (and being used by quite a few OEMs) http://www.digital.com/alphaoem/dsc-pc264dp.html It's got two 500 MHz 21264s, each with 4 Megs of L2 cache and 6 * 64 bit PCI slots. OK, I'll have to convinve Simtec to produce a carrier board with 64 bit PCI, but well..... you never know.... 64-bit addressing would eliminate the scalability problem as well :-) greetings, Mark. From m.j.s.vandoesburg@student.utwente.nl Wed, 27 Jan 1999 05:18:57 -0500 Date: Wed, 27 Jan 1999 05:18:57 -0500 From: Mark van Doesburg m.j.s.vandoesburg@student.utwente.nl Subject: Beoboard idea One is that Simtec will also offer CPU modules with 32 Megs. In this case, you presumably would be able to get 128 modules in the 4 Gigbyte memory space - let's subtract 512 megs for the host, which leaves120 modules. You are completly correct on this one. The other is that I believe that by configuring the 21285 PCI Bridge you can declare parts of the memory as local, and other parts as public. If so, it might be possible to have, say, 32 Megs of local memory and 32 Megs available for shared memory per processor. Again, you'd be able to get back up to 120 CPUs... This is possible but not without a performance penalty. It also conflicts with other plans I have with the system, I want to create shared memory segments distributed on all the processors to eliminate the protocol overhead. greetings, Mark. From gerry@cs.tamu.edu Wed, 27 Jan 1999 06:59:07 -0500 Date: Wed, 27 Jan 1999 06:59:07 -0500 From: Gerry Creager gerry@cs.tamu.edu Subject: Beoboard idea "Jessen, Per" wrote: > > > -----Original Message----- > > From: Andy [mailto:andy@meshplc.co.uk] > > Sent: 27 January 1999 03:54 > > To: Alvin Starr; Simon Thorpe > [snip] > > However, does it have to be switched mode? There are plenty > > of standard > > power supplies available at this level with very good > > tolerances and ripple > > characteristics, designed primarily for radio transmitters > > and the like. > > These should prove amply capable of running a computer setup. > > Absolutely right. The problem is their efficiency, and the sizes > needed. As an example - take 8 dual PPRO motherboards. The one > I'm looking at is the P6NDI from AIR. The specs say: ... > And it's the 5V at 136A that's the killer. Power-wise it's nothing, but > a linear powersupply that regulates e.g. 10V down to 5V at that current > will produce a lot of heat. You could probably heat your house with that > :-) While admittedly the Texas winters are somewhat mild, I *DO* heat my workshop with an older spectrum analyzer with a linear power supply! > Other problems: > > Weight (+1Kw transformers are BIG) > Size (ditto, plus regulation circuitry, mainly transistors, plus > heatsinks) As Per noted earlier, efficiency is the key, although his numbers are a bit optimistic. I've a friend who's an absolute genius at power supply design, and has designed 'em for Compaq, now at Dell. I've had a hand in some design work, and when we updated the power supplies for the Shuttle Amateur Radio Experiment, we went from linear supplies documented at about 65% efficiency to some switchers at about 95%. In "real life" I'm more used to the 80-85% range, but as technology has improved, so have the switching supplies. As an aside, early in the design work for Space Station Freedom, back 10 or so years, we had an innovative design for power distribution, where solar energy was collected and managed as direct current to the batteries, then was transformed using switching power supplies to the 100 khz range for transmission about the Station. There are a number of safety and efficiency benefits to this design, but it was a bit radical for some of the managers. In order to "test" whether it would work, they ordered some of the engineering team to hook up some of the Vaxes to a testbed power grid operating in this mode, and see what happened. Not designed for that frequency, the power supplies in the Vaxen were not happy, and they transmitted their displeasure throughout the cabinets. Resulted in some high degree of damage and led more to salvage than repair. No degree of explanation by any of the power engineers involved could convince the marginally competent (from an engineering standpoint) managers that the management "test" had been flawed because not all power supplies were created equally! Gerry Creager Mapping Sciences Laboratory Texas A&M University From ferguson.joseph@nesc.epa.gov Wed, 27 Jan 1999 07:08:55 -0500 Date: Wed, 27 Jan 1999 07:08:55 -0500 From: Joseph M. Ferguson ferguson.joseph@nesc.epa.gov Subject: Beoboard idea Mark van Doesburg wrote: > > Douglas Eadline wrote: > > And what is the limit of CPUs you can put in the machine. This > will limit problem size (scalability). > > The number of CPU's you can put into the machine is limited by the 4GB > PCI memory space. For performance reasons I map all memory of all SA's > into the PCI memory space, this limits the number of CPU's to 64 if you > use 64MB of RAM per CPU. Obviously the host system needs some of the > space as well so i guess the limit will be 56 CPU's. Multiple hosts with > 56 SA's can be networked together of course. > > greetings, > > Mark. There is info on the Intel web site about the "profusion" chip set which supports a 36-bit address space. This could change the arithmetic used above. NOTE: Some assembly required ;) -- Joe Ferguson ferguson.joseph@nesc.epa.gov Lockheed Martin Corp. NESC Support, EPA-RTP -- NC Voice: (919)-541-3716 From enano@ceu.fi.udc.es Wed, 27 Jan 1999 07:20:26 -0500 Date: Wed, 27 Jan 1999 07:20:26 -0500 From: Miguel Barreiro Paz enano@ceu.fi.udc.es Subject: Beoboard idea > >It's got two 500 MHz 21264s, each with 4 Megs of L2 cache and 6 > >* 64 bit PCI slots. OK, I'll have to convinve Simtec to produce > >a carrier board with 64 bit PCI, but well..... you never > >know.... AFAIK you can plug a 32-bit PCI board on a 64-bit slot. Am I just plain wrong? > 64-bit addressing would eliminate the scalability problem as we Yes, for the time being... :-) However, even for say 54 SA-110 (9 full carrier boards) a non-switched PCI backplane would become a bottleneck for very communication-intensive codes, I suppose. Anyway - wouldn't we need 64-bit ARMs also to get in the truckload-range number or CPUs? Regards, From ferguson.joseph@nesc.epa.gov Wed, 27 Jan 1999 07:23:53 -0500 Date: Wed, 27 Jan 1999 07:23:53 -0500 From: Joseph M. Ferguson ferguson.joseph@nesc.epa.gov Subject: Beoboard idea Simon Thorpe wrote: > > >Kragen Wrote: > > >> If you're using a standard Pentium II motherboard with four free PCI slots, > >> you should be able to get 32 processors in. > > > >I haven't seen a lot of mobos with 4 PCI slots, but I haven't looked in > >a year or so. > > Actually there are some with 5 PCI slots. Mind you, you might well want to > keep one slot free for a nice fast ethernet board. Graphics is no problem > now that there's a separate AGP graphics slot. > > >And I thought the design had been revised to only have > >six SAs per PCI card instead of 8? > > Not sure - I suspect that it's really a question of how to replace the > 3-way PCI bridge from HiNT (www.hint.com) that was originally planned. > Unfortunately, that particular component won't be available for another few > months, so there will have to be an alternative interim solution... I think > Simtec are working on that one right now.... > > >And sebringring.com apparently has plans for much bigger PCI > >backplanes. > > Absolutely. Their web page (www.sebringring.com) has me drooling... > > >> So a PCI backplane with 128 CPUs could be booted in under a second). > > > >Well, more than likely, it'll take under a second to get Linux to start > >booting on all those CPUs; it might take half a minute to finish > >booting. > > You're of course right that it will presumably take some time for the > kernel to sort itself out, but the fact is that this will all be happening > in parallel, so that the total boot time would still only be the value for > a single CPU + 1 second. That doesn't sound too bad ;-) > Cheers > > Simon Check the "Industrial PC" suppliers. You'll find lots of interesting combinations: Many PCI slots, I seem to recall as many as 14 (using PCI-PCI bridges, of course) and segmented passive backplanes with up to four or five independent systems, all designed to mount in one 20-slot rack-mount housing. Many of these suppliers also have "flash" technology devices that act like fast disks and can be used for booting, compiled modules, etc.: anything that is reasonably stable over time. -- Joe Ferguson ferguson.joseph@nesc.epa.gov Lockheed Martin Corp. NESC Support, EPA-RTP -- NC Voice: (919)-541-3716 From thorpe@cerco.ups-tlse.fr Wed, 27 Jan 1999 07:52:25 -0500 Date: Wed, 27 Jan 1999 07:52:25 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea >Miguel wrote: > >It's got two 500 MHz 21264s, each with 4 Megs of L2 cache and 6 > >* 64 bit PCI slots. OK, I'll have to convinve Simtec to produce > >a carrier board with 64 bit PCI, but well..... you never > >know.... > AFAIK you can plug a 32-bit PCI board on a 64-bit slot. Am I just >plain wrong? You certainly can in the new G3 Macs, which come with 3 free 64-bit 33 MHz PCI slots. I gather you can use any 32 bit card with no problems. It's just that it would be a shame not to take advantage of the extra bandwidth. After all, Simtec is currently planning to throw in the carrier board if you buy a fully loaded one. And changing the carrier board shouldn't be too difficult, since it only has a couple of PCI bridges and lots of SO-DIMM connectors on it - at least that's how I imagine it. Actually, it shouldn't be too long before 64bit 66 MHz PCI becomes available, and when it does, I'll just have to take out all my lovely CPU modules and put them in a new fancier carrier board... :-) >> 64-bit addressing would eliminate the scalability problem as we > > Yes, for the time being... :-) However, even for say 54 SA-110 (9 >full carrier boards) a non-switched PCI backplane would become a >bottleneck for very communication-intensive codes, I suppose. > > Anyway - wouldn't we need 64-bit ARMs also to get in the >truckload-range number or CPUs? > The point is though, that as long as local memory requirements don't go over 4 Gigabytes, you can always link up your motherboards using "standard" beowulf connections - i.e. Fast or Gigabit ethernet. To put it another way, if someone's already got say 16 Pentium II motherboards set up as a beowulf cluster and they want more processing power, they will have a choice between either (i) adding a whole pile of extra Pentium IIs (or alphas, or whatever) and (ii) putting StrongARM based multiprocessor boards in the existing machines. I think I know what I'd rather do (but there again, I don't need floating point). Cheers Simon __________________ Simon Thorpe Centre de Recherche Cerveau et Cognition 133, route de Narbonne 31062 Toulouse France Tel 33 (0)5 62 17 28 03. Fax 33 (0)5 62 17 28 09 http://www.cerco.ups-tlse.fr/~modele/index.html __________________ From ferguson.joseph@nesc.epa.gov Wed, 27 Jan 1999 09:19:29 -0500 Date: Wed, 27 Jan 1999 09:19:29 -0500 From: Joseph M. Ferguson ferguson.joseph@nesc.epa.gov Subject: Beoboard idea Dave Hart wrote: > > At 07:23 AM 1/27/1999 -0500, you wrote: > > >Check the "Industrial PC" suppliers. > > Any names? recommendations?? > > -- > David Hart http://php.indiana.edu/~dhart > Research Computing Support 812-855-2632 > University Information Technology Services Indiana University David: Try an Altavista search on "industrial computer*"; you'll get lots of hits with some fertile ground near the head of the list. -- Joe Ferguson ferguson.joseph@nesc.epa.gov Lockheed Martin Corp. NESC Support, EPA-RTP -- NC Voice: (919)-541-3716 From mack@nesc.epa.gov Wed, 27 Jan 1999 09:42:10 -0500 Date: Wed, 27 Jan 1999 09:42:10 -0500 From: Joseph Mack mack@nesc.epa.gov Subject: networking data, global /proc space Dave New wrote: > > > [Dave New] IIRC, DG had a special ECL chip as in emitter coupled logic (fast and lots of power, ie hot)? > that sat on the local bus or their own > processor and looked like shared L3 coherent > cache amongst the local and remote processors. > didn't know any of this, thanks > This is essentially what is required to do > 'transparent' SCI-based NUMA. A PCI adapter > solution doesn't support processor-local cache > across the PCI bus, nor virtual circuits, which > the Intel processors need to achieve symmetrical > Intel MP v1.4 interrupt processing. SCI can be > used to transport the MP 1.4-required three-wire > APIC protocol across box boundaries. transporting the APIC protocol... is this after you have the ECL bridge, or can you do this independantly of the ECL bridge? Thanks Joe -- Joseph Mack, Senior Systems Engineer, Lockheed Martin National Environmental Supercomputer Center (NESC), mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA From NewD@esi.com Wed, 27 Jan 1999 10:19:12 -0500 Date: Wed, 27 Jan 1999 10:19:12 -0500 From: Dave New NewD@esi.com Subject: networking data, global /proc space > -----Original Message----- > From: Joseph Mack [SMTP:mack@nesc.epa.gov] > Sent: Wednesday, January 27, 1999 9:40 AM > To: Dave New > Cc: Andreas C. Doering; beowulf@beowulf.gsfc.nasa.gov > Subject: Re: networking data, global /proc space > > Dave New wrote: > > > > > > [Dave New] IIRC, DG had a special ECL chip > > as in emitter coupled logic (fast and lots of power, ie hot)? [Dave New] I'm afraid so. It's one of the ways that companies are able to achieve the switching speeds they need to run SCI train protocols at 1 GByte/sec. The trick, of course, is to interface the ECL circuitry to the 'normal' TTL-like levels used in the rest of the computer. This is still being tinkered with, I'd say. Eventually, someone will come out with a commodity-priced chip that just drops onto a processor's local bus, that provides the local copy of the L3 cache, as well as the level-shifting and driving circuits for the networking fabric. It needs a market with sufficient volume with that kind of need (less than 1 microsecond cache coherency latency for a 'miss') to make it happen. > > that sat on the local bus or their own > > processor and looked like shared L3 coherent > > cache amongst the local and remote processors. > > > > > didn't know any of this, thanks > > > This is essentially what is required to do > > 'transparent' SCI-based NUMA. A PCI adapter > > solution doesn't support processor-local cache > > across the PCI bus, nor virtual circuits, which > > the Intel processors need to achieve symmetrical > > Intel MP v1.4 interrupt processing. SCI can be > > used to transport the MP 1.4-required three-wire > > APIC protocol across box boundaries. > > transporting the APIC protocol... is this after you have the > ECL bridge, or can you do this independantly of the ECL bridge? [Dave New] The idea is that SCI can act as a transport or wrapper for many different other protocols. Besides the NUMA coherent cache protocol, you can establish IP links over an SCI fabric, or tunnel other protocols through it, like the Intel APIC protocol. Software protocol stacks, like IP, can effectively be tunneled through SCI adapter boards like the Dolphin PCI interconnect; you only need to write the appropriate media layer hooks to use an SCI fabric instead of Ethernet, for instance. A hardware protocol though, like APIC (or L3 NUMA coherent cache), is a different kind of animal. You need direct hardware support attached at the right spot in the system that can convert the hardware protocol into something that can tunneled across the SCI fabric. This means getting real cozy with the processor's local bus (typically) to implement things like NUMA on SCI, and a semi-intelligent inteface chip to do the hardware protocol to SCI conversion. > Thanks Joe > > -- > Joseph Mack, Senior Systems Engineer, Lockheed Martin > National Environmental Supercomputer Center (NESC), > mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA [Dave New] Cheers, -- DaveN From mack@nesc.epa.gov Wed, 27 Jan 1999 11:57:41 -0500 Date: Wed, 27 Jan 1999 11:57:41 -0500 From: Joseph Mack mack@nesc.epa.gov Subject: NUMA beowulf (Was: networking data, global /proc space) Dave New wrote: > > > as in emitter coupled logic (fast and lots of power, ie hot)? > [Dave New] I'm afraid so. It's one of the OK > fabric. It needs a market with sufficient > volume with that kind of need (less than > 1 microsecond cache coherency latency for > a 'miss') to make it happen. > [Dave New] The idea is that SCI can act as > a transport or wrapper for many different > other protocols. Besides the NUMA coherent OK got that > A hardware protocol though, like APIC > (or L3 NUMA coherent cache), is a > different kind of animal. You need > direct hardware support attached at > the right spot in the system that can > convert the hardware protocol into something > that can tunneled across the SCI fabric. > > This means getting real cozy with the > processor's local bus (typically) to > implement things like NUMA on SCI, and > a semi-intelligent inteface chip to do the > hardware protocol to SCI conversion. Having looked at DG's stuff http://www.dg.com/aviion/html/av_25000_enterprise_server.html (see 4th para in section called "Successive Generations of ccNUMA servers" where they have a chip sitting on the memory bus making 2 dual Xeons appear as a quad processor) and http://www.dg.com/about/html/av25000_foundation.html where DG appears to have rewritten the OS to run a transparently NUMA machine, I'm wondering how close we are to a commodity NUMA machine (they say they have only 1 copy of the OS running for all the nodes). Is this machine truly cache coherent, have they got the OS all threaded up so that all the nodes appear as one machine or are they message passing as well (I don't know and can't tell from the web pages)? So then I go to Scali at http://www.scali.com/ who are using SCI connects and message passing on Linux/Intel or Solaris/Sparc to make what looks like a Beowulf with low latency (I assume 1usec). This isn't a cache coherent NUMA machine so we aren't there yet, but it doesn't seem all that far away. I haven't found out yet whether Scali is adding anything on top of the linux SCI drivers from Dolphinics http://www.dolphinics.no/ to make your life easier or whether you can do the whole thing yourself using the Dolphinics code. Joe -- Joseph Mack, Senior Systems Engineer, Lockheed Martin National Environmental Supercomputer Center (NESC), mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA From deadline@plogic.com Wed, 27 Jan 1999 13:01:11 -0500 Date: Wed, 27 Jan 1999 13:01:11 -0500 From: Douglas Eadline deadline@plogic.com Subject: NUMA beowulf (Was: networking data, global /proc space) > > > where DG appears to have rewritten the OS to run a transparently > NUMA machine, I'm wondering how close we are to a commodity NUMA > machine (they say they have only 1 copy of the OS Not very. The point about Beowulf is that many of the parts are commodity. I doubt these will every be sold in large numbers or at lest enough to make them a resonablepercent of the overal cost. > running for all the nodes). Is this machine truly > cache coherent, have they got the OS all threaded up so > that all the nodes appear as one machine or are they message > passing as well (I don't know and can't tell from the web pages)? > > So then I go to Scali at > > http://www.scali.com/ > > who are using SCI connects and message passing on Linux/Intel > or Solaris/Sparc to make what looks like a Beowulf with low > latency (I assume 1usec). This isn't a cache coherent NUMA machine > so we aren't there yet, but it doesn't seem all that far away. > I haven't found out yet whether Scali > is adding anything on top of the linux SCI drivers from Dolphinics > > http://www.dolphinics.no/ > > to make your life easier or whether you can do the whole thing yourself > using the Dolphinics code. One thing we have found it that while NUMA allows a shared memory paradigm for the user (easier programming) optimization is not so easy. A good NUMA compiler must know where to place the data optimally so as to avoid expensive cache misses. Message passing on the other hand, can be optimized a bit easier (relatively speaking as both problems are hard) The HP/Convex machines are good examples of SCI/NUMA. Doug ------------------------------------------------------------------- Paralogic, Inc. | PEAK | Voice:+610.861.6960 115 Research Drive | PARALLEL | Fax:+610.861.8247 Bethlehem, PA 18017 USA | PERFORMANCE | http://www.plogic.com ------------------------------------------------------------------- From NewD@esi.com Wed, 27 Jan 1999 13:21:10 -0500 Date: Wed, 27 Jan 1999 13:21:10 -0500 From: Dave New NewD@esi.com Subject: Beoboard idea > Check the "Industrial PC" suppliers. You'll find lots > of interesting combinations: Many PCI slots, I seem to recall as many as > 14 (using PCI-PCI bridges, of course) and segmented passive backplanes > with up to four or five independent systems, all designed to mount in > one 20-slot rack-mount housing. Many of these suppliers also have > "flash" technology devices > that act like fast disks and can be used for booting, > compiled modules, etc.: anything that is reasonably > stable over time. [Dave New] Been there, done that, and got the scars to prove it... I've listened to this thread for sometime, and several times thought about saying something to those that are pushing things with lots o' PCI slots, and making claims for 132 MBytes/sec speeds, and also, I think for what sounds like fondly hoping for PCI 'broadcast' capabilities. Sad to say, PCI is not good for any of those claims: 1. The PCI electrical (and timing) specification is very strict, and allows (among other things) for only 10 loads, and the card-edge connectors themselves count for one of those loads. Therefore, a motherboard with four slots (8 loads) is pushing it; it only leaves two effective loads for the motherboard chipset to use up, usually the host bridge and an ISA bridge. Also, the trace lengths for PCI signals are strictly controlled, to maintain correct clock skew and signal levels. A motherboard with five or more PCI slots requires a bridge. Bridge support amongst some popular operating systems is poor, at best (some bridges are also poor excuses for a bridge, as well). The 'industrial' PC's you see advertised are an attempt to appease those that want to use cheap desktop-style cards in a rugged case, possibly with lots o' PCI slots. The PICMG 'standard' that most of these systems use essentially ignore the PCI electrical and timing specifications, creating no end of clock skew and other bus problems. The worst offender is the long routing of the PCI signals from the CPU 'bridge' connector near the front of the motherboard, to the first PCI slot contained near the rear. If violates most all of the PCI timing and electrical specifications. The result is an unstable system that crashes mysteriously depending on the exact mix and location of various vendor's boards. Folks that bring PICMG-related problems to the PCI sig maillist are usually ignored, or in persistant cases are told where to get off. PICMG is considered an ill-thought-out 'renegade' spinoff of PCI. If what you want is an 'industrial' PCI solution (to take advantage of fast backplane communications) then you should look closely at Compact PCI (CPCI). Although, CPCI might also be considered a 'renegade' by the PCI purists, they at least, seem to have done their homework. The CPCI group studied the electrical and timing issues of moving to an industrial VME-style cardcage, with gas-tight Euro-style connectors, and exhaustively modeled the signal characteristics. The result was a modification to the PCI electrical specification, adding ballast resistors to the signals where they enter the cards, to impedance match what is essentially a reflective transmission line on the backplane. The CPCI has all this information in a white paper that can be downloaded from their site, and the simulation code is available for those that want to duplicate the results. An example of what can be done on Compact PCI is Corel Computer's 'Zaphod' project, where they have a number of MIPS modules running in a Compact PCI bus, in a standard rack mount. This may not strike you as COTS enough to qualify for a Beowulf label, but if you are interested in stable PCI configurations with large amounts of uptime, I would avoid PICMG stuff. If you would prefer multiple x86 processors on CPCI, the recent availability (finally) of an x86 host bridge that tolerates such an arrangement has resulted in at least one vendor, Ziatech, to offer a CPCI board based on this chipset. I'm sure there will be others. 2. Repeat after me: 132 MBytes/sec is a marketing fantasy. Ain't no such thing. It is a theoretically infinitely long maximum burst rate. There are many reasons why this will never be seen in practice. First of all, most devices that talk on PCI, if they burst at all, burst in cache-line chunks (say, 16, 32, or 64 bytes). Add in the overhead of 5, 6, or 7 PCI bus cycles per burst (depending on whether hidden arbitration is used and whether one or two address cycles are used for 64-bit addressing; and we are assuming that all device decoding is done in fast mode, which is also almost never true if an ISA bridge is in the system doing subtractive decoding), and that 132 MByte/sec throughput starts to drop drastically. Add to that the idea that you usually are trying to get something else useful done in the system, and this means that you might have to wait to get access to the bus in the first place. And the biggest bus hog of all usually turns out to be the CPU itself. In most all Intel host bridge chipsets I've encountered, the PCI bus arbiter is programmed by default to give the CPU consistently the highest priority for all PCI bus transactions. Thus, if you are heavily using the PCI bus from the CPU, you can effectively starve another PCI bus master from getting any significant throughput. The fact that the bus arbiter is executing a fairness algorithm doesn't guarantee what you might consider 'fair' from *your* viewpoint. If the host bridge arbiter algorithm can be modified, this information is usually only revealed to system integrators that sign a non-disclosure agreement to see the chip 'errata'. Without these undocumented register- level changes, vendors like Data Translation were unable to even get reliable 20 MByte/sec video transfers across a Triton chipset bus from a PCI framegrabber (they shipped their boards with a utility program that tinkered with the undocumented register settings in the Triton chipset, to change the priority scheme to round-robin). To top it all off, busmastering PCI devices (when they are trying to write CPU memory, like the above framegrabber) are competing not only for PCI bus access, but also local processor bus access to the main memory, 'behind' the PCI host bridge. Contention with the local CPU's instructions fetches when L2 cache misses occur typically blocks a PCI busmaster from writing to the main memory, unless the host bridge is very smart about cache coherency, etc., etc. The upshot is that earlier Intel chipsets oft times did not support burst writes into main memory across the host bridge (they would force PCI disconnects on every burst attempt, causing horrible throughput), and in more recent chipsets, although things have gotten a lot better, the most you could possibly hope for in a sustained burst might be 80 MBytes/sec, and that's if there is *no* contention for either the PCI bus or the main memory. In other words, don't count on it. A good rule of thumb is to expect only about a quarter of the stated burst rate in a loaded system, a far cry from 132 MBytes/sec. Kind of reminds you of loaded Ethernet, doesn't it? 3. PCI broadcasting, or "why can't I write to multiple targets simultaneously?" Well, you can. There are two ways of doing it, neither of which are supported in any significant way that you could generally make use of. The PCI spec allows for a special cycle type, to 'broadcast' certain system level information to PCI devices. Where this might be used is to send power management directives or other similar info around. I've not encountered places where this has been used, yet, although the new PCI power management spec may make use of it. My impression is that chipset (and software) support of this feature is pretty non-existant. It is also very low bandwidth, (non-burstable) and probably not suitable for what most folks would like to use it for. 'VGA palette snooping' is really a means to have a passive listener on the PCI bus, and have it officially supported by bridge chipsets that may need to pass video palette write cycles downstream, even though an upstream video board may already be decoding the cycles. Usually only a restrictive address range (of interest only to graphic accelerator boards) is passed along. The assumption is the passive listener is at least as fast (always) as the active listener's decoder is. Otherwise, the passive listener may miss some palette information written to the main video board. Delays through bridges make this a dicey proposition, at best. Most systems have propagation of palette snooping disable by default, as almost no one has ever used it. Since PCI is kind of a transaction protocol (I say kind of, because it lacks a complete handshake for certain kinds of bus cycle situations), 'broadcasting' can only be done if a bunch of passive listeners eaves- drop on an active conversation with a particular PCI target. If you have complete control over the PCI interface design of your proposed passive listeners, then you could put together an impressive broadcast setup (as long as you keep in mind the speed requirements, because none of the passive listeners will be able to suspend or force retries on any of the transactions they are eavesdropping on). On the other hand, most designers use a drop-in chipset for PCI interfacing (for a lot of reasons, not the least of which is that designing your own PCI interface is a relatively daunting task; it has fairly strict electrical and timing requirements, and a complex state machine architecture), and there are no available off-the-shelf chipsets (that I'm aware of) that support the kind of passive broadcast listening that some folks might like. So, it is doable within the PCI specification, but it is not required, so it is not implemented. A similar situation exists with bursting to I/O addresses. It is doable, but it is not a requirement, so no one's chipset supports it. It would require a 'private' arrangement between a master and target, and without a standard in place to codify that arrangement, no one will commit a proprietary solution to silicon. Well, I'm sure I've beat these issues to death and then some, but I felt it necessary to stop a lot of what looked like hopefully-optimistic mis-information about PCI from being propagated any further in the Beowulf community. PCI was originally designed for very local, high-speed peripheral arrangements, but it is being contorted into all kinds of other uses, including some that just plain don't work. It is not a 'one size fits all' kind of interface, and no one should put up with the grief caused by mis-guided efforts like the PICMG. Constructive comments are certainly welcome, but flames from PICMG supporters can go to /dev/null... Cheers, -- DaveN =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- =-= Dave New, NewD@elcsci.com | Neutrinos have mass? ESI Vision Products Division | I didn't know they were Catholic... 3980 Ranchero Drive | Ann Arbor, MI 48108 | Opinions expressed are mine. | PGP 2.6 (734) 332-7010 VOX | 08 12 9F AF 5B 3E B2 9B 6F DC 66 5A 41 0B AB 29 (734) 332-7077 FAX > From NewD@esi.com Wed, 27 Jan 1999 14:44:01 -0500 Date: Wed, 27 Jan 1999 14:44:01 -0500 From: Dave New NewD@esi.com Subject: NUMA beowulf (Was: networking data, global /proc space) > -----Original Message----- > From: Joseph Mack [SMTP:mack@nesc.epa.gov] > Sent: Wednesday, January 27, 1999 11:56 AM > To: Dave New > Cc: Andreas C. Doering; beowulf@beowulf.gsfc.nasa.gov > Subject: NUMA beowulf (Was: networking data, global /proc space) > [Dave New] > > A hardware protocol though, like APIC > > (or L3 NUMA coherent cache), is a > > different kind of animal. You need > > direct hardware support attached at > > the right spot in the system that can > > convert the hardware protocol into something > > that can tunneled across the SCI fabric. > > > > This means getting real cozy with the > > processor's local bus (typically) to > > implement things like NUMA on SCI, and > > a semi-intelligent inteface chip to do the > > hardware protocol to SCI conversion. > > Having looked at DG's stuff > > > http://www.dg.com/aviion/html/av_25000_enterprise_server.html > > (see 4th para in section called "Successive Generations of ccNUMA > servers" > where they have a chip sitting on the memory bus making 2 dual Xeons > appear as a quad processor) [Dave New] Yep. That's traditionally the way to do it. > and > > http://www.dg.com/about/html/av25000_foundation.html > > > where DG appears to have rewritten the OS to run a transparently > NUMA machine, I'm wondering how close we are to a commodity NUMA > machine (they say they have only 1 copy of the OS > running for all the nodes). Is this machine truly > cache coherent, have they got the OS all threaded up so > that all the nodes appear as one machine or are they message > passing as well (I don't know and can't tell from the web pages)? [Dave New] I would say that if they say they are running one copy of the OS across all the nodes, then they are running cache coherent NUMA. There's no reason to resort to message passing (for the OS) in that case. By the same token, an SMP-compliant x86 OS (like NT or Linux) could run on such an architecture implemented as NUMA, as long as all memory, I/O and interrupts are symmetrical with respect to the nodes (Intel MB 1.4 requirement). As far as an SMP OS would be concerned it would be running on a single machine with multiple, tightly-coupled processors. SCI's fabric is one enabling technology to make that magic work. > So then I go to Scali at > > http://www.scali.com/ > > who are using SCI connects and message passing on Linux/Intel > or Solaris/Sparc to make what looks like a Beowulf with low > latency (I assume 1usec). This isn't a cache coherent NUMA machine > so we aren't there yet, but it doesn't seem all that far away. > I haven't found out yet whether Scali > is adding anything on top of the linux SCI drivers from Dolphinics > > http://www.dolphinics.no/ > > to make your life easier or whether you can do the whole thing yourself > using the Dolphinics code. [Dave New] Hard to say. There terminology is a bit confusing to the uninitiated. The SMP they keep referring to is really only within a box boundary; everything else is done using MPI between boxes. They claim 6 usec MPI latency. The Dolpin interconnects will not be able to give you true SMP over an entire cluster, because they plug into the PCI bus. You really need a processor bus-local interconnect. Although the PCI bus can support cacheable memory, I've never seen anyone put their main memory across a PCI bus from the processors; it would be *very* slow for single byte/word/doubleword accesses. PCI is a "bursty" bus. Single accesses are penalized heavily by all the transaction overhead. > Joe > -- > Joseph Mack, Senior Systems Engineer, Lockheed Martin > National Environmental Supercomputer Center (NESC), > mailto:mack@nesc.epa.gov ph# 919-541-0007, RTP, NC, USA [Dave New] Cheers, -- DaveN From thorpe@cerco.ups-tlse.fr Thu, 28 Jan 1999 04:02:31 -0500 Date: Thu, 28 Jan 1999 04:02:31 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea > [Dave New] Been there, done that, and got the > scars to prove it... > > I've listened to this thread for > sometime, and several times thought about saying > something to those that are pushing things with > lots o' PCI slots, and making claims for 132 > MBytes/sec speeds, and also, I think for > what sounds like fondly hoping for PCI > 'broadcast' capabilities. > > Sad to say, PCI is not good for any of those > claims: Dave, Your comments on the limits of PCI are much appreciated and informative - I learnt a lot from reading them. I have a number of comments though. The first is that I am well aware of the real advantages of using the "traditional" beowulf approach of using ethernet for communications. In particular, you are absolutely right that PCI doesn't really have a usable equivalent of multicasting and broadcasting. A bit of history. Wwhen I first started talking with Neil Carson and his colleagues at Chaltech (february 98), my original idea was that I wanted a minimal computing node, composed of (i) a CPU, (ii) some RAM, and (iii) a fast ethernet controller. The idea was to put the three items in as small a space as possible and then link together lots of them with fast ethernet. In particular, I was really keen to be able to use multicasting as a very efficient way of selectively sending messages between the nodes. It was patiently pointed out to me that you can't just wire up a StrongARM to an ethernet controller - the only practical way is to go via Digital's 21285 PCI Bridge chip. Not such a bad idea really - they don't cost much, and it allows the CPU to talk directly with the memory without (I believe) interfering with the PCI bus. Similarly, data transfers between the PCI bus (i.e. the ethernet controller) and the SDRAM can occur without the CPU being involved - it could continue number crunching data in its L1 cache. So at that stage, the idea was CPU/mem/PCI bridge/ethernet. But then it was realised that since the PCI bus link was there, you might as well leave open the possiblity of doing direct local processor to processor communications. In that way you might have (say) 3 CPU/memory modules sharing the same ethernet controller. But then, we realised that if you now add a separate PCI bridge, and put all the components on a PCI board, you could put the whole lot into a single PCI slot, and use a host computer to load Linux onto each CPU, and generally monitor whats going on. Finally, Gareth (Simpson) had the idea of using HiNT's forthcoming 3-way PCI bridge, so that on board you could have 2 separate 4 device PCI busses, linked to the hosts bus. One PCI bus would run down each side of the board, each with 4 modules connected. This would allow any combination of transfers either (i) between the four local modules on each separate PCI bus, (ii) between CPUs on both sides of the board, and (iii) with other PCI devices, including the host, via the main PCI bus. (Now it turns out that HiNTs 3-way PCI bridge chip has been delayed, so Gareth's having to come up with a substitute - and I don't know yet what it will be). Now, I'm quite happy to repeat after that there ain't no such thing as 132 Mbytes/s on a single PCI bus. Nnevertheless, the overall communications bandwidth in a PC with four slots, each with a board with two separate four element PCI busses, could be as much as 9 times whatever value you give me, because there would be effectively 9 different PCI busses in the system. Does that seem right to you? Obviously, to get the system to work that well, tasks would have to be split between the different CPUs in a very intelligent way, so that as much communication as possible is done between CPUs sharing the same 4 processor section - difficult, but not (I think) impossible - it will depend very much on the task. Now the other point is that the approach that Chaltech have taken is to put the CPU/SDRAM/Bridge combo on a separate 7 by 7 cm PCB, with a (very cheap) SO-DIMM connector. To put it mildly, this leaves quite a few options open. For example: 1) For those who like the idea of being able to boot Linux over the PCI bus, and are happy to use PCI for message passing you could use the current proposition - namely stick a few of these boards in a standard Linux based PC. And use the host's ethernet connector to wire up several machines if so desired. 2) In the not too distant future, one could easily imagine producing a SO-DIMM compatible fast ethernet module, so that on each side of the carrier board there would be 3 CPU modules sharing one ethernet link. That way, the host's PCI bus doesn't get used at all. 3) For the true beowulf fan (like you Dave?) who wants a fast ethernet link per CPU, you could have two ethernet links and two CPU modules on each side of the board. That way, the host's PCI bus would essentially only be used for booting and for providing the power supply. Isn't that what you would like - A PC with maybe 16 CPU modules and 16 fast ethernet links? 4) If you insist on having local hard-disk access for each CPU, Simtec (or someone else) might be persuaded to produce an IDE controller module. In that case you might (on each side of the carrier PCI board) have 2 CPU modules, and ethernet controller, and a hard disk controller. Oh, and while I'm at it, can I please have a SCSI-2 module, oh, and Firewire... etc. etc. 5) Suppose someone wants to try out Sebring's PCI ring system. All that would be required would be to produce an alternative carrier board with a Sebring chip on each side. The Sebring chip is capable of supporting 4 33MHz 32 bit PCI devices - that's all we need. No problems with dodgy PCI hardware in that case. 6) Suppose that next years Pentium III motherboards come equiped with 64 bit 66MHz PCI (anyone want to take bets). All that's needed would be to replace the host-carrier board bridge chip with something more substantial, and I'll be able to plug in all the StrongARM CPU modules that I bought this year. I like that.... 7) I take your points about the dubious design of some of the multiple PCI backplane cages. However, if I happen to have 16 of these boards, it would be no great deal to try them out. If it works, great. If it doesn't work then I will have wasted a bit of money buying the back plane - but that's no big problem - they only cost a few hundred dollars. If there are enough people out there interested in trying different backplanes there's a reasonable chance that one of us might find one that's sufficiently well designed to get it to work... To summarize - if people think that the definition of "beowulf" excludes using PCI, then I'd be quite happy to stop submerging the beowulf mailing list with irrelevent and over-the-top hype. Those who are interested can keep tuned in to the sa-beowulf list. But to be honest, I think that the CPU module idea has a lot of potential, even for people like yourself... Only time will tell. Best wishes Simon Thorpe From hinsen@cnrs-orleans.fr Thu, 28 Jan 1999 06:31:50 -0500 Date: Thu, 28 Jan 1999 06:31:50 -0500 From: Konrad Hinsen hinsen@cnrs-orleans.fr Subject: Alpha vs. Intel We are planning to set up a small Beowulf system for a research group in biomolecular simulation. Since (at least initially) most jobs will be plain serial, single-CPU performance matters and that's why we were considering an Alpha-based system. On the other hand, software support for Alphas seems to be quite a bit weaker than for Intel-based Linux machines. All of our applications are available in source code, mostly C, but also some legacy Fortran code. So the main problem seems to be compilers. Could anyone using an Alpha cluster please comment on the current situation? Are there any compilers other than the GNU/EGCS suite? Is GNU/EGCS performance sufficient for numerical work in C and C++? How bad is g77 performance? What about Fortran-90? I suppose NAG-F90 is available, but how is performance? And is there anything else that you wish you had? -- ------------------------------------------------------------------------------- Konrad Hinsen | E-Mail: hinsen@cnrs-orleans.fr Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69 Rue Charles Sadron | Fax: +33-2.38.63.15.17 45071 Orleans Cedex 2 | Deutsch/Esperanto/English/ France | Nederlands/Francais ------------------------------------------------------------------------------- From Doug.Hughes@Eng.Auburn.EDU Thu, 28 Jan 1999 09:54:29 -0500 Date: Thu, 28 Jan 1999 09:54:29 -0500 From: Doug Hughes Doug.Hughes@Eng.Auburn.EDU Subject: Alpha vs. Intel > >We are planning to set up a small Beowulf system for a research group >in biomolecular simulation. Since (at least initially) most jobs will >be plain serial, single-CPU performance matters and that's why we were >considering an Alpha-based system. On the other hand, software support >for Alphas seems to be quite a bit weaker than for Intel-based Linux >machines. All of our applications are available in source code, mostly >C, but also some legacy Fortran code. So the main problem seems to be >compilers. > >Could anyone using an Alpha cluster please comment on the current >situation? Are there any compilers other than the GNU/EGCS suite? >Is GNU/EGCS performance sufficient for numerical work in C and C++? >How bad is g77 performance? What about Fortran-90? I suppose NAG-F90 >is available, but how is performance? And is there anything else >that you wish you had? >-- We're doing the same thing for physics (well, the physics department is, I'm just a technical consultant). I have 'some' very subjective answers: g77 performance is bad.. very bad when compared with Digital Unix fortran compiler. The NAG licensing is very restrictive and seems expensive. Haven't tried it yet. Microway NBS compiler is fair. much better than g77, but not as good as Digital (on limited runs so far). Fortran 90 has limited availability right now. And the MPI stuff on alpha is kind of quirky. The automatic LAM/MPI startup script seems to fail trying to rsh to the different machine for no apparent reason.. Seems to be buggy network code.. Don't know. But if you start up the lamd's as described in the manual section it works fine. The MPICH stuff will not complete the test suite. No C benchmarks though. -- ____________________________________________________________________________ Doug Hughes Engineering Network Services System/Net Admin Auburn University doug@eng.auburn.edu From langzhi@fizik.upm.edu.my Thu, 28 Jan 1999 10:47:18 -0500 Date: Thu, 28 Jan 1999 10:47:18 -0500 From: Eng ler ...(admin) langzhi@fizik.upm.edu.my Subject: Beowulf with PII 400 MHZ Benchmark - HELP ! We just build up our beowulf clusters. Running Mandrake 5.2 with PVM 3.4 and MPICH 1.1.1 installed and running properly. All 8 clusters is connected via 3 Com SuperStackII 3300 Switch (10/100). All clusters are PII 400MHZ with S3 GX2 AGP , Intel 440BX Motherboard with 64MB SDRAM PC100. The master have 128MB SDRAM. We installed ParkBench and run the MATMUL_pvm benchmark. The benchmark result give us around 60 Mflops for 2 machines and 120 Mflops for 4 Macfhines. We also run the NPB2.1 Parallel benchmark and most of the benchmark program (bt,lu ... with SAMPLE) give around 50 Mflops ... The quetions : How is the benchmarking result ? Is it too bad ? (Coz i though it can do much better ). Anyone out there using same or almost same config and archive much higher benchmark result ?? Any idea to boost the performance ?? thanks !! From Christopher.Bohn@afit.af.mil Thu, 28 Jan 1999 11:26:30 -0500 Date: Thu, 28 Jan 1999 11:26:30 -0500 From: Bohn, Christopher A Christopher.Bohn@afit.af.mil Subject: Beowulf with PII 400 MHZ Benchmark - HELP ! Good day, I don't know about ParkBench and MATMUL_pvm, but as for NPB, you can get a MUCH better performance by using a larger problem size. The S-class problem is >not< large enough to tax your system. And I'm a bit surprised that you were even able to run it. I'm particurly thinking of LU, since that's the one I've been modifying for my thesis. The S-class problem on LU is a 12x12x12-element domain, and the unmodified LU will try to break that into eight 3x6x12-element subdomains. Except that the unmodified LU won't let a subdomain have fewer than four elements in any dimension, so it should've aborted. When running the unmodified LU from NPB2.3 on eight processors [1], I'm getting around 350Mflops for the A-class (64x64x64). Try running the benchmarks with larger problem sizes. Take care, cb [1] one 333MHz PII, six 400MHz PII's, and one 450MHz PII, each with 128MB SDRAM [2], interconnected with Intel Express 510T Switch, using RH5.0 & MPICH 1.1.0, as the closest configuration to yours that I'm running. [2] this is not a significant difference between your cluster & ours, as the entire A-class application requires 40MB for the entire data set. *-*-*-*-*-*-*-* Capt Christopher A. Bohn Graduate Student, Electrical (digital) Engineering Air Force Institute of Technology Phone (937)255-3636 (DSN 785) AFIT/EN638 Lab x4606 Voicemail x6638 2950 P St, Box 4638 email Christopher.Bohn@afit.af.mil Wright-Patterson AFB OH 45433-7765 EngrBohn@aol.com http://members.aol.com/EngrBohn/ *-*-*-*-*-*-*-* > -----Original Message----- > From: Eng ler ...(admin) [SMTP:langzhi@fizik.upm.edu.my] > Sent: Thursday, January 28, 1999 10:48 PM > To: beowulf list > Subject: Beowulf with PII 400 MHZ Benchmark - HELP ! > > We just build up our beowulf clusters. Running Mandrake 5.2 with PVM 3.4 > and MPICH 1.1.1 installed and running properly. All 8 clusters is > connected via 3 Com SuperStackII 3300 Switch (10/100). All clusters are > PII 400MHZ with S3 GX2 AGP , Intel 440BX Motherboard with 64MB SDRAM > PC100. The master have 128MB SDRAM. > > We installed ParkBench and run the MATMUL_pvm benchmark. > The benchmark result give us around 60 Mflops for 2 machines and 120 > Mflops for 4 Macfhines. > > We also run the NPB2.1 Parallel benchmark and most of the benchmark > program (bt,lu ... with SAMPLE) give around 50 Mflops ... > > The quetions : > How is the benchmarking result ? Is it too bad ? (Coz i though it can do > much better ). > > Anyone out there using same or almost same config and archive much higher > benchmark result ?? > > Any idea to boost the performance ?? > > thanks !! > From NewD@esi.com Thu, 28 Jan 1999 11:33:23 -0500 Date: Thu, 28 Jan 1999 11:33:23 -0500 From: Dave New NewD@esi.com Subject: Beoboard idea > -----Original Message----- > From: thorpe@cerco.ups-tlse.fr [SMTP:thorpe@cerco.ups-tlse.fr] > Sent: Thursday, January 28, 1999 4:04 AM > To: Dave New > Cc: beowulf@beowulf.gsfc.nasa.gov; sa-beowulf@kragen.dnaco.net > Subject: RE: Beoboard idea > > > > [Dave New] Been there, done that, and got the > > scars to prove it... > > > > I've listened to this thread for > > sometime, and several times thought about saying > > something to those that are pushing things with > > lots o' PCI slots, and making claims for 132 > > MBytes/sec speeds, and also, I think for > > what sounds like fondly hoping for PCI > > 'broadcast' capabilities. > > > > Sad to say, PCI is not good for any of those > > claims: > > > Dave, > Your comments on the limits of PCI are much appreciated and > informative - I learnt a lot from reading them. I have a number of > comments > though. [Dave New] Thanks. > The first is that I am well aware of the real advantages of using the > "traditional" beowulf approach of using ethernet for communications. In > particular, you are absolutely right that PCI doesn't really have a usable > equivalent of multicasting and broadcasting. [Dave New] Sorry. Didn't mean to 'talk down' to you. I was trying to hit several different subjects that had been floating around on this thread, and happened to pick on your message as the one to originally reply to. > A bit of history. Wwhen I first started talking with Neil Carson and his > colleagues at Chaltech (february 98), my original idea was that I wanted a > minimal computing node, composed of (i) a CPU, (ii) some RAM, and (iii) a > fast ethernet controller. The idea was to put the three items in as small > a > space as possible and then link together lots of them with fast ethernet. > In particular, I was really keen to be able to use multicasting as a very > efficient way of selectively sending messages between the nodes. > > It was patiently pointed out to me that you can't just wire up a StrongARM > to an ethernet controller - the only practical way is to go via Digital's > 21285 PCI Bridge chip. Not such a bad idea really - they don't cost much, > and it allows the CPU to talk directly with the memory without (I believe) > interfering with the PCI bus. Similarly, data transfers between the PCI > bus > (i.e. the ethernet controller) and the SDRAM can occur without the CPU > being involved - it could continue number crunching data in its L1 cache. [Dave New] Correct. For most current chipsets, as long as the CPU and the PCI bus do not concurrently need access to the main memory, things can proceed at a reasonable bandwidth. Of course, the CPU will still have the advantage, mainly because it has a lower latency and higher bandwidth access to the main memory than any PCI bus device has had, so far. > So at that stage, the idea was CPU/mem/PCI bridge/ethernet. But then it > was > realised that since the PCI bus link was there, you might as well leave > open the possiblity of doing direct local processor to processor > communications. In that way you might have (say) 3 CPU/memory modules > sharing the same ethernet controller. > > But then, we realised that if you now add a separate PCI bridge, and put > all the components on a PCI board, you could put the whole lot into a > single PCI slot, and use a host computer to load Linux onto each CPU, and > generally monitor whats going on. > > Finally, Gareth (Simpson) had the idea of using HiNT's forthcoming 3-way > PCI bridge, so that on board you could have 2 separate 4 device PCI > busses, > linked to the hosts bus. One PCI bus would run down each side of the > board, > each with 4 modules connected. This would allow any combination of > transfers either (i) between the four local modules on each separate PCI > bus, (ii) between CPUs on both sides of the board, and (iii) with other > PCI > devices, including the host, via the main PCI bus. [Dave New] At this point, it's Only a Matter of Software(tm) 8-). > (Now it turns out that HiNTs 3-way PCI bridge chip has been delayed, so > Gareth's having to come up with a substitute - and I don't know yet what > it > will be). [Dave New] One almost-3-way bridge that I've looked into is the Intel i960Rx. It really only has a primary and secondary PCI bus, but it also has a fast i960-style 'local bus' that you can hang memory or other devices off of (probably not other processors, though). Additionally, it's an I2O-compliant component, and (at least so far) IxWorks RTOS is bundled with it, to make it an intelligent I/O controller with essentially a PCI local bus and a fast i960-local bus with I2O message passing and realtime threading. Being already familiar with the i960Cx family of processors and VxWorks (having written image processing drivers for SIMD arrays using them), the i960Rx stuff doesn't scare me off, but others might find the whole i960/IxWorks business a put-off. The 'bridge' has three scatter-gather DMA engines built in, two between the primary PCI and local bus, and one between the local bus and secondary PCI. Because there isn't a direct DMA path between primary and secondary PCI, the i960 local memory has to be used as a buffer. This might not be all bad, since it offloads handling the DMA scatter/gather initialization onto the i960, and gives a place for synchronization FIFO's to reside. I2O/IxWorks already has primitives for doing DMA resource management, so you could have a larger number of processes competing for DMA access than there are DMA channels. Electrically, the part is found wanting, though. Originally designed to run the two PCI busses completely independently of each other, the current Intel errata (excuse, 'updated spec') says that the primary and secondary PCI clocks must be tied together (i.e. frequency and phase coherent), and that the part is still dynamic, meaning that you can't use it in systems that have 100 KHz clock 'green' modes, or 'spread-spectrum' clocking, where the clock frequency is changed on a per-cycle basis. The i960Rx clocking can't follow such changes, and will croak. So, it isn't recommended for applications where you don't have control over what the system clock is doing. Intel has stated that this is now a 'feature' (being non- PCI 2.1 compliant) and will not fix this in future tapeouts. This, from the guys that pushed PCI compliance in the first place... All that aside, I wonder what effect having some local controller that understands MPI (or similar) would bring to the picnic. Maybe nothing at all. So far, the (few) commercial applications of I2O have centered around RAID arrays, and other 'enterprise' computing. Until recently, I2O was kept under wraps, but now that the specification has gone 'open doc', so to speak, maybe some interest will start. SCO seems to think that UDI would be perfect as a standard for writing the low-level device drivers for I2O nodes. They are trying to interest the Open Source community in this. > Now, I'm quite happy to repeat after that there ain't no such thing as 132 > Mbytes/s on a single PCI bus. Nnevertheless, the overall communications > bandwidth in a PC with four slots, each with a board with two separate > four > element PCI busses, could be as much as 9 times whatever value you give > me, > because there would be effectively 9 different PCI busses in the system. > Does that seem right to you? Obviously, to get the system to work that > well, tasks would have to be split between the different CPUs in a very > intelligent way, so that as much communication as possible is done between > CPUs sharing the same 4 processor section - difficult, but not (I think) > impossible - it will depend very much on the task. [Dave New] I agree. A traffic analysis is a good start, when considering this kind of approach. If you get detailed enough, you will probably find reasons why some communications need to happen outside your local busses. Then, you need to find out if you can minimize it, or if there is enough bandwidth to go around. The hard part of this analysis is that most chipsets don't publish their arbiter algorithm(s), or have a good discussion of localized contention issues. These are crucial to a good analysis. Frequently you are forced to prototype a system with the components you want to use (an expensive, time-consuming proposition) and then fabricate test cases that you believe will put the kind of loads on the busses you think you will see, then verify it with USD$30K logic analyzers. I have found it hard to get companies to 'sign up' for this kind of upfront analysis. They would rather believe the hype, and forge ahead with a beta build, and empirical testing (it worked all last night with our program, so ship it!). > Now the other point is that the approach that Chaltech have taken is to > put > the CPU/SDRAM/Bridge combo on a separate 7 by 7 cm PCB, with a (very > cheap) > SO-DIMM connector. To put it mildly, this leaves quite a few options open. > For example: > > 1) For those who like the idea of being able to boot Linux over the PCI > bus, and are happy to use PCI for message passing you could use the > current > proposition - namely stick a few of these boards in a standard Linux based > PC. And use the host's ethernet connector to wire up several machines if > so > desired. > > 2) In the not too distant future, one could easily imagine producing a > SO-DIMM compatible fast ethernet module, so that on each side of the > carrier board there would be 3 CPU modules sharing one ethernet link. That > way, the host's PCI bus doesn't get used at all. > > 3) For the true beowulf fan (like you Dave?) who wants a fast ethernet > link > per CPU, you could have two ethernet links and two CPU modules on each > side > of the board. That way, the host's PCI bus would essentially only be used > for booting and for providing the power supply. Isn't that what you would > like - A PC with maybe 16 CPU modules and 16 fast ethernet links? [Dave New] Maybe. I'm straddling an industrial line, here. There are many folks that think that off-the-shelf stuff (read PC) would be neat, but traditionally, our stuff has been on VME bus, built in to wirebond machines (for example) that do 10-20 wirebonds a second. Moving wirebonding probes that quickly for wirebonding at that speed produces a *lot* of noise and vibration (it's driven with compressed air), and 'regular' desktop-style PCI card-slot arrangements vibrate the cards right out of the slots! So, this explains the interest in Compact PCI, not only for its electrical characteristics, but for its mechanical ones, as well. We already boot across the VME backplane from a master M68K CPU, so doing this kind of thing across the PCI bus would be a matter of re-writing the transport layer. Currently this is all done with VxWorks or QNX (well, at least it's not CE... 8-) > 4) If you insist on having local hard-disk access for each CPU, Simtec (or > someone else) might be persuaded to produce an IDE controller module. In > that case you might (on each side of the carrier PCI board) have 2 CPU > modules, and ethernet controller, and a hard disk controller. Oh, and > while > I'm at it, can I please have a SCSI-2 module, oh, and Firewire... etc. > etc. [Dave New] A PMC site (or maybe PCMCIA) would do nicely. Then everyone could pick their favorite interface. As for me, it might be a digital camera interface, for multi-tap line scan, streaming at 25 MBytes/sec per tap 8-). > 5) Suppose someone wants to try out Sebring's PCI ring system. All that > would be required would be to produce an alternative carrier board with a > Sebring chip on each side. The Sebring chip is capable of supporting 4 > 33MHz 32 bit PCI devices - that's all we need. No problems with dodgy PCI > hardware in that case. > > 6) Suppose that next years Pentium III motherboards come equiped with 64 > bit 66MHz PCI (anyone want to take bets). All that's needed would be to > replace the host-carrier board bridge chip with something more > substantial, > and I'll be able to plug in all the StrongARM CPU modules that I bought > this year. I like that.... [Dave New] Don't hold your breath for 66 MHz PCI. The 64-bit stuff is easy (and there are already things out that use it). The 66 MHz stuff is hard, because the original 33 MHz design didn't do a good job of accounting for eventually doubling the clock (and cutting all the setup times in half). The 66 MHz spec was an appendix to the original 33 MHz spec, with a lot less thought put into whether anyone would be able to actually put together hardware that could talk to such a bus, with such tight timings, and such varying loads. AGP is one result of this. Impatient (or perhaps giving up) with waiting for someone to implement a multiple-slot 66 MHz PCI bus and cards, Intel forged ahead with a simpler two-agent-only AGP bus to accelerate access to video boards. The timing and electrical specs are much easier to meet, when you can accurately predict the loads on the bus (there are always two agents, never more). Several folks on the PCI sig maillist have stated their doubts about anyone ever producing a real 66 MHz general purpose bus, based on the PCI specification as it now stands. Recent changes to the spec are more concerned with facilitating Wintel power management schemes, than 'fixing' 66 MHz. Instead, some folks have started to look into high speed synchronous serial designs to work around the timing limitations of PCI (and parallel busses, in general). This kind of brings us back to things like SCI, and all its poor offspring, like FireWire, USB, FibreChannel, and SLDRAM, which are essentially early dumbed-down subsets of the Real Thing(tm), (SCI 8-). > 7) I take your points about the dubious design of some of the multiple PCI > backplane cages. However, if I happen to have 16 of these boards, it would > be no great deal to try them out. If it works, great. If it doesn't work > then I will have wasted a bit of money buying the back plane - but that's > no big problem - they only cost a few hundred dollars. If there are enough > people out there interested in trying different backplanes there's a > reasonable chance that one of us might find one that's sufficiently well > designed to get it to work... [Dave New] I'm not sure where you've seen PICMG systems for 'a few hundred dollars'. The two systems we got from ICS here for evaluation weighed in at a cool USD$6K each. This included a Pentium 200 MMX (fast at the time) with video and 64 MBytes of RAM, a 2 GByte drive, and CD-ROM. The motherboard had seven PCI slots, four of them behind a DEC bridge mounted on the motherboard. I don't recall how many ISA slots, but there were plenty of them, as well. Otherwise, it was pretty stripped down, but because it is sold as an industrial system, there is always a hefty price tag attached. To be completely fair to ICS, the system was built like a tank, and had a considerable amount of EMI gasketing around every conceivable opening, including a separate lower compartment for the drives, since the drive openings are so hard to individual shield. We could still vibrate the boards out of their sockets, by bolting the system on the bottom rack shelf of our customer's wirebonder, though. The desktop bracket and card-slot solution is horrible for those kind of environments. I much prefer Eurocard-style racks for that. > > To summarize - if people think that the definition of "beowulf" > excludes using PCI, then I'd be quite happy to stop submerging the beowulf > mailing list with irrelevent and over-the-top hype. Those who are > interested can keep tuned in to the sa-beowulf list. But to be honest, I > think that the CPU module idea has a lot of potential, even for people > like > yourself... Only time will tell. [Dave New] Oh no, I don't think that Beowulf excludes PCI, I mostly wanted to help folks think realistically about what can and can't be done across PCI. I still think a bunch of modules plugged into PCI is a good idea, but I'm concerned about the flood of PICMG systems, and folks' temptation to use that as a standard form factor. I would much rather see folks do these modules on Compact PCI (or at least on CPCI PMC carriers), which would get us a much better electrical and mechanical platform. Maybe not as cheap as desktop machines, but as I mentioned above, you may be hard-pressed to find a good supply of inexpensive PICMG motherboards and cabinets, as well. Instead, the telephony industry now seems to be driving Compact PCI heavily, and as volumes ramp up, you should see prices drop quickly for things like CPCI racks, CPUs and power supplies. Also, check out the IEEE P1996 standards initiative. This is a high-reliability Compact PCI bus specification, with some interesting features, like 64-bit addressability and WAN networking over fibre or SCI. Meant for things like telephony switches or a widely distributed traffic light control system, linked together over fibre. Yummm! Once you get a 256-plus node machine running, you don't want to be debugging timing problems in a PICMG system... 8-) > Best wishes > > Simon Thorpe > [Dave New] Cheers, -- DaveN > From gstovall@nortelnetworks.com Thu, 28 Jan 1999 12:13:23 -0500 Date: Thu, 28 Jan 1999 12:13:23 -0500 From: Greg Stovall gstovall@nortelnetworks.com Subject: Beoboard idea Dave, I've used the i960Rx in several prototype devices, so I would like to add a few comments to yours. * The i960Rx is really found wanting when it comes to cpu power. My measurements shows that it runs approximately 1/16 as fast as a Dell GXa/266 (266MHz PII, 66MHz SDRAM). Doesn't execute our protocol stacks very quickly. * There *is* a direct DMA path between the primary and secondary busses. We are able to disable the processor and just use the built-in bridge to access the network devices on the secondary bus from our host processor, no problem. * The new i960RN adds a 64bit/66MHz bus plus 100MHz i960Rx core, but this is a really inadequate leap. I don't know why Intel did this. My guess is they're just trying to squeeze a little more mileage from the architecture before scrapping it in favor of the StrongArm. * Ramix (http://www.ramix.com) has a nice array of i960Rx PMC cards if you are interested in the device anyway. * We've ported RTEMS (http://www.oarcorp.com), a free RTOS, to these cards, if anyone desires to avoid the high VxWorks costs. Ramix has the software available. > [Dave New] One almost-3-way bridge that I've looked > into is the Intel i960Rx. It really only has a > primary and secondary PCI bus, but it also has a > fast i960-style 'local bus' that you can hang > memory or other devices off of (probably not > other processors, though). Additionally, it's > an I2O-compliant component, and (at least so far) > IxWorks RTOS is bundled with it, to make it an intelligent > I/O controller with essentially a PCI local bus > and a fast i960-local bus with I2O message > passing and realtime threading. Being already familiar > with the i960Cx family of processors and VxWorks > (having written image processing drivers for SIMD > arrays using them), the i960Rx stuff doesn't scare > me off, but others might find the whole i960/IxWorks > business a put-off. > > The 'bridge' has three scatter-gather DMA engines > built in, two between the primary PCI and local > bus, and one between the local bus and secondary > PCI. Because there isn't a direct DMA path between > primary and secondary PCI, the i960 local memory > has to be used as a buffer. This might not be > all bad, since it offloads handling the DMA > scatter/gather initialization onto the i960, > and gives a place for synchronization FIFO's > to reside. I2O/IxWorks already has primitives > for doing DMA resource management, so you > could have a larger number of processes competing > for DMA access than there are DMA channels. > > Electrically, the part is found wanting, though. > Originally designed to run the two PCI busses > completely independently of each other, the current > Intel errata (excuse, 'updated spec') says that > the primary and secondary PCI clocks must be > tied together (i.e. frequency and phase coherent), > and that the part is still dynamic, meaning that > you can't use it in systems that have 100 KHz > clock 'green' modes, or 'spread-spectrum' clocking, > where the clock frequency is changed on a per-cycle > basis. The i960Rx clocking can't follow such > changes, and will croak. So, it isn't recommended > for applications where you don't have control > over what the system clock is doing. Intel has > stated that this is now a 'feature' (being non- > PCI 2.1 compliant) and will not fix this in > future tapeouts. This, from the guys that > pushed PCI compliance in the first place... > > All that aside, I wonder what effect having some > local controller that understands MPI (or similar) > would bring to the picnic. Maybe nothing at > all. So far, the (few) commercial applications > of I2O have centered around RAID arrays, and > other 'enterprise' computing. Until recently, > I2O was kept under wraps, but now that the > specification has gone 'open doc', so to > speak, maybe some interest will start. SCO > seems to think that UDI would be perfect as > a standard for writing the low-level device > drivers for I2O nodes. They are trying to > interest the Open Source community in this. > From lindahl@cs.virginia.edu Thu, 28 Jan 1999 12:48:20 -0500 Date: Thu, 28 Jan 1999 12:48:20 -0500 From: Greg Lindahl lindahl@cs.virginia.edu Subject: Alpha vs. Intel > I have 'some' very subjective answers: > > g77 performance is bad.. very bad when compared with Digital Unix fortran > compiler. http://www.cs.virginia.edu/~lindahl/spec/ g77 can give as little as half the performance of the Digital Unix compiler, or as much as a few percent faster. It's very important to use the new fast math libraries; many of the people complaining about really awful performance are using codes that depend on the library. I have a copy of the NAG F95 compiler which I would be happy to try on a real F9x code, if anyone wanted to send me one. > Fortran 90 has limited availability right now. Other than the NAG compiler, yes. I would suspect that the new Nag compiler probably will be about the same compared to the Digital compiler as g77. > And the MPI stuff on alpha > is kind of quirky. I've never had a problem with it (both mpich and my own Legion MPI implementation), so I suspect this is operator error. -- greg From NewD@esi.com Thu, 28 Jan 1999 12:52:37 -0500 Date: Thu, 28 Jan 1999 12:52:37 -0500 From: Dave New NewD@esi.com Subject: Beoboard idea > -----Original Message----- > From: Greg Stovall [SMTP:gstovall@nortelnetworks.com] > Sent: Thursday, January 28, 1999 12:12 PM > To: Dave New; 'thorpe@cerco.ups-tlse.fr' > Cc: beowulf@beowulf.gsfc.nasa.gov; sa-beowulf@kragen.dnaco.net > Subject: RE: Beoboard idea > > Dave, I've used the i960Rx in several prototype devices, so I would like > to > add a few comments to yours. [Dave New] Thanks. > * The i960Rx is really found wanting when it comes to cpu power. My > measurements shows that it runs approximately 1/16 as fast as a Dell > GXa/266 > (266MHz PII, 66MHz SDRAM). Doesn't execute our protocol stacks very > quickly. [Dave New] It *is* true that the i960 itself is not a blazing fast processor, but in the cases that I was looking at, it was not being asked to do a lot of fast processing itself -- just act as the 'baton waver' for a nest of proprietary SIMD array processors. As such the heavy throughput path was not through the i960. Instead the SIMD array was processing digital video fed directly to them, and the results being passed to the i960 for host-based post-processing via shared memory. > * There *is* a direct DMA path between the primary and secondary > busses. We are able to disable the processor and just use the built-in > bridge to access the network devices on the secondary bus from our host > processor, no problem. [Dave New] What I mean is that there is no DMA scatter-gather engine hooked between the primary and secondary PCI busses. While it is true that you can just use the part as a bridge, then where is the win in having a processor running an IRTOS in the bridge? > * The new i960RN adds a 64bit/66MHz bus plus 100MHz i960Rx core, but > this is a really inadequate leap. I don't know why Intel did this. My > guess is they're just trying to squeeze a little more mileage from the > architecture before scrapping it in favor of the StrongArm. [Dave New] I assume the 66 MHz mentioned is the local i960 bus, not one of the PCI busses. As far as Intel's embedded roadmap is concerned, the i960 has always been the poor stepchild of their processor family. The design was purchased (from Number Nine, I believe, back in the early 32-bit days), so it has that 'not invented here' taint to it. i960's are never fabbed in the same plants that the top-of-the-line x86 stuff is. Rather, after the newest and fastest fabs are fully amortized, then Intel uses them to fab all the 2nd-tier parts, including their embedded line. This means pedestrian line sizes, and clock speeds. That has always annoyed me, but the saving throw (at the time) for the i960CA superscaler part was that we could arrange to run two-clock loops that saturated the processor local bus (Address, Data, Address, Data, etc.) to drive our SIMD arrays. Using this setup, with a 25 MHz i960CA and 128 parallel processors, we could get 500 MBytes/sec of binary morphology throughput. Pretty good for a three-board 6U VME sandwich, which included video-in-a-window, four camera inputs, and several megs of main and image processing memory. These are the systems I mentioned that are doing 10-20 wirebond inspections a second on Intel processors, among other things. Talk about using a computer to build itself... 8-) Not sure what Intel will do with the StrongARM stuff. Seemed at first they were going great guns, then nothing much out of them for a while. Seems like their embedded group is typically always eclipsed by what their desktop computing group is doing. > > * Ramix (http://www.ramix.com) has a nice array of i960Rx PMC cards if > you are interested in the device anyway. [Dave New] Thanks. I'll take a look. > * We've ported RTEMS (http://www.oarcorp.com), a free RTOS, to these > cards, if anyone desires to avoid the high VxWorks costs. Ramix has the > software available. [Dave New] I monitor the crossgcc list, where RTEMS has been discussed. I haven't tried it myself, but it seems rather nice, from other folk's reports. Interesting that you mention VxWorks. IxWorks is bundled with the i960Rx, not VxWorks. IxWorks is free, but the Tornado development kit from Wind River for writing applications for the IRTOS is not. Also, the last time I checked, Wind River (amazingly enough) had not decided to write an I2O host module for VxWorks, so you could run it on a host processor talking to their IxWorks running in the i960Rx. They were leaving that to the 'enterprise' computing OSes, like NT and NetWare. When I pointed out that this stuff could be of use to the embedded community, they seemed surprised at that idea. I also could not get any good figures from them for I2O message passing latency in IxWorks on typical i960Rx setups. From your initial comments, maybe I can see why. Why publish any specifications that might be embarassing? In a way, it was fortunate that we never pursued that line much further. I2O has not evolved (yet, if ever) into something that can be considered a commodity protocol. Support for it (even after it went open doc) in the open software community appears to be just about nil. [Dave New] Cheers, -- DaveN > From Doug.Hughes@Eng.Auburn.EDU Thu, 28 Jan 1999 12:52:57 -0500 Date: Thu, 28 Jan 1999 12:52:57 -0500 From: Doug Hughes Doug.Hughes@Eng.Auburn.EDU Subject: Alpha vs. Intel > >> And the MPI stuff on alpha >> is kind of quirky. > >I've never had a problem with it (both mpich and my own Legion MPI >implementation), so I suspect this is operator error. > Very likely! However, when the readme/install instructions don't work, and the errors returned are undocumented I become a bit suspicious. I become a bit suspicious. (I'm new to MPI, but very old to installing and testing software) -- ____________________________________________________________________________ Doug Hughes Engineering Network Services System/Net Admin Auburn University doug@eng.auburn.edu From JayPrince@usa.net Thu, 28 Jan 1999 13:54:43 -0500 Date: Thu, 28 Jan 1999 13:54:43 -0500 From: JayPrince JayPrince@usa.net Subject: Beoboard idea This SA based beowulf sounds great. Can someone put me in touch with the person coordinating the modules? I'd like to buy one (probably).... I'd like to see the pinnout of the connector first, though (so I can figure out if it will work in my project.) I don't want to build a multiprocessor machine, acutally, but these modules would be great CPUs for a linux PDA. Could one be configured to host the PCI bus? This way, I could build a carrier card that had my IO on it and use the SO-DIMM as my CPU daughter card. Thanks for any advice- Jay -- "Honor your errors. A trick will only work for a while, until everybody else is doing it. To advance... requires a new game. But the process of going outside the conventional method...is indistinguishable from error... Evolution can be thought of as systematic error management." --Kevin Kelly From john.dubberley@nrlssc.navy.mil Thu, 28 Jan 1999 14:32:38 -0500 Date: Thu, 28 Jan 1999 14:32:38 -0500 From: john dubberley john.dubberley@nrlssc.navy.mil Subject: Beowulf with PII 400 MHZ Benchmark - HELP ! "Bohn, Christopher A" wrote: > Good day, > > I don't know about ParkBench and MATMUL_pvm, but as for NPB, you can get a > MUCH better performance by using a larger problem size. The S-class problem > is >not< large enough to tax your system. And I'm a bit surprised that you > were even able to run it. I'm particurly thinking of LU, since that's the > one I've been modifying for my thesis. The S-class problem on LU is a > 12x12x12-element domain, and the unmodified LU will try to break that into > eight 3x6x12-element subdomains. Except that the unmodified LU won't let a > subdomain have fewer than four elements in any dimension, so it should've > aborted. > > When running the unmodified LU from NPB2.3 on eight processors [1], I'm > getting around 350Mflops for the A-class (64x64x64). Try running the > benchmarks with larger problem sizes. > It seems every time a scientist gets a hold of a parallel machine for the first time they want to split up a matrix inversion or a matrix multiplication especially when the problem clearly fits into memory (64*64*64 is only 262k elements). Latency and bandwidth uses eat these problems alive. The smarter solution is to do parallel inversions. The most important attribute of a good parallel programmer is knowing where _not_ to break a problem into parallel parts. Sorry to be a bit grumpy about my pet peeve. -john dubberley From thorpe@cerco.ups-tlse.fr Thu, 28 Jan 1999 16:21:38 -0500 Date: Thu, 28 Jan 1999 16:21:38 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea Hi Jay, Good to have you on board (groan...) >This SA based beowulf sounds great. Can someone put me in touch with the >person coordinating the modules? Not much point in trying to get one from me (I'm just a customer). You should contact Neil Carson (neil@causality.com) and/or Simtec if you want to buy stuff. > I'd like to buy one (probably).... I'd >like to see the pinnout of the connector first, though (so I can figure >out if it will work in my project.) According to Neil, Gareth (Simpson) who does the hardware design is happy to provide details of the pinout wiring on the SODIMM connector to anyone who's interested (Now isn't that an enlightened attitude). You can contact him at Gareth@simtec.demon.co.uk. I get the impression that he doesn't react quite as quickly to email as I do though (probably 'cos he's ploughing through piles of PCI bridge data sheets at this very moment). >I don't want to build a >multiprocessor machine, acutally, but these modules would be great CPUs >for a linux PDA. Sounds reasonable to me. >Could one be configured to host the PCI bus? This way, I could build a >carrier card that had my IO on it and use the SO-DIMM as my CPU daughter >card. Just let us all know if you can get your I/O to work. Maybe we could all use it... Cheers Simon From thorpe@cerco.ups-tlse.fr Thu, 28 Jan 1999 16:23:01 -0500 Date: Thu, 28 Jan 1999 16:23:01 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea > Dave New wrote: >> (Now it turns out that HiNTs 3-way PCI bridge chip has been delayed, so >> Gareth's having to come up with a substitute - and I don't know yet what >> it >> will be). > [Dave New] One almost-3-way bridge that I've looked > into is the Intel i960Rx. Thanks for the info on the i960Rx. I hope that Gareth finds the time to listen in on this. I suspect, though, that he'll be looking for something as simple as possible. Hopefully, it'll be something that allows him to put all 8 CPU modules on one carrier board. But he's under quite a bit of pressure (from people like me) to get something done fast, so it may be that the first multiprocessor carrier won't be quite as sophisticated as we might like (the HiNTs should be along in a few months anyway). >> 3) For the true beowulf fan (like you Dave?) who wants a fast ethernet >> link >> per CPU, you could have two ethernet links and two CPU modules on each >> side >> of the board. That way, the host's PCI bus would essentially only be used >> for booting and for providing the power supply. Isn't that what you would >> like - A PC with maybe 16 CPU modules and 16 fast ethernet links? > [Dave New] Maybe. I'm straddling an industrial > line, here. There are many folks that think that > off-the-shelf stuff (read PC) would be neat, but > traditionally, our stuff has been on VME bus, built > in to wirebond machines (for example) that do 10-20 > wirebonds a second. Moving wirebonding probes that > quickly for wirebonding at that speed produces a > *lot* of noise and vibration (it's driven with > compressed air), and 'regular' desktop-style PCI > card-slot arrangements vibrate the cards right out > of the slots! So, this explains the interest in > Compact PCI, not only for its electrical > characteristics, but for its mechanical ones, as well. Again as a bit of history, one of Chaltech's original suggestions was to put all of this on a Eurocard format, using Compact PCI. It was only later when we thought about all those people out there with PCs running Linux that it was decided to go for the simpler PCI board approach. No reason why those SODIMM modules couldn't be mounted on a Compact PCI system though... >>Oh, and >> while >> I'm at it, can I please have a SCSI-2 module, oh, and Firewire... etc. >> etc. > [Dave New] A PMC site (or maybe PCMCIA) would > do nicely. Then everyone could pick their favorite > interface. As for me, it might be a digital camera > interface, for multi-tap line scan, streaming at > 25 MBytes/sec per tap 8-). Now there's a very sensible suggestion! Gareth...? >> 7) I take your points about the dubious design of some of the multiple PCI >> backplane cages. However, if I happen to have 16 of these boards, it would >> be no great deal to try them out. If it works, great. If it doesn't work >> then I will have wasted a bit of money buying the back plane - but that's >> no big problem - they only cost a few hundred dollars. If there are enough >> people out there interested in trying different backplanes there's a >> reasonable chance that one of us might find one that's sufficiently well >> designed to get it to work... > [Dave New] I'm not sure where you've seen PICMG systems > for 'a few hundred dollars'. Hmm. I'll have to check, but my recollection was that the 17 PCI slot backplane I saw was being sold for around 2500 FF (say $500, here in France). I generally assume that everything is always twice as expensive here as in the states :-( Mind you, for that you don't get the Single Board Computer, or the power supply or the Case. Simon From neil@causality.com Thu, 28 Jan 1999 16:40:41 -0500 Date: Thu, 28 Jan 1999 16:40:41 -0500 From: Neil A. Carson neil@causality.com Subject: Beoboard idea On the bridges, certainly AFAIR the three currently evaluated choices were: a) Two zero-latency bridges. b) One of the DEC transparent bridges c) One of the DEC non-transparent bridges ("drawbridge") a was the cheapest option, and was fairly easy to program. b was the easiest to program, but might not be as efficient as a and the lead times are longer. c was cool in that it would offer a 64-bit pci capability, but would be much harder to program. I don't know how Simpson made up his mind in the end. Neil -- Neil A. Carson From NewD@esi.com Thu, 28 Jan 1999 16:52:26 -0500 Date: Thu, 28 Jan 1999 16:52:26 -0500 From: Dave New NewD@esi.com Subject: Beoboard idea > -----Original Message----- > From: thorpe@cerco.ups-tlse.fr [SMTP:thorpe@cerco.ups-tlse.fr] > Sent: Thursday, January 28, 1999 4:24 PM > To: Dave New > Cc: beowulf@beowulf.gsfc.nasa.gov; sa-beowulf@kragen.dnaco.net > Subject: RE: Beoboard idea > [Dave New] > >> 7) I take your points about the dubious design of some of the multiple > PCI > >> backplane cages. However, if I happen to have 16 of these boards, it > would > >> be no great deal to try them out. If it works, great. If it doesn't > work > >> then I will have wasted a bit of money buying the back plane - but > that's > >> no big problem - they only cost a few hundred dollars. If there are > enough > >> people out there interested in trying different backplanes there's a > >> reasonable chance that one of us might find one that's sufficiently > well > >> designed to get it to work... > > [Dave New] I'm not sure where you've seen PICMG systems > > for 'a few hundred dollars'. > > > Hmm. I'll have to check, but my recollection was that the 17 PCI slot > backplane I saw was being sold for around 2500 FF (say $500, here in > France). I generally assume that everything is always twice as expensive > here as in the states :-( > Mind you, for that you don't get the Single Board Computer, or the power > supply or the Case. [Dave New] Oh. I wasn't thinking 'bare board' when you gave the original price. I suppose you can get PICMG backplanes by themselves far cheaper than for USD 'thousands', but then you still have to find a suitable cabinet to house the admittedly unusual form factor -- a 'regular' AT or ATX style cabinet won't do. A have yet to find a cheap PICMG cabinet. Also, USD$500 for an almost-passive (except maybe for a DEC bridge) backplane still seems kind of excessive, when you compare it to an entire PC system going for only about USD$200 more. I recently purchased an AMD K6-366 system w/64 MBytes, 6.4 GByte drive, 32x CD-ROM, 8MB i740 AGP, wave table sound, mouse, keyboard, and 3 1/2" floppy in a mid-tower ATX case for USD$799. Kind of hard to beat those economics (if you can find stuff like that for those kinds of prices where you are). That's what kind of powers the 'pile o' PC's' aspect of Beowulf. > Simon > [Dave New] Cheers, -- DaveN > > From m.j.s.vandoesburg@student.utwente.nl Thu, 28 Jan 1999 19:20:16 -0500 Date: Thu, 28 Jan 1999 19:20:16 -0500 From: Mark van Doesburg m.j.s.vandoesburg@student.utwente.nl Subject: Beoboard idea This SA based beowulf sounds great. Can someone put me in touch with the person coordinating the modules? You can try: neil@causality.com I'd like to buy one (probably).... I'd like to see the pinnout of the connector first, though (so I can figure out if it will work in my project.) I don't want to build a multiprocessor machine, acutally, but these modules would be great CPUs for a linux PDA. Could one be configured to host the PCI bus? This way, I could build a carrier card that had my IO on it and use the SO-DIMM as my CPU daughter card. It should be possible. I'm not sure if they have made a jumber to configure the board as a central function however, if they have done this, the answer is yes. greetings, Mark. From hanno@lv.parnu.ee Fri, 29 Jan 1999 09:27:19 -0500 Date: Fri, 29 Jan 1999 09:27:19 -0500 From: Hanno Saks hanno@lv.parnu.ee Subject: About Extreme CD Hi! Can I find somewhere Extreme-Linux with kernel at least 2.0.36 or what to do if 2.0.33 kernel did not recognized my PCI but I want to install Extreme? Hanno Saks LAN Admin Parnu Town Government Development board Estonia From thorpe@cerco.ups-tlse.fr Fri, 29 Jan 1999 10:25:17 -0500 Date: Fri, 29 Jan 1999 10:25:17 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea >On the bridges, certainly AFAIR the three currently evaluated choices >were: > >a) Two zero-latency bridges. >b) One of the DEC transparent bridges >c) One of the DEC non-transparent bridges ("drawbridge") > >a was the cheapest option, and was fairly easy to program. b was the >easiest to program, but might not be as efficient as a and the lead >times are longer. c was cool in that it would offer a 64-bit pci >capability, but would be much harder to program. > >I don't know how Simpson made up his mind in the end. > > Neil > Neil, Not sure if I understand what a non-transparent bridge is. I guess that means that if a CPU on one board wants to write data in the memory of a CPU on another board, then it won't be able to do it by just directly copying data to a particular address, but rather it would have to use the intricacies of I20 messaging - is that right? Any idea how much extra latency is added by using a more standard DEC bridge, as opposed to the zero-latency bridges that HiNT are producing? Is it one PCI clock (i.e. 30 ns) or is there normally even more delay? In any case, it seems to me that in the short term, Gareth would be better to go for a) or b), since what we all really want is to have something to work with a.s.a.p. I think we can assume that there won't be too many people wanting to stick the boards into 64 bit PCI just yet, and in anycase, since the CPU modules can be transfered to a more sophisticated PCI bus architecture later, nobody would be wasting much money by going for a simpler 32-bit design now. Simon __________________ Simon Thorpe Centre de Recherche Cerveau et Cognition 133, route de Narbonne 31062 Toulouse France Tel 33 (0)5 62 17 28 03. Fax 33 (0)5 62 17 28 09 http://www.cerco.ups-tlse.fr/~modele/index.html __________________ From m.j.s.vandoesburg@student.utwente.nl Fri, 29 Jan 1999 11:58:23 -0500 Date: Fri, 29 Jan 1999 11:58:23 -0500 From: Mark van Doesburg m.j.s.vandoesburg@student.utwente.nl Subject: Beoboard idea >On the bridges, certainly AFAIR the three currently evaluated choices >were: > >a) Two zero-latency bridges. >b) One of the DEC transparent bridges >c) One of the DEC non-transparent bridges ("drawbridge") > >a was the cheapest option, and was fairly easy to program. b was the >easiest to program, but might not be as efficient as a and the lead >times are longer. c was cool in that it would offer a 64-bit pci >capability, but would be much harder to program. > >I don't know how Simpson made up his mind in the end. > > Neil I would like to know the part numbers of the bridges he is considering. Especially the p# of the non-transparent bridge, too see how difficult it would be to program and if it doesn't conflict with the functions I want to implement. (Would it, for example, allow shared memory segments) greetings, Mark. From juels@arep.med.harvard.edu Fri, 29 Jan 1999 12:32:16 -0500 Date: Fri, 29 Jan 1999 12:32:16 -0500 From: Philip I. Juels juels@arep.med.harvard.edu Subject: About Extreme CD Are you having the same problems as I? I have Asus P2B motherboards and when I install Extreme.Linux, I have problems with Disk Druid...it tells me I haven't set up a swap partition depite the fact that I did. Philip Juels philip_juels@harvard.edu Hanno Saks wrote: > Hi! > > Can I find somewhere Extreme-Linux with kernel at least 2.0.36 or what > to do if 2.0.33 kernel did not recognized my PCI but I want to install > Extreme? > > Hanno Saks > LAN Admin > Parnu Town Government > Development board > Estonia From gareth@simtec.demon.co.uk Fri, 29 Jan 1999 15:03:23 -0500 Date: Fri, 29 Jan 1999 15:03:23 -0500 From: Gareth Simpson gareth@simtec.demon.co.uk Subject: Beoboard idea On Fri, 29 Jan 1999 17:51:42 +0100, m.j.s.vandoesburg@student.utwente.nl said: > >On the bridges, certainly AFAIR the three currently evaluated choices > >were: > > > >a) Two zero-latency bridges. > >b) One of the DEC transparent bridges > >c) One of the DEC non-transparent bridges ("drawbridge") > > > >a was the cheapest option, and was fairly easy to program. b was the > >easiest to program, but might not be as efficient as a and the lead > >times are longer. c was cool in that it would offer a 64-bit pci > >capability, but would be much harder to program. > > > >I don't know how Simpson made up his mind in the end. > > > > Neil > > I would like to know the part numbers of the bridges he is considering. > Especially the p# of the non-transparent bridge, too see how difficult > it would be to program and if it doesn't conflict with the functions I > want to implement. (Would it, for example, allow shared memory segments) > > greetings, > > Mark. > Unfortunately the ideal dual-bus Hint bridge chip is not due to ship samples till June/July so we have to use a regular device. On evaluation, their zero delay bridge does not have transaction buffers so in a multiple CPU application, the bus will block completely if the destination bus is busy. We then started looking at the second generation Digital devices that offer both deep transaction buffers and the ability to buffer up to three independant transactions. At the same time, we decided that at only a slight increase in cost, the move to a 64bit priamary bus would improve communication bandwidth between nodes when fitted to a 64 bit backplane. 32 and 64 bit cards can still freely communicate with eachother. As nobody has given me a good case why we should/shouldn't use a transparent bridge, we had decided to use the DEC 21554 64 bit non-transparent bridge. If there is a case for using transparent devices then we will use the 64/32bit DEC 21153. The '53 requires that it is configured by an external host processor and won't allow the onboard CPUs to configure their own node. So a passive backplane full of cards is not possible. There would need to be an additional host CPU on the primary bus to configure everything. The '554 requires one local (or default EEPROM config) to be set up from the local secondary side. The device then appears as a single PCI address block. Address translation is also possible via the bridge. As any external config. cycles are blocked, it is not posible for an external CPU to access any of the local local registers. Could be useful if a 'black box' approach is required. This type of bridge allows config cycles to pass out, enabling any CPU on a node to probe the external PCI bus for devices. Comments welcome, as it's still not too late to make the change as we havn't quite started the layout yet. Gareth. As an aside, here is an update on the production CPU modules: Good news: We've measured the power dissipation of our 64Mb modules and at 50MHz memory and 230MHz cache clock, it consumes just unfer 2W. The boards have also been run with memory at 64MHz and cpu at 279MHz with an increase in disiapation to 2.4W. With these results, we will be configuring the default memory clock to be 60MHz instead of the current 50. Not so good news: There has been a slight delay on production CPU modules as we made a subtle mistake on routing that would prevent in-circuit reprogramming of the onboard FLASH device. While this did not affect the running of the module and would still enable the ROM to be self-programmed, it would have made host reprogramming the FLASH chip impossible. The circuit has been corrected and the new outer layers are to be sent for manufacture for Monday. Boards are currently drilled and waiting for the new outer layers to be completed. We hope this has only delayed finshed boards by only a few days. -- Gareth... --------- Gareth Simpson Email: gareth@simtec.demon.co.uk Design Engineer, Tel: +44 1772 812863 / 816485 Simtec Electronics. Fax: +44 1772 816426 From thorpe@cerco.ups-tlse.fr Sat, 30 Jan 1999 04:35:42 -0500 Date: Sat, 30 Jan 1999 04:35:42 -0500 From: Simon Thorpe thorpe@cerco.ups-tlse.fr Subject: Beoboard idea Hey Gareth, great to hear from you!! >Gareth Simpson wrote: >Unfortunately the ideal dual-bus Hint bridge chip is not due to ship samples >till June/July so we have to use a regular device. On evaluation, their >zero delay bridge does not have transaction buffers so in a multiple >CPU application, the bus will block completely if the destination bus is >busy. So the zero latency bridges are definitely out then. There may nevertheless be quite a few people who would like to see carrier board with the 3-way HiNT bridge when you can get hold of it - it would after all effectively double the communications bandwidth for local processor-to-processor transfers >We then started looking at the second generation Digital devices that offer >both deep transaction buffers and the ability to buffer up to three >independant transactions. At the same time, we decided that at only a slight >increase in cost, the move to a 64bit priamary bus would improve communication >bandwidth between nodes when fitted to a 64 bit backplane. 32 and 64 bit >cards can still freely communicate with eachother. Excellent - I hadn't realised that it could be so simple. Presumably it will need some extra programming though (from Mark, Neil and Mark?). I've got three free 64 bit PCI slots in my new G3 Mac - and as soon as the LinuxPPC people have got a version up and running I could try it out. Since the new Macs cost less than $1500, they could actually end up being quite a nice host. Both the Dec chips can handle up to 9 PCI devices on the secondary side. The only problem is that I can't see how you're going to get 9 CPU modules on the PCI carrier board ;-). Oh all right, I'll go for 8 (see comments on power consumption) :-) >As nobody has given me a good case why we should/shouldn't use a transparent >bridge, we had decided to use the DEC 21554 64 bit non-transparent bridge. >If there is a case for using transparent devices then we will use the 64/32bit >DEC 21153. > >The '53 requires that it is configured by an external host processor and >won't allow the onboard CPUs to configure their own node. So a passive >backplane full of cards is not possible. There would need to be an >additional host CPU on the primary bus to configure everything. I honestly can't see why this could be thought of as a real advantage - seems more like a disadvantage to me. >The '554 requires one local (or default EEPROM config) to be set up from >the local secondary side. Presumably, you could have just one EEPROM on the main carrier board.... >The device then appears as a single PCI >address block. Address translation is also possible via the bridge. >As any external config. cycles are blocked, it is not posible for an >external CPU to access any of the local local registers. Could be >useful if a 'black box' approach is required. This type of bridge >allows config cycles to pass out, enabling any CPU on a node to probe the >external PCI bus for devices. This sounds excellent to me. I can quite imagine some people wanting to "black box" the boards (I might even make use of it myself...). On the other hand, there would still be no problem for those who want a truly Open system. >Comments welcome, as it's still not too late to make the change as we havn't >quite started the layout yet. > >Gareth. > >As an aside, here is an update on the production CPU modules: > >Good news: > >We've measured the power dissipation of our 64Mb modules and at 50MHz memory >and 230MHz cache clock, it consumes just unfer 2W. The boards have also >been run with memory at 64MHz and cpu at 279MHz with an increase in >disiapation to 2.4W. With these results, we will be configuring the >default memory clock to be 60MHz instead of the current 50. This is really nice. Since PCI slots are rated at 25 W, this means that you can get 8 CPU modules on without needing any extra power supply cables. It also means that packing 32 processors into a standard PC case would still only use a maximum of about 75 Watts - easily within the load capabilities of even the simplest sub $1000 PC. Is there any hope of allowing overclocking to 279 MHz? Do you think that this would be too dangerous? Any chance that it might be possible to vary CPU clock speed with a setting on the carrier board? Doesn't the clock frequency depend on the voltage applied to the VCO?? - if so, then one could maybe imagine that users could tweak clock speed to get the best performance on their particular problem... >Not so good news: > >There has been a slight delay on production CPU modules as >we made a subtle mistake on routing that would prevent in-circuit >reprogramming of the onboard FLASH device. While this did not affect >the running of the module and would still enable the ROM to be >self-programmed, it would have made host reprogramming the FLASH >chip impossible. The circuit has been corrected and the new outer >layers are to be sent for manufacture for Monday. Boards are currently >drilled and waiting for the new outer layers to be completed. >We hope this has only delayed finshed boards by only a few days. > Programmability for the FLASH memory is definitely essential. I can certainly hang on for a few more days.... Cheers, thanks and keep up the good work Simon From efinch@cais.com Sat, 30 Jan 1999 19:28:08 -0500 Date: Sat, 30 Jan 1999 19:28:08 -0500 From: Ed Finch efinch@cais.com Subject: How to get the local IP address after booting with DHCP? Greetings! I'm in the process of building a Beowulf cluster. I would like to boot the slave nodes via DHCP served from the master node. I've got it all working, but the client doesn't know its own IP address after boot. The address is stored in a file under /etc/dhcp..., but /etc/hosts isn't updated. If I configure /etc/resolv.conf to check files first, for example, and try an nslookup command on the local machine's name, it gives an error that the name can't be resolved. How is this supposed to work? Best regards, Ed -- Q: Why do PCs have a reset button on the front? A: Because they are expected to run Microsoft operating systems. From rajkumar@dgs.monash.edu.au Sun, 31 Jan 1999 04:55:03 -0500 Date: Sun, 31 Jan 1999 04:55:03 -0500 From: Rajkumar Buyya rajkumar@dgs.monash.edu.au Subject: Task Force on Cluster Computing (and CFP for session at PDPTA'99) Dear Colleague, I am happy to inform you the following: --- 1. Formation of, IEEE CS Task Force of Cluster Computing (TFCC) Please see, TFCC website: http://www.dgs.monash.edu.au/~rajkumar/tfcc/ for more details. If you are interested in knowing more about Task Force activities, please subscribe to TFCC mailing list. See TFCC website for further information. Looking forward for your active participation in the TFCC activities. -- 2. I have organized a Technical Session on: Cluster Computing (URL: http://www.dgs.monash.edu.au/~rajkumar/cctea.html) at PDPTA'99 conference to be held in Las Vegas. I have enclosed its CFP for your kind consideration. If possible, please distribute this CFP to your colleagues interested in Cluster Computing. Looking forward receiving your research paper for this session. --- Best wishes, Sincerely, Rajkumar Buyya - http://www.dgs.monash.edu.au/~rajkumar/ ----------------------------------------------------------------------------- TECHNICAL SESSION Cluster Computing Technologies, Environments, and Applications (CC-TEA) URL: http://www.bigfoot.com/~buyya/cctea.html http://www.dgs.monash.edu.au/~rajkumar/cctea.html in 1999 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'99) http://www.jhpc.org/pdpta June 28 - July 1, 1999 Monte Carlo Resort, Las Vegas, Nevada, USA Call for Papers In recent years, the availability of high-speed networks and high-performance microprocessors is making networks of computers an appealing vehicle for cost-effective parallel computing. Clusters/networks of computers (workstations/PCs/SMPs) built using commodity hardware and software are playing a major role in redefining the concept of supercomputing. As a whole, clusters are becoming an alternative to MPPs and supercomputers in many areas of application. The focus of this special session will be on both hardware and software aspects of computing on clusters. Topics of interest include, but are not limited to: Cluster Hardware (Cluster of PCs, Workstations, or SMPs) High Perormance Communication Networks and Interfaces Light Weight Communication Protocols Cluster Middleware Single System Image Infrastructure System Availabilty Infrastructure Issues in Building Scalable Services File Systems and Parallel I/O Job and Resource Management Data Distribution and Load Balancing Programming Paradigms/Environment for Clusters Message Passing Systems such as MPI and PVM for Clusters Problem Solving Environments for Clusters Tools for Operating and Managing Clusters Java for High Performance Computing Algorithms for Solving Problems on Clusters Scientific, Engineering, and Commercial Applications on Clusters Submission should include authors names, affiliations, addresses, fax and phone numbers, email addresses, on the cover page. Submission implies the willingness of at least one of the authors to register and present the paper. Please submit full paper not exceeding 5 (or 7) single spaced pages in file of the paper (preferably viewable by ghostview); also send a separate email with authors' names, addresses, title and abstract of the paper. Hard copies should be sent only if electronic submission is not possible. All papers selected for this session will be published in the PDPTA'99 conference proceedings. Please send full paper for consideration to this technical session to: Session Organizer and Chair: Rajkumar Buyya School of Computing Science and Software Engineering Monash University Room No. 130, Bld No. 63, Clayton Campus Melbourne, Vic. 3168, Australia Office phone: +61-3-9905 1502 Office fax: +61-3-9905 3574 Email: rajkumar@dgs.monash.edu.au / buyya@bigfoot.com WWW: http:www.dgs.monash.edu.au/~rajkumar / http://www.bigfoot.com/~buyya Important Dates: Draft Papers due on: March 15, 1999 Notification of Acceptance: April 5, 1999 Camera Ready Papers and Preregistration due on: May 1, 1999 PDPTA'99 Conference: June 28 - July 1, 1999 ------------------------------------------------------------------------ For information regarding the PDPTA'99 Conference, contact: Prof. Hamid R. Arabnia General Chair, PDPTA'99 The University of Georgia Department of Computer Science 415 Graduate Studies Research Center Athens, Georgia 30602-7404, USA Tel: (706) 542-3480 Fax: (706) 542-2966 email: hra@cs.uga.edu Authors who would like to submit papers to be considered for other technical sessions should mail their papers to Hamid R. Arabnia by March 1, 1999. ----------------------------------------------------------------------------- From rajkumar@dgs.monash.edu.au Sun, 31 Jan 1999 05:01:54 -0500 Date: Sun, 31 Jan 1999 05:01:54 -0500 From: Rajkumar Buyya rajkumar@dgs.monash.edu.au Subject: TF on Cluster Computing (and CFP for session at PDPTA'99) Dear Colleague, I am happy to inform you the following: --- 1. Formation of, IEEE CS Task Force of Cluster Computing (TFCC) Please see, TFCC website: http://www.dgs.monash.edu.au/~rajkumar/tfcc/ for more details. If you are interested in knowing more about Task Force activities, please subscribe to TFCC mailing list. See TFCC website for further information. Looking forward for your active participation in the TFCC activities. -- 2. I have organized a Technical Session on: Cluster Computing (URL: http://www.dgs.monash.edu.au/~rajkumar/cctea.html) at PDPTA'99 conference to be held in Las Vegas. I have enclosed its CFP for your kind consideration. If possible, please distribute this CFP to your colleagues interested in Cluster Computing. Looking forward receiving your research paper for this session. --- Best wishes, Sincerely, Rajkumar Buyya - http://www.dgs.monash.edu.au/~rajkumar/ ----------------------------------------------------------------------------- TECHNICAL SESSION Cluster Computing Technologies, Environments, and Applications (CC-TEA) URL: http://www.bigfoot.com/~buyya/cctea.html http://www.dgs.monash.edu.au/~rajkumar/cctea.html in 1999 International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA'99) http://www.jhpc.org/pdpta June 28 - July 1, 1999 Monte Carlo Resort, Las Vegas, Nevada, USA Call for Papers In recent years, the availability of high-speed networks and high-performance microprocessors is making networks of computers an appealing vehicle for cost-effective parallel computing. Clusters/networks of computers (workstations/PCs/SMPs) built using commodity hardware and software are playing a major role in redefining the concept of supercomputing. As a whole, clusters are becoming an alternative to MPPs and supercomputers in many areas of application. The focus of this special session will be on both hardware and software aspects of computing on clusters. Topics of interest include, but are not limited to: Cluster Hardware (Cluster of PCs, Workstations, or SMPs) High Perormance Communication Networks and Interfaces Light Weight Communication Protocols Cluster Middleware Single System Image Infrastructure System Availabilty Infrastructure Issues in Building Scalable Services File Systems and Parallel I/O Job and Resource Management Data Distribution and Load Balancing Programming Paradigms/Environment for Clusters Message Passing Systems such as MPI and PVM for Clusters Problem Solving Environments for Clusters Tools for Operating and Managing Clusters Java for High Performance Computing Algorithms for Solving Problems on Clusters Scientific, Engineering, and Commercial Applications on Clusters Submission should include authors names, affiliations, addresses, fax and phone numbers, email addresses, on the cover page. Submission implies the willingness of at least one of the authors to register and present the paper. Please submit full paper not exceeding 5 (or 7) single spaced pages in file of the paper (preferably viewable by ghostview); also send a separate email with authors' names, addresses, title and abstract of the paper. Hard copies should be sent only if electronic submission is not possible. All papers selected for this session will be published in the PDPTA'99 conference proceedings. Please send full paper for consideration to this technical session to: Session Organizer and Chair: Rajkumar Buyya School of Computing Science and Software Engineering Monash University Room No. 130, Bld No. 63, Clayton Campus Melbourne, Vic. 3168, Australia Office phone: +61-3-9905 1502 Office fax: +61-3-9905 3574 Email: rajkumar@dgs.monash.edu.au / buyya@bigfoot.com WWW: http:www.dgs.monash.edu.au/~rajkumar / http://www.bigfoot.com/~buyya Important Dates: Draft Papers due on: March 15, 1999 Notification of Acceptance: April 5, 1999 Camera Ready Papers and Preregistration due on: May 1, 1999 PDPTA'99 Conference: June 28 - July 1, 1999 ------------------------------------------------------------------------ For information regarding the PDPTA'99 Conference, contact: Prof. Hamid R. Arabnia General Chair, PDPTA'99 The University of Georgia Department of Computer Science 415 Graduate Studies Research Center Athens, Georgia 30602-7404, USA Tel: (706) 542-3480 Fax: (706) 542-2966 email: hra@cs.uga.edu Authors who would like to submit papers to be considered for other technical sessions should mail their papers to Hamid R. Arabnia by March 1, 1999. -----------------------------------------------------------------------------