diskless nodes? (was Re: Xbox clusters?)

[Bogdan Costescu]

On Thu, 6 Dec 2001, Velocet wrote:

> if you wanted it to be faster, just add more networks to the server...
> ... LVD 160 controllers are PRETTY damn fast. Put it all on a PCI-X 
> board 64bit/133MHz pci ...
> ...then just start using multiple servers...
> Hell even put 2 or 3 NICs in each node...

Maybe I'm over-looking something, but you do all these to replace one 
inexpensive IDE disk per node ? Here in Germany, I can get a resonably 
fast and big IDE disk for 2-3 times the price of a (3Com, of course 8-)) 
100Mbit network card; for multiple network cards per node, you also need 
to get multiple/big switches and a fat pipe (be it 100Mbit bonded or 
Gigabit - bonded or not) for the NFS server, which needs to still provide 
a pretty large storage capacity. Err, I think that I'll stick to local 
disks...

[Maybe from another message] The fact that you are saving over NFS 
temporary files which are only supposed to be accessed by the node which 
created them is in fact a good thing. If other nodes would need to access 
them as well, you get into coherency problems, want to disable client 
caching and loose performance big time; however, in this case having the 
temporary files on local disks or on a NFS server doesn't make much 
difference.

> I'd go insane configuring things if I had disk-full nodes.

I think that you are sticking together 2 things: having local disks and 
booting/using root FS from them. One of our clusters (which evolved from a 
diskless state to an almost disk-full state) boots over network (RARP, 
root FS on the main node), while the local disks are available exclusively 
for scratch/temporary files. This way the network (one 100Mbit card per 
node) is used almost exclusively for message passing; accesing the root FS 
is just a negligible part of the traffic and input and output files (dealt 
with only on the first node of a job) are not so big either. And BTW, we 
are doing things not very different from what you do: mainly MM (CHARMM) 
and QM/MM (CHARMM + GAMESS, NWChem).

However, I'm not saying that our solution is the best - just that it fits 
our needs. That's all that matters, right ?

-- 
Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu at IWR.Uni-Heidelberg.De