<HTML><BODY style="word-wrap: break-word; -khtml-nbsp-mode: space; -khtml-line-break: after-white-space; "><BR><DIV><DIV>On Dec 14, 2006, at 2:33 PM, Donald Becker wrote:</DIV><BR class="Apple-interchange-newline"><BLOCKQUOTE type="cite"><DIV>On Sat, 9 Dec 2006, Joe Landman wrote:</DIV> <BLOCKQUOTE type="cite"><DIV>Guy Coates wrote:</DIV> <BLOCKQUOTE type="cite"><DIV>At what node count does the nfs-root model start to break down? Does anyone</DIV><DIV>have any rough numbers with the number of clients you can support with a generic</DIV><DIV>linux NFS server vs a dedicated NAS filer?</DIV> </BLOCKQUOTE><DIV><BR></DIV><DIV>If you use warewulf or the new perceus variant, it creates a ram disk</DIV><DIV>which is populated upon boot. Thats one of the larger transients. Then</DIV><DIV>you nfs mount applications, and home directories. I haven't looked at</DIV><DIV>Scyld for a while, but I seem to remember them doing something like this.</DIV> </BLOCKQUOTE><DIV><BR></DIV><DIV>I forgot to finish my reply to this message earlier this week. Since I'm </DIV><DIV>in the writing mood today, I've finished it.</DIV><DIV><BR></DIV><DIV><BR></DIV><DIV>Just when were getting past "diskless" being being misinterpreted as</DIV><DIV>"NFS root"...</DIV></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>I prefer the term "stateless" to describe Warewulf and Perceus provisioning model (stateless installs may have local disks for swap and scratch/data space).</DIV></DIV><DIV><BR><BLOCKQUOTE type="cite"><DIV>RAMdisk Inventory</DIV><DIV><BR></DIV><DIV>We actually have five (!) different types of ramdisks over the system</DIV><DIV>(see the descriptions below). But it's the opposite of the Warewulf</DIV><DIV>approach. Our architecture is a consistent system model, so we</DIV><DIV>dynamically build and update the environment on nodes. Warewulf-like</DIV><DIV>ramdisk system only catch part of what we are doing:</DIV></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>The stateless provisioning model has a very different goal then Scyld's Bproc implementation and thus a comparison is misleading.</DIV><BR><BLOCKQUOTE type="cite"><DIV><BR></DIV><DIV>The Warewulf approach</DIV><DIV> - Uses a manually selected subset distribution on the compute node </DIV><DIV>ramdisk.</DIV><DIV> While still very large, it's never quite complete. No matter how </DIV><DIV>useless</DIV><DIV> you think some utility is, there is probably some application out </DIV><DIV>there</DIV><DIV> that depends on it.</DIV></BLOCKQUOTE><BLOCKQUOTE type="cite"><DIV> - The ramdisk image is very large and it has to be completely downloaded </DIV><DIV>at</DIV><DIV> boot time just when the server is extremely.</DIV><DIV> - Supplements the ramdisk with NFS, combining the problems of both.(*) </DIV><DIV>The</DIV><DIV> administrator and users to learn and think about how both fail.</DIV></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>I suppose that under some circumstances these observations maybe applicable, but with that said... I have not heard of any of the Warewulf or stateless Perceus *users* sharing these opinions.</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>Regarding the various cluster implementations: there is not one size fits all, and all of the toolkits and implementation methods have tradeoffs. Rather then point out the problems in the various cluster solutions, I would just like to reiterate that people should evaluate what fits their needs best and utilize what works best for them in their environment.</DIV><BR><BLOCKQUOTE type="cite"><DIV><BR></DIV><DIV>(*1) That said, combining a ramdisk root with NFS is still far more</DIV><DIV>scalable and somewhat more robust than using solely NFS. With careful</DIV><DIV>administration most of the executables will be on the ramdisk, allowing</DIV><DIV>the server to support more nodes and reducing the likelihood of</DIV><DIV>failures.</DIV></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV><DIV>Well said, I agree. There are also some general policies that will work reasonably well and doesn't require much system specific tuning (if any).</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV><BR class="khtml-block-placeholder"></DIV><BLOCKQUOTE type="cite"><DIV>The phrase "careful administration" should be read as "great for demos,</DIV><DIV>and when the system is first configured, but degrades over time". The</DIV><DIV>type of people that leap to configure the ramdisk properly the first</DIV><DIV>time are generally not the same type that will be there for long-term</DIV><DIV>manual tuning. </DIV></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV>What an odd way of looking at it.</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV>Great for demos but not for a long term solution because it degrades over time???<DIV><BR class="khtml-block-placeholder"></DIV><DIV>If you are referring to careless people or admins mucking up the virtual node file systems I think the physical muckage would be the least of the concerns when these people have root. Not to mention blaming the cluster toolkit or provisioning model for allowing the users the freedom and flexibility to do what they want is misidentifying the problem.</DIV><BR><BLOCKQUOTE type="cite"><DIV> Either they figure out why we designed around dynamic,</DIV><DIV>consistent caching and re-write, or the system will degrade over time.</DIV></BLOCKQUOTE><DIV><BR class="khtml-block-placeholder"></DIV>Why would a system not built around "dynamic consistent caching and re-write" degrade over time?</DIV><DIV><BR class="khtml-block-placeholder"></DIV><DIV><DIV><BR class="khtml-block-placeholder"></DIV></DIV>Many thanks!<DIV>Greg<BR><DIV> <SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><SPAN class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: 0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; white-space: normal; widows: 2; word-spacing: 0px; "><DIV>--</DIV><DIV>Greg Kurtzer</DIV><DIV><A href="mailto:gmk@runlevelzero.net">gmk@runlevelzero.net</A></DIV><DIV><BR class="khtml-block-placeholder"></DIV><BR class="Apple-interchange-newline"></SPAN></SPAN> </DIV><BR></DIV></BODY></HTML>