<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">-- <br>Olli-Pekka Lehto<br>Development Manager, Computing Platforms </div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">CSC - IT Center for Science Ltd.<br>E-Mail: <a href="mailto:olli-pekka.lehto@csc.fi">olli-pekka.lehto@csc.fi</a> // Tel: +358 50 381 8604 // skype: oplehto // twitter: @ople<br></div></div>
</div>
<br><div><div>On 06 Nov 2015, at 19:15, Joe Landman <<a href="mailto:landman@scalableinformatics.com">landman@scalableinformatics.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">On 11/04/2015 04:06 AM, <a href="mailto:tegner@renget.se">tegner@renget.se</a> wrote:<br><blockquote type="cite">Hi,<br><br>I need to set up a diskless cluster. Some years ago I did this using<br>Perceus, <a href="http://www.perceus.org/">http://www.perceus.org/</a>. Wored nicely, but since I know there<br>are several other systems around; e.g., Warewulf,<br><a href="http://warewulf.lbl.gov/trac">http://warewulf.lbl.gov/trac</a> or<br><a href="http://www.admin-magazine.com/HPC/Articles/Warewulf-Cluster-Manager-Master-and-Compute-Nodes">http://www.admin-magazine.com/HPC/Articles/Warewulf-Cluster-Manager-Master-and-Compute-Nodes</a>,<br>and Onesis, <a href="http://onesis.org/">http://onesis.org/</a> I just wanted to check what people are<br>using these days.<br><br>is there one system in particular which is more actively developed? Or<br>do people tend to use different, more "basic" tools these days?<br></blockquote><br>Open source: Warewulf is your go-to system. Does clustering, very well. Xcat is out there and in use (actually Xcat2 from what I see/hear). Onesis was neat when I played with it a while ago, but I am not sure how up to date it is.<br><br>Closed source: Bright's system, probably a few others<br><br>We took the track of developing our own, as we needed much greater flexibility than cluster distros allowed. This was also due, in part, to linux distros and their (famously) broken init/startup bits, not to mention how they handle networking, and many other things. Don't get me started on their kernels.<br><br></blockquote></div><br><div><br></div><div>Yeah.. At some point you tend to hit some sort of ceiling with the cluster distros and start to work around the limitations manually. At some point you then reach a point where you wonder what was the point of going with a cluster distro anyway. </div><div><br></div><div>Warewulf has worked quite well as it does the basic stuff pretty well and then gets out of the way. However evan that has had it’s limitations and idiosyncracies.</div><div><br></div><div>Now we’re working on a new stack which will likely consists of something like: </div><div>- Ansible for config management of the base system</div><div>- EasyBuild+lmod for config management of the user software stack</div><div>- collectd+Graphite+Grafana for monitoring</div><div>- ELK for log analytics and correlation</div><div><br></div><div>There are quite a lot of work-in-progress repos related to this in our GitHub: <a href="https://github.com/CSC-IT-Center-for-Science">https://github.com/CSC-IT-Center-for-Science</a></div><div><br></div><div>Provisioning is a bit of an open issue. It would be nice to have something that the flexibility to work not only for HPC but also for general-purpose server deployments. A simple Kickstart+PXE is a starting poiint now there’s some promising tools coming from the scale-out world such as Razor, Foreman, Crowbar etc. It would be interesting to hear if people have experience in using these, especially on a reasonable scale (~1000 nodes). </div><div><br></div><div>O-P</div><div><br></div><div><br></div><div><br></div></body></html>