[Beowulf] [EXTERNAL] Re: Have machine, will compute: ESXi or bare metal?

[Skylar Thompson]

Thanks for the pointers, Jim! I had this fear that there wasn't something I
was understanding with zeroconf addressing so it's good to know that other
people have had similar ideas. :)

With BCCD, we have a huge variety of NICs that we've supported over the
years, and even have had people run clusters over wifi (sometimes even
intentionally) though obviously a shared medium is bad for performance and latency.

On Wed, Feb 12, 2020 at 09:13:30PM +0000, Lux, Jim (US 337K) wrote:
> I've used zeroconf quite effectively on a couple clusters of beagleboards.  If you do it, then you can let the nodes use DHCP to get their IP addresses, which is handy if you're sharing a WiFi network, for instance. 
> 
> On the other hand, for a "training experience", having to go through the process of manually assigning IP addresses, and node identifiers (hostname) and keeping them all straight is a useful thing.  Having done that, you really appreciate zeroconf/bonjour and DHCP.
> 
> My two configurations are:
> Head node is a Macbook running OS X
> Configuration 1: Macbook using Wireless to connect to the "internet" and wired to connect to the cluster, which is 4 BeagleBoard Green with wired ethernet.
> Configuration 2: Macbook using wireless to connect to the "internet" and wired to an 802.11b/g access point, then the cluster is 4 BeagleBoard Green Wireless.
> 
> In both cases, I wind up setting up a bridge from the cluster to the outside world via the macbook, so that I can run things like "apt get" on the nodes to install software.
> I use a combination of screen, ssh, and pdsh to work with the nodes, and scp (with pdsh) to move files around.
> 
> Note that with beagles (and Rpis) you usually use a "network over USB" to get it up originally (the gadget interface)
> 
> On 2/11/20, 7:57 PM, "Beowulf on behalf of Skylar Thompson" <beowulf-bounces at beowulf.org on behalf of skylar.thompson at gmail.com> wrote:
> 
>     On Tue, Feb 11, 2020 at 06:25:24AM +0800, Benson Muite wrote:
>     > 
>     > 
>     > On Tue, Feb 11, 2020, at 9:31 AM, Skylar Thompson wrote:
>     > > On Sun, Feb 09, 2020 at 10:46:05PM -0800, Chris Samuel wrote:
>     > > > On 9/2/20 10:36 pm, Benson Muite wrote:
>     > > > 
>     > > > > Take a look at the bootable cluster CD here:
>     > > > > http://www.littlefe.net/
>     > > > 
>     > > > From what I can see BCCD hasn't been updated for just over 5 years, and the
>     > > > last email on their developer list was Feb 2018, so it's likely a little out
>     > > > of date now.
>     > > > 
>     > > > http://bccd.net/downloads
>     > > > 
>     > > > http://bccd.net/pipermail/bccd-developers/
>     > > > 
>     > > > On the other hand their TRAC does list some ticket updates a few months ago,
>     > > > so perhaps there are things going on but Skylar needs more hands?
>     > > > 
>     > > > https://cluster.earlham.edu/trac/bccd-ng/report/1?sort=created&asc=0&page=1
>     > > 
>     > > Wow, I had no idea people on the Beowulf list were still thinking of BCCD.
>     > > :)
>     > > 
>     > > I've been working on a major BCCD update for a while now (modern Debian,
>     > > better node auto-detection) but a combination of life interference and a
>     > > shift in focus for the project to curriculum development has slowed me
>     > > down.
>     > > 
>     > > At the end of the day, BCCD has three has three main goals:
>     > > 
>     > > 1. Non-destructive in its default mode
>     > > 2. Simple ("just press enter")
>     > > 3. Ready with pedagogically-useful ("validated, verified, and accredited")
>     > > curriculum modules
>     > > 
>     > > One thing I'm hoping can come out of this major update is to decouple the BCCD
>     > > from the underlying distribution, since that's been a barrier for some
>     > > people in using BCCD. That's just an aspiration right now, but we'll see
>     > > where it goes.
>     > 
>     > Can you give more details on how you expect to decouple it?
>     
>     There's a couple things that are tightly integrated with the init process
>     now:
>     
>     * Network setup - prompts the user for information (which NIC to run on,
>       whether to use DHCP to assign addresses, etc.), detects other BCCD
>       systems on the network, etc.
>     
>     * SSH public key broadcast (pkbcast) - needs to run after a user logs in to
>       ensure that authorized_keys is setup on other participating systems.
>     
>     The network setup in particular is a challenge in the systemd world, since
>     getting STDIN from systemd-invoked processes is not trivial. We've also had
>     some users wanting better integration between networking and desktop
>     applications, which pushed me to try to make use of the existing
>     systemd/networkd toolchain rather than rolling our own tooling.
>     
>     Right now the challenge is the node auto-detection, though I'm hoping that
>     we might be able to use mDNS and zeroconf-assigned addressing rather than
>     depending on custom DHCP tags which have been problematic in the past.
>     zeroconf might also mitigate the biggest problem we have in workshops:
>     someone jumping the gun and starting up a head node before we're ready to
>     go.
>     
>     While I'm not the biggest fan of systemd, it does have the potential to
>     allow us to get away from custom scripts and use functionality common
>     across more than one distribution.
>     
>     -- 
>     Skylar
>     _______________________________________________
>     Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>     To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>     
> 

-- 
Skylar