[Beowulf] Your thoughts on the latest RHEL drama?

[Andrew M.A. Cater]

On Mon, Jun 26, 2023 at 02:27:23PM -0400, Prentice Bisbal via Beowulf wrote:

> Beowulfers,
> 
> By now, most of you should have heard about Red Hat's latest to eliminate
> any competition to RHEL. If not, here's some links:
> 
> Red Hat's announcement:
> https://www.redhat.com/en/blog/furthering-evolution-centos-stream
> <https://www.redhat.com/en/blog/furthering-evolution-centos-stream>
> 
> Alma Linux's response:
> https://almalinux.org/blog/impact-of-rhel-changes/
> 
> Rocky Linux's response:
> https://rockylinux.org/news/2023-06-22-press-release/
> 
> Software Freedom Conservancy's anaylsis of the situation:
> https://sfconservancy.org/blog/2023/jun/23/rhel-gpl-analysis/
> <https://sfconservancy.org/blog/2023/jun/23/rhel-gpl-analysis/>
> 
> I'm writing to get your thoughts on this situation, as well as see what
> plans of action you are considering moving forward.
> 

HPC is a *very* small world: Moving beyond the hand built Beowulf clusters
this list started with, the top 500 have thousands of nodes. All run Linux
- many of them still run Red Hat but mediated by the cluster hardware vendors,
system integrators, specialist interconnect providers - or a one-stop shop.

HPE (ex-Cray) will integrate a whole computer, tweak the linux, add secret
sauce and performance tuning, for example, with various sub teams and field
engineers doing this.

The large clusters will have their own engineers - if you were to ask a
Red Hat certified architect to deal with a large supercomputer, he might well
be sunk (and that's before you consider catering to the specialist requirements
of the user communities who want to run meteo forecasting models or whatever).

> Here are my thoughts:
> 
> This is Red Hat biting the hands that feed them. Red Hat went from a small
> company operating out of a basement to a large global company thanks to
> open-source software. My first exposure to Linux was Red Hat Linux 4 in
> December 1996. I bought a physical, shrink-wrapped version with the
> commercial Metro-X X server to start learning Linux at home in my spare time
> shortly after graduation from college. I chose RHL because everything I read
> said RPM made it super easy to install and manage software (perfect for
> noobs like me), and the Metro-X X-server was far superior to any open-source
> X-server available at the time (which was just Xfree86, really). I felt good
> about giving RH my $40 for this not just because it would make it easier for
> me to learn Linux, but because it seemed like Red Hat were really the
> company that was going to take this underdog operating system and make it
> famous.
> 

Back in the early days of RGB, Don Becker and so on, Extreme Linux was a
single CD Beowulf in a disk. I suggested that the community went Debian
at the time, I think - but meh.

If you have several hundred / several thousand machines - whether as a 
cluster or as a web hosting datacentre - you'll have your own engineers.
CentOS was ideal because it would allow you to use any server that was
qualified to run Red Hat (HP/IBM/Dell/Lenovo) and know it would "just work"
The marginal cost savings are small but significant - not least because you
wouldn't have to keep count on licence numbers/struggle with Satellite or
similar.

I've seen people who want to become Red Hat certified engineers/sysadmins
who used CentOS to do this. Red Hat make $$$$ for certifications if you
follow them all through. Red Hat are losing expertise and goodwill
from Fedora/CentOS/Rocky/Alma folks who might still have suggested 
fixes/reported bugs/acted as advocates for the RPM way.

The long term denizens of this list probably don't care *which* Linux it
is that underlies the workload - Linux is Linux is Linux - if you're 
having to deal with user requirements that transcend distro packaging, your
customisations will mean you can't get support and you become your own expert.

> They certainly achieved that goal, but along the way, I've seen them do a
> lot of anti-open-source things that I didn't like, leading me to change my
>  image of them from champion of the underdog to the "Microsoft of Linux" to
> whatever my low opinion of them is now (Backstabber? Ingrate? Hypocrite?):
> 
> 1. When they weren't making any money off a product they were giving away
> for free (Red Hat Linux, and "duh!"), they came out with an "Enterprise"
> version, that would still GPL-compliant, but you'd have to pay for
> subscriptions to get access to their update mechanism. To get people to buy
> into this model, they started spreading fear, uncertainty, and doubt (FUD),
> about "non-enterprise" Linux distributions, saying that any Linux
> distribution other than Red Hat Enterprise Linux (RHEL) wasn't reliable for
> use in any kind of enterprise that needed reliability.
> 

Anybody else remember the Red Hat customisations and gcc 2.96? Other
 unfortunate Red Hat problems like breaking RPM ... it wasn't a picnic
to run commercial Red Hat at the time

> 2. When spreading FUD didn't work, RH killed of RHL entirely. If you wanted
> a free version of Red Hat, your only option was Rawhide, which was their
> development version for the next generation of RHEL, which was too unstable
> and unpredictable for enterprise needs (of course).

<big snip>
> 
> Not long after, RHEL eliminates CentOS as a competitor by changing it to
> "CentOS  Stream" so it's no longer a competitor to RHEL. CentOS Stream is
> now a development version of sorts for RHEL, but I thought that was exactly
> what Fedora was for.
> 
> 5. When Alma and Rocky pop-up to fill the void created by the killing of
> CentOS, RH does what it can to eliminate their access from RHEL source code
> so they can't be competitiors to RHEL, which brings us to today.
> 
> Somewhere around event #3 is when I started viewing RHEL from as the MS of
> the Linux world for obvious reasons. It seems that RH is determined to make
> RHEL a monopoly of the "Enterprise Linux" market. Yes, I know there's Ubuntu
> and SLES, but Ubuntu is viewed as a desktop more than a server OS (IMO), and
> SLES hasn't really caught on, at least not in the US.
> 
> I feel that every time the open-source community ratchets up efforts to
> preserve free alternatives to RHEL, RH ratchets up their efforts to
> eliminate any competition, so trying to stick with a free alternative to
> RHEL is ultimately going to be futile, so know is a good time to consider
> changing to a different line of Linux distro.
> 
There's an element of laziness from hardware vendors / specialist software
vendors here (high end interconnects / EDA or whatever massively expensive
CAD _demand_ Red Hat often at a specific point release from which you can't
update safely) and so the default is RHEL.

> With RH (and IBM?) so focused on market dominance/profits, it's not a
> stretch to think they they'll eventually "say no" to supporting anything
> other than x86 and POWER processors, since the other processors don't have
> enough market share to make it profitable, or compete with IBM's offerings. 
> I mean, right now it's extremely rare to find any commercial application
> that supports anything other than x86_64 (other than Mac applications that
> now support Apple's M processors, which is a relatively new development).
> 

The Beowulf / HPC / SPC world is pretty much either X86 or ultra-bespoke
ARM anyway or depends on GPU for the hard yards.

> My colleagues here agree with my conclusions about the future of RHEL and,
> we are certainly giving the thought of moving away from RHEL some serious
> consideration, but it's certainly not going to be cheap or easy. What are
> you thinking/doing about this?
> 

As a long term Debian user (and lurker here with occasional comments for
about 25 years) move to Debian for everything: tell your hardware vendors
to get with the programme. Ubuntu if not - but there's a few things that
don't sit well like snaps though in this community they're probably irrelevant.
> -- 
> Prentice

There's a gap in the archives for a couple of years, I think. Those with
the longest memories here may remember my good friend Martin Wheeler from
the 90s. He died a short while ago and I'm heading to his funeral tomorrow.
I'll remember him for all sorts of things - not least for persuading me
that this list was worth following.

It's been our great pleasure to follow this community - polite, helpful,
 technical and a model of how to do this technical stuff well on the 
Internet. I thank you all for the example you have set, are setting
and continue to set.

Andrew Cater
[amacater at einval.org / amacater at debian.org]
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf