BlueOnyx 5211R Development

Some updates and ramblings about the ongoing BlueOnyx 5211R development on EL9.

With the year 2021 coming to an end I thought I'd give an update on the BlueOnyx 5211R development, which had been started on 4th November 2021.

RedHat Enterprise Linux 9 Beta

With the release of the RHEL9 Beta in early November I installed it and used it to kickstart the development of the next version of BlueOnyx, which would be 5211R if we follow our existing numerology.

The idea here is to use RHEL9 Beta as approximation of what to expect whenever RHEL9 comes out for real and to be ready for whatever AlmaLinux or RockyLinux port that will be based on it.

The RHEL9 Beta had a few surprises, such as using OpenSSL 3.0.0-beta2 (!!) and anything that uses OpenSSL was compiled against that hot pile of unfinished garbage. With the usual results, such as some programs, tools and libraries not working correctly, or not being present yet, as they couldn't be built (yet) against OpenSSL 3.0.0 - let alone a Beta2 of that! Even "subversion" (which we use as SVN) wouldn't allow me to check out our repository code via HTTPS, as the TLS implementation in subversion was broken after compiling it against OpenSSL 3.0.0-beta2.

Many of the OpenSSL related issues went away as RedHat rolled out patches and updates for the RHEL9 beta and it is by now using an OpenSSL-3.0.0 that has left the beta stages. However, some SSL related issues remain for now - usually with some older libraries or programs.

The Good, the Bad and the Ugly

Those of you who are familiar with my review style of new RedHat releases already know that this headline is by now pretty established. :p

The Good

Let me start with the good things: RHEL9 holds fewer surprises and dramatic sacrifices of features, functions and old habits than RHEL8 or RHEL7 did. Because by now our pain tolerance is already pretty high. We're used to Systemd, we're used to the craptastic AppStream functionality that got forced into YUM (which is now actually DNF, but never mind) and we also survived the Pythoncalypse by having both Python2 and Python3 installed on the same server and bits and pieces of the OS using one or the other.

As it is now RHEL9 Beta ships with the following components which usually interest us the most:

• Apache v2.4.51
• Nginx v1.20.1
• PHP v8.0.12
• MariaDB v10.5.12
• Perl v5.32.1
• Sendmail v8.16
• Postfix v3.5.9
• ProFTPd v1.3.7c
• Dovecot v2.3.16

Which all in itself is sort of fine. Those services whose age will bother us eventually? We'll replace them with more modern versions if needed and PHP and MariaDB will always be available in the BlueOnyx shop in a plenitude of versions given enough time. But as far as decent modernity goes? It'll do. For now.

The Ugly

The somewhat uglier part is that a few things have been deprecated for good now. Which was as needed as it (in practical terms) is unwelcome. But yeah, it had to be done. So Python2 is now gone and anything that needs Python needs to be Python3 ready. Furthermore: Old style InitV/Upstart scripts no longer work. There is no longer a wrapper that allows Systemd to handle these as well.

There were a few BlueOnyx related services that still used old style init scripts, such as AdmServ, Poprelayd, Jailkit and something else I already did forget. They now also had to get proper Systemd Unit-Files, which was sort of painless - except for AdmServ.

The really ugly parts of RHEL9 deal with PHP as far as we're concerned.

On all currently supported BlueOnyx versions Vsites can use one of the following options for PHP:

• Disabled
• DSO
• DSO + mod_ruid2
• suPHP
• PHP-FPM

The two DSO versions obviously need it that Apache loads the libphp.so module. But as Apache sucks (compared to Nginx), it cannot do HTTP/2 *and* load PHP as DSO. On RHEL8 HTTP/2 was therefore usually disabled, because it was also in a rather buggy and early state of implementation to begin with. Now it has matured and the fine folks at RedHat flipped the switch: HTTP/2 is enabled by default in Apache and their PHP is now build and shipped without the DSO. The SRPM still has provisions for the DSO in it, but they're disabled and the RPM that would contain the DSO is not published to the RedHat YUM repos anyway.

So we have a choice there: I could build and ship the DSO separately and then you will have a choice if you want HTTP/2 *or* PHP with DSO support. This is both bad and ugly, because ideally the DSO needs to match the version of the rest of the PHP install and as RedHat will keep the PHP patched and will publish updates, so do I need to do as well. And I'll of course only learn after the fact when "upstream" publishes a new PHP.

So I haven't fully decided yet which way we'll go there, but to rock the boat the least the default install of BlueOnyx 5211R will most likely have HTTP/2 enabled in Apache and DSO not available at all. Or turned off *if* we provide it as an option. For ease during migrations I will probably add an internal mechanism that maps Vsites with one of the DSO related PHP configs to use suPHP or PHP-FPM instead to retain functionality.

There is a little more ugliness to report:

Xinetd is gone. For good.

But we can do without indeed. Since BlueOnyx 5210R our ProFTPd is no longer using Xinetd and the only relevant service that *might* (fat chance!) still use Xinetd (if enabled) was the Telnet server. Which we had provided as a very, very last ditch effort at remote management if everything else would fail. And of course that cursed and insecure thing was always turned off by default. But for the sake of tradition we still had it, because any BlueOnyx and BlueQuartz and Cobalt had always had a Telnet server. But with the inclusion of Shellinabox and the prevalence of other remote administration means such as VNC or Cockpit we really should finally toss the Telnet server where it belongs: Into the trash bin of history.

Another ugliness is that SELinux can no longer be disabled by just setting a flag in a config file and doing a reboot. Nope. It also needs to be disabled by kernel boot option:

grubby --update-kernel ALL --args selinux=0

But that's fine. We can live with that.

A further ugliness is that - at this time - there is no working NTPd available for either RHEL9 beta nor Fedora Core 35, as it won't compile against the currently used GLIBC version. So for now? We won't have NTPd either. But this might change.

The Bad

At the same time that RHEL9 Beta was released, the latest Fedora Core 35 was also released. That's the usual rhythm of things over at RedHat. Whenever a new Beta of their upcoming Enterprise Linux comes out, there will also be a release of their playground OS Fedora Core, which uses the same technology and versions of software.

As usual the Fedora Core YUM repositories are chock full of all goodies, while the Beta repository of the new RedHat Enterprise Linux is as empty as a treasury after you elect socialist into power and let them run amok for a few years. I'm old enough to be used to that, so I cursed a little less than usual, rolled up my sleeves and dug in to leech SRPMs from FC35 and rebuild them on RHEL9 Beta to assemble a build environment to be able to build a BlueOnyx.

By now I can curse fluently in German, English and Spanish, with a bit of Yiddish, Russian and Finish strewn in. Because off and on everyone needs good "Perkele!", right?

For a build environment I need a working "subversion" and the one shipped with RHEL 9 Beta initially couldn't use HTTPS, while our SVN forces all connections to HTTPS. So I had to tussle with the code repository configs first - after finding out why the shipped "subversion" was broken.

That out of the way I usually want an easy way to build fully up to date Perl modules straight out of the latest releases on Meta CPAN. The tool "cpanspec" exists for that reason. The FC35 repos have it, but (of course - sigh!) RedHat 9 Beta doesn't. And so far I've chased dozens of dozens of dependencies of "cpanspec" to the nth degree and I *still* don't have it working as there are still unresolved dependencies. I then put "cpanspec" aside for the time being and moved on.

Next I tried to build "mock". Which you *really* want to have to build RPMs. Many Perl of Python related SRPMs want or need mock, or you're getting all sorts of weirdness during builds. So I chased dependencies in order to rebuild the FC35 "mock" on RHEL 9. After having built hundreds of RPMs for dependencies I again had to throw the towel, as I was (again) running into circular dependencies where RPM A needs RPM B to build. And RPM B needs RPM A to be built. And both also need RPM X, Y and Z, which conveniently need either RPM A or RPM B being present.

Great, isn't it?

Then I said "Fuck it!" and went on to build BlueOnyx itself. There are a few modules that have dependencies that aren't fulfilled yet. But I got the majority of the modules built eventually. And most of their easier dependencies sorted.

Today I tried to build Mailman, the mailing list manager. On RHEL8 (and clones) it still uses Python2.7. On FC35 there is a Mailman3 RPM/SRPM, which uses Python3. So I tried to rebuild that. And down the rabbit hole of dependencies it went again:

[root@5211r SRPMS]# rpmbuild --rebuild mailman3-3.3.4-5.el9.src.rpm
Installing mailman3-3.3.4-5.el9.src.rpm
error: Failed build dependencies:
python3-aiosmtpd >= 1.4.1 is needed by mailman3-3.3.4-5.el9.noarch
python3-alembic is needed by mailman3-3.3.4-5.el9.noarch
python3-atpublic is needed by mailman3-3.3.4-5.el9.noarch
python3-authheaders >= 0.9.2 is needed by mailman3-3.3.4-5.el9.noarch
python3-authres >= 1.0.1 is needed by mailman3-3.3.4-5.el9.noarch
python3-click >= 8.0 is needed by mailman3-3.3.4-5.el9.noarch
python3-falcon > 1.0.0 is needed by mailman3-3.3.4-5.el9.noarch
python3-flufl-bounce is needed by mailman3-3.3.4-5.el9.noarch
python3-flufl-i18n >= 2.0.1 is needed by mailman3-3.3.4-5.el9.noarch
python3-flufl-lock >= 3.1 is needed by mailman3-3.3.4-5.el9.noarch
python3-flufl-testing is needed by mailman3-3.3.4-5.el9.noarch
python3-gunicorn is needed by mailman3-3.3.4-5.el9.noarch
python3-lazr-config is needed by mailman3-3.3.4-5.el9.noarch
python3-lazr-smtptest is needed by mailman3-3.3.4-5.el9.noarch
python3-mock is needed by mailman3-3.3.4-5.el9.noarch
python3-nose2 is needed by mailman3-3.3.4-5.el9.noarch
python3-passlib >= 1.6.0 is needed by mailman3-3.3.4-5.el9.noarch
python3-sqlalchemy1.3 >= 1.2.3 is needed by mailman3-3.3.4-5.el9.noarch
python3-zope-component is needed by mailman3-3.3.4-5.el9.noarch
python3-zope-configuration is needed by mailman3-3.3.4-5.el9.noarch

I've followed the dependencies for "python-aiosmtpd" now 14 levels deep. Means: It needs something else to build that needs something else first before THAT can be built, which again requires something else first. Until I hit another unresolvable circular dependency.

That's really frustrating.

I'm feeding three BlueOnyx 5211R related YUM repositories at this time:

• RPMS (Directly related to BlueOnyx)
• OSRPMS (OS related dependencies and libraries)
• SRPMs (Contains the SRPMs of anything we need that's not in the BlueOnyx SVN)

To give you some statistics:

• RPMS: 575 RPMs as of now
• OSRPMS: 337 RPMs as of now
• SRPMs: 238 SRPMs as of now

My "work" directory currently has 49 SRPMs that I need to build in order to satisfy the currently known open dependencies. And none of those builds as of yet, as they themselves have so far unresolved dependencies. Some of them being circular dependencies.

This is mostly a matter of elbow grease and determination and I *can* eventually overcome this dependency hell one way or another. But every fucking time RedHat and Fedora kick the version number can down the alley for their major versions, they leave system integrators such as us struggling even more. Fedora for example has the bloody ridiculous habit that if something can be compiled against a library that allows to display the date and time in Mayan calendar format? Then you can bloody well be sure that you need to build the library for the Mayan Calendar handling first and then compile against it. Which means that most SRPMs from the FC35 repos have a ridiculous amount of dependencies.

It gets worse with their build environments, such as for Perl and Python. Which require stuff like Moose, MouseX, Mock, Virtualenv and other goodies, which then all end up with a couple of circular dependencies that can't be resolved without breaking the dependency chain by doing a manual install into a disposable virtual environment to begin with.

And like said: Every new RHEL version the list of dependencies grows. Last time around with RHEL8 I was done after rebuilding 230 SRPMs. Now I've done 238 SRPM rebuilds and I'm only between 30-40% there. I can fully understand the Amazon Cloud people that they tossed RedHat into the garbage bin and went to straight build their own Linux instead. They, too, quoted the Fedora Core dependency hell, which you *have* to tackle with if you need anything that the barebone and almost empty RHEL repositories don't provide.

What awaits us?

The above are teething problems and they will be overcome eventually. RHEL9 and its clones will for sure provide a stable and reliable platform for the next BlueOnyx version. That I have no doubt of.

We don't yet know when RHEL9 leaves beta (I suspect perhaps in February) and when AlmaLinux and RockyLinux will ship their RHEL9 clones. I have more confidence in an early availability of AlmaLinux 9, so we just might get a first Release Candidate perhaps within a month of the first RHEL9 release. That would perhaps be March 2022.

Until then I want to have a rough draft of BlueOnyx 5211R ready, in which most of the functionality of BlueOnyx is more or less working. As usual it would be wise to let the dust settle if you want a stable production environment, so if you're cautious, then plan on using BlueOnyx 5211R in production sometime around May 2022 - give or take.

Some more running commentary about the development can be found on the BlueOnyx Mailing List and if there are major news about the progress of the development, then I will also publish them here.

In any case: Many thanks for your interest in BlueOnyx and we all wish you a Happy New Year and all the best for 2022!