BlueOnyx EL6 ISOs - developer rant

The last six days I tried to build updated BlueOnyx 5207R/5208R ISO's and did run again into some FUBAR-SNAFU of epic proportions.

A couple of months ago I had set up five different VPS's for building ISO's for the following platforms:

• CentOS 6.6 x86_64: For building BlueOnyx CentOS 5108R/5208R ISO's.
• CentOS 6.6 i686: For building BlueOnyx CentOS 5107R/5207R ISO's.
• Scientific Linux 6.6 x86_64: For building BlueOnyx Scientifc Linux 5108R/5208R ISO's.
• Scientific Linux 6.6 i686: For building BlueOnyx Scientific Linux 5107R/5207R ISO's.
• CentOS 7.0 x86_64: For building BlueOnyx 5209R ISO's.

As the last 5207R/5208R ISO's were from early December and we published many updates, I thought it would be prudent to roll up new and updated ISOs that contain all these OS related and BlueOnyx related updates.

My first mistake:

Running "yum update" on the build boxes to make them current.

My second mistake:

Trusting RedHat to not fuck things up royally. But they did.

My build boxes for EL6 based BlueOnyx ISO's use Revisor (modified from the Scientific Linux 6 build) and so far that always had worked six times out of eight and produced good results. On EL7 I now use Pungi 3.12 and that works so relieably, that we could even do unattended nightly ISO builds if we ever wanted to.

However, this time around on my EL6 build boxes not one of the created ISO's would work. They all failed to verify the SHA256 checksum of the RPMs on Anaconda installs off the created ISO's. It took several days of bug hunting and trouble shooting to get to the core of this. My usually good "Google-fu" didn't turn up much that seemed to be related to the issue.

When I did find something that seemed related, it got such very helpful replies from the CentOS guys like this:

Which is basically the equivalent of saying "Go frack yourself ... with a cactus." Even if the original posted pointed out a very valid bug which prevents the spinning of updated ISO's that just contain the stuff that they ship.

So I tried a few things. Set up a fresh new CentOS 6.6, fully updated. Started over. Installed Pungi, all dependencies, installed and set up my local repositories and modified my working EL7 Pungi setup to work on EL6.

It build ISO's just fine. No issues whatsoever. But: Same as with Revisor. An install off those ISO's failed as the RPMs failed the SHA256 checksum tests of Anaconda.

I had a very helpful soul come to the rescue and he gave me some pointers. Otherwise I'd have given up for the time being. I'll leave his name out for now, but many of you know him and hold him as dearly as I do. So I went back and forth between older and newer Anaconda versions. Started experimenting with a modifed "createrepo" and a modified Anaconda. Tried SHA1 instead of SHA256. It was a really wild goose chase that accomplished very little.

I still have around +200 tabs open in my browser from Anaconda related bug reports on the RedHat bugracker, the CentOS bugtracker, StackOverflow and various blogs. The most helpful (but very convoluted) one was this. It provided no clear shot at the issue, but various leads to follow up on.

It pointed me into the right direction:

As soon as you try to respin a CentOS ISO *and* enable the Updates repository, you're busted. The resulting ISO will fail with so many error messages that it's no longer funny. The real one being: Inability to verify the SHA256 checksums of the RPMs and failure to install them.

It's not the Updates repository or general usage of it. It's the "nss" related RPMs in it:

• nss
• nss-softokn
• nss-softokn-freebl
• nss-sysinit
• nss-tools
• nss-util

If you include them in your updated ISO's and let Anaconda build the fist/second stage installer of your ISO use them, then you're busted. I don't know exactly which of these NSS related RPMs is the culprit and after spending +100 hours on debugging it I'm actually way past caring.

Fact of the matter is: If you build your ISO with the NSS RPMs out of the Base repository and include anything else from the Updates repository (except for the NSS related RPMs), then the CD just works fine. As soon as you also include the NSS RPMs from the Updates repository, you get a broken ISO.

After installing off the new ISO with the older NSS RPMs you can run "yum update" and can update them just fine with no ill side effects.

So what's the issue? Simply put: ALL Anaconda versions of EL6 up to and including the most current anaconda-13.21.229-1 are incompatible with the NSS updates from September/November 2014.

Shame on you, RedHat, shame on you, CentOS. Especially for telling valid complainers that they should go and frack themselves when the ball is clearly in your court. I'm not even going to file a bug report. It's simply not worth my time if the most likely answer is just another politely worded "FU!".

Why am I excluding Scientific Linux in this rant? Because ... their most up to date NSS and Anaconda actually do work together. Just build an ISO off the 6x (rolling) repository and you get a working and fully updated EL6.6 with no issues. Go figure!

In the meantime the first new ISO has been uploaded: BlueOnyx-5208R-CentOS-6.6-20150224.iso and the Scientific Linux based 5208R will be up in a minute as well. The rest of the updated 5207R/5208R ISO's should be available within the next 2-3 days.