BlueOnyx 5209R on EL7

As I'm about to wrap up the development of 5209R on CentOS7 it's time for me to post some thoughts on the process.

If we as a people had just a limited amount of curses that we can utter during our lifetime, then I would have run out long before getting half way through the development of 5209R.

I can only repeat what I once posted on the BlueOnyx developer list to sum it up:

"If I were in a room with Lennart Poettering and Hitler and had a gun with only two bullets? I'd shoot Poettering twice. Just to be sure."

Poettering is the guy who said "the Open Source community is full of assholes". But actually he's the bodily orifice which we have to "thank" for having Systemd as Init service in EL7 and pretty much all alternative OS's that we could have used instead. This bloated excuse of an Init-service has more issues than a dope addicted alcoholic street walker. It's utterly illogic, has stuff that doesn't work and is missing things that a +20 year old Initd could do on the sidelines while being just a few kilobytes large monolithic binary.

So yes. We did have our issues getting a lot of stuff to work with Systemd, but we eventually got there through many work arounds and trial and error. The results aren't that bad, but I could have done entirely without it.

While working on it I also cleaned up the install procedure of BlueOnyx and pulled all post-install scripts from the blueonyx-cd-installer RPM (that's only on the ISO's) into base-blueonyx instead. That way it is now entirely possible to install BlueOnyx out of YUM on a "naked" CentOS7. Without tarball-installer. Just install a minimal CentOS7, configure the network and install the blueonyx-yumconf RPM for 5209R. Then do a "yum groupinstall blueonyx", reboot and run "/root/network_settings.sh" after the reboot.

Such a clean setup also makes the post-install section of the ISO install a lot shorter and cleaner.

As for new features in BlueOnyx 5209R: The last three weeks I wrapped up the Apache 2.4 specific changes. There were some issues with getting suPHP to work. At first I thought it would be impossible to get it to work. So we also started to look for alternatives such as PHP-FPM. As I got the basic PHP-FPM configured, the suPHP issue kinda solved itself. But not stopping there I went ahead full speed and used this opportunity.

So BlueOnyx 5209R now fully supports the following options for PHP enabled Vsites:

1.) PHP via DSO. Same as before.

2.) PHP via DSO + mod_ruid2: This new option allows to run PHP via the DSO, but executes PHP scripts with the UID and GID of one of the Vsite siteAdmins. Which is secure and pretty fast.

3.) suPHP: As before. suPHP executes PHP scripts with the UID and GID of one of the Vsite siteAdmins. But it now also works fine for subdomains without them having a separate php.ini. Instead they use the php.ini of the Vsite they belong to.

4.) PHP-FPM via mod_fcgid: This executes PHP scripts via the PHP-FPM daemon and mod_fcid. PHP scripts are also executed with the UID and GID of the Vsite siteAdmin. Works also well for subdomains and is a fast and secure way of executing PHP scripts. In our configuration each Vsite with PHP-FPM enabled gets a separate PHP-FPM pool.

The changed PHP options created some issues with portability of Vsites. So as it was a cmuExport generated on a 5209R would not cmuImport on anything but a 5209R. Because the database schemas have extra key/value pairs that older BlueOnyx versions don't have.

This required an updated Cobalt Migration Utility (2.93) which discards these extra key/value pairs when a 5209R cmuExport is imported on an older BlueOnyx. That seems to work fine now and I could easily import a 5209R cmuExport on a BlueOnyx 5106R.

As I was getting to this point where the general functionality is there and all obvious bugs are gone it was time to look at CD building to roll up a BlueOnyx 5209R Beta ISO image.

Hence I googled "CentOS 7 respin" to find out what has changed since the last time I looked. I was already aware that there would be no Revisor available for CentOS 7. Which is what I use to build the CentOS 5, CentOS 6 and SL 6 ISO images of BlueOnyx.

So I looked at Pungi again. Some people report success with it, but the process is (as it always has been) pretty obscure and frustrating.

The recommendations boil down to using either the livecd-tools or the livemedia-creator. It is possible to do a hardisk install off a LiveCD. So for starters I pondered using that as a testbed to check how sane our RPMs were.

So I set up a VirtualBox VM of a basic CentOS7 with 50GB of space and made sure that both /tmp and /var are on the / partition which I gave everything but the 800 MB reserved for /boot.

Then I grabbed the RPMs from http://people.centos.org/arrfab/CentOS7/LiveMedia/RPMS/ and YUM installed them to make sure all dependencies are fullfilled. Then I did a ...

#> yum install anaconda livecd-tools system-config-kickstart

... and let it grab around 500MB of dependencies. That should round out the crux of my "5209R Toaster" for building ISO's.

That provided me with two options to get going:

#> LANG=C livecd-creator --config=5209r-live-cd.cfg --fslabel=BlueOnyx-5209R

#> LANG=C livemedia-creator --make-iso --no-virt --image-name="BlueOnyx-5209R" --ks=5209r.ks

That of course requires to have a proper *.cfg for the LiveCD or a working kickstart for the ISO. I hacked some basic ones together, but the farthest I got was with livemedia-creator. The RedHat docs were also kinda helpful.

It quickly barfed on an RPM conflict. Our "analog" RPM provides /usr/bin/analog. The RPM "anaconda", which we need for the install also has a binary of the same name in the same place. For different purposes. So I had to rebuild our "analog" RPM to create our binary as /usr/bin/analogbx instead <sigh>.

Once that was sorted the CD build process grabbed 1369 RPMs and installed them into the install image. And then on processing the image it died, because the image couldn't be unmounted as CCEd was running inside it. \o/

So I went back and cleaned up our sausalito-cce-server RPM's post-install scripts to not start CCEd on installs, but only on upgrades. Fun and games: That still didn't work. CCEd got started by something else instead.

As a work around I then opened a second "root" console and during the CD-Building I simply ran this in it:

#> watch "killall -9 cced & killall-9 pperld"

That's a bit of a dirty work around and I'll sort out the underlying problem in that RPM a bit later. The above work around kills off any spawned CCEd and pperld process and allows the install image to be created just fine:

2015-02-01 20:49:55,448: Disk Image install successful
2015-02-01 20:49:55,558: working dir is /var/tmp/tmpHEb3z8
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on /var/tmp/tmpHEb3z8/images/install.img, block size 131072.
[...]
2015-02-01 21:07:12,146: Truncating isolabel to 32 chars: Red Hat Enterprise Linux 7.0 x86

That reminded me to give this "5209R Toaster" VPS more CPU cores next time around to speed up the mksquashfs procedure, which really takes a long time on just one core. Almost 18 minutes in this case on a single core <sigh>.

After that was done I got as far as this:

2015-02-01 21:08:44,761: Building boot.iso
2015-02-01 21:08:44,959: running x86.tmpl
2015-02-01 21:09:09,340: template command error in x86.tmpl:
2015-02-01 21:09:09,340:   install usr/share/syslinux/isolinux.bin isolinux
2015-02-01 21:09:09,426:   IOError: nothing matching /var/tmp/lorax.imgutils.lqPyk0/usr/share/syslinux/isolinux.bin in /

Where it barfed and died. At this point the whole procedure had been running for one hour. A Google search for this error message returned just two results. One of them being this.

So I had to extend my %packages section in the kickstart script to this:

%packages --nobase --ignoremissing
@core
anaconda
grub2
grub2-efi
efibootmgr
shim
authconfig
chrony
-postfix
memtest86+
syslinux
@BlueOnyx
%end

Which added grub2-efi, memtest86+ and efibootmgr, which I previously didn't have. They're actually not mentioned in the docs either. Nice! Well, once that was done I started another run. I wondered how many more I need until I'd get something, but that knowledge would probably have discouraged me at this point.

A local YUM repository copy would sure help to speed things up. Time to get that sorted as well.

As things got a bit repetitive at this point I threw in a small shell script to help with the cleanup and build:

#!/bin/sh
/bin/rm -f /var/tmp/BlueOnyx-5209R
anaconda-cleanup
livemedia-creator --make-iso --no-virt --image-name="BlueOnyx-5209R" --ks=5209r.ks --resultdir=/home/build_cd/ --logfile=5209r.log

The anaconda-cleanup doesn't remove everything from a previous build attempt. But it frees up the mounts that it has created during previous aborted runs, which sure is nice. But /var/tmp still needs to be dealt with separately.

That did finish a bit sooner now (with 4 cores instead of one) and ended like this:

 96.22% done, estimate finish Sun Feb  1 22:10:16 2015
 97.36% done, estimate finish Sun Feb  1 22:10:17 2015
 98.51% done, estimate finish Sun Feb  1 22:10:17 2015
 99.65% done, estimate finish Sun Feb  1 22:10:18 2015
Total translation table size: 6483
Total rockridge attributes bytes: 3694
Total directory bytes: 14336
Path table size(bytes): 108
Max brk space used 0
436536 extents written (852 MB)
Inserting md5sum into iso image...
md5 = 83d8ad645b33c2d4755b53159e3ffeb7
Inserting fragment md5sums into iso image...
fragmd5 = 45a8949dea39c9c94bf564763d6d5899daa25544ffb8c41f9c21a3e4ad7e
frags = 20
Setting supported flag to 0
2015-02-01 22:10:40,940: Disk image erased
2015-02-01 22:11:57,592: SUMMARY
2015-02-01 22:11:57,592: -------
2015-02-01 22:11:57,593: Logs are in /home/install/cd_test
2015-02-01 22:11:57,593: Results are in /home/build_cd/

And lo and behold: There is the ISO, with 852MB in size.

So I fired up another VPS and booted off that ISO. It booted pretty darn fast and threw me a login prompt. Hmmkay. Logged in as "root" with password "blueonyx" (as defined in the kickstart) and got a shell. "ps axf" and "rpm -qa" confirmed: This is a LiveCD of BlueOnyx 5209R on CentOS 7. How neat!

Now BlueOnyx isn't really designed to run off a LiveCD. Our network scripts had a few issues there and as there is no /etc/fstab our / didn't get mounted with quotas enabled. Hence user "admin" didn't get created by the CCEd constructors and therefore we cannot login to the GUI. BUT: This is actually still neat and can be tweaked to get a working LiveCD later on.

So while that was fun, let's go back and try Pungi <sigh>.

First we get the EL7 Pungi RPM and install Epel for EL7 for dependencies.

I then set up the basic build structure:

#> mkdir /home/build_cd

I copied my 5209r.ks kickstart script over and created a shell script named make_cd.sh with this in it:

#!/bin/sh
/bin/rm -Rf /home/build_cd/work/
/bin/rm -Rf /home/build_cd/5209R/x86_64/os
pungi --name=BlueOnyx --ver=5209R --destdir=/home/build_cd --nosource --nodebuginfo --all-stages --bugurl=http://www.blueonyx.it --isfinal --config=/home/build_cd/5209r.ks --cachedir=/var/tmp/

I needed to create the subdirectories as well as it turned out:

#> mkdir -p /home/build_cd/BlueOnyx/x86_64/

Made the script executable and let it run. Ok, this looks different. Off the same kickstart Pungi insists on pulling 1632 RPMs instead of the 1369 RPMs that livemedia-creator wanted. The real number of needed RPMs is closer to 1200 than 1300 actually. So even if this finishes OK there will be much chaff to cut from the resulting ISO. Like for example ... it pulled KDE. Last time I ckecked my own kickstart I didn't see KDE getting mentioned, so ... that was surprising. But at the end it started to remove whole groups and packages again, so .... that approach is a bit weird.

Eventually Pungi failed with:

ERROR:program:mount: unknown filesystem type 'hfsplus'
mount: unknown filesystem type 'hfsplus'

Looks like I need the kmod-hfsplus kernel module. The ELrepo has it in the testing repository. So I grabbed that from there and then loaded it via "insmod /lib/modules/3.10.0-123.el7.x86_64/extra/hfsplus/hfsplus.ko".

That turned out to be a bad idea. When building the UEFI image at the end of the procedure I got a kernel panic and a reboot. Which sounds familliar. The problem is related to the fact that Pungi tries to build a Macintosh compatible UEFI image, which can't be done on EL7. There is an unofficial patch to turn off that stuff. So I made a few edits to the files that my Pungi 3.12 RPM had brought aboard and replicated what the patch had done.

And that actually did run through without any errors and produced two ISO's

BlueOnyx-DVD-x86_64-5209R.iso (1.5 GB)
BlueOnyx-netinst-x86_64-5209R.iso (373MB)

Cool beans. Off to install them! And that went surprisingly well. The new ISO's use the graphical installer (with default branding - which will get changed). The package group "BlueOnyx" has to be selected manually and also the partition scheme has to be set manually. The installed OS had a couple of minor issues, which were quickly addressed with BlueOnyx updates.

So there is just a little cleanup to do. Some "unfattening" of the DVD and some cosmetics. Also some post-install scripts to make sure that user "root" gets directed to the setup.sh script. This will probably take a few days to complete, as each build takes roughly 40-60 minutes.

But we're on a good track here and it looks like Pungi will be our choice for CD-Building for El7.

Update: After two days of playing around with Pungi I'm now at a point where the Installation CD for BlueOnyx 5209R is almost done. I fiddled the Pungi pocedure into our existing Makefile for CD builds. We now use Pungi with the "-G -C -B" switches to let it do everything but the actual ISO generation. We then have "make import" import the parts from the Pungi work directory that contains the CD structure that we need. We let some parts remain static (like our modified isolinux/isolinux.cfg and the boot.msg (and a few other bits and pieces). We also copy some extra stuff aboard like our license, the readme and things like that. Finally "make iso" is run and builds the ISO of the DVD and implants the md5sum.

The branding wasn't as much of a challenge as I thought. I downloaded the centos-logos SRPM, took it apart and renamed it blueonyx-logos. With a slightly higher release number. It's simply set to obsolete centos-logos and in the anaconda subdirectory I replaced the CentOS logos with our own.

Lastly: Anaconda. I also took the SRPM for it from CentOS and adjusted the patch that contains the path to the EULA to point to our BlueOnyx license instead. I also took out the patch that makes XFS the prefered filesystem, as XFS isn't useful for us due to the different quota handling.

The only sad part about Pungi is that it's pretty greedy. The ISO contains Gnome and KDE, even if we don't need it. Our "make import" has to drop these RPMs, or I have to specifically exclude them in the package list of the kickstart scripts.

All in all: Pungi saved me a lot of time. I still wish for a Revisor for EL7, though. That would have saved me the most time, as our existing build setup was designed for it in first place.