CD Building
The toaster gets warmed up to drop out new ISO images ... and the frustration level rises.
After the massive UTF-8 update it was time again to roll up new ISO images.
CD-Building, day one:
For our Scientific Linux based BlueOnyx 5107R and 5108R this was just a matter of minutes:
cd /home/build_cd
make installer
make revisor
make import
make iso
make storage
Done!
OK, granted: The Makefile and the scripting effort behind it initially had taken quite a bit of hacking. But it's just automating an already perfect mechanism for CD-building a little further. I use Revisor to roll up the BlueOnyx 5107R and 5108R CDs and it automatically resolves all dependencies, downloads the latest RPMs that we need and downloads it all into a directory that already contains everything that a bootable install CD requires. It also rolls up a first and second stage Linux kernel, initrd image and everything. All that's left to do is to burn off the directory as an ISO image.
For the CentOS-5 based BlueOnyx 5106R I haven't bothered to roll up a new ISO image since BlueOnyx-5.6-20110621.iso, which was created on 21st June 2011. So this is really ancient now and could use an update, too.
Sadly, the guys at CentOS have not found it necessary to release a Revisor for CentOS-5 or CentOS-6, although it is available upstream in RHEL. Sure, Karanbir uses buildinstall to roll up the CentOS ISO's, so anyone wanting to use something else that RHEL already supports out of the box (such as Revisor) gets the short stick. Which is kind of hillarious considering that Scientific Linux apparently never had a problem with that. Scientific Linux encourages respins of the OS and really bends over backwards to explain how to do it AND provides the tools for it. Namely a Revisor that works out of the box and is served out off the usual SL repositories. Kudos and heartfelt thanks to the Scientific Linux team and a raised middle finger to team CentOS for being 'holier-than-thou' and ignorant snobs.
Epel has Revisors for CentOS5 and CentOS6, but these are just plain rebuilds of the RedHat Revisor SRPMs and don't really work that well on CentOS. Unless you do a lot of hacking.
Well, there is of course Pungi. Works similar as Revisor, but uses different methods. With the right handling the end result is the same: You get an updated local directory which contains everything your need for a bootable install CD.
There is no Pungi for CentOS-5 either. Not officially - of course. And the ones I found required tons of dependencies that the CentOS-5 repository didn't have either. Epel had most of them, but to add injury to insult: Pungi requires a newer Anaconda than either CentOS-5 or Epel provide. The need for the newer Anaconda became painfully evident during the course of this endeavour.
Hence Pungi didn't work out for me either.
So after hours of frustration I finally went back to a semi-manual build method for 5106R ISO images <sigh>.
Bascially I used ...
yum -y groupinstall --downloadonly blueonyx core Base anaconda --downloaddir=/home/build_cd/i386/CentOS/ --installroot=/home/build_cd/tmp/
... to trick YUM into just resolving the dependencies and to download (but not to install) all needed RPMs in their latest version into the directory of our CD-build where the RPMs usually reside.
This requires some extras such as the downloadonly extension for YUM, but that's not a big deal.
However, as we cannot use Pungi or Revisor, the rest of the CD layout has to be stuffed in manually. The /isolinux directory, the /images directory and whatever else is needed has to be filled in manually. Like ripped off an official CentOS-5.8 ISO image.
But even then the stuff in the /repodata directory has to be re-generated to match the RPMs that we now have on the CD. So off to run genhdlist and pkgorder with the right options to do just that. Only to finally remember that a run of createrepo ought to suffice nowadays. So let's use createrepo instead.
So after some fiddling and manually generating the pkg-oder file, I finally had everything together for an updated BlueOnyx 5106R on CentOS-5.8 ISO image. Right?
Of course not!
The fun really started there. The new ISO image now installs, but runs into an "exception" during setting of the timezone, claiming that /mnt/sysimage/etc/localtime is on a read-only medium. While this could be an Anaconda problem, it's most likely an issue in the stage2 installer of the CD itself.
OK, then back to the last resort. Let's use Anaconda's buildinstall to throw the ISO together. Which will also generate the stage2 installer.
Well: That didn't get me very far either, as it didn't build the required (and matching) /images/minstg2.img and /images/stage2.img that our CD installer needs to boot:
Failure to communicate with kernel device-mapper driver. Check that device-mapper is available in the kernel.
Of course: Device-Mapper is not available inside an OpenVZ VPS, nor can it easily be activated. Did I mention that the "box" that I build the CDs on is an OpenVZ VPS? No? Well ... that never was an issue before and Scientific Linux's Revisor works perfectly in an OpenVZ VPS. And it also generates the stage2 images from scratch - without any complaint. But of course it uses an Anaconda that's a few lightyears ahead of the one on CentOS-5!
So let's see: That's 12 hours of work that did go nowhere. CD building is already - under the best of circumstances - a painful ordeal. I wish it wouldn't get unnecessarily complicated by the upstream vendor (i.e.: CentOS) by withholding the right tools required for that purpose.
<ARRRRRGH!!>
OK, another day, another try. Tomorrow I'll try buildinstall again inside a CentOS-5 KVM. Or I will try to trick Revisor on Scientific Linux 5 into building CentOS5 ISO's. Or use the RHEL5 revisor from Epel on CentOS-5. Whatever.
CD-Building, day two:
So it is time to try something different to get an updated CentOS-5 based BlueOnyx CD built:
I installed a CentOS-5.6 + BlueOnyx off our old CD inside a KVM on Aventurin{e}. After YUM updating that box to v5.8 I did the cardinal sin of activating a ton of extra YUM repositories (which you should never do on a production box). So in went Epel, Dag and a few others.
After installing anaconda, anaconda-runtime, revisor, pungi and a lot of other needed bits and pieces for CD-Building I imported my CD build tree from the 5106R VPS that I was using so far. This also brought my custom Revisor configs aboard.
A first run of Revisor was promising. Anaconda's buildinstall was also running more or less, but failed due to some missing dependencies such as busybox and patch, which I had forgotten to grab. This was quickly rectified with YUM. Next issue was that the CD-install of BlueOnyx mounts /tmp with the noexec and nosuid bits. But during the creation of the stage2 images both Revisor and buildinstall heavily use /tmp and both the noexec and nosuid are preventing them from doing their jobs. After removing these bits for /tmp from /etc/fstab I rebooted the KVM for the change to take effect.
When the VPS didn't answer to SSH for several minutes, I checked with VNC what was up and there we went:
Unable to load Selinux Policy. Machine is in enforcing mode. Halting now. Kernel Panic!
Great! After rebooting the VPS yet again, I hurried to fire up Vinagre to access the KVM console in time to see the Grub boot menu. In there I appended enforcing=0 to the boot line of the default kernel and then the KVM booted just fine. That parameter then also went directly into /boot/grub/grub.conf to spare me these kind of troubles during the next reboot.
Another trial run of Anaconda's buildinstall was now getting a bit further. But as space in /tmp was running out (it still had stuff from the last Revisor run in it) it only managed to build /images/minstg2.img, but buggered out on building the /images/stage2.img that our CD installer needs to boot.
After cleaning out /tmp another run went through just fine:
Building minstg.img Running mksquashfs /tmp/instimage.dir.3298 /tmp/minstg2.img -all-root -no-fragments Creating little endian 3.0 filesystem on /tmp/minstg2.img.3298, block size 65536. Little endian filesystem, data block size 65536, compressed data, compressed metadata, no fragments Wrote /home/build_cd/i386/images/minstg2.img (40948k)... Building stage2.img Running mksquashfs /tmp/treedir.23942/instimage /tmp/instimage.img.3298 -all-root -no-fragments Creating little endian 3.0 filesystem on /tmp/instimage.img.3298, block size 65536. Wrote /home/build_cd/i386/images/stage2.img (93768k)
Yeah, that looks better. After confirming that everything was in place, I rolled up a first ISO image. With 798MB of size it was too large, but that's hardly surprising, as the /images directory still contains the XEN and PXE boot images as well as the netboot image. But for a first test that'll do. Off to install the ISO into a KVM to see what happens. The KVM booted off into the CD installer. No problems. Right, the splash image is stock and the boot menu reminds me that I forgot to subsitute our custom isolinux.cfg and boot.msg. So booting finally got me into the graphical installer. As we need our custom kickstart (which wouldn't engage this way) I aborted this try.
Back to the work directory: I cleaned out erronous images from /boot and fixed the boot menus in /isolinux and burned another ISO. This time the size is 746M which is still too much <sigh>.
After booting the image with "nolvm noapic" the installer launches just fine and goes through all the magic without hitch.
Until it dies as spectacularly as with the stage2 images from our old CentOS-5.6 CD during the attempt to set the time zone of the installed system:
Just great. \o/ Back to the drawing boards.
So what could the problem be? Most likely it's an issue with Anaconda. After lots and lots of digging I finally found a hint that it could be related to this issue. After all, some of our BlueOnyx RPMs try to set the timezone and modify /etc/localtime and I think one RPM sets /etc/localtime during installation of said RPM. When it does, then Anaconda craps its pants when it tries to set the timezone at the end of the install and before the %post install section.
How to tackle that? Well ... I could take the second stage images apart and could modify the respective Anaconda script. Or I could use a newer Anaconda than the stock 11.1.2.250-1 that comes with CentOS-5.8.
Scientific Linux 6 uses Anaconda anaconda-13.21.149-1 for example. So I grabbed the SRPM for that and tried to rebuild it on CentOS-5. It did run into a lot of BuildRequires issues and after clearing those, I got this far:
checking for uname... yes
checking for utime... yes
checking for wcwidth... yes
checking for a Python interpreter with version >= 2.5... none
configure: error: no suitable Python interpreter found
error: Bad exit status from /var/tmp/rpm-tmp.1208 (%build)
Rrrrrrright. I forgot. We need Python 2.5 or better for newer Anaconda's. And CentOS-5 is still stuck with Python-2.4 <sigh>.
So that leaves two choices:
a) To modify the base-timezone.mod constructors setTimeZone.pl and syncTimeZone.pl to not run during an install from the CD.
b) To modify the existing CentOS-5 Anaconda 11.1.2.250-1 to fix it's timezone.py not to bugger out if /etc/localtime has already been set up.
So I went for plan (b) and rebuilt Anaconda 11.1.2.250-1 with the fixes to timezone.py from anaconda-13.21.149-1. Another few Revisor runs later I did run into several new problems:
The resulting ISO was getting larger and larger. No matter what I did to the Revisor configs, certain optional RPMs got installed that I defenitely do not want. Like half the X11-server, the XEN kernel and what not. The second problem was that the stage2 kernel still contained the same timezone.py that it had before I modified anaconda, bumped its version number and subsituted it from a local YUM repository and had it installed on the build box, too.
With that the second day of CD building ends. Some new things were learned, some new frustrations built up without getting much further onto the way towards a new BlueOnyx 5106R ISO image.
The fun will be continued tomorrow <sigh>.
CD-Building, day three:
After some reflection of last nights oddities I checked and found out that the CD contained an anaconda that was not served out of the local YUM repository. So it didn't contain the latest modifications to timezone.py that I had made.
However, this also gave me the insight that both Revisor and buildinstall apparently use the Anaconda of the CD (and not the Anaconda installed on the build box?) to roll up the stage2 boot images of the CD. However, in later tests I wasn't so sure about that anymore and just made sure that I had the same anaconda installed and ready for usage on the CD through the local yum repository.
After making sure that the correct modified anaconda RPM was in my local YUM repository and after another Revisor run, I finally had a CD with that had the substituted timezone.py in the stage2 image. Off for a test install, which got this far:
That went well then! As we can see, timezone.py buggered out because of a syntax error. Apparently it uses some Python-2.5 syntax, which Python-2.4 can't handle? So be it.
Next attempt:
Rebuilt Anaconda again. This time with the original timezone.py back in again, but this time I commented out the lines that deal with modifying /etc/localtime:
if not os.access(fromFile, os.R_OK):
log.error("Timezone to be copied (%s) doesn't exist" % fromFile)
# else:
# try:
# shutil.copyfile(fromFile, instPath + "/etc/localtime")
# except OSError, (errno, msg):
# log.error("Error copying timezone (from %s): %s" % (fromFile, msg))
After rebuilding the Anaconda RPM and stuffing it into the /Packages directory (the local YUM repository as well) I didn't run Revisor again, but just triggered the rebuild of the stage2 images:
/usr/lib/anaconda-runtime/buildinstall --product BlueOnyx --variant 5106R-i386 --version 5106R --release "BlueOnyx 5106R" --prodpath Packages /srv/revisor/5106R-i386/os/i386/
Then I rolled up the ISO again (for what feels like the 100th time) and tried another test install. Actually, this was KVM test install #35 in three days <sigh>.
But let's see what we got now:
Wow. At long, long last. \o/
Sadly, the resulting ISO image is still 822MB large, which is a bit too chubby for a CD sized media. So a few buckets of fat need to be trimmed off. First things overboard: kernel-xen, kernel-debug and kernel-PAE. Oh? Why did the php53-* RPMs come aboard alongside the usual PHP? Away with these as well. Hmm ... 756MB now. Still too much.
Also, a test of the installed OS revealed some glitches. In VNC the setup menu looked a bit garbled. Additionally - after flying through the setup blindly - the GUI didn't work, as mod_ssl wasn't installed. A quick check revealed that the comps.xml file in the BlueOnyx 5106R YUM repository indeed hasn't listed it as mandatory RPM. So that needed fixing. But while I was at it and considering that this comp.xml was new, I compared the RPM list of a "good" 5106R and this new 5106R KVM with each others and found a few missing "must have's". So I fixed the comps.xml to add these missing items and rebuilt the repository. Likewise I noticed that the GUI setup wizard and the login page were shown to me in Danish instead of English. A quick look under the hood showed a small problem in I18n.php, which needed a fix to that module.
Now it was time for another Revisor run and rolling up yet another ISO for testing. Hmpf ... 768MB again. More fat trimming first. Oh, what's that? Why do we have two separage java RPM's, both 38MB of size? One can go. While we're at it, throw japanese-fonts (20MB) and man-pages-ja (8MB) away, too. GCC? Not needed. Let's see ... now we're talking: 681MB. I could really swing the nerf bat and trim it down a little further (some of the libs look like we don't need them). But every extra MB saved will take more and more effort and testing. On day three of this endeavour I don't have the steam for that anymore.
What's left to do: The image on the splash screen of the CD shows (c) 2008-2011 and would need to be replaced. Which would require fiddling with Adobe Fireworks (my graphics tool of choice), saving a GIF with reduced palette, using giftopnm and ppmtolss16 to turn it into a splash.lss file. And more testing. Sorry, can't be bothered right now.
This concludes a 48 hours work week and it's only just Wednesday early afternoon. If I weren't my own boss, I'd complain and would ask for a raise. ;-)
Instead I just invest a few more minutes to adapt my existing Makefiles to automate the building of updated ISO's in the future, so that ...
cd /home/build_cd
make installer
make revisor
make import
make iso
make storage
... again will yield an updated ISO within a matter of minutes.
Conclusions:
Respinning the media on CentOS-5 is a much more tricky endeavour than it ought to be. The reasons for this are:
- Revisor is missing from upstream.
- Pungi missing from upstream.
- The CentOS-5 supplied Anaconda has known show stopping bugs, so buildinstall is not an option either.
- buildinstall fails building the stage2 images if the build box runs on OpenVZ (device-mapper required!)
- The Revisor from Epel doesn't honor the usual mechanisms to exclude certain packages.
- Python-2.4 prevents usage of newer self-built Anaconda.
All in all this situation is outright ridiculous. I don't really mind Karanbir telling everyone to use buildinstall. If it were working to begin with and you wouldn't smash your heads into this wall of bricks there to begin with! But thanks to the messed up Anaconda from upstream this is ain't really an option either. Well, thanks for the bad advice and the wasted time! \o/
<Sigh>
← Return