CD-Building for CentOS6
Due to popular demand I set out to build 5107R and 5108R ISO's based on CentOS-6.2.
After my recent stint with building an updated CentOS-5.8 ISO for BlueOnyx 5106R I was less than thrilled to do the same for CentOS6 and 5107R and 5108R. After all, I have come to hate CentOS with a passion.
CD-Building, Day 1:
First steps involved setting up an OpenVZ VPS based on CentOS-6.2-x86_64 on an Aventurin{e} 6106R node.
With the easy part then being over, things got interesting. I fully YUM updated the VPS, installed some "must have" stuff such as alpine, mc, autoconf, automake, gcc, anaconda and what not.
I then went to Epel and a couple of other third party repositories in a hunt for a Revisor for EL6, but I only found the one for Scientific Linux 6. Ok, so off to build the latest Revisor from the sources as outlined here. That was fairly simple.
I then proceeded to install the local build environment that I use to build the Scientific Linux 6.2 based 5108R. It's basically a local YUM repository for the blueonyx-cd-installer RPM and a set of makefiles and scripts that pounces through the various steps involved building our CDs.
It builds the latest blueonyx-cd-installer RPM and publishes it to the local YUM repository. Then it runs Revisor, imports the Revisor output directory into our work directory and strips out a few unneeded bits and pices. Finally it builds the ISO, implants the MD5 sum and publishes the ISO to the download server.
A few adjustments had to be made and the "stock" Revisor needed to be populated with the right config files. This was mostly done by copying the existing SL-6.2-5108R configs and just changing a few bits and pieces where applicable.
The first run of Revisor quickly terminated with this error message:
No Package Matching kernel-xen. This is a required package.
Ok, I had seen that before. RHEL6 clones no longer have a XEN kernel, so no wonder why it fails here. It is more of a surprise that the latest Revisor from GIT still has that requirement in it. It even complains when I specifically add ...
-kernel-xen
... to my kickstart config and set ...
dependency_resolve_allow_conflicts=1
in revisor.conf to allow RPM dependency problems.
Eventually I found a ticket on Fedorahosted which mentioned the problem and outlined a possible resolve.
So I edited /usr/lib/python2.6/site-packages/revisor/cfg.py and removed all "kernel-xen" lines from it.
The next Revisor run got a little further, but also croaked:
Creating Repository Information: ######################################## 100.0% Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/revisor/__init__.py", line 528, in run
self.base.run()
File "/usr/lib/python2.6/site-packages/revisor/base.py", line 108, in run
self.cli.run()
File "/usr/lib/python2.6/site-packages/revisor/cli.py", line 44, in run
self.base.lift_off()
File "/usr/lib/python2.6/site-packages/revisor/base.py", line 885, in lift_off
self.buildInstallationMedia()
File "/usr/lib/python2.6/site-packages/revisor/base.py", line 1167, in buildInstallationMedia
self.plugins.exec_hook('pre_exec_buildinstall')
File "/usr/lib/python2.6/site-packages/revisor/plugins.py", line 193, in exec_hook
exec("self.%s.%s()" % (plugin,hook))
File "<string>", line 1, in <module>
File "/usr/lib/python2.6/site-packages/revisor/modmock/__init__.py", line 58, in pre_exec_buildinstall
from mock.backend import Root
ImportError: No module named mock.backend
Traceback occurred, please report a bug at http://fedorahosted.org/revisor
/var/tmp/revisor/var/lib/rpm: No such file or directory
A Google search later I was looking at this, but quite frankly: The installed Revisor and installed Mock are newer than that bug report. So I just resorted to uninstalling "revisor-mock" and tried again. This time Revisor managed to get a little further:
Creating Repository Information: ######################################## 100.0% Initting progress bar for Building Installation Images Building Installation Images: 0.0% Setting rundir to /var/tmp/revisor-rundir Running command: bash -x /usr/lib/anaconda-runtime/buildinstall --product BlueOnyx --variant
5108R-x86_64-updates --version 5108R --release "BlueOnyx 5108R" --prodpath Packages --bugurl
Your distribution's bug reporting URL --yumconf /etc/revisor/conf.d/revisor-c6-x86_64-updates.conf
/var/tmp/revisor-install/5108R/5108R-x86_64-updates/x86_64/os
Extra information: /var/tmp/revisor-rundir False None
bash: ++ pwd
Building Installation Images: 0.0% bash: + CWD=/var/tmp/revisor-rundir
Building Installation Images: 0.0% bash: + PRODUCTPATH=anaconda
Building Installation Images: 0.0% bash: + '[' 15 -gt 0 ']'
Building Installation Images: 0.0% Got an error from bash (return code 1)
/var/tmp/revisor/var/lib/rpm: No such file or directory
Note the text marked in red above. There is an apostrophe in it and that isn't taken too well there, as it's counted as unclosed quote. Hard coding the "bugurl" to "Nothing" in revisor.conf got around this, but the error message didn't change. So something else was amiss. After 20 minutes of Google search and trial and error, I found out that the command line string that Revisor uses to roll the Stage2 images ...
/usr/lib/anaconda-runtime/buildinstall --product BlueOnyx --variant 5108R-x86_64-updates --version 5108R --release "BlueOnyx 5108R" --prodpath Packages --bugurl nothing --yumconf /etc/revisor/conf.d/revisor-c6-x86_64-updates.conf /var/tmp/revisor-install/5108R/5108R-x86_64-updates/x86_64/os
... simply does not work with this version of anaconda-runtime's buildinstall. \o/ But this one ...
/usr/lib/anaconda-runtime/buildinstall --version 5108R --brand BlueOnyx --product BlueOnyx --release "BlueOnyx 5108R" /var/tmp/revisor-install/5108R/5108R-x86_64-updates/x86_64/os/
... goes through, although it produces a fair chunk of error messages and/or warnings:
[root@c664 build_cd]# /usr/lib/anaconda-runtime/buildinstall --version 5108R --brand BlueOnyx --product BlueOnyx --release "BlueOnyx 5108R" /var/tmp/revisor-install/5108R/5108R-x86_64-updates/x86_64/os/
Running buildinstall...
/tmp/buildinstall.tree.AqlpHD /home/build_cd
anaconda-13.21.149-1.el6.centos.x86_64.rpm | 3.0 MB 00:00
/home/build_cd
Building images...
Assembling package list...
Mon Apr 9 01:35:32 MSD 2012 Expanding packages...
install-info: No such file or directory for /usr/share/info/sed.info.gz
install-info: No such file or directory for /usr/share/info/libidn.info.gz
install-info: No such file or directory for /usr/share/info/which.info.gz
install-info: No such file or directory for /usr/share/info/grep.info.gz
install-info: No such file or directory for /usr/share/info/groff.gz
install-info: No such file or directory for /usr/share/info/ipc.info
install-info: No such file or directory for /usr/share/info/gnupg.info
/proc is empty (not mounted ?)
awk: cmd. line:1: fatal: cannot open file `/etc/fstab' for reading (No such file or directory)
/proc is empty (not mounted ?)
/proc is empty (not mounted ?)
/proc is empty (not mounted ?)
/proc is empty (not mounted ?)
SELinux: Could not downgrade policy file /etc/selinux/targeted/policy/policy.24, searching for an older version.
SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.24: No such file or directory
install-info: No such file or directory for /usr/share/info/grub.info.gz
install-info: No such file or directory for /usr/share/info/multiboot.info.gz
install-info: No such file or directory for /usr/share/info/wget.info.gz
Mon Apr 9 01:45:09 MSD 2012 Installing files
[...]
All in all it did run into a couple of pages of error messages and warnings. Especially one about missing "device-mapper" had me worried that again this buildinstall will not work under OpenVZ where device-mapper is not available.
But in the end it dumped the stage2 images out, although it remains to be seen if they will work. So I modified the rest of my scripts and makefiles to import whatever bits and pieces we need from the Revisor and buildinstall run into our CD-building directory. That directory then contained 818 MB of data and the resulting ISO had a size 815 MB. So that's roughly 115 MB more than it should have, but for a first test it'll do. Off to install it inside a KVM.
Right off the bat I discovered a small glitch with the isolinux.cfg file, which shows the boot menu of the CD. That was easily fixed and another iSO was rolled. After formatting the disks, Anaconda did run into an error:
Well, that is basically what you get when you run Revisor with the switch that allows missing dependencies. So in this case the RPM "glib" is missing from the CD. Which is kinda funny. Sure, our comps.xml that defines "BlueOnyx" doesn't specifically list it. But it ought to have gotten it anyway during the resolving of the dependencies. So I edited the kickstart script and added "glib" as requirement. Just to be sure it also went into the comps.xml file of the 5108R YUM repository. Then off to build another CD. And it still failed on getting "glib". A closer look revealed that "glib" is in the SL YUM repository, but not in the CentOS YUM repository <sigh>.
So I took the SL-6.2 "glib" and shoved it into the BlueOnyx 5108R YUM repository, too.
Next I edited /usr/lib/python2.6/site-packages/revisor/cfg.py again to throw out a lot of the X11 related dependencies that Revisor forces me to grab for no good reason. That started a struggle with dependencies which either Revisor insisted on during the conflict resolution phase and those that Anaconda required for building the Stage2 images. It took several edits of the kickstart.cf to add or remove ignores and edits of the Revisor yumconf to ignore or not ignore certain RPMs. After that the resulting ISO was down to 769 MB with still some undesired stuff on it. Off to install a KVM with it.
That went well then <sigh>. Part of the problem here is that the stage2 installer doesn't honour ignoring dependency issues and Anaconda won't let me remove certain unneeded stuff like redhat-lsb-printing. OK, how to fix that? Part of the problem is to know where to fix it. In Revisor to fix the stage2 image? In Anaconda?
For starters I edited /usr/lib/python2.6/site-packages/revisor/pungi.py to get Revisor to build the stage2 images for CentOS-6 without bugging out. For that the appending of the --prodpath and --yumconf switches to buildinstall had to be commented out.
After another ISO build and still running into dependency issues during Stage2 dependency checks, I finally did the sensible thing to compare my GIT installed and now modified Revisor with the Revisor from SL-6.2. Off the bat I found similar modifications to pungi.py that I had already done. But it went a lot further than that.
So all things considered I will probably be better off to give the Scientific Linux 6.2 Revisor a try on CentOS 6.2. But that'll be left for another day.
CD-Building, Day 2:
After not getting far with the latest Revisor from GIT, it was time to look for an alternative. After comparing the GIT built Revisors a bit more closely with the SL-6.2 Revisor I must say that I'm a bit surprised. Both at the excellent job that the Scientific Linux maintainers did there - and the lack of RHEL6 provisions in the latest Revisour sourced off the official GIT repository. After all, it is not like RHEL6 came out just yesterday <sigh>.
As I didn't (yet) want to loose my custom Revisor install entirely, I cloned the CD-building VPS. On the clone I then uninstalled the custom built Revisor and installed the x86_64 version of the SL-6.2 Revisor.
After bringing my custom BlueOnyx related Revisor config files back, I started to build the first 5108R ISO with the SL-6.2 Revisor. Bloody hell - what a difference. It worked like a charm all the way through - including the building of the Stage2 images. Just one small glitch: The Packages directory contained the "sl-release" RPM instead of the "centos-release" RPM, although my /etc/revisor/revisor.conf had these lines in them:
release_pkgs = ^centos-release.*$
release_files = GPL RPM-GPG-KEY-beta RPM-GPG-KEY-CentOS-6 RPM-GPG-KEY-CentOS-Debug-6 RPM-GPG-KEY-CentOS-Security-6 RPM-GPG-KEY-CentOS-Testing-6
That was a bit surprising. So I did two things to mitigate that. I specifically added the dependency centos-release to my kickstart file and also copied the above section into the model section of my revisor.conf:
[5108R-x86_64-updates]
description = BlueOnyx 5108R for x86_64
main = /etc/revisor/conf.d/revisor-c6-x86_64-updates.conf
product_name = BlueOnyx
product_path = Packages
iso_label = BlueOnyx
iso_basename = BX-Updated
architecture = x86_64
version = 5108R
version_from = RHEL6
getsource = 0
#use kickstart file to determine what is included in compose
kickstart_manifest = 1
#product_img = /etc/revisor/SL6/images/sl6-product.img
release_pkgs = ^centos-release.*$
distroverpkg = centos-release
release_files = GPL RPM-GPG-KEY-beta RPM-GPG-KEY-CentOS-6 RPM-GPG-KEY-CentOS-Debug-6 RPM-GPG-KEY-CentOS-Security-6 RPM-GPG-KEY-CentOS-Testing-6
mode_respin = 1
That made the difference. Also note that I added the "distroverpkg" line that I had found while going through the command line options listed somewhere in the Python source code of Revisor.
The generated directory for this 5108R CD was roughly 650 MB and the resulting ISO turned out to be 647 MB. Why the already immense savings right off the start? That's another kudos to the Scientific Linux maintainers: They ripped out all the crappy dependencies that one doesn't really need. So now off to install the new ISO.
The install again did choke on dependency issues. But this time around we have a much more streamlined ISO, so I can afford to bring the missing dependencies back all in one go. With the added dependencies the size of the ISO is 679 MB, which is still within acceptable limits for now. Off for another test install via KVM. This time the dependency resolution goes through.
Surprise, surprise: BlueOnyx 5108R on CentOS-6.2 installed just fine off the new ISO. After the install I rebooted the KVM and realized that the "blueonyx-cd-installer" RPM was missing <sigh>. True enough, the local repository was missing from my Revisor configs. Once that was fixed, it was off for another Revisor run.
And this one finally installed fine. No problems during KVM install or after the reboot and setup. A brief testing of the GUI and a look under the hood showed no defects either. So that one appears to be tightly wrapped up. \o/
Now off to build a BlueOnyx 5107R on CentOS-6.2 <sigh>.
I had never tried to build an i386/i686 ISO on an x86_64 box with Revisor. Never had to, because typically we have at least one "buildbox" for each BlueOnyx version. Adding just another VPS for building BlueOnyx 5107R on CentOS-6.2 appears wasteful if there is no real need for it. So I gave Revisor a try to roll up the 32-bit ISO on the 64-bit buildbox. After creating the right config files for that purpose I fired up Revisor and let it run. It started nicely and grabbed all the right RPMs and downloaded them nicely. Then it set out to build the Stage2 images for the CD and that's where it failed hard:
bash: + /tmp/buildinstall.tree.vs0ZYN/maketreeinfo.py --family=BlueOnyx --variant=5107R-i386-updates --version=5107R --arch=x86_64 --outfile=/var/tmp/revisor-install/5107R/5107R-i386-updates/i386/os/.treeinfo [...] bash: ++ echo 'cannot find package kernel' bash: cannot find package kernel
The problem is highligted in red in the output above. The scripts that generate the images incorrectly assume x86_64 instead of the i386 specified in the config. That's because the anaconda-runtime scripts to build the images use "uname -m" to override the architecture and to use whatever architecture the build box has. That's all stuff you'll have a hard time finding in the manpages <sigh>.
All in all I wasn't really surprised and I'll have to set up another VPS instead to roll up ISO's for BlueOnyx 5107R on CentOS-6.2. But again that's enought frustration for one (long) night and a job for another day.
CD-Building, Day 3:
With the hardest parts now out of the way, I set out to set up a 32-bit CentOS-6.2 OpenVZ VPS on Aventurin{e} 6106R and yum updated it. I then installed and activated the YUM repositories for EPEL and BlueOnyx and set out to install the same RPMs as on the 64-bit CentOS-6.2 CD building VPS. To make sure everything was identical, I checked which RPMs were installed on the 64-box and made sure all of these were installed on the 32-bit build box as well. Naturally I had to bring aboard Pungi or Revisor as well.
So once everything but Revisor and Pungi was installed, I grabbed those as well from the 32-bit Scientific Linux 6.2 YUM repository.
Finally I copied the Revisor configs, build directory and local yum repository over from the 64-bit buildbox to the 32-bit buildbox.
The first run of Revisor already produced a good result and finished without fail. The ISO built from that turned out to have a site of 653 MB, which is a pretty awesome size and had me worried for a moment that some stuff was missing. But it was all there.
Off to install it in a KVM.
For a moment I was a bit worried, as the install seemed slow to a crawl once it had reached 100%. But when checking the various consoles I could see that it was still doing the finalization of the install. Eventually I got the screen that I had been waiting for:
After rebooting the KVM I quickly went through the initial setup and did run a few tests. It all checked out and we now also have a BlueOnyx 5107R on CentOS-6.2.
Conclusions:
The respinning of installation media on CentOS 6 is certainly less frustrating than on CentOS 5. But it only gets really easy if you use the Revisor from the much superiour RHEL6 clone and the sister project of CentOS: Scientific Linux.
- The CentOS supplied Anaconda is OK and holds up nicely.
- Revisor and Pungi are missing from CentOS
- Latest Revisor from GIT has no RHEL6 provisions and is unuseable
- Revisor from Scientific Linux 6 works perfectly on CentOS 6
- That Revisor & Anaconda don't depend on device-mapper and work in OpenVZ containers
So I tip my hat towards the Scientific Linux maintainers again for delivering a superiour job. Well done, ladies and gentlemen. Thank you!
← Return