5108R: Progress over a bumpy road

Development of he 64-bit BlueOnyx continued today.

After a good nights sleep (and nightmares about RPM dependencies) work on BlueOnyx 5108R continued today. After a full workday of 12 hours on it I'm currently rolling the third ISO image of it.

However, it was a really bumpy road to get this far.

After I had the basic build environment set up and all dependencies sorted, I compiled all BlueOnyx modules and shoved them onto a local repository and from there (after signing) to the public repository.

A few odds and sods didn't compile first time around due to some hard coded 32-bit oddities, but that was expected. I fixed them along the way and did a few SVN commits to save the changes.

However, when I had almost all done (except Zend Guard Loader, IonCube Loader and a few other non-essentials), I noticed that CCEd was failing. Like failing HARD:

Sep 15 06:57:03 5108r kernel: __ratelimit: 7 callbacks suppressed
Sep 15 06:57:03 5108r kernel: cced[22437]: segfault at 180 ip 00007feffcbe9aa7 sp 00007fffb047ac80 error 4 in libc-2.12.so[7feffcba2000+186000]

It segfaulted on every SET transaction. Meaning: Whenever a write action to our CODB-database happened.

I played around with 'gdb' and the core-dumps, sifted through the sources, tried a few things and got nowhere. After four hours of poking around I was ready to throw the towel and asked for help on the developers list. Maybe someone firmer with C than me could figure it out.

But eventually I read through the backtrace again and discovered something that made me suspicious:

Sometime last year we added some debugging lines to CCE to be able to keep an eye on a locking issue that had often troubled us. Back in August I removed those 'LOGDEBUG' lines from CCE and released updated RPMs without these provisions.

However: The RPMs that I had built on 5108R still had these. And these logging attempts were causing the segfaults! So I checked the local SVN on the 5107R build box and lo and behold: I had forgotten to commit the 'LOGDEBUG'-removal to SVN! So on my 5108R build box I was one version behind as far as the CCE sources were concerned.

Once that was sorted, the segfaults were history. Yay!

Then it got a bit bumpy again: The 'admin' user didn't get created, as I had forgotton to enable quotas for my /home partition on the build box. Once that was sorted, I got as far as to the initial web based setup wizard, where I was greeted with a cracklib error upon attempting to set a password.

After some poking around, I noticed a matching bug report on PECL. On 64-bit it worked with the wrong integer format. They had the code fixed in SVN, but not in the downloadable tarball. Well, that was sorted quickly by just getting the updated file from their SVN.

Getting one step further, PAM and pam_abl started to get into the way. After all on 64-bit our libraries no longer reside in /lib/*, but /lib64/*. This required a few Makefile and Specfile updates in various places and making the code detect upon building if the target is 32-bit or 64-bit. But that was rather trivial.

Finally being able to get past the initial web based setup, I briefly checked the GUI and everything seemed to work fine. Creating a site failed with "Can't create system group", but I partially already expected that and turned to fixing this as well.

Finally the missing SRPMs were built (IonCube Loader, Zend Guard Loader) after fetching the right 64-bit sources for them from the vendor sites. I also built the RPM for the Solarspeed BlueOnyx repository, although I first had to create a 64-bit repository for 5108R. But this was trivial as well.

While I had been fuming over the segfault issue further above, I had already installed, configured and used 'Revisor' to roll up a 5108R ISO image. I finally tweaked our usual build environment in so far, that I could automate the builds of 5108R ISOs as much as we already have that in place for the 5107R ISO's.

In the time that it took me to write this article, Revisor has downloaded all the needed RPMs from the YUM repository and rolled up another 5108R ISO image. Which I'll be installing in a VirtualBox VPS in a moment.

There will probably be some lingering issues left, but all in all the road to a (more or less working) 5108R was less bumpy than I imagined. Three days of work (or a 40-hour Monday to Friday week for slackers) and it's almost done.

Not bad.