BlueOnyx 5211R Development Diary
Progress report on the BlueOnyx 5211R development
So far I mainly wrote on the BlueOnyx Mailing-List about the ongoing effort to develop a BlueOnyx 5211R on (or for) AlmaLinux 9 / RockyLinux 9 / RHEL9. Let me give you a detailed overview over the state of the BlueOnyx 5211R development here.
But before I do that, please note that developing BlueOnyx 5211R is a monumental task that requires a lot of effort and dedication. If you use BlueOnyx and would like to use BlueOnyx 5211R in the future? Then please consider supporting the development with a donation.
|
How it started
The preparation for this had started in November 2021 when RHEL9 Beta had been released. As it's always the case: Such a RHEL Beta is just a preview of the OS and lacking in many regards. I quickly realized that many of the dependencies that BlueOnyx needs were not satisfied by the RHEL 9 Beta YUM repositories and (of course) at that time there was no EPEL 9 or any other repository that would satisfy our needs. So a tedious process began and over the course of months I built roughly seven thousand RPMs (I kid you not) to populate the BlueOnyx 5211R development YUM repositories.
That got me to a point where I had mostly all required software lined up, but was missing MailMan (our mailing list manager) and a couple of other odds and sods that can be best described as non-essentials. The GUI was (mostly) working, but the old CodeIgniter 3 was creaking a little when running on PHP-8.0 and throwing the odd errors here and there. These issues were eventually ironed out. Still: Some non-essentials needed to be built and with these I was banging my head against the wall, as the RHEL 9 Beta made this a really tedious endeavor due to unsatisfied build dependencies and long list of dependencies for anything I was porting over from Fedora Core releases that *had* the SRPMs of what I needed.
Eventually I set the development aside waiting for the real RHEL 9 or an AlmaLinux 9 or RockyLinux 9 to be released.
AlmaLinux 9 Release
This finally happened in late May 2022. I cloned my RHEL 9 Beta development VPS and did an "in situ" transformation of it to AlmaLinux 9. By now EPEL 9 had also been created and been pre-populated with a lot of RPMs. Some of which I already had built myself in the months leading up to this point. So I went through the BlueOnyx 5211R development YUM repositories with a fine toothed comb to replace some of my older RPMs with newer ones from EPEL 9, untangled some dependency or naming issues of RPMs. There were also some small glitches to fix that resulted from differences between the RHEL 9 Beta and AlmaLinux 9 itself.
Still a few dependencies could not yet be satisfied to get the full functionality of 5210R baked into 5211R:
Mailman:
The old Mailman mailing list manager on 5210R still uses Python v2 and this has now (finally!) been deprecated. EL9 ships exclusively with Python v3 instead of having Python v2 and Python v3 installed side by side as EL8 did. There is a Python v3 capable Mailman3 around and newer releases of Fedora Core have it. EL9 does not and neither on EPEL9 or elsewhere I found something that would help us kickstart this. The problem being a mountain of Python related dependencies that need to be satisfied. Many of them being circular dependencies. So for building RPM "C" you need to build RPMs "A" and "B". But "B" needs "C" and "A" to be present in order to build. On RHEL 9 Beta I had spent *weeks* on this and couldn't shake it loose.
So for now 5211R will have no Mailman. This can eventually be rectified in the long run.
Z-Push ActiveSync:
Z-Push is an open-source application to synchronize ActiveSync compatible devices such as mobile phones, tablets and Outlook 2013 and above. We had it in BlueOnyx 5210R and prior to allow people to get Push notification to their devices for Email related activity. However, Z-Push seems to be a dead project and the latest PHP version it runs on is PHP-7.4. With PHP-8.0 (or newer) on EL9 we can't use it until newer versions of Z-Push arrive. But like said: It could be that this never happens as that project appears to be somewhat dead'ish.
IonCube / CodeIgniter:
Let me combine these two topics, as they are related. IonCube is used to encode PHP scripts to make the code unreadable to humans. It also serves a bit as an accelerator. Our PHP on a stock BlueOnyx ships with IonCube Loaders to enable the PHP to execute IonCube encoded PHP scripts. No part of the BlueOnyx GUI itself is encoded in IonCube to maintain the ideals of Open Source.
However: The GUI pages of commercially sold PKGs from the BlueOnyx Shop use IonCube to protect the license management that is built into these commercial offerings. So naturally the BlueOnyx GUI needs to run an IonCube enabled PHP for that to work.
The PHP framework that BlueOnyx uses since 2014 is CodeIgniter and the BlueOnyx GUI is built on top of that. BlueOnyx 5210R used CodeIgniter 3, which since a while has been superseded by CodeIgniter 4 (current version: v4.2.5 at the time of the writing of this article). There is no easy way to upgrade from CodeIgniter 3 to CodeIgniter 4. Too many things have changed - in many parts for the better. CodeIgniter 4 supports all recent PHP versions up to PHP-8.1. CodeIgniter 3 on the other hand kinda tops out at PHP-7.4. It was barely running on PHP-8.0 and croaking in a few places, but that could be fixed with some modest effort and most of that had already been done by me in the months leading up to this point. But there was no way in hell that it would run on PHP-8.1, as that introduces too many changes.
When building the IonCube RPM for BlueOnyx 5211R I suddenly realized that there was no IonCube loader for PHP-8.0! The IonCube support confirmed that they had skipped PHP-8.0 and gone straight to PHP-8.1 "to save development time".
It was now time to make decisions:
Release 5211R with CodeIgniter 3, PHP-8.0 and SourceGuardian as IonCube replacement? They have PHP-8.0 (and PHP-8.1) loaders.
Benefit: Shorter development time, faster release of 5211R.
Drawback: Maintaining an almost obsolete fork of CodeIgniter 3 throughout the entire life-cycle of EL9. Which will result in an even bigger development effort for BlueOnyx 5212R on EL10 whenever that comes out.
Rewrite the 5211R GUI with CodeIgniter 4, switch AdmServ to a custom built PHP-8.1 maintained by BlueOnyx and either keep IonCube and/or switch to SourceGuardian?
Benefit: More future proof GUI due to more modern CodeIgniter and more modern PHP. Shorter development time for subsequent BlueOnyx/Aventurin{e} releases. Possibility to rectify some bad design choices or less than optimal functions in BlueOnyx subroutines.
Drawback: Lots of work porting the GUI from CodeIgniter 3 to CodeIgniter 4, maintaining a separate PHP version out of the BlueOnyx YUM repos.
After some soul searching (and lots of cursing and cussing!) it was decided to do the right thing and go the extra mile for the sake of modernity: CodeIgniter 4, PHP-8.1, IonCube.
AdmServ, admserv-php-fpm.service, admserv-php-8.1.9-1
AdmServ still runs on Apache (a separate instance of it) and on 5211R that uses HTTP/2 out of the box. That means: PHP cannot be loaded as DSO. So AdmServ uses PHP-FPM. To be more precise: It's own instance of it, which is started as service "admserv-php-fpm". This now uses the PHP provided by "admserv-php", which is installed in /home/solarspeed/admserv-php/. We will provide rolling releases for it whenever a new PHP-8.1 comes out.
CodeIgniter 4
CodeIgniter v4.2.5 is installed in /usr/sausalito/ui/chorizo/ci4/ (instead of /usr/sausalito/ui/chorizo/ci/ where CodeIgniter 3 used to reside on 5210R and older). During the decision making phase I spent a few weeks both on making CodeIgnier 3 PHP-8.0 ready as well as familiarizing myself with CodeIgniter 4 to get a feel on how difficult it would be to re-do the GUI in it.
Off the bat I was facing several problems with CodeIgniter 4:
Routing:
That refers to the association between the browser URL and the decision making process of the framework which module page it should load based on the given URL. In the BlueOnyx GUI's so far we have a neat separation of modules. Here is an example from a BlueOnyx 5210R:
[root@5210r ~]# tree -d -L 2 /usr/sausalito/ui/chorizo/ci/application/modules/ /usr/sausalito/ui/chorizo/ci/application/modules/ ├── base │ ├── alpine │ ├── am │ ├── apache │ ├── api │ ├── backupcontrol │ ├── console │ ├── ddns │ ├── disk │ ├── dns │ ├── email │ ├── ftp │ ├── gui │ ├── import │ ├── istat │ ├── login │ ├── maillists │ ├── mailman │ ├── mailmangui │ ├── mysql │ ├── network │ ├── nginx │ ├── phpmyadmin │ ├── phpsysinfo │ ├── podman │ ├── power │ ├── raid │ ├── remote │ ├── shell │ ├── sitestats │ ├── snmp │ ├── ssh │ ├── ssl │ ├── subdomains │ ├── support │ ├── swupdate │ ├── system │ ├── telnet │ ├── time │ ├── user │ ├── vsite │ └── wizard ├── Compass │ ├── base │ └── webapps └── solarspeed ├── apf ├── av_spam ├── awstats ├── fail2ban ├── firewall └── openvpn
52 directories
As you can see: There are the directories "base", "Compass" and "solarspeed". Any GUI module in "base" is a stock BlueOnyx module. Anything in "Compass" is a commercial module made by Compass Networks and any module in "solarspeed" is a commercial module from SOLARSPEED.NET.
To realize such a separation of modules and to allow CodeIgniter 3 to find them we had used a third party CodeIgniter 3 module from "Wiredesignz" that we had heavily modified as well. This module isn't available for CodeIgniter 4.
To add insult to injury: While CodeIgniter 4 allows a wide range of different routing options: None of them fit our needs or came even close.
Solution: After a couple of days of haggling, Google searches and forum trolling we hacked our own routing mechanism together which ties into an existing CodeIgniter 4 class. Surprisingly it's just 4-5 lines of code. /facepalm
Central Scratchpad Data Storage
I couldn't find a better heading for this, so let me explain what I mean: The way CodeIgniter works is this: You supply the browser with an URL. The URL matches a route in CodeIgniter and that loads a Controller Class. Example /base/login/login.php for our login page. Or /base/vsite/vsiteList.php for the class that displays the list of Virtual Sites.
That controller then loads a metric fuckton (this is scientific term!) of other PHP Classes, Libraries and Helpers to do all kinds of funky things. Like handling Cookies and Session data, accessing our CCE backend, executing commands or manipulating files, showing you the text of the GUI in your specified language, rendering of GUI elements and lastly: Displaying you the GUI page output in HTML, CSS, jQuery, AJAX and JavaScript.
As things are many of these PHP Classes and Libraries need access to the same fundamental data: Your username, your sessionID, the "System" Object in CODB, your "User" Object in CODB, the language settings you have chosen and more. Many of these Classes and Libraries need access to CCE/CODB. For that they need to use CCEClient() and ServerScriptHelper() in PHP. Ideally we need to establish the connection and authentication to CCE only once and can re-use it until the end of your session. But the way CodeIgniter works? It kinda locks Classes and Libraries and Helpers into their own Sandbox an from some of them you then have issues accessing already aggregated data or object instances in other Classes.
In CodeIgniter 3 we had the option to use...
$CI =& get_instance();... in subordinate Classes, Helpers and Libraries to build a bridge and allow access to data that we had willfully exposed in other Classes, Helpers and Libraries. However, this was absent in CodeIgniter 4 and we had to hack support for it back into the framework.
Once that was done, we needed a place very early in the Controller execution where we would run preparatory tasks such as pre-loading Classes, Helpers and Libraries, initializing CCEd, verifying and managing access levels, determining language and locale settings and much more. In CodeIgniter 3 we had hacked this into "Wiredesignz" third party routing module. Luckily CodeIgniter 4 has something called a BaseController, which fits that purpose of preloading and pre-executing code on Controller execution perfectly.
So that is where all these logistical baggage went.
OMG! NOTHING works! \o/
So I started with the Login page of the GUI. It's not much, right? The BlueOnyx graphics on top, two form fields (username / password), HTTP/HTTPS selector and the login button. What got rendered in the browser? A pretty detailed error list.
But that was expected. After all: This "simple" login page? It requires access to CCEd, i18n for the locales, ServerScriptHelper, ArrayPacker and the encoding library. None of which CodeIgniter 4 yet had. It took a week to adjust the existing essential libraries and helpers to CodeIgniter 4. And that was a lot less work than I had suspected.
So right now we have a login page that renders, which is already a very big step. Because many of the essential fundamentals such as the ability to handle the nine different languages of the GUI via i18n is sorted, we can "talk" with CCE and do CODB transactions and rendering of basic GUI elements work.
OMG, it is working! \o/
There is another neat feature of CodeIgniter 4 when the development mode is activated. You can expand a bar at the bottom (or top) of the screen that allows you to toggle through various statistics. It shows loading time for the page and memory usage, loaded modules, loaded routes, some logs and more. One darn interesting part is this:
It allows profiling the application. This is usually somewhat bare, but you can set custom checkpoints in your code that then get aggregated into the display. Here is an example of that in the code:
When I first had the login page working, I had average load times of 900ms for it. That has now been trimmed down (as can be seen in the first image above) to 138ms. The profiling and custom check-pointing made it really easy to pinpoint and identify time sinks in the code.
Just some examples:
The GUI needs a working CCEd connection. So we sure want to know if CCEd is up! And if it is not up? Then we don't just want to throw an error, but restart CCEd, make sure it now runs and then continue with the code execution while skipping the whole "Hey! Something is broken!" error message part.
It turned out that the method I had used to check if CCEd was up was taking 140-150ms, whereas I could also do it in about 20-25ms. Worse: The time consuming check if CCEd was up? It wasn't just done once, but twice. Once in CCE.php and once in ServerScriptHelper.php.
Likewise: In a few places for lack of access to an already elsewhere opened CCEd connection some Classes had to fire up their own instance. Which means raise the connection, authenticate, parse the auth result, do the transaction, parse the result. Now with a central instance in BaseController we can skip these redundancies. Not only for CCEd connections, but also for the previously heavy getCookie() usage to fetch username and sesssionId and/or locale settings and subsequent access to an $i18n Object that is already open and ready for usage ever since BaseController fired.
So just for the login page we're looking at the speed increase (so far) of seven times faster.
BUT: This is only half the story. \o/
The way things are in CodeIgniter 4? I can cut down on a lot of baggage or legacy code as it is no longer needed. The old code that I port over or the new code that I have to write? I can directly and with onboard tools profile and "speed check" it to see if things performs better this way or that. This is finding bottle-necks made easy! In itself CodeIgniter 4 is also a little faster and PHP-8.1 *and* the combination of HTTP/2 and using PHP-FPM speeds things up nicely.
I'm excited to tell:
There is quite a bit of progress and the way it looks now? The new GUI will have quite a performance increase.
As there is still a whole mountain of GUI pages and libraries to port I can't make any solid promises or predictions when BlueOnyx 5211R will be ready for release. At the end of this month I might be able to make a tentative prediction on that based on the progress that I was able to make until then.
← Return