BlueOnyx: Sauce::Service

We overhauled a core feature of BlueOnyx to make service restarts more seamless.

BlueOnyx: Sauce::Service

In BlueOnyx all service related transactions are performed through our Sauce::Service Perl Module. It allows us to check if a service is enabled, if it is running and we can also enable or disable it or change its run state. Like restarting or stopping it.

This all usually works quite well and Sauce::Service over the years has already seen some overhauls and improvements. Yet one issue remained:

Ever since we integrated PHP-FPM a restart of Apache (and Nginx) usually also entails restarting PHP-FPM. This creates a long list of service restart requests and (sadly) many of them are also quite repetitive.

This has now been solved by a redesign of the Sauce::Service Perl module and the respective YUM update has been published for BlueOnyx 5209R, 5210R and 5211R.

General functionality of Sauce::Service

Sauce::Service has always consisted of three parts and this has remained unchanged:

- Sauce::Service

Toplevel functions for service handling

- Sauce::Service::Client

Pushes service related transactions to Sauce::Service::Daemon

- Sauce::Service::Daemon

Processes the queued service related systemctl calls

In a typical CCE handler or constructor we can restart services in one of two ways:

Sauce::Service::service_run_init('service-name', 'restart', 'nobg');
Sauce::Service::service_run_init('service-name', 'restart');

The first one (the one with the 'nobg' parameter) tells Sauce::Service to not queue the request into the background, but to execute it directly and wait with the processing until the restart has happened.

The second request (without the 'nobg' parameter) tells Sauce::Service to queue the request and to batch-process them.

This batch-processing was done by Sauce::Service::Client, which fired up a short lived instance of the Sauce::Service::Daemon. Which then processed the queued requests and if there were no more to process, it would eventually stop.

This batch-processing was a really good idea from the original Cobalt developers, but it had its flaws. Sauce::Service::Client talked to Sauce::Service::Daemon via a Unix socket and any daemon using Unix sockets can either listen on the bloody socket, or it can process the transactions. But not both at the same time. That they also uses ALARM handlers and SigChld calls to trigger runs of the queue-processing didn't make it any easier, as these have their quirks as well. This did lead to some "interesting" runtime issues and general shenanigans.

It was a genius stroke at the time of its inception +25 years ago, but sub-optimal for modern times.

Functionality of the modernized Sauce::Service

Sauce::Service itself remained mostly unchanged.

Sauce::Service::Client no longer starts Sauce::Service::Daemon and its only job now is to write a flat file database into /usr/sausalito/services/. These contain the name of the service to be handled, WHAT to do (restart, stop, whatever) and a timestamp. If a database entry is already present for a specific service, then this entry will be updated instead. It gets a new time-stamp and the instruction what to do (stop, restart, reload) may get upgraded or downgraded depending on the situation.

Sauce::Service::Daemon is now a fully fledged daemon that runs persistently in the background. It is started via "systemctl restart sauce_serviced.service" and is enabled and running by default. If it fails, Systemd will restart it the same way it makes sure that cced.init is always running:

[root@5211r ~]# systemctl status sauce_serviced.service  
● sauce_serviced.service - Sauce Service Daemon 
     Loaded: loaded (/usr/lib/systemd/system/sauce_serviced.service; enabled; preset: disabled) 
     Active: active (running) since Mon 2023-08-14 11:49:02 -05; 5h 13min ago 
   Main PID: 1846032 (sauce_serviced) 
      Tasks: 1 (limit: 151932) 
     Memory: 4.7M 
        CPU: 339ms 
     CGroup: /system.slice/sauce_serviced.service 
             └─1846032 sauce_serviced 
 
Aug 14 11:49:02 5211r.smd.net systemd[1]: Starting Sauce Service Daemon... 
Aug 14 11:49:02 5211r.smd.net sauce_serviced[1846032]: Daemon.pm: sauce_serviced[1846032] Ready to accept requests 
Aug 14 11:49:02 5211r.smd.net systemd[1]: Started Sauce Service Daemon.

The only job of Sauce::Service::Daemon is now to periodically (every 15 seconds) check /usr/sausalito/services/ for transaction files and if it finds any, it checks their timestamps. If a transaction file is older than 10 seconds, it will be processed and the queued systemctl transaction will be executed. A "stop" request will be processed almost immediately, anything else must wait for its turn.

This means that the usual ...

Sauce::Service::service_run_init('<service>', 'restart');

... (as long as no 'nobg' is added to it) will NOT be executed right away. But with a delay. This delay allows us to weed out almost all the redundant restarts for the same services that otherwise might happen in short order.

Results:

Say you save the PHP settings in the GUI and it may take between 30 seconds and a minute until the related services are restarted. You don't have to wait that long in the GUI, though, as the restarts happen in the background after all SET transactions against CODB are done. So once you hit Save, you fairly quickly see the GUI page reload and show the results, while Sauce::Service::Daemon is still doing its magic.

The main benefit is that Systemd no longer is drowned in a flood of redundant service restarts, which enhances stability. Due to the background processing of these service restarts the GUI is a bit more dynamic as well.