Thoughts on Systemd

There is currently an interesting Systemd related discussion on the Linux kernel developer list. My take on this - and Systemd in general.

As you might know, I have my gripes with Systemd and the more I see or hear about it, the more I think that Linux is not doing itself a favour here.

One thing is sure: Systemd is one of the most controversial projects in Linux-land right now. How controversial? So controversial that Lennart Poettering, one of systemd’s developers, even claims that horrible people have been pooling Bitcoins to hire a hitman on him. Which - if you ask me - sounds like a splendid idea. Where can I contribute?

The Good.

Jokes aside and on a more reasonable level: The number of Linux distributions that does not include Systemd is shrinking. Debian just released 'Jessie' and it comes with Systemd as well. The only good thing? It also includes SysVinit and both actually seem to coexist under 'Jessie'. One of Debians reasons for switching to Systemd? GNOME is becoming more dependent on it over time. Which is one of Debian’s stated reasons for switching back to GNOME. Systemd is everywhere. And it cannot be avoided. Even if you wanted to.

I am sure the Systemd guys have been spitting nails about Debian 'Jessie' including both Systemd and SysVinit, as they do their best to make Systemd the only init service on Linux and even actively discourage RPM builders from including both Systemd 'Unit'-files and traditional init scripts in the same RPM. Only a few RPM builders actively resist or contest that kind of bullshit.

Such as the SpamAssassin developers. They started to include Systemd Unit scripts as an afterthought in their tarball and RPM Specfile and the resulting RPM then contained Upstart, SysVinit and Systemd support. When the Systemd guys got bitchy about it, the SpamAssassin guys did the right thing: They told them to stuff it where the sun doesn't shine. I applaud them for that and wish more people would do that.

The Bad.

Even Systemd critics (such as myself) largely agree that SysV is old and could use a replacement or servious overhaul. But Systemd is much more than an init service and contains many other bits of functionality. However, Systemd is a software suite, not just an init system. 

The Ugly.

The systemd project also contains logind, a daemon that manages user logins, and journald, an event-logging system that controversially writes to binary files and not text ones. Systemd has also absorbed the udev project and its code, which handles the management of virtual device files in the /dev/ directory and events when devices are plugged in and unplugged. The list goes on and on: systemd also includes a cron-style task scheduler and networkd, a daemon for managing network connections.

More recently, systemd is gaining consoled, a user-mode console daemon that can be used when Linux’s virtual terminal code is stripped out of the kernel itself. The kernel developers seem happy to get this stuff out of the kernel and into user-space. But some people have to be thinking: Does Systemd really have to take over this as well?

The Liars.

As you might (or might not) know, the Systemd guys built a new kernel component called "kdbus". It was supposed to be part of the Linux kernel starting with Linux 4.1. But now the first release candidate of Linux 4.1 came out without kdbus-support. Once a release candidate is out, new features have to wait for the next release and that window of opportunity is closed.

So what happened? See for yourself over here. This whole thread on the Kernel developer list is about the Systemd guys reasoning why their kdbus is so much more efficient than the old dbus and that 'something has to be done' to include it, because 'this crap cannot go on like this'. So Linus stepped in, took kdbus and tried to build it. It blew up right into his face with the usual crappy build problems between 'configure' and 'pkg-config' that entail most of these Systemd components and sub-projects. Their code just plain stinks.

The fun part: Linus runs a profiler to find out the reason and source of the reported slowness. And guess what? It's not the kernel-related part, but Systemd's horrible userspace integration. And that sweeps the entire 'we need to replace dbus with kdbus due to speed reasons!' off the table. In Linus' own words:

My guess is that pretty much the entirely of the quoted kdbus "speedup" 
isn't because it speeds up any kernel side thing, it's because it avoids 
the user-space crap in the dbus server.

IOW, all the people who say that it's about avoiding context switches 
are probably just full of shit. It's not about context switches, it's 
about bad user-level code.

Eric W. Biederman goes even one step further and places some very well worded criticism:

[...] you are pushing entirely too hard for this code to get it.  When
someone pushes as hard as you are doing, inevitiable problems get
through.

[...] it is my professional opinion that the code smells.  There are all
sort of missteps and oversights that indicate that almost certainly that
something important has been overlooked.

I do not believe this code is yet up to the standards we want for core
kernel code.

[...]

Further by refusing to tease apart the pieces.  By refusing to allow
other people time to understand this code.  By refusing to give an inch
and admit anyone else has a valid point real problems, and real issues
can not be revealed and fixed.

With such a pig headed direction I do not believe that kdbus is in any
sense ready for merging.

What do we learn from that? The Systemd guys are actually lying to get their crappy code into the kernel. It's neither needed, because it is not faster. Because their code is as bit as worse performance wise than the stuff it actually tries to replace. But worse? The claim that it is more secure falls flat on its face as well, as their code is not just put together badly, but also willingly and "by design" exposes things to userspace that it shouldn't. Bluntly put: They've been found with the hand in the cookie jar. Which Linus explains here:

So I really don't understand why that part is even controversial.
kdbus wasn't meant to be some generic IPC mechanism. It is meant as a
way to talk to system daemons.

So the whole "capabilities and user information" is really to me a
non-issue. It's clearly required information, and if you don't want to
expose it, you damn well have absolutely *zero* business talking to
system daemons.

Really, it's that simple.

But things like "comm" and the cmdline? That makes me nervous. There
are real privacy issues there. Sure, maybe you think it's useful for
debugging, but the very fact that you think it's useful for debugging
makes me suspect you might be logging it (for future debugging). And
quite frankly, I don't think you should be logging things like that.
Yes, yes, if you're a system admin, you can find those things out, but
they should *not* be something that you just end up logging by mistake
or because "it's easy and all the information is right there".

If somebody is printing something, it shouldn't matter if it's "lpr"
or "firefox http://horses.and.trannyporn.my.little.pony.com/" that
does the printing.

And you can go "but we don't log it" all you want. It's still a bad
idea. Sane people should refuse to allow a system service to see those
kinds of things by default, for a very simple reason: it's none of
their business.

The Future.

/soapbox mode on:

What I say next is totally speculative, but think about it for a moment. If you wanted to "compromise" and "infiltrate" Linux, how would you go about it? The Kernel developers are very bullshit resistant and even though there have been cases where someone tried to sneak some "fishy" code into the Kernel, this never got very far. There are too many watchful eyes auditing that code and that is a very good thing.

On the other hand: Look at OpenSSL. Almost the entire internet depends on it and until recently it was just maintained by one clueless, overworked and underpaid maintainer, who actually got paid by a fishy organization that has an agenda influenced by the US government. Once people started to look closer, a lot of ticking time bombs have been found and defused, but there is still a long way to go.

If someone wanted to "infiltrate" Linux, he would need to submit a "must have" killer-application that everyone wants. Preferably a very invasive one that operates on both the kernel and userspace level and has it's slimy tentacles in everything from the moment the box boots. It doesn't have to be incredibly better than what it replaces. Just make it reasonably better. Once you have the foot in the door make it spread out and replace other system components as you go along. All in the sake of "providing a better integration", where in fact you're just trying to make it more difficult to ever replace this monster of an Init service with something less invasive. Just look at the GNOME project which fell for it hook, bait and sinker in a way that GNOME now won't run well unless you have Systemd to begin with.

In my opinion the conflicts on the Linux kernel developer list are just the begining of a much larger conflict that has the potential to do a lot of damage to Linux as a whole. The Systemd developers are getting more influence due to the widespread adaption of Systemd and they use this gained "throwing weight" to force more of their shitty code and shitty procedures onto us. What's worse: They have a really shitty track record when it comes to security and performance, too. What's worse: Instead of accepting valid criticism and suggested improvements from authorities such as Linus Thorvalds they get all defensive, lie, cheat and continue to push their own agenda and come back with more crappy code.

This is far from being over and I'm not sure where it'll take us. But I'm sure it won't be pretty. In my not so humble opinion Systemd has to go. The sooner, the better. But I'm realistic enough to know that it won't happen. So strap in. It'll be a rough ride.