Controversial Lennart Poettering Finds His Place In Linux Community - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Software // Operating Systems
09:41 AM
Charles Babcock
Charles Babcock
Connect Directly

Controversial Lennart Poettering Finds His Place In Linux Community

The systemd developer produced an initialization system that no one liked, but everyone adopted for use with Linux containers. Are his days as a target over?

IoT Drives The Future Of Connected Cars
IoT Drives The Future Of Connected Cars
(Click image for larger view and slideshow.)

As Lennart Poettering talked about systemd (pronounced system D), I was struck not so much by his reputation as the enfant terrible of the Linux Kernel development process and more by the fact that he's a very bright programmer, commanding attention in an auditorium full of bright programmers.

Poettering took the podium at CoreOS Fest in San Francisco this week to talk about one of his favorite subjects, systemd, which he co-authored, and how it will help run Linux containers. I'm not going to pretend that I absorbed every nuance. There was a lot of nspawn and description=foobar stuff in it, and how all the stdout+stderr goes into the System Log and will never be lost. I'm still working on some of those details.

But mainly systemd is the controversial replacement for an old, familiar Linux initialization program, init. Systemd is about starting Linux faster, launching services, and when necessary, killing services off. It does so many things that init hadn't done before that it brought down the wrath of established Linux contributors and community as being contrary to the first rule of Unix development: Basic functions should be done as simply as possible. Systemd is a lot more complicated than init and tries to do a lot more. You can see where Poettering, perhaps in his naïveté and enthusiasm, ran afoul of this open source community.

Lennart Poettering takes the Fest podium.

Lennart Poettering takes the Fest podium.

Except that he and co-developer Kay Sievers have won. As a senior software engineer at Red Hat, Poettering received some validation when Red Hat adopted systemd into Fedora in place of init, and then into Red Hat Enterprise Linux itself. The Debian Linux distribution adopted it as well, which meant that CEO Mark Shuttleworth at Canonical, against his better judgment, was forced to follow. Ubuntu is based on Debian. Shuttleworth already had commissioned his own init alternative in Ubuntu, Mir display server, and he had no enthusiasm for systemd, but Debian's move forced his hand.

Poettering got a hearing at CoreOS Fest this week because CoreOS, a version of Linux optimized for running containers, uses systemd. Red Hat's Atomic Host, another Linux version optimized for running Linux containers, also uses it. It turns out systemd is good for fast startup and for killing off of containers, an operational point heavily in its favor now that containers are on everyone's to-do list.

Bucking The Boycott systemd Movement

But getting this far hasn't been easy. The Linux development process is known for its brutally direct comments on the code that contributors submit. That's a good thing because so many people want to make additions to Linux, which is already on the cusp of 20 million lines of code. A lot of discrimination rightly should be applied to what code gets in.

Over the course of 20 years, some of the discriminators have gotten rather good at their job, and Poettering, a German citizen, at first was greeted with a withering storm of criticism. Systemd contained 69 packages or code modules, compared to the much more compact init. It relied on declarative language commands, an approved coding concept in many settings, but an affront to init's existing use of shell scripts.

Prior to systemd, Poettering was the developer of PulseAudio, a sound server, and Avahi, a zeroconf protocol implementation for network device discovery. He had been active in 36 other projects. He started systemd in 2010, and, as it was picked up by more Linux users, a group of Linux traditionalists started a BoycottSystemd website and refused to use any distribution that picked it up.

[Want to learn more about Red Hat's commitment to containers? See Red Hat Launches Atomic Host To Run Containers.]

A sample from the BoycottSystemd site:

Systemd flies in the face of Unix philosophy: "do one thing and do it well." … Its responsibilities grossly exceed that of an init systems, as it [Systemd] goes on to handle power management, device management, mount points, cron, disk encryption, socket API/inetd, syslog, network configuration, login/session. ... Keep it simple, stupid.

But Poettering was busy mounting his response. In a January 2013 post, "The Biggest Myths," Poettering responded to his critics. By going beyond what init could do, systemd initiated socket-activated containers, which allowed a user "to drastically increase the density of your cloud system" -- meaning run more containers per host, a widely sought attribute.  

Systemd was not incompatible with shell scripts, "We just don't use them," he wrote. On the contrary it could work with 99% of them, if the user chose to employ them. He set out to refute "a few myths." When he was done, he had rebutted 30 specific criticisms of systemd in detail.

Last fall, in an Oct. 6 post to Google+, Poettering, 33 at the time, went further and said the Linux community had become too intolerant of newcomers with new ideas, a complaint that has been voiced before. He wrote:

The Open Source community is full of a**holes, and I probably more than most others am one of their favorite targets. I get hate mail for hacking on Open Source. People have started multiple "petitions" on petition websites, asking me to stop working. ... Recently, people start collecting Bitcoins to hire a hitman for me (this really happened!). Just the other day some idiot posted a "song" on YouTube, a creepy work filled with expletives about me and suggestions of violence. People post websites about boycotting my projects, containing pretty personal attacks.

Some of the criticism appears to be driven, not only because Poettering and Sievers weren’t doing the unglamorous follow-up work of fixing bugs, but also because they were weren't conforming to both spoken and unspoken rules about how the kernel could be modified.

Last year, the tension between systemd developers and the kernel process came to a head as a bug

Page 2: Linus Torvalds stokes the controversy.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
1 of 2
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Apprentice
6/28/2016 | 10:31:07 AM
Mir display server was not an alternative to systemd. There is no relation between the two (you're thinking of the Mir vs. Wayland debacle). Canonical's alternative init system was called upstart.
User Rank: Apprentice
5/11/2015 | 1:39:35 PM
kfv653: Error in article regarding Ubuntu ,Systemd and Mir
With respect, as a Linux user for almost 20 years, and a developer for over 25 years, I must disagree on almost every point you make. 

I read your discussion about Linux very carefully, and I cannot impress this enough: anyone who claims Linux is a desktop OS is "deranged" or living in a dream.  Linux was never intended to support "out of the box" commercial software NOR is it intended as a desktop operating system. 

In order to do either of those things, Linux would have been designed so companies could compile their Windows and other applications on Linux. It's not possible at the present time without extensive changes. Microsoft made certain of that when they withdrew support from the POSIX standard in 2000. The only exception is .NET code, which Microsoft recently opensourced, and that has been depreciated in favor of Windows Runtime as of Windows 8.  
If you want a desktop OS with "shrink-wrap" software, use Windows or MacOS.  It is what they are designed for.  You will be a lot happier.  Linux is designed first and foremost to be a UNIX server operating system.


As for your points regarding systemd:

 "People who want to have an older style init can still have it that way be configuring it how they want, SystemD supports the old SysV init as well as its new facilities."

 Sorry, but that is only partially true.  Systemd is 'mostly' compatible but will break a number of System 5 scripts. Systemd notes this in their own documentation.  For example, there are no inherited variables.  Systemd is not a "drop in replacement" for System 5.

"The way things were before is that it was impossible to do this."

Not true.  There were several service daemons to do these tasks.  Systemd just replaces all or most of them under a common framework, which consequentially also makes the software incompatible with everything except Linux.  For that reason, I do not consider it to be a good outcome.

"THe people who oppose it claim to be system administrators, they must not be very good system administrators if they find it so difficult to understand the basic model of systemd."

No, actually what they object to is combining all of the server services into a compound system.  The Systemd daemon itself is not as much the problem as the fact that its developers seem hell bent on replacing every service daemon with one that depends on systemd to work.  This adds attack surface to the operating system, e.g. if systemd is vulnerable, then all of the services that use systemd code are also.  It creates a point of failure, because it not only initializes the operating system, but actively manages daemons in session.  If the management portion of systemd crashes, the whole system goes down, and needs a hard restart.   

"If they claim to be system administrators, why dont they just use the system V failities to run their jobs, if thats what they want?"

Because they do not work properly in many cases, under systemd. See above or read systemd's own wiki.

 "I think what they are oppoising really is other administrators being able to actually have some control over initialization because previously with the Linux kernel and init it was hard to hook to certain system events and trace what is going on in the system. It was sort of a black box that administrators had little introspection over."

 Actually in my experience, syslogd solved that problem very well.  Systemd's binary logs have had a history of corruption problems.  The developers' response?  Restart the system and erase the logs because they are useless.  Doing that means that you won't find out what happened.  Sorry, but no.  I will use systemd on a personal desktop, but never on a server when keeping a paying customer hangs on whether I can fix a problem.

THis is because systemd DOES NOT take away anyones options it only adds additional options ..."

No, systemd is not the problem.  The Linux distributors are the problem.  They are the ones forcing their users to adopt systemd, rather than leaving the decision to the end user to decide what they need.  THAT is what is making people angry.

The only common versions of Linux that do not force their users to use systemd are Slakware and Gentoo, who have it as an optional install.  I would include Debian in that list provisionally, but future versions may not leave the option to not use it open.

"I think they are doing immense damage to the operating system because of this. I seriously question the intent of these people."

I think they have every right to complain when systemd causes real world problems.  Just as they have every right to complain when Windows crashes.  It does not "damage" anything to say that a problem is a problem.  I'd consider it to be far more damaging that the systemd developers ignore criticism as they do. It took Linus Torvalds suspending Key Sievers' merging privileges for causing kernel crashes before Sievers "got the message".

Charlie Babcock
Charlie Babcock,
User Rank: Author
5/11/2015 | 1:01:29 AM
In Ubuntu, systemd displaced Canonical's preferred Upstart.
I have read about two modules developed by Canonical for its Ubuntu distribution that it was forced to back away from. Randy P533 is right. I should have referred to Upstart as the one that systemd displaced.
User Rank: Apprentice
5/10/2015 | 6:28:01 PM
Re: Error in article regarding Ubuntu ,Systemd and Mir
I do not believe everyone hates systemd, in fact it seems to be a small, but loud, minority that do. I have worked with systemd and have found little issue with it. People who want to have an older style init can still have it that way be configuring it how they want, SystemD supports the old SysV init as well as its new facilities. systemd is sort of independensable and can be of great utility in being able to schedule jobs to be able to run on certain system events. The way things were before is that it was impossible to do this. systemd has give us only more flexibility and control over the system, and people who oppose it seem to oppose the administrator being able to have the control to be able to configure events in such a fine grained way that systemd allows.

I really don't know what the problem is. I didnt find any problems with the concept, its actually pretty useful. THe people who oppose it claim to be system administrators, they must not be very good system administrators if they find it so difficult to understand the basic model of systemd. If they claim to be system administrators, why dont they just use the system V failities to run their jobs, if thats what they want? I think what they are oppoising really is other administrators being able to actually have some control over initialization because previously with the Linux kernel and init it was hard to hook to certain system events and trace what is going on in the system. It was sort of a black box that administrators had little introspection over. 

Because systemd just provides additional facilities rather than forces anything on anyone, there really is not way to oppose it because people who want things to use old system V type init can do so, so thats obviously NOT what the uproar is about, those who oppose it seems to be trying to stop Linux from being any more powerful or flexible, they are te ones who oppose choice and flexibility in configuration because they want to deny the ability and customizability that systemd provides. THis is because systemd DOES NOT take away anyones options it only adds additional options so by opposing it they are trying to force thei own way of doing things on others. I think they are doing immense damage to the operationg system because of this. I seriously question the intent of these people. 


Another fact is while Linux people claim to be so superior and immensely powerful, and the OS to basically to have always been this perfectly useable thing the fact is that even for system administrators Linux has not been a particular flexible operating system for many years in terms of the out of the box software that was available, many Linux systems were down right inflexible and difficult to configure, and the OS has long suffered from a lack of well developed desktop applications, making it almost useless as a desktop OS. It had an HTTP server but this is far from really being able to do a serious server development platform, database systems for instance with failover support necessary in an enterprise environment were a long time coming. MySQL always lagged well behind Oracle. It was good enough for a small web site but when you are talking massive operations where downtime is not an option, it would fall way short. The point of the criticism is that the systems need improvenent but, I am making a point that whenever anyone does try to do something to make the system not so rigid that they are shot down by these people who have this crazy idea that Linux in 1992 was the most wonderful thing and that somehow not having a database with transactions and failover, redundancy and systems necessary for high availability was a wonderful thing. These people basically do not care about anyone else and any other use cases but their own and their own are often quite narrow purposed.
User Rank: Apprentice
5/9/2015 | 3:24:38 AM
Systemd and Mr. Poettering
With respect to your point of view, systemd's reputation with the community has suffered because systemd has caused real problems.

Firstly, it has caused problems with the Linux kernel itself.  This would be similar to someone writing a driver that crashes Windows to the "blue screen of death."  Generally, that is counterproductive, and when asked to fix systemd, its creators attempt to ignore responsibility.  Linus suspended Kay Sievers for this, and in my opinion, it was justly deserved.

The second concern is compatibility. In spite of claims to the contrary, systemd is not entirely compatible with System 5. Nor is systemd compatible with itself. Like so many undeciplined efforts these days, systemd does not guarantee a minimal fixed interface standard regardless of version.  This means that a single version update to systemd can cause your system become inoperable.

Third, Unix code is typically written to the POSIX standard, ensuring that it can easily be ported from one operating system to the next.  Systemd does not follow the POSIX standard. That in itself is not a problem.  There are many Unixes that  use code that violates the standard.  The problem is that projects such as Gnome are now building themselves around systemd, which makes them either crippled or unusable on anything except Linux. That flies directly in the face of the longstanding tradition of the Unix roots of the opensource community, and deliberately tying what was once generic code to Linux only has angered a great many.

Fourth, Systemd's goal is overreaching and forces homogenization.  In spite of what people howl, that it replaces init is of no concern.  Init is not actually part of the POSIX standard.  The problem is that the systemd project has been consistantly attempting to subsume other projects into itself, and suffers from poor project management.  Systemd makes legacy software incompatible, with no concern as to use cases.  It forces every Linux system to use the same software or become incompatible.  This might sound like a good thing to some, but one of the reasons that Linux is more resistant to being tampered with over Windows is diversity.  A diversity that the creators of systemd have set themselves to eliminate.

Lastly, systemd's precieved advantages can be acheived through less invasive means, as other projects, such as uselessd have set out to prove.
User Rank: Apprentice
5/8/2015 | 8:59:22 PM
Error in article regarding Ubuntu ,Systemd and Mir
Yea.  Good article but confidence in the  authors basic linux knowledge is low.....

There is a major difference between an init system and a display server.  Article states "Shuttleworth already had commissioned his own init alternative in Ubuntu, Mir display server".  The fact is Shuttleworth had commissioned his own init alternative in Ubuntu, --- Upstart --, and lost out to Systemd.  He has in fact commisioned his own Display server called MIR.  The rest of the community is appearing to put their support for Wayland as the next generation Display server, so Shuttleworth may be 'shut out' again.  

Charlie Babcock
Charlie Babcock,
User Rank: Author
5/8/2015 | 7:44:01 PM
VMs were hard to get into the kernel; containers too?
The merit of the code rules in the Linux community, right? So why have Poettering and Sievers taken so much heat? In my view, the Linux Kernel maintainers, especially Torvalds, were slow to develop any interest in virtual machines, and it took an artful second try by Qumranet to get a hypervisor (KIVM) folded into the kernel's operations in 2007. That meant the Linux community was five years behind VMware's proprietary ESX Server 1.0. Now containers have come along, with the initiative to get them included in Linux operations coming from outside the Kernel development process, primarily from Google. Understood that systemd, the initialization system that's best for containers, is non-conforming in several ways. But the merit -- the usefulness -- of the code still rules, right? 
CIOs Face Decisions on Remote Work for Post-Pandemic Future
Joao-Pierre S. Ruth, Senior Writer,  2/19/2021
11 Ways DevOps Is Evolving
Lisa Morgan, Freelance Writer,  2/18/2021
CRM Trends 2021: How the Pandemic Altered Customer Behavior Forever
Jessica Davis, Senior Editor, Enterprise Apps,  2/18/2021
White Papers
Register for InformationWeek Newsletters
Current Issue
2021 Top Enterprise IT Trends
We've identified the key trends that are poised to impact the IT landscape in 2021. Find out why they're important and how they will affect you.
Flash Poll