domenica 17 novembre 2013

It's just an init system

The recent debate for the choice of the default init system in Debian has tickled my curiosity to know more about the current situation of init systems and at the same time has led me to form a certain opinion on this matter, which I'm going to share here. It should be noted that I'm currently employed by Canonical (I take the opportunity to underline that the opinions expressed here are just my personal ones), and while this might have had some influence on the forming of my opinion, in all honesty I claim to not have noticed any of that — though indeed I'll leave the verdict to you, if you'll be patient enough to read the whole post.

From the little knowledge I have of these init systems, I'll start by saying that I like Upstart (and I'm a satisfied user of it), as well as systemd and OpenRC. I find the design of Upstart to be elegant, modular, and the implementation could be set as an example for students of what good code is. On the other hand, when in 2010 I first read about systemd from Lennart's blog, I was almost blown away: it seemed to me that every detail had been considered to make the booting phase as efficient as possible, and I was looking forward to its development. And I should state very clearly that — despite not having yet tried it — I now believe that all promises were met, and that systemd is an excellent init system. As for OpenRC, I know really little about it, other than it's out there, it works, it's maintained and is in progress of being ported to FreeBSD.

That said, my wish is that Debian goes with anything but systemd. And that of course is not limited to Debian, but I wish the same for any distribution which is pondering a choice for the init system. Why so? As I said, systemd is an excellent init system, it's used in several distributions, and I would be very happy if other init systems took inspiration from it for all the good stuff it has to offer. My concern is better expressed by quoting Lennart himself:
I believe ultimately this really boils down to this: the Linux userspace plumbing layer is nowadays developed to a big part in the systemd source tree. Ignoring this means you constantly have to work with half-baked systems, where you combine outdated components which do not belong to together into something that in many facets might work but is hardly integrated or consistent. Or to put this another way: you are at a road fork: either you take the path where you use the stuff that the folks doing most of the Linux  core OS development work on (regardless if they work at Red Hat, Intel, Suse, Samsung or wherever else) or you use the stuff Canonical is working on (which in case it isn't obvious is well... "limited").
This text, turned into the opposite perspective, excellently summarizes the danger that comes with going with systemd: entering a road which hopefully continues in green and flowery fields, but from which it will be very very difficult to escape, should you want to. The more distributions go with systemd, the more the Linux world will be tied to it. The code of udev was merged into the systemd project last year, and Kay Sievers (the udev maintainer) assured that "the udev built from the systemd source tree will stay compatible with non-systemd init systems for a long time". How long? It seems just logical that the more distributions use systemd, the less effort will be spent to make udev usable outside on non-systemd systems. And, as Lennart wrote, it's not just udev. Already now logind requires systemd, and tomorrow it could be cgroups, kdbus, Wayland and what not. But if it's true that I like systemd, why am I complaining? How could this be a bad thing? We should simply all switch to the solution that currently is the best one, which in the opinion of many is systemd, and forget worrying, fighting and (more concretely) spending time in maintaining other init systems (and related plumbing stuff).

Well, the reason is that we don't know where the currently flowery road will lead us to. Could it lead to some bad place? If you think of a future where nearly everyone will be walking on this road, well, the bad case is really unlikely: there is a solid company behind the project (Red Hat), and possibly the greatest developers that have been worked on Linux. The fact that almost everyone will be using systemd should simply make it stronger and better. Just look at the Linux kernel.
However... we would still be locked on a road. This is what the text I quoted above says, after all: if you get out of the road, you lose logind (with GNOME as of now, and possible other DEs will follow), kdbus, udev, maybe even Wayland. More precisely, you don't lose them, but you'll have to come up with a system which offers all the equivalent required functionality. It's not impossible, but it would demand a great amount of work. And here I guess that even the most patient of my readers will have asked this question for the third time: "but why the hell would I want to jump off this road?".
Because you still can, as of now. You can write a better udev. You can write a better logind. You can write a better init system, a new one, you can even start from scratch, as a hobby project. You can write it in Go. You can innovate, you can come up with some revolutionary idea (as in some ways was systemd when it was conceived). Once all the Linux plumbing layer is tied together, changing stuff will be much more difficult — especially if you are a single individual with great ideas but limited time to implement them. Contributing changes upstream is not always easy, and it becomes more difficult as the complexity of the project grows. Not anyone can afford develop a new feature in a branch, and keep it in there until it's ready to be merged, rebasing it and solving merge conflicts every other day, for months. And it's likely that the more revolutionary your idea is, the deeper and more disruptive your changes will be, to the point that keeping them in a branch would be impractical. This is why I wish that the components which form the software stack remain separate: because you might want (or need) to replace them. You don't like Upstart? Fine, use OpenRC. OpenRC sucks? Fine, use something else. But in the future you won't have this choice with systemd, and probably with any other component of the Linux plumbing layer, once it's become part of a single entity.

I liked it when systemd came out. And I want such a change to be able to happen again.

Etichette: ,

domenica 3 novembre 2013

Mappero: public source code, CLA, Qt5 port

Mappero has always been distributed under a GPL licence. However, since when I started selling Mappero Geotagger (which is built from the same source), I decided not to publish the source code in a public repository, but only to provide it to those who made an explicit request to obtain it.

I spent some time reconsidering the matter, and I've finally decided to let the source code live in a public repository. I also setup a mailing list for it. And indeed I welcome code contributions, however there's a small catch: a CLA. While Mappero is distributed under the GPLv3 licence, I request that all contributors send me an e-mail in which they give me the right to re-licence their contribution under any licence published by the Free Software Foundation.

Since I believe that the busiest time for my involvement with speculo has passed, I expect to be able to spend some more time developing Mappero. The qt5 port is more or less working, but most of the cool features are missing, so it's little more than a map viewer at the moment (Mappero Geotagger, however, is fully working under Qt5!).

Here you can see Mappero running on an Ubuntu Touch powered Nexus 4. Pinch zooming and GPS are not yet working, but I promise they'll be there in less than a week. Also I found a nasty bug which can cause the application to crash when downloading map tiles, and I'll fix it ASAP (I'm mentioning it just so that I won't be flooded with identical bug reports now :-) ).

Etichette: , , , , ,

sabato 2 novembre 2013

speculo, or shared memory made easy

The last few months I've been dedicating most of my (very little) free time to a new project: speculo, a library which implements IPC (Inter-Process Communication) on top of shared memory. Since developers appreciate conciseness and minimalism, here's a description of speculo in a few bullet points:
  • written in C
  • POSIX (tested in Linux and FreeBSD)
  • small (~850 LOC)
  • well commented (~400 lines)
  • good test coverage
  • zerocopy
  • lockless
  • one writer, many readers
  • data is written and read in chunks of arbitrary size
  • a data chunk becomes visible to the readers as soon as the writer commits it
  • data chunks can have an expiration time
  • data chunks can be obsoleted by a newer copy
  • garbage collector
  • no file descriptors are permanently kept open
  • no change notification
But here I probably need to write some paragraphs to explain a couple of points.  Except for a few memory addresses which hold the state of the memory area and which are atomically updated and guarded with memory barriers, all the data written to the shared memory object is immutable. This in particular guarantees that readers have a consistent access to the data, which will not change under their eyes. Data chunks are only appended, which means that the shared memory object can only grow. However, data chunks can be marked as expired (if they have an expiration time associated with them) or obsoleted (if a new chunk is said to replace their contents), which means that not all of the data which is written in the SHM object is actually valid. The readers' functions know this, and skip over the invalid data.
At some point the garbage collector will kick in, when the conditions specified by the writer are met or when the SHM area is completely full. All the chunks which are still valid will be copied over to a new SHM object, and then the SHM objects will be atomically switched. Readers will be able to complete their ongoing reads, and transparently move on to the new SHM object as they request to read a new data chunk.

The above means that speculo can be used in at least two ways:
  • to implement a data stream: the writer fills the shared memory area with short-lived data chunks
  • sharing changeable blocks of data
The first use-case doesn't need much of an explanation; it's usual message passing between a writer and some readers. The second use-case means having a writer expose one or more blocks of data to the readers; if the data gets updated, readers can re-read it and obtain the latest version.

On the minus side, speculo doesn't implement any mechanism of change notification: when the writer writes a new data chunk, or updates an existing one, readers won't be notified of it (though they will get the new data, if they access the shared memory area). This was done intentionally, because the scope of speculo is just to make using shared memory a little easier, and to integrate with existing IPC mechanisms, such as sockets or D-Bus (or even new IPC mechanisms).

The project is in such a state where the declared functionality is working, though I wouldn't swear about its complete reliability (more tests are needed). The current API is not fixed in stone, and on the contrary I'm thinking of a couple of changes to apply. Also, after seeing the implementation of shm_open in Linux and FreeBSD, I'm seriously considering dropping the usage of it in favour of plain open, which would then allow for per-user namespaces. So, I think this is a good time to join the project, to shape it the way it can be useful to the largest number of people.

Last but not least, the link to the project page: https://gitlab.com/mardy/speculo
There is not much documentation in the wiki yet; I recommend checking out the code and running make doc to build the HTML documentation (requires doxygen). To run the tests, use make check (requires pkg-config and check).
A mailing list is up here.

Etichette: , , , ,