Reinstating an iconic error message

Ludovic Courtès — dubna 1, 2023

Software development is a social process. What might be a “bug” for someone might well be a “feature” for someone else. The Guix project rediscovered it the hard way when, after “fixing a bug” that had been present in Guix System for years, it was confronted with an uproar in its user base.

In this post we look at why developers considered the initial behavior a “bug”, why users on the contrary had come to rely on it, and why developers remained blind to it. A patch to reinstate the initial behavior is being reviewed. This post is also an opportunity for us Guix developers to extend our apologies to our users whose workflow was disrupted.

The crux of the matter

Anyone who’s used Guix System in the past has seen this message on the console during the boot process:

error in finalization thread: Success

The following picture shows a typical boot screen (with additional messages in the same vein):

If you have never seen it before, it may look surprising to you. Guix System users lived with it literally for years; the message became a hint that the boot process was, indeed, successful.

A few months ago, a contributor sought to satisfy their curiosity by finding the origin of the message. It did look like a spurious error message, after all, and perhaps the right course of action would be to address the problem at its root—or so they thought.

As it turns out, the message originated in Guile—check out the Guile manual if you’re curious about finalization. Investigation revealed two things: first, that this perror call in Guile was presumably reporting the wrong error code—this was fixed.

The second error—the core of the problem—lied in Guix System itself. Remember that, in its quest of memory safety™, statelessness, and fun, Guix System does it all in Guile Scheme—well, except for the kernel (for now). As soon as Linux has booted, Guix System spawns Guile to run boot code that’s in its initial RAM disk (“initrd”). Right before executing shepherd, its service manager, as PID 1, the initrd code would carelessly close all the file descriptors above 2 to make sure they do not leak into PID 1. The problem—you guessed it—is that one of them was the now-famous file descriptor of the finalization thread’s pipe; the finalization thread would quickly notice and boom!

error in finalization thread: Success

Our intrepid developers thought: “hey, we found it! Let’s fix it!”. And so they did.

Breaking user workflows

This could have been the end of the story, but there’s more to it than software. As Xkcd famously captured, this was bound to break someone’s workflow. Indeed, had developers paid more attention to what users had to say, they would have known that the status quo was preferable.

For some time now, users had shown that they held the error/success message deep in their heart. The message was seen on the blackboard at the Ten Years of Guix celebration, as a motto, as a rallying cry, spontaneously put on display:

What’s more, a fellow NixOS hacker and Guix enthusiast, beguiled by this powerful message, designed stickers and brought them to FOSDEM in February 2023:

The sticker design builds upon the “test pilot” graphics made by Luis Felipe for the 1.3.0 release. The test pilot has a bug on its helmet. In a way, the drawing and error message both represent, metaphorically, a core tenet of Guix as a project; just like Haskell is avoiding success at all costs, Guix seems trapped in an error/success quantum state.

Had it gone too far? Was calling it a “bug” the demonstration of the arrogance of developers detached from the reality of the community?

Fixing our mistakes

Those who installed Guix System starting from version 1.4.0 have been missing out on the error/success boot message. The patch submitted today finally reinstates that message. The review process will determine whether consensus is to enable it by default—as part of %base-service—or whether to make it optional—after all, we also need to accommodate the needs of new users who never saw this message. This will allow users to restore their workflow, while also ensuring that those freshly printed stickers remain relevant.

This incident had broader consequences in the project. It led some to suggest that we, finally, set up a request-for-comment (RFC) kind of process that would give all the community a say on important topics—a process most large free software projects have developed in one form or another. Such a process could have prevented this incident: instead of arrogantly labeling it as a “bug”, developers would have proposed an RFC to remove the message; the discussion period, most likely, would have made it clear that removal was not a desirable outcome and we would all have moved on.

This incident made many users uncomfortable, but we are glad that it is now being addressed. The lessons learned will be beneficial to the project for the years to come.

Credits

Test pilot by Luis Felipe distributed under the terms of CC-BY-SA 4.0; sticker design distributed under CC-BY-SA 4.0 as well. Blackboard picture by Julien Lepiller under CC0; sticker pictures under CC0.

Many thanks to the anonymous sticker provider!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.