Guile 3 & Guix

Ludovic Courtès — 24 січня 2020 року

Version 3.0 of GNU Guile, an implementation of the Scheme programming language, was released just last week. This is a major milestone for Guile, which gets compiler improvements and just-in-time (JIT) native code generation, leading to significant performance improvements over 2.2. It’s also great news for all the users of Guile, and in particular for Guix!

This post discusses what it means for Guix to migrate to Guile 3 and how that migration is already taking place.

Guile in Guix

Most users interact with Guix through its command-line interface, and we work hard to make it as approachable as possible. As any user quickly notices, Guix uses the Scheme programming language uniformly for its configuration—from channels to manifests and operating systems—and anyone who starts packaging software knows that package definitions are in fact Scheme code as well.

This is a significant departure from many other, and in particular from Nix. While Nix defines several domain-specific languages (DSLs) for these aspects—the Nix language but also specific configuration languages—Guix chooses Scheme as the single language for all this, together with the definition of high-level embedded domain-specific languages (EDSLs).

It goes beyond that: in Guix System, all the things traditionally implemented in C or as a set of Perl or shell scripts are implemented in Scheme. That includes the init system, package builds, the initial RAM disk (initrd), system tests, and more. Because this leads to several layers of Scheme code, executed at different points in time, Guix includes a code staging mechanism built upon the nice properties of Scheme.

Why do that? The arguments, right from the start, were twofold: using a general-purpose language allows us to benefit from its implementation tooling, and having interfaces for “everything” in Scheme makes it easy for users to navigate their distro or OS code and to reuse code to build new features or applications. Guix developers benefit from the ease of code reuse every day; demonstrative examples include the use of Guix container facilities in the init system, the development of many tools providing facilities around packages, the implementation of additional user interfaces, and work on applications that use Guix as a library such as the Guix Workflow Language and Guix-Jupyter.

As for the benefits of the host general-purpose language, these are rather obvious: Guix developers benefit from an expressive language, an optimizing compiler, a debugger, a powerful read-eval-print loop (REPL), an interactive development environment, and all sorts of libraries. Moving to Guile 3 should add to that better performance, essentially for free. To be comprehensive, Guile 3 may well come with a set of brand new bugs too, but so far we seem to be doing OK!

Migrating to Guile 3

What does it mean for Guix to migrate to Guile 3? We’ve seen above different ways in which Guix relies on Guile. In short, we can say that migration is threefold:

1. Guix is a distro that ships Guile-related packages. Like any other distro, it will have to upgrade its guile package to 3.0 and to ensure packages that depend on it and updated as well.
2. Guix is a program written in Guile. As such, we need to make sure that all its dependencies (half a dozen of Guile libraries) work with Guile 3 and that Guix itself runs fine with Guile 3.
3. Guix ties together operating system components. In particular, the init system (the Shepherd) and other boot-time facilities will also migrate.

The packages

Updating the distro is the boring part, but it’s best to get it right. Guix makes it possible to have unrelated versions of variants of packages in different environments or different profiles, which is very nice. We’ll have performed a smooth transition if users and tools see that the packages named guile and guile-ssh (say) transparently move from Guile 2.2 to 3.0, in lockstep.

Put differently, most of the upgrade work upon a programming language version bump deals with conventions, and in particular package names. Currently, guile corresponds to the 2.2 stable series and all the guile-* packages are built against it. In the meantime, the package for Guile 3 is named guile-next and packages built against it are called guile3.0-*. Over the last few weeks we created guile3.0- variants for most Guile packages, something that’s easily achieved with Guix.

The big switch will consist in renaming all current guile-* packages to guile2.2-* packages, for use with the legacy 2.2 series, and renaming all the guile3.0-* packages to guile-*. We will switch soon, but before getting there, we’re making sure important packages are available for 3.0.

Guix-the-program

A more interesting part is “porting” Guix itself from Guile 2.2 to Guile 3. It seems that developers have become wary of 2-to-3 transitions for programming languages. Fear not! Switching from Guile 2 to Guile 3 turned out to be an easy task. In fact, very little changed in the language itself; what did change—e.g., semantics on fine points of the module system, support for structured exceptions—is either optional or backwards-compatible.

As Guile 2.9 pre-releases trickled in, we started testing all the Guile libraries Guix relies on against 2.9. For the vast majority of them, all we had to do was to update their configure.ac to allow builds with 3.0.

Guix itself was a bit more work, mostly because it’s a rather large code base with a picky test suite. The bit that required most work has to do with the introduction of declarative modules, an optional semantic change in modules to support more compiler optimizations. We had several “white-box tests” where tests would happily peek at private module bindings through the magical-evil @@ operator. Because we chose to enable declarative modules, we also had to adjust our tests to no longer do that. And well, that’s about it!

At that point, we were able to create a guile3.0-guix package variant, primarily for testing purposes. Soon after, we told guix pull to build Guix with 3.0 instead of 2.2. Thus, Guix users who upgrade will transparently find themselves running Guix on Guile 3.0.

The main benefit is improved performance. Guile 3 is known to be up to 32 times faster than Guile 2.2 on some micro-benchmarks. Assessing the performance gains on a “real-world” application like Guix is the real test. What would be a relevant benchmark? At its core, Guix is essentially a compiler from high-level descriptions of packages, operating systems, and the like, to low-level build instructions (derivations). Thus, a good benchmark is a command that exercises little more than this compilation step:

guix build libreoffice ghc-pandoc guix --dry-run --derivation

or:

guix system build config.scm --dry-run --derivation

On x86_64, the guix build command above on Guile 3 is 7% faster than on Guile 2.2, and guix system build, which is more computing-intensive, is 10% faster (heap usage is ~5% higher). This is lower than the skyrocketing speedups observed on some microbenchmarks, but it’s probably no surprise: these guix commands are short-lived (a couple of seconds) and they’re rather I/O- and GC-intensive—something JIT compilation cannot help with.

On 32-bit ARM, we temporarily disabled JIT due to a bug; there we observe a slight slowdown compared to 2.2. This can be explained by the fact that virtual machine (VM) instructions in 3.0 are lower-level than in 2.2 and will hopefully be more than compensated for when JIT is re-enabled.

Gluing it all together

The last part of the Guile 3 migration has to do with how Guix, and in particular Guix System, glues things together. As explained above, Guix manipulates several stages of Scheme code that will run a different points in time.

Firstly, the code that runs package builds, such as the one that runs ./configure && make && make install, is Guile code. Currently that code runs on Guile 2.2, but on the next major rebuild-the-world upgrade, we will switch to Guile 3.

Additionally, Guix produces Scheme code consumed by the Shepherd, by GNU mcron, and for the graphical installer. These will soon switch to Guile 3 as well. This kind of change is made easy by the fact that both the package definitions and the staged code that depends on those packages live in the same repository.

Long live Guile 3!

Migrating Guix to Guile 3 is a bit of work because of the many ways Guix interacts with Guile and because of the sheer size of the code base. For a “2-to-3” transition though, it was easy. And fundamentally, it remains a cheap transition compared to what it brings: better performance and new features. That’s another benefit of using a general-purpose language.

Thumbs up to everyone involved in its development, and long live Guile 3!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.