The Big Change
Making cross-cutting changes over a large code base is difficult, but it's occasionally necessary if we are to keep the code base tidy and malleable. With almost 800K source lines of code, Guix can reasonably be called a large code base. One might argue that almost 80% of this code is package definitions, which “doesn't count”. Reality is that it does count, not only because those package definitions are regular Scheme code, but also they are directly affected by the big change Guix has just undergone.
This post looks at what’s probably the biggest change Guix has seen since it started nine years ago and that anyone writing packages will immediately notice: simplified package inputs. Yes, we just changed how each of the 20K packages plus those in third-party channels can declare their dependencies. Before describing the change, how we implemented it, and how packagers can adapt, let’s first take a look at the previous situation and earlier improvements that made this big change possible.
Packages and inputs
Packages in Guix are defined using a domain-specific language embedded in the Scheme programming language—an EDSL, for the programming language geeks among us. This is a departure from other designs such as that of Nix, which comes with a dedicated language, and it gives packagers and users access to a rich programming interface while retaining the purely declarative style of package definitions.
This package EDSL is one of the oldest bits of Guix and was described in a 2013 paper. Although embedded in Scheme, the package “language” was designed to be understandable by people who’ve never practiced any kind of Lisp before—you could think of it as a parenthesized syntax for JSON or XML. It’s reasonable to say that it’s been successful at that: of the 600+ people who contributed to Guix over the years, most had never written Scheme or Lisp before. The example given in that paper remains a valid package definition:
(define hello (package (name "hello") (version "2.8") (source (origin (method url-fetch) (uri (string-append "mirror://gnu/hello/hello-" version ".tar.gz")) (sha256 (base32 "0wqd8...")))) (build-system gnu-build-system) (arguments '(#:configure-flags `("--disable-color" ,(string-append "--with-gawk=" (assoc-ref %build-inputs "gawk"))))) (inputs `(("gawk" ,gawk))) (synopsis "GNU Hello") (description "An illustration of GNU's engineering practices.") (home-page "http://www.gnu.org/software/hello/") (license gpl3+)))
Of particular interest here is the
inputs field, which lists
build-time dependencies. Here there’s only one, GNU Awk; it has an
,gawk bit lets us insert the value
gawk variable, which is another package. We can list more
dependencies like so:
(inputs `(("gawk" ,gawk) ("gettext" ,gnu-gettext) ("pkg-config" ,pkg-config)))
Quite a bit of boilerplate. Unless you’re into Lisp, this probably
looks weird to you—manageable, but weird. What’s the deal with this
backquote and those commas? The backquote is shorthand for
and commas are shorthand for
unquote; it’s a facility that Lisp and
Scheme provide to construct
Lispers couldn’t live without quasiquote, it’s wonderful. Still, exposing newcomers to this syntax has always been uncomfortable; in tutorials you’d end up saying “yeah don’t worry, just write it this way”. Our goal though is to empower users by giving them abstractions they can comprehend, hopefully providing a smooth path towards programming without noticing. This seemingly opaque backquote-string-unquote construct works against that goal.
Then, you ask, why did Guix adopt this unpleasant syntax for inputs in
the first place? Input syntax had to satisfy one requirement: that it’d
be possible for “build-side code” to refer to a specific input. And
what’s build-side code? It’s code that appears in the package
definition that is staged for later execution—code that’s only
evaluated when the package is actually built. The bit that follows
#:configure-flags in the example above is build-side code: it’s an
expression evaluated if and when the package gets built. That
#:configure-flags expression refers to the gawk package to construct a
"--with-gawk=/gnu/store/…-gawk-5.0.1"; it does so by
referring to the special
%build-inputs variable, which happens to
contain an association
that maps input labels to file names. The
"gawk" label in
here to allow build-side code to get at an
input’s file name.
Still here? The paragraphs above are a demonstration of the shortcoming of this approach. That we have to explain so much suggests we’re lacking an abstraction that would make the whole pattern clearer.
G-expressions and self-referential records
The missing abstraction came to Guix a couple of years later:
Without going into the details, which were covered elsewhere, notably
in a research article,
g-expressions, or “gexps”, are traditional Lisp
s-expressions (“sexps”) on
steroids. A gexp can contain a package record or any other “file-like
object” and, when that gexp is serialized for eventual execution, the
package is replaced by its
/gnu/store/… file name.
Gexps have been used since 2014–2015 in all of Guix System and they’ve been great to work with, but package definitions were stuck with old-style sexps. One reason is that a big chunk of the code that deals with packages and build systems had to be ported to the gexp-related programming interfaces; a first attempt had been made long ago but performance was not on par with that of the old code, so postponing until that was addressed seemed wiser. The second reason was that using gexps in package definitions could be so convenient that packagers might unwillingly find themselves creating inconsistent packages.
We can now rewrite our
hello package such that configure flags are
expressed using a gexp:
(define hello (package (name "hello") ;; … (arguments (list #:configure-flags #~`("--disable-color" ,(string-append "--with-gawk=" #$gawk)))) (inputs `(("gawk" ,gawk)))))
The reference inserted here with
#$ is synonymous for
ungexp, the gexp equivalent of traditional
unquote) refers to the
gawk variable. This is more concise and semantically clearer
(assoc-ref %build-inputs "gawk") snippet we had before.
Now suppose you define a package variant using this common idiom:
(define hello-variant (package (inherit hello) (name "hello-variant") (inputs `(("gawk" ,gawk-4.0)))))
Here the intent is to create a package that depends on a different
version of GNU Awk—the hypothetical
gawk-4.0 instead of
#:configure-flags gexp of this variant still refers to
gawk variable, contrary to what the
inputs field prescribes; in
effect, this variant depends on the two Awk versions.
To address this discrepancy, we needed a new linguistic device, to put
it in a fancy way. It arrived in
in the form of self-referential records. Within a field such as the
arguments field above, it’s now possible to refer to
get at the value of the package being defined. (If you’ve done
object-oriented programming before, you might be thinking that we just
self pointer, and there’s some truth in
it. :-)) With a bit of syntactic sugar, we can rewrite the example above
so that it refers to the Awk package that appears in its own
(define hello (package (name "hello") ;; … (arguments (list #:configure-flags #~(list (string-append "--with-gawk=" #$(this-package-input "gawk"))))) (inputs `(("gawk" ,gawk)))))
With this in place, we can take advantage of gexps in package definitions while still supporting the common idiom to define package variants, wheee!
That was a long digression from our input label theme but, as you can see, all this interacts fairly tightly.
Getting rid of input labels
Now that we have gexps and self-referential records, it looks like we can finally get rid of input labels: they’re no longer strictly necessary because we can insert packages in gexps instead of looking them up by label in build-side code. We “just” need to devise a backward compatibility plan…
Input labels are pervasive; they’re visible in three contexts:
- in the
inputsfields of package definitions;
- on the build side with the
inputskeyword parameter of build phases;
- in the Scheme programming interface since
package-inputsand related functions are expected to return a list of labeled inputs.
We’re brave but not completely crazy, so we chose to focus on #1 for now—it’s also the most visible of all three—, with an plan to incrementally address #2, leaving #3 for later.
To allow for label-less inputs, we augmented the record interface with
This feature allows us to define a procedure that inspects and
transforms the value specified for the
propagated-inputs. Currently that procedure reintroduces input
when they’re missing. In a sense, we’re just changing the surface
syntax but under the hood everything is unchanged. With this change,
our example above can now be written like this:
(define hello (package (name "hello") ;; … (inputs (list gawk))))
Much nicer, no? The effect is even more pleasant for packages with a number of inputs:
(define-public guile-sqlite3 (package (name "guile-sqlite3") ;; … (build-system gnu-build-system) (native-inputs (list autoconf automake guile-3.0 pkg-config)) (inputs (list guile-3.0 sqlite))))
That’s enough to spark joy to anyone who’s taught the previous syntax. Currently this is transformed into something like:
(define-public guile-sqlite3 (package ;; … (native-inputs (map add-input-label (list autoconf automake guile-3.0 pkg-config)))))
… where the
add-input-label function turns a package into a
label/package pair. It does add a little bit of run-time overhead, but
nothing really measurable.
There are also cases where package definitions, in particular for
package variants, would directly manipulate input lists as returned by
package-inputs and related procedures. It’s a case where packagers
had to be aware of input labels, and they would typically use
association list (or “alist”) manipulation
and similar construct—this is context #3 above. To replace those
idioms, we defined a higher-level construct that does not assume input
labels. For example, a common idiom when defining a package variant
with additional dependencies goes like this:
(define hello-with-additional-dependencies (package (inherit hello) (name "hello-with-bells-and-whistles") (inputs `(("guile" ,guile-3.0) ("libtextstyle" ,libtextstyle) ,@(package-inputs hello)))))
The variant defined above adds two inputs to those of
introduced a macro,
which allows packagers to express that in a higher-level (and less
cryptic) fashion, in a way that does not refer to input labels. Using
this other linguistic device (ha ha!), the snippet above becomes:
(define hello-with-additional-dependencies (package (inherit hello) (name "hello-with-bells-and-whistles") (inputs (modify-inputs (package-inputs hello) (prepend guile-3.0 libtextstyle)))))
modify-inputs advantageously subsumes
whatnot when willing to replace or remove inputs, like so:
(modify-inputs (package-inputs hello) (replace "gawk" my-special-gawk) (delete "guile"))
On the build side—context #2 above—, we also provide new procedures that
allow packagers to avoid relying on input labels:
Instead of having build phases that run code like:
(lambda* (#:key inputs #:allow-other-keys) ;; Replace references to “/sbin/ip” by references to ;; the actual “ip” command in /gnu/store. (substitute* "client/scripts/linux" (("/sbin/ip") ;; Look up the input labeled “iproute”. (string-append (assoc-ref inputs "iproute") "/sbin/ip"))))
… you’d now write:
(lambda* (#:key inputs #:allow-other-keys) ;; Replace references to “/sbin/ip” by references to ;; the actual “ip” command in /gnu/store. (substitute* "client/scripts/linux" (("/sbin/ip") ;; Search “/sbin/ip” among all the inputs. (search-input-file inputs "/sbin/ip"))))
Nothing revolutionary but a couple of advantages: code is no longer tied
to input labels or package names, and
search-input-file raises an
exception when the file is not found, which is better than building an
incorrect file name.
That was a deep dive into packaging! If you’re already packaging software for Guix, you hopefully see how to do things “the new way”. Either way, it’s interesting to see the wide-ranging implications of what initially looks like a simple change. Things get complex when you have to consider all the idioms that 20,000 packages make use of.
Adapting to the new style
It’s nice to have a plan to change the style of package definitions, but how do you make it happen concretely? The last thing we want is, for the next five years, to have to explain two package styles to newcomers instead of one.
now returns packages in the new style. But what about existing package
Fortunately, most of the changes above can be automated: that’s the job
command that we added for this purpose, but which may eventually be
extended to make package definitions prettier in all sorts of ways.
From a checkout, one can run:
./pre-inst-env guix style
Whenever it can be done safely, package inputs in every package
definition are rewritten to the new style: removing input labels, and
modify-inputs where appropriate. If you maintain your own
channel, you can also run it for your packages:
guix style -L /path/to/channel my-package1 my-package2 …
We recommend waiting until the next Guix release is out though, which
could be a month from now, so that your channel remains usable by those
who pinned an older revision of the
guix style a couple of days ago on the whole repository,
leading to the biggest
in Guix history:
460 files changed, 37699 insertions(+), 49782 deletions(-)
Woohoo! Less than 15% of the packages (not counting Rust packages, which are a bit special) have yet to be adjusted. In most cases, package inputs that were left unchanged are either those where we cannot automatically determine whether the change would break the package, for instance because build-side code expects certain labels, and those that are “complex”—e.g., inputs that include conditionals.
The key challenge here was making sure
guix style transformations are
correct; by default, we even want to be sure that changes introduced by
guix style do not trigger a rebuild—that package
To achieve that,
guix style correlates the source code of each
package definition with the corresponding live package record. That
allows it to answer questions such as “is this label the name of the
input package”. That way, it can tell that, say:
(inputs `(("libx11" ,libx11) ("libxcb" ,libxcb)))
can be rewritten without introducing a rebuild, because labels match actual package names, whereas something like this cannot, due to label mismatches:
(inputs `(("libX11" ,libx11) ("xcb" ,libxcb)))
guix style can also determine situations where changes would trigger a
rebuild but would still be “safe”—without any observable effect. You
can force it to make such changes by running:
guix style --input-simplification=safe
Because we’re all nitpicky when it comes to code formatting,
guix style had to produce nicely formatted code, and to make local changes
as opposed to rewriting complete package definitions. Lisps are famous
for being homoiconic,
which comes in handy in such a situation.
But the tools at our disposal are not capable enough for this
application. First, Scheme’s standard
procedure, which reads an sexp (or an abstract syntax tree if you will)
from a byte stream and returns it, does not preserve comments.
Obviously we’d rather not throw away comments, so we came up with our
read variant that preserves comments. Similarly, we have a custom
pretty printer that can write comments, allowing it to achieve changes
@@ -2749,18 +2707,17 @@ (define-public debops "debops-debops-defaults-fall-back-to-less.patch")))) (build-system python-build-system) (native-inputs - `(("git" ,git))) + (list git)) (inputs - `(("ansible" ,ansible) - ("encfs" ,encfs) - ("fuse" ,fuse) - ("util-linux" ,util-linux) ;for umount - ("findutils" ,findutils) - ("gnupg" ,gnupg) - ("which" ,which))) + (list ansible + encfs + fuse + util-linux ;for umount + findutils + gnupg + which)) (propagated-inputs - `(("python-future" ,python-future) - ("python-distro" ,python-distro))) + (list python-future python-distro)) (arguments `(#:tests? #f
The pretty printer also has special rules for input lists. For
instance, lists of five inputs or less go into a single line, if
possible, whereas longer lists end up with one input per line, which is
often more convenient, especially when visualizing diffs. It also has
rules to format
modify-inputs in the same way we’d do it in Emacs:
@@ -171,9 +170,9 @@ (define-public arcan-sdl (inherit arcan) (name "arcan-sdl") (inputs - `(("sdl" ,sdl) - ,@(fold alist-delete (package-inputs arcan) - '("libdrm")))) + (modify-inputs (package-inputs arcan) + (delete "libdrm") + (prepend sdl))) (arguments
Overall that makes
guix style a pretty fun meta-project!
“Worse is better” or “the Right Thing”?
There are several lessons here. One is that having an embedded
domain-specific language is what makes such changes possible: yes
package definitions have a declarative field, but we do not hide the
fact that their meaning is determined by the broader Guix framework,
starting with the
(guix packages) module, which defines the package
record type and associated procedures. Having a single repository
containing both package definitions and “the package manager” is also
a key enabler; we can change the framework, add new linguistic tools,
and adjust package definitions at the same time. This is in contrast,
for instance, with the approach taken by Nix, where the language
implementation lives separately from the package collection.
Another one is that a strong, consistent community leads to consistent changes—not surprisingly in fact. It’s a pleasure to see that we, collectively, can undertake such overarching changes and all look in the same direction.
The last lesson is in how we approach design issues in a project that is now a little more than nine years old. Over these nine years it’s clear that we have usually favored “the right thing” in our design—but not at any cost. This whole change is an illustration of this. It was clear from the early days of Guix that input labels and the associated machinery were suboptimal. But it took us a few years to design an implement the missing pieces: G-expressions, self-referential records, and the various idioms that allow package definitions to benefit from these. In the meantime, we built a complete system and a community and we gained experience. We cannot claim we attained the right thing, if such a thing exists, but certainly package definitions today are closer to the declarative ideal and easier to teach.
It’s here today!
This big change, along with countless other improvements and package
upgrades, is just one
guix pull away! After months of development, we
have just merged the “core updates” branch bringing so many new
things—from GNOME 41, to GCC 10 by default, to hardened Python packages
and improved executable startup
This paves the way for the upcoming release, most likely labeled “1.4”,
unless a closer review of the changes that have landed leads us to think
“2.0” would be more appropriate… Stay tuned!
About GNU Guix
GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.
Related topics:Scheme API