https://guix.gnu.org/feeds/blog/performance.atomGNU Guix — Blog — Performancefeed author nameGNU Guixhttps://guix.gnu.org/themes/initial/img/icon.png2024-03-20T10:57:59Zhttps://guix.gnu.org/blog/2021/from-guix-environment-to-guix-shell//From ‘guix environment’ to ‘guix shell’Ludovic Courtès2021-10-26T15:00:00Z2021-10-26T15:00:00Z There are times when what looked like the right design choice some years
back comes out as an odd choice as time passes. The beloved guix environment
tool is having that fate. Its command-line interface has become
non-intuitive and annoying for the most common use cases. Since it
could not be changed without breaking compatibility in fundamental ways,
we devised a new command meant to progressively replace it; guix shell —that’s
the name we unimaginatively ended up with—has just landed after a
three-week review period , itself a
followup to discussions and hesitations on the…<p>There are times when what looked like the right design choice some years
back comes out as an odd choice as time passes. The beloved <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-environment.html"><code>guix environment</code></a>
tool is having that fate. Its command-line interface has become
non-intuitive and annoying for the most common use cases. Since it
could not be changed without breaking compatibility in fundamental ways,
we devised a new command meant to progressively replace it; <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-shell.html"><code>guix shell</code></a>—that’s
the name we unimaginatively ended up with—has just landed after <a href="https://issues.guix.gnu.org/50960">a
three-week review period</a>, itself a
followup to discussions and hesitations on the best course of action.</p><p>This post introduces <code>guix shell</code>, how it differs from <code>guix environment</code>, the choices we made, and why we hope you will like it.</p><h1>The story of <code>guix environment</code></h1><p>The <code>guix environment</code> command <a href="https://lists.gnu.org/archive/html/guix-devel/2014-10/msg00111.html">started its life in
2014</a>,
when Guix was a two-year old baby and the whole community could fit in a
small room. It had one purpose: “to assist hackers in creating
reproducible development environments”. It was meant to be similar in
spirit to <a href="https://docs.python.org/3/tutorial/venv.html">VirtualEnv</a> or
<a href="https://bundler.io/">Bundler</a>, but <em>universal</em>—not limited to a single
language. You would run:</p><pre><code>guix environment inkscape</code></pre><p>… and obtain an interactive shell with all the packages needed to hack
on Inkscape; in that shell, the relevant environment variables—<code>PATH</code>,
<code>CPATH</code>, <code>PKG_CONFIG_PATH</code>, and so on—would automatically point to a
profile created on the fly and containing the compiler, libraries, and
tools Inkscape depends on, but not Inkscape itself.</p><p>Only a year later did it become clear that there are cases where one
would want to create an environment containing specific packages, rather
than an environment containing <em>the dependencies</em> of packages. To
address that, David Thompson <a href="https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00561.html">proposed the <code>--ad-hoc</code>
option</a>:</p><pre><code>guix environment --ad-hoc inkscape -- inkscape</code></pre><p>… would create an environment containing only Inkscape, and would then
launch the <code>inkscape</code> command in that environment. Many features were
added over the years, such as <a href="https://guix.gnu.org/en/blog/2015/container-provisioning-with-guix/">the invaluable <code>--container</code>
option</a>,
but these two modes, development and “ad hoc”, are the guts of it.</p><p>Fast forward six years: today, there’s consensus that the name
<code>--ad-hoc</code> is confusing for newcomers and above all, that the “ad hoc”
mode should be the default. This is the main problem that <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-environment.html"><code>guix shell</code></a>
addresses.</p><h1>Doing what you’d expect</h1><p>Changing the default mode from “development environment” to “ad hoc” is
technically easy, but how to do that without breaking compatibility is
harder. This led to <a href="https://issues.guix.gnu.org/38529">lengthy
discussions</a>, including proposals of
mechanisms to choose between the new and old semantics.</p><p>In the end, keeping the <code>guix environment</code> name while allowing it to
have different semantics was deemed dangerous. For one thing, there’s
lots of material out there that demoes <code>guix environment</code>—blog posts,
magazine articles, on-line courses—and it would have been impossible to
determine whether they refer to the “new” or to the “old” semantics. We
<a href="https://issues.guix.gnu.org/38529#17">reached the conclusion</a> that it
would be easier to use a new command name and to eventually deprecate
<code>guix environment</code>.</p><p>With <code>guix shell</code>, the default is to create an environment that contains
the packages that appear on the command line; to launch Inkscape, run:</p><pre><code>guix shell inkscape -- inkscape</code></pre><p>The <code>--ad-hoc</code> option is gone! Likewise, to spawn an ephemeral
development environment containing Python and a couple of libraries,
run:</p><pre><code>guix shell python python-numpy python-scipy -- python3</code></pre><p>Now, if you want, say, the development environment of Inkscape, add the
<code>--development</code> or <code>-D</code> option right before:</p><pre><code>guix shell -D inkscape</code></pre><p>You can add Git and GDB on top of it like so:</p><pre><code>guix shell -D inkscape git gdb</code></pre><p>(Note that <code>-D</code> only applies to the immediately following package,
<code>inkscape</code> in this case.) It’s more concise and more natural than with
<code>guix environment</code>. As can be seen <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-shell.html">in the
manual</a>,
all the other options supported by <code>guix environment</code> remain available
in <code>guix shell</code>.</p><h1>Short-hands for development environments</h1><p>A convention that’s become quite common is for developers to provide a
<code>guix.scm</code> at the top of their project source tree, so that others can
start a development environment right away:</p><pre><code>guix environment -l guix.scm</code></pre><p>The <code>guix.scm</code> file would contain a <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">package
definition</a>
for the project at hand, <a href="https://notabug.org/cwebber/guile-gcrypt/src/master/guix.scm">as in this
example</a>.
This option is known as <code>-f</code> in <code>guix shell</code>, for consistency with other
commands, and the equivalent command is:</p><pre><code>guix shell -D -f guix.scm</code></pre><p>Since all Guix commands accept a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html#index-manifest">“manifest”</a>
with <code>-m</code>, another option is to provide a <code>manifest.scm</code> file and to
run:</p><pre><code>guix shell -m manifest.scm</code></pre><p>“Wouldn’t it be nice if <code>guix shell</code> would automatically follow these
conventions when not given any argument?”, some
<a href="https://lists.gnu.org/archive/html/guix-devel/2017-08/msg00300.html">suggested</a>.
As in the case of Bundler, <a href="https://direnv.net/">direnv</a>, or typical
build tools from Meson to Make, having a default file name can save
typing and contribute to a good user experience for frequently-used
commands. In this spirit, <code>guix shell</code> automatically loads <code>guix.scm</code>
or <code>manifest.scm</code>, from the current directory or an ancestor thereof,
such that entering a project to hack on it is as simple as:</p><pre><code>cd ~/my/project/src
guix shell</code></pre><p>Worry not: <code>guix shell</code> loads <code>guix.scm</code> or <code>manifest.scm</code> <em>if and only
if</em> you have first added its directory to
<code>~/.config/guix/shell-authorized-directories</code>. Otherwise <code>guix shell</code>
warns you and prints a hint that you can copy/paste if you want to
authorize the directory.</p><h1>Caching environments</h1><p>With that in place, <code>guix shell</code> can pretty much fill the same role as
direnv and similar tools, with one difference though: speed. When all
the packages are already in store, <code>guix shell</code> can take one to a few
seconds to run, depending on the package set, on whether you’re using a
solid state device (SSD) or a “spinning” hard disk, and so on. It’s
acceptable but prohibitively slow for direnv-like use cases.</p><p>To address that, <code>guix shell</code> maintains a profile cache for the <code>-D -f guix.scm</code> and <code>-m manifest.scm</code> cases. On a hot cache, it runs in
0.1 second. All it has to do is fork a shell with the right environment
variable definitions; it does not talk to <code>guix-daemon</code>, and it does not
even read <code>guix.scm</code> or <code>manifest.scm</code> (it’s possible to forcefully
update the cache with <code>--rebuild-cache</code>).</p><p>That makes <code>guix shell</code> usable even for short-lived commands like
<code>make</code>:</p><pre><code>guix shell -- make</code></pre><p>Hopefully it’ll change the way we use the tool!</p><h1>The shell doctor</h1><p>While revamping this command-line interface, the idea of a “shell
doctor” came up. In interactive use, <code>guix shell</code> sets environment
variables and spawns a shell, but it’s not uncommon for the shell to
mess up with the whole environment. Why? Because, <a href="https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html">contrary to
documented
practice</a>,
it’s quite common for users to define or override environment variables
in the startup files of non-login shells, <code>~/.bashrc</code> for Bash,
<code>~/.zshrc</code> for Zsh. Instead, environment variable definitions should go
to the startup file of <em>login</em> shells—<code>~/.bash_profile</code>, <code>~/.profile</code>,
or similar. But let’s face it: it’s a subtle distinction that few of us
know or care about.</p><p>As a result, users of Guix, especially on distros other than Guix
System, would often be disappointed when running <code>guix environment --pure</code> and <em>yet</em> find that <code>PATH</code> contains non-Guix entries, that
there’s a bogus <code>LD_LIBRARY_PATH</code> definition, and whatnot. Now, they
can call the doctor, so to speak, to obtain a diagnosis of the health of
their shell by adding the <code>--check</code> flag:</p><pre><code>guix shell --check python python-numpy</code></pre><p>The command creates an environment containing Python and NumPy, spawns
an interactive shell, checks the environment variables as seen by the
shell, and prints a warning if <code>PATH</code> or <code>PYTHONPATH</code> in this case have
been overridden. It does not tell users where the problem comes from—it
cannot guess—but it tells them if something’s wrong, which is a first
step.</p><p>Of course, the best way to sidestep these problems is to pass
<code>--container</code>, which gives a fresh, isolated environment that does not
contain those startup files. That’s not always an option though, for
instance on systems lacking support for unprivileged user namespaces, so
<code>--check</code> comes in handy there.</p><h1>Try it!</h1><p>Just run <code>guix pull</code> to get this shiny new <code>guix shell</code> thingie!</p><p>If you don’t feel ready yet, that’s OK: <code>guix environment</code> won’t
disappear overnight. We have a <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-environment.html">written
commitment</a>
to keep it around until May, 1st 2023. Though overall, we hope you’ll
find the <code>guix shell</code> interface easier to use and compelling enough that
you’ll be willing to switch overnight!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2021/taming-the-stat-storm-with-a-loader-cache//Taming the ‘stat’ storm with a loader cacheLudovic Courtès2021-08-02T15:00:00Z2021-08-02T15:00:00Z It was one of these days where some of us on IRC were rehashing that old
problem—that application startup in Guix causes a
“ stat storm”—and lamenting the
lack of a solution when suddenly, Ricardo
proposes what,
in hindsight, looks like an obvious solution: “maybe we could use a
per-application ld cache?”. A moment where collective thinking exceeds
the sum of our individual thoughts. The result is one of the many
features that made it in the core-updates branch, slated to be merged
in the coming weeks, one that reduces application startup time. ELF files and their…<p>It was one of these days where some of us on IRC were rehashing that old
problem—that application startup in Guix causes a
“<a href="https://linux.die.net/man/2/stat"><code>stat</code></a> storm”—and lamenting the
lack of a solution when suddenly, Ricardo
<a href="https://logs.guix.gnu.org/guix/2020-11-24.log#183934">proposes</a> what,
in hindsight, looks like an obvious solution: “maybe we could use a
per-application ld cache?”. A moment where collective thinking exceeds
the sum of our individual thoughts. The result is one of the many
features that made it in the <code>core-updates</code> branch, slated to be merged
in the coming weeks, one that reduces application startup time.</p><h1>ELF files and their dependencies</h1><p>Before going into detail, let’s look at what those “<code>stat</code> storms” look
like and where they come from. Loading an
<a href="https://en.wikipedia.org/wiki/Executable_and_Linkable_Format">ELF</a>
executable involves loading the shared libraries (the <code>.so</code> files, for
“shared objects”) it depends on, recursively. This is the job of the
<em>loader</em> (or <em>dynamic linker</em>), <code>ld.so</code>, which is part of the GNU C
Library (glibc) package. What shared libraries an executable like that
of Emacs depends on? The <code>ldd</code> command answers that question:</p><pre><code>$ ldd $(type -P .emacs-27.2-real)
linux-vdso.so.1 (0x00007fff565bb000)
libtiff.so.5 => /gnu/store/l1wwr5c34593gqxvp34qbwdkaf7xhdbd-libtiff-4.2.0/lib/libtiff.so.5 (0x00007fd5aa2b1000)
libjpeg.so.62 => /gnu/store/5khkwz9g6vza1n4z8xlmdrwhazz7m8wp-libjpeg-turbo-2.0.5/lib/libjpeg.so.62 (0x00007fd5aa219000)
libpng16.so.16 => /gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/libpng16.so.16 (0x00007fd5aa1e4000)
libz.so.1 => /gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libz.so.1 (0x00007fd5aa1c2000)
libgif.so.7 => /gnu/store/bpw826hypzlnl4gr6d0v8m63dd0k8waw-giflib-5.2.1/lib/libgif.so.7 (0x00007fd5aa1b8000)
libXpm.so.4 => /gnu/store/jgdsl6whyimkz4hxsp2vrl77338kpl0i-libxpm-3.5.13/lib/libXpm.so.4 (0x00007fd5aa1a4000)
[…]
$ ldd $(type -P .emacs-27.2-real) | wc -l
89</code></pre><p>(If you’re wondering why we’re looking at <code>.emacs-27.2-real</code> rather than
<code>emacs-27.2</code>, it’s because in Guix the latter is a tiny shell wrapper
around the former.)</p><p>To load a graphical program like Emacs, the loader needs to load more
than 80 shared libraries! Each is in its own <code>/gnu/store</code> sub-directory
in Guix, one directory per package.</p><p>But how does <code>ld.so</code> know where to find these libraries in the first
place? In Guix, during the link phase that produces an ELF file
(executable or shared library), we tell the
<a href="https://en.wikipedia.org/wiki/Linker_%28computing%29">linker</a> to
populate the <code>RUNPATH</code> entry of the ELF file with the list of
directories where its dependencies may be found. This is done by
passing
<a href="https://sourceware.org/binutils/docs/ld/Options.html#index-_002drpath_003ddir"><code>-rpath</code></a>
options to the linker, which Guix’s <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ld-wrapper.in">“linker
wrapper”</a>
takes care of. The <code>RUNPATH</code> is the <em>run-time library search path</em>:
it’s a colon-separated list of directories where <code>ld.so</code> will look for
shared libraries when it loads an ELF file. We can look at the
<code>RUNPATH</code> of our Emacs executable like this:</p><pre><code>$ objdump -x $(type -P .emacs-27.2-real) | grep RUNPATH
RUNPATH /gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib:/gnu/store/l1wwr5c34593gqxvp34qbwdkaf7xhdbd-libtiff-4.2.0/lib:/gnu/store/5khkwz9g6vza1n4z8xlmdrwhazz7m8wp-libjpeg-turbo-2.0.5/lib:[…]</code></pre><p>This <code>RUNPATH</code> has 39 entries, which roughly corresponds to the number
of direct dependencies of the executable—dependencies are listed as
<code>NEEDED</code> entries in the ELF file:</p><pre><code>$ objdump -x $(type -P .emacs-27.2-real) | grep NEED | head
NEEDED libtiff.so.5
NEEDED libjpeg.so.62
NEEDED libpng16.so.16
NEEDED libz.so.1
NEEDED libgif.so.7
NEEDED libXpm.so.4
NEEDED libgtk-3.so.0
NEEDED libgdk-3.so.0
NEEDED libpangocairo-1.0.so.0
NEEDED libpango-1.0.so.0
$ objdump -x $(type -P .emacs-27.2-real) | grep NEED | wc -l
52</code></pre><p>(Some of these <code>.so</code> files live in the same directory, which is why
there are more <code>NEEDED</code> entries than directories in the <code>RUNPATH</code>.)</p><p>A system such as Debian that follows the <a href="https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard">file system hierarchy
standard</a>
(FHS), where all libraries are in <code>/lib</code> or <code>/usr/lib</code>, does not have to
bother with <code>RUNPATH</code>: all <code>.so</code> files are known to be found in one of
these two “standard” locations. Anyway, let’s get back to our initial
topic: the “<code>stat</code> storm”.</p><h1>Walking search paths</h1><p>As you can guess, when we run Emacs, the loader first needs to locate
and load the 80+ shared libraries it depends on. That’s where things
get pretty inefficient: the loader will search each <code>.so</code> file Emacs
depends on in one of the 39 directories listed in its <code>RUNPATH</code>.
Likewise, when it finally finds <code>libgtk-3.so</code>, it’ll look for its
dependencies in each of the directories in its <code>RUNPATH</code>. We can see
that at play by tracing system calls with the
<a href="https://strace.io/"><code>strace</code></a> command:</p><pre><code>$ strace -c emacs --version
GNU Emacs 27.2
Copyright (C) 2021 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
55.46 0.006629 3 1851 1742 openat
16.06 0.001919 4 422 mmap
11.46 0.001370 2 501 477 stat
4.79 0.000573 4 122 mprotect
3.84 0.000459 4 111 read
2.45 0.000293 2 109 fstat
2.34 0.000280 2 111 close
[…]
------ ----------- ----------- --------- --------- ----------------
100.00 0.011952 3 3325 2227 total</code></pre><p>For this simple <code>emacs --version</code> command, the loader and <code>emacs</code> probed
for more than 2,200 files, with the
<a href="https://linux.die.net/man/2/openat"><code>openat</code></a> and
<a href="https://linux.die.net/man/2/stat"><code>stat</code></a> system calls, and most of
these probes were unsuccessful (counted as “errors” here, meaning that
the call returned an error). The fraction of “erroneous” system calls
is no less than 67% (2,227 over 3,325). We can see the desperate search
of <code>.so</code> files by looking at individual calls:</p><pre><code>$ strace -e openat,stat emacs --version
[…]
openat(AT_FDCWD, "/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/l1wwr5c34593gqxvp34qbwdkaf7xhdbd-libtiff-4.2.0/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/5khkwz9g6vza1n4z8xlmdrwhazz7m8wp-libjpeg-turbo-2.0.5/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/haswell/x86_64/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/haswell/x86_64", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/haswell/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/haswell", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/x86_64/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/x86_64", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/tls", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/haswell/x86_64/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/haswell/x86_64", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/haswell/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/haswell", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/x86_64/libpng16.so.16", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/x86_64", 0x7ffe428a1c70) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/gnu/store/3x2kak8abb6z2klch72kfff2qxzv00pj-libpng-1.6.37/lib/libpng16.so.16", O_RDONLY|O_CLOEXEC) = 3
[…]</code></pre><p>Above is the sequence where we see <code>ld.so</code> look for <code>libpng16.so.16</code>,
searching in locations where we <em>know</em> it’s not going to find it. A bit
ridiculous. How does this affect performance? The impact is small in
the most favorable case—on a hot cache, with fast solid state device
(SSD) storage. But it likely has a visible effect in other cases—on a
cold cache, with a slower spinning hard disk drive (HDD), on a network
file system (NFS).</p><h1>Enter the per-package loader cache</h1><p>The idea that Ricardo submitted, using a loader cache, makes a lot of
sense: we know from the start that <code>libpng.so</code> may only be found in
<code>/gnu/store/…-libpng-1.6.37</code>, no need to look elsewhere. In fact, it’s
not new: glibc has had such a cache “forever”; it’s the
<code>/etc/ld.so.cache</code> file you can see on FHS distros and which is
typically created by running
<a href="https://linux.die.net/man/8/ldconfig"><code>ldconfig</code></a> when a package has
been installed. Roughly, the cache maps library <code>SONAME</code>s, such as
<code>libpng16.so.16</code>, to their file name on disk, say
<code>/usr/lib/libpng16.so.16</code>.</p><p>The problem is that this cache is inherently system-wide: it assumes
that there is only <em>one</em> <code>libpng16.so</code> on the system; any binary that
depends on <code>libpng16.so</code> will load it from its one and only location.
This models perfectly matches the FHS, but it’s at odds with the
flexibility offered by Guix, where several variants or versions of the
library can coexist on the system, used by different applications.
That’s the reason why Guix and other non-FHS distros such as NixOS or
GoboLinux typically <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/base.scm?id=a92dfbce30777de6ca05031e275410cf9f56c84c#n716">turn
off</a>
that feature altogether… and pay the cost of those <code>stat</code> storms.</p><p>The insight we gained on that Tuesday evening IRC conversation is that
we could <em>adapt</em> glibc’s loader cache to our setting: instead of a
system-wide cache, we’d have a <em>per-application loader cache</em>. As one
of the last package <a href="https://guix.gnu.org/manual/en/html_node/Build-Phases.html">build
phases</a>,
we’d run <code>ldconfig</code> to create <code>etc/ld.so.cache</code> within that package’s
<code>/gnu/store</code> sub-directory. We then need to modify the loader so it
would look for <code>${ORIGIN}/../etc/ld.so.cache</code> instead of
<code>/etc/ld.so.cache</code>, where <code>${ORIGIN}</code> is the location of the ELF file
being loaded. A discussion of these changes is <a href="https://issues.guix.gnu.org/44899">in the issue
tracker</a>; you can see <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/patches/glibc-dl-cache.patch?id=0236013cd0fc86ff4a042885c735e3f36a7f5c25">the glibc
patch</a>
and the new <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gnu-build-system.scm?id=0236013cd0fc86ff4a042885c735e3f36a7f5c25#n735"><code>make-dynamic-linker-cache</code> build
phase</a>.
In short, the <code>make-dynamic-linker-cache</code> phase computes the set of
direct and indirect dependencies of an ELF file using the
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gremlin.scm?id=0236013cd0fc86ff4a042885c735e3f36a7f5c25#n265"><code>file-needed/recursive</code></a>
procedure and derives from that the library search path, creates a
temporary <code>ld.so.conf</code> file containing this search path for use by
<code>ldconfig</code>, and finally runs <code>ldconfig</code> to actually build the cache.</p><p>How does this play out in practice? Let’s try an <code>emacs</code> build that
uses this new loader cache:</p><pre><code>$ strace -c /gnu/store/ijgcbf790z4x2mkjx2ha893hhmqrj29j-emacs-27.2/bin/emacs --version
GNU Emacs 27.2
Copyright (C) 2021 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
28.68 0.002909 26 110 13 openat
25.13 0.002549 26 96 read
20.41 0.002070 4 418 mmap
9.34 0.000947 10 90 pread64
6.60 0.000669 5 123 mprotect
4.12 0.000418 3 107 1 newfstatat
2.19 0.000222 2 99 close
[…]
------ ----------- ----------- --------- --------- ----------------
100.00 0.010144 8 1128 24 total</code></pre><p>Compared to what we have above, the total number of system calls has
been divided by 3, and the fraction of erroneous system calls goes from
67% to 0.2%. Quite a difference! We count on you, dear users, to <a href="https://guix.gnu.org/en/contact">let
us know</a> how this impacts load time for
you.</p><h1>Flexibility without <code>stat</code> storms</h1><p>With <a href="https://www.gnu.org/software/stow">GNU Stow</a> in the 1990s, and
then Nix, Guix, and other distros, the benefits of flexible file layouts
rather than the rigid Unix-inherited FHS have been demonstrated—nowadays
I see it as an antidote to opaque and bloated application bundles à la
Docker. Luckily, few of our system tools have FHS assumptions baked in,
probably in large part thanks to GNU’s insistence on a <a href="https://www.gnu.org/prep/standards/html_node/Directory-Variables.html">rigorous
installation directory
categorization</a>
in the early days rather than hard-coded directory names. The loader
cache is one of the few exceptions. Adapting it to a non-FHS context is
fruitful for Guix and for the other distros and packaging tools in a
similar situation; perhaps it could become an option in glibc proper?</p><p>This is not the end of <code>stat</code> storms, though. Interpreters and language
run-time systems rely on search paths—<code>GUILE_LOAD_PATH</code> for Guile,
<code>PYTHONPATH</code> for Python, <code>OCAMLPATH</code> for OCaml, etc.—and are equally
prone to stormy application startups. Unlike ELF, they do not have a
mechanism akin to <code>RUNPATH</code>, let alone a run-time search path cache. We
have yet to find ways to address these.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2021/getting-bytes-to-disk-more-quickly//Getting bytes to disk more quicklyLudovic Courtès2021-03-26T14:00:00Z2021-03-26T14:00:00Z Let’s face it: functional package managers like Guix provide unequaled
support for reproducibility and transactional upgrades, but the price to
pay is that users often spend a fair amount of time downloading (or
building) packages. Download times are okay on day-to-day use but
they’re a pain point when performing large upgrades or when installing
Guix System for the first time. With that in mind, it should come as no surprise that Michael
Stapelberg’s excellent 2020 Arch Linux Conference
talk
and the installation speed achieved by distri were a great motivation
boost. Michael proposes radical ideas to speed up package installation,
such as downloading and…<p>Let’s face it: functional package managers like Guix provide unequaled
support for reproducibility and transactional upgrades, but the price to
pay is that users often spend a fair amount of time downloading (or
building) packages. Download times are okay on day-to-day use but
they’re a pain point when performing large upgrades or when installing
Guix System for the first time.</p><p>With that in mind, it should come as no surprise that Michael
Stapelberg’s <a href="https://media.ccc.de/v/arch-conf-online-2020-6387-distri-researching-fast-linux-package-management">excellent 2020 Arch Linux Conference
talk</a>
and the installation speed achieved by distri were a great motivation
boost. Michael proposes radical ideas to speed up package installation,
such as downloading and mounting ready-to-use SquashFS images. Not
everything can be transposed to Guix as-is, but it certainly got us
thinking.</p><p>This article dives into improvements made over the last few months that
will be in the upcoming 1.2.1 release, and which are already <a href="https://guix.gnu.org/manual/en/html_node/Upgrading-Guix.html">one <code>guix pull</code>
away</a>;
they all contribute to making <a href="https://guix.gnu.org/manual/en/html_node/Substitutes.html">substitute
download</a> and
installation “faster”. This is an evolution of the existing mechanisms
rather than a revolution, but one that users will surely welcome.</p><h1>Reducing latency</h1><p>One of the first things we notice is latency between subsequent
substitute downloads. It may sound ridiculous, but <code>guix-daemon</code> would
spawn one helper process (running the internal <code>guix substitute</code>
command) for each substitute download. That means that not only would
it create a new process each time, it would also be unable to reuse
connections to the substitute servers. This is particularly noticeable
and wasteful when downloading many substitutes in a row, such as when
installing Guix System for the first time.</p><p>The latency in between subsequent substitute downloads was primarily
this: <code>fork</code> (creating a new process), TCP connection establishment and
TLS connection handshake. When running <code>perf timechart record guix build …</code> (what a wonderful tool <a href="https://perf.wiki.kernel.org/"><code>perf timechart</code></a> is!), we get this Gantt
diagram, which gives an idea of all the time wasted creating these
processes, waiting for them to initialize and connect to the substitute
server, and so on:</p><p><img src="https://guix.gnu.org/static/blog/img/substitute-chart-one-process-per-substitute.png" alt="Gantt diagram without substitute agent." /></p><p>Why was it done this way? Because the daemon, written in C++ and
inherited from Nix, would historically delegate substitution to helper
programs, with a flexible but naive protocol between the daemon and
those helper programs.</p><p>Back in December, we <a href="https://issues.guix.gnu.org/45018">tackled this
issue</a>
(<a href="https://issues.guix.gnu.org/45253">followup</a>): the daemon would now
launch a single <code>guix substitute</code> process and reuse it for subsequent
substitute downloads. In turn, <code>guix substitute</code> would cache
connections so we save time on connection establishment.</p><p>Another observation we made is that <code>guix-daemon</code> implemented
post-processing steps after each download that would contribute to
latency. Once <code>guix substitute</code> had completed a download and extracted
the archive to the store, the daemon would traverse this store item a
couple of times to reset file timestamps/permissions (“<em>metadata
canonicalization</em>”), to deduplicate files, and to verify the integrity
of the whole store item. This is I/O-intensive and particularly
wasteful for store items with many files. Our next step was to delegate
all this work to <code>guix substitute</code>, which allows it to <a href="https://issues.guix.gnu.org/45253">pipeline archive
extraction, integrity checks, deduplication, and metadata
canonicalization</a>. All this happens
as the bytes flow in and no extra traversal is needed.</p><p>The speedup offered by these optimizations depends on several factors,
including the latency of your network, the speed of your CPU and that of
your hard disk, and it grows with the number of substitutes fetched in a
row. What we can say is that even in a “favorable”
situation—low-latency network, fast CPU, fast storage device—it
definitely feels snappier.</p><h1>Increasing bandwidth</h1><p>With the latency issue pretty much solved, the next step was to look at
bandwidth. When we <a href="https://guix.gnu.org/en/blog/2019/substitutes-are-now-available-as-lzip/">introduced lzip-compressed
substitutes</a>
back in 2019, the assumption was that a much higher compression ratio
(compared to gzip), would inevitably translate to faster downloads.
That is true… but only for some bandwidth/CPU power configurations.</p><p>Specifically, it turns out that, for someone with a fast connection,
such as fiber-to-the-home (FTTH), <a href="https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html">downloads of lzip-compressed
substitutes are actually
CPU-bound</a>.
In other words, the limiting factor is the processing time to decompress
those lzip’d archives—not the available bandwidth. Lzip currently
achieves the best compression ratios so far, and that’s great, but
decompression is compute-intensive.</p><p>Ever-increasing bandwidth is what drove the design and implementation of
newer compression methods, such as “Z standard”, colloquially known as
<a href="https://facebook.github.io/zstd/">“zstd”</a>. The decompression speed of
zstd is on the order of 3.5 higher than that of gzip and an order of
magnitude higher than that of lzip; but unlike the venerable
<a href="http://www.oberhumer.com/opensource/lzo">lzo</a>, zstd achieves high
compression ratios: at level 19, the compression ratio of zstd is lower
than that of lzip but in the same ballpark. Guillaume Le Vaillant
<a href="https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00097.html">provided</a>
an insightful comparison of gzip, lzip, and zstd on a plot showing their
decompression speed as a function of the compression ratio:</p><p><img src="https://guix.gnu.org/static/blog/img/decompression-speed-plot.png" alt="Decompression speed vs. compression ratio of gzip, lzip, and zstd." /></p><p>There are several takeaways. First, zstd decompression is always faster
than the alternatives. Second, zstd compresses better than gzip
starting from level 2, so gzip “loses” on both criteria. Last, zstd at
level 19 achieves a compression ratio comparable to lzip level 5 or
6—lzip level 9 remains better in that regard.</p><p>With brand new <a href="https://notabug.org/guile-zstd/guile-zstd">Guile bindings to
zstd</a>, we were able to add
zstd support to <a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-publish.html"><code>guix publish</code></a>
and <code>guix substitute</code>. But what policy should be adopted for the
official substitute server at <code>ci.guix.gnu.org</code>?</p><p>Our goal is to maximize substitute download speed. In some
configurations, for example on relatively slow connections, fetching
lzip substitutes is not CPU-bound; in those cases, fetching lzip
substitutes remains the fastest option. And of course, there are these
other configurations where lzip decompression is the limiting factor,
and where we’d rather fetch a slightly less compressed zstd substitute
(or even a much less compressed gzip substitutes, sometimes).</p><p>The solution we came up with is a naive but efficient solution:
<a href="https://issues.guix.gnu.org/47137">client-side adaptive compression
selection</a>. When it fetches and
decompresses a substitute, <code>guix substitute</code> monitors its CPU usage as
reported by
<a href="https://www.gnu.org/software/libc/manual/html_node/Processor-Time.html#index-times"><code>times</code></a>.
If the user time to wall-clock time ratio is close to one, that probably
means the substitute fetch and decompression process was CPU-bound; in
that case, choose a compression method that yields faster decompression
next time—assuming the substitute server offers several options.
Conversely, if CPU usage is close to zero, the process is probably
network-bound, so next time we’ll choose a better-compressed option.</p><p>The official server at <code>ci.guix.gnu.org</code> now provides zstd-compressed
substitutes in addition to lzip and gzip. An <a href="https://lists.gnu.org/archive/html/guix-devel/2021-03/msg00333.html">extra
complication</a>
is that we cannot drop gzip compression just yet because pre-1.1.0 Guix
installations understand nothing but gzip. To ease transition, the
server will probably offer substitutes in all three compression formats
for one more year or so.</p><p>There’s a good reason to upgrade your daemon though: you’ll be able to
benefit from those zstd substitutes and if you have high bandwidth,
you’ll quickly see the difference!</p><h1>Grabbing substitutes from your neighbors</h1><p>When it comes to increasing download speeds, another option is to
download from your neighbors rather than from a server far away. This
is particularly useful at the workplace, where machines around you are
likely to already have the store items you’re looking for, or, say, at
Guix gatherings—at least when we can meet physically again…</p><p>The idea had been <a href="https://guix.gnu.org/en/blog/2017/reproducible-builds-a-status-update/">floating
around</a>
for some time as a direct benefit of <a href="https://reproducible-builds.org/">reproducible
builds</a> and co-maintainer Mathieu
Othacehe <a href="https://othacehe.org/substitute-server-discovery.html">recently implemented
it</a>. As Mathieu
explains in his blog post, <code>guix publish</code> can now advertise itself on
the local network using the mDNS/DNS-SD protocol <em>via</em>
<a href="https://avahi.org">Avahi</a>; when <code>guix-daemon</code> is passed the
<code>--discover</code> option, <code>guix substitute</code> automatically discovers local
substitute servers and adds them as the preferred download location.
The Guix System installation image even allows you to enable it to speed
up the installation process:</p><p><img src="https://guix.gnu.org/static/blog/img/installer-substitute-discovery.png" alt="Enabling substitute server discovery when installing Guix System." /></p><p>Again, that only speeds things up if substitute servers use a
compression method with fast decompression, and with either a cache or
fast compression if they compress things on the fly. Zstd comes in
handy for that: Guillaume’s measurements show that <a href="https://lists.gnu.org/archive/html/guix-devel/2021-01/pnglYLfp1DXNC.png">zstd compression is
inexpensive at low compression levels, while achieving higher
compression ratios than
gzip</a>.</p><h1>Going further</h1><p>Put together, these changes have the potential to noticeably improve
user experience. But as I wrote in the introduction, it’s not
revolutionary either—users still have to download all these things.</p><p>The next big step might come from fetching substitutes over
peer-to-peer, content-addressed networks such as
<a href="https://issues.guix.gnu.org/33899">IPFS</a> or
<a href="https://issues.guix.gnu.org/46800#3">GNUnet</a>. Their content-addressed
nature could allow users to download less. The performance
characteristics of these networks is less clear though.</p><p>There is still value in a plain HTTP-based substitute protocol like the
one currently used that is easy to set up, though. In that spirit, an
option would be to “upgrade” the existing substitute protocol to take
advantage of content-addressability. After all, the daemon already
performs file-level deduplication and there’s <a href="https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00258.html">a fair amount of
identical files between subsequent builds of the same
package</a>.
So… <a href="https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00080.html">what if we only downloaded those files not already available
locally?</a>
This idea is appealing but we need to go beyond the prototyping phase to
get a better idea of its viability.</p><p>For completeness, another option currently investigated by the Nix
developers is that of “<em>content-addressed derivations</em>”. While
currently store file names contain a hash of the <em>inputs</em> used to
produce them, the idea of content-addressed derivations is to make it a
content hash; that way, if an input change has no effect on the build
result, the output is the same and nothing needs to be re-downloaded
(this is what Eelco Dolstra described as the <em>intensional model</em> in
Chapter 6 of his <a href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">seminal PhD
thesis</a>). This option is
appealing, also for other reasons, but it’s a fundamental change with
what looks like a <a href="https://github.com/tweag/rfcs/blob/cas-rfc/rfcs/0062-content-addressed-paths.md">high implementation complexity and transition
cost</a>.
We have yet to gauge the pros and cons of following this approach.</p><p>Until then, we hope Guix 1.2.1 will bring your faster substitutes and
happiness!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/grafts-continued//Grafts, continuedLudovic Courtès2020-05-06T15:00:00Z2020-05-06T15:00:00Z Guix includes a mechanism called grafts that allows us to provide
users with security
updates
in a timely fashion, even for core packages deep down in the dependency
graph. Most users value the benefits of grafts, but newcomers were also
unavoidably struck by what turned out to be the undesirable side effect
of our graft implementation on user experience. This had been a
well-known problem for a while, but 1.1.0 finally addressed these
issues . This article recaps how grafts are implemented, what problems that
caused, and how we solved it. It’s a deep dive into core Guix,…<p>Guix includes a mechanism called <em>grafts</em> that allows us to provide
users with <a href="https://guix.gnu.org/manual/en/html_node/Security-Updates.html">security
updates</a>
in a timely fashion, even for core packages deep down in the dependency
graph. Most users value the benefits of grafts, but newcomers were also
unavoidably struck by what turned out to be the undesirable side effect
of our graft implementation on user experience. This had been a
well-known problem for a while, but <a href="https://guix.gnu.org/blog/2020/gnu-guix-1.1.0-released/">1.1.0 finally addressed these
issues</a>.</p><p>This article recaps how grafts are implemented, what problems that
caused, and how we solved it. It’s a deep dive into core Guix, and I
hope it’ll be insightful to all and intriguing to the functional
programming geeks among us!</p><h1>What’s this “graft” thing anyway?</h1><p>Grafts were introduced in the early days of Guix to <a href="https://guix.gnu.org/blog/2016/timely-delivery-of-security-updates/">address probably
the main practical shortcomings of functional software
deployment</a>.
In a nutshell, functional deployment as implemented by Nix and Guix
means that, when a package changes, everything that depends on it must
be rebuilt (or re-downloaded). To deploy a security fix in the C
library or in Bash, you would thus need to rebuild everything. Even
with a huge build farm, that can significantly delay the deployment of
fixes; users would find themselves either rebuilding things locally or,
at best, re-downloading binaries of everything.</p><p>To address this, Guix developers can instead specify a <em>replacement</em> in
a <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definition</a>.
If we have a bug-fix for, say, libc, developers would (1) define a
package for the fixed libc, and (2) add a <code>replacement</code> field in the
original libc package pointing to that fixed package. The effect is
that <em>only the bug-fix libc needs to be built</em>. When building a
package, the bug-fix libc is automatically <em>grafted onto that package</em>,
such that the resulting package refers to the bug-fix libc. <a href="https://guix.gnu.org/manual/en/html_node/Security-Updates.html">See the
manual</a>
for more.</p><p>When “lowering” a high-level <a href="https://guix.gnu.org/manual/en/html_node/Defining-Packages.html">package
definition</a>
to a low-level
<a href="https://guix.gnu.org/manual/en/html_node/Derivations.html"><em>derivation</em></a>,
Guix traverses the package dependency graph and identifies a set of
potentially applicable grafts. Why “potentially applicable”? Consider
this scenario: assume <code>perl</code> has a <code>replacement</code>; <code>coreutils</code> has a
dependency on <code>perl</code>, but it’s a build-time dependency: <code>coreutils</code> does
not depend on <code>perl</code> at run time. Thus, <code>coreutils</code> can be used as is,
there is no need to graft it.</p><p>But how do we know whether a dependency is a built-time-only dependency?
The <a href="https://guix.gnu.org/manual/en/html_node/package-Reference.html#index-package"><code>native-inputs</code>
field</a>
of a package usually lists build-time dependencies, but it’s more of a
hint. Ultimately, the set of run-time dependencies, which we call the
<em>references</em>, is the subset of the build-time dependencies that the
garbage collector (GC) in the build daemon finds <em>in the build
result</em>—Section 5.5.1 of <a href="http://nixos.org/~eelco/pubs/phd-thesis.pdf">Eelco Dolstra’s PhD
thesis</a> describes how the
GC
scans for references. In our example, we first have to actually build
<code>coreutils</code> before we can tell whether it depends on <code>perl</code> at
run time.</p><p>Guix arranges to graft only when necessary. In this example, <code>guix build coreutils</code> would return the same as <code>guix build coreutils --no-grafts</code>. Conversely, since <code>inkscape</code> has a run-time dependency on
<code>perl</code>, <code>guix build inkscape</code> returns a derivation that grafts the
<code>perl</code> replacement onto the original <code>inkscape</code> build result, the one
returned by <code>guix build inkscape --no-grafts</code>. The (simplified)
dependency graph of the derivation for the grafted <code>inkscape</code> looks like
this:</p><p><img src="https://guix.gnu.org/static/blog/img/inkscape-graft.svg" alt="Dependency graph of the graft derivation of Inkscape." /></p><p>Grafts are a form of what <a href="https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf"><em>Build Systems à la
Carte</em></a>
by Mokhov <em>et al.</em> (a good read!) refers to as <em>dynamic dependencies</em>:
grafting depends on intermediate <em>build results</em>.</p><p>Still here? With the background in place, let’s look at the problems
that arose.</p><h1>Grafts, the user interface, and performance</h1><p>Conceptually, to decide whether to graft a package, we examine the
references of the build result of the ungrafted package. However, we
usually want <code>guix install</code> & co. to first display an overall build
plan, especially when invoked with <code>--dry-run</code>:</p><pre><code>$ guix install inkscape
The following package will be installed:
inkscape 1.0
71.3 MB will be downloaded:
/gnu/store/xq64iaxx2gmlcgnipj31wjxlf1yd2g2p-gsl-2.6
/gnu/store/zrmhnj3pwchn2msphgnwzwd3q89m46rn-aspell-0.60.8
/gnu/store/5g31zf21lk8nrfd2b8qrm19nwdm9gir9-potrace-1.16
/gnu/store/qpr7387bsjs7rpl6flrwdvn2zbzh5bch-ghostscript-with-cups-9.52
/gnu/store/7y3lvk3xf4im8n44337mc6y0ccysvfia-font-dejavu-2.37
/gnu/store/95n3zvzxzq2bxvdvz292cg04757ij30x-cups-minimal-2.3.1
…</code></pre><p>To accommodate that, the pre-1.1.0 implementation of grafts did the
following: when
<a href="https://guix.gnu.org/manual/en/html_node/Substitutes.html">substitutes</a>
were enabled, it would get the list of references of ungrafted packages
from substitutes; only when substitutes for an ungrafted package are
missing would it first try to build that package. Thus, when
substitutes are available, <code>guix install</code> and similar commands would be
able to display the build plan upfront. However, when a packages had no
substitutes, you’d see Guix start building it without saying a word
about the build plan, which was <a href="https://issues.guix.gnu.org/issue/28310">arguably
confusing</a>.</p><p>But it’s worse than that. Grafting is per-package, so every time you
would lower a package to a derivation, you would need to answer the
question “does <em>this</em> specific package have substitutes, and if so,
should it be grafted?” The end result was <a href="https://issues.guix.gnu.org/issue/22990">poor resource usage and
terrible user interface
feedback</a>. For every package
that is a graft candidate, the user would see that infamous line:</p><pre><code>updating substitutes from 'https://ci.guix.gnu.org'...</code></pre><p>The problem was particularly acute when building whole systems with
<code>guix system</code> because there would typically be a large number of such
packages. Furthermore, each of these lines would correspond to
(roughly) a single HTTP GET request on a fresh TLS connection. That can
be slow… and annoying. Perhaps to some users this <code>updating substitutes</code> stuttering was the proof of the developers’ incompetence
and perhaps, truth be told, to some of us developers it was a small
price to pay for the sophistication of grafts.</p><p>For users who disable substitutes and build everything locally, the
situation wasn’t much better: all the packages candidate for grafting
would be built one by one, thereby missing parallelization opportunities
as specified by
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html"><code>--max-jobs</code></a>.</p><h1>Gathering dynamic dependencies</h1><p>To address this, all these individual dynamic dependencies need to be
gathered somehow instead of being treated one by one. Conceptually, we
would like to, roughly, do a first pass lowering packages to derivations
as if grafting was disabled, build all these derivations, and then do a
second pass to determine which packages in the graph need to be grafted and
to compute the relevant grafting derivation. That would address the
performance issue: we’d now have as much parallelism as possible, so we
wouldn’t query substitutes or build packages one by one. If we reify
that second pass to the user interface code, it also addresses the user
interface issue by allowing it to display, possibly, two build plans:
the “ungrafted” one followed by the grafted one.</p><p>The problem is that our API is inherently serial: the
<code>package-derivation</code> function takes <em>one</em> package, lowers it, and
returns its derivation:</p><pre><code class="language-scheme">(use-modules (guix)
(gnu packages base)
(gnu packages inkscape))
(define s (open-connection))
(package-derivation s coreutils)
⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv => /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug /gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
(package-derivation s coreutils #:graft? #f)
⇒ #<derivation /gnu/store/rpfdbax1py483m9ydhvp73s7dgmn6xh4-coreutils-8.31.drv => /gnu/store/jkj7wxybgcpdamkl6fz7wwbb1ak5wxvk-coreutils-8.31-debug /gnu/store/zibwkb5xavnv6z3gzknfqjsxb9b0izh0-coreutils-8.31 7f6c92e3a000>
(package-derivation s inkscape)
⇒ #<derivation /gnu/store/jzm2zsq8m0rj8wdsmikc0p2ik0cprrcf-inkscape-0.92.4.drv => /gnu/store/clj8rjnsip8a35hyd9nf4l65w7ahn0gs-inkscape-0.92.4 7f6c9c15b730>
(package-derivation s inkscape #:graft? #f)
⇒ #<derivation /gnu/store/psd31x1fq0v2g594z217mh56xzg21dym-inkscape-0.92.4.drv => /gnu/store/zz28ckjwfxwkx3gsm8sc452kmvfiky6y-inkscape-0.92.4 7f6c90ad4f50></code></pre><p>Lowering includes dealing with grafts, and
that’s why we ended up with one-by-one inefficiencies. An option would
be to make all the API “plural”: have <code>package-derivation</code> and its
friends accept a <em>list</em> of packages instead of a single one. That would
be a huge amount of work and the end result would be unpleasant to use:
it’s easier to reason one-by-one.</p><p>The solution implemented in 1.1.0 instead starts from this observation:
the call graph of <code>package-derivation</code> mirrors the package graph. Thus,
we could gather dynamic dependencies using <a href="https://guix.gnu.org/manual/en/html_node/The-Store-Monad.html">monad
trickery</a>
or using “control effects”. We went for the latter, which didn’t have
the “contamination” problem of monads and led to simpler code.</p><p>The starting point is that, by definition, code with dynamic
dependencies necessarily calls
<a href="https://guix.gnu.org/manual/en/html_node/The-Store.html#index-build_002dderivations"><code>build-derivations</code></a>.
Taking advantage of <a href="https://www.gnu.org/software/guile/manual/html_node/Prompts.html">delimited continuations in
Guile</a>,
<code>build-derivations</code> is instrumented to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=041b340da409078951267b6a8c43b27716e6b7ec">abort to a “build handler”
prompt</a>
when it’s called. The build handler receives the list of derivations to
build along with a continuation to invoke to resume the aborted
computation and start building things. User interface code can install
a build handler that displays what’s going to be built:</p><pre><code class="language-scheme">(with-build-handler (lambda (continue store things mode)
(show-what-to-build store things)
(continue #t))
…)</code></pre><p>To implement dry runs, simply omit the call to <code>continue</code> and nothing
will be built. (This is a slightly simplified artist view, see
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=07ce23e011d18460e7ff5553d4ff640f7073075b"><code>build-notifier</code></a>
for the real thing.)</p><p>Now, we need to take advantage of this mechanism to gather the
individual <code>build-derivations</code> calls so we can later emit a single
<code>build-derivations</code> call for all the gathered derivations. The goal is
to effectively gather all the calls for ungrafted packages, build them
all at once, and then resume graft computation.</p><p>To achieve that, we write a build handler that, when invoked, returns an
<code><unresolved></code> object that captures what to build and the continuation.
In addition, we provide a primitive to <em>introduce parallelism</em> such
that, if a dynamic dependency is encountered, we keep going and attempt
to compute as much as possible without resolving that dynamic
dependency. These are <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c40bf5816cb3ffb59920a61f71bd34b53cac3637"><code>build-accumulator</code> and
<code>map/accumulate-builds</code></a>.
<code>map/accumulate-builds</code> is like <code>map</code>, except that it accumulates and
gathers <code>build-derivations</code> request.</p><p>By using <code>map/accumulate-builds</code> instead of <code>map</code> in a few
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=584dfdac3795541ff020aca3f488ceaf2ddd7fc3">key</a>
<a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=25af35fa32bf6c991510406a330d4a42bd5beba8">places</a>,
we obtain a good approximation of what we wanted, as illustrated in this
run:</p><pre><code>$ guix install zile-on-guile vim-full
The following packages will be installed:
zile-on-guile 2.4.14-0.fd09781
vim-full 8.2.0411
9.4 MB will be downloaded:
/gnu/store/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781
/gnu/store/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411
downloading from https://ci.guix.gnu.org/nar/lzip/dnj9wljcck9vdwgp7dwxk00qnnk1g3c5-vim-full-8.2.0411...
vim-full-8.2.0411 8.9MiB 7.6MiB/s 00:01 [##################] 100.0%
downloading from https://ci.guix.gnu.org/nar/lzip/vf7w4yiax38ra7x8aqqvbnc38c0ldgpm-zile-on-guile-2.4.14-0.fd09781...
zile-on-guile-2.4.14-0.fd09781 140KiB 1.8MiB/s 00:00 [##################] 100.0%
The following derivation will be built:
/gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv
The following graft will be made:
/gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv
applying 8 grafts for /gnu/store/4n6dmg6iwjg0adpcvqygr9wgsnclswss-vim-full-8.2.0411.drv...
building /gnu/store/d9xms78917w67xq71pqsx5x9s6dmq6d7-profile.drv...</code></pre><p>What we see above is first a build plan that downloads binaries for the
two ungrafted packages, followed by a build plan for one grafting
derivations: we have successfully preserved parallelism.</p><p>The solution resembles the <code>suspending</code> scheduler discussed in the <em>à
la Carte</em> paper, though decomposition is not as principled as what the
paper describes. It remains an approximation and not the
optimal way to deal with dynamic dependencies. There are still
situations <a href="https://issues.guix.gnu.org/issue/40612">where that shows</a>,
but overall, it’s a significant improvement. Unlike <a href="https://issues.guix.gnu.org/issue/22990#7">other solutions
prototyped before</a>, this one
has the advantage of being orthogonal and simple: less than <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=c40bf5816cb3ffb59920a61f71bd34b53cac3637">100 new
lines of
code</a>,
and even <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=4b75a7060058bc2e959dcb4145067f6bba3e34e5">about 30 lines
removed</a>
from the graft implementation. That alone contributes a lot to the
author’s satisfaction. :-)</p><h1>Interlude: a scatter/gather pattern?</h1><p>In the end, we’re just gathering all the <code>build-derivations</code> calls,
turning them into a single call, and finally calling all the original
site continuations with the result. The same kind of issue shows up
when dealing with sequences of remote procedure calls (RPCs) and HTTP
requests, and it seems there’s a more general pattern lurking here.
Consider code like this:</p><pre><code class="language-scheme">(map (lambda (thing)
(http-get (thing->url thing)))
lst)</code></pre><p>Wouldn’t it be nice if we could somehow capture all the <code>http-get</code>
calls, turn them into a series of <a href="https://en.wikipedia.org/wiki/HTTP_pipelining">pipelined GET
requests</a>, and resume the
continuations with their result?</p><p>I haven’t found a standard functional pattern to address this and would
welcome ideas!</p><h1>Dynamic dependencies of all shapes</h1><p>We have seen how Guix deals with dynamic dependencies. Nix supports a
similar but limited form of dynamic dependencies through
the <code>import</code> primitive of the
Nix language, <a href="https://github.com/NixOS/nix/blob/master/src/libexpr/primops.cc#L74">which can take the result of a derivation
build</a>;
it does not attempt to gather the resulting <code>buildPaths</code> calls.</p><p>If we take a step back, we can see that Nix and Guix actually support
other forms of dynamic dependencies. For example, it’s possible to
write derivations whose result is a function of the reference graph of
another derivation’s build result. This is achieved in Guix by passing
the <a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html#index-gexp_002d_003ederivation"><code>#:references-graphs</code> argument of
<code>gexp->derivation</code></a>,
which leads the build daemon to <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/nix/libstore/build.cc?id=298fb2907e3f432cea7dee9f58e89ab8d9dbd56f#n1763">include the reference graph in the
build
environment</a>.</p><p>Another form of dynamic dependency is <em>derivation-building derivations</em>
or <em>recursive derivations</em>, which were <a href="https://github.com/NixOS/nix/pull/3205">recently implemented in
Nix</a>. It supports another form
of dynamic dependency where the build process of a derivation can itself
create and build derivations (these are <a href="https://en.wikipedia.org/wiki/Parallel_task_scheduling_problem"><em>moldable
tasks</em></a>
in scheduling parlance). It’s a great feature because in a nutshell, it
allows Nix to be used not only to compose packages, but also at a finer
grain as part of a package build process.</p><p>Guix supports yet another form of dynamic dependencies. The newfangled
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-deploy.html"><code>guix deploy</code>
tool</a>
works by <a href="https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial/">evaluating g-expressions (gexps)
remotely</a>.
For example, before actually deploying an operating system, it first
runs code on the remote node to perform sanity checks: checking whether
the declared file system UUIDs or labels are valid, checking whether
additional kernel modules should be added to the initial RAM disk, and
so forth. To do that,
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/remote.scm#n109"><code>remote-eval</code></a>
first builds a derivation that produces a Scheme program, deploys it
along with all its dependencies on that target machine, runs it, and
retrieves the result. This form of dynamic dependency also benefits
from the gathering machinery discussed above.</p><h1>Conclusion</h1><p>This is a long article on what may look like a fine point of Guix design
and implementation, but there’s much to say about it! Grafts are key to
the use of functional deployment in production because they enable quick
security updates, and it’s a lot better if they don’t harm the user
experience.</p><p>The pre-1.1.0 implementation of grafts had a negative impact on the user
interface and on performance, which was due to the sequential handling
of grafts, one package at a time. In 1.1.0 we addressed it by using
delimited continuations to gather dynamic dependencies such as grafts,
perform builds in bulk, and resume each derivation computation.</p><p>As it turned out, the implementation of dynamic dependencies raises lots
of interesting design and implementation issues, and it’s probably not
the end of the road!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/guile-3-and-guix//Guile 3 & GuixLudovic Courtès2020-01-24T15:00:00Z2020-01-24T15:00:00Z Version 3.0 of GNU Guile, an implementation of the Scheme programming
language , was released just last
week .
This is a major milestone for Guile, which gets compiler improvements
and just-in-time (JIT) native code generation, leading to significant
performance improvements over 2.2. It’s also great news for all the
users of Guile, and in particular for Guix! This post discusses what it means for Guix to migrate to Guile 3 and
how that migration is already taking place. Guile in Guix Most users interact with Guix through its command-line interface, and we
work hard to make it…<p>Version 3.0 of GNU Guile, an implementation of the <a href="https://schemers.org">Scheme programming
language</a>, <a href="https://www.gnu.org/software/guile/news/gnu-guile-300-released.html">was released just last
week</a>.
This is a major milestone for Guile, which gets compiler improvements
and just-in-time (JIT) native code generation, leading to significant
performance improvements over 2.2. It’s also great news for all the
users of Guile, and in particular for Guix!</p><p><img src="https://guix.gnu.org/static/blog/img/guile-3.png" alt="Guile 3 logo." /></p><p>This post discusses what it means for Guix to migrate to Guile 3 and
how that migration is already taking place.</p><h1>Guile in Guix</h1><p>Most users interact with Guix through its command-line interface, and we
work hard to make it as approachable as possible. As any user quickly
notices, Guix uses the Scheme programming language uniformly for its
configuration—from
<a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html">channels</a>
to
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html#profile_002dmanifest">manifests</a>
and <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">operating
systems</a>—and
anyone who starts packaging software knows that <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">package
definitions</a>
are in fact Scheme code as well.</p><p>This is a significant departure from many other, and in particular from
<a href="https://nixos.org/nix/">Nix</a>. While Nix defines several
domain-specific languages (DSLs) for these aspects—the <a href="https://nixos.org/nix/manual/#chap-writing-nix-expressions">Nix
language</a>
but also specific
<a href="https://nixos.org/nix/manual/#sec-conf-file">configuration</a>
<a href="https://nixos.org/nix/manual/#chap-distributed-builds">languages</a>—Guix
chooses Scheme as the single language for all this, <a href="https://hal.inria.fr/hal-00824004/en">together with the
definition of high-level embedded domain-specific languages
(EDSLs)</a>.</p><p>It goes beyond that: in Guix System, all the things traditionally
implemented in C or as a set of Perl or shell scripts are implemented in
Scheme. That includes <a href="https://www.gnu.org/software/shepherd/">the init
system</a>, <a href="https://guix.gnu.org/manual/en/html_node/Build-Systems.html">package
builds</a>,
<a href="https://guix.gnu.org/manual/en/html_node/Initial-RAM-Disk.html">the initial RAM disk
(initrd)</a>,
<a href="https://guix.gnu.org/blog/2016/guixsd-system-tests/">system tests</a>, and
more. Because this leads to several layers of Scheme code, executed at
different points in time, Guix includes a <a href="https://hal.inria.fr/hal-01580582/en"><em>code staging</em>
mechanism</a> built upon the nice
properties of Scheme.</p><p>Why do that? The arguments, right from the start, were twofold: using a
general-purpose language allows us to benefit from its implementation
tooling, and having interfaces for “everything” in Scheme makes it easy
for users to navigate their distro or OS code and to reuse code to build
new features or applications. Guix developers benefit from the ease of
code reuse every day; demonstrative examples include the <a href="https://guix.gnu.org/blog/2017/running-system-services-in-containers/">use of Guix
container facilities in the init
system</a>,
the development of
<a href="https://guix.gnu.org/manual/devel/en/html_node/Development.html">many</a>
<a href="https://guix.gnu.org/manual/devel/en/html_node/Utilities.html">tools</a>
providing facilities around packages, the implementation of additional
<a href="https://emacs-guix.gitlab.io/website/">user</a>
<a href="https://github.com/UMCUGenetics/hpcguix-web/">interfaces</a>, and work on
applications that use Guix as a library such as the <a href="https://www.guixwl.org/">Guix Workflow
Language</a> and
<a href="https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/">Guix-Jupyter</a>.</p><p>As for the benefits of the host general-purpose language, these are
rather obvious: Guix developers benefit from an expressive language, an
optimizing compiler, a debugger, a powerful <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html">read-eval-print loop
(REPL)</a>,
an <a href="https://nongnu.org/geiser/">interactive development environment</a>,
and all sorts of libraries. Moving to Guile 3 should add to that better
performance, essentially for free. To be comprehensive, Guile 3 may
well come with a set of brand new bugs too, but so far we seem to be
doing OK!</p><h1>Migrating to Guile 3</h1><p>What does it mean for Guix to migrate to Guile 3? We’ve seen above
different ways in which Guix relies on Guile. In short, we can say that
migration is threefold:</p><ol><li>Guix is a distro that ships Guile-related packages. Like any other
distro, it will have to upgrade its <code>guile</code> package to 3.0 and to
ensure packages that depend on it and updated as well.</li><li>Guix is a program written in Guile. As such, we need to make sure
that all its dependencies (half a dozen of Guile libraries) work
with Guile 3 and that Guix itself runs fine with Guile 3.</li><li>Guix ties together operating system components. In particular, the
init system (the Shepherd) and other boot-time facilities will also
migrate.</li></ol><h2>The packages</h2><p>Updating the distro is the boring part, but it’s best to get it right.
Guix makes it possible to have unrelated versions of variants of
packages in different environments or different profiles, which is very
nice. We’ll have performed a smooth transition if users and tools see
that the packages named <code>guile</code> and <code>guile-ssh</code> (say) transparently move
from Guile 2.2 to 3.0, <em>in lockstep</em>.</p><p>Put differently, most of the upgrade work upon a programming language
version bump deals with conventions, and in particular package names.
Currently, <code>guile</code> corresponds to the 2.2 stable series and all the
<code>guile-*</code> packages are built against it. In the meantime, the package
for Guile 3 is named <code>guile-next</code> and packages built against it are
called <code>guile3.0-*</code>. Over the last few weeks we created <code>guile3.0-</code>
variants for most Guile packages, something that’s <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=89a99d53f56c7c383659d821c28286b6d71e458d">easily achieved with
Guix</a>.</p><p>The big switch will consist in renaming all current <code>guile-*</code> packages
to <code>guile2.2-*</code> packages, for use with the legacy 2.2 series, and
renaming all the <code>guile3.0-*</code> packages to <code>guile-*</code>. We will switch
soon, but before getting there, we’re making sure important packages are
available for 3.0.</p><h2>Guix-the-program</h2><p>A more interesting part is “porting” Guix itself from Guile 2.2 to
Guile 3. It seems that developers have become wary of 2-to-3
transitions for programming languages. Fear not! Switching from
Guile 2 to Guile 3 turned out to be an easy task. In fact, very little
changed in the language itself; what did change—e.g., semantics on fine
points of the module system, support for structured exceptions—is either
optional or backwards-compatible.</p><p>As Guile 2.9 pre-releases trickled in, we started testing all the Guile
libraries Guix relies on against 2.9. For the vast majority of them,
all we had to do was to <a href="https://gitlab.com/gnutls/gnutls/commit/763e31d351933222281bf9c11ff0bddb89bb701d">update their <code>configure.ac</code> to allow builds
with
3.0</a>.</p><p>Guix itself was a bit more work, mostly because it’s a rather large code
base with a picky test suite. The bit that required most work has to do
with the introduction of <a href="https://www.gnu.org/software/guile/manual/html_node/Declarative-Modules.html"><em>declarative
modules</em></a>,
an optional semantic change in modules to support more compiler
optimizations. We had several <a href="https://en.wikipedia.org/wiki/White-box_testing">“white-box
tests”</a> where tests
would happily peek at private module bindings through <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html#index-_0040_0040">the magical-evil
<code>@@</code>
operator</a>.
Because we chose to enable declarative modules, we also had to adjust
our tests to no longer do that. And well, that’s about it!</p><p>At that point, we were able to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=da7651806102d637253cb9f5677b96d6a178fc05">create a <code>guile3.0-guix</code> package
variant</a>,
primarily for testing purposes. Soon after, we told <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>
to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8234fe653e61d0090138cbd4c48d877568355439">build Guix with 3.0 instead of
2.2</a>.
Thus, Guix users who upgrade will transparently find themselves running
Guix on Guile 3.0.</p><p>The main benefit is improved performance. Guile 3 is known to be <a href="https://www.gnu.org/software/guile/news/gnu-guile-300-released.html">up to
32 times faster than
Guile 2.2</a>
on some micro-benchmarks. Assessing the performance gains on a
“real-world” application like Guix is the real test. What would be a
relevant benchmark? At its core, Guix is essentially a compiler from
high-level descriptions of packages, operating systems, and the like, to
low-level build instructions
(<a href="https://guix.gnu.org/manual/devel/en/html_node/Derivations.html"><em>derivations</em></a>).
Thus, a good benchmark is a command that exercises little more than this
compilation step:</p><pre><code>guix build libreoffice ghc-pandoc guix --dry-run --derivation</code></pre><p>or:</p><pre><code>guix system build config.scm --dry-run --derivation</code></pre><p>On x86_64, the <code>guix build</code> command above on Guile 3 is 7% faster than
on Guile 2.2, and <code>guix system build</code>, which is more
computing-intensive, is 10% faster (heap usage is ~5% higher). This is
lower than the skyrocketing speedups observed on some microbenchmarks,
but it’s probably no surprise: these <code>guix</code> commands are short-lived (a
couple of seconds) and they’re rather I/O- and GC-intensive—something
JIT compilation cannot help with.</p><p>On 32-bit ARM, we temporarily disabled JIT <a href="https://issues.guix.gnu.org/issue/39208">due to a
bug</a>; there we observe a slight
<em>slowdown</em> compared to 2.2. This can be explained by the fact that
<a href="https://wingolog.org/archives/2018/01/17/instruction-explosion-in-guile">virtual machine (VM) instructions in 3.0 are lower-level than in
2.2</a>
and will hopefully be more than compensated for when JIT is re-enabled.</p><h2>Gluing it all together</h2><p>The last part of the Guile 3 migration has to do with how Guix, and in
particular Guix System, glues things together. As explained above, Guix
manipulates several stages of Scheme code that will run a different
points in time.</p><p>Firstly, the code that runs package builds, such as <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gnu-build-system.scm">the one that runs
<code>./configure && make && make install</code></a>,
is Guile code. Currently that code runs on Guile 2.2, but on the next
major rebuild-the-world upgrade, we will switch to Guile 3.</p><p>Additionally, Guix produces Scheme code consumed by <a href="https://www.gnu.org/software/shepherd">the
Shepherd</a>, by
<a href="https://www.gnu.org/software/mcron">GNU mcron</a>, and for <a href="https://guix.gnu.org/manual/en/html_node/Guided-Graphical-Installation.html">the graphical
installer</a>.
These will soon switch to Guile 3 as well. This kind of change is made
easy by the fact that both the package definitions and the staged code
that depends on those packages live in the same repository.</p><h1>Long live Guile 3!</h1><p>Migrating Guix to Guile 3 is a bit of work because of the many ways Guix
interacts with Guile and because of the sheer size of the code base.
For a “2-to-3” transition though, it was easy. And fundamentally, it
remains a cheap transition compared to what it brings: better
performance and new features. That’s another benefit of using a
general-purpose language.</p><p>Thumbs up to everyone involved in its development, and long live
Guile 3!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/substitutes-are-now-available-as-lzip//Substitutes are now available as lzipLudovic Courtès2019-06-17T14:30:00Z2019-06-17T14:30:00Z For a long time, our build farm at ci.guix.gnu.org has been delivering
substitutes
(pre-built binaries) compressed with gzip. Gzip was never the best
choice in terms of compression ratio, but it was a reasonable and
convenient choice: it’s rock-solid, and zlib made it easy for us to have
Guile
bindings
to perform in-process compression in our multi-threaded guix publish
server. With the exception of building software from source, downloads take the
most time of Guix package upgrades. If users can download less,
upgrades become faster, and happiness ensues. Time has come to improve
on this, and starting…<p>For a long time, our build farm at ci.guix.gnu.org has been delivering
<a href="https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html">substitutes</a>
(pre-built binaries) compressed with gzip. Gzip was never the best
choice in terms of compression ratio, but it was a reasonable and
convenient choice: it’s rock-solid, and zlib made it easy for us to have
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/zlib.scm">Guile
bindings</a>
to perform in-process compression in our multi-threaded <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-publish.html"><code>guix publish</code></a>
server.</p><p>With the exception of building software from source, downloads take the
most time of Guix package upgrades. If users can download less,
upgrades become faster, and happiness ensues. Time has come to improve
on this, and starting from early June, Guix can publish and fetch
<a href="https://nongnu.org/lzip/">lzip</a>-compressed substitutes, in addition to
gzip.</p><h1>Lzip</h1><p><a href="https://nongnu.org/lzip/">Lzip</a> is a relatively little-known
compression format, initially developed by Antonio Diaz Diaz ca. 2013.
It has several C and C++ implementations with surprisingly few lines of
code, which is always reassuring. One of its distinguishing features is
a very good compression ratio with reasonable CPU and memory
requirements, <a href="https://nongnu.org/lzip/lzip_benchmark.html">according to benchmarks published by the
authors</a>.</p><p><a href="https://nongnu.org/lzip/lzlib.html">Lzlib</a> provides a well-documented C
interface and Pierre Neidhardt set out to write bindings for that
library, which eventually landed as the <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/lzlib.scm"><code>(guix lzlib)</code>
module</a>.</p><p>With this in place we were ready to start migrating our tools, and then
our build farm, to lzip compression, so we can all enjoy smaller
downloads. Well, easier said than done!</p><h1>Migrating</h1><p>The compression format used for substitutes is not a core component like
it can be in “traditional” binary package formats <a href="https://lwn.net/Articles/789449/">such as
<code>.deb</code></a> since Guix is conceptually a
“source-based” distro. However, deployed Guix installations did not
support lzip, so we couldn’t just switch our build farm to lzip
overnight; we needed to devise a transition strategy.</p><p>Guix asks for the availability of substitutes over HTTP. For example, a
question such as:</p><blockquote><p>“Dear server, do you happen to have a binary of
<code>/gnu/store/6yc4ngrsig781bpayax2cg6pncyhkjpq-emacs-26.2</code> that I could download?”</p></blockquote><p>translates into prose to an HTTP GET of
<a href="https://ci.guix.gnu.org/6yc4ngrsig781bpayax2cg6pncyhkjpq.narinfo">https://ci.guix.gnu.org/6yc4ngrsig781bpayax2cg6pncyhkjpq.narinfo</a>,
which returns something like:</p><pre><code>StorePath: /gnu/store/6yc4ngrsig781bpayax2cg6pncyhkjpq-emacs-26.2
URL: nar/gzip/6yc4ngrsig781bpayax2cg6pncyhkjpq-emacs-26.2
Compression: gzip
NarHash: sha256:0h2ibqpqyi3z0h16pf7ii6l4v7i2wmvbrxj4ilig0v9m469f6pm9
NarSize: 134407424
References: 2dk55i5wdhcbh2z8hhn3r55x4873iyp1-libxext-1.3.3 …
FileSize: 48501141
System: x86_64-linux
Deriver: 6xqibvc4v8cfppa28pgxh0acw9j8xzhz-emacs-26.2.drv
Signature: 1;berlin.guixsd.org;KHNpZ25hdHV…</code></pre><p>(This narinfo format is inherited from <a href="https://nixos.org/nix/">Nix</a> and
implemented
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/substitute.scm?id=121d9d1a7a2406a9b1cbe22c34343775f5955b34#n283">here</a>
and
<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/scripts/publish.scm?id=121d9d1a7a2406a9b1cbe22c34343775f5955b34#n265">here</a>.)
This tells us we can download the actual binary from
<code>/nar/gzip/…-emacs-26.2</code>, and that it will be about 46 MiB (the
<code>FileSize</code> field.) This is what <code>guix publish</code> serves.</p><p>The trick we came up with was to allow <code>guix publish</code> to advertise
several URLs, one per compression format. Thus, for recently-built
substitutes, we get something <a href="https://ci.guix.gnu.org/mvhaar2iflscidl0a66x5009r44fss15.narinfo">like
this</a>:</p><pre><code>StorePath: /gnu/store/mvhaar2iflscidl0a66x5009r44fss15-gimp-2.10.12
URL: nar/gzip/mvhaar2iflscidl0a66x5009r44fss15-gimp-2.10.12
Compression: gzip
FileSize: 30872887
URL: nar/lzip/mvhaar2iflscidl0a66x5009r44fss15-gimp-2.10.12
Compression: lzip
FileSize: 18829088
NarHash: sha256:10n3nv3clxr00c9cnpv6x7y2c66034y45c788syjl8m6ga0hbkwy
NarSize: 94372664
References: 05zlxc7ckwflz56i6hmlngr86pmccam2-pcre-8.42 …
System: x86_64-linux
Deriver: vi2jkpm9fd043hm0839ibbb42qrv5xyr-gimp-2.10.12.drv
Signature: 1;berlin.guixsd.org;KHNpZ25hdHV…</code></pre><p>Notice that there are two occurrences of the <code>URL</code>, <code>Compression</code>, and
<code>FileSize</code> fields: one for gzip, and one for lzip. Old Guix instances
will just pick the first one, gzip; newer Guix will pick whichever
supported method provides the smallest <code>FileSize</code>, usually lzip. This
will make migration trivial in the future, should we add support for
other compression methods.</p><p>Users need to upgrade their Guix daemon to benefit from lzip. On a
“foreign distro”, simply run <code>guix pull</code> as root. On standalone Guix
systems, run <code>guix pull && sudo guix system reconfigure /etc/config.scm</code>. In both cases, the daemon has to be restarted, be it
with <code>systemctl restart guix-daemon.service</code> or with <code>herd restart guix-daemon</code>.</p><h1>First impressions</h1><p>This new gzip+lzip scheme has been deployed on ci.guix.gnu.org for a
week. Specifically, we run <code>guix publish -C gzip:9 -C lzip:9</code>, meaning
that we use the highest compression ratio for both compression methods.</p><p>Currently, only a small subset of the package substitutes are available
as both lzip and gzip; those that were already available as gzip have
not been recompressed. The following Guile program that taps into the
API of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-weather.html"><code>guix weather</code></a>
allows us to get some insight:</p><pre><code class="language-scheme">(use-modules (gnu) (guix)
(guix monads)
(guix scripts substitute)
(srfi srfi-1)
(ice-9 match))
(define all-packages
(@@ (guix scripts weather) all-packages))
(define package-outputs
(@@ (guix scripts weather) package-outputs))
(define (fetch-lzip-narinfos)
(mlet %store-monad ((items (package-outputs (all-packages))))
(return
(filter (lambda (narinfo)
(member "lzip" (narinfo-compressions narinfo)))
(lookup-narinfos "https://ci.guix.gnu.org" items)))))
(define (lzip/gzip-ratio narinfo)
(match (narinfo-file-sizes narinfo)
((gzip lzip)
(/ lzip gzip))))
(define (average lst)
(/ (reduce + 0 lst)
(length lst) 1.))</code></pre><p>Let’s explore this at the
<a href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a>:</p><pre><code class="language-scheme">scheme@(guile-user)> (define lst
(with-store s
(run-with-store s (fetch-lzip-narinfos))))
computing 9,897 package derivations for x86_64-linux...
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
scheme@(guile-user)> (length lst)
$4 = 2275
scheme@(guile-user)> (average (map lzip/gzip-ratio lst))
$5 = 0.7398994395478715</code></pre><p>As of this writing, around 20% of the package substitutes are
available as lzip, so take the following stats with a grain of salt.
Among those, the lzip-compressed substitute is on average 26% smaller
than the gzip-compressed one. What if we consider only packages bigger
than 5 MiB uncompressed?</p><pre><code class="language-scheme">scheme@(guile-user)> (define biggest
(filter (lambda (narinfo)
(> (narinfo-size narinfo)
(* 5 (expt 2 20))))
lst))
scheme@(guile-user)> (average (map lzip/gzip-ratio biggest))
$6 = 0.5974238562384483
scheme@(guile-user)> (length biggest)
$7 = 440</code></pre><p>For those packages, lzip yields substitutes that are 40% smaller on
average. Pretty nice! Lzip decompression is slightly more
CPU-intensive than gzip decompression, but downloads are
bandwidth-bound, so the benefits clearly outweigh the costs.</p><h1>Going forward</h1><p>The switch from gzip to lzip has the potential to make upgrades “feel”
faster, and that is great in itself.</p><p>Fundamentally though, we’ve always been looking in this project at
peer-to-peer solutions with envy. Of course, the main motivation is to
have a community-supported and resilient infrastructure, rather than a
centralized one, and that vision goes <a href="https://www.gnu.org/software/guix/blog/2017/reproducible-builds-a-status-update/">hand-in-hand with reproducible
builds</a>.</p><p>We started working on <a href="https://issues.guix.gnu.org/issue/33899">an extension to publish and fetch
substitutes</a> over
<a href="https://ipfs.io/">IPFS</a>. Thanks to its content-addressed nature, IPFS
has the potential to further reduce the amount of data that needs to be
downloaded on an upgrade.</p><p>The good news is that IPFS developers are also <a href="https://github.com/ipfs/package-managers">interested in working
with package manager
developers</a>, and I bet
there’ll be interesting discussions at <a href="https://camp.ipfs.io/">IPFS
Camp</a> in just a few days. We’re eager to pursue
our IPFS integration work, and if you’d like to join us and hack the
good hack, <a href="https://www.gnu.org/software/guix/contact/">let’s get in
touch!</a></p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>