https://guix.gnu.org/feeds/blog/cuirass.atomGNU Guix — Blog — Cuirassfeed author nameGNU Guixhttps://guix.gnu.org/themes/initial/img/icon.png2024-03-20T10:58:00Zhttps://guix.gnu.org/blog/2023/from-development-environments-to-continuous-integrationthe-ultimate-guide-to-software-development-with-guix//From development environments to continuous integration—the ultimate guide to software development with GuixLudovic Courtès2023-06-05T16:30:00Z2023-06-05T16:30:00Z Guix is a handy tool for developers; guix shell ,
in particular, gives a standalone development environment for your
package, no matter what language(s) it’s written in. To benefit from
it, you have to initially write a package definition and have it either
in Guix proper, in a channel, or directly upstream as a guix.scm file.
This last option is appealing: all developers have to do to get set up
is clone the project's repository and run guix shell , with no
arguments—we looked at the rationale for guix shell in an earlier
article . …<p>Guix is a handy tool for developers; <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-shell.html"><code>guix shell</code></a>,
in particular, gives a standalone development environment for your
package, no matter what language(s) it’s written in. To benefit from
it, you have to initially write a package definition and have it either
in Guix proper, in a channel, or directly upstream as a <code>guix.scm</code> file.
This last option is appealing: all developers have to do to get set up
is clone the project's repository and run <code>guix shell</code>, with no
arguments—we looked at the rationale for <code>guix shell</code> in <a href="https://guix.gnu.org/en/blog/2021/from-guix-environment-to-guix-shell/">an earlier
article</a>.</p><p>Development needs go beyond development environments though. How can
developers perform continuous integration of their code in Guix build
environments? How can they deliver their code straight to adventurous
users? This post describes a set of files developers can add
to their repository to set up Guix-based development
environments, continuous integration, and continuous delivery—all at
once.</p><blockquote><p><strong>Update</strong>: Check out <a href="https://guix.gnu.org/cookbook/en/html_node/Software-Development.html">the
Cookbook</a>
for an up-to-date and translated version of this tutorial!</p></blockquote><h1>Getting started</h1><p>How do we go about “Guixifying” a repository? The first step, as we’ve
seen, will be to add a <code>guix.scm</code> at the root of the repository in
question. We’ll take <a href="https://www.gnu.org/software/guile">Guile</a>
as an example in this post: it’s written in Scheme (mostly) and C, and
has a number of dependencies—a C compilation tool chain, C libraries,
Autoconf and its friends, LaTeX, and so on. The resulting <code>guix.scm</code>
looks like the usual <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">package
definition</a>,
just without the <code>define-public</code> bit:</p><pre><code class="language-scheme">;; The ‘guix.scm’ file for Guile, for use by ‘guix shell’.
(use-modules (guix)
(guix build-system gnu)
((guix licenses) #:prefix license:)
(gnu packages autotools)
(gnu packages base)
(gnu packages bash)
(gnu packages bdw-gc)
(gnu packages compression)
(gnu packages flex)
(gnu packages gdb)
(gnu packages gettext)
(gnu packages gperf)
(gnu packages libffi)
(gnu packages libunistring)
(gnu packages linux)
(gnu packages pkg-config)
(gnu packages readline)
(gnu packages tex)
(gnu packages texinfo)
(gnu packages version-control))
(package
(name "guile")
(version "3.0.99-git") ;funky version number
(source #f) ;no source
(build-system gnu-build-system)
(native-inputs
(append (list autoconf
automake
libtool
gnu-gettext
flex
texinfo
texlive-base ;for "make pdf"
texlive-epsf
gperf
git
gdb
strace
readline
lzip
pkg-config)
;; When cross-compiling, a native version of Guile itself is
;; needed.
(if (%current-target-system)
(list this-package)
'())))
(inputs
(list libffi bash-minimal))
(propagated-inputs
(list libunistring libgc))
(native-search-paths
(list (search-path-specification
(variable "GUILE_LOAD_PATH")
(files '("share/guile/site/3.0")))
(search-path-specification
(variable "GUILE_LOAD_COMPILED_PATH")
(files '("lib/guile/3.0/site-ccache")))))
(synopsis "Scheme implementation intended especially for extensions")
(description
"Guile is the GNU Ubiquitous Intelligent Language for Extensions,
and it's actually a full-blown Scheme implementation!")
(home-page "https://www.gnu.org/software/guile/")
(license license:lgpl3+))</code></pre><p>Quite a bit of boilerplate, but now someone who’d like to hack on Guile
just needs to run:</p><pre><code>guix shell</code></pre><p>That gives them a shell containing all the dependencies of Guile: those
listed above, but also <em>implicit dependencies</em> such as the GCC tool
chain, GNU Make, sed, grep, and so on. The chef’s recommendation:</p><pre><code>guix shell --container --link-profile</code></pre><p>That gives a shell in an isolated container, and all the dependencies
show up in <code>$HOME/.guix-profile</code>, which plays well with caches such as
<a href="https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.70/html_node/Cache-Files.html"><code>config.cache</code></a>
and absolute file names recorded in generated <code>Makefile</code>s and the likes.
The fact that the shell runs in a container brings peace of mind:
nothing but the current directory and Guile’s dependencies is visible
inside the container; nothing from the system can possibly interfere
with your development.</p><h1>Level 1: Building with Guix</h1><p>Now that we have a package definition, why not also take advantage of it
so we can build Guile with Guix? We had left the <code>source</code> field empty,
because <code>guix shell</code> above only cares about the <em>inputs</em> of our
package—so it can set up the development environment—not about the
package itself.</p><p>To build the package with Guix, we’ll need to fill out the <code>source</code>
field, along these lines:</p><pre><code class="language-scheme">(use-modules (guix)
(guix git-download) ;for ‘git-predicate’
…)
(define vcs-file?
;; Return true if the given file is under version control.
(or (git-predicate (current-source-directory))
(const #t))) ;not in a Git checkout
(package
(name "guile")
(version "3.0.99-git") ;funky version number
(source (local-file "." "guile-checkout"
#:recursive? #t
#:select? vcs-file?))
…)</code></pre><p>Here’s what we changed:</p><ol><li>We added <code>(guix git-download)</code> to our set of imported modules, so we
can use its <code>git-predicate</code> procedure.</li><li>We defined <code>vcs-file?</code> as a procedure that returns true when passed
a file that is under version control. For good measure, we add a
fallback case for when we’re not in a Git checkout: always return
true.</li><li>We set <code>source</code> to a
<a href="https://guix.gnu.org/manual/devel/en/html_node/G_002dExpressions.html#index-local_002dfile"><code>local-file</code></a>—a
recursive copy of the current directory (<code>"."</code>), limited to files
under version control (the <code>#:select?</code> bit).</li></ol><p>From there on, our <code>guix.scm</code> file serves a second purpose: it lets us
build the software with Guix. The whole point of building with Guix is
that it’s a “clean” build—you can be sure nothing from your working tree
or system interferes with the build result—and it lets you test a
variety of things. First, you can do a plain native build:</p><pre><code>guix build -f guix.scm</code></pre><p>But you can also build for another system (possibly after setting up
<a href="https://guix.gnu.org/manual/devel/en/html_node/Daemon-Offload-Setup.html">offloading</a>
or <a href="https://guix.gnu.org/manual/devel/en/html_node/Virtualization-Services.html#index-emulation">transparent
emulation</a>):</p><pre><code>guix build -f guix.scm -s aarch64-linux -s riscv64-linux</code></pre><p>… or cross-compile:</p><pre><code>guix build -f guix.scm --target=x86_64-w64-mingw32</code></pre><p>You can also use <a href="https://guix.gnu.org/manual/devel/en/html_node/Package-Transformation-Options.html">package transformation
options</a>
to test package variants:</p><pre><code># What if we built with Clang instead of GCC?
guix build -f guix.scm \
--with-c-toolchain=guile@3.0.99-git=clang-toolchain
# What about that under-tested configure flag?
guix build -f guix.scm \
--with-configure-flag=guile@3.0.99-git=--disable-networking</code></pre><p>Handy!</p><h1>Level 2: The repository as a channel</h1><p>We now have a Git repository containing (among other things) a package
definition. Can’t we turn it into a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html"><em>channel</em></a>?
After all, channels are designed to ship package definitions to users,
and that’s exactly what we’re doing with our <code>guix.scm</code>.</p><p>Turns out we can indeed turn it into a channel, but with one caveat: we
must create a separate directory for the <code>.scm</code> file(s) of our channel
so that <code>guix pull</code> doesn’t load unrelated <code>.scm</code> files when
someone pulls the channel—and in Guile, there are lots of them! So
we’ll start like this, keeping a top-level <code>guix.scm</code> symlink for the
sake of <code>guix shell</code>:</p><pre><code>mkdir -p .guix/modules
mv guix.scm .guix/modules/guile-package.scm
ln -s .guix/modules/guile-package.scm guix.scm</code></pre><p>To make it usable as part of a channel, we
need to turn our <code>guix.scm</code> file into a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Package-Modules.html">module</a>:
we do that by changing the <code>use-modules</code> form at the top to a
<code>define-module</code> form. We also need to actually <em>export</em> a package
variable, with <code>define-public</code>, while still returning the package value
at the end of the file so we can still use <code>guix shell</code> and <code>guix build -f guix.scm</code>. The end result looks like this (not repeating things that
haven’t changed):</p><pre><code class="language-scheme">(define-module (guile-package)
#:use-module (guix)
#:use-module (guix git-download) ;for ‘git-predicate’
…)
(define vcs-file?
;; Return true if the given file is under version control.
(or (git-predicate (dirname (dirname (current-source-directory))))
(const #t))) ;not in a Git checkout
(define-public guile
(package
(name "guile")
(version "3.0.99-git") ;funky version number
(source (local-file "../.." "guile-checkout"
#:recursive? #t
#:select? vcs-file?))
…))
;; Return the package object define above at the end of the module.
guile</code></pre><p>We need one last thing: a <a href="https://guix.gnu.org/manual/devel/en/html_node/Package-Modules-in-a-Sub_002ddirectory.html"><code>.guix-channel</code>
file</a>
so Guix knows where to look for package modules in our repository:</p><pre><code class="language-scheme">;; This file lets us present this repo as a Guix channel.
(channel
(version 0)
(directory ".guix/modules")) ;look for package modules under .guix/modules/</code></pre><p>To recap, we now have these files:</p><pre><code>.
├── .guix-channel
├── guix.scm → .guix/modules/guile-package.scm
└── .guix
└── modules
└── guile-package.scm</code></pre><p>And that’s it: we have a channel! (We could do better and support
<a href="https://guix.gnu.org/manual/devel/en/html_node/Specifying-Channel-Authorizations.html"><em>channel
authentication</em></a>
so users know they’re pulling genuine code. We’ll spare you the details
here but it’s worth considering!) Users can pull from this channel by
<a href="https://guix.gnu.org/manual/devel/en/html_node/Specifying-Additional-Channels.html">adding it to
<code>~/.config/guix/channels.scm</code></a>,
along these lines:</p><pre><code class="language-scheme">(append (list (channel
(name 'guile)
(url "https://git.savannah.gnu.org/git/guile.git")
(branch "main")))
%default-channels)</code></pre><p>After running <code>guix pull</code>, we can see the new package:</p><pre><code>$ guix describe
Generation 264 May 26 2023 16:00:35 (current)
guile 36fd2b4
repository URL: https://git.savannah.gnu.org/git/guile.git
branch: main
commit: 36fd2b4920ae926c79b936c29e739e71a6dff2bc
guix c5bc698
repository URL: https://git.savannah.gnu.org/git/guix.git
commit: c5bc698e8922d78ed85989985cc2ceb034de2f23
$ guix package -A ^guile$
guile 3.0.99-git out,debug guile-package.scm:51:4
guile 3.0.9 out,debug gnu/packages/guile.scm:317:2
guile 2.2.7 out,debug gnu/packages/guile.scm:258:2
guile 2.2.4 out,debug gnu/packages/guile.scm:304:2
guile 2.0.14 out,debug gnu/packages/guile.scm:148:2
guile 1.8.8 out gnu/packages/guile.scm:77:2
$ guix build guile@3.0.99-git
[…]
/gnu/store/axnzbl89yz7ld78bmx72vpqp802dwsar-guile-3.0.99-git-debug
/gnu/store/r34gsij7f0glg2fbakcmmk0zn4v62s5w-guile-3.0.99-git</code></pre><p>That’s how, as a developer, you get your software delivered directly into
the hands of users! No intermediaries, yet no loss of transparency and
provenance tracking.</p><p>With that in place, it also becomes trivial for anyone to create Docker
images, Deb/RPM packages, or a plain tarball with <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pack.html"><code>guix pack</code></a>:</p><pre><code># How about a Docker image of our Guile snapshot?
guix pack -f docker -S /bin=bin guile@3.0.99-git
# And a relocatable RPM?
guix pack -f rpm -R -S /bin=bin guile@3.0.99-git</code></pre><h1>Bonus: Package variants</h1><p>We now have an actual channel, but it contains only one package. While
we’re at it, we can <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Package-Variants.html">define package
variants</a>
in our <code>guile-package.scm</code> file, variants that we want to be able to
test as Guile developers—similar to what we did above with
transformation options. We can add them like so:</p><pre><code class="language-scheme">;; This is the ‘.guix/modules/guile-package.scm’ file.
(define-module (guile-package)
…)
(define-public guile
…)
(define (package-with-configure-flags p flags)
"Return P with FLAGS as additional 'configure' flags."
(package/inherit p
(arguments
(substitute-keyword-arguments (package-arguments p)
((#:configure-flags original-flags #~(list))
#~(append #$original-flags #$flags))))))
(define-public guile-without-threads
(package
(inherit (package-with-configure-flags guile
#~(list "--without-threads")))
(name "guile-without-threads")))
(define-public guile-without-networking
(package
(inherit (package-with-configure-flags guile
#~(list "--disable-networking")))
(name "guile-without-networking")))
;; Return the package object defined above at the end of the module.
guile</code></pre><p>We can build these variants as regular packages once we’ve pulled the
channel. Alternatively, from a checkout of Guile, we can run a command like
this one from the top level:</p><pre><code>guix build -L $PWD/.guix/modules guile-without-threads</code></pre><h1>Level 3: Setting up continuous integration</h1><p>This channel becomes even more interesting once we set up <a href="https://en.wikipedia.org/wiki/Continuous_integration">continuous
integration</a> (CI).
There are several ways to do that.</p><p>You can use one of the mainstream continuous integration tools, such as
GitLab-CI. To do that, you need to make sure you run jobs in a Docker
image or virtual machine that has Guix installed. If we were to do that
in the case of Guile, we’d have a job that runs a shell command like
this one:</p><pre><code>guix build -L $PWD/.guix/modules guile@3.0.99-git</code></pre><p>Doing this works great and has the advantage of being easy to achieve on
your favorite CI platform.</p><p>That said, you’ll really get the most of it by using
<a href="https://guix.gnu.org/en/cuirass">Cuirass</a>, a CI tool designed for and
tightly integrated with Guix. Using it is more work than using a hosted
CI tool because you first need to set it up, but that setup phase is
greatly simplified if you use its Guix System
<a href="https://guix.gnu.org/manual/devel/en/html_node/Continuous-Integration.html">service</a>.
Going back to our example, we give Cuirass a spec file that goes like
this:</p><pre><code class="language-scheme">;; Cuirass spec file to build all the packages of the ‘guile’ channel.
(list (specification
(name "guile")
(build '(channels guile))
(channels
(append (list (channel
(name 'guile)
(url "https://git.savannah.gnu.org/git/guile.git")
(branch "main")))
%default-channels))))</code></pre><p>It differs from what you’d do with other CI tools in two important ways:</p><ul><li>Cuirass knows it’s tracking <em>two</em> channels, <code>guile</code> and <code>guix</code>.
Indeed, our own <code>guile</code> package depends on many packages provided by
the <code>guix</code> channel—GCC, the GNU libc, libffi, and so on. Changes to
packages from the <code>guix</code> channel can potentially influence our
<code>guile</code> build and this is something we’d like to see as soon as
possible as Guile developers.</li><li>Build results are not thrown away: they can be distributed as
<a href="https://guix.gnu.org/manual/devel/en/html_node/Substitutes.html"><em>substitutes</em></a>
so that users of our <code>guile</code> channel transparently get pre-built
binaries!</li></ul><p>From a developer’s viewpoint, the end result is this <a href="https://ci.guix.gnu.org/jobset/guile">status
page</a> listing <em>evaluations</em>: each
evaluation is a combination of commits of the <code>guix</code> and <code>guile</code>
channels providing a number of <em>jobs</em>—one job per package defined in
<code>guile-package.scm</code> times the number of target architectures.</p><p>As for substitutes, they come for free! As an example, since our
<code>guile</code> jobset is built on ci.guix.gnu.org, which runs <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-publish.html"><code>guix publish</code></a>
in addition to Cuirass, one automatically gets substitutes for <code>guile</code>
builds from ci.guix.gnu.org; no additional work is needed for that.</p><h1>Bonus: Build manifest</h1><p>The Cuirass spec above is convenient: it builds every package in our
channel, which includes a few variants. However, this might be
insufficiently expressive in some cases: one might want specific
cross-compilation jobs, transformations, Docker images, RPM/Deb
packages, or even system tests.</p><p>To achieve that, you can write a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Writing-Manifests.html"><em>manifest</em></a>.
The one we have for Guile has entries for the package variants we
defined above, as well as additional variants and cross builds:</p><pre><code class="language-scheme">;; This is ‘.guix/manifest.scm’.
(use-modules (guix)
(guix profiles)
(guile-package)) ;import our own package module
(define* (package->manifest-entry* package system
#:key target)
"Return a manifest entry for PACKAGE on SYSTEM, optionally cross-compiled to
TARGET."
(manifest-entry
(inherit (package->manifest-entry package))
(name (string-append (package-name package) "." system
(if target
(string-append "." target)
"")))
(item (with-parameters ((%current-system system)
(%current-target-system target))
package))))
(define native-builds
(manifest
(append (map (lambda (system)
(package->manifest-entry* guile system))
'("x86_64-linux" "i686-linux"
"aarch64-linux" "armhf-linux"
"powerpc64le-linux"))
(map (lambda (guile)
(package->manifest-entry* guile "x86_64-linux"))
(cons (package
(inherit (package-with-c-toolchain
guile
`(("clang-toolchain"
,(specification->package
"clang-toolchain")))))
(name "guile-clang"))
(list guile-without-threads
guile-without-networking
guile-debug
guile-strict-typing))))))
(define cross-builds
(manifest
(map (lambda (target)
(package->manifest-entry* guile "x86_64-linux"
#:target target))
'("i586-pc-gnu"
"aarch64-linux-gnu"
"riscv64-linux-gnu"
"i686-w64-mingw32"
"x86_64-linux-gnu"))))
(concatenate-manifests (list native-builds cross-builds))</code></pre><p>We won’t go into the details of this manifest; suffice to say that it
provides additional flexibility. We now need to tell Cuirass to build
this manifest, which is done with a spec slightly different from the
previous one:</p><pre><code class="language-scheme">;; Cuirass spec file to build all the packages of the ‘guile’ channel.
(list (specification
(name "guile")
(build '(manifest ".guix/manifest.scm"))
(channels
(append (list (channel
(name 'guile)
(url "https://git.savannah.gnu.org/git/guile.git")
(branch "main")))
%default-channels))))</code></pre><p>We changed the <code>(build …)</code> part of the spec to <code>'(manifest ".guix/manifest.scm")</code> so that it would pick our manifest, and that’s
it!</p><h1>Wrapping up</h1><p>We picked Guile as the running example in this post and you can see the
result here:</p><ul><li><a href="https://git.savannah.gnu.org/cgit/guile.git/tree/.guix-channel?id=cd57379b3df636198d8cd8e76c1bfbc523762e79"><code>.guix-channel</code></a>;</li><li><a href="https://git.savannah.gnu.org/cgit/guile.git/tree/.guix/modules/guile-package.scm?id=cd57379b3df636198d8cd8e76c1bfbc523762e79"><code>.guix/modules/guile-package.scm</code></a>
with the top-level <code>guix.scm</code> symlink;</li><li><a href="https://git.savannah.gnu.org/cgit/guile.git/tree/.guix/manifest.scm?id=cd57379b3df636198d8cd8e76c1bfbc523762e79"><code>.guix/manifest.scm</code></a>.</li></ul><p>These days, repositories are commonly peppered with dot files for
various tools: <code>.envrc</code>, <code>.gitlab-ci.yml</code>, <code>.github/workflows</code>,
<code>Dockerfile</code>, <code>.buildpacks</code>, <code>Aptfile</code>, <code>requirements.txt</code>, and whatnot.
It may sound like we’re proposing a bunch of <em>additional</em> files, but in
fact those files are expressive enough to <em>supersede</em> most or all of
those listed above.</p><p>With a couple of files, we get support for:</p><ul><li>development environments (<code>guix shell</code>);</li><li>pristine test builds, including for package variants and for
cross-compilation (<code>guix build</code>);</li><li>continuous integration (with Cuirass or with some other tool);</li><li>continuous delivery to users (<em>via</em> the channel and with pre-built
binaries);</li><li>generation of derivative build artifacts such as Docker images or
Deb/RPM packages (<code>guix pack</code>).</li></ul><p>At the Guix headquarters, we’re quite happy about the result. We’ve
been building a unified tool set for reproducible software deployment;
this is an illustration of how you as a developer can benefit
from it!</p><h1>Acknowledgments</h1><p>Thanks to Attila Lendvai, Brian Cully, and Ricardo Wurmus for providing
feedback on an earlier draft of this post.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2021/cuirass-10-released//Cuirass 1.0 releasedMathieu Othacehe2021-03-31T09:00:00Z2021-03-31T09:00:00Z We are pleased to announce the release of
Cuirass version 1.0, after almost five years
of development and around 700 commits from 14 contributors. Cuirass is the GNU Guix continuous integration software. It's a general
purpose build automation server written in GNU
Guile that checks out sources from VCS
repositories, execute build jobs and store build results in a
database. Cuirass also provides a web interface to monitor the build results. Cuirass is running on the GNU Guix build farm . Since January, the project is funded through the NGI0 PET Fund, a fund
established by …<p>We are pleased to announce the release of
<a href="http://guix.gnu.org/cuirass/">Cuirass</a> version 1.0, after almost five years
of development and around 700 commits from 14 contributors.</p><p>Cuirass is the GNU Guix continuous integration software. It's a general
purpose build automation server written in <a href="https://www.gnu.org/software/guile/">GNU
Guile</a> that checks out sources from VCS
repositories, execute build jobs and store build results in a
database. Cuirass also provides a web interface to monitor the build results.</p><p>Cuirass is running on the <a href="https://ci.guix.gnu.org">GNU Guix build farm</a>.</p><p>Since January, the project is funded through the NGI0 PET Fund, a fund
established by <a href="https://nlnet.nl/">NLnet</a> with financial support from the
European Commission's Next Generation, as explained
<a href="https://othacehe.org/gnu-guix-continuous-integration---nlnet-grant.html">here</a>.</p><p>Thanks to this support, we were able to speed up the developments and finally
propose a first release of this software. Many things have changed in Cuirass
over the years and now is the perfect time to give it a try.</p><p>Here are the highlights of this new release.</p><h3>Database</h3><p>Let's start with the database that is the core of Cuirass. Until recently
Cuirass was using an SQLite3 database. This technological choice proved to be
quite challenging, and we had some troubles to make it scale as discussed
<a href="https://lists.gnu.org/archive/html/guix-devel/2021-01/msg00127.html">here</a>.</p><p>Cuirass now uses a PostgreSQL database, bringing the performance issues to an
end while providing much more stability. Almost all the SQL queries are
covered by test cases.</p><h3>Specifications</h3><p>In order to build some derivations, Cuirass first needs to be told what to
build. Originally, an obscure association list describing the requested build
jobs had to be passed to Cuirass.</p><p>Cuirass now operates on specification records that are described
<a href="http://guix.gnu.org/cuirass/manual/html_node/Specifications.html#Specifications">here</a>.
This input format is much more easy to understand for the user. It relies on
Guix <a href="https://guix.gnu.org/manual/en/html_node/Channels.html">channels</a>, which
are well-adopted.</p><p>Here are a few different specifications examples.</p><p>This will build all the packages of the <code>my-channel</code> channel.</p><pre><code class="language-scheme">(list (specification
(name "my-channel")
(build '(channels my-channel))
(channels
(cons (channel
(name 'my-channel)
(url "https://my-channel.git"))
%default-channels))))</code></pre><p>This will build the <code>linux-libre</code> package on the default Guix channel master
branch.</p><pre><code class="language-scheme">(list (specification
(name "my-linux")
(build '(packages "linux-libre"))))</code></pre><p>Finally, this will build the packages declared in the <code>my-manifest</code> file of
the <code>my-channel</code> channel, against the <code>core-updates</code> branch of the default
Guix channel.</p><pre><code class="language-scheme">(list (specification
(name "my-core-manifest")
(build '(manifests "my-manifest"))
(channels
(list (channel
(name 'my-channel)
(url "https://my-channel.git"))
(channel
(name 'guix)
(url %default-channel-url)
(branch "core-updates"))))))</code></pre><p>For people willing to spare some parens, a specification edition form has been
implemented in the Web interface.</p><p><img src="/static/blog/img/cuirass-specification-form.png" alt="Specification creation" /></p><p>The Cuirass home page has also been updated to reflect this new input format.</p><h3>Notifications</h3><p>This feature was rightfully requested many times as this is a basic of any
respectable CI system. Cuirass can now report failing and fixed builds in
three different ways:</p><ul><li><p>By email on the
<a href="https://lists.gnu.org/mailman/listinfo/guix-ci">guix-ci@gnu.org</a> mailing
list.</p></li><li><p>Using Mastodon thanks to the
<a href="https://framagit.org/prouby/guile-mastodon">Guile-Mastodon</a> bindings.</p></li><li><p>Using the RSS feed available <a href="http://ci.guix.gnu.org/events/rss/">here</a>.</p></li></ul><h3>New build mode</h3><p>The traditional way of building things in Cuirass is to send batches of
derivations that need to be built to the local <a href="https://guix.gnu.org/manual/devel/en/guix.html#Invoking-guix_002ddaemon">Guix
daemon</a>.
The daemon can possibly
<a href="https://guix.gnu.org/manual/devel/en/guix.html#Daemon-Offload-Setup">offload</a>
those builds to other machines. While it's probably the most sensible way to
proceed, this solution doesn't scale well and suffers from some limitations.</p><ul><li><p>There's no way to influence the scheduling decisions of the Guix daemon.
It's quite delicate to prioritize builds or build machines from Cuirass.</p></li><li><p>The Guix daemon doesn't offer much feedback. Cuirass needs to parse the
debug output of the daemon to detect build events such as start and stop
events.</p></li><li><p>Using a unique daemon means using unique build parameters such as build
<code>timeout</code> and <code>max-silent-time</code> properties. Some packages have different build
properties and Cuirass cannot honor them.</p></li><li><p>When relying heavily on offloading, the Guix daemon scales badly. Builds
that often take a longer time to complete, such as emulated builds can
saturate the build queue.</p></li></ul><p>For all those reasons, using a new build mode seemed like a necessary evil.
The rationale behind this new build mode is to have Cuirass communicate
directly with the Guix daemons of all the offloading machines. Instead of
dealing with a single, local, Guix daemon, Cuirass can now interact with
several Guix daemons on remote machines.</p><p>The build jobs are not submitted to the local Guix daemon. Instead, a remote
server dispatches build requests to the connect remote workers, according to
the build priorities.</p><p>The remote server and the connected workers communicate using <a href="https://zeromq.org/">ZeroMQ</a> over
TCP. The workers are able to discover the remote server using <a href="https://www.avahi.org/">Avahi</a>.</p><p>The built items are exchanged as
<a href="https://guix.gnu.org/manual/en/html_node/Substitutes.html">substitutes</a> by
spawning <a href="https://guix.gnu.org/manual/devel/en/guix.html#Invoking-guix-publish">Guix
publish</a>
servers both on the remote server and on each connected remote worker.</p><p>It seems more complex, and it is indeed more complex. However, the
performance gains are real.</p><p><img src="/static/blog/img/cuirass-idle-chart.png" alt="Build machines CPU idle percentage" /></p><p>This chart shows the CPU idle time percentage of the GNU Guix build farm
machines. The introduction of the remote building mechanism around January
2021 results in a much higher activity of the connected machines.</p><p>This remote build mode also unlocked new features such as:</p><ul><li><p>The live streaming of build logs from remote workers to Cuirass so that they
can be browsed in real time through the web interface.</p></li><li><p>The support for <code>timeout</code> and <code>max-silent-time</code> package properties.</p></li><li><p>The support for specification and package priorities.</p></li><li><p>The new "Workers status" and "Machine status" pages allowing to closely
monitor remote machine activities.</p></li></ul><p>The workers status page is accessible <a href="http://ci.guix.gnu.org/workers">here</a>.</p><p><img src="/static/blog/img/cuirass-workers-status.png" alt="Workers status page" /></p><p>The machine status page is accessible
<a href="http://ci.guix.gnu.org/machine/hydra-guix-101">here</a>.</p><p><img src="/static/blog/img/cuirass-machine-status.png" alt="Machine status page" /></p><h3>Web interface</h3><p>Besides the features related to the specification record introduction, several
improvements were brought to the Web interface.</p><p>Some administration actions that previously required manual SQL intervention
can now be performed directly through the Web interface.</p><p>Any Cuirass administrator can now:</p><ul><li>Add a specification</li><li>Edit a specification</li><li>Delete a specification</li><li>Cancel an evaluation pending builds</li><li>Retry all builds of an evaluation</li><li>Retry an evaluation</li><li>Restart a build</li></ul><p>The build page was also improved to display the build weather and a build
history.</p><p><img src="/static/blog/img/cuirass-build-page.png" alt="Build page" /></p><p>Several issues were also fixed such as the broken pagination and the negative
build duration.</p><h3>Metrics</h3><p>Cuirass computes periodically various metrics such as:</p><ul><li>Average evaluation duration per specification (seconds).</li><li>Difference between newly added derivations and built derivations per day.</li><li>Average time required for an evaluation to start its builds.</li><li>Evaluation completion speed.</li><li>Sum of currently pending builds.</li><li>Builds count per machine during the last day.</li><li>Percentage of failed evaluations.</li></ul><p><img src="/static/blog/img/cuirass-metrics.png" alt="Metrics" /></p><p>Those metrics can be browsed <a href="http://ci.guix.gnu.org/metrics">here</a>.</p><h3>Documentation</h3><p>The Cuirass documentation is now updated to reflect those changes and can be
browsed <a href="http://guix.gnu.org/cuirass/manual/">online</a>.</p><p>The release itself is available on Cuirass <a href="https://guix.gnu.org/en/cuirass/">home
page</a>.</p><p>The Guix's Cuirass package as well as the Cuirass
<a href="https://guix.gnu.org/manual/devel/en/guix.html#Continuous-Integration">service</a> were also updated.</p><h3>Going further</h3><p>The NLNet grant will allow me to keep working on Cuirass for a couple more
months. This will hopefully help us to:</p><ul><li><p>Connect more armhf/aarch64 machines to the build farm.</p></li><li><p>Fix the build dependencies <a href="https://issues.guix.gnu.org/46402">issue</a></p></li><li><p>Add a substitutes availability API and its counterpart in GNU Guix to
improve the the <code>channel-with-substitute-available</code>
<a href="https://guix.gnu.org/manual/devel/en/guix.html#Channels-with-Substitutes">procedure</a>
to take a manifest argument. This way, the <code>guix pull</code> command can be
instructed to only update to Guix revisions where the manifest packages are
all substitutable.</p></li></ul><p>This release is an important milestone as, combined with the recent substitute
<a href="https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/">improvements</a>,
the whole substitute availability & download speed situation is now largely
mitigated, at least on Intel architectures.</p><p>Don't hesitate to run your own Cuirass server to build stuff ahead of the GNU
Guix build farm, or to build your custom channels. Also feel free to share
the features you would like to see in the next Cuirass release.</p>