https://guix.gnu.org/feeds/blog/programming-interfaces.atomGNU Guix — Blog — Programming interfacesfeed author nameGNU Guixhttps://guix.gnu.org/themes/initial/img/icon.png2024-03-20T10:57:56Zhttps://guix.gnu.org/blog/2023/write-package-definitions-in-a-breeze//Write package definitions in a breezeLudovic Courtès, Philippe Virouleau2023-11-24T14:30:00Z2023-11-24T14:30:00Z More than 28,000 packages are available in Guix today, not counting
third-party channels. That’s a lot—the 5th largest GNU/Linux
distro ! But it’s nothing if the one package you
care about is missing. So even you, dear reader, may one day find
yourself defining a package for your beloved deployment tool. This post
introduces a new tool poised to significantly lower the barrier to
writing new packages. Introducing Guix Packager Defining
packages
for Guix is not all that hard but, as always, it’s much harder the first
time you do it, especially when starting from a blank page…<p>More than 28,000 packages are available in Guix today, not counting
third-party channels. That’s a lot—the <a href="https://repology.org/">5th largest GNU/Linux
distro</a>! But it’s nothing if the one package you
care about is missing. So even you, dear reader, may one day find
yourself defining a package for your beloved deployment tool. This post
introduces a new tool poised to significantly lower the barrier to
writing new packages.</p><h1>Introducing Guix Packager</h1><p><a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">Defining
packages</a>
for Guix is not all that hard but, as always, it’s much harder the first
time you do it, especially when starting from a blank page and/or not
being familiar with the programming environment of Guix. <a href="https://guix-hpc.gitlabpages.inria.fr/guix-packager/">Guix
Packager</a> is a new
web user interface to get you started—<a href="https://guix-hpc.gitlabpages.inria.fr/guix-packager/">try
it!</a>. It arrived
right in time as an aid to the <a href="https://hpc.guix.info/events/2023/workshop/program/#how-to-get-started-writing-guix-packages">packaging
tutorial</a>
given last week at the Workshop on Reproducible Software Environments.</p><p><img src="/static/blog/img/guix-packager.gif" alt="Screenshot showing the Guix Packager interface." /></p><p>The interface aims to be intuitive: fill in forms on the left
and it produces a correct, ready-to-use package definition on the right.
Importantly, it helps you avoid pitfalls that trip up many newcomers:</p><ul><li>When you add a dependency in one of the “Inputs” fields, it adds the
right variable name in the generated code <em>and</em> imports the right
<a href="https://guix.gnu.org/manual/devel/en/html_node/Package-Modules.html">package
module</a>.</li><li>Likewise, you can choose a license and be sure the <code>license</code> field
will refer to the right variable representing that license.</li><li>You can turn tests on and off, and add configure flags. These
translate to a valid <code>arguments</code> field of your package, letting you
discover the likes of <a href="https://guix.gnu.org/cookbook/en/html_node/A-Scheme-Crash-Course.html">keyword
arguments</a>
and
<a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html">G-expressions</a>
without having to first dive into the manual.</li></ul><p>Pretty cool, no?</p><h1>Implementation</h1><p>All the credit for this tool goes to co-worker and intrepid hacker
Philippe Virouleau. A unique combination of paren aversion and web
development superpowers—unique in the Guix community—led
Philippe to develop the whole thing in a glimpse (says Ludovic!).</p><p>The purpose was to provide a single view to be able to edit a package recipe,
therefore the application is a <em>single-page application</em> (SPA) written in
using the UI library Philippe is most comfortable with: <a href="https://react.dev/">React</a>,
and <a href="https://mui.com/material-ui/">MaterialUI</a> for styling the components.
It's built with <a href="https://www.typescriptlang.org/">TypeScript</a>, and the library
part actually defines all the types needed to manipulate Guix packages and their
components (such as build systems or package sources).
One of the more challenging parts was to be able to provide fast and helpful “search as you
type” results over the 28k+ packages. It required a combination of
MaterialUI's <a href="https://mui.com/material-ui/react-autocomplete/#virtualization">virtualized inputs</a>,
as well as caching the packages data in the browser's local storage,
when possible (packaging metadata itself is fetched from
<code>https://guix.gnu.org/packages.json</code>, a generic representation of the
current package set).</p><p>While the feature set provides a great starting point, there are still a few
things that may be worth implementing. For instance, only the GNU and
CMake build systems are supported so far; it would make sense to include
a few others (Python-related ones might be good candidates).</p><p>Running a local (development) version of the application can happen on
top of Guix, since—obviously—it's been developed with the <code>node</code>
version packaged in Guix, using the quite standard <code>packages.json</code> for
JavaScript dependencies installed through <code>npm</code>. <a href="https://gitlab.inria.fr/guix-hpc/guix-packager/">Contributions
welcome!</a></p><h1>Lowering the barrier to entry</h1><p>This neat tool complements a set of steps we’ve taken over time to make
packaging in Guix approachable. Indeed, while package definitions are
actually code written in the Scheme language, the <code>package</code> “language”
was designed <a href="https://arxiv.org/abs/1305.4584">from the get-go</a> to be
fully declarative—think JSON with parens instead of curly braces and
semicolons. More recently we <a href="https://guix.gnu.org/en/blog/2021/the-big-change/">simplified the way package inputs are
specified</a> with an
eye on making package definitions less intimidating.</p><p>The <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-import.html"><code>guix import</code>
command</a>
also exists to make it easier to simplify packaging: it can generate a
package definition for anything available in other package
repositories such as PyPI, CRAN, Crates.io, and so forth. If your
preference goes to curly braces rather than parens, it can also <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-import.html#index-JSON_002c-import">convert
a JSON package
description</a>
to Scheme code. Once you have your first <code>.scm</code> file, <code>guix build</code>
prints hints for common errors such missing module imports (those
<code>#:use-module</code> stanzas). We also put effort into providing <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">reference
documentation</a>,
a <a href="https://guix.gnu.org/en/videos/2020/packaging-part-two/">video
tutorial</a>, and
a <a href="https://guix.gnu.org/cookbook/en/html_node/Packaging-Tutorial.html">tutorial for more complex
packages</a>.</p><p>Do share your experience <a href="https://guix.gnu.org/contact">with us</a> and
until then, happy packaging!</p><h1>Acknowledgments</h1><p>Thanks to Felix Lechner and Timothy Sample for providing feedback on an
earlier draft of this post.</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2023/dissecting-guix-part-3-g-expressions//Dissecting Guix, Part 3: G-Expressions(2023-04-19T12:30:00Z2023-04-19T12:30:00Z Welcome back to Dissecting Guix !
Last time, we discussed monads ,
the functional programming idiom used by Guix to thread a store connection
through a series of store-related operations. Today, we'll be talking about a concept rather more specific to Guix:
g-expressions . Being an implementation of the Scheme language, Guile is built
around s-expressions , which can
represent, as the saying goes, code as data , thanks to the simple structure of
Scheme forms. As Guix's package recipes are written in Scheme, it naturally needs some way to
represent code that is to be…<p>Welcome back to <a href="https://guix.gnu.org/en/blog/tags/dissecting-guix">Dissecting Guix</a>!
Last time, we discussed <a href="https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad">monads</a>,
the functional programming idiom used by Guix to thread a store connection
through a series of store-related operations.</p><p>Today, we'll be talking about a concept rather more specific to Guix:
<em>g-expressions</em>. Being an implementation of the Scheme language, Guile is built
around <a href="https://en.wikipedia.org/wiki/S-expression"><em>s-expressions</em></a>, which can
represent, as the saying goes, <em>code as data</em>, thanks to the simple structure of
Scheme forms.</p><p>As Guix's package recipes are written in Scheme, it naturally needs some way to
represent code that is to be run only when the package is built. Additionally,
there needs to be some way to reference dependencies and retrieve output paths;
otherwise, you wouldn't be able to, for instance, create a phase to install a
file in the output directory.</p><p>So, how do we implement this "deferred" code? Well, initially Guix used plain
old s-expressions for this purpose.</p><h1>Once Upon a Time</h1><p>Let's say we want to create a store item that's just a symlink to the
<code>bin/irssi</code> file of the <code>irssi</code> package. How would we do that with an
s-expression? Well, the s-expression itself, which we call the <em>builder</em>, is
fairly simple:</p><pre><code class="language-scheme">(define sexp-builder
`(let* ((out (assoc-ref %outputs "out"))
(irssi (assoc-ref %build-inputs "irssi"))
(bin/irssi (string-append irssi "/bin/irssi")))
(symlink bin/irssi out)))</code></pre><p>If you aren't familliar with the "quoting" syntax used to create s-expressions,
I strongly recommend that you read the excellent Scheme Primer; specifically,
section 7, <a href="https://spritely.institute/static/papers/scheme-primer.html#scheme-lists-and-cons"><em>Lists and
"cons"</em></a>
and section 11, <a href="https://spritely.institute/static/papers/scheme-primer.html#scheme-extensibility"><em>On the extensibility of Scheme (and Lisps in
general)</em></a></p><p>The <code>%outputs</code> and <code>%build-inputs</code> variables are bound within builder scripts to
<em>association lists</em>, which are lists of pairs that act like key/value stores,
for instance:</p><pre><code class="language-scheme">'(("foo" . "bar")
("floob" . "blarb")
("fvoolag" . "bvarlag"))</code></pre><p>To retrieve values from association lists, which are often referred to as
<em>alists</em>, we use the <code>assoc-ref</code> procedure:</p><pre><code class="language-scheme">(assoc-ref '(("boing" . "bouncy")
("floing" . "flouncy"))
"boing")
⇒ "bouncy"</code></pre><p><code>%outputs</code>, as the name might suggest, maps derivation output names to the paths
of their respective store items, the default output being <code>out</code>, and
<code>%build-inputs</code> maps inputs labels to their store items.</p><p>The builder is the easy part; we now need to turn it into a derivation and tell
it what <code>"irssi"</code> actually refers to. For this, we use the
<code>build-expression->derivation</code> procedure from <code>(guix derivations)</code>:</p><pre><code class="language-scheme">(use-modules (guix derivations)
(guix packages)
(guix store)
(gnu packages guile)
(gnu packages irc))
(with-store store
(let ((guile-3.0-drv (package-derivation store guile-3.0))
(irssi-drv (package-derivation store irssi)))
(build-expression->derivation store "irssi-symlink" sexp-builder
#:guile-for-build guile-3.0-drv
#:inputs `(("irssi" ,irssi-drv)))))
⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …></code></pre><p>There are several things to note here:</p><ul><li>The inputs <em>must</em> all be derivations, so we need to first convert the packages
using <code>package-derivation</code>.</li><li>We need to explicitly set <code>#:guile-for-build</code>; there's no default value.</li><li>The <code>build-expression->derivation</code> and <code>package-derivation</code> procedures are
<em>not</em> monadic, so we need to explicitly pass them the store connection.</li></ul><p>The shortcomings of using s-expressions in this way are numerous: we have to
convert everything to a derivation before using it, and <em>inputs are not an
inherent aspect of the builder</em>. G-expressions were designed to overcome these
issues.</p><h1>Premortem Examination</h1><p>A g-expression is fundamentally a record of type <code><gexp></code>, which is, naturally,
defined in <code>(guix gexp)</code>. The two most important fields of this record type,
out of a total of five, are <code>proc</code> and <code>references</code>; the former is a procedure
that returns the equivalent s-expression, the latter a list containing
everything from the "outside world" that's used by the g-expression.</p><p>When we want to turn the g-expression into something that we can actually run as
code, we combine these two fields by first building any g-expression inputs that
can become derivations (leaving alone those that cannot), and then passing the
built <code>references</code> as the arguments of <code>proc</code>.</p><p>Here's an example g-expression that is essentially equivalent to our
<code>sexp-builder</code>:</p><pre><code class="language-scheme">(use-modules (guix gexp))
(define gexp-builder
#~(symlink #$(file-append irssi "/bin/irssi")
#$output))</code></pre><p><code>gexp-builder</code> is far more concise than <code>sexp-builder</code>; let's examine the syntax
and the <code><gexp></code> object we've created. To make a g-expression, we use the <code>#~</code>
syntax, equivalent to the <code>gexp</code> macro, rather than the <code>quasiquote</code> backtick
used to create s-expressions.</p><p>When we want to embed values from outside as references, we use <code>#$</code>, or
<code>ungexp</code>, which is, in appearance if not function, equivalent to <code>unquote</code>
(<code>,</code>). <code>ungexp</code> can accept any of four reference types:</p><ul><li>S-expressions (strings, lists, etc), which will be embedded literally.</li><li>Other g-expressions, embedded literally.</li><li>Expressions returning any sort of object that can be lowered into a
derivation, such as <code><package></code>, embedding that object's <code>out</code> store item; if
the expression is specifically a symbol bound to a buildable object, you can
optionally follow it with a colon and an alternative output name, so
<code>package:lib</code> is permitted, but <code>(get-package):lib</code> isn't.</li><li>The symbol <code>output</code>, embedding an output path. Like symbols bound to
buildable objects, this can be followed by a colon and the output name that
should be used rather than the default <code>out</code>.</li></ul><p>All these reference types will be represented by <code><gexp-input></code> records in the
<code>references</code> field, except for the last kind, which will become <code><gexp-output></code>
records. To give an example of each type of reference (with the return value
output formatted for easier reading):</p><pre><code class="language-scheme">(use-modules (gnu packages glib))
#~(list #$"foobar" ;s-expression
#$#~(string-append "foo" "bar") ;g-expression
#$(file-append irssi "/bin/irssi") ;buildable object (expression)
#$glib:bin ;buildable object (symbol)
#$output:out) ;output
⇒ #<gexp (list #<gexp-input "foobar":out>
#<gexp-input #<gexp (string-append "foo" "bar") …>:out>
#<gexp-input #<file-append #<package irssi@1.4.3 …> "/bin/irssi">:out>
#<gexp-input #<package glib@2.70.2 …>:bin>
#<gexp-output out>) …></code></pre><p>Note the use of <code>file-append</code> in both the previous example and <code>gexp-builder</code>;
this procedure produces a <code><file-append></code> object that builds its first argument
and is embedded as the concatenation of the first argument's output path and the
second argument, which should be a string. For instance,
<code>(file-append irssi "/bin/irssi")</code> builds <code>irssi</code> and expands to
<code>/gnu/store/…-irssi/bin/irssi</code>, rather than the <code>/gnu/store/…-irssi</code> that the
package alone would be embedded as.</p><p>So, now that we have a g-expression, how do we turn it into a derivation? This
process is known as <em>lowering</em>; it entails the use of the aptly-named
<code>lower-gexp</code> monadic procedure to combine <code>proc</code> and <code>references</code> and produce a
<code><lowered-gexp></code> record, which acts as a sort of intermediate representation
between g-expressions and derivations. We can piece apart this lowered form to
get a sense of what the final derivation's builder script would look like:</p><pre><code class="language-scheme">(define lowered-gexp-builder
(with-store store
(run-with-store store
(lower-gexp gexp-builder))))
(lowered-gexp-sexp lowered-gexp-builder)
⇒ (symlink
"/gnu/store/…-irssi-1.4.3/bin/irssi"
((@ (guile) getenv) "out"))</code></pre><p>And there you have it: a s-expression compiled from a g-expression, ready to be
written into a builder script file in the store. So, how exactly do you turn
this into said derivation?</p><p>Well, it turns out that there isn't an interface for turning lowered
g-expressions into derivations, only one for turning regular g-expressions into
derivations that first uses <code>lower-gexp</code>, then implements the aforementioned
conversion internally, rather than outsourcing it to some other procedure, so
that's what we'll use.</p><p>Unsurprisingly, that procedure is called <code>gexp->derivation</code>, and unlike its
s-expression equivalent, it's monadic. (<code>build-expression->derivation</code> and
other deprecated procedures were in Guix since before the monads system
existed.)</p><pre><code class="language-scheme">(with-store store
(run-with-store store
(gexp->derivation "irssi-symlink" gexp-builder)))
⇒ #<derivation /gnu/store/…-irssi-symlink.drv => /gnu/store/…-irssi-symlink …></code></pre><p>Finally, we have a g-expression-based equivalent to the derivation we earlier
created with <code>build-expression->derivation</code>! Here's the code we used for the
s-expression version in full:</p><pre><code class="language-scheme">(define sexp-builder
`(let* ((out (assoc-ref %outputs "out"))
(irssi (assoc-ref %build-inputs "irssi"))
(bin/irssi (string-append irssi "/bin/irssi")))
(symlink bin/irssi out)))
(with-store store
(let ((guile-3.0-drv (package-derivation store guile-3.0))
(irssi-drv (package-derivation store irssi)))
(build-expression->derivation store "irssi-symlink" sexp-builder
#:guile-for-build guile-3.0-drv
#:inputs `(("irssi" ,irssi-drv)))))</code></pre><p>And here's the g-expression equivalent:</p><pre><code class="language-scheme">(define gexp-builder
#~(symlink #$(file-append irssi "/bin/irssi")
#$output))
(with-store store
(run-with-store store
(gexp->derivation "irssi-symlink" gexp-builder)))</code></pre><p>That's a lot of complexity abstracted away! For more complex packages and
services, especially, g-expressions are a lifesaver; you can refer to the output
paths of inputs just as easily as you would a string constant. You do, however,
have to watch out for situations where <code>ungexp-native</code>, written as <code>#+</code>, would
be preferable over regular <code>ungexp</code>, and that's something we'll discuss later.</p><p>A brief digression before we continue: if you'd like to look inside a <code><gexp></code>
record, but you'd rather not build anything, you can use the
<code>gexp->approximate-sexp</code> procedure, which replaces all references with dummy
values:</p><pre><code class="language-scheme">(gexp->approximate-sexp gexp-builder)
⇒ (symlink (*approximate*) (*approximate*))</code></pre><h1>The Lowerable-Object Hardware Shop</h1><p>We've seen two examples already of records we can turn into derivations, which
are generally referred to as <em>lowerable objects</em> or <em>file-like objects</em>:</p><ul><li><code><package></code>, a Guix package.</li><li><code><file-append></code>, which wraps another lowerable object and appends a string to
the embedded output path when <code>ungexp</code>ed.</li></ul><p>There are many more available to us. Recall from the previous post,
<a href="https://guix.gnu.org/en/blog/2023/dissecting-guix-part-2-the-store-monad"><em>The Store Monad</em></a>,
that Guix provides the two monadic procedures <code>text-file</code> and <code>interned-file</code>,
which can be used, respectively, to put arbitrary text or files from the
filesystem in the store, returning the path to the created item.</p><p>This doesn't work so well with g-expressions, though; you'd have to wrap each
<code>ungexp</code>ed use of either of them with
<code>(with-store store (run-with-store store …))</code>, which would be quite tedious.
Thankfully, <code>(guix gexp)</code> provides the <code>plain-file</code> and <code>local-file</code> procedures,
which return equivalent lowerable objects. This code example builds a directory
containing symlinks to files greeting the world:</p><pre><code class="language-scheme">(use-modules (guix monads)
(ice-9 ftw)
(ice-9 textual-ports))
(define (build-derivation monadic-drv)
(with-store store
(run-with-store store
(mlet* %store-monad ((drv monadic-drv))
(mbegin %store-monad
;; BUILT-DERIVATIONS is the monadic version of BUILD-DERIVATIONS.
(built-derivations (list drv))
(return (derivation-output-path
(assoc-ref (derivation-outputs drv) "out"))))))))
(define world-greeting-output
(build-derivation
(gexp->derivation "world-greeting"
#~(begin
(mkdir #$output)
(symlink #$(plain-file "hi-world"
"Hi, world!")
(string-append #$output "/hi"))
(symlink #$(plain-file "hello-world"
"Hello, world!")
(string-append #$output "/hello"))
(symlink #$(plain-file "greetings-world"
"Greetings, world!")
(string-append #$output "/greetings"))))))
;; We turn the list into multiple values using (APPLY VALUES …).
(apply values
(map (lambda (file-path)
(let* ((path (string-append world-greeting-output "/" file-path))
(contents (call-with-input-file path get-string-all)))
(list path contents)))
;; SCANDIR from (ICE-9 FTW) returns the list of all files in a
;; directory (including ``.'' and ``..'', so we remove them with the
;; second argument, SELECT?, which specifies a predicate).
(scandir world-greeting-output
(lambda (path)
(not (or (string=? path ".")
(string=? path "..")))))))
⇒ ("/gnu/store/…-world-greeting/greetings" "Greetings, world!")
⇒ ("/gnu/store/…-world-greeting/hello" "Hello, world!")
⇒ ("/gnu/store/…-world-greeting/hi" "Hi, world!")</code></pre><p>Note that we define a procedure for building the output; we will need to build
more derivations in a very similar fashion later, so it helps to have this to
reuse instead of copying the code in <code>world-greeting-output</code>.</p><p>There are many other useful lowerable objects available as part of the gexp
library. These include <code>computed-file</code>, which accepts a gexp that builds
the output file, <code>program-file</code>, which creates an executable Scheme script in
the store using a g-expression, and <code>mixed-text-file</code>, which allows you to,
well, mix text and lowerable objects; it creates a file from the concatenation
of a sequence of strings and file-likes. The
<a href="https://guix.gnu.org/manual/en/html_node/G_002dExpressions.html">G-Expressions</a>
manual page has more details.</p><p>So, you may be wondering, at this point: there's so many lowerable objects
included with the g-expression library, surely there must be a way to define
more? Naturally, there is; this is Scheme, after all! We simply need to
acquaint ourselves with the <code>define-gexp-compiler</code> macro.</p><p>The most basic usage of <code>define-gexp-compiler</code> essentially creates a procedure
that takes as arguments a record to lower, the host system, and the target
system, and returns a derivation or store item as a monadic value in
<code>%store-monad</code>.</p><p>Let's try implementing a lowerable object representing a file that greets the
world. First, we'll define the record type:</p><pre><code class="language-scheme">(use-modules (srfi srfi-9))
(define-record-type <greeting-file>
(greeting-file greeting)
greeting?
(greeting greeting-file-greeting))</code></pre><p>Now we use <code>define-gexp-compiler</code> like so; note how we can use <code>lower-object</code>
to compile down any sort of lowerable object into the equivalent store item or
derivation; essentially, <code>lower-object</code> is just the procedure for applying the
right gexp-compiler to an object:</p><pre><code class="language-scheme">(use-modules (ice-9 i18n))
(define-gexp-compiler (greeting-file-compiler
(greeting-file <greeting-file>)
system target)
(lower-object
(let ((greeting (greeting-file-greeting greeting-file)))
(plain-file (string-append greeting "-greeting")
(string-append (string-locale-titlecase greeting) ", world!")))))</code></pre><p>Let's try it out now. Here's how we could rewrite our greetings directory
example from before using <code><greeting-file></code>:</p><pre><code class="language-scheme">(define world-greeting-2-output
(build-derivation
(gexp->derivation "world-greeting-2"
#~(begin
(mkdir #$output)
(symlink #$(greeting-file "hi")
(string-append #$output "/hi"))
(symlink #$(greeting-file "hello")
(string-append #$output "/hello"))
(symlink #$(greeting-file "greetings")
(string-append #$output "/greetings"))))))
(apply values
(map (lambda (file-path)
(let* ((path (string-append world-greeting-2-output
"/" file-path))
(contents (call-with-input-file path get-string-all)))
(list path contents)))
(scandir world-greeting-2-output
(lambda (path)
(not (or (string=? path ".")
(string=? path "..")))))))
⇒ ("/gnu/store/…-world-greeting-2/greetings" "Greetings, world!")
⇒ ("/gnu/store/…-world-greeting-2/hello" "Hello, world!")
⇒ ("/gnu/store/…-world-greeting-2/hi" "Hi, world!")</code></pre><p>Now, this is probably not worth a whole new gexp-compiler. How about something
a bit more complex? Sharp-eyed readers who are trying all this in the REPL may
have noticed the following output when they used <code>define-gexp-compiler</code>
(formatted for ease of reading):</p><pre><code class="language-scheme">⇒ #<<gexp-compiler>
type: #<record-type <greeting-file>>
lower: #<procedure … (greeting-file system target)>
expand: #<procedure default-expander (thing obj output)>></code></pre><p>Now, the purpose of <code>type</code> and <code>lower</code> is self-explanatory, but what's this
<code>expand</code> procedure here? Well, if you recall <code>file-append</code>, you may realise
that the text produced by a gexp-compiler for embedding into a g-expression
doesn't necessarily have to be the exact output path of the produced derivation.</p><p>There turns out to be another way to write a <code>define-gexp-compiler</code> form that
allows you to specify <em>both</em> the lowering procedure, which produces the
derivation or store item, and the expanding procedure, which produces the text.</p><p>Let's try making another new lowerable object; this one will let us build a
Guile package and expand to the path to its module directory. Here's our
record:</p><pre><code class="language-scheme">(define-record-type <module-directory>
(module-directory package)
module-directory?
(package module-directory-package))</code></pre><p>Here's how we define both a compiler and expander for our new record:</p><pre><code class="language-scheme">(use-modules (gnu packages guile)
(guix utils))
(define lookup-expander (@@ (guix gexp) lookup-expander))
(define-gexp-compiler module-directory-compiler <module-directory>
compiler => (lambda (obj system target)
(let ((package (module-directory-package obj)))
(lower-object package system #:target target)))
expander => (lambda (obj drv output)
(let* ((package (module-directory-package obj))
(expander (or (lookup-expander package)
(lookup-expander drv)))
(out (expander package drv output))
(guile (or (lookup-package-input package "guile")
guile-3.0))
(version (version-major+minor
(package-version guile))))
(string-append out "/share/guile/site/" version))))</code></pre><p>Let's try this out now:</p><pre><code class="language-scheme">(use-modules (gnu packages guile-xyz))
(define module-directory-output/guile-webutils
(build-derivation
(gexp->derivation "module-directory-output"
#~(symlink #$(module-directory guile-webutils) #$output))))
(readlink module-directory-output/guile-webutils)
⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/3.0"
(scandir module-directory-output/guile-webutils)
⇒ ("." ".." "webutils")
(define module-directory-output/guile2.2-webutils
(build-derivation
(gexp->derivation "module-directory-output"
#~(symlink #$(module-directory guile2.2-webutils) #$output))))
(readlink module-directory-output/guile2.2-webutils)
⇒ "/gnu/store/…-guile-webutils-0.1-1.d309d65/share/guile/site/2.2"
(scandir module-directory-output/guile2.2-webutils)
⇒ ("." ".." "webutils")</code></pre><p>Who knows why you'd want to do this, but it certainly works! We've looked at
why we need g-expressions, how they work, and how to extend them, and we've now
only got two more advanced features to cover: cross-build support, and modules.</p><h1>Importing External Modules</h1><p>Let's try using one of the helpful procedures from the <code>(guix build utils)</code>
module in a g-expression.</p><pre><code class="language-scheme">(define simple-directory-output
(build-derivation
(gexp->derivation "simple-directory"
#~(begin
(use-modules (guix build utils))
(mkdir-p (string-append #$output "/a/rather/simple/directory"))))))</code></pre><p>Looks fine, right? We've even got a <code>use-modules</code> in th--</p><pre><code class="language-Scheme">ERROR:
1. &store-protocol-error:
message: "build of `/gnu/store/…-simple-directory.drv' failed"
status: 100</code></pre><p>OUTRAGEOUS. Fortunately, there's an explanation to be found in the Guix build
log directory, <code>/var/log/guix/drvs</code>; locate the file using the first two
characters of the store hash as the subdirectory, and the rest as the file name,
and remember to use <code>zcat</code> or <code>zless</code>, as the logs are gzipped:</p><pre><code class="language-scheme">Backtrace:
9 (primitive-load "/gnu/store/…")
In ice-9/eval.scm:
721:20 8 (primitive-eval (begin (use-modules (guix build #)) (?)))
In ice-9/psyntax.scm:
1230:36 7 (expand-top-sequence ((begin (use-modules (guix ?)) #)) ?)
1090:25 6 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
1222:19 5 (parse _ (("placeholder" placeholder)) ((top) #(# # ?)) ?)
259:10 4 (parse _ (("placeholder" placeholder)) (()) _ c&e (eval) ?)
In ice-9/boot-9.scm:
3927:20 3 (process-use-modules _)
222:17 2 (map1 (((guix build utils))))
3928:31 1 (_ ((guix build utils)))
3329:6 0 (resolve-interface (guix build utils) #:select _ #:hide ?)
ice-9/boot-9.scm:3329:6: In procedure resolve-interface:
no code for module (guix build utils)</code></pre><p>It turns out <code>use-modules</code> can't actually find <code>(guix build utils)</code> at all.
There's no typo; it's just that to ensure the build is isolated, Guix builds
<code>module-import</code> and <code>module-importe-compiled</code> directories, and sets the
<em>Guile module path</em> within the build environment to contain said directories,
along with those containing the Guile standard library modules.</p><p>So, what to do? Turns out one of the fields in <code><gexp></code> is <code>modules</code>, which,
funnily enough, contains the names of the modules which will be used to build
the aforementioned directories. To add to this field, we use the
<code>with-imported-modules</code> macro. (<code>gexp->derivation</code> <em>does</em> provide a <code>modules</code>
parameter, but <code>with-imported-modules</code> lets you add the required modules
directly to the g-expression value, rather than later on.)</p><pre><code class="language-scheme">(define simple-directory-output
(build-derivation
(gexp->derivation "simple-directory"
(with-imported-modules '((guix build utils))
#~(begin
(use-modules (guix build utils))
(mkdir-p (string-append #$output "/a/rather/simple/directory")))))))
simple-directory-output
⇒ "/gnu/store/…-simple-directory"</code></pre><p>It works, yay. It's worth noting that while passing just the list of modules to
<code>with-imported-modules</code> works in this case, this is only because
<code>(guix build utils)</code> has no dependencies on other Guix modules. Were we to try
adding, say, <code>(guix build emacs-build-system)</code>, we'd need to use the
<code>source-module-closure</code> procedure to add its dependencies to the list:</p><pre><code class="language-scheme">(use-modules (guix modules))
(source-module-closure '((guix build emacs-build-system)))
⇒ ((guix build emacs-build-system)
(guix build gnu-build-system)
(guix build utils)
(guix build gremlin)
(guix elf)
(guix build emacs-utils))</code></pre><p>Here's another scenario: what if we want to use a module not from Guix or Guile
but a third-party library? In this example, we'll use <a href="https://github.com/aconchillo/guile-json">guile-json
</a>, a library for converting between
S-expressions and <a href="https://json.org">JavaScript Object Notation</a>.</p><p>We can't just <code>with-imported-modules</code> its modules, since it's not part of Guix,
so <code><gexp></code> provides another field for this purpose: <code>extensions</code>. Each of
these extensions is a lowerable object that produces a Guile package directory;
so usually a package. Let's try it out using the <code>guile-json-4</code> package to
produce a JSON file from a Scheme value within a g-expression.</p><pre><code class="language-scheme">(define helpful-guide-output
(build-derivation
(gexp->derivation "json-file"
(with-extensions (list guile-json-4)
#~(begin
(use-modules (json))
(mkdir #$output)
(call-with-output-file (string-append #$output "/helpful-guide.json")
(lambda (port)
(scm->json '((truth . "Guix is the best!")
(lies . "Guix isn't the best!"))
port))))))))
(call-with-input-file
(string-append helpful-guide-output "/helpful-guide.json")
get-string-all)
⇒ "{\"truth\":\"Guix is the best!\",\"lies\":\"Guix isn't the best!\"}"</code></pre><p>Amen to that, <code>helpful-guide.json</code>. Before we continue on to cross-compilation,
there's one last feature of <code>with-imported-modules</code> you should note. We can
add modules to a g-expression by name, but we can also create entirely new ones
using lowerable objects, such as in this pattern, which is used in several
places in the Guix source code to make an appropriately-configured
<code>(guix config)</code> module available:</p><pre><code class="language-scheme">(with-imported-modules `(((guix config) => ,(make-config.scm))
…)
…)</code></pre><p>In case you're wondering, <code>make-config.scm</code> is found in <code>(guix self)</code> and
returns a lowerable object that compiles to a version of the <code>(guix config)</code>
module, which contains constants usually substituted into the source code at
compile time.</p><h1>Native <code>ungexp</code></h1><p>There is another piece of syntax we can use with g-expressions, and it's called
<code>ungexp-native</code>. This helps us distinguish between native inputs and regular
host-built inputs in cross-compilation situations. We'll cover
cross-compilation in detail at a later date, but the gist of it is that it
allows you to compile a derivation for one architecture X, the target, using a
machine of architecture Y, the host, and Guix has excellent support for it.</p><p>If we cross-compile a g-expression G that <em>non-natively</em> <code>ungexp</code>s L1, a
lowerable object, from architecture Y to architecture X, both G and L1 will be
compiled for architecture X. However, if G <em>natively</em> <code>ungexp</code>s L1, G will be
compiled for X and L1 for Y.</p><p>Essentially, we use <code>ungexp-native</code> in situations where there would be no
difference between compiling on different architectures (for instance, if <code>L1</code>
were a <code>plain-file</code>), or where using L1 built for X would actually <em>break</em> G
(for instance, if <code>L1</code> corresponds to a compiled executable that needs to be run
during the build; the executable would fail to run on Y if it was built for X.)</p><p>The <code>ungexp-native</code> macro naturally has a corresponding reader syntax, <code>#+</code>, and
there's also <code>ungexp-native-splicing</code>, which is written as <code>#+@</code>. These two
pieces of syntax are used in the same way as their regular counterparts.</p><h1>Conclusion</h1><p>What have we learned in this post? To summarise:</p><ul><li>G-expressions are essentially abstractions on top of s-expressions used in
Guix to stage code, often for execution within a build environment or a
Shepherd service script.</li><li>Much like you can <code>unquote</code> external values within a <code>quasiquote</code> form, you
can <code>ungexp</code> external values to access them within a <code>gexp</code> form. The key
difference is that you may use not only s-expressions with <code>ungexp</code>, but other
g-expressions and lowerable objects too.</li><li>When a lowerable object is used with <code>ungexp</code>, the g-expression ultimately
receives the path to the object's store item (or whatever string the lowerable
object's expander produces), rather than the object itself.</li><li>A lowerable object is any record that has a "g-expression compiler" defined
for it using the <code>define-gexp-compiler</code> macro. G-expression compilers always
contain a <code>compiler</code> procedure, which converts an appropriate record into a
derivation, and sometimes an <code>expander</code> procedure, which produces the string
that is to be expanded to within g-expressions when the object is <code>ungexp</code>ed.</li><li>G-expressions record the list of modules available in their environment, which
you may expand using <code>with-imported-modules</code> to add Guix modules, and
<code>with-extensions</code> to add modules from third-party Guile packages.</li><li><code>ungexp-native</code> may be used within g-expressions to compile lowerable objects
for the host rather than the target system in cross-compilation scenarios.</li></ul><p>Mastering g-expressions is essential to understanding Guix's inner workings, so
the aim of this blog post is to be as thorough as possible. However, if you
still find yourself with questions, please don't hesitate to stop by at the IRC
channel <code>#guix:libera.chat</code> and mailing list <code>help-guix@gnu.org</code>; we'll be glad
to assist you!</p><p>Also note that due to the centrality of g-expressions to Guix, there exist a
plethora of alternative resources on this topic; here are some which you may
find useful:</p><ul><li>Arun Isaac's
<a href="https://www.systemreboot.net/post/deploy-scripts-using-g-expressions">post</a>
on using g-expressions with <code>guix deploy</code>.</li><li>Marius Bakke's
<a href="https://gexp.no/blog/guix-drops-part-3-g-expressions.html">"Guix Drops" post</a>
which explains g-expressions in a more "top-down" way.</li><li>This 2020
<a href="https://archive.fosdem.org/2020/schedule/event/gexpressionsguile/">FOSDEM talk</a>
by Christopher Marusich on the uses of g-expressions.</li><li>And, of course, the one and only original
<a href="https://hal.inria.fr/hal-01580582v1">g-expression paper</a> by Ludovic Courtès,
the original author of Guix.</li></ul><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2023/dissecting-guix-part-2-the-store-monad//Dissecting Guix, Part 2: The Store Monad(2023-02-20T21:30:00Z2023-02-20T21:30:00Z Hello again! In the last post ,
we briefly mentioned the with-store and run-with-store macros. Today, we'll
be looking at those in further detail, along with the related monad library and
the %store-monad ! Typically, we use monads to chain operations together, and the %store-monad is
no different; it's used to combine operations that work on the Guix store (for
instance, creating derivations, building derivations, or adding data files to
the store). However, monads are a little hard to explain, and from a distance, they seem to
be quite incomprehensible. …<p>Hello again!</p><p>In <a href="https://guix.gnu.org/en/blog/2023/dissecting-guix-part-1-derivations/">the last post</a>,
we briefly mentioned the <code>with-store</code> and <code>run-with-store</code> macros. Today, we'll
be looking at those in further detail, along with the related monad library and
the <a href="https://guix.gnu.org/manual/devel/en/html_node/The-Store-Monad.html"><code>%store-monad</code></a>!</p><p>Typically, we use monads to chain operations together, and the <code>%store-monad</code> is
no different; it's used to combine operations that work on the Guix store (for
instance, creating derivations, building derivations, or adding data files to
the store).</p><p>However, monads are a little hard to explain, and from a distance, they seem to
be quite incomprehensible. So, I want you to erase them from your mind for now.
We'll come back to them later. And be aware that if you can't seem to get your
head around them, it's okay; you can understand most of the architecture of Guix
without understanding monads.</p><h1>Yes, No, Maybe So</h1><p>Let's instead implement another M of functional programming, <em><code>maybe</code></em> values,
representing a value that may or may not exist. For instance, there could be a
procedure that attempts to pop a stack, returning the result if there is one, or
<code>nothing</code> if the stack has no elements.</p><p><code>maybe</code> is a very common feature of statically-typed functional languages, and
you'll see it all over the place in Haskell and OCaml code. However, Guile is
dynamically typed, so we usually use ad-hoc <code>#f</code> values as the "null value"
instead of a proper "nothing" or "none".</p><p>Just for fun, though, we'll implement a proper <code>maybe</code> in Guile. Fire up that
REPL once again, and let's import a bunch of modules that we'll need:</p><pre><code class="language-scheme">(use-modules (ice-9 match)
(srfi srfi-9))</code></pre><p>We'll implement <code>maybe</code> as a record with two fields, <code>is?</code> and <code>value</code>. If the
value contains something, <code>is?</code> will be <code>#t</code> and <code>value</code> will contain the thing
in question, and if it's empty, <code>is?</code>'ll be <code>#f</code>.</p><pre><code class="language-scheme">(define-record-type <maybe>
(make-maybe is? value)
maybe?
(is? maybe-is?)
(value maybe-value))</code></pre><p>Now we'll define constructors for the two possible states:</p><pre><code class="language-scheme">(define (something value)
(make-maybe #t value))
(define (nothing)
(make-maybe #f #f)) ;the value here doesn't matter; we'll just use #f</code></pre><p>And make some silly functions that return optional values:</p><pre><code class="language-scheme">(define (remove-a str)
(if (eq? (string-ref str 0) #\a)
(something (substring str 1))
(nothing)))
(define (remove-b str)
(if (eq? (string-ref str 0) #\b)
(something (substring str 1))
(nothing)))
(remove-a "ahh")
⇒ #<<maybe> is?: #t value: "hh">
(remove-a "ooh")
⇒ #<<maybe> is?: #f value: #f>
(remove-b "bad")
⇒ #<<maybe> is?: #t value: "ad"></code></pre><p>But what if we want to compose the results of these functions?</p><h1>Keeping Your Composure</h1><p>As you might have guessed, this is not fun. Cosplaying as a compiler backend
typically isn't.</p><pre><code class="language-scheme">(let ((t1 (remove-a "abcd")))
(if (maybe-is? t1)
(remove-b (maybe-value t1))
(nothing)))
⇒ #<<maybe> is?: #t value: "cd">
(let ((t1 (remove-a "bbcd")))
(if (maybe-is? t1)
(remove-b (maybe-value t1))
(nothing)))
⇒ #<<maybe> is?: #f value: #f></code></pre><p>I can almost hear the heckling. Even worse, composing three:</p><pre><code class="language-scheme">(let* ((t1 (remove-a "abad"))
(t2 (if (maybe-is? t1)
(remove-b (maybe-value t1))
(nothing))))
(if (maybe-is? t2)
(remove-a (maybe-value t2))
(nothing)))
⇒ #<<maybe> is?: #t value: "d"></code></pre><p>So, how do we go about making this more bearable? Well, one way could be to
make <code>remove-a</code> and <code>remove-b</code> accept <code>maybe</code>s:</p><pre><code class="language-scheme">(define (remove-a ?str)
(match ?str
(($ <maybe> #t str)
(if (eq? (string-ref str 0) #\a)
(something (substring str 1))
(nothing)))
(_ (nothing))))
(define (remove-b ?str)
(match ?str
(($ <maybe> #t str)
(if (eq? (string-ref str 0) #\b)
(something (substring str 1))
(nothing)))
(_ (nothing))))</code></pre><p>Not at all pretty, but it works!</p><pre><code class="language-scheme">(remove-b (remove-a (something "abc")))
⇒ #<<maybe> is?: #t value: "c"></code></pre><p>Still, our procedures now require quite a bit of boilerplate. Might there be a
better way?</p><h1>The Ties That <code>>>=</code> Us</h1><p>First of all, we'll revert to our original definitions of <code>remove-a</code> and
<code>remove-b</code>, that is to say, the ones that take a regular value and return a
<code>maybe</code>.</p><pre><code class="language-scheme">(define (remove-a str)
(if (eq? (string-ref str 0) #\a)
(something (substring str 1))
(nothing)))
(define (remove-b str)
(if (eq? (string-ref str 0) #\b)
(something (substring str 1))
(nothing)))</code></pre><p>What if we tried introducing higher-order procedures (procedures that accept
other procedures as arguments) into the equation? Because we're functional
programmers and we have an unhealthy obsession with that sort of thing.</p><pre><code class="language-scheme">(define (maybe-chain maybe proc)
(if (maybe-is? maybe)
(proc (maybe-value maybe))
(nothing)))
(maybe-chain (something "abc")
remove-a)
⇒ #<<maybe> is?: #t value: "bc">
(maybe-chain (nothing)
remove-a)
⇒ #<<maybe> is?: #f value: #f></code></pre><p>It lives! To make it easier to compose procedures like this, we'll define a
macro that allows us to perform any number of sequenced operations with only one
composition form:</p><pre><code class="language-scheme">(define-syntax maybe-chain*
(syntax-rules ()
((_ maybe proc)
(maybe-chain maybe proc))
((_ maybe proc rest ...)
(maybe-chain* (maybe-chain maybe proc)
rest ...))))
(maybe-chain* (something "abad")
remove-a
remove-b
remove-a)
⇒ #<<maybe> is?: #t value: "d"></code></pre><p>Congratulations, you've just implemented the <code>bind</code> operation, commonly written
as <code>>>=</code>, for our <code>maybe</code> type. And it turns out that a monad is just any
container-like value for which <code>>>=</code> (along with another procedure called
<code>return</code>, which wraps a given value in the simplest possible form of a monad)
has been implemented.</p><p>A more formal definition would be that a monad is a mathematical object composed
of three parts: a type, a <code>bind</code> function, and a <code>return</code> function. So, how do
monads relate to Guix?</p><h1>New Wheel, Old Wheel</h1><p>Now that we've reinvented the wheel, we'd better learn to use the original
wheel. Guix provides a generic, high-level monads library, along with the two
generic monads <code>%identity-monad</code> and <code>%state-monad</code>, and the Guix-specific
<code>%store-monad</code>. Since <code>maybe</code> is not one of them, let's integrate our version
into the Guix monad system!</p><p>First we'll import the module that provides the aforementioned library:</p><pre><code class="language-scheme">(use-modules (guix monads))</code></pre><p>To define a monad's behaviour in Guix, we simply use the <code>define-monad</code> macro,
and provide two procedures: <code>bind</code>, and <code>return</code>.</p><pre><code class="language-scheme">(define-monad %maybe-monad
(bind maybe-chain)
(return something))</code></pre><p><code>bind</code> is just the procedure that we use to compose monadic procedure calls
together, and <code>return</code> is the procedure that wraps values in the most basic form
of the monad. A properly implemented <code>bind</code> and <code>return</code> must follow the
so-called <em>monad laws</em>:</p><ol><li><code>(bind (return x) proc)</code> must be equivalent to <code>(proc x)</code>.</li><li><code>(bind monad return)</code> must be equivalent to just <code>monad</code>.</li><li><code>(bind (bind monad proc-1) proc-2)</code> must be equivalent to
<code>(bind monad (lambda (x) (bind (proc-1 x) proc-2)))</code>.</li></ol><p>Let's verify that our <code>maybe-chain</code> and <code>something</code> procedures adhere to the
monad laws:</p><pre><code class="language-scheme">(define (mlaws-proc-1 x)
(something (+ x 1)))
(define (mlaws-proc-2 x)
(something (+ x 2)))
;; First law: the left identity.
(equal? (maybe-chain (something 0)
mlaws-proc-1)
(mlaws-proc-1 0))
⇒ #t
;; Second law: the right identity.
(equal? (maybe-chain (something 0)
something)
(something 0))
⇒ #t
;; Third law: associativity.
(equal? (maybe-chain (maybe-chain (something 0)
mlaws-proc-1)
mlaws-proc-2)
(maybe-chain (something 0)
(lambda (x)
(maybe-chain (mlaws-proc-1 x)
mlaws-proc-2))))
⇒ #t</code></pre><p>Now that we know they're valid, we can use the <code>with-monad</code> macro to tell Guix
to use these specific implementations of <code>bind</code> and <code>return</code>, and the <code>>>=</code>
macro to thread monads through procedure calls!</p><pre><code class="language-scheme">(with-monad %maybe-monad
(>>= (something "aabbc")
remove-a
remove-a
remove-b
remove-b))
⇒ #<<maybe> is?: #t value: "c"></code></pre><p>We can also now use <code>return</code>:</p><pre><code class="language-scheme">(with-monad %maybe-monad
(return 32))
⇒ #<<maybe> is?: #t value: 32></code></pre><p>But Guix provides many higher-level interfaces than <code>>>=</code> and <code>return</code>, as we
will see. There's <code>mbegin</code>, which evaluates monadic expressions without binding
them to symbols, returning the last one. This, however, isn't particularly
useful with our <code>%maybe-monad</code>, as it's only really usable if the monadic
operations within have side effects, just like the non-monadic <code>begin</code>.</p><p>There's also <code>mlet</code> and <code>mlet*</code>, which <em>do</em> bind the results of monadic
expressions to symbols, and are essentially equivalent to a chain of
<code>(>>= MEXPR (lambda (BINDING) ...))</code>:</p><pre><code class="language-scheme">;; This is equivalent...
(mlet* %maybe-monad ((str -> "abad") ;non-monadic binding uses the -> symbol
(str1 (remove-a str))
(str2 (remove-b str1)))
(remove-a str2))
⇒ #<<maybe> is?: #t value: "d">
;; ...to this:
(with-monad %maybe-monad
(>>= (return "abad")
(lambda (str)
(remove-a str))
(lambda (str1)
(remove-b str1))
(lambda (str2)
(remove-a str2))))</code></pre><p>Various abstractions over these two exist too, such as <code>mwhen</code> (a <code>when</code> plus an
<code>mbegin</code>), <code>munless</code> (an <code>unless</code> plus an <code>mbegin</code>), and <code>mparameterize</code>
(dynamically-scoped value rebinding, like <code>parameterize</code>, in a monadic context).
<code>lift</code> takes a procedure and a monad and creates a new procedure that returns
a monadic value.</p><p>There are also interfaces for manipulating lists wrapped in monads; <code>listm</code>
creates such a list, <code>sequence</code> turns a list of monads into a list wrapped in a
monad, and the <code>anym</code>, <code>mapm</code>, and <code>foldm</code> procedures are like their non-monadic
equivalents, except that they return lists wrapped in monads.</p><p>This is all well and good, you may be thinking, but why does Guix need a monad
library, anyway? The answer is technically that it doesn't. But building on
the monad API makes a lot of things much easier, and to learn why, we're going
to look at one of Guix's built-in monads.</p><h1>In a State</h1><p>Guix implements a monad called <code>%state-monad</code>, and it works with single-argument
procedures returning two values. Behold:</p><pre><code class="language-scheme">(with-monad %state-monad
(return 33))
⇒ #<procedure 21dc9a0 at <unknown port>:1106:22 (state)></code></pre><p>The <code>run-with-state</code> value turns this procedure into an actually useful value,
or, rather, two values:</p><pre><code class="language-scheme">(run-with-state (with-monad %state-monad (return 33))
(list "foo" "bar" "baz"))
⇒ 33
⇒ ("foo" "bar" "baz")</code></pre><p>What can this actually do for us, though? Well, it gets interesting if we do
some <code>>>=</code>ing:</p><pre><code class="language-scheme">(define state-seq
(mlet* %state-monad ((number (return 33)))
(state-push number)))
result
⇒ #<procedure 7fcb6f466960 at <unknown port>:1484:24 (state)>
(run-with-state state-seq (list 32))
⇒ (32)
⇒ (33 32)
(run-with-state state-seq (list 30 99))
⇒ (30 99)
⇒ (33 30 99)</code></pre><p>What is <code>state-push</code>? It's a monadic procedure for <code>%state-monad</code> that takes
whatever's currently in the first value (the primary value) and pushes it onto
the second value (the state value), which is assumed to be a list, returning the
old state value as the primary value and the new list as the state value.</p><p>So, when we do <code>(run-with-state result (list 32))</code>, we're passing <code>(list 32)</code> as
the initial state value, and then the <code>>>=</code> form passes that and <code>33</code> to
<code>state-push</code>. What <code>%state-monad</code> allows us to do is thread together some
procedures that require some kind of state, while essentially pretending the
state value is stored globally, like you might do in, say, C, and then retrieve
both the final state and the result at the end!</p><p>If you're a bit confused, don't worry. We'll write some of our own
<code>%state-monad</code>-based monadic procedures and hopefully all will become clear.
Consider, for instance, the
<a href="https://en.wikipedia.org/wiki/Fibonacci_number">Fibonacci sequence</a>, in which
each value is computed by adding the previous two. We could use the
<code>%state-monad</code> to compute Fibonacci numbers by storing the previous number as
the primary value and the number before that as the state value:</p><pre><code class="language-scheme">(define (fibonacci-thing value)
(lambda (state)
(values (+ value state)
value)))</code></pre><p>Now we can feed our Fibonacci-generating procedure the first value using
<code>run-with-state</code> and the second using <code>return</code>:</p><pre><code class="language-scheme">(run-with-state
(mlet* %state-monad ((starting (return 1))
(n1 (fibonacci-thing starting))
(n2 (fibonacci-thing n1)))
(fibonacci-thing n2))
0)
⇒ 3
⇒ 2
(run-with-state
(mlet* %state-monad ((starting (return 1))
(n1 (fibonacci-thing starting))
(n2 (fibonacci-thing n1))
(n3 (fibonacci-thing n2))
(n4 (fibonacci-thing n3))
(n5 (fibonacci-thing n4)))
(fibonacci-thing n5))
0)
⇒ 13
⇒ 8</code></pre><p>This is all very nifty, and possibly useful in general, but what does this have
to do with Guix? Well, many Guix store-based operations are meant to be used
in concert with yet another monad, called the <code>%store-monad</code>. But if we look at
<code>(guix store)</code>, where <code>%store-monad</code> is defined...</p><pre><code class="language-scheme">(define-alias %store-monad %state-monad)
(define-alias store-return state-return)
(define-alias store-bind state-bind)</code></pre><p>It was all a shallow façade! All the "store monad" is is a special case of the
state monad, where a value representing the store is passed as the state value.</p><h1>Lies, Damned Lies, and Abstractions</h1><p>We mentioned that, technically, we didn't need monads for Guix. Indeed, many
(now deprecated) procedures take a store value as the argument, such as
<code>build-expression->derivation</code>. However, monads are far more elegant and
simplify store code by quite a bit.</p><p><code>build-expression->derivation</code>, being deprecated, should never of course be
used. For one thing, it uses the "quoted build expression" style, rather than
G-expressions (we'll discuss gexps another time). The best way to create a
derivation from some basic build code is to use the new-fangled
<code>gexp->derivation</code> procedure:</p><pre><code class="language-scheme">(use-modules (guix gexp)
(gnu packages irc))
(define symlink-irssi
(gexp->derivation "link-to-irssi"
#~(symlink #$(file-append irssi "/bin/irssi") #$output)))
⇒ #<procedure 7fddcc7b81e0 at guix/gexp.scm:1180:2 (state)></code></pre><p>You don't have to understand the <code>#~(...)</code> form yet, only everything surrounding
it. We can see that this <code>gexp->derivation</code> returns a procedure taking the
initial state (store), just like our <code>%state-monad</code> procedures did, and like we
used <code>run-with-state</code> to pass the initial state to a <code>%state-monad</code> monadic
value, we use our old friend <code>run-with-store</code> when we have a <code>%store-monad</code>
monadic value!</p><pre><code class="language-scheme">(define symlink-irssi-drv
(with-store store
(run-with-store store
symlink-irssi)))
⇒ #<derivation /gnu/store/q7kwwl4z6psifnv4di1p1kpvlx06fmyq-link-to-irssi.drv => /gnu/store/6a94niigx4ii0ldjdy33wx9anhifr25x-link-to-irssi 7fddb7ef52d0></code></pre><p>Let's just check this derivation is as expected by reading the code from the
builder script.</p><pre><code class="language-scheme">(define symlink-irssi-builder
(list-ref (derivation-builder-arguments symlink-irssi-drv) 1))
(call-with-input-file symlink-irssi-builder
(lambda (port)
(read port)))
⇒ (symlink
"/gnu/store/hrlmypx1lrdjlxpkqy88bfrzg5p0bn6d-irssi-1.4.3/bin/irssi"
((@ (guile) getenv) "out"))</code></pre><p>And indeed, it symlinks the <code>irssi</code> binary to the output path. Some other,
higher-level, monadic procedures include <code>interned-file</code>, which copies a file
from outside the store into it, and <code>text-file</code>, which copies some text into it.
Generally, these procedures aren't used, as there are higher-level procedures
that perform similar functions (which we will discuss later), but for the sake
of this blog post, here's an example:</p><pre><code class="language-scheme">(with-store store
(run-with-store store
(text-file "unmatched-paren"
"( <paren@disroot.org>")))
⇒ "/gnu/store/v6smacxvdk4yvaa3s3wmd54lixn1dp3y-unmatched-paren"</code></pre><h1>Conclusion</h1><p>What have we learned about monads? The key points we can take away are:</p><ol><li>Monads are a way of composing together procedures and values that are wrapped
in containers that give them extra context, like <code>maybe</code> values.</li><li>Guix provides a high-level monad library that compensates for Guile's lack of
static typing or an interface-like system.</li><li>The <code>(guix monads)</code> module provides the state monad, which allows you to
thread state through procedures, allowing you to essentially pretend it's a
global variable that's modified by each procedure.</li><li>Guix uses the store monad frequently to thread a store connection through
procedures that need it.</li><li>The store monad is really just the state monad in disguise, where the state
value is used to thread the store object through monadic procedures.</li></ol><p>If you've read this post in its entirety but still don't yet quite get it, don't
worry. Try to modify and tinker about with the examples, and ask any questions
on the IRC channel <code>#guix:libera.chat</code> and mailing list at <code>help-guix@gnu.org</code>,
and hopefully it will all click eventually!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2023/dissecting-guix-part-1-derivations//Dissecting Guix, Part 1: Derivations(2023-01-04T12:30:00Z2023-01-04T12:30:00Z To a new user, Guix's functional architecture can seem quite alien, and possibly
offputting. With a combination of extensive #guix -querying, determined
manual-reading, and plenty of source-perusing, they may eventually figure out
how everything fits together by themselves, but this can be frustrating and
often takes a fairly long time. However, once you peel back the layers, the "Nix way" is actually rather
elegant, if perhaps not as simple as the mutable, imperative style implemented
by the likes of dpkg and
pacman .
This series of blog posts will cover basic Guix concepts, taking a "ground-up"
approach…<p>To a new user, Guix's functional architecture can seem quite alien, and possibly
offputting. With a combination of extensive <code>#guix</code>-querying, determined
manual-reading, and plenty of source-perusing, they may eventually figure out
how everything fits together by themselves, but this can be frustrating and
often takes a fairly long time.</p><p>However, once you peel back the layers, the "Nix way" is actually rather
elegant, if perhaps not as simple as the mutable, imperative style implemented
by the likes of <a href="https://wiki.debian.org/dpkg"><code>dpkg</code></a> and
<a href="https://wiki.archlinux.org/title/pacman"><code>pacman</code></a>.
This series of blog posts will cover basic Guix concepts, taking a "ground-up"
approach by dealing with lower-level concepts first, and hopefully make those
months of information-gathering unnecessary.</p><p>Before we dig in to Guix-specific concepts, we'll need to learn about one
inherited from <a href="https://nixos.org">Nix</a>, the original functional package manager
and the inspiration for Guix; the idea of a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Derivations.html"><em>derivation</em></a>
and its corresponding <em>store items</em>.</p><p>These concepts were originally described by Eelco Dolstra, the original author
of Nix, in their <a href="https://edolstra.github.io/pubs/phd-thesis.pdf">PhD thesis</a>;
see <em>§ 2.1 The Nix store</em> and <em>§ 2.4 Store Derivations</em>.</p><h1>Store Items</h1><p>As you probably know, everything that Guix builds is stored in the <em>store</em>,
which is almost always the <code>/gnu/store</code> directory. It's the job of the
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix_002ddaemon.html"><code>guix-daemon</code></a>
to manage the store and build things. If you run
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-build.html"><code>guix build PKG</code></a>,
<code>PKG</code> will be built or downloaded from a substitute server if available, and a
path to an item in the store will be displayed.</p><pre><code>$ guix build irssi
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3</code></pre><p>This item contains the final result of building <a href="https://irssi.org"><code>irssi</code></a>.
Let's peek inside:</p><pre><code>$ ls $(guix build irssi)
bin/ etc/ include/ lib/ share/
$ ls $(guix build irssi)/bin
irssi*</code></pre><p><code>irssi</code> is quite a simple package. What about a more complex one, like
<a href="https://docs.gtk.org/glib"><code>glib</code></a>?</p><pre><code>$ guix build glib
/gnu/store/bx8qq76idlmjrlqf1faslsq6zjc6f426-glib-2.73.3-bin
/gnu/store/j65bhqwr7qq7l77nj0ahmk1f1ilnjr3a-glib-2.73.3-debug
/gnu/store/3pn4ll6qakgfvfpc4mw89qrrbsgj3jf3-glib-2.73.3-doc
/gnu/store/dvsk6x7d26nmwsqhnzws4iirb6dhhr1d-glib-2.73.3
/gnu/store/4c8ycz501n2d0xdi4blahvnbjhd5hpa8-glib-2.73.3-static</code></pre><p><code>glib</code> produces five <code>/gnu/store</code> items, because it's possible for a package to
produce multiple <a href="https://guix.gnu.org/manual/en/html_node/Packages-with-Multiple-Outputs.html">outputs</a>.
Each output can be referred to separately, by suffixing a package's name with
<code>:OUTPUT</code> where supported. For example, this
<a href="https://guix.gnu.org/manual/en/html_node/Invoking-guix-package.html"><code>guix install</code></a>
invocation will add <code>glib</code>'s <code>bin</code> output to your profile:</p><pre><code>$ guix install glib:bin</code></pre><p>The default output is <code>out</code>, so when you pass <code>glib</code> by itself to that command,
it will actually install <code>glib:out</code> to the profile.</p><p><code>guix build</code> also provides the <code>--source</code> flag, which produces the store item
corresponding to the given package's downloaded source code.</p><pre><code>$ guix build --source irssi
/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz
$ guix build --source glib
/gnu/store/d22wzjq3xm3q8hwnhbgk2xd3ph7lb6ay-glib-2.73.3.tar.xz</code></pre><p>But how does Guix know how to build these store outputs in the first place?
That's where derivations come in.</p><h1><code>.drv</code> Files</h1><p>You've probably seen these being printed by the Guix program now and again.
Derivations, represented in the daemon's eyes by <code>.drv</code> files, contain
instructions for building store items. We can retrieve the paths of these
<code>.drv</code> files with the <code>guix build --derivations</code> command:</p><pre><code>$ guix build --derivations irssi
/gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv</code></pre><p><code>guix build</code> can actually also accept derivation paths as an argument, in lieu
of a package, like so:</p><pre><code>$ guix build /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv
/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3</code></pre><p>Let's look inside this derivation file.</p><pre><code>Derive([("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3","","")],[("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv",["out"]),("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv",["out"]),("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv",["out"]),("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv",["out"]),("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv",["out"]),("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv",["out"]),("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv",["out"]),("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv",["out"])],["/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],"x86_64-linux","/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile",["--no-auto-compile","-L","/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import","-C","/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled","/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"],[("allowSubstitutes","0"),("guix properties","((type . graft) (graft (count . 2)))"),("out","/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3"),("preferLocalBuild","1")])</code></pre><p>It's... not exactly human-readable. We could try to format it and break it
down, but it'd still be pretty hard to understand, since <code>.drv</code> files contain no
labels for the fields or any other human-readable indicators. Instead, we're
going to explore derivations in a Guile REPL.</p><h1>Exploring Guix Interactively</h1><p>Before we continue, we'll want to start a REPL, so that we can try out the Guix
Guile API interactively. To run a REPL in the terminal, simply
<a href="https://guix.gnu.org/manual/devel/en/html_node/Using-Guix-Interactively.html">call <code>guix repl</code></a>.</p><p>If you're using Emacs, you can instead install
<a href="https://nongnu.org/geiser">Geiser</a>, which provides a comfortable Emacs UI for
various Lisp REPLs, invoke <code>guix repl --listen=tcp:37146 &</code>, and type
<code>M-x geiser-connect RET RET RET</code> to connect to the running Guile instance.</p><p>Your <code>.guile</code> file may contain code for enabling colours and readline bindings
that Geiser will choke on. The default Guix System <code>.guile</code> contains code to
suppress these features when <code>INSIDE_EMACS</code> is set, so you'll need to run
<code>guix repl</code> like this:</p><pre><code class="language-sh">INSIDE_EMACS=1 guix repl --listen=tcp:37146 &</code></pre><p>There are a few Guix modules we'll need. Run this Scheme code to import them:</p><pre><code class="language-scheme">(use-modules (guix)
(guix derivations)
(guix gexp)
(guix packages)
(guix store)
(gnu packages glib)
(gnu packages irc))</code></pre><p>We now have access to the store, G-expression, package, and derivation APIs,
along with the <code>irssi</code> and <code>glib</code> <code><package></code> objects.</p><h1>Creating a <code><derivation></code></h1><p>The Guix API for derivations revolves around the <code><derivation></code> record, which is
the Scheme representation of that whole block of text surrounded by
<code>Derive(...)</code>. If we look in <code>guix/derivations.scm</code>, we can see that it's
defined like this:</p><pre><code class="language-scheme">(define-immutable-record-type <derivation>
(make-derivation outputs inputs sources system builder args env-vars
file-name)
derivation?
(outputs derivation-outputs) ; list of name/<derivation-output> pairs
(inputs derivation-inputs) ; list of <derivation-input>
(sources derivation-sources) ; list of store paths
(system derivation-system) ; string
(builder derivation-builder) ; store path
(args derivation-builder-arguments) ; list of strings
(env-vars derivation-builder-environment-vars) ; list of name/value pairs
(file-name derivation-file-name)) ; the .drv file name</code></pre><p>With the exception of <code>file-name</code>, each of those fields corresponds to a field
in the <code>Derive(...)</code> form. Before we can examine them, though, we need to
figure out how to <em>lower</em> that <code>irssi</code> <code><package></code> object into a derivation.</p><p><code>guix repl</code> provides the <code>,lower</code> command to create derivations quickly,
as shown in this sample REPL session:</p><pre><code>scheme@(guile-user)> ,use (guix)
scheme@(guile-user)> ,use (gnu packages irc)
scheme@(guile-user)> irssi
$1 = #<package irssi@1.4.3 gnu/packages/irc.scm:153 7f3ff98e0c60>
scheme@(guile-user)> ,lower irssi
$2 = #<derivation /gnu/store/drjfddvlblpr635jazrg9kn5azd9hsbj-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7f3ff7782d70>
;; Below we use the $N variable automatically bound by the REPL.
scheme@(guile-user)> (derivation-system $2)
$3 = "x86_64-linux"</code></pre><p>Since <code>,lower</code> is a REPL command, however, we can't use it in proper Scheme
code. It's quite useful for exploring specific derivations interactively, but
since the purpose of this blog post is to explain how things work inside, we're
going to use the pure-Scheme approach here.</p><p>The procedure we need to use to turn a high-level object like <code><package></code> into a
derivation is called <code>lower-object</code>; more on that in a future post. However,
this doesn't initially produce a derivation:</p><pre><code class="language-scheme">(pk (lower-object irssi))
;;; (#<procedure 7fe17c7af540 at guix/store.scm:1994:2 (state)>)</code></pre><p><code>pk</code> is an abbreviation for the procedure <code>peek</code>, which takes the given object,
writes a representation of it to the output, and returns it. It's especially
handy when you want to view an intermediate value in a complex expression.</p><p>The returned object is a monadic value (more on those in the next post on
monads) that needs to be evaluated in the context of a store connection. We do
this by first using <code>with-store</code> to connect to the store and bind the connection
to a name, then wrapping the <code>lower-object</code> call with <code>run-with-store</code>:</p><pre><code class="language-scheme">(define irssi-drv
(pk (with-store %store
(run-with-store %store
(lower-object irssi)))))
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fe1902b6140>)
(define glib-drv
(pk (with-store %store
(run-with-store %store
(lower-object glib)))))
;;; (#<derivation /gnu/store/81qqs7xah2ln39znrji4r6xj85zi15bi-glib-2.70.2.drv => /gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin /gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug /gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2 /gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static 7fe17ca13b90>)</code></pre><p>And we have liftoff! Now we've got two <code><derivation></code> records to play with.</p><h1>Exploring <code><derivation></code></h1><h2><code><derivation-output></code></h2><p>The first "argument" in the <code>.drv</code> file is <code>outputs</code>, which tells the Guix
daemon about the outputs that this build can produce:</p><pre><code class="language-scheme">(define irssi-outputs
(pk (derivation-outputs irssi-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3" hash-algo: #f hash: #f recursive?: #f>)))
(pk (assoc-ref irssi-outputs "out"))
(define glib-outputs
(pk (derivation-outputs glib-drv)))
;;; ((("bin" . #<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>) ("debug" . #<<derivation-output> path: "/gnu/store/22mkp8cr6rxg6w8br9q8dbymf51b44m8-glib-2.70.2-debug" hash-algo: #f hash: #f recursive?: #f>) ("out" . #<<derivation-output> path: "/gnu/store/a6qb5arvir4vm1zlkp4chnl7d8qzzd7x-glib-2.70.2" hash-algo: #f hash: #f recursive?: #f>) ("static" . #<<derivation-output> path: "/gnu/store/y4ak268dcdwkc6lmqfk9g1dgk2jr9i34-glib-2.70.2-static" hash-algo: #f hash: #f recursive?: #f>)))
(pk (assoc-ref glib-outputs "bin"))
;;; (#<<derivation-output> path: "/gnu/store/lp7k9ygvpwxgxjvmf8bix8d2aar0azr7-glib-2.70.2-bin" hash-algo: #f hash: #f recursive?: #f>)</code></pre><p>It's a simple association list mapping output names to <code><derivation-output></code>
records, and it's equivalent to the first "argument" in the <code>.drv</code> file:</p><pre><code>[ ("out", "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3", "", "")
]</code></pre><p>The <code>hash-algo</code> and <code>hash</code> fields are for storing the content hash and the
algorithm used with that hash for what we term a <em>fixed-output derivation</em>,
which is essentially a derivation where we know what the hash of the content
will be in advance. For instance, <code>origin</code>s produce fixed-output derivations:</p><pre><code class="language-scheme">(define irssi-src-drv
(pk (with-store %store
(run-with-store %store
(lower-object (package-source irssi))))))
;;; (#<derivation /gnu/store/mcz3vzq7lwwaqjb8dy7cd69lvmi6d241-irssi-1.4.3.tar.xz.drv => /gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz 7fe17b3c8d70>)
(define irssi-src-outputs
(pk (derivation-outputs irssi-src-drv)))
;;; ((("out" . #<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)))
(pk (assoc-ref irssi-src-outputs "out"))
;;; (#<<derivation-output> path: "/gnu/store/cflbi4nbak0v9xbyc43lamzl4a539hhb-irssi-1.4.3.tar.xz" hash-algo: sha256 hash: #vu8(185 63 113 82 35 163 34 230 127 66 182 26 8 165 18 174 41 227 75 212 165 61 127 34 55 102 102 10 170 90 4 52) recursive?: #f>)</code></pre><p>Note how the <code>hash</code> and <code>hash-algo</code> now have values.</p><p>Perceptive readers may note that the <code><derivation-output></code> has four fields,
whereas the tuple in the <code>.drv</code> file only has three (minus the label). The
serialisation of <code>recursive?</code> is done by adding the prefix <code>r:</code> to the
<code>hash-algo</code> field, though its actual purpose is difficult to explain, and is out
of scope for this post.</p><h2><code><derivation-input></code></h2><p>The next field is <code>inputs</code>, which corresponds to the second field in the <code>.drv</code>
file format:</p><pre><code>[ ("/gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv", ["out"]),
("/gnu/store/baqpbl4wck7nkxrbyc9nlhma7kq5dyfl-guile-2.0.14.drv", ["out"]),
("/gnu/store/bfirgq65ndhf63nn4q6vlkbha9zd931q-openssl-1.1.1l.drv", ["out"]),
("/gnu/store/gjwpqzvfhz13shix6a6cs2hjc18pj7wy-module-import-compiled.drv", ["out"]),
("/gnu/store/ij8651x4yh53hhcn6qw2644nhh2s8kcn-glib-2.70.2.drv", ["out"]),
("/gnu/store/jg2vv6yc2yqzi3qzs82dxvqmi5k21lhy-irssi-1.4.3.drv", ["out"]),
("/gnu/store/qggpjl9g6ic3cq09qrwkm0dfsdjf7pyr-glibc-utf8-locales-2.33.drv", ["out"]),
("/gnu/store/zafabw13yyhz93jwrcz7axak1kn1f2cx-openssl-1.1.1s.drv", ["out"])
]</code></pre><p>Here, each tuple specifies a derivation that needs to be built before this
derivation can be built, and the outputs of the derivation that the build
process of this derivation uses. Let's grab us the Scheme equivalent:</p><pre><code class="language-scheme">(define irssi-inputs
(pk (derivation-inputs irssi-drv)))
;;; [a fairly large amount of output]
(pk (car irssi-inputs))
;;; (#<<derivation-input> drv: #<derivation /gnu/store/9mv9xg4kyj4h1cvsgrw7b9x34y8yppph-glib-2.70.2.drv => /gnu/store/2jj2mxn6wfrcw7i85nywk71mmqbnhzps-glib-2.70.2 7fe1902b6640> sub-derivations: ("out")>)</code></pre><p>Unlike <code>derivation-outputs</code>, <code>derivation-inputs</code> maps 1:1 to the <code>.drv</code>
form; the <code>drv</code> field is a <code><derivation></code> to be built, and the
<code>sub-derivations</code> field is a list of outputs.</p><h2>Builder Configuration</h2><p>The other fields are simpler; none of them involve new records. The third is
<code>derivation-sources</code>, which contains a list of all store items used in the build
which aren't themselves built using derivations, whereas <code>derivation-inputs</code>
contains the dependencies which are.</p><p>This list usually just contains the path to the Guile <em>build script</em> that
realises the store items when run, which we'll examine in a later post, and
the path to a directory containing extra modules to add to the build script's
<code>%load-path</code>, called <code>/gnu/store/...-module-import</code>.</p><p>The next field is <code>derivation-system</code>, which specifies the system type (such as
<code>x86_64-linux</code>) we're building for. Then we have <code>derivation-builder</code>, pointing
to the <code>guile</code> executable that runs the build script; and the second-to-last is
<code>derivation-builder-arguments</code>, which is a list of arguments to pass to
<code>derivation-builder</code>. Note how we use <code>-L</code> and <code>-C</code> to extend the Guile
<code>%load-path</code> and <code>%load-compiled-path</code> to include the <code>module-import</code> and
<code>module-import-compiled</code> directories:</p><pre><code class="language-scheme">(pk (derivation-system irssi-drv))
;;; ("x86_64-linux")
(pk (derivation-builder irrsi-drv))
;;; ("/gnu/store/hnr4r2d0h0xarx52i6jq9gvsrlc3q81a-guile-2.0.14/bin/guile")
(pk (derivation-builder-arguments irrsi-drv))
;;; (("--no-auto-compile" "-L" "/gnu/store/af18nrrsk98c5a71h3fifnxg1zi5mx7y-module-import" "-C" "/gnu/store/6rkkvvb7pl1l9ng8vvywvwf357vhm3va-module-import-compiled" "/gnu/store/qnrwmby5cwqdqxyiv1ga6azvakmdvgl7-irssi-1.4.3-builder"))</code></pre><p>The final field contains a list of environment variables to set before we start
the build process:</p><pre><code class="language-scheme">(pk (derivation-builder-environment-vars irssi-drv))
;;; ((("allowSubstitutes" . "0") ("guix properties" . "((type . graft) (graft (count . 2)))") ("out" . "/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3") ("preferLocalBuild" . "1")))</code></pre><p>The last record field, <code>derivation-file-name</code> contains the path to the <code>.drv</code>
file, and so isn't represented in a serialised derivation.</p><h1>Utilising <code><derivation></code></h1><p>Speaking of serialisation, to convert between the <code>.drv</code> text format and the
Scheme <code><derivation></code> record, you can use <code>write-derivation</code>, <code>read-derivation</code>,
and <code>read-derivation-from-file</code>:</p><pre><code class="language-scheme">(define manual-drv
(with-store %store
(derivation %store "manual"
"/bin/sh" '())))
(write-derivation manual-drv (current-output-port))
;;; -| Derive([("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example","","")],[],[],"x86_64-linux","/bin/sh",[],[("out","/gnu/store/kh7fais2zab22fd8ar0ywa4767y6xyak-example")])
(pk (read-derivation-from-file (derivation-file-name irssi-drv)))
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb3798788c0>)
(call-with-input-file (derivation-file-name irssi-drv)
read-derivation)
;;; (#<derivation /gnu/store/zcgmhac8r4kdj2s6bcvcmhh4k35qvihx-irssi-1.4.3.drv => /gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3 7fb37ad19e10>)</code></pre><p>You can realise <code><derivation></code>s as store items using the <code>build-derivations</code>
procedure:</p><pre><code class="language-scheme">(use-modules (ice-9 ftw))
(define irssi-drv-out
(pk (derivation-output-path
(assoc-ref (derivation-outputs irssi-drv) "out"))))
;;; ("/gnu/store/v5pd69j3hjs1fck4b5p9hd91wc8yf5qx-irssi-1.4.3")
(pk (scandir irssi-drv-out))
;;; (#f)
(pk (with-store %store (build-derivations %store (list irssi-drv))))
;;; (#t)
(pk (scandir irssi-drv-out))
;;; (("." ".." "bin" "etc" "include" "lib" "share"))</code></pre><h1>Conclusion</h1><p>Derivations are one of Guix's most important concepts, but are fairly easy to
understand once you get past the obtuse <code>.drv</code> file format. They provide the
Guix daemon with the initial instructions that it uses to build store items
like packages, origins, and other file-likes such as <code>computed-file</code> and
<code>local-file</code>, which will be discussed in a future post!</p><p>To recap, a derivation contains the following fields:</p><ol><li><code>derivation-outputs</code>, describing the various output paths that the derivation
builds</li><li><code>derivation-inputs</code>, describing the other derivations that need to be built
before this one is</li><li><code>derivation-sources</code>, listing the non-derivation store items that the
derivation depends on</li><li><code>derivation-system</code>, specifying the system type a derivation will be compiled
for</li><li><code>derivation-builder</code>, the executable to run the build script with</li><li><code>derivation-builder-arguments</code>, arguments to pass to the builder</li><li><code>derivation-builder-environment-vars</code>, variables to set in the builder's
environment</li></ol><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2022/keeping-ones-home-tidy//Keeping one’s home tidyLudovic Courtès2022-03-21T15:30:00Z2022-03-21T15:30:00Z How much effort to recreate your work environment when you switch to a
new machine? What would it take to roll back to your previous
environment once you’ve noticed a program no longer behaves as expected?
What about sharing your environment with friends of yours? These are
some of the things that Guix
Home ,
which landed in Guix as a “technology preview” in September 2021, aims
to make effortless, reliable, and fun. In a nutshell, Guix Home brings the fully declarative configuration of
Guix
System
to home directories. With Guix System, users and administrators provide
a configuration file that defines the…<p>How much effort to recreate your work environment when you switch to a
new machine? What would it take to roll back to your previous
environment once you’ve noticed a program no longer behaves as expected?
What about sharing your environment with friends of yours? These are
some of the things that <a href="https://guix.gnu.org/manual/devel/en/html_node/Home-Configuration.html">Guix
Home</a>,
which landed in Guix as a “technology preview” in September 2021, aims
to make effortless, reliable, and fun.</p><p>In a nutshell, Guix Home brings the <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">fully declarative configuration of
Guix
System</a>
to home directories. With Guix System, users and administrators provide
a configuration file that defines the operating system configuration;
with Guix Home, users provide a configuration file that defines the
configuration of their work environment in their home directory—their
<em>home environment</em>. That configuration is meant to <em>stand alone</em>, to
describe all the relevant aspects of your work environment. But what
exactly goes in a home environment?</p><h1>“Dot files” don’t live in a vacuum</h1><p>Among seasoned Unix-style users, we often equate “home environment” with
“dot files”—configuration files in our home directory, from <code>~/.bashrc</code>
and <code>~/.ssh/config</code> to <code>~/.emacs</code> and everything under <code>~/.config</code>.
These files are precious and many store them under version control, to
keep track of changes made to their configuration. That’s a good idea,
but is that all it takes to describe the home environment? To roll
back to a previous version?</p><p>Of course not. Dot files don’t exist in a vacuum; at the very least,
your home environment is not just a set of dot files, but also a set of
installed packages. They work together: if you upgrade a package, the
corresponding dot file might need to be adjusted; if a package is
missing, its dot file is not of any use. Sometimes a home environment
contains additional things: daemons (programs that run in the
background), or periodically executed jobs.</p><p>Guix Home goes beyond dot files: it lets you declare and
instantiate all these aspects that make up your home environment.</p><h1>Genesis</h1><p>Guix Home was initially developed by Andrew Tropin as part of the <a href="https://git.sr.ht/~abcdw/rde">rde
project</a>; it was integrated in Guix proper
six months ago. I am writing this as an adopter and contributor, but
there were a number of earlier adopters and earlier contributors. In
fact, despite being still very much under development, the tool has
already attracted a number of excited users eager to find a way to keep
their home tidy!</p><p>The idea of writing down a declaration of your home environment that you
can reproduce anytime is a natural followup to everything Guix does—you
could already declare a package set in a
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html#index-profile-manifest">manifest</a>
or even a <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">complete operating
system</a>.
It had been floating around, in Nix land with <a href="https://github.com/nix-community/home-manager">Home
Manager</a> and in Guix land
with the now-defunct <a href="https://framagit.org/tyreunom/guix-home-manager/">Guix Home
Manager</a> by Julien
Lepiller. The latter was similar to today’s Guix Home, but went one
step further by making your home directory read-only—yes, <em>read-only</em>!
The main advantage is that it would ensure statelessness—you’d be sure
that absolutely <em>all</em> your home configuration is under Guix Home
Manager’s control; sub-directories containing mutable data would have to
be explicitly declared. The downside is that it raised the barrier to
entry: you’d have to either switch entirely, or not use it at all. Guix
Home takes a more pragmatic approach and happily coexists with
configuration managed “the old way”.</p><h1>Getting started</h1><p>To get started, you need a Home configuration file. There’s
<a href="https://guix.gnu.org/manual/devel/en/html_node/Declaring-the-Home-Environment.html">documentation</a>,
but as always, starting from a blank page is a bit intimidating. So
instead of starting from a blank page, you can let <code>guix home import</code>
generate an initial config for you:</p><pre><code>guix home import ~/src/guix-config</code></pre><p>This will create the <code>~/src/guix-config</code> directory and populate it with a
bunch of files among which <code>home-configuration.scm</code> along these lines:</p><pre><code class="language-scheme">(use-modules (gnu home)
(gnu packages)
(gnu services)
(guix gexp)
(gnu home services shells))
(home-environment
(packages
(map (compose list specification->package+output)
(list "emacs-geiser-guile"
"emacs-geiser"
"pinentry-emacs"
"emacs-exwm"
"gnome-maps"
"pipe-viewer"
"emacs"
"pavucontrol"
"git"
"xterm"
"qemu"
"openssh")))
(services
(list (service home-bash-service-type
(home-bash-configuration
(aliases
'(("grep" . "grep --color=auto")
("ll" . "ls -l")
("ls" . "ls -p --color=auto")
("qemu" . "qemu-system-x86_64 -enable-kvm -m 512")
("rm" . "rm --one-file-system")))
(bashrc
(list (local-file "/home/charlie/src/guix-config/.bashrc"
"bashrc")))
(bash-profile
(list (local-file
"/home/charlie/src/guix-config/.bash_profile"
"bash_profile"))))))))</code></pre><p><code>guix home import</code> automatically added the packages of <code>~/.guix-profile</code>
to the <code>packages</code> field. Because I’m using
<a href="https://www.gnu.org/software/bash">Bash</a>, it also added an instance of
<a href="https://guix.gnu.org/manual/devel/en/html_node/Shells-Home-Services.html"><code>home-bash-service-type</code></a>
with aliases extracted from my <code>~/.bashrc</code>; it also made copies of
<code>~/.bashrc</code> and <code>~/.bash_profile</code> and refers to them.</p><p>Now that I have an initial configuration, I can first test it in an
<em>isolated container</em>:</p><pre><code>guix home container ~/src/guix-config/home-configuration.scm</code></pre><p>This command gives an interactive shell in a container where my home
environment, as declared in <code>home-configuration.scm</code>, is deployed.
There I can see my home directory as it would look like if I deploy my
home environment “for real”: I can see my <code>~/.bashrc</code> and co., I can
check that all the packages declared are in <code>$PATH</code> and visible in
<code>~/.guix-home</code>, and so on. And all this is safe: my actual home
directory has been left unchanged!</p><p>Once satisfied with my configuration, I can instantiate it:</p><pre><code>guix home reconfigure ~/src/guix-config/home-configuration.scm</code></pre><p>At that point, my actual home directory corresponds to that
configuration. Some of my dot files are now provided by Guix Home, and
thus they’re symbolic links (“symlinks”) to their read-only copy in
<code>/gnu/store</code>:</p><pre><code>$ ls -l ~/.bashrc ~/.bash_profile
lrwxrwxrwx 1 charlie users 56 Mar 7 15:46 /home/charlie/.bash_profile -> /gnu/store/lpdydssyyxx9n0xvp2jmv7yqgyr2pcg3-bash_profile
lrwxrwxrwx 1 charlie users 50 Mar 7 15:46 /home/charlie/.bashrc -> /gnu/store/kxc0j4i05sib04vf92nr8xxkb8isdfn7-bashrc</code></pre><p>But don’t worry: before creating those symlinks, <code>guix home reconfigure</code>
created backups of existing files under
<code>~/TIMESTAMP-guix-home-legacy-configs-backup</code>, where <code>TIMESTAMP</code> is a
Unix-style timestamp.</p><p>And voilà, I have my first Guix Home generation!</p><pre><code>$ guix home describe
Generation 1 Mar 07 2022 15:46:20 (current)
file name: /var/guix/profiles/per-user/charlie/guix-home-1-link
canonical file name: /gnu/store/qr1c5jpfrj815ncv6yr2lfdgs8nq8kkn-home
channels:
guix:
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 3ac1366648f933f7244c2d0b9926f7ba5d92a113
configuration file: /gnu/store/xfgasfms9rhhigyj7i8za77zpqx6zbhn-configuration.scm</code></pre><p><code>guix home describe</code> shows provenance tracking we know and love from
Guix System: all the info we need to redeploy the same home environment
elsewhere, or at a different point in time. It’s also information <code>guix home reconfigure</code> relies on to make sure you never accidentally
<em>downgrade</em> you home environment to an older Guix revision.</p><h1>Going further</h1><p>Alright, at this point, you might be thinking that it’s a lot of fuss
but the “only” benefit over dot files under version control is that
<code>guix home</code> also takes care of installing packages. Guix Home really
shines once you use higher-level services, and when you start composing
services together.</p><p>To the example above, in the <code>services</code> field, we can add a service
declaration that runs
<a href="https://guix.gnu.org/manual/devel/en/html_node/Desktop-Home-Services.html#index-home_002dredshift_002dservice_002dtype">Redshift</a>,
a program that adjusts the display color temperature according to the
time of day:</p><pre><code class="language-scheme">(service home-redshift-service-type
(home-redshift-configuration
(location-provider 'manual)
(latitude 35.81) ;northern hemisphere
(longitude -0.80))) ;west of Greenwich</code></pre><p>The effect is that, as soon as we log in, under Xorg, Redshift will be
started in the background as a <a href="https://guix.gnu.org/manual/devel/en/html_node/Shepherd-Home-Service.html">Shepherd
service</a>.
The Home-generated <code>~/.profile</code> takes care of spawning <code>shepherd</code>, which
in turn spawns the <code>redshift</code> service:</p><pre><code>$ herd status
Started:
+ root
+ redshift</code></pre><p>We gained another thing here: a consistent, unified configuration
language. Instead of learning Redshift’s configuration file format, we
define a <code>home-redshift-configuration</code> record, right in Scheme. Under
the hood, that configuration is converted into Redshift’s file format;
any error is caught at configuration time, when running <code>guix home reconfigure</code>, and we can be sure that Redshift is passed a valid
configuration file.</p><p>We can similarly define a <a href="https://guix.gnu.org/manual/devel/en/html_node/Mcron-Home-Service.html#index-home_002dmcron_002dservice_002dtype">periodic mcron
job</a>,
for example one that updates a
<a href="https://www.gnu.org/software/idutils">GNU Idutils</a> search database
(that’s a pretty convenient and speedy way to look for code or
documents!):</p><pre><code class="language-scheme">(simple-service 'idutils home-mcron-service-type
;; Every day at 12:15 and 19:15.
(list #~(job '(next-minute-from (next-hour '(12 19)) '(15))
(string-append #$idutils "/bin/mkid \
-o $HOME/.idutils/src.db $HOME/src"))))</code></pre><p>Again, <code>guix home</code> creates a Shepherd service that start mcron with a
configuration file containing definitions for periodic jobs, which we
can inspect <em>via</em> <code>herd</code>:</p><pre><code>$ herd schedule mcron | head -5
Sun Mar 20 19:15:00 2022 +0000
/gnu/store/2d026nan309qkci968k8gpa8fcv9q4mv-idutils-4.6/bin/mkid -o $HOME/.idutils/src $HOME/src
Mon Mar 21 12:15:00 2022 +0000
/gnu/store/2d026nan309qkci968k8gpa8fcv9q4mv-idutils-4.6/bin/mkid -o $HOME/.idutils/src $HOME/src</code></pre><h1>Services, composed</h1><p>If you already use Guix System, all the above certainly looks familiar:
Guix Home builds upon the <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Services.html">service
framework</a>
that powers Guix System; Home services are defined in the <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/home/services"><code>(gnu home services …)</code> module
tree</a>.</p><p>That framework lets us define relations among
“services”, in a broad sense, and how services <em>extend</em> each other—in
the example above, <code>redshift</code> and <code>mcron</code> both extend <code>shepherd</code> by
giving it a daemon to take care of. We can see those relations at play
by running:</p><pre><code>guix home extension-graph home-configuration.scm</code></pre><p>… which, for the configuration described above, gives a graph that looks
like this:</p><p><img src="/static/blog/img/home-extension-graph.svg" alt="Extension graph for home services." /></p><p>We see <code>redshift</code>, <code>mcron</code>, and <code>shepherd</code>, but we also see lower-level
services that <code>guix home</code> instantiates for us, such as the <code>profile</code>
service which takes care of deploying packages listed in the <code>packages</code>
field under <code>~/.guix-home/profile</code>. Each arrow denotes a <em>service
extension</em>. You can <a href="https://guix.gnu.org/manual/devel/en/html_node/Service-Composition.html">read
more</a>
(and <a href="https://archive.fosdem.org/2017/schedule/event/composingsystemservicesinguixsd/">view
more!</a>)
about service composition. To satisfy our math and
functional-programming geek audience, we should mention that service
types and their extension operation form a
<a href="https://en.wikipedia.org/wiki/Monoid">monoid</a>.</p><h1>What’s next?</h1><p>Let’s be clear: Guix Home is pretty new and chances are that <code>guix home search</code>—the command to search for services by keyword—won’t give you the
service you’re looking for. There’s also a bunch of open questions
left, such as how to reuse services initially defined for Guix System in
cases where they could be equally useful in Guix
Home—<a href="https://guix.gnu.org/manual/devel/en/html_node/Networking-Services.html#index-syncthing_002dservice_002dtype">Syncthing</a>,
for example.</p><p>But while it’s still a “technology preview”, it’s already a tool that
tinkerers can play with and benefit from. Patches adding new services
have already been proposed; maybe your favorite service is next?
Consider
<a href="https://guix.gnu.org/manual/devel/en/html_node/Contributing.html">contributing</a>.</p><p>With a new release and ten-year anniversary coming up, we’re happy to
celebrate with a tool that extends the reach of declarative and
reproducible deployment!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package manager and
an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects user
freedom</a>.
Guix can be used on top of any system running the Hurd or the Linux
kernel, or it can be used as a standalone operating system distribution
for i686, x86_64, ARMv7, AArch64 and POWER9 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2020/guile-3-and-guix//Guile 3 & GuixLudovic Courtès2020-01-24T15:00:00Z2020-01-24T15:00:00Z Version 3.0 of GNU Guile, an implementation of the Scheme programming
language , was released just last
week .
This is a major milestone for Guile, which gets compiler improvements
and just-in-time (JIT) native code generation, leading to significant
performance improvements over 2.2. It’s also great news for all the
users of Guile, and in particular for Guix! This post discusses what it means for Guix to migrate to Guile 3 and
how that migration is already taking place. Guile in Guix Most users interact with Guix through its command-line interface, and we
work hard to make it…<p>Version 3.0 of GNU Guile, an implementation of the <a href="https://schemers.org">Scheme programming
language</a>, <a href="https://www.gnu.org/software/guile/news/gnu-guile-300-released.html">was released just last
week</a>.
This is a major milestone for Guile, which gets compiler improvements
and just-in-time (JIT) native code generation, leading to significant
performance improvements over 2.2. It’s also great news for all the
users of Guile, and in particular for Guix!</p><p><img src="https://guix.gnu.org/static/blog/img/guile-3.png" alt="Guile 3 logo." /></p><p>This post discusses what it means for Guix to migrate to Guile 3 and
how that migration is already taking place.</p><h1>Guile in Guix</h1><p>Most users interact with Guix through its command-line interface, and we
work hard to make it as approachable as possible. As any user quickly
notices, Guix uses the Scheme programming language uniformly for its
configuration—from
<a href="https://guix.gnu.org/manual/devel/en/html_node/Channels.html">channels</a>
to
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-package.html#profile_002dmanifest">manifests</a>
and <a href="https://guix.gnu.org/manual/devel/en/html_node/Using-the-Configuration-System.html">operating
systems</a>—and
anyone who starts packaging software knows that <a href="https://guix.gnu.org/manual/devel/en/html_node/Defining-Packages.html">package
definitions</a>
are in fact Scheme code as well.</p><p>This is a significant departure from many other, and in particular from
<a href="https://nixos.org/nix/">Nix</a>. While Nix defines several
domain-specific languages (DSLs) for these aspects—the <a href="https://nixos.org/nix/manual/#chap-writing-nix-expressions">Nix
language</a>
but also specific
<a href="https://nixos.org/nix/manual/#sec-conf-file">configuration</a>
<a href="https://nixos.org/nix/manual/#chap-distributed-builds">languages</a>—Guix
chooses Scheme as the single language for all this, <a href="https://hal.inria.fr/hal-00824004/en">together with the
definition of high-level embedded domain-specific languages
(EDSLs)</a>.</p><p>It goes beyond that: in Guix System, all the things traditionally
implemented in C or as a set of Perl or shell scripts are implemented in
Scheme. That includes <a href="https://www.gnu.org/software/shepherd/">the init
system</a>, <a href="https://guix.gnu.org/manual/en/html_node/Build-Systems.html">package
builds</a>,
<a href="https://guix.gnu.org/manual/en/html_node/Initial-RAM-Disk.html">the initial RAM disk
(initrd)</a>,
<a href="https://guix.gnu.org/blog/2016/guixsd-system-tests/">system tests</a>, and
more. Because this leads to several layers of Scheme code, executed at
different points in time, Guix includes a <a href="https://hal.inria.fr/hal-01580582/en"><em>code staging</em>
mechanism</a> built upon the nice
properties of Scheme.</p><p>Why do that? The arguments, right from the start, were twofold: using a
general-purpose language allows us to benefit from its implementation
tooling, and having interfaces for “everything” in Scheme makes it easy
for users to navigate their distro or OS code and to reuse code to build
new features or applications. Guix developers benefit from the ease of
code reuse every day; demonstrative examples include the <a href="https://guix.gnu.org/blog/2017/running-system-services-in-containers/">use of Guix
container facilities in the init
system</a>,
the development of
<a href="https://guix.gnu.org/manual/devel/en/html_node/Development.html">many</a>
<a href="https://guix.gnu.org/manual/devel/en/html_node/Utilities.html">tools</a>
providing facilities around packages, the implementation of additional
<a href="https://emacs-guix.gitlab.io/website/">user</a>
<a href="https://github.com/UMCUGenetics/hpcguix-web/">interfaces</a>, and work on
applications that use Guix as a library such as the <a href="https://www.guixwl.org/">Guix Workflow
Language</a> and
<a href="https://hpc.guix.info/blog/2019/10/towards-reproducible-jupyter-notebooks/">Guix-Jupyter</a>.</p><p>As for the benefits of the host general-purpose language, these are
rather obvious: Guix developers benefit from an expressive language, an
optimizing compiler, a debugger, a powerful <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html">read-eval-print loop
(REPL)</a>,
an <a href="https://nongnu.org/geiser/">interactive development environment</a>,
and all sorts of libraries. Moving to Guile 3 should add to that better
performance, essentially for free. To be comprehensive, Guile 3 may
well come with a set of brand new bugs too, but so far we seem to be
doing OK!</p><h1>Migrating to Guile 3</h1><p>What does it mean for Guix to migrate to Guile 3? We’ve seen above
different ways in which Guix relies on Guile. In short, we can say that
migration is threefold:</p><ol><li>Guix is a distro that ships Guile-related packages. Like any other
distro, it will have to upgrade its <code>guile</code> package to 3.0 and to
ensure packages that depend on it and updated as well.</li><li>Guix is a program written in Guile. As such, we need to make sure
that all its dependencies (half a dozen of Guile libraries) work
with Guile 3 and that Guix itself runs fine with Guile 3.</li><li>Guix ties together operating system components. In particular, the
init system (the Shepherd) and other boot-time facilities will also
migrate.</li></ol><h2>The packages</h2><p>Updating the distro is the boring part, but it’s best to get it right.
Guix makes it possible to have unrelated versions of variants of
packages in different environments or different profiles, which is very
nice. We’ll have performed a smooth transition if users and tools see
that the packages named <code>guile</code> and <code>guile-ssh</code> (say) transparently move
from Guile 2.2 to 3.0, <em>in lockstep</em>.</p><p>Put differently, most of the upgrade work upon a programming language
version bump deals with conventions, and in particular package names.
Currently, <code>guile</code> corresponds to the 2.2 stable series and all the
<code>guile-*</code> packages are built against it. In the meantime, the package
for Guile 3 is named <code>guile-next</code> and packages built against it are
called <code>guile3.0-*</code>. Over the last few weeks we created <code>guile3.0-</code>
variants for most Guile packages, something that’s <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=89a99d53f56c7c383659d821c28286b6d71e458d">easily achieved with
Guix</a>.</p><p>The big switch will consist in renaming all current <code>guile-*</code> packages
to <code>guile2.2-*</code> packages, for use with the legacy 2.2 series, and
renaming all the <code>guile3.0-*</code> packages to <code>guile-*</code>. We will switch
soon, but before getting there, we’re making sure important packages are
available for 3.0.</p><h2>Guix-the-program</h2><p>A more interesting part is “porting” Guix itself from Guile 2.2 to
Guile 3. It seems that developers have become wary of 2-to-3
transitions for programming languages. Fear not! Switching from
Guile 2 to Guile 3 turned out to be an easy task. In fact, very little
changed in the language itself; what did change—e.g., semantics on fine
points of the module system, support for structured exceptions—is either
optional or backwards-compatible.</p><p>As Guile 2.9 pre-releases trickled in, we started testing all the Guile
libraries Guix relies on against 2.9. For the vast majority of them,
all we had to do was to <a href="https://gitlab.com/gnutls/gnutls/commit/763e31d351933222281bf9c11ff0bddb89bb701d">update their <code>configure.ac</code> to allow builds
with
3.0</a>.</p><p>Guix itself was a bit more work, mostly because it’s a rather large code
base with a picky test suite. The bit that required most work has to do
with the introduction of <a href="https://www.gnu.org/software/guile/manual/html_node/Declarative-Modules.html"><em>declarative
modules</em></a>,
an optional semantic change in modules to support more compiler
optimizations. We had several <a href="https://en.wikipedia.org/wiki/White-box_testing">“white-box
tests”</a> where tests
would happily peek at private module bindings through <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html#index-_0040_0040">the magical-evil
<code>@@</code>
operator</a>.
Because we chose to enable declarative modules, we also had to adjust
our tests to no longer do that. And well, that’s about it!</p><p>At that point, we were able to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=da7651806102d637253cb9f5677b96d6a178fc05">create a <code>guile3.0-guix</code> package
variant</a>,
primarily for testing purposes. Soon after, we told <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>
to <a href="https://git.savannah.gnu.org/cgit/guix.git/commit/?id=8234fe653e61d0090138cbd4c48d877568355439">build Guix with 3.0 instead of
2.2</a>.
Thus, Guix users who upgrade will transparently find themselves running
Guix on Guile 3.0.</p><p>The main benefit is improved performance. Guile 3 is known to be <a href="https://www.gnu.org/software/guile/news/gnu-guile-300-released.html">up to
32 times faster than
Guile 2.2</a>
on some micro-benchmarks. Assessing the performance gains on a
“real-world” application like Guix is the real test. What would be a
relevant benchmark? At its core, Guix is essentially a compiler from
high-level descriptions of packages, operating systems, and the like, to
low-level build instructions
(<a href="https://guix.gnu.org/manual/devel/en/html_node/Derivations.html"><em>derivations</em></a>).
Thus, a good benchmark is a command that exercises little more than this
compilation step:</p><pre><code>guix build libreoffice ghc-pandoc guix --dry-run --derivation</code></pre><p>or:</p><pre><code>guix system build config.scm --dry-run --derivation</code></pre><p>On x86_64, the <code>guix build</code> command above on Guile 3 is 7% faster than
on Guile 2.2, and <code>guix system build</code>, which is more
computing-intensive, is 10% faster (heap usage is ~5% higher). This is
lower than the skyrocketing speedups observed on some microbenchmarks,
but it’s probably no surprise: these <code>guix</code> commands are short-lived (a
couple of seconds) and they’re rather I/O- and GC-intensive—something
JIT compilation cannot help with.</p><p>On 32-bit ARM, we temporarily disabled JIT <a href="https://issues.guix.gnu.org/issue/39208">due to a
bug</a>; there we observe a slight
<em>slowdown</em> compared to 2.2. This can be explained by the fact that
<a href="https://wingolog.org/archives/2018/01/17/instruction-explosion-in-guile">virtual machine (VM) instructions in 3.0 are lower-level than in
2.2</a>
and will hopefully be more than compensated for when JIT is re-enabled.</p><h2>Gluing it all together</h2><p>The last part of the Guile 3 migration has to do with how Guix, and in
particular Guix System, glues things together. As explained above, Guix
manipulates several stages of Scheme code that will run a different
points in time.</p><p>Firstly, the code that runs package builds, such as <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gnu-build-system.scm">the one that runs
<code>./configure && make && make install</code></a>,
is Guile code. Currently that code runs on Guile 2.2, but on the next
major rebuild-the-world upgrade, we will switch to Guile 3.</p><p>Additionally, Guix produces Scheme code consumed by <a href="https://www.gnu.org/software/shepherd">the
Shepherd</a>, by
<a href="https://www.gnu.org/software/mcron">GNU mcron</a>, and for <a href="https://guix.gnu.org/manual/en/html_node/Guided-Graphical-Installation.html">the graphical
installer</a>.
These will soon switch to Guile 3 as well. This kind of change is made
easy by the fact that both the package definitions and the staged code
that depends on those packages live in the same repository.</p><h1>Long live Guile 3!</h1><p>Migrating Guix to Guile 3 is a bit of work because of the many ways Guix
interacts with Guile and because of the sheer size of the code base.
For a “2-to-3” transition though, it was easy. And fundamentally, it
remains a cheap transition compared to what it brings: better
performance and new features. That’s another benefit of using a
general-purpose language.</p><p>Thumbs up to everyone involved in its development, and long live
Guile 3!</p><h4>About GNU Guix</h4><p><a href="https://guix.gnu.org">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial//Managing Servers with GNU Guix: A TutorialJakob L. Kreuze2019-11-04T15:00:00Z2019-11-04T15:00:00Z The outcome of this year's
GSoC
is a Guile-based programming interface named guix deploy for
automatically creating, upgrading, and changing the configurations of
machines running the Guix System. The tool is comparable to
Ansible or
NixOps , but makes use of the system
configuration facilities provided by Guix. A post from earlier this
summer
described an early version of the programming interface, but we're
already a few months into autumn, so it's time for guide on how you
can use guix deploy in production. Simple case: managing a home server If the machine you…<p>The outcome of this year's
<a href="https://summerofcode.withgoogle.com/projects/#5232565294727168">GSoC</a>
is a Guile-based programming interface named <code>guix deploy</code> for
automatically creating, upgrading, and changing the configurations of
machines running the Guix System. The tool is comparable to
<a href="https://www.ansible.com/">Ansible</a> or
<a href="https://nixos.org/nixops/">NixOps</a>, but makes use of the system
configuration facilities provided by Guix. A <a href="http://guix.gnu.org/blog/2019/towards-guix-for-devops/">post from earlier this
summer</a>
described an early version of the programming interface, but we're
already a few months into autumn, so it's time for guide on how you
can use <code>guix deploy</code> in production.</p><h4>Simple case: managing a home server</h4><p>If the machine you need to manage is already running the Guix System,
it shouldn't be too hard to incorporate <code>guix deploy</code> into your
workflow. All that's needed is the <code><operating-system></code> declaration
you've been passing to <code>guix system reconfigure</code> and some information
about the machine (specifically its IP address and architecture). The
<code>guix deploy</code> command is invoked with the filename of a "deployment
specification" as an argument, whose contents should look something
like this:</p><pre><code class="language-scheme">;; Module imports
(use-modules (gnu) (guix))
(use-service-modules networking ssh)
(use-package-modules bootloaders)
;; Operating system description
(define os
(operating-system
(locale "en_US.utf8")
(timezone "America/New_York")
(keyboard-layout (keyboard-layout "us" "altgr-intl"))
(bootloader (bootloader-configuration
(bootloader grub-bootloader)
(target "/dev/sda")
(keyboard-layout keyboard-layout)))
(file-systems (cons* (file-system
(mount-point "/")
(device "/dev/sda1")
(type "ext4"))
%base-file-systems))
(host-name "alyssas-home-server")
(users (cons* (user-account
(name "alyssa")
(comment "Alyssa")
(group "users")
(home-directory "/home/alyssa")
(supplementary-groups
'("wheel" "netdev" "audio" "video")))
%base-user-accounts))
(sudoers-file (plain-file "sudoers" "\
root ALL=(ALL) ALL
%wheel ALL=NOPASSWD: ALL\n"))
(services (append
(list (service openssh-service-type
(openssh-configuration
(permit-root-login #t)))
(service dhcp-client-service-type))
%base-services))))
;; List of machines to deploy
(list (machine
(operating-system os)
(environment managed-host-environment-type)
(configuration (machine-ssh-configuration
(host-name "alyssa-p-hacker.tld")
(system "i686-linux")
(identity "/path/to/ssh-key")))))</code></pre><p>Even if Scheme isn't your forté, parts of this should look familiar if
you've used Guix before. The "operating system description" section
in particular is something you might use with <code>guix system reconfigure</code>. What's new is the last part: We construct a <code>list</code>
containing one <code>machine</code> of the <code>managed-host-environment-type</code>, for
which we've specified that <code>os</code> is the <code>operating-system</code>
declaration that we want to install on it, and that we can connect to
it using the parameters specified by the <code>machine-ssh-configuration</code>.</p><p>Let's take a step back for a moment and explain what a <code>machine</code> is.
<code>guix deploy</code> aims to support a number of different use-cases, which
we abstract as "environment types". We'll see other environment types
later in this article, but the general idea is that these environments
specify how resources should be "provisioned" or created. For
example, an environment type designed for working with a Virtual
Private Server (VPS) provider might make calls the provider's API to
request a virtual machine before installing the <code>machine</code>'s
<code>operating-system</code> declaration on it.</p><p>The environment type used in this example,
<code>managed-host-environment-type</code>, is intended for machines that are
already running Guix System and are accessible over SSH. It expects
that the <code>configuration</code> field of the <code>machine</code> be an instance of
<code>machine-ssh-configuration</code>, whose available fields are described in
the
<a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-deploy.html">manual</a>.
This gives <code>guix deploy</code> the information it needs to connect to the
machine's SSH daemon.</p><p>Running <code>guix deploy</code> with this file would build the "operating system
closure" of <code>os</code> -- a bundle of the packages, configuration
files, and other dependencies necessary to realize that configuration
-- for the architecture specified by <code>system</code> (in this case
<code>i686-linux</code>), send it over SSH to <code>alyssa-p-hacker.tld</code>, and then
remotely "activate" the configuration by creating a new system
generation and upgrading running services. Sweet! Upgrading our
single server setup has been reduced to an endeavour involving just
over a dozen keystrokes.</p><h4>More advanced case: managing a virtual private server deployment</h4><p>One server not cutting it for you? <code>guix deploy</code> can still help.
Suppose we run a web service that we'd like to split up across
multiple machines for performance reasons.</p><pre><code class="language-scheme">(define %forum-server-count 4)
(define (forum-server n)
(operating-system
(host-name (format #f "forum-server-~a" n))
...
(services (append (list (service httpd-service-type
(httpd-configuration
...)))
%base-services))))
(map (lambda (n)
(machine
(system (forum-server n))
(environment digital-ocean-environment-type)
(configuration (digital-ocean-configuration
(region "nyc3")
(size "s-1vcpu-1gb")
(enable-ipv6 #t)))))
(iota %forum-server-count))</code></pre><p>This example isn't as concrete as the first one; I'm intentionally
omitting parts of the configuration to make the example clearer.
Here, we automate the creation of <code>%forum-server-count</code> Digital Ocean
"droplets" in their <code>NYC3</code> region by creating a list of 4 machines.</p><p>Assuming that the environment variable <code>GUIX_DIGITAL_OCEAN_TOKEN</code> is
properly set, running <code>guix deploy</code> with this file will do much of the
same as the previous example. The difference is that four virtual
machines will be automatically created on Digital Ocean.</p><p>One important thing to note about the <code>digital-ocean-environment-type</code>
is that, currently, it <em>does not</em> automatically clean up unused
virtual machines. If you change something in the deployment
specification and run <code>guix deploy</code> again, the virtual machines from
the previous deployment will remain until you destroy them yourself.</p><h4>A quick peek into the internals of <code>digital-ocean-environment-type</code></h4><p>It would be an overstatement to say that the process of implementing a
new environment type is easy, but a fair amount of the work has
already been done for you. We'll use the definition of
<code>digital-ocean-environment-type</code> as an example.</p><pre><code class="language-scheme">(define digital-ocean-environment-type
(environment-type
(machine-remote-eval digital-ocean-remote-eval)
(deploy-machine deploy-digital-ocean)
(roll-back-machine roll-back-digital-ocean)
(name 'digital-ocean-environment-type)
(description "Provisioning of \"droplets\": virtual machines
provided by the Digital Ocean virtual private server (VPS) service.")))</code></pre><p>The <code>environment-type</code> record specifies a small amount of metadata
(<code>name</code> and <code>description</code>), as well as the names of three procedures:
one for remotely evaluating a
<a href="http://guix.gnu.org/manual/en/html_node/G_002dExpressions.html#G_002dExpressions">G-Expression</a>
on the host (<code>machine-remote-eval</code>), one for deploying an
<code>operating-system</code> declaration to the host, and one for rolling the
host back one generation.</p><p>This might sound like a lot, but the pattern for these high-level
environment types is to somehow obtain a machine running Guix System,
set up an SSH daemon, and then delegate to
<code>managed-host-environment-type</code>. <code>digital-ocean-remote-eval</code> is a
pretty good example of this:</p><pre><code class="language-scheme">(define (digital-ocean-remote-eval target exp)
"Internal implementation of 'machine-remote-eval' for MACHINE instances with
an environment type of 'digital-ocean-environment-type'."
(mlet* %store-monad ((name (droplet-name target))
(network -> (droplet-public-ipv4-network name))
(address -> (hash-ref network "ip_address"))
(ssh-key -> (digital-ocean-configuration-ssh-key
(machine-configuration target)))
(delegate -> (machine
(inherit target)
(environment managed-host-environment-type)
(configuration
(machine-ssh-configuration
(host-name address)
(identity ssh-key)
(system "x86_64-linux"))))))
(machine-remote-eval delegate exp)))</code></pre><p>As you can see, you could reasonably go about implementing an
environment type without ever having to learn what a G-Expression is.
Here, <code>droplet-name</code> derives the name of the droplet from the
machine's <code>operating-system</code> declaration, the information necessary to
connect to the droplet is found using <code>droplet-public-ipv4-network</code>,
and that's used to create machine of <code>managed-host-environment-type</code>.</p><h4>In conclusion</h4><p>I sincerely hope that <code>guix deploy</code> proves to be a useful to anyone
dealing with system administration or software development.
Transactional upgrades should provide peace of mind to those managing
servers (it's worth noting that few existing tools are capable of
recovering from failed deployments), and I believe that
procedurally-generated deployment configurations could very well be
the future of distribution for software such as web services: Imagine
if setting up a <a href="https://joinmastodon.org">Mastodon</a> instance were as
easy as downloading a Scheme file and handing it off to <code>guix deploy</code>.
The ease of writing code that generates code isn't the only benefit of
using Guile for something like this. Guile is a general-purpose
programming language, so more advanced tooling can reasonably be built
atop <code>guix deploy</code>. A GTK or Emacs DevOps interface, perhaps? (If
that idea sounds outlandish, consider that the latter has <a href="https://emacs-guix.gitlab.io/website/">already
happened</a> for the package
management interface.)</p><p>It's been a great summer working alongside everyone in the Guix
community. <code>guix deploy</code> is brand new (and a little unstable!), but
we've had enthusiastic adoption by several on the mailing lists who
were quick to report any issues they found. I'd like to thank
everyone on the <code>#guix</code> IRC channel and the mailing lists who got me
up to speed with the code, answered my questions, gave feedback when I
submitted my patches, and put <code>guix deploy</code> under the pressure of use
in production. And of course, I want to thank my mentors Christopher
Lemmer Webber and David Thompson. I had to make some hard design
decisions, but this was made easier thanks to the guidance of two
experienced Guix veterans.</p><p>Oh, and this isn't a goodbye. I really feel I've found my place as a
Guix contributor, and I can't wait to see what the future will bring
for <code>guix deploy</code>. Catch ya on the mailing lists!</p><h4>Editor's note</h4><p>Thank you for all of your hard work, Jakob!</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2019/towards-guix-for-devops//Towards Guix for DevOpsJakob L. Kreuze2019-07-12T19:00:00Z2019-07-12T19:00:00Z Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to
Guix. Since May, I've been working on a DevOps automation tool for the Guix
System, which we've been calling guix deploy . The idea for a Guix DevOps tool has been making rounds on the mailing lists for
some time now. Years, in fact; Dave Thompson and Chris Webber put together a
proof-of-concept for it way back in 2015. Thus, we've had plenty of time to gaze
upon the existing tools for this sort of thing -- Ansible , NixOps -- and
fantasize about a…<p>Hey, there! I'm Jakob, a Google Summer of Code intern and new contributor to
Guix. Since May, I've been working on a DevOps automation tool for the Guix
System, which we've been calling <code>guix deploy</code>.</p><p>The idea for a Guix DevOps tool has been making rounds on the mailing lists for
some time now. Years, in fact; Dave Thompson and Chris Webber put together a
proof-of-concept for it way back in 2015. Thus, we've had plenty of time to gaze
upon the existing tools for this sort of thing -- <a href="https://www.ansible.com/">Ansible</a>, <a href="https://nixos.org/nixops/">NixOps</a> -- and
fantasize about a similar tool, albeit with the expressive power of Guile scheme
and the wonderful system configuration facilities of Guix. And now, those
fantasies are becoming a reality.</p><p>"DevOps" is a term that might be unfamiliar to a fair number of Guix users. I'll
spare you the detour to Wikipedia and give a brief explanation of what <code>guix deploy</code> does.</p><p>Imagine that you've spent the afternoon playing around with Guile's <code>(web)</code>
module, developing software for a web forum. Awesome! But a web forum with no
users is pretty boring, so you decide to shell out a couple bucks for a virtual
private server to run your web forum. You feel that Wildebeest admirers on the
internet deserve a platform of their own for discussion, and decide to dedicate
the forum to that.</p><p>As it turns out, <em>C. gnou</em> is a more popular topic than you ever would have
imagined. Your web forum soon grows in size -- attracting hundreds of thousands
of simultaneous users. Despite Guile's impressive performance characteristics,
one lowly virtual machine is too feeble to support such a large population of
Wildebeest fanatics. So you decide to use Apache as a load-balancer, and shell
out a couple more bucks for a couple more virtual private servers. Now you've
got a problem on your hands; you're the proud owner of five or so virtual
machines, and you need to make sure they're all running the most recent version
of either your web forum software or Apache.</p><p>This is where <code>guix deploy</code> comes into play. Just as you'd use an
<code>operating-system</code> declaration to configure services and user accounts on a
computer running the Guix System, you can now use that same <code>operating-system</code>
declaration to remotely manage any number of machines. A "deployment" managing
your Wildebeest fan site setup might look something like this:</p><pre><code class="language-scheme">...
;; Service for our hypothetical guile web forum application.
(define guile-forum-service-type
(service-type (name 'guile-forum)
(extensions
(list (service-extension shepherd-root-service-type
guile-forum-shepherd-service)
(service-extension account-service-type
(const %guile-forum-accounts))))
(default-value (guile-forum-configuration))
(description "A web forum written in GNU Guile.")))
...
(define %forum-server-count 4)
(define (forum-server n)
(operating-system
(host-name (format #f "forum-server-~a" n))
...
(services
(append (list (service guile-forum-service-type
(guile-forum-configuration
"GNU Fan Forum!")))
%base-services))))
(define load-balancer-server
(operating-system
(host-name "load-balancer-server"
...
(services
(append (list (service httpd-service-type
(httpd-configuration
...)))
%base-services)))))
;; One machine running our load balancer.
(cons (machine
(system load-balancer-server)
(environment manged-host-environment-type)
(configuration (machine-ssh-configuration
...)))
;; And a couple running our forum software!
(let loop ((n 1)
(servers '()))
(if (> n %forum-server-count)
servers
(loop (1+ n)
(cons (machine
(system (forum-server n))
(environment manged-host-environment-type)
(configuration (machine-ssh-configuration
...)))
servers)))))</code></pre><p>The take-away from that example is that there's a new <code>machine</code> type atop the
good ol' <code>operating-system</code> type, specifying how the machine should be
<em>provisioned</em>. The version of <code>guix deploy</code> that's currently on the master
branch only supports <code>managed-host-environment-type</code>, which is used for machines
that are already up and running the Guix System. Provisioning, in that sense,
only really involves opening an SSH connection to the host. But I'm sure you can
imagine a <code>linode-environment-type</code> which automatically sets up a virtual
private server through Linode, or a <code>libvirt-environment-type</code> that spins up a
virtual machine for running your services. Those types are what I'll be working
on in the coming months, in addition to cleaning up the code that's there now.</p><p>And yes, you did read that right. <code>guix deploy</code> is on the Guix master branch
right now! In fact, we've already done a successful deployment right here on
<a href="http://ci.guix.gnu.org/">ci.guix.gnu.org</a>. So, if this sounds as though it'd be up your alley, run <code>guix pull</code>, crack open the <a href="https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-deploy.html">manual</a>, and let us know how it goes!</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager and an advanced distribution of the GNU system that <a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects
user
freedom</a>.
Guix can be used on top of any system running the kernel Linux, or it
can be used as a standalone operating system distribution for i686,
x86_64, ARMv7, and AArch64 machines.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. When used as a standalone
GNU/Linux distribution, Guix offers a declarative, stateless approach to
operating system configuration management. Guix is highly customizable
and hackable through <a href="https://www.gnu.org/software/guile">Guile</a>
programming interfaces and extensions to the
<a href="http://schemers.org">Scheme</a> language.</p>https://guix.gnu.org/blog/2018/a-packaging-tutorial-for-guix//A packaging tutorial for GuixPierre Neidhardt2018-10-10T16:00:00Z2018-10-10T16:00:00Z Introduction GNU Guix stands out as the hackable package manager, mostly because it uses
GNU Guile , a powerful high-level programming language, one of the Scheme
dialects from the Lisp family . Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simple languages. Use functions, structures, macros and all of Scheme expressiveness for your
package definitions. Inheritance makes it easy to customize a package by inheriting from it and
modifying only what is needed.…<h1>Introduction</h1><p>GNU Guix stands out as the <em>hackable</em> package manager, mostly because it uses
<a href="https://www.gnu.org/software/guile/">GNU Guile</a>, a powerful high-level programming language, one of the <a href="https://en.wikipedia.org/wiki/Scheme_(programming_language)">Scheme</a>
dialects from the <a href="https://en.wikipedia.org/wiki/Lisp_(programming_language)">Lisp family</a>.</p><p>Package definitions are also written in Scheme, which empowers Guix in some very
unique ways, unlike most other package managers that use shell scripts or
simple languages.</p><ul><li><p>Use functions, structures, macros and all of Scheme expressiveness for your
package definitions.</p></li><li><p>Inheritance makes it easy to customize a package by inheriting from it and
modifying only what is needed.</p></li><li><p>Batch processing: the whole package collection can be parsed, filtered and
processed. Building a headless server with all graphical interfaces stripped
out? It's possible. Want to rebuild everything from source using specific
compiler optimization flags? Pass the <code>#:make-flags "..."</code> argument to the
list of packages. It wouldn't be a stretch to think <a href="https://wiki.gentoo.org/wiki/USE_flag">Gentoo USE flags</a> here,
but this goes even further: the changes don't have to be thought out
beforehand by the packager, they can be <em>programmed</em> by the user!</p></li></ul><p>The following tutorial covers all the basics around package creation with Guix.
It does not assume much knowledge of the Guix system nor of the Lisp language.
The reader is only expected to be familiar with the command line and to have some
basic programming knowledge.</p><blockquote><p><strong>Update</strong>: Check out <a href="https://guix.gnu.org/cookbook/en/html_node/Packaging-Tutorial.html">the
Cookbook</a>
for an up-to-date and translated version of this tutorial!</p></blockquote><h1>A "Hello World" package</h1><p>The <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">“Defining Packages” section of the manual</a> introduces the basics of Guix
packaging. In the following section, we will partly go over those basics again.</p><p><code>GNU hello</code> is a dummy project that serves as an idiomatic example for
packaging. It uses the GNU build system (<code>./configure && make && make install</code>).
Guix already provides a package definition which is a perfect example to start
with. You can look up its declaration with <code>guix edit hello</code> from the
command line. Let's see how it looks:</p><pre><code class="language-scheme">(define-public hello
(package
(name "hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, GNU world: An example GNU package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+)))</code></pre><p>As you can see, most of it is rather straightforward. But let's review the
fields together:</p><ul><li><strong>name:</strong> The project name. Using Scheme conventions, we prefer to keep it
lower case, without underscore and using dash-separated words.</li><li><p><strong>source:</strong> This field contains a description of the source code origin. The
<code>origin</code> record contains these fields:</p><ol><li>The method, here <code>url-fetch</code> to download via HTTP/FTP, but other methods
exist, such as <code>git-fetch</code> for Git repositories.</li><li>The URI, which is typically some <code>https://</code> location for <code>url-fetch</code>. Here
the special <code>mirror://gnu</code> refers to a set of well known locations, all of
which can be used by Guix to fetch the source, should some of them fail.</li><li>The <code>sha256</code> checksum of the requested file. This is essential to ensure
the source is not corrupted. Note that Guix works with base32 strings,
hence the call to the <code>base32</code> function.</li></ol></li><li><strong>build-system:</strong> This is where the power of abstraction provided by the Scheme
language really shines: in this case, the <code>gnu-build-system</code>
abstracts away the famous <code>./configure && make && make install</code> shell invocations. Other build systems include the
<code>trivial-build-system</code> which does not do anything and requires
from the packager to program all the build steps, the
<code>python-build-system</code>, the <code>emacs-build-system</code>, <a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html">and many
more</a>.</li><li><strong>synopsis:</strong> It should be a concise summary of what the package
does. For many packages a tagline from the
project's home page can be used as the synopsis.</li><li><strong>description:</strong> Same as for the synopsis, it's fine to re-use the project
description from the homepage. Note that Guix uses Texinfo
syntax.</li><li><strong>home-page:</strong> Use HTTPS if available.</li><li><strong>license:</strong> See <code>guix/licenses.scm</code> in the project source for a full list.</li></ul><p>Time to build our first package! Nothing fancy here for now: we will stick to a
dummy "my-hello", a copy of the above declaration.</p><p>As with the ritualistic "Hello World" taught with most programming languages,
this will possibly be the most "manual" approach. We will work out an ideal
setup later; for now we will go the simplest route.</p><p>Save the following to a file <code>my-hello.scm</code>.</p><pre><code class="language-scheme">(use-modules (guix packages)
(guix download)
(guix build-system gnu)
(guix licenses))
(package
(name "my-hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, Guix world: An example custom Guix package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+))</code></pre><p>We will explain the extra code in a moment.</p><p>Feel free to play with the different values of the various fields. If you
change the source, you'll need to update the checksum. Indeed, Guix refuses to
build anything if the given checksum does not match the computed checksum of the
source code. To obtain the correct checksum of the package declaration, we
need to download the source, compute the sha256 checksum and convert it to
base32.</p><p>Thankfully, Guix can automate this task for us; all we need is to provide the
URI:</p><pre><code class="language-sh">$ guix download mirror://gnu/hello/hello-2.10.tar.gz
Starting download of /tmp/guix-file.JLYgL7
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz...
following redirection to `https://mirror.ibcp.fr/pub/gnu/hello/hello-2.10.tar.gz'...
…10.tar.gz 709KiB 2.5MiB/s 00:00 [##################] 100.0%
/gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i</code></pre><p>In this specific case the output tells us which mirror was chosen.
If the result of the above command is not the same as in the above snippet,
update your <code>my-hello</code> declaration accordingly.</p><p>Note that GNU package tarballs come with an OpenPGP signature, so you
should definitely check the signature of this tarball with <code>gpg</code> to
authenticate it before going further:</p><pre><code class="language-sh">$ guix download mirror://gnu/hello/hello-2.10.tar.gz.sig
Starting download of /tmp/guix-file.03tFfb
From https://ftpmirror.gnu.org/gnu/hello/hello-2.10.tar.gz.sig...
following redirection to `https://ftp.igh.cnrs.fr/pub/gnu/hello/hello-2.10.tar.gz.sig'...
….tar.gz.sig 819B 1.2MiB/s 00:00 [##################] 100.0%
/gnu/store/rzs8wba9ka7grrmgcpfyxvs58mly0sx6-hello-2.10.tar.gz.sig
0q0v86n3y38z17rl146gdakw9xc4mcscpk8dscs412j22glrv9jf
$ gpg --verify /gnu/store/rzs8wba9ka7grrmgcpfyxvs58mly0sx6-hello-2.10.tar.gz.sig /gnu/store/hbdalsf5lpf01x4dcknwx6xbn6n5km6k-hello-2.10.tar.gz
gpg: Signature made Sun 16 Nov 2014 01:08:37 PM CET
gpg: using RSA key A9553245FDE9B739
gpg: Good signature from "Sami Kerola <kerolasa@iki.fi>" [unknown]
gpg: aka "Sami Kerola (http://www.iki.fi/kerolasa/) <kerolasa@iki.fi>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 8ED3 96E3 7E38 D471 A005 30D3 A955 3245 FDE9 B739</code></pre><p>You can then happily run</p><pre><code class="language-sh">$ guix package --install-from-file=my-hello.scm</code></pre><p>You should now have <code>my-hello</code> in your profile!</p><pre><code class="language-sh">$ guix package --list-installed=my-hello
my-hello 2.10 out
/gnu/store/f1db2mfm8syb8qvc357c53slbvf1g9m9-my-hello-2.10</code></pre><p>We've gone as far as we could without any knowledge of Scheme. Now is the right
time to introduce the minimum we need from the language before we can proceed.</p><h1>A Scheme crash-course</h1><p>As we've seen above, basic packages don't require much Scheme knowledge, if
at all. But as you progress and your desire to write more and more complex
packages grows, it will become both necessary and empowering to hone your Lisper
skills.</p><p>Since an extensive Lisp course is very much out of the scope of this tutorial,
we will only cover some basics here.</p><blockquote><p><strong>Update</strong>: Check out <a href="https://guix.gnu.org/cookbook/en/html_node/A-Scheme-Crash-Course.html">the
Cookbook</a>
for an up-to-date and translated version of this introduction to Scheme.</p></blockquote><p>Guix uses the Guile implementation of Scheme. To start playing with the
language, install it with <code>guix package --install guile</code> and start a <a href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a> by
running <code>guile</code> from the command line.</p><p>Alternatively you can also run <code>guix environment --ad-hoc guile -- guile</code> if
you'd rather not have Guile installed in your user profile.</p><p>In the following examples we use the <code>></code> symbol to denote the REPL prompt, that
is, the line reserved for user input. See <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Interactively.html">the Guile manual</a> for more details on
the REPL.</p><ul><li><p>Scheme syntax boils down to a tree of expressions (or <em>s-expression</em> in Lisp
lingo). An expression can be a literal such as numbers and strings, or a
compound which is a parenthesized list of compounds and literals. <code>#t</code> and
<code>#f</code> stand for the booleans "true" and "false", respectively.</p><p>Examples of valid expressions:</p><pre><code class="language-scheme">> "Hello World!"
"Hello World!"
> 17
17
> (display (string-append "Hello " "Guix" "\n"))
"Hello Guix!"</code></pre></li><li><p>This last example is a function call embedded in another function call. When
a parenthesized expression is evaluated, the first term is the function and
the rest are the arguments passed to the function. Every function returns the
last evaluated expression as value.</p></li><li><p>Anonymous functions are declared with the <code>lambda</code> term:</p><pre><code class="language-scheme">> (lambda (x) (* x x))
#<procedure 120e348 at <unknown port>:24:0 (x)></code></pre><p>The above lambda returns the square of its argument. Since everything is an
expression, the <code>lambda</code> expression returns an anonymous function, which can
in turn be applied to an argument:</p><pre><code class="language-scheme">> ((lambda (x) (* x x)) 3)
9</code></pre></li><li><p>Anything can be assigned a global name with <code>define</code>:</p><pre><code class="language-scheme">> (define a 3)
> (define square (lambda (x) (* x x)))
> (square a)
9</code></pre></li><li><p>Procedures can be defined more concisely with the following syntax:</p><pre><code class="language-scheme">(define (square x) (* x x))</code></pre></li><li><p>A list structure can be created with the <code>list</code> procedure:</p><pre><code class="language-scheme">> (list 2 a 5 7)
(2 3 5 7)</code></pre></li><li><p>The <em>quote</em> disables evaluation of a parenthesized expression: the first term
is not called over the other terms. Thus it effectively returns a list of
terms.</p><pre><code class="language-scheme">> '(display (string-append "Hello " "Guix" "\n"))
(display (string-append "Hello " "Guix" "\n"))
> '(2 a 5 7)
(2 a 5 7)</code></pre></li><li><p>The <em>quasiquote</em> disables evaluation of a parenthesized expression until a
comma re-enables it. Thus it provides us with fine-grained control over what
is evaluated and what is not.</p><pre><code class="language-scheme">> `(2 a 5 7 (2 ,a 5 ,(+ a 4)))
(2 a 5 7 (2 3 5 7))</code></pre><p>Note that the above result is a list of mixed elements: numbers, symbols (here
<code>a</code>) and the last element is a list itself.</p></li><li><p>Multiple variables can be named locally with <code>let</code>:</p><pre><code class="language-scheme">> (define x 10)
> (let ((x 2)
(y 3))
(list x y))
(2 3)
> x
10
> y
ERROR: In procedure module-lookup: Unbound variable: y</code></pre><p>Use <code>let*</code> to allow later variable declarations to refer to earlier
definitions.</p><pre><code class="language-scheme">> (let* ((x 2)
(y (* x 3)))
(list x y))
(2 6)</code></pre></li><li><p>The keyword syntax is <code>#:</code>, it is used to create unique identifiers. See also
the <a href="https://www.gnu.org/software/guile/manual/html_node/Keywords.html">Keywords section in the Guile manual</a>.</p></li><li><p>The percentage <code>%</code> is typically used for read-only global variables in the
build stage. Note that it is merely a convention, like <code>_</code> in C. Scheme Lisp
treats <code>%</code> exactly the same as any other letter.</p></li><li><p>Modules are created with <code>define-module</code>. For instance</p><pre><code class="language-scheme">(define-module (guix build-system ruby)
#:use-module (guix store)
#:export (ruby-build
ruby-build-system))</code></pre><p>defines the module <code>ruby</code> which must be located in
<code>guix/build-system/ruby.scm</code> somewhere in <code>GUILE_LOAD_PATH</code>. It depends on
the <code>(guix store)</code> module and it exports two symbols, <code>ruby-build</code> and
<code>ruby-build-system</code>.</p></li></ul><p>For a more detailed introduction, check out <a href="http://www.troubleshooters.com/codecorn/scheme_guile/hello.htm">Scheme at a Glance</a>, by Steve Litt.</p><p>One of the reference Scheme books is the seminal <em>Structure and Interpretation
of Computer Programs</em>, by Harold Abelson and Gerald Jay Sussman, with Julie
Sussman. You'll find a free copy <a href="https://mitpress.mit.edu/sites/default/files/sicp/index.html">online</a>, together with <a href="https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/">videos of the lectures
by the authors</a>. The book is available in Texinfo format as the <code>sicp</code> Guix
package. Go ahead, run <code>guix package --install sicp</code> and start reading with
<code>info sicp</code> (or with the Emacs Info reader). An unofficial ebook <a href="https://sarabander.github.io/sicp/">is also
available</a>.</p><p>You'll find more books, tutorials and other resources at <a href="https://schemers.org/">https://schemers.org/</a>.</p><h1>Setup</h1><p>Now that we know some Scheme basics we can detail the different possible setups
for working on Guix packages.</p><p>There are several ways to set up a Guix packaging environment.</p><p>We recommend you work directly on the Guix source checkout since it makes it
easier for everyone to contribute to the project.</p><p>But first, let's look at other possibilities.</p><h3>Local file</h3><p>This is what we previously did with <code>my-hello</code>. With the Scheme basics we've
covered, we are now able to explain the leading chunks. As stated in <code>guix package --help</code>:</p><pre><code class="language-sh">-f, --install-from-file=FILE
install the package that the code within FILE
evaluates to</code></pre><p>Thus the last expression <em>must</em> return a package, which is the case in our
earlier example.</p><p>The <code>use-modules</code> expression tells which of the modules we need in the file.
Modules are a collection of values and procedures. They are commonly called
"libraries" or "packages" in other programming languages.</p><h3>GUIX_PACKAGE_PATH</h3><p><em>Note: Starting from Guix 0.16, the more flexible Guix "channels" are the
preferred way and supersede <code>GUIX_PACKAGE_PATH</code>. See below.</em></p><p>It can be tedious to specify the file from the command line instead of simply
calling <code>guix package --install my-hello</code> as you would do with the official
packages.</p><p>Guix makes it possible to streamline the process by adding as many "package
declaration paths" as you want.</p><p>Create a directory, say <code>~./guix-packages</code> and add it to the <code>GUIX_PACKAGE_PATH</code>
environment variable:</p><pre><code class="language-sh">$ mkdir ~/guix-packages
$ export GUIX_PACKAGE_PATH=~/guix-packages</code></pre><p>To add several directories, separate them with a colon (<code>:</code>).</p><p>Our previous <code>my-hello</code> needs some adjustments though:</p><pre><code class="language-scheme">(define-module (my-hello)
#:use-module (guix licenses)
#:use-module (guix packages)
#:use-module (guix build-system gnu)
#:use-module (guix download))
(define-public my-hello
(package
(name "my-hello")
(version "2.10")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnu/hello/hello-" version
".tar.gz"))
(sha256
(base32
"0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"))))
(build-system gnu-build-system)
(synopsis "Hello, Guix world: An example custom Guix package")
(description
"GNU Hello prints the message \"Hello, world!\" and then exits. It
serves as an example of standard GNU coding practices. As such, it supports
command-line arguments, multiple languages, and so on.")
(home-page "https://www.gnu.org/software/hello/")
(license gpl3+)))</code></pre><p>Note that we have assigned the package value to an exported variable name with
<code>define-public</code>. This is effectively assigning the package to the <code>my-hello</code>
variable so that it can be referenced, among other as dependency of other
packages.</p><p>If you use <code>guix package --install-from-file=my-hello.scm</code> on the above file, it
will fail because the last expression, <code>define-public</code>, does not return a
package. If you want to use <code>define-public</code> in this use-case nonetheless, make
sure the file ends with an evaluation of <code>my-hello</code>:</p><pre><code class="language-scheme">; ...
(define-public my-hello
; ...
)
my-hello</code></pre><p>This last example is not very typical.</p><p>Now <code>my-hello</code> should be part of the package collection like all other official
packages. You can verify this with:</p><pre><code class="language-sh">$ guix package --show=my-hello</code></pre><h3>Guix channels</h3><p>Guix 0.16 features channels, which is very similar to <code>GUIX_PACKAGE_PATH</code> but
provides better integration and provenance tracking. Channels are not
necessarily local, they can be maintained as a public Git repository for
instance. Of course, several channels can be used at the same time.</p><p>See the <a href="http://guix.info/manual/en/Channels.html">“Channels” section in the manual</a> for setup details.</p><h3>Direct checkout hacking</h3><p>Working directly on the Guix project is recommended: it reduces the friction
when the time comes to submit your changes upstream to let the community benefit
from your hard work!</p><p>Unlike most software distributions, the Guix repository holds in one place both
the tooling (including the package manager) and the package definitions. This
choice was made so that it would give developers the flexibility to modify the
API without breakage by updating all packages at the same time. This reduces
development inertia.</p><p>Check out the official <a href="https://git-scm.com/">Git</a> repository:</p><pre><code class="language-sh">$ git clone https://git.savannah.gnu.org/git/guix.git</code></pre><p>In the rest of this article, we use <code>$GUIX_CHECKOUT</code> to refer to the location of
the checkout.</p><p>Follow the instruction from the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html">"Contributing" chapter</a> in the manual to set up the
repository environment.</p><p>Once ready, you should be able to use the package definitions from the
repository environment.</p><p>Feel free to edit package definitions found in <code>$GUIX_CHECKOUT/gnu/packages</code>.</p><p>The <code>$GUIX_CHECKOUT/pre-inst-env</code> script lets you use <code>guix</code> over the package
collection of the repository.</p><ul><li><p>Search packages, such as Ruby:</p><pre><code class="language-sh">$ cd $GUIX_CHECKOUT
$ ./pre-inst-env guix package --list-available=ruby
ruby 1.8.7-p374 out gnu/packages/ruby.scm:119:2
ruby 2.1.6 out gnu/packages/ruby.scm:91:2
ruby 2.2.2 out gnu/packages/ruby.scm:39:2</code></pre></li><li><p>Build a package, here Ruby version 2.1:</p><pre><code class="language-sh">$ ./pre-inst-env guix build --keep-failed ruby@2.1
/gnu/store/c13v73jxmj2nir2xjqaz5259zywsa9zi-ruby-2.1.6</code></pre></li><li><p>Install it to your user profile:</p><pre><code class="language-sh">$ ./pre-inst-env guix package --install ruby@2.1</code></pre></li><li><p>Check for common mistakes:</p><pre><code class="language-sh">$ ./pre-inst-env guix lint ruby@2.1</code></pre></li></ul><p>Guix strives at maintaining a high packaging standard; when contributing to the
Guix project, remember to</p><ul><li>follow the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Coding-Style.html">coding style</a>,</li><li>and review the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Submitting-Patches.html">check list</a> from the manual.</li></ul><p>Once you are happy with the result, you are welcome to send your contribution to
make it part of Guix. This process is also detailed in the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Contributing.html">manual</a>.</p><p>It's a community effort so the more join in, the better Guix becomes!</p><h1>Extended example</h1><p>The above "Hello World" example is as simple as it goes. Packages can be more
complex than that and Guix can handle more advanced scenarios. Let's look at
another, more sophisticated package (slightly modified from the source):</p><pre><code class="language-scheme">(define-module (gnu packages version-control)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix utils)
#:use-module (guix packages)
#:use-module (guix git-download)
#:use-module (guix build-system cmake)
#:use-module (gnu packages ssh)
#:use-module (gnu packages web)
#:use-module (gnu packages pkg-config)
#:use-module (gnu packages python)
#:use-module (gnu packages compression)
#:use-module (gnu packages tls))
(define-public my-libgit2
(let ((commit "e98d0a37c93574d2c6107bf7f31140b548c6a7bf")
(revision "1"))
(package
(name "my-libgit2")
(version (git-version "0.26.6" revision commit))
(source (origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/libgit2/libgit2/")
(commit commit)))
(file-name (git-file-name name version))
(sha256
(base32
"17pjvprmdrx4h6bb1hhc98w9qi6ki7yl57f090n9kbhswxqfs7s3"))
(patches (search-patches "libgit2-mtime-0.patch"))
(modules '((guix build utils)))
(snippet '(begin
;; Remove bundled software.
(delete-file-recursively "deps")
#t))))
(build-system cmake-build-system)
(outputs '("out" "debug"))
(arguments
`(#:tests? #t ; Run the test suite (this is the default)
#:configure-flags '("-DUSE_SHA1DC=ON") ; SHA-1 collision detection
#:phases
(modify-phases %standard-phases
(add-after 'unpack 'fix-hardcoded-paths
(lambda _
(substitute* "tests/repo/init.c"
(("#!/bin/sh") (string-append "#!" (which "sh"))))
(substitute* "tests/clar/fs.h"
(("/bin/cp") (which "cp"))
(("/bin/rm") (which "rm")))
#t))
;; Run checks more verbosely.
(replace 'check
(lambda _ (invoke "./libgit2_clar" "-v" "-Q")))
(add-after 'unpack 'make-files-writable-for-tests
(lambda _ (for-each make-file-writable (find-files "." ".*")))))))
(inputs
`(("libssh2" ,libssh2)
("http-parser" ,http-parser)
("python" ,python-wrapper)))
(native-inputs
`(("pkg-config" ,pkg-config)))
(propagated-inputs
;; These two libraries are in 'Requires.private' in libgit2.pc.
`(("openssl" ,openssl)
("zlib" ,zlib)))
(home-page "https://libgit2.github.com/")
(synopsis "Library providing Git core methods")
(description
"Libgit2 is a portable, pure C implementation of the Git core methods
provided as a re-entrant linkable library with a solid API, allowing you to
write native speed custom Git applications in any language with bindings.")
;; GPLv2 with linking exception
(license license:gpl2))))</code></pre><p>(In those cases were you only want to tweak a few fields from a package
definition, you should rely on inheritance instead of copy-pasting everything.
See below.)</p><p>Let's discuss those fields in depth.</p><h3><code>git-fetch</code> method</h3><p>Unlike the <code>url-fetch</code> method, <code>git-fetch</code> expects a <code>git-reference</code> which takes
a Git repository and a commit. The commit can be any Git reference such as
tags, so if the <code>version</code> is tagged, then it can be used directly. Sometimes
the tag is prefixed with a <code>v</code>, in which case you'd use <code>(commit (string-append "v" version))</code>.</p><p>To ensure that the source code from the Git repository is stored in a unique
directory with a readable name we use <code>(file-name (git-file-name name version))</code>.</p><p>Note that there is also a <code>git-version</code> procedure that can be used to derive the
version when packaging programs for a specific commit.</p><h3>Snippets</h3><p>Snippets are quoted (i.e. non-evaluated) Scheme code that are a means of patching
the source. They are a Guix-y alternative to the traditional <code>.patch</code> files.
Because of the quote, the code in only evaluated when passed to the Guix daemon
for building.</p><p>There can be as many snippet as needed.</p><p>Snippets might need additional Guile modules which can be imported from the
<code>modules</code> field.</p><h3>Inputs</h3><p>First, a syntactic comment: See the quasi-quote / comma syntax?</p><pre><code class="language-scheme">(native-inputs
`(("pkg-config" ,pkg-config)))</code></pre><p>is equivalent to</p><pre><code class="language-scheme">(native-inputs
(list (list "pkg-config" pkg-config)))</code></pre><p>You'll mostly see the former because it's shorter.</p><p>There are 3 different input types. In short:</p><ul><li><strong>native-inputs:</strong> Required for building but not runtime – installing a package
through a substitute won't install these inputs.</li><li><strong>inputs:</strong> Installed in the store but not in the profile, as well as being
present at build time.</li><li><strong>propagated-inputs:</strong> Installed in the store and in the profile, as well as
being present at build time.</li></ul><p>See <a href="https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html">the package reference in the manual</a> for more details.</p><p>The distinction between the various inputs is important: if a dependency can be
handled as an <em>input</em> instead of a <em>propagated input</em>, it should be done so, or
else it "pollutes" the user profile for no good reason.</p><p>For instance, a user installing a graphical program that depends on a
command line tool might only be interested in the graphical part, so there is no
need to force the command line tool into the user profile. The dependency is a
concern to the package, not to the user. <em>Inputs</em> make it possible to handle
dependencies without bugging the user by adding undesired executable files (or
libraries) to their profile.</p><p>Same goes for <em>native-inputs</em>: once the program is installed, build-time
dependencies can be safely garbage-collected.
It also matters when a substitute is available, in which case only the <em>inputs</em>
and <em>propagated inputs</em> will be fetched: the <em>native inputs</em> are not required to
install a package from a substitute.</p><h3>Outputs</h3><p>Just like how a package can have multiple inputs, it can also produce multiple
outputs.</p><p>Each output corresponds to a separate directory in the store.</p><p>The user can choose which output to install; this is useful to save space or
to avoid polluting the user profile with unwanted executables or libraries.</p><p>Output separation is optional. When the <code>outputs</code> field is left out, the
default and only output (the complete package) is referred to as <code>"out"</code>.</p><p>Typical separate output names include <code>debug</code> and <code>doc</code>.</p><p>It's advised to separate outputs only when you've shown it's worth it: if the
output size is significant (compare with <code>guix size</code>) or in case the package is
modular.</p><h3>Build system arguments</h3><p>The <code>arguments</code> is a keyword-value list used to configure the build process.</p><p>The simplest argument <code>#:tests?</code> can be used to disable the test suite when
building the package. This is mostly useful when the package does not feature
any test suite. It's strongly recommended to keep the test suite on if there is
one.</p><p>Another common argument is <code>:make-flags</code>, which specifies a list of flags to
append when running make, as you would from the command line. For instance, the
following flags</p><pre><code class="language-scheme">#:make-flags (list (string-append "prefix=" (assoc-ref %outputs "out"))
"CC=gcc")</code></pre><p>translate into</p><pre><code class="language-sh">$ make CC=gcc prefix=/gnu/store/...-<out></code></pre><p>This sets the C compiler to <code>gcc</code> and the <code>prefix</code> variable (the installation
directory in Make parlance) to <code>(assoc-ref %outputs "out")</code>, which is a build-stage
global variable pointing to the destination directory in the store (something like
<code>/gnu/store/...-my-libgit2-20180408</code>).</p><p>Similarly, it's possible to set the "configure" flags.</p><pre><code class="language-scheme">#:configure-flags '("-DUSE_SHA1DC=ON")</code></pre><p>The <code>%build-inputs</code> variable is also generated in scope. It's an association
table that maps the input names to their store directories.</p><p>The <code>phases</code> keyword lists the sequential steps of the build system. Typically
phases include <code>unpack</code>, <code>configure</code>, <code>build</code>, <code>install</code> and <code>check</code>. To know
more about those phases, you need to work out the appropriate build system
definition in <code>$GUIX_CHECKOUT/guix/build/gnu-build-system.scm</code>:</p><pre><code class="language-scheme">(define %standard-phases
;; Standard build phases, as a list of symbol/procedure pairs.
(let-syntax ((phases (syntax-rules ()
((_ p ...) `((p . ,p) ...)))))
(phases set-SOURCE-DATE-EPOCH set-paths install-locale unpack
bootstrap
patch-usr-bin-file
patch-source-shebangs configure patch-generated-file-shebangs
build check install
patch-shebangs strip
validate-runpath
validate-documentation-location
delete-info-dir-file
patch-dot-desktop-files
install-license-files
reset-gzip-timestamps
compress-documentation)))</code></pre><p>Or from the REPL:</p><pre><code class="language-scheme">> (add-to-load-path "/path/to/guix/checkout")
> ,module (guix build gnu-build-system)
> (map first %standard-phases)
(set-SOURCE-DATE-EPOCH set-paths install-locale unpack bootstrap
patch-usr-bin-file patch-source-shebangs configure
patch-generated-file-shebangs build check install patch-shebangs strip
validate-runpath validate-documentation-location delete-info-dir-file
patch-dot-desktop-files install-license-files reset-gzip-timestamps
compress-documentation)</code></pre><p>If you want to know more about what happens during those phases, consult the
associated procedures.</p><p>For instance, as of this writing the definition of <code>unpack</code> for the GNU build
system is</p><pre><code class="language-scheme">(define* (unpack #:key source #:allow-other-keys)
"Unpack SOURCE in the working directory, and change directory within the
source. When SOURCE is a directory, copy it in a sub-directory of the current
working directory."
(if (file-is-directory? source)
(begin
(mkdir "source")
(chdir "source")
;; Preserve timestamps (set to the Epoch) on the copied tree so that
;; things work deterministically.
(copy-recursively source "."
#:keep-mtime? #t))
(begin
(if (string-suffix? ".zip" source)
(invoke "unzip" source)
(invoke "tar" "xvf" source))
(chdir (first-subdirectory "."))))
#t)</code></pre><p>Note the <code>chdir</code> call: it changes the working directory to where the source was
unpacked.
Thus every phase following the <code>unpack</code> will use the source as a working
directory, which is why we can directly work on the source files.
That is to say, unless a later phase changes the working directory to something
else.</p><p>We modify the list of <code>%standard-phases</code> of the build system with the
<code>modify-phases</code> macro as per the list of specified modifications, which may have
the following forms:</p><ul><li><code>(add-before PHASE NEW-PHASE PROCEDURE)</code>: Run <code>PROCEDURE</code> named <code>NEW-PHASE</code> before <code>PHASE</code>.</li><li><code>(add-after PHASE NEW-PHASE PROCEDURE)</code>: Same, but afterwards.</li><li><code>(replace PHASE PROCEDURE)</code>.</li><li><code>(delete PHASE)</code>.</li></ul><p>The <code>PROCEDURE</code> supports the keyword arguments <code>inputs</code> and <code>outputs</code>. Each
input (whether <em>native</em>, <em>propagated</em> or not) and output directory is referenced
by their name in those variables. Thus <code>(assoc-ref outputs "out")</code> is the store
directory of the main output of the package. A phase procedure may look like
this:</p><pre><code class="language-scheme">(lambda* (#:key inputs outputs #:allow-other-keys)
(let (((bash-directory (assoc-ref inputs "bash"))
(output-directory (assoc-ref outputs "out"))
(doc-directory (assoc-ref outputs "doc"))
; ...
#t)</code></pre><p>The procedure must return <code>#t</code> on success. It's brittle to rely on the return
value of the last expression used to tweak the phase because there is no
guarantee it would be a <code>#t</code>. Hence the trailing <code>#t</code> to ensure the right value
is returned on success.</p><h3>Code staging</h3><p>The astute reader may have noticed the quasi-quote and comma syntax in the
argument field. Indeed, the build code in the package declaration should not be
evaluated on the client side, but only when passed to the Guix daemon. This
mechanism of passing code around two running processes is called <a href="https://arxiv.org/abs/1709.00833">code staging</a>.</p><h3>"Utils" functions</h3><p>When customizing <code>phases</code>, we often need to write code that mimics the
equivalent system invocations (<code>make</code>, <code>mkdir</code>, <code>cp</code>, etc.) commonly used during
regular "Unix-style" installations.</p><p>Some like <code>chmod</code> are native to Guile. See the <a href="https://www.gnu.org/software/guile/manual/html_node">Guile reference manual</a> for a
complete list.</p><p>Guix provides additional helper functions which prove especially handy in the
context of package management.</p><p>Some of those functions can be found in
<code>$GUIX_CHECKOUT/guix/guix/build/utils.scm</code>. Most of them mirror the behaviour
of the traditional Unix system commands:</p><ul><li><strong>which:</strong> Like the <code>which</code> system command.</li><li><strong>find-files:</strong> Akin to the <code>find</code> system command.</li><li><strong>mkdir-p:</strong> Like <code>mkdir -p</code>, which creates all parents as needed.</li><li><strong>install-file:</strong> Similar to <code>install</code> when installing a file to a (possibly
non-existing) directory. Guile has <code>copy-file</code> which works
like <code>cp</code>.</li><li><strong>copy-recursively:</strong> Like <code>cp -r</code>.</li><li><strong>delete-file-recursively:</strong> Like <code>rm -rf</code>.</li><li><strong>invoke:</strong> Run an executable. This should be used instead of <code>system*</code>.</li><li><strong>with-directory-excursion:</strong> Run the body in a different working directory,
then restore the previous working directory.</li><li><strong>substitute*:</strong> A "sed-like" function.</li></ul><h3>Module prefix</h3><p>The license in our last example needs a prefix: this is because of how the
<code>license</code> module was imported in the package, as <code>#:use-module ((guix licenses) #:prefix license:)</code>. The <a href="https://www.gnu.org/software/guile/manual/html_node/Using-Guile-Modules.html">Guile module import mechanism</a> gives the user full
control over namespacing: this is needed to avoid clashes between, say, the
<code>zlib</code> variable from <code>licenses.scm</code> (a <em>license</em> value) and the <code>zlib</code> variable
from <code>compression.scm</code> (a <em>package</em> value).</p><h1>Other build systems</h1><p>What we've seen so far covers the majority of packages using a build system
other than the <code>trivial-build-system</code>. The latter does not automate anything
and leaves you to build everything manually. This can be more demanding and we
won't cover it here for now, but thankfully it is rarely necessary to fall back
on this system.</p><p>For the other build systems, such as ASDF, Emacs, Perl, Ruby and many more, the
process is very similar to the GNU build system except for a few specialized
arguments.</p><p>Learn more about build systems in</p><ul><li><a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html#Build-Systems">the manual, section 4.2 Build systems</a>,</li><li>the source code in the <code>$GUIX_CHECKOUT/guix/build</code> and
<code>$GUIX_CHECKOUT/guix/build-system</code> directories.</li></ul><h1>Programmable and automated package definition</h1><p>We can't repeat it enough: having a full-fledged programming language at hand
empowers us in ways that reach far beyond traditional package management.</p><p>Let's illustrate this with some awesome features of Guix!</p><h3>Recursive importers</h3><p>You might find some build systems good enough that there is little to do at all
to write a package, to the point that it becomes repetitive and tedious after a
while. A <em>raison d'être</em> of computers is to replace human beings at those
boring tasks. So let's tell Guix to do this for us and create the package
definition of an R package from CRAN (the output is trimmed for conciseness):</p><pre><code class="language-sh">$ guix import cran --recursive walrus
(define-public r-mc2d
; ...
(license gpl2+)))
(define-public r-jmvcore
; ...
(license gpl2+)))
(define-public r-wrs2
; ...
(license gpl3)))
(define-public r-walrus
(package
(name "r-walrus")
(version "1.0.3")
(source
(origin
(method url-fetch)
(uri (cran-uri "walrus" version))
(sha256
(base32
"1nk2glcvy4hyksl5ipq2mz8jy4fss90hx6cq98m3w96kzjni6jjj"))))
(build-system r-build-system)
(propagated-inputs
`(("r-ggplot2" ,r-ggplot2)
("r-jmvcore" ,r-jmvcore)
("r-r6" ,r-r6)
("r-wrs2" ,r-wrs2)))
(home-page "https://github.com/jamovi/walrus")
(synopsis "Robust Statistical Methods")
(description
"This package provides a toolbox of common robust statistical tests, including robust descriptives, robust t-tests, and robust ANOVA. It is also available as a module for 'jamovi' (see <https://www.jamovi.org> for more information). Walrus is based on the WRS2 package by Patrick Mair, which is in turn based on the scripts and work of Rand Wilcox. These analyses are described in depth in the book 'Introduction to Robust Estimation & Hypothesis Testing'.")
(license gpl3)))</code></pre><p>The recursive importer won't import packages for which Guix already has package
definitions, except for the very first.</p><p>Not all applications can be packaged this way, only those relying on a select
number of supported systems. Read about the full list of importers in the <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-import.html">guix
import section</a> of the manual.</p><h3>Automatic update</h3><p>Guix can be smart enough to check for updates on systems it knows. It can
report outdated package definitions with</p><pre><code class="language-sh">$ guix refresh hello</code></pre><p>In most cases, updating a package to a newer version requires little more than
changing the version number and the checksum. Guix can do that automatically as
well:</p><pre><code class="language-sh">$ guix refresh hello --update</code></pre><h3>Inheritance</h3><p>If you've started browsing the existing package definitions, you might have
noticed that a significant number of them have a <code>inherit</code> field:</p><pre><code class="language-scheme">(define-public adwaita-icon-theme
(package (inherit gnome-icon-theme)
(name "adwaita-icon-theme")
(version "3.26.1")
(source (origin
(method url-fetch)
(uri (string-append "mirror://gnome/sources/" name "/"
(version-major+minor version) "/"
name "-" version ".tar.xz"))
(sha256
(base32
"17fpahgh5dyckgz7rwqvzgnhx53cx9kr2xw0szprc6bnqy977fi8"))))
(native-inputs
`(("gtk-encode-symbolic-svg" ,gtk+ "bin")))))</code></pre><p>All unspecified fields are inherited from the parent package. This is very
convenient to create alternative packages, for instance with different source,
version or compilation options.</p><h1>Getting help</h1><p>Sadly, some applications can be tough to package. Sometimes they need a patch to
work with the non-standard filesystem hierarchy enforced by the store.
Sometimes the tests won't run properly. (They can be skipped but this is not
recommended.) Other times the resulting package won't be reproducible.</p><p>Should you be stuck, unable to figure out how to fix any sort of packaging
issue, don't hesitate to ask the community for help.</p><p>See the <a href="https://www.gnu.org/software/guix/contact/">Guix homepage</a> for information on the mailing lists, IRC, etc.</p><h1>Conclusion</h1><p>This tutorial was a showcase of the sophisticated package management that Guix
boasts. At this point we have mostly restricted this introduction to the
<code>gnu-build-system</code> which is a core abstraction layer on which more advanced
abstractions are based.</p><p>Where do we go from here? Next we ought to dissect the innards of the build
system by removing all abstractions, using the <code>trivial-build-system</code>: this
should give us a thorough understanding of the process before investigating some
more advanced packaging techniques and edge cases.</p><p>Other features worth exploring are the interactive editing and debugging
capabilities of Guix provided by the Guile REPL.</p><p>Those fancy features are completely optional and can wait; now is a good time
to take a well-deserved break. With what we've introduced here you should be
well armed to package lots of programs. You can get started right away and
hopefully we will see your contributions soon!</p><h1>References</h1><ul><li><p>The <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">package reference in the manual</a></p></li><li><p><a href="https://gitlab.com/pjotrp/guix-notes/blob/master/HACKING.org">Pjotr’s hacking guide to GNU Guix</a></p></li><li><p><a href="https://www.gnu.org/software/guix/guix-ghm-andreas-20130823.pdf">"GNU Guix: Package without a
scheme!"</a>,
by Andreas Enge</p></li></ul><h1>Notices</h1><p><a href="http://creativecommons.org/licenses/by-sa/4.0/"><img src="https://i.creativecommons.org/l/by-sa/4.0/88x31.png" alt="CC-BY-SA" title="CC-BY-SA
4.0" /></a></p><p>This work is licensed under a Creative Commons Attribution-ShareAlike 4.0
International License.</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager for the GNU system. The Guix System Distribution or GuixSD is
an advanced distribution of the GNU system that relies on GNU Guix and
<a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects the user's
freedom</a>.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. Guix uses low-level
mechanisms from the Nix package manager, except that packages are
defined as native <a href="https://www.gnu.org/software/guile">Guile</a> modules,
using extensions to the <a href="http://schemers.org">Scheme</a> language. GuixSD
offers a declarative approach to operating system configuration
management, and is highly customizable and hackable.</p><p>GuixSD can be used on an i686, x86_64, ARMv7, and AArch64 machines. It
is also possible to use Guix on top of an already installed GNU/Linux
system, including on mips64el and aarch64.</p>https://guix.gnu.org/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my//Multi-dimensional transactions and rollbacks, oh my!Ludovic Courtès2018-07-24T14:30:00Z2018-07-24T14:30:00Z One of the highlights of version
0.15.0
was the overhaul of guix pull ,
the command that updates Guix and its package collection. In Debian
terms, you can think of guix pull as: apt-get update && apt-get install apt Let’s be frank, guix pull does not yet run as quickly as this
apt-get command—in the “best case”, when pre-built binaries are
available, it currently runs in about 1m30s on a recent laptop. More
about the performance story in a future post… One of the key features of…<p>One of the <a href="https://www.gnu.org/software/guix/blog/2018/gnu-guix-and-guixsd-0.15.0-released/">highlights of version
0.15.0</a>
was the overhaul of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-pull.html"><code>guix pull</code></a>,
the command that updates Guix and its package collection. In Debian
terms, you can think of <code>guix pull</code> as:</p><pre><code>apt-get update && apt-get install apt</code></pre><p>Let’s be frank, <code>guix pull</code> does not yet run as quickly as this
<code>apt-get</code> command—in the “best case”, when pre-built binaries are
available, it currently runs in about 1m30s on a recent laptop. More
about the performance story in a future post…</p><p>One of the key features of the new <code>guix pull</code> is the ability to <em>roll
back</em> to previous versions of Guix. That’s a distinguishing feature
that opens up new possibilities.</p><h1>“Profile generations”</h1><p>Transactional upgrades and rollbacks have been a distinguishing feature
of Guix since Day 1. They come for free as a consequence of the
functional package management model inherited from the Nix package
manager. To many users, this alone is enough to justify using a
functional package manager: if an upgrade goes wrong, you can always
roll back. Let’s recap how this all works.</p><p>As a user, you install packages in your own <em>profile</em>, which defaults to
<code>~/.guix-profile</code>. Then from time to time you update Guix and its
package collection:</p><pre><code>$ guix pull</code></pre><p>This updates <code>~/.config/guix/current</code>, giving you an updated <code>guix</code>
executable along with an updated set of packages. You can now upgrade
the packages that are in your profile:</p><pre><code>$ guix package -u
The following packages will be upgraded:
diffoscope 93 → 96 /gnu/store/…-diffoscope-96
emacs 25.3 → 26.1 /gnu/store/…-emacs-26.1
gimp 2.8.22 → 2.10.4 /gnu/store/…-gimp-2.10.4
gnupg 2.2.7 → 2.2.9 /gnu/store/…-gnupg-2.2.9</code></pre><p>The upgrade creates a new <em>generation</em> of your profile—the previous
generation of your profile, with diffoscope 93, emacs 25.3, and so on is
still around. You can list profile generations:</p><pre><code>$ guix package --list-generations
Generation 1 Jun 08 2018 20:06:21
diffoscope 93 out /gnu/store/…-diffoscope-93
emacs 25.3 out /gnu/store/…-emacs-25.3
gimp 2.8.22 out /gnu/store/…-gimp-2.8.22
gnupg 2.2.7 out /gnu/store/…-gnupg-2.2.7
python 3.6.5 out /gnu/store/…-python-3.6.5
Generation 2 Jul 12 2018 12:42:08 (current)
- diffoscope 93 out /gnu/store/…-diffoscope-93
- emacs 25.3 out /gnu/store/…-emacs-25.3
- gimp 2.8.22 out /gnu/store/…-gimp-2.8.22
- gnupg 2.2.7 out /gnu/store/…-gnupg-2.2.7
+ diffoscope 96 out /gnu/store/…-diffoscope-96
+ emacs 26.1 out /gnu/store/…-emacs-26.1
+ gimp 2.10.4 out /gnu/store/…-gimp-2.10.4
+ gnupg 2.2.9 out /gnu/store/…-gnupg-2.2.9</code></pre><p>That shows our two generations with the diff between Generation 1 and
Generation 2. We can at any time run <code>guix package --roll-back</code> and get
our previous versions of gimp, emacs, and so on. Each generation is
just a bunch of symlinks to those packages, so what we have looks like
this:</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/guix-pull1.png" alt="Image of the profile generations." /></p><p>Notice that python was not updated, so it’s shared between both
generations. And of course, all the dependencies that didn’t change in
between—e.g., the C library—are shared among all packages.</p><h1><code>guix pull</code> generations</h1><p>Like I wrote above, <code>guix pull</code> brings the latest set of package
definitions from Git <code>master</code>. The Guix package collection usually
contains only the latest version of each package; for example, current
<code>master</code> only has version 26.1 of Emacs and version 2.10.4 of the GIMP
(there are notable exceptions such as GCC or Python.) Thus, <code>guix package -i gimp</code>, from today’s master, can only install gimp 2.10.4.
Often, that’s not a problem: you can keep old profile generations
around, so if you really need that older version of Emacs, you can run
it from your previous generation.</p><p>Still, having <code>guix pull</code> keep track of the changes to Guix and its
package collection is useful. Starting from 0.15.0, <code>guix pull</code> creates
a new generation, just like <code>guix package</code> does. After you’ve run <code>guix pull</code>, you can now list Guix generations as well:</p><pre><code>$ guix pull -l
Generation 10 Jul 14 2018 00:02:03
guix 27f7cbc
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: origin/master
commit: 27f7cbc91d1963118e44b14d04fcc669c9618176
Generation 11 Jul 20 2018 10:44:46
guix 82549f2
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: origin/master
commit: 82549f2328c59525584b92565846217c288d8e85
14 new packages: bsdiff, electron-cash, emacs-adoc-mode,
emacs-markup-faces, emacs-rust-mode, inchi, luakit, monero-gui,
nethack, openbabel, qhull, r-txtplot, stb-image, stb-image-write
52 packages upgraded: angband@4.1.2, aspell-dict-en@2018.04.16-0,
assimp@4.1.0, bitcoin-core@0.16.1, botan@2.7.0, busybox@1.29.1,
…
Generation 12 Jul 23 2018 15:22:52 (current)
guix fef7bab
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: origin/master
commit: fef7baba786a96b7a3100c9c7adf8b45782ced37
20 new packages: ccrypt, demlo, emacs-dired-du,
emacs-helm-org-contacts, emacs-ztree, ffmpegthumbnailer,
go-github-com-aarzilli-golua, go-github-com-kr-text,
go-github-com-mattn-go-colorable, go-github-com-mattn-go-isatty,
go-github-com-mgutz-ansi, go-github-com-michiwend-golang-pretty,
go-github-com-michiwend-gomusicbrainz, go-github-com-stevedonovan-luar,
go-github-com-wtolson-go-taglib, go-github-com-yookoala-realpath,
go-gitlab-com-ambrevar-damerau, go-gitlab-com-ambrevar-golua-unicode,
guile-pfds, u-boot-cubietruck
27 packages upgraded: c-toxcore@0.2.4, calibre@3.28.0,
emacs-evil-collection@20180721-2.5d739f5,
…</code></pre><p>The nice thing here is that <code>guix pull</code> provides high-level information
about the differences between two subsequent generations of Guix.</p><p>In the end, Generation 1 of our profile was presumably built with Guix
Generation 11, while Generation 2 of our profile was built with Guix
Generation 12. We have a clear mapping between Guix generations as
created by <code>guix pull</code> and profile generations as created with <code>guix package</code>:</p><p><img src="https://www.gnu.org/software/guix/static/blog/img/guix-pull3.png" alt="Image of the Guix generations." /></p><p>Each generation created by <code>guix pull</code> corresponds to one commit in the
Guix repo. Thus, if I go to another machine and run:</p><pre><code>$ guix pull --commit=fef7bab</code></pre><p>then I know that I get the exact same Guix instance as my Generation 12
above. From there I can install diffoscope, emacs, etc. and I know I’ll
get the exact same binaries as those I have above, thanks to
<a href="https://reproducible-builds.org/docs/definition/">reproducible builds</a>.</p><p>These are very strong guarantees in terms of reproducibility and
provenance tracking—properties that are
<a href="https://github.com/canonical-websites/snapcraft.io/issues/651">typically</a>
<a href="https://lwn.net/Articles/752982/">missing</a> from “applications bundles”
à la Docker.</p><p>In addition, you can easily run an older Guix. For instance, this is
how you would install the version of gimp that was current as of
Generation 10:</p><pre><code>$ ~/.config/guix/current-10-link/bin/guix package -i gimp</code></pre><p>At this point your profile contains gimp coming from an old Guix along
with packages installed from the latest Guix. Past and present coexist
in the same profile. The historical dimension of the profile no longer
matches exactly the history of Guix itself.</p><h1>Composing Guix revisions</h1><p>Some people have expressed interest in being able to compose packages
coming from different revisions of Guix—say to create a profile
containing old versions of Python and NumPy, but also the latest and
greatest GCC. It may seem far-fetched but it has very real
applications: there are large collections of scientific packages and in
particular bioinformatics packages that don’t move as fast as our
beloved flagship free software packages, and users may require ancient
versions of some of the tools.</p><p>We could keep old versions of many packages but maintainability costs
would grow exponentially. Instead, Guix users can take advantage of the
version control history of Guix itself to mix and match packages coming
from different revisions of Guix. As shown above, it’s already possible
to achieve this by running the <code>guix</code> program off the generation of
interest. It does the job, but can we do better?</p><p>In the process of enhancing <code>guix pull</code> we developed a high-level API
that allows an instance of Guix to “talk” to a different instance of
Guix—<a href="https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32115">an
“inferior”</a>. It’s
what allows <code>guix pull</code> to display the list of packages that were added
or upgraded between two revisions. The next logical step will be to
provide seamless integration of packages coming from an inferior. That
way, users would be able to refer to “past” package graphs right from <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#index-profile-manifest">a
profile
manifest</a>
or from the command-line. Future work!</p><h1>On coupling</h1><p>The time traveler in you might be wondering: Why are package definitions
coupled with the package manager, doesn’t it make it harder to compose
packages coming from different revisions? Good point!</p><p>Tight coupling certainly complicates this kind of composition: we can’t
just have any revision of Guix load package definitions from any other
revision; this could fail altogether, or it could provide a different
build result. Another potential issue is that <code>guix pull</code>ing an older
revision not only gives you an older set of packages, it also gives you
older tools, bug-for-bug.</p><p>The reason for this coupling is that a package definition <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html">like this
one</a>
doesn’t exist in a vacuum. Its meaning is defined by the implementation
of <a href="https://www.gnu.org/software/guix/manual/en/html_node/package-Reference.html">package
objects</a>,
by
<a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html"><code>gnu-build-system</code></a>,
by a number of <a href="https://arxiv.org/abs/1305.4584">lower-level</a>
<a href="https://hal.inria.fr/hal-01580582/en">abstractions</a> that are all
defined as extensions of the Scheme language in Guix itself, and
ultimately by <a href="https://www.gnu.org/software/guile/">Guile</a>, which
implements the language Guix is written in. Each instance created by
<code>guix pull</code> brings all these components. Because Guix is implemented as
a set of programming language extensions and libraries, that package
definitions depend on all these parts becomes manifest. Instead of
being frozen, the APIs and package definitions evolve together, which
gives us developers a lot of freedom on the changes we can make.</p><p><a href="https://nixos.org/nix/">Nix</a> results from a different design choice.
Nix-the-package-manager implements the Nix language, which acts as a
“frozen” interface. Package definitions in Nixpkgs are written in that
language, and a given version of Nix can <em>possibly</em> interpret both
current and past package definitions without further ado. The Nix
language does evolve though, so at one point an old Nix inevitably
<a href="https://github.com/NixOS/nixpkgs/blob/master/lib/minver.nix">becomes unable to evaluate a new
Nixpkgs</a>,
and <em>vice versa</em>.</p><p>These two approaches make different tradeoffs. Nix’ loose coupling
simplifies the implementation and makes it easy to compose old and new
package definitions, to some extent; Guix’ tight coupling makes such
composition more difficult to implement, but it leaves developers more
freedom and, we hope, may support “time travels” over longer period of
times. Time will tell!</p><h1>It’s like driving a DeLorean</h1><p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/1/15/TeamTimeCar.com-BTTF_DeLorean_Time_Machine-OtoGodfrey.com-JMortonPhoto.com-05.jpg/800px-TeamTimeCar.com-BTTF_DeLorean_Time_Machine-OtoGodfrey.com-JMortonPhoto.com-05.jpg" alt="Inside the cabin of the DeLorean time machine in “Back to the Future.”" /></p><p><em><a href="https://commons.wikimedia.org/wiki/File:TeamTimeCar.com-BTTF_DeLorean_Time_Machine-OtoGodfrey.com-JMortonPhoto.com-05.jpg">Picture</a>
of a DeLorean cabin by Oto Godfrey and Justin Morton, under <a href="https://creativecommons.org/licenses/by-sa/4.0/deed.en">CC-BY-SA
4.0</a>.</em></p><p>That profile generations are kept around already gave users a time
machine of sorts—you can always roll back to a previous state of your
software environment. With the addition of roll-back support for <code>guix pull</code>, this adds another dimension to the time machine: you can
roll-back to a previous state of Guix itself and from there create
alternative futures or even mix bits from the past with bits from the
present. We hope you’ll enjoy it!</p>https://guix.gnu.org/blog/2018/customize-guixsd-use-stock-ssh-agent-everywhere//Customize GuixSD: Use Stock SSH Agent Everywhere!Chris Marusich2018-05-26T17:00:00Z2018-05-26T17:00:00Z I frequently use SSH. Since I don't like typing my password all the
time, I use an SSH agent. Originally I used the GNOME Keyring as my
SSH agent, but recently I've switched to using the ssh-agent from
OpenSSH. I accomplished this by doing the following two things: Replace the default GNOME Keyring with a custom-built version that
disables the SSH agent feature. Start my desktop session with OpenSSH's ssh-agent so that it's
always available to any applications in my desktop session. Below, I'll show you in…<p>I frequently use SSH. Since I don't like typing my password all the
time, I use an SSH agent. Originally I used the GNOME Keyring as my
SSH agent, but recently I've switched to using the <code>ssh-agent</code> from
OpenSSH. I accomplished this by doing the following two things:</p><ul><li><p>Replace the default GNOME Keyring with a custom-built version that
disables the SSH agent feature.</p></li><li><p>Start my desktop session with OpenSSH's <code>ssh-agent</code> so that it's
always available to any applications in my desktop session.</p></li></ul><p>Below, I'll show you in detail how I did this. In addition to being
useful for anyone who wants to use OpenSSH's <code>ssh-agent</code> in GuixSD, I
hope this example will help to illustrate how GuixSD enables you to
customize your entire system to be just the way you want it!</p><h1>The Problem: GNOME Keyring Can't Handle My SSH Keys</h1><p>On GuixSD, I like to use the <a href="https://www.gnome.org">GNOME desktop
environment</a>. GNOME is just one of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Desktop-Services.html">the
various desktop environments that GuixSD
supports</a>.
By default, the GNOME desktop environment on GuixSD comes with a lot
of goodies, including the <a href="https://wiki.gnome.org/Projects/GnomeKeyring">GNOME
Keyring</a>, which is
GNOME's integrated solution for securely storing secrets, passwords,
keys, and certificates.</p><p>The GNOME Keyring has many useful features. One of those is <a href="https://wiki.gnome.org/Projects/GnomeKeyring/Ssh">its SSH
Agent feature</a>.
This feature allows you to use the GNOME Keyring as an SSH agent.
This means that when you invoke a command like <code>ssh-add</code>, it will add
the private key identities to the GNOME Keyring. Usually this is
quite convenient, since it means that GNOME users basically get an SSH
agent for free!</p><p>Unfortunately, up until <a href="https://www.gnome.org/news/2018/03/gnome-3-28-released/">GNOME 3.28 (the current
release)</a>,
the GNOME Keyring's SSH agent implementation was not as complete as
the stock SSH agent from OpenSSH. As a result, <a href="https://bugzilla.gnome.org/show_bug.cgi?id=775981">earlier versions of
GNOME Keyring did not support many use
cases</a>. This was a
problem for me, since GNOME Keyring couldn't read my modern SSH keys.
To make matters worse, by design the SSH agent for GNOME Keyring and
OpenSSH both use the same environment variables (e.g.,
<code>SSH_AUTH_SOCK</code>). This makes it difficult to use OpenSSH's
<code>ssh-agent</code> everywhere within my GNOME desktop environment.</p><p>Happily, starting with GNOME 3.28, <a href="https://bugzilla.gnome.org/show_bug.cgi?id=775981">GNOME Keyring delegates all SSH
agent functionality to the stock SSH agent from
OpenSSH</a>. They
have removed their custom implementation entirely. This means that
today, I could solve my problem simply by using the most recent
version of GNOME Keyring. I'll probably do just that when the new
release gets included in Guix. However, when I first encountered this
problem, GNOME 3.28 hadn't been released yet, so the only option
available to me was to customize GNOME Keyring or remove it entirely.</p><p>In any case, I'm going to show you how I solved this problem by
modifying the default GNOME Keyring from the Guix package collection.
The same ideas can be used to customize any package, so hopefully it
will be a useful example. And what if you don't use GNOME, but you do
want to use OpenSSH's <code>ssh-agent</code>? In that case, you may still need
to customize your GuixSD system a little bit. Let me show you how!</p><h1>The Solution: <code>~/.xsession</code> and a Custom GNOME Keyring</h1><p>The goal is to make OpenSSH's <code>ssh-agent</code> available everywhere when we
log into our GNOME desktop session. First, we must arrange for
<code>ssh-agent</code> to be running whenever we're logged in.</p><p>There are many ways to accomplish this. For example, I've seen people
implement shell code in their shell's start-up files which basically
manages their own <code>ssh-agent</code> process. However, I prefer to just
start <code>ssh-agent</code> once and not clutter up my shell's start-up files
with unnecessary code. So that's what we're going to do!</p><h1>Launch OpenSSH's <code>ssh-agent</code> in Your <code>~/.xsession</code></h1><p>By default, GuixSD uses the <a href="https://sourceforge.net/projects/slim.berlios">SLiM desktop
manager</a>. When you log
in, SLiM presents you with a menu of so-called "desktop sessions",
which correspond to the desktop environments you've declared in your
<a href="https://www.gnu.org/software/guix/manual/en/html_node/operating_002dsystem-Reference.html">operating system
declaration</a>.
For example, if you've added the
<a href="https://www.gnu.org/software/guix/manual/en/html_node/Desktop-Services.html">gnome-desktop-service</a>
to your operating system declaration, then you'll see an option for
GNOME at the SLiM login screen.</p><p>You can further customize your desktop session with the <code>~/.xsession</code>
file. The contract for this file in GuixSD is the same as it is for
many GNU/Linux distributions: <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/xorg.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n392">if it exists, then it will be
executed</a>.
The arguments passed to it will be the command line invocation that
would normally be executed to start the desktop session that you
selected from the SLiM login screen. Your <code>~/.xsession</code> is expected
to do whatever is necessary to customize and then start the specified
desktop environment. For example, when you select GNOME from the SLiM
login screen, your <code>~/.xsession</code> file will basically be executed like
this (for the exact execution mechanism, please refer to the source
code linked above):</p><pre><code class="language-shell">$ ~/.xsession gnome-session</code></pre><p>The upshot of all this is that the <code>~/.xsession</code> is an <em>ideal</em> place
to set up your SSH agent! If you start an SSH agent in your
<code>~/.xsession</code> file, you can have the SSH agent available everywhere,
automatically! Check it out: Put this into your <code>~/.xsession</code> file,
and make the file executable:</p><pre><code class="language-shell">#!/run/current-system/profile/bin/bash
exec ssh-agent "$@"</code></pre><p>When you invoke <code>ssh-agent</code> in this way, it executes the specified
program in an environment where commands like <code>ssh-add</code> just work. It
does this by setting environment variables such as <code>SSH_AUTH_SOCK</code>,
which programs like <code>ssh-add</code> find and use automatically. Because
GuixSD allows you to customize your desktop session like this, you can
use any SSH agent you want in any desktop environments that you want,
automatically!</p><p>Of course, if you're using GNOME Keyring version 3.27 or earlier (like
I was), then this isn't quite enough. In that case, the SSH agent
feature of GNOME Keyring will override the environment variables set
by OpenSSH's <code>ssh-agent</code>, so commands like <code>ssh-add</code> will wind up
communicating with the GNOME Keyring instead of the <code>ssh-agent</code> you
launched in your <code>~/.xsession</code>. This is bad because, as previously
mentioned, GNOME Keyring version 3.27 or earlier doesn't support as
many uses cases as OpenSSH's <code>ssh-agent</code>.</p><p>How can we work around this problem?</p><h1>Customize the GNOME Keyring</h1><p>One heavy-handed solution would be to remove GNOME Keyring entirely.
That would work, but then you would lose out on all the other great
features that it has to offer. Surely we can do better!</p><p>The GNOME Keyring documentation
<a href="https://wiki.gnome.org/Projects/GnomeKeyring/Ssh">explains</a> that one
way to disable the SSH agent feature is to include the
<code>--disable-ssh-agent</code> configure flag when building it. Thankfully,
Guix provides some ways to customize software in <em>exactly</em> this way!</p><p>Conceptually, we "just" have to do the following two things:</p><ul><li><p>Customize the existing <code>gnome-keyring</code> package.</p></li><li><p>Make the <code>gnome-desktop-service</code> use our custom <code>gnome-keyring</code>
package.</p></li></ul><h1>Create a Custom GNOME Keyring Package</h1><p>Let's begin by defining a custom <code>gnome-keyring</code> package, which we'll
call <code>gnome-keyring-sans-ssh-agent</code>. With Guix, we can do this in
less than ten lines of code:</p><pre><code class="language-scheme">(define-public gnome-keyring-sans-ssh-agent
(package
(inherit gnome-keyring)
(name "gnome-keyring-sans-ssh-agent")
(arguments
(substitute-keyword-arguments
(package-arguments gnome-keyring)
((#:configure-flags flags)
`(cons "--disable-ssh-agent" ,flags))))))</code></pre><p>Don't worry if some of that code is unclear at first. I'll clarify it
now!</p><p>In Guix, a <code><package></code> record like the one above is defined by a macro
called <code>define-record-type*</code> (<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/records.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n178">defined in the file guix/records.scm in
the Guix
source</a>).
It's similar to an <a href="https://www.gnu.org/software/guile/manual/en/html_node/SRFI_002d9-Records.html#SRFI_002d9-Records">SRFI-9
record</a>.
The <code>inherit</code> feature of this macro is very useful: it creates a new
copy of an existing record, overriding specific fields in the new copy
as needed.</p><p>In the above, we define <code>gnome-keyring-sans-ssh-agent</code> to be a copy of
the <code>gnome-keyring</code> package, and we use <code>inherit</code> to change the <code>name</code>
and <code>arguments</code> fields in that new copy. We also use the
<code>substitute-keyword-arguments</code> macro (<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/guix/utils.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n345">defined in the file
guix/utils.scm in the Guix
source</a>)
to add <code>--disable-ssh-agent</code> to the list of <a href="https://www.gnu.org/software/guix/manual/en/html_node/Build-Systems.html">configure
flags</a>
defined in the <code>gnome-keyring</code> package. The effect of this is to
define a new GNOME Keyring package that is built exactly the same as
the original, but in which the SSH agent is disabled.</p><p>I'll admit this code may seem a little opaque at first, but all code
does when you first learn it. Once you get the hang of things, you
can customize packages any way you can imagine. If you want to learn
more, you should read the docstrings for the <code>define-record-type*</code> and
<code>substitute-keyword-arguments</code> macros in the Guix source code. It's
also very helpful to <code>grep</code> the source code to see examples of how
these macros are used in practice. For example:</p><pre><code class="language-shell">$ # Search the currently installed Guix for the current user.
$ grep -r substitute-keyword-arguments ~/.config/guix/latest
$ # Search the Guix Git repository, assuming you've checked it out here.
$ grep -r substitute-keyword-arguments ~/guix</code></pre><h1>Use the Custom GNOME Keyring Package</h1><p>OK, we've created our own custom GNOME Keyring package. Great! Now,
how do we use it?</p><p>In GuixSD, the GNOME desktop environment is treated as a <a href="https://www.gnu.org/software/guix/manual/en/html_node/Services.html">system
service</a>. To
make GNOME use our custom GNOME Keyring package, we must somehow
customize the <code>gnome-desktop-service</code> (<a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/desktop.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n795">defined in the file
gnu/services/desktop.scm</a>)
to use our custom package. How do we customize a service? Generally,
the answer depends on the service. Thankfully, many of GuixSD's
services, including the <code>gnome-desktop-service</code>, follow a similar
pattern. In this case, we "just" need to pass a custom
<code><gnome-desktop-configuration></code> record to the <code>gnome-desktop-service</code>
procedure in our operating system declaration, like this:</p><pre><code class="language-scheme">(operating-system
...
(services (cons*
(gnome-desktop-service
#:config my-gnome-desktop-configuration)
%desktop-services)))</code></pre><p>Here, the <code>cons*</code> procedure just adds the GNOME desktop service to the
<code>%desktop-services</code> list, returning the new list. For details, please
refer to <a href="https://www.gnu.org/software/guile/manual/en/html_node/List-Constructors.html#index-cons_002a">the Guile
manual</a>.</p><p>Now the question is: what should <code>my-gnome-desktop-configuration</code> be?
Well, if we examine <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/services/desktop.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n799">the definition of this record type in the Guix
source</a>,
we see the following:</p><pre><code class="language-scheme">(define-record-type* <gnome-desktop-configuration> gnome-desktop-configuration
make-gnome-desktop-configuration
gnome-desktop-configuration
(gnome-package gnome-package (default gnome)))</code></pre><p>The <code>gnome</code> package referenced here is a "meta" package: it exists
only to aggregate many GNOME packages together, including
<code>gnome-keyring</code>. To see <a href="https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/gnome.scm?id=263c9941a1e523b360ca9f42d1ed6b11e6e6e285#n5977">its
definition</a>,
we can simply invoke <code>guix edit gnome</code>, which <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-edit.html#Invoking-guix-edit">opens the file where
the package is
defined</a>:</p><pre><code class="language-scheme">(define-public gnome
(package
(name "gnome")
(version (package-version gnome-shell))
(source #f)
(build-system trivial-build-system)
(arguments '(#:builder (mkdir %output)))
(propagated-inputs
;; TODO: Add more packages according to:
;; <https://packages.debian.org/jessie/gnome-core>.
`(("adwaita-icon-theme" ,adwaita-icon-theme)
("baobab" ,baobab)
("font-cantarell" ,font-cantarell)
[... many packages omitted for brevity ...]
("gnome-keyring" ,gnome-keyring)
[... many packages omitted for brevity ...]
(synopsis "The GNU desktop environment")
(home-page "https://www.gnome.org/")
(description
"GNOME is the graphical desktop for GNU. It includes a wide variety of
applications for browsing the web, editing text and images, creating
documents and diagrams, playing media, scanning, and much more.")
(license license:gpl2+)))</code></pre><p>Apart from being a little long, this is <a href="https://www.gnu.org/software/guix/manual/en/html_node/Defining-Packages.html#Defining-Packages">just a normal package
definition</a>.
We can see that <code>gnome-keyring</code> is included in the list of
<code>propagated-inputs</code>. So, we need to create a replacement for the
<code>gnome</code> package that uses our <code>gnome-keyring-sans-ssh-agent</code> instead
of <code>gnome-keyring</code>. The following package definition accomplishes
that:</p><pre><code class="language-scheme">(define-public gnome-sans-ssh-agent
(package
(inherit gnome)
(name "gnome-sans-ssh-agent")
(propagated-inputs
(map (match-lambda
((name package)
(if (equal? name "gnome-keyring")
(list name gnome-keyring-sans-ssh-agent)
(list name package))))
(package-propagated-inputs gnome)))))</code></pre><p>As before, we use <code>inherit</code> to create a new copy of the <code>gnome</code>
package that overrides the original <code>name</code> and <code>propagated-inputs</code>
fields. Since Guix packages are just defined using good old scheme,
we can use existing language features like
<a href="https://www.gnu.org/software/guile/manual/en/html_node/List-Mapping.html#index-map"><code>map</code></a>
and
<a href="https://www.gnu.org/software/guile/manual/en/html_node/Pattern-Matching.html#Pattern-Matching"><code>match-lambda</code></a>
to manipulate the list of propagated inputs. The effect of the above
is to create a new package that is the same as the <code>gnome</code> package but
uses <code>gnome-keyring-sans-ssh-agent</code> instead of <code>gnome-keyring</code>.</p><p>Now that we have <code>gnome-sans-ssh-agent</code>, we can create a custom
<code><gnome-desktop-configuration></code> record and pass it to the
<code>gnome-desktop-service</code> procedure as follows:</p><pre><code class="language-scheme">(operating-system
...
(services (cons*
(gnome-desktop-service
#:config (gnome-desktop-configuration
(gnome-package gnome-sans-ssh-agent)))
%desktop-services)))</code></pre><h1>Wrapping It All Up</h1><p>Finally, you need to run the following commands as <code>root</code> to create
and boot into the new <a href="https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-system.html">system
generation</a>
(replace <code>MY-CONFIG</code> with the path to the customized operating system
configuration file):</p><pre><code class="language-shell"># guix system reconfigure MY-CONFIG
# reboot</code></pre><p>After you log into GNOME, any time you need to use SSH, the stock SSH
agent from OpenSSH that you started in your <code>~/.xsession</code> file will be
used instead of the GNOME Keyring's SSH agent. It just works! Note
that it still works even if you select a non-GNOME desktop session
(like XFCE) at the SLiM login screen, since the <code>~/.xsession</code> is not
tied to any particular desktop session,</p><p>In the unfortunate event that something went wrong and things just
aren't working when you reboot, don't worry: with GuixSD, you can
safely roll back to the previous system generation via <a href="https://www.gnu.org/software/guix/manual/en/html_node/Using-the-Configuration-System.html#index-roll_002dback_002c-of-the-operating-system">the usual
mechanisms</a>.
For example, you can run this from the command line to roll back:</p><pre><code class="language-shell"># guix system roll-back
# reboot</code></pre><p>This is one of the great benefits that comes from the fact that <a href="https://www.gnu.org/software/guix/manual/en/html_node/Introduction.html#Introduction">Guix
follows the functional software deployment
model</a>.
However, note that because the <code>~/.xsession</code> file (like many files in
your home directory) is not managed by Guix, you must manually undo
the changes that you made to it in order to roll back fully.</p><h1>Conclusion</h1><p>I hope this helps give you some ideas for how you can customize your
own GuixSD system to make it exactly what you want it to be. Not only
can you customize your desktop session via your <code>~/.xsession</code> file,
but Guix also provides tools for you to modify any of the default
packages or services to suit your specific needs.</p><p>Happy hacking!</p><h1>Notices</h1><p><a href="http://creativecommons.org/publicdomain/zero/1.0/"><img src="https://licensebuttons.net/p/zero/1.0/88x31.png" alt="CC0" title="CC0 1.0
Universal" /></a></p><p>To the extent possible under law, Chris Marusich has waived all
copyright and related or neighboring rights to this article,
"Customize GuixSD: Use Stock SSH Agent Everywhere!". This work is
published from: United States.</p><p>The views expressed in this article are those of Chris Marusich and do
not necessarily reflect the views of his past, present, or future
employers.</p><h4>About GNU Guix</h4><p><a href="https://www.gnu.org/software/guix">GNU Guix</a> is a transactional package
manager for the GNU system. The Guix System Distribution or GuixSD is
an advanced distribution of the GNU system that relies on GNU Guix and
<a href="https://www.gnu.org/distros/free-system-distribution-guidelines.html">respects the user's
freedom</a>.</p><p>In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. Guix uses low-level
mechanisms from the Nix package manager, except that packages are
defined as native <a href="https://www.gnu.org/software/guile">Guile</a> modules,
using extensions to the <a href="http://schemers.org">Scheme</a> language. GuixSD
offers a declarative approach to operating system configuration
management, and is highly customizable and hackable.</p><p>GuixSD can be used on an i686, x86_64 and armv7 machines. It is also
possible to use Guix on top of an already installed GNU/Linux system,
including on mips64el and aarch64.</p>https://guix.gnu.org/blog/2014/guix-at-the-2014-gnu-hackers-meeting//Guix at the 2014 GNU Hackers MeetingLudovic Courtès2014-10-11T00:00:00+02002014-10-11T00:00:00+0200 The Guix talk of this summer's GNU Hackers Meeting is now available (get the slides ). DOWNLOAD VIDEO (WebM, 60 minutes) It gives an introduction to Guix from a user's viewpoint, and covers topics such as features for GNU maintainers, programming interfaces, declarative operating system configuration, status of the GNU/Hurd port, and the new Emacs and Web interfaces---with a bunch of demos. Do not miss other fine talks from the GHM . Many thanks to everyone who took care of the video recordings. …<p>The Guix talk of this summer's GNU Hackers Meeting is now available (get the <a href="https://www.gnu.org/software/guix/guix-ghm-20140815.pdf">slides</a>).</p><video src="https://audio-video.gnu.org/video/ghm2014/2014-08--courtes--were-building-the-gnu-system--ghm.webm" controls="controls"><div class="action-box centered-text"><a class="button-big" href="https://audio-video.gnu.org/video/ghm2014/2014-08--courtes--were-building-the-gnu-system--ghm.webm">DOWNLOAD VIDEO</a><p>(WebM, 60 minutes)</p></div></video><p>It gives an introduction to Guix from a user's viewpoint, and covers topics such as features for GNU maintainers, programming interfaces, declarative operating system configuration, status of the GNU/Hurd port, and the new Emacs and Web interfaces---with a bunch of demos.</p><p>Do not miss <a href="http://audio-video.gnu.org/video/ghm2014/">other fine talks from the GHM</a>. Many thanks to everyone who took care of the video recordings.</p>https://guix.gnu.org/blog/2013/guix-at-the-european-lisp-symposium//Guix at the European Lisp SymposiumLudovic Courtès2013-05-21T00:00:00+02002013-05-21T00:00:00+0200 A paper presenting the design of Guix's Scheme API and packaging language has been accepted for the 2013 European Lisp Symposium (ELS) . ELS will take place in Madrid on June 3-4. …<div><p>A <a href="http://arxiv.org/abs/1305.4584">paper</a> presenting the design of Guix's Scheme API and packaging language has been accepted for the <a href="http://www-sop.inria.fr/members/Manuel.Serrano/conferences/els13.html">2013 European Lisp Symposium (ELS)</a>. ELS will take place in Madrid on June 3-4.<br /></p></div>