Childhurds and GNU/Hurd substitutes

Jan (janneke) Nieuwenhuizen, Ludovic Courtès, Mathieu Othacehe — 8. oktober, 2020

A lot has happened since our Hello Hurd post beginning of April. No, not nearly as much as we joked on April 1st , but more than enough to share and be proud of.

Building a Hurd virtual machine

As some of you noticed, the previous hacks to build a Hurd virtual machine (VM) were removed and no longer work; using Guix you can now build a GNU/Hurd VM just like you would build a GNU/Linux VM:

guix system disk-image -t hurd-raw bare-hurd.tmpl

This cross-compiles all the relevant packages for GNU/Hurd—specifically the i586-pc-gnu triplet—and produces a VM image:

/gnu/store/n7jkfajw0fzp975hv0b9v18r9bbr961q-disk-image

You can build it and start it from your GNU/Linux machine with this command:

qemu-system-i386 -enable-kvm -m 512 -snapshot -hda \
  $(guix system disk-image -t hurd-raw bare-hurd.tmpl)

We are using this ready-made, minimal GNU/Hurd operating system description gnu/system/examples/bare-hurd.tmpl that looks suprisingly familiar:

(use-modules (gnu) (gnu system hurd) (guix utils))
(use-service-modules ssh)
(use-package-modules ssh)

(define %hurd-os
  (operating-system
    (inherit %hurd-default-operating-system)
    (bootloader (bootloader-configuration
                 (bootloader grub-minimal-bootloader)
                 (target "/dev/sdX")))
    (file-systems (cons (file-system
                          (device (file-system-label "my-root"))
                          (mount-point "/")
                          (type "ext2"))
                        %base-file-systems))
    (host-name "guixygnu")
    (timezone "Europe/Amsterdam")
    (packages (cons openssh-sans-x %base-packages/hurd))
    (services (cons (service openssh-service-type
                             (openssh-configuration
                              (openssh openssh-sans-x)
                              (use-pam? #f)
                              (port-number 2222)
                              (permit-root-login #t)
                              (allow-empty-passwords? #t)
                              (password-authentication? #t)))
               %base-services/hurd))))

%hurd-os

and it can be customized just like a GNU/Linux operating system description. The end result is a full-blown Guix System with the Shepherd managing system services and all that—finally we can run herd on the Hurd.

A lot of things had to be in place to support this, we worked on

• Adding multiboot support to grub-minimal-bootloader; a nice standard that Linux sadly does not support,
• Support for the Hurd and cross-building of the core packages listed in %base-packages/hurd,

• Support for the Hurd and cross-building of the system services in %base-services/hurd,

  • Add hurd-console-service,

  • Add hurd-getty-service,

  • Add hurd-default-essential-services,

  • Add Hurd support to %boot-service

  • Add a hurd-startup-service,

  • Add Hurd support to activation-service,

  • Add Hurd support for “syscalls”: define-c-struct, ioctl, readdir*, sockaddr-in, getxattr, setxattr,

counting some ~200 patches by ten people over six months; including generic cross-compilation fixes and support, and Hurd fixes and support.

Also, we finished the passive translator settings over extended attributes (xattrs) for the Hurd and for Linux.

You may notice that we are using the new disk-image command rather than the old vm-image. One of the big hurdles in producing a VM image was the way Guix produces VM images: it would run a target QEMU, e.g. qemu-arm. That does not work for the Hurd, as there is no qemu-hurd. Without going into the hairy details, when Ludo and Janneke were—three patch sets, 50 messages and 13 days later—almost ready to give up, Mathieu came to the rescue with his brand-new implementation of the disk-image command. At the time, Hurd work was done on the wip-hurd branch and the disk-image work on wip-disk-image. Soon after, Mathieu proposed an explosive mix of the two branches; we managed to create the first Hurd system that really felt like Guix System.

The new implementation of the disk-image command was followed by the introduction of an --image-type or -t option. This option allows to produce disk images targeting different supports. The hurd-raw and hurd-qcow2 image types, producing respectively a raw Hurd disk-image and a Hurd QCOW2 disk-image were introduced. They can be used this way:

guix system disk-image -t hurd-raw bare-hurd.tmpl
guix system disk-image -t hurd-qcow2 bare-hurd.tmpl

This mechanism providing much more flexibility in the Guix System image generation will be described in a future blog post.

We also offer downloads of continuously built (actually cross-built) Guix System on GNU/Hurd (~350 MiB).

Substitutes

While amazing to be able to just run the Hurd in a VM, development without substitutes is a real pain; a Guix system needs substitutes. How to go about that? We have a build farm but currently Hurd only runs on ancient hardware: not an option. We need a machine running Guix/Hurd to offload Hurd build jobs to.

The proposed solution was to automate the building and running of a Guix VM into a new Guix service: the so-called hurd-vm or childhurd service. We would add something like:

(service hurd-vm-service-type)

to a couple of our build nodes to add a hird of virtual Hurd machines to our build farm.

The Childhurd service

The Hurd—being based on a microkernel—has some beautiful built-in "virtualization" possibilites that have their own naming: neighborhurds, subhurds. Similarly, we are adding childhurds to this mix: a childhurd is a GNU/Hurd VM running on GNU/Linux and managed by Guix.

When you are running Guix System, building a Hurd VM manually is no longer necessary. Just add the hurd-vm service to your operating system description:

(service hurd-vm-service-type
         (hurd-vm-configuration
          (disk-size (* 12 (expt 2 30))) ;12GiB
          (memory-size 1024)))           ; 1GiB

and this will build a childhurd for you when you reconfigure your system. This childhurd can be stopped and started just like any other service:

herd stop hurd-vm
herd start childhurd

(childhurd is an alias for hurd-vm).

WARNING

It is highly addictive.

Having shaved this yak, let’s not lose sight that our initial goal was to offload builds to those GNU/Hurd VMs. The childhurd service does all the heavy lifting. To offload from GNU/Linux to my childhurd, I added this to my /etc/guix/machines.scm:

 (build-machine
  (name "localhost")
  (systems (list "i586-gnu"))
  (host-key "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHZsrZ63zs+AhWbVJgYq6j1h2rgQGrWKCokpR2/Q/Jzy root@guixygnu")
  (port 10022)  ;the Hurd VM has SSH listening on that port
  (user "root")
  (private-key "/home/janneke/.ssh/id_rsa_childhurd"))

That can be used to transparently offload builds from GNU/Linux to the childhurd:

$ uname -o
GNU/Linux
$ guix build hello -s i586-gnu
The following derivation will be built:
   /gnu/store/jqdvjhcxxnbq370y8i2c973c9zfiqrgl-hello-2.10.drv
[…]
guix offload: sending 1 store item (12 MiB) to 'localhost'...
offloading '/gnu/store/jqdvjhcxxnbq370y8i2c973c9zfiqrgl-hello-2.10.drv' to 'localhost'...
[…]
/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
$ file -L $(guix build hello -s i586-gnu)/bin/hello
/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10/bin/hello: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /gnu/store/9vs3gkp6svam82zw7vjlml7iiarcs11c-glibc-2.31/lib/ld.so.1, for GNU/Hurd 0.0.0, not stripped

Hurrah!

Hurd substitutes

Last Friday we produced the very first GNU Hello substitute for the Hurd:

StorePath: /gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
URL: nar/gzip/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
Compression: gzip
FileSize: 61822
URL: nar/lzip/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
Compression: lzip
FileSize: 52887
NarHash: sha256:0g4k1kppjs5148ynm4zw4x1kpaby67npc3ws6s7y7hf0il1cgryk
NarSize: 204328
References: 803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10 9vs3gkp6svam82zw7vjlml7iiarcs11c-glibc-2.31 bwvd5338kfm0vsc4i9xvh48vdxr5ywrz-gcc-7.5.0-lib
System: i586-gnu
Deriver: jqdvjhcxxnbq370y8i2c973c9zfiqrgl-hello-2.10.drv
Signature: 1;berlin.guix.gnu.org;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjMDVGOEY5NjMxRUU5QzcxM0REQUNBRTYwNUNCNjJBNzlDNUY4NEVFQTIwMjc5OERBNTQ3NURCOUU2Q0FBRDMwMSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDVBMzkzMTgwOUY1RkQyMTdGMDM4MUVDMTJEODYyNzIyOEYyNjJGRDA4MTcxQjREMzZBNEM0RjBBNjZEQkY4NSMpCiAgIChzICMwQTc2RjZGNENCOTMzQTczNzA4QkNGMzRGREExMzkyOTRGQTQxREQzQTUwQkEwOUE0ODRCQUQyOTA4MjQ5ODIxIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjOEQxNTZGMjk1RDI0QjBEOUE4NkZBNTc0MUE4NDBGRjJEMjRGNjBGN0I2QzQxMzQ4MTRBRDU1NjI1OTcxQjM5NCMpCiAgICkKICApCiApCg==

For development, porting and fixing of packages, you can use a Childhurd configuration like this:

(use-modules (srfi srfi-1) (guix packages) (guix records))
(use-package-modules base compression file gawk gdb hurd less m4 package-management)

(define guix-packages
  (filter-map input->package
              (fold alist-delete (package-direct-inputs guix)
                    '("glibc-utf8-locales" "graphviz" "po4a"))))

(operating-system
  (inherit %hurd-vm-operating-system)
  (users ...)
  (packages
    (cons* diffutils file findutils gawk gdb-minimal git-minimal
           gnu-make grep gzip less m4 openssh-sans-x tar xz
           (append
            guix-packages
            (delete guile-3.0 %base-packages/hurd)))))

… which allows working from a Git clone:

15:51:05 janneke@dundal:~/src/guix/master [env]
$ ssh -A janneke@childhurd
Last login: Sat Oct  3 15:31:55 2020 from 10.0.2.2


  This is the GNU Hurd.  Welcome.

janneke@childhurd ~$ git config --global url."git+ssh://git.sv.gnu.org/srv/git/".insteadOf gnu:
janneke@childhurd ~$ git clone gnu:guix
Cloning into 'guix'...
remote: Counting objects: 394436, done.
remote: Compressing objects: 100% (85728/85728), done.
remote: Total 394436 (delta 309572), reused 392294 (delta 307893)
Receiving objects: 100% (394436/394436), 137.05 MiB | 1.18 MiB/s, done.
Resolving deltas: 100% (309572/309572), done.
Updating files: 100% (2199/2199), done.

but before we continue let's first check the weather:

janneke@childhurd ~$ guix weather
computing 11079 package derivations for i586-gnu...
looking for 11521 store items on https://ci.guix.gnu.org...
updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
https://ci.guix.gnu.org
  1.5% substitutes available (169 out of 11521)
  at least 443.3 MiB of nars (compressed)
  966.7 MiB on disk (uncompressed)
  0.012 seconds per request (142.2 seconds in total)
  81.0 requests per second
[..]

this gives an idea of how young this project is. Any day now, more Hurd substitutes will follow. Now let's configure and build Guix:

janneke@childhurd ~$ cd guix
janneke@childhurd ~/guix$ guix environment --bootstrap \
  --ad-hoc gcc-toolchain@7 libgcrypt zlib
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
The following derivations will be built:
   /gnu/store/gxq5flc8kwpn999dw9xxvldy9xfd0q2x-profile.drv
   /gnu/store/vfkjnwgl4ckyklrl2z4q8x2vnlrwwyfr-gcc-toolchain-7.5.0.drv
   /gnu/store/zh6snj49ayrpw24jn7whpzygj1fpy9cm-module-import-compiled.drv

66.3 MB will be downloaded
[..]
building profile with 3 packages...
janneke@childhurd ~/guix [env]$ ./bootstrap
[..]
janneke@childhurd ~/guix [env]$ ./configure --with-courage\
  --localstatedir=/var --sysconfdir=/etc
[..]
janneke@childhurd ~/guix [env]$ make
[..]
janneke@childhurd ~/guix [env]$ ./pre-inst-env guix build hello
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
0.1 MB will be downloaded:
   /gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10
substituting /gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10...
downloading from https://ci.guix.gnu.org/nar/lzip/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10 ...
 hello-2.10  52KiB                    258KiB/s 00:00 [##################] 100.0%

/gnu/store/803q5wapfnmr91ag8d9dzwabkbdxz3ay-hello-2.10

just like we are used to do…almost. We are using --bootstrap and a targeted --ad-hoc to avoid dependencies like libx11, python-minimal, and other packages that do not build yet.

Isolated build environments

To help achieve reproducible builds, Guix builds packages in isolated build environments: build environments contain nothing but the inputs explicitly declared in the package definition—not even /bin/sh. Build environments also lack network access. On GNU/Linux this is achieved by running builds in separate namespaces. Besides, the environment contains device nodes and “special” file systems that are usually expected to be available: /dev/null, /dev/pts, /dev/shm, /proc, the loopback networking device, and so on. (The exact contents are documented.)

On GNU/Linux, these special files and file systems are implemented by the kernel. Guix only cares about user-land software, meaning that these devices “leak” from the host kernel instead of being an explicit “input” of derivations, but that’s OK, that’s the deal: the kernel and hardware are considered outside of Guix’s control.

What about GNU/Hurd, though? In GNU/Hurd, /dev/null, a Linux-compatible /proc, the TCP/IP stack necessary to implement the loopback device, and even support for pipes are all implemented in user-land: writing to /dev/null amounts to talking to the /hurd/null service (or translator), operations on AF_INET sockets translate to remote procedure calls (RPCs) to /servers/socket/2, which the /hurd/pfinet program listens to, and so on.

That raises an interesting question: what should the build environment contain on GNU/Hurd? So far our GNU/Hurd builds were made in non-isolated environments; we have just started implementing support for isolated builds but we’ll have to answer that question first. If we stick to our approach—every piece of user-land software must be an explicit input of the build process—then code that implements TCP/IP, /dev/null, or even pipe should be an explicit input of any build process that needs those facilities.

This principled approach can push the notion of controlled, reproducible build environments to a whole new level. For example, we’ve had cases where the choice of the root file system—e.g., ext4 vs. Btrfs—has an observable effect on software behavior, leading to concrete issues such as test failures in one case and not in the other. On GNU/Hurd, build processes could run their own root file system, doing away with this kind of discrepancy.

On the other hand, there are practical issues that cannot be ignored: virtually all build processes need these facilities so they’ll need to be set up one way or another. Also, one could argue that things like /dev/null have a well-defined interface that’s set in stone and that, consequently, how they’re implemented does not matter at all. Can we say the same of the TCP/IP stack though? Maybe not. A line needs to be drawn somewhere.

We have yet to decide where to draw the line and to precisely define what the build environment contains on GNU/Hurd. These questions are closely related to bootstrapping issues we notably discussed at the 2019 Reproducible Builds Summit. Tricky, but exciting.

What's next?

In an earlier post we tried to answer the question “Why bother with the Hurd anyway?” An obvious question because it is all too easy to get discouraged, to downplay or underestimate the potential social impact of GNU and the Hurd.

We tried to make Hurd development as easy and as pleasant as we could. As you have seen, things start to work pretty nicely and there is still plenty of work to do in Guix. But in a way this is “merely packaging” the amazing work of others. Some of the real work that needs to be done and which is being discussed and is in progress right now includes:

• user-space driver/modern hardware support
• audio support (this is sponsored by NLnet, thanks!),
• SMP support,
• 64-bit/ x86_64 support (!), as well as porting to non-Intel CPUs.

All these tasks look daunting, and indeed that’s a lot of work ahead. But the development environment is certainly an advantage. Take an example: surely anyone who’s hacked on device drivers or file systems before would have loved to be able to GDB into the code, restart it, add breakpoints and so on—that’s exactly the experience that the Hurd offers. As for Guix, it will make it easy to test changes to the micro-kernel and to the Hurd servers, and that too has the potential to speed up development and make it a very nice experience.

Join #guix and #hurd on libera.chat or the mailing lists and get involved!

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

About the GNU Hurd

The GNU Hurd is the GNU project's replacement for the Unix kernel. It is a collection of servers that run on the Mach microkernel to implement file systems, network protocols, file access control, and other features that are implemented by the Unix kernel or similar kernels (such as Linux). More info.

The mission of the GNU Hurd project is to create a general-purpose kernel suitable for the GNU operating system, which is viable for everyday use, and gives users and programs as much control over their computing environment as possible.