Guix

QA on non-Intel at Guix Days

Efraim Flashner — February 7, 2019

During the second day of Guix Days (a FOSDEM fringe event) we split up into smaller working groups based on our areas of interest. I led a group which aimed to address some of the package issues which exist on non-Intel architectures. Of course not everyone has access to an ARM board, but with the qemu-binfmt-service service it is possible to use QEMU and the binfmt_misc functionality of the Linux kernel to emulate these systems. Many have reported that this system emulation is comparable in speed to many of the available ARM boards on the market. Yet another possibility would be to do the hacking on an x86_64 system and, when we had a working prototype, to test it with QEMU or on actual ARM hardware.

Our group decided to tackle Go, which was lacking support in Guix on armhf and aarch64. Upon checking the build logs from Cuirass and the source code for Go we determined that Go did indeed require the gold linker from the GNU Binutils. We didn't want to modify the copy of Binutils in Guix since it is part of our bootstrap story, so we quickly put together a new package definition which added the configure flag to enable gold.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     (substitute-keyword-arguments (package-arguments binutils)
       ((#:configure-flags flags)
        `(cons "--enable-gold=default" ,flags))))))

This was an obvious first step, and one which we knew would fail. Had it been this easy gold would have been enabled back in 2012 when it was first added. Our error came in the form of one of the binaries not being able to link against libstdc++.so, which is in the gcc:lib output. This was quickly added and we were off and building again.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     (substitute-keyword-arguments (package-arguments binutils)
       ((#:configure-flags flags)
        `(cons "--enable-gold=default" ,flags))))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

Once again this failed. What were we missing? The correct paths were included, the file was indeed in the gcc:lib output. We inspected the original binutils package again noticed that it was built against a static libgcc, so of course it wouldn't find the shared library. In order to work quickly we copied the configure flags rather than inheriting them from binutils and tried our build again.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     (substitute-keyword-arguments (package-arguments binutils)
       ((#:configure-flags flags)
        `(cons* "--enable-gold=default"
                "--enable-new-dtags"
                "--with-lib-path=/no-ld-lib-path"
                "--enable-install-libbfd"
                "--enable-deterministic-archives"))))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

This time we made it through the full build phase and we knew we were almost there. Our enthusiasm was quickly dampened when we got the error during the tests: unable to find the 'dc' program. What is this dc program? This isn't any package any of us had heard of before. It definitely wasn't packaged in Guix. A quick apt-cache search dc search in Ubuntu showed they didn't have package either. A second search of Ubuntu, apt-file search dc | grep '/bin/dc' quickly showed us it was in the bc package, and soon we were building binutils-gold again.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     (substitute-keyword-arguments (package-arguments binutils)
       ((#:configure-flags flags)
        `(cons* "--enable-gold=default"
                "--enable-new-dtags"
                "--with-lib-path=/no-ld-lib-path"
                "--enable-install-libbfd"
                "--enable-deterministic-archives"))))
    (native-inputs
     `(("bc" ,bc)))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

Approaching the end of the check phase we soon ran into another error, there was an unpatched /bin/sh somewhere in the source code which was generated during the check phase. Based on the build logs we were able to track down approximately where the code should be, so we downloaded the source tar xf $(guix build --source binutils) and started looking. There were many obvious /bin/sh calls which we cross-referenced with the build logs and the patch-source-shebangs phase, and this left us with some code in gold/Makefile.in, which by default is not included in the patch-source-shebangs and would need to be fixed manually.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'patch-source-shebangs 'patch-more-shebangs
           (lambda _
             (substitute* "gold/Makefile.in"
               (("/bin/sh") (which "sh")))
             #t)))
       ,@(substitute-keyword-arguments (package-arguments binutils)
         ((#:configure-flags flags)
          `(cons* "--enable-gold=default"
                  "--enable-new-dtags"
                  "--with-lib-path=/no-ld-lib-path"
                  "--enable-install-libbfd"
                  "--enable-deterministic-archives")))))
    (native-inputs
     `(("bc" ,bc)))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

One more build cycle later and we did it! /gnu/store/…-binutils-gold-2.31.1 existed! We now did two things, we copied our patch over to an aarch64 build machine and we started cleaning up our package definition on our x86_64 build machine, where we knew we had a working package definition.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'patch-source-shebangs 'patch-more-shebangs
           (lambda _
             (substitute* "gold/Makefile.in"
               (("/bin/sh") (which "sh")))
             #t)))
       ,@(substitute-keyword-arguments (package-arguments binutils)
         ((#:configure-flags flags)
          `(cons* "--enable-gold=default"
                  (delete "LDFLAGS=-static-libgcc" ,flags))))))
    (native-inputs
     `(("bc" ,bc)))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

Fortunately for us the changes in the code worked on x86_64 and we still got a working binutils-gold output. On our aarch64 side the build was progressing nicely and everything seemed fine... until we suddenly were presented with big red errors about unrelocatable code. How could it? Everything was working so well! Undeterred, we built the source again, this time targeting armhf and were unfortunately presented with similar errors. Deciding to address the test failures later (It's ARM! It's not as well tested as other architectures! Right?) we disabled the tests and unsurprisingly binutils-gold built on both aarch64 and armhf.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'patch-source-shebangs 'patch-more-shebangs
           (lambda _
             (substitute* "gold/Makefile.in"
               (("/bin/sh") (which "sh")))
             #t)))
       ,@(substitute-keyword-arguments (package-arguments binutils)
         ((#:tests? _ #f) #f)
         ((#:configure-flags flags)
          `(cons* "--enable-gold=default"
                  (delete "LDFLAGS=-static-libgcc" ,flags))))))
    (native-inputs
     `(("bc" ,bc)))
    (inputs
     `(("gcc:lib" ,gcc "lib")))))

Now for the real test. Due to bootstrapping issues with Go and aarch64, aarch64 uses Go@1.4 built for armhf. Go@1.11 failed to build until now because it was missing the gold linker. Surely using the gold linker would be a good test if our package worked. Since Go for aarch64 is 'more complex' due to the bootstrapping using armhf's Go, we decided to test armhf first. binutils-gold was added and our build started.

    (native-inputs
     `(("go" ,go-1.4)
+      ,@(match (%current-system)
+          ((or "armhf-linux" "aarch64-linux")
+           `(("gold" ,binutils-gold)))
+          (_ `()))
       ,@(package-native-inputs go-1.4)))

First build, success! /gnu/store/…-go-1.11.5 exists! OK, but does it actually work? guix build syncthing --system=armhf-linux. /gnu/store/…-syncthing-1.0.0 exists too! A quick check of guix refresh --list-dependent go@1.4 showed that we had unlocked 176 new packages for armhf. Even better, since they had all failed by default due to go@1.11 failing to build, for each package that did build meant one fewer package which failed to build which should take a big bite out of our build failures.

Our next test was syncthing for aarch64. /gnu/store/…-go-1.11.5 exists! /gnu/store/…-syncthing-1.0.0 ... does not. "unknown architecture 'armv7-a'." It seems that Go is confused which architecture it is building for. Unfortunately we were reaching the end of our time for hacking, so that will have to wait for another day. All that was left now was the test failures on binutils-gold for the ARM systems. Some attempts at cargo-culting other code failed (per-architecture tests we had and overriding flags in substitute-keyword-arguments we had, but not together), but after some attempts we were able to create a working package definition we were happy with.

(define-public binutils-gold
  (package
    (inherit binutils)
    (name "binutils-gold")
    (arguments
     `(#:phases
       (modify-phases %standard-phases
         (add-after 'patch-source-shebangs 'patch-more-shebangs
           (lambda _
             (substitute* "gold/Makefile.in"
               (("/bin/sh") (which "sh")))
             #t)))
       ,@(substitute-keyword-arguments (package-arguments binutils)
         ; Upstream is aware of unrelocatable test failures on arm*.
         ((#:tests? _ #f)
          (if (any (cute string-prefix? <> (or (%current-target-system)
                                               (%current-system)))
                   '("i686" "x86_64"))
              '#t '#f))
         ((#:configure-flags flags)
          `(cons* "--enable-gold=default"
                 (delete "LDFLAGS=-static-libgcc" ,flags))))))
     (native-inputs
      `(("bc" ,bc)))
     (inputs
      `(("gcc:lib" ,gcc "lib")))))

This patch was pushed to the master branch as 28317d499034b00cf1f08a9efd39bd2bc3425b19, and the commit following uses it as a native-input for Go@1.9 and Go@1.11. Go@1.4 was added in June 2016 and Go@1.6 that August, with our first go packages being added in October 2017. That same October Go@1.4 had support limited to Intel and armhf and in October 2018, in an effort to work toward a resolution, a patch was added to have aarch64 use Go@1.4 built for armhf for it's bootstrap path. Basically since the addition of the Go language support into Guix there was not a time when it was usable on armhf or aarch64. Hopefully we will soon finish getting full Go support on aarch64 and we can move all 352 dependents of Go@1.4 from "failing" to "succeeding" and have these architectures better supported.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the kernel Linux, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.