Do the binaries provided by this server really correspond to the source
code it claims to build? Is a package build process deterministic?
These are the questions the
guix challenge command attempts to
The former is obviously an important question: Before using a substitute server (see Substitutes), one had better verify that it provides the right binaries, and thus challenge it. The latter is what enables the former: If package builds are deterministic, then independent builds of the package should yield the exact same result, bit for bit; if a server provides a binary different from the one obtained locally, it may be either corrupt or malicious.
We know that the hash that shows up in /gnu/store file names is
the hash of all the inputs of the process that built the file or
directory—compilers, libraries, build scripts,
etc. (see Introduction). Assuming deterministic build processes,
one store file name should map to exactly one build output.
guix challenge checks whether there is, indeed, a single
mapping by comparing the build outputs of several independent builds of
any given store item.
The command output looks like this:
$ guix challenge --substitute-urls="https://ci.guix.gnu.org https://guix.example.org" updating list of substitutes from 'https://ci.guix.gnu.org'... 100.0% updating list of substitutes from 'https://guix.example.org'... 100.0% /gnu/store/…-openssl-1.0.2d contents differ: local hash: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q https://ci.guix.gnu.org/nar/…-openssl-1.0.2d: 0725l22r5jnzazaacncwsvp9kgf42266ayyp814v7djxs7nk963q https://guix.example.org/nar/…-openssl-1.0.2d: 1zy4fmaaqcnjrzzajkdn3f5gmjk754b43qkq47llbyak9z0qjyim differing files: /lib/libcrypto.so.1.1 /lib/libssl.so.1.1 /gnu/store/…-git-2.5.0 contents differ: local hash: 00p3bmryhjxrhpn2gxs2fy0a15lnip05l97205pgbk5ra395hyha https://ci.guix.gnu.org/nar/…-git-2.5.0: 069nb85bv4d4a6slrwjdy8v1cn4cwspm3kdbmyb81d6zckj3nq9f https://guix.example.org/nar/…-git-2.5.0: 0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73 differing file: /libexec/git-core/git-fsck /gnu/store/…-pius-2.1.1 contents differ: local hash: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax https://ci.guix.gnu.org/nar/…-pius-2.1.1: 0k4v3m9z1zp8xzzizb7d8kjj72f9172xv078sq4wl73vnq9ig3ax https://guix.example.org/nar/…-pius-2.1.1: 1cy25x1a4fzq5rk0pmvc8xhwyffnqz95h2bpvqsz2mpvlbccy0gs differing file: /share/man/man1/pius.1.gz … 6,406 store items were analyzed: - 4,749 (74.1%) were identical - 525 (8.2%) differed - 1,132 (17.7%) were inconclusive
In this example,
guix challenge first scans the store to
determine the set of locally-built derivations—as opposed to store
items that were downloaded from a substitute server—and then queries
all the substitute servers. It then reports those store items for which
the servers obtained a result different from the local build.
As an example,
guix.example.org always gets a different answer.
ci.guix.gnu.org agrees with local builds, except in the
case of Git. This might indicate that the build process of Git is
non-deterministic, meaning that its output varies as a function of
various things that Guix does not fully control, in spite of building
packages in isolated environments (see Features). Most common
sources of non-determinism include the addition of timestamps in build
results, the inclusion of random numbers, and directory listings sorted
by inode number. See https://reproducible-builds.org/docs/, for
To find out what is wrong with this Git binary, the easiest approach is to run:
guix challenge git \ --diff=diffoscope \ --substitute-urls="https://ci.guix.gnu.org https://guix.example.org"
This automatically invokes
diffoscope, which displays detailed
information about files that differ.
Alternatively, we can do something along these lines (see Invoking guix archive):
$ wget -q -O - https://ci.guix.gnu.org/nar/lzip/…-git-2.5.0 \ | lzip -d | guix archive -x /tmp/git $ diff -ur --no-dereference /gnu/store/…-git.2.5.0 /tmp/git
This command shows the difference between the files resulting from the
local build, and the files resulting from the build on
ci.guix.gnu.org (see Comparing and Merging Files in Comparing and Merging Files). The
works great for text files. When binary files differ, a better option
is Diffoscope, a tool that helps
visualize differences for all kinds of files.
Once you have done that work, you can tell whether the differences are due
to a non-deterministic build process or to a malicious server. We try
hard to remove sources of non-determinism in packages to make it easier
to verify substitutes, but of course, this is a process that
involves not just Guix, but a large part of the free software community.
In the meantime,
guix challenge is one tool to help address
If you are writing packages for Guix, you are encouraged to check
ci.guix.gnu.org and other substitute servers obtain the
same build result as you did with:
$ guix challenge package
where package is a package specification such as
The general syntax is:
guix challenge options [packages…]
When a difference is found between the hash of a locally-built item and that of a server-provided substitute, or among substitutes provided by different servers, the command displays it as in the example above and its exit code is 2 (other non-zero exit codes denote other kinds of errors).
The one option that matters is:
Consider urls the whitespace-separated list of substitute source URLs to compare to.
Upon mismatches, show differences according to mode, one of:
Show the list of files that differ.
Invoke Diffoscope, passing it two directories whose contents do not match.
When command is an absolute file name, run command instead of Diffoscope.
Do not show further details about the differences.
Thus, unless --diff=none is passed,
downloads the store items from the given substitute servers so that it
can compare them.
Show details about matches (identical contents) in addition to information about mismatches.