2.1 Email

Email build notifications can be enabled using the following record.

Data Type: email ¶

from

The email From field, as a string.

to

The email To field, as a string.

server

The mail server connection string. Cuirass uses the mailutils package. Hence the server can be specified as a remote SMTP mailbox (see SMTP Mailboxes in Mailutils) or as a program mailbox (see Program Mailboxes in Mailutils).

2.2 Mastodon

Mastodon build notifications can be enabled using the following record.

Data Type: mastodon ¶

The Mastodon credentials must be defined as Cuirass parameters, see Parameters.

2.3 RSS

Cuirass is proposing a build notification RSS feed at the following URL:

• ‘http://cuirass-url/events/rss[?specification=spec]’ By default build notifications are sent for all specifications. If the specification argument is passed, they can be restricted to the spec specification.

5.1 With the local Guix daemon

This is the default build mechanism. Once the build jobs are evaluated, they are sent to the local Guix daemon. Cuirass then listens to the Guix daemon output to detect the various build events.

While this mode doesn’t require any particular configuration, it doesn’t scale well. The scheduling decisions of the Guix daemon are opaque and often suboptimal.

When Cuirass is used to build a large amount of jobs, the remote build mechanism described below should be preferred.

5.2 With the remote build mechanism.

This mode is harder to setup but scales way better. This is the build mode that is used on the GNU Guix build farm at https://ci.guix.gnu.org. The build jobs are not submitted to the local Guix daemon. Instead, a remote server dispatches build requests to the connect remote workers, according to the build priorities.

The remote server and the connected workers communicate using ZMQ over TCP. The workers are able to discover the remote server using Avahi.

The built items are exchanged as substitutes (see Substitutes in Guix) by spawning Guix publish servers both on the remote server and on each connected remote worker.

It can be enabled this way:

• Start the cuirass register process with the build-remote command line argument, see Invocation. This way, the registration process does not submit the new build jobs to the local Guix daemon.
• Start the cuirass remote-server process to dispatch the build jobs to the connected workers.
• Start at least one cuirass remote-worker process on any machine of the local network to actually perform the builds and report their status.

Note that some Cuirass features are only available when using this build mode. That’s the case for:

• The build priority support.
• The notification mechanism, see Notifications.
• The transmission of timeout and max-silent-time package properties to the Guix daemon.
• The live build log mechanism of the Web interface.
• The workers status page of the Web inferface accessible at ‘http://cuirass-url/workers’.

The easiest way to setup such an infrastructure is to rely on the GNU Guix Cuirass services definitions (see Continuous Integration in Guix).

Alternatively, there is a systemd service unit file installed at $CUIRASS_PREFIX/lib/systemd/system/cuirass-remote-worker.service to launch the cuirass remote-worker on systems which use systemd. It can be installed and enabled as follows:

sudo install -o root -g root -m 644 \
     $CUIRASS_PREFIX/lib/systemd/system/cuirass-remote-worker.service \
     /etc/systemd/system
sudo systemctl enable cuirass-remote-worker.service
sudo systemctl start cuirass-remote-worker.service

6.1 Invoking cuirass register

The usual way to invoke cuirass registration process is as follows:

cuirass register --specifications specs

This starts a Cuirass registration instance building specs and storing the results using the default PostgreSQL database.

Additionally the following options can be used.

--one-shot

Instead of executing cuirass as a daemon looping over the jobs. Only evaluate and build the specifications once.

--cache-directory=directory

directory is the place where the VCS repositories used by the jobs are stored.

--specifications=specifications-file

-S specifications-file

Add the specifications defined in specifications-file in the job database before launching the evaluation and build processes.

--database=database

-D database

Use database as the database containing the jobs and the past build results. Since Cuirass uses PostgreSQL as a database engine, database must be a string such as "dbname=cuirass host=localhost". By default, Cuirass uses the following connection string: dbname=cuirass host=/var/run/postgresql".

--build-remote

Use the remote build mechanism, whereby build distribution is handled by a separate cuirass remote-server process (see below). This option can increase throughput and resilience when distributing builds to a large number of build machines.

When this option is omitted, builds are started using the “normal” Guix way, by asking guix-daemon to build them.

--parameters=parameters-file

-P parameters-file

Read parameters from the given parameters-file. The supported parameters are described here (see Parameters).

--ttl=duration

Cuirass registers build results as garbage collector (GC) roots, thereby preventing them from being deleted by the GC. The --ttl option instructs it to keep those GC roots live for at least duration—e.g., 1m for one month, 2w for two weeks, and so on. The default is 30 days.

Those GC roots are typically stored in /var/guix/gcroots/profiles/per-user/user/cuirass, where user is the user under which Cuirass is running.

--interval=n

-I n

Wait at most n seconds between each poll.

Note: We recommend notifying Cuirass when it should evaluate a jobset—e.g., because new code has been pushed to a channel’s Git repository—and passing a high value of n like 3600 (an hour): this will reduce network traffic as Cuirass will update its local checkouts only when needed, plus every n seconds “just in case”.

To trigger a jobset evaluation, issue an HTTP GET request, for example with a command along these lines:

wget --post-data="" -O /dev/null \
  https://cuirass.example.org/jobset/jobset/hook/evaluate

You would typically run that command as a push hook on the servers that host the Git repositories relevant to jobset.

Our course, you may only do this for repositories you control. For other repositories, periodic polling in unavoidable.

--threads=n

Use up to n kernel threads.

n should be lower than or equal to the number of CPU cores on the machine. In general though, having a large n is not very useful since the work of Cuirass is primarily I/O-bound—on the contrary, large values of n may increase overhead. The default value should be appropriate for most cases.

--version

-V

Display the actual version of cuirass.

--help

-h

Display an help message that summarize all the options provided.

6.2 Invoking cuirass web

The usual way to invoke the cuirass web server is as follows:

cuirass web

This starts a Cuirass web server on the default port. Additionally the following options can be used.

--database=database

-D database

Use database as the database containing the jobs and the past build results. Since Cuirass uses PostgreSQL as a database engine, database must be a string such as "dbname=cuirass host=localhost". By default, Cuirass uses the following connection string: dbname=cuirass host=/var/run/postgresql".

--parameters=parameters-file

-P parameters-file

Read parameters from the given parameters-file. The supported parameters are described here (see Parameters).

--port=num

-p num

Make the HTTP interface listen on port num. Use port 8080 by default.

--listen=host

Make the HTTP interface listen on network interface for host. Use localhost by default.

--version

-V

Display the actual version of cuirass.

--help

-h

Display an help message that summarize all the options provided.

6.3 Invoking cuirass remote-server

The remote-server command starts a daemon that is able to communicate with remote-worker processes; it is used in conjunction with cuirass register --build-remote (see above). Its role is to answer build requests from the workers, by sending back derivations that must be built.

On build completion it updates the database accordingly and possibly fetches build substitutes. The remote-server and remote-worker processes communicate using ZMQ over TCP.

Additionally the following options can be used.

--backend-port=port

The TCP port for communicating with remote-worker processes using ZMQ. It defaults to 5555.

--log-port=port

The TCP port of the log server. It defaults to 5556.

--publish-port=port

The TCP port of the publish server. It defaults to 5557.

--parameters=parameters-file

-P parameters-file

Read parameters from the given parameters-file. The supported parameters are described here (see Parameters).

--database=database

-D database

Use database PostgreSQL connection string.

--cache=directory

Use directory to cache build log files.

--log-expiry=duration

Periodically delete build logs older than duration, where ‘2m’ means “2 months”, ‘10d’ means “10 days”, and so on. The default duration is 6 months.

--trigger-substitute-url=URL

Once a substitute is successfully fetched, trigger substitute baking at URL.

--user=user

Change privileges to user as soon as possible—i.e., once the signing key has been read.

--no-publish

Do not start a publish server and ignore the publish-port argument. This can be useful if there is a standalone publish server standing next to the remote server.

--public-key=file

--private-key=file

Use the specific files as the public/private key pair used to sign the store items being published.

--version

-V

Display the actual version of cuirass.

--help

-h

Display an help message that summarize all the options provided.

6.4 Invoking cuirass remote-worker

The remote-worker command starts a daemon that is able to communicate with a remote-server process. Its role is to request builds to the remote-server, perform them and report their status.

The remote-worker is able to discover a remote-server process on the local network using Avahi and connect to it.

Additionally the following options can be used.

--workers=count

Start count parallel workers. It defaults to 1.

--publish-port=port

The TCP port of the publish server. It defaults to 5558.

--server=ip-address

Do not use Avahi discovery and connect to the given remote-server IP address.

--systems=systems

Only request builds for the given systems. It defaults to (list (%current-system)).

substitute-urls (default: %default-substitute-urls)

The list of URLs where to look for substitutes by default.

--public-key=file

--private-key=file

Use the specific files as the public/private key pair used to sign the store items being published.

--version

-V

Display the actual version of cuirass.

--help

-h

Display an help message that summarize all the options provided.

8.1 Evaluation information

Single evaluation

It is possible to query the Cuirass web server for evaluation information. The dedicated API is "/api/evaluation?id=eval-id" where eval-id is the unique id associated to the evaluation in database.

For instance, querying a local Cuirass web server can be done with curl and jq to format the JSON response :

$ curl -s "http://localhost:8080/api/evaluation?id=1" | jq

{
  "id": 1,
  "specification": "master",
  "status": 0,
  "timestamp": 1615289011,
  "checkouttime": 1615289011,
  "evaltime": 1615289655,
  "checkouts": [
    {
      "commit": "bd311f5a6ccbd4696ac77f0426a036bb375a2f10",
      "channel": "guix",
      "directory": "/gnu/store/6978xw9vs4ybg2pc3g736p1dba2056yl-guix-bd311f5"
    }
  ]
}

The nominal output is a JSON object whose fields are described hereafter.

id

The unique build id.

specification

The associated specification name, as a string.

status

The evaluation status, as an integer. Possible values are :

-1 -> started
0 -> succeeded
1 -> failed
2 -> aborted

checkouttime

The timestamp after channel checkout.

evaltime

The timestamp after evaluation completion.

checkouts

The evaluation checkouts as a JSON object.

Multiple evaluations

The latest evaluations list can be obtained with the API "/api/evaluations". The output is a JSON array of evaluations. Evaluations are represented as in the "/api/evaluation?id=eval-id" API.

This request accepts a mandatory parameter and an optional one.

nr

Limit query result to nr elements. This parameter is mandatory.

spec

Only consider evaluations that are part of the given spec specification.

8.2 Triggering an Evaluation

One can send an HTTP POST request to trigger the evaluation of a jobset, along these lines:

wget --post-data="" -O /dev/null \
  https://cuirass.example.org/jobset/jobset/hook/evaluate

A good idea is to do that from the post-push hook of the relevant Git repository.

8.3 Build information

It is possible to query Cuirass web server for build informations. The dedicated API is "/build/build-id" where build-id is the unique id associated to the build in database.

The build information can also be queried by output. For example, ‘/output/kg9mirg6xbvzcp0a98v7326n1nvvwgsj-hello-2.10’ will return the details of the output, along with the build if available.

$ curl -s "http://localhost:8080/build/2" | jq

{
  "id": 2,
  "jobset": "guix",
  "job": "acpica-20150410-job",
  "timestamp": 1501347493,
  "starttime": 1501347493,
  "stoptime": 1501347493,
  "buildoutputs": {
    "out": {
      "path": "/gnu/store/6g3njhfzqpdm335s7qhvmwvs5l7gcbq1-acpica-20150410"
    }
  },
  "system": "x86_64-linux",
  "nixname": "acpica-20150410",
  "buildstatus": 0,
  "weather": 0,
  "busy": 0,
  "priority": 0,
  "finished": 1,
  "buildproducts": null
}

If requested build-id is not known, the HTTP code 404 is answered with a JSON error message. For example:

$ curl -s "http://localhost:8080/build/fff"

{"error" : "Build with ID fff doesn't exist."}

The nominal output is a JSON object whose fields are described hereafter.

id

The unique build id.

jobset

The associated specification name, as a string.

job

The associated job-name, as a string.

timestamp

Timestamp taken at build creation time.

starttime

Timestamp taken at build start time.

stoptime

Timestamp taken at build stop time.

buildoutputs

Build outputs as a JSON object. The keys names are referring to the eventual output names. The associated value is another JSON object which only key is path. path value is the output directory in store as a string.

system

System name of the build, as a string.

nixname

Derivation name, as a string.

buildstatus

Build status, as an integer. Possible values are :

0 -> succeeded
1 -> failed
2 -> failed dependency
3 -> failed other
4 -> cancelled

weather

Build weather, as an integer.

-1 -> unknown
0 -> new-success
1 -> new-failure
2 -> still-succeeding
3 -> still-failing

busy

Whether the build is pending, as an integer.

priority

Build priority, as an integer.

finished

Build finished, as an integer.

buildproducts

Build products in store as a JSON object.

8.4 Build raw log output

It is possible to ask Cuirass for the raw build output log with the API "/build/build-id/log/raw" where build-id is the unique id associated to the build in database.

The output is a raw text, for example:

$ curl http://localhost:8080/build/2/log/raw

starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
...

If requested build-id is not known, the HTTP code 404 is answered with a JSON error message. For example:

$ curl -s "http://localhost:8080/build/fff/log/raw"

{"error" : "Build with ID fff doesn't exist."}

8.5 Jobs

The list of jobs associated with a given evaluation can be obtained with the API "/api/jobs". The output is a JSON array of jobs.

This request accepts a mandatory parameter and multiple optional ones.

evaluation

The evaluation id. This parameter is mandatory.

names

Filter query result to jobs which names are part of the given names list, a comma separated list of job names.

system

Filter query result to jobs with the given system.

For example, to ask for the jobs of evaluation 12 for x86_64-linux:

$ curl "http://localhost:8080/api/jobs?evaluation=12&system=x86_64-linux"

or the emacs and emacs-minimal jobs of evaluation 12 for x86_64-linux:

$ curl "http://localhost:8080/api/jobs?evaluation=12&names=emacs.x86_64-linux,emacs-minimal.x86_64-linux"

The nominal output is a JSON object whose fields are described hereafter.

build

The unique build id associated with the job.

status

The build status, as an integer.

name

The job name, as a string.

8.6 Jobs history

The history of jobs across the last evaluations of a given specification can be optained with the API "/api/jobs/history".

This request accepts three mandatory parameters.

specification

The specification name. This parameter is mandatory.

names

Filter query result to jobs which names are part of the given names list, a comma separated list of job names. This parameter is mandatory.

nr

Limit query result to nr elements. This parameter is mandatory.

For example, to ask for the history of emacs.x86_64-linux and emacs-minimal.x86_64-linux jobs of the master specification over the last 10 evaluations:

$ curl "http://localhost:8080/api/jobs/history?spec=master&names=emacs.x86_64-linux,emacs-minimal.x86_64-linux&nr=10"

The nominal output is a JSON array which objects have the following field:

evaluation

The unique evaluation id.

checkouts

The evaluation checkouts as a JSON array.

name

The jobs list for this evaluation, as a JSON array.

8.7 Dashboard registration

The user can register a dashboard using the "/api/dashboard/register" API. This request accepts two mandatory parameters.

specification

The specification name. This parameter is mandatory.

names

The dashboard jobs names, where names is a comma separated list of job names. This parameter is mandatory.

For example, to register a dashboard for the emacs.x86_64-linux and emacs-minimal.x86_64-linux jobs of the master specification:

$ curl "http://localhost:8080/api/dashboard/register?spec=master&names=emacs.x86_64-linux,emacs-minimal.x86_64-linux"

The nominal output is a JSON object which contains a unique field:

id

The registered dashboard id.

The dashboard is then accessible at the following address: http://localhost:8080/dashboard/<id>.

8.8 Latest builds

The list of latest builds can be obtained with the API "/api/latestbuilds". The output is a JSON array of builds. Builds are represented as in the "/build/build-id" API.

This request accepts a mandatory parameter and multiple optional ones.

nr

Limit query result to nr elements. This parameter is mandatory.

jobset

Filter query result to builds with the given jobset.

job

Filter query result to builds with the given job name.

system

Filter query result to builds with the given system.

For example, to ask for the ten last builds:

$ curl "http://localhost:8080/api/latestbuilds?nr=10"

or the five last builds where jobset “guix”:

$ curl "http://localhost:8080/api/latestbuilds?nr=5&jobset=guix"

If no builds matching given parameters are found, an empty JSON array is returned.

8.9 Queued builds

The list of queued builds can be obtained with the API "/api/queue". The output is a JSON array of builds. Builds are represented as in the "/build/build-id" API.

This request accepts a mandatory parameter.

nr

Limit query result to nr elements. This parameter is mandatory.

9.1 Specifications

This table stores specifications describing the repositories from whence Cuirass fetches code and the environment in which it will be processed. Entries in this table must have values for the following text fields:

name

This field holds the name of the specification. This field is also the primary key of this table.

channels

The channels to be fetched by Cuirass as an SEXP string.

build_ouputs

The build outputs to be saved by Cuirass as an SEXP string.

notifications

The build notifications to be sent by Cuirass as an SEXP string.

period

The specification periodicity as an integer.

priority

The specification priority relatively to the other specifications, as an integer ranging from 0 to 9 where 0 is the higher priority and 9 the lowest.

systems

The systems for which build jobs must be evaluated, as a comma separated list.

9.2 Checkouts

When a specification is processed, the repositories must be downloaded at a certain revision as specified. The download is called a checkout. The Checkouts table stores the new checkouts for every specification when it is being processed.

The Checkouts table has the following columns:

specification

The specification associated with the checkout.

revision

The revision of the checkout. Within the same specification, two checkouts can’t be identical: they can’t have the same revision.

evaluation

The evaluation that was triggered by the addition of that new checkout.

channel

The channel associated with the checkout.

directory

The directory into which the checkout was extracted.

timestamp

The checkout insertion timestamp.

9.3 Evaluations

An evaluation relates a specification with the revision of the repository specified therein. Builds (see below) belong to a specific evaluation.

The Evaluations table has the following columns:

id

This is an automatically incrementing numeric identifier.

specification

This field holds the name of a specification from the Specifications table.

status

This integer field hold the evaluation status. Possible values are:

• started (-1)
• succeeded (0)
• failed (1)
• aborted (2)

timestamp

The timestamp at evaluation insertion.

checkout

The timestamp after channel checkout.

evaltime

The timestamp after evaluation completion.

9.4 Jobs

This table contains all the jobs associated with a given evaluation. If a new job produces build outputs that are not already stored inside the Outputs table then, it is added to the Builds table.

name

This text field holds the job name.

evaluation

This integer field references the evaluation identifier from the Evaluations table, indicating to which evaluation this job belongs.

build

This integer field references the build id from the Builds table, corresponding to this job.

status

This integer field holds the build status of the above build. The goal of this duplication is to avoid expensive joins between the Jobs and Builds table.

system

This text field holds the system name of the above build. This field is duplicated for the same reason as the above status field.

9.5 Builds

This table holds records of the derivations and their build status. Note that a job will be registered here only if its derivation doesn’t already exist.

derivation

This text field holds the absolute name of the derivation file that resulted in this build.

evaluation

This integer field references the evaluation identifier from the Evaluations table, indicating to which evaluation this build belongs.

job_name

This text field holds the name of the job.

system

This text field holds the system name of the derivation.

nix_name

This text field holds the name of the derivation —e.g., coreutils-8.24.

worker

This text field references the name of worker performing the build from the Workers table.

log

This text field holds the absolute file name of the build log file.

status

This integer field holds the build status of the derivation.

last_status

This integer field holds the build status of the previous job evaluation.

weather

This integer field holds the weather of the build. Possible values are:

• unknown (-1)
• new-success (0)
• new-failure (1)
• still-succeeding (2)
• still-failing (3)

priority

The build priority relatively to the other builds with the same job_name, as an integer ranging from 0 to 99 where 0 is the higher priority and 99 the lowest.

max_silent

This integer field holds the number of seconds of silence after which a build process times out.

timeout

This integer field holds the number of seconds of activity after which a build process times out.

timestamp

This integer field holds a timestamp taken at build creation time.

starttime

This integer field holds a timestamp taken at build start time. Currently, it has the same value as the timestamp above.

stoptime

This integer field holds a timestamp taken at build stop time. Currently, it has the same value as the timestamp above.

9.6 BuildDependencies

This table contains the dependencies between the builds registered in the Builds table.

source

This integer field holds a reference to the build id from the Builds table.

target

This integer field holds a reference to the build id from the Builds table, indicating that the source build id depends from the target build id.

9.7 Outputs

This table keep tracks for every eventual build outputs. Each build stored in Builds table may have zero (if it has failed), one or multiple outputs.

derivation

This field holds the derivation of a build from the Builds table.

name

This text field holds the name of the output.

path

This text field holds the path of the output.

9.8 Metrics

This table contains several metrics that are recorded by the metrics fiber periodically.

field

This text field holds the application field of the metric.

type

This integer field holds the type of the metric.

path

This float field holds the value of the metric.

evaltime

The metric insertion timestamp.

9.9 BuildProducts

This table contains the saved build products, that are proposed to download through the web interface.

build

This integer field holds a reference to the build id from the Builds table, the build product belongs to.

type

This text field holds the build product type.

file_size

This integer field holds build product size in bytes.

checksum

This text field holds the build product checksum.

path

This text field holds the build product absolute store path.

9.10 Notifications

This table contains the notifications that are queued for sending.

id

This is an automatically incrementing numeric identifier.

type

This text field holds the SEXP representation of the notification.

build

This integer fields references the build id associated with the notification.

9.11 Workers

This table contains the registered workers when Cuirass is using the remote building mechanism.

name

This text field holds the worker name. This field is also the primary key of this table.

address

This text field holds the worker IP address.

machine

This text field holds the worker machine name.

systems

This text field holds the systems that are supported by the worker, as a comma separated list of systems.

last_seen

This integer field holds the timestamp of the last communication with the worker.

9.12 Dashboards

This table contains the user registered Dashboards.

id

This is an automatically incrementing numeric identifier.

specification

This field holds the name of a specification from the Specifications table.

jobs

This text field holds a list of comma separated job names.