Building packages targeting psABIs
Starting with version 2.33, the GNU C library (glibc) grew the capability to search for shared libraries using additional paths, based on the hardware capabilities of the machine running the code. This was a great boon for x86_64, which was first released in 2003, and has seen many changes in the capabilities of the hardware since then. While it is extremely common for Linux distributions to compile for a baseline which encompasses all of an architecture, there is performance being left on the table by targeting such an old specification and not one of the newer revisions.
One option used internally in glibc and in some other performance-critical
libraries is indirect functions, or
IFUNCs (see also
here)
The loader, ld.so
uses them to pick function implementations optimized for
the available CPU at load time. GCC's functional multi-versioning
(FMV) generates several
optimized versions of functions, using the IFUNC mechanism so the approprate
one is selected at load time. These are strategies which most
performance-sensitive libraries do, but not all of them.
With the --tune
using package
transformation
option, Guix implements so-called package
multi-versioning
which creates package variants using compiler flags set to use optimizations
targeted for a specific CPU.
Finally - and we're getting to the central topic of this post! - glibc since
version 2.33 supports another approach: ld.so
would search not just the
/lib
folder, but also the glibc-hwcaps
folders, which for x86_64 included
/lib/glibc-hwcaps/x86-64-v2
, /lib/glibc-hwcaps/x86-64-v3
and
/lib/glibc-hwcaps/x86-64-v4
, corresponding to the psABI micro-architectures
of the x86_64 architecture (psABI stands for processor supplement of the
application binary interface and refers to the
document that specifies
among other things those x86-64-v*
levels).
This means that if a library was compiled against
the baseline of the architecture then it should be installed in /lib
, but if
it were compiled a second time, this time using (depending on the build
instructions) -march=x86-64-v2
, then the libraries could be installed in
/lib/glibc-hwcaps/x86-64-v2
and then glibc, using ld.so
, would choose the
correct library at runtime.
These micro-architectures aren't a perfect match for the different hardware available, it is often the case that a particular CPU would satisfy the requirements of one tier and part of the next but would therefore only be able to use the optimizations provided by the first tier and not by the added features that the CPU also supports.
This of course shouldn't be a problem in Guix; it's possible, and even
encouraged, to adjust packages to be more useful for one's needs. The problem
comes from the search paths: ld.so
will only search for the glibc-hwcaps
directory if it has already found the base library in the preceding /lib
directory. This isn't a problem for distributions following the File System
Hierarchy (FHS), but for Guix we will need to ensure that all the different
versions of the library will be in the same output.
With a little bit of planning this turns out to not be as hard as it sounds.
Lets take for example, the GNU Scientific
Library, gsl, a math library which helps
with all sorts of numerical analysis. First we create a procedure to generate
our 3 additional packages, corresponding to the psABIs that are searched for in
the glibc-hwcaps
directory.
(define (gsl-hwabi psabi)
(package/inherit gsl
(name (string-append "gsl-" psabi))
(arguments
(substitute-keyword-arguments (package-arguments gsl)
((#:make-flags flags #~'())
#~(append (list (string-append "CFLAGS=-march=" #$psabi)
(string-append "CXXFLAGS=-march=" #$psabi))
#$flags))
((#:configure-flags flags #~'())
#~(append (list (string-append "--libdir=" #$output
"/lib/glibc-hwcaps/" #$psabi))
#$flags))
;; The building machine can't necessarily run the code produced.
((#:tests? _ #t) #f)
((#:phases phases #~%standard-phases)
#~(modify-phases #$phases
(add-after 'install 'remove-extra-files
(lambda _
(for-each (lambda (dir)
(delete-file-recursively (string-append #$output dir)))
(list (string-append "/lib/glibc-hwcaps/" #$psabi "/pkgconfig")
"/bin" "/include" "/share"))))))))
(supported-systems '("x86_64-linux" "powerpc64le-linux"))
(properties `((hidden? . #t)
(tunable? . #f)))))
We remove some directories and any binaries since we only want the libraries
produced from the package; we want to use the headers and any other bits from
the main
package. We then combine all of the pieces together to produce a
package which can take advantage of the hardware on which it is run:
(define-public gsl-hwcaps
(package/inherit gsl
(name "gsl-hwcaps")
(arguments
(substitute-keyword-arguments (package-arguments gsl)
((#:phases phases #~%standard-phases)
#~(modify-phases #$phases
(add-after 'install 'install-optimized-libraries
(lambda* (#:key inputs outputs #:allow-other-keys)
(let ((hwcaps "/lib/glibc-hwcaps/"))
(for-each
(lambda (psabi)
(copy-recursively
(string-append (assoc-ref inputs (string-append "gsl-" psabi))
hwcaps psabi)
(string-append #$output hwcaps psabi))
'("x86-64-v2" "x86-64-v3" "x86-64-v4"))))))))
(native-inputs
(modify-inputs (package-native-inputs gsl)
(append (gsl-hwabi "x86-64-v2")
(gsl-hwabi "x86-64-v3")
(gsl-hwabi "x86-64-v4"))))
(supported-systems '("x86_64-linux"))
(properties `((tunable? . #f)))))
In this case the size of the final package is increased by about 13 MiB, from 5.5 MiB to 18 MiB. It is up to you if the speed-up from providing an optimized library is worth the size trade-off.
To use this package as a replacement build input in a package
package-input-rewriting/spec
is a handy tool:
(define use-glibc-hwcaps
(package-input-rewriting/spec
;; Replace some packages with ones built targeting custom packages build
;; with glibc-hwcaps support.
`(("gsl" . ,(const gsl-hwcaps)))))
(define-public inkscape-with-hwcaps
(package
(inherit (use-glibc-hwcaps inkscape))
(name "inkscape-with-hwcaps")))
Of the Guix supported architectures, x86_64-linux and powerpc64le-linux can both benefit from this new capability.
Through the magic of newer versions of GCC and LLVM it is safe to use these
libraries in place of the standard libraries while compiling packages; these
compilers know about the glibc-hwcap
directories and will purposefully link
against the base library during build time, with glibc's ld.so
choosing the
optimized library at runtime.
One possible use case for these libraries is creating guix pack
s
of packages to run on other systems. By substituting these libraries it
becomes possible to crate a guix pack
which will have better performance than
a standard package used in a guix pack
. This works even when the included
libraries don't make use of the IFUNCs from glibc or functional
multi-versioning from GCC. Providing optimized yet portable pre-compiled
binaries is a great way to take advantage of this feature.
About GNU Guix
GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.
In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.
Unless otherwise stated, blog posts on this site are copyrighted by their respective authors and published under the terms of the CC-BY-SA 4.0 license and those of the GNU Free Documentation License (version 1.3 or later, with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts).