Adding a fully-bootstrapped Mono

unmush — 2024-12-29

We used to have a Mono package. It was introduced on August 8 2016 by commit 763b3d50b6249b43fceda51445bbeb1f5f5fd7d0, at Mono version 4.4.1.0, but it was later discovered in April of 2022 that the release tarball that it was built from included prebuilt binaries. Further research revealed that these binaries were not optional. Due to this, a decision was made to remove the Mono package, carried out on September 1, 2022.

We now once again have a Mono package, due to a patch series I submitted on November 29, which after some revisions was committed on December 22. This patch series introduced a full, 17-mono-package sequence that takes us from a mono-1.2.6 built fully from source to mono-6.12.0 built fully from source, using only packages that already have full bootstrap paths. I make no promise that this is the shortest or most optimal path, but it exists and I have verified it works.

As I've spent what is probably an unreasonable amount of time working toward this, I thought I'd share some of my thoughts, experiences, and commentary. Sorry in advance if it gets a bit rambly or lecture-ish.

Prologue

I started down this road because someone I'm working on a project with decided to depend on a C# package that requires C# 12.0 features, and my personal Mono package based on the tarball releases (which include bootstrap binaries) only went up to C# 7.0. This meant that the C# package in question de facto required strictly Microsoft's (er, I mean, "the .NET foundation"'s) .NET implementation — hereafter referred to as "dotnet" — and a very recent version no less. The bootstrapping story with dotnet is very bad; even beginning to untangle it would probably require a relatively modern C# compiler, and something that at least sort of understands MSBuild. And there's not much point to "bootstrapping" dotnet from something that isn't bootstrapped itself. So I figured I may as well start with Mono.

History

While Mono is today probably the most well-known alternative to Microsoft's .NET offerings, it is not the only one. Indeed, in the early 2000s there were at least 2 competing free software implementations: Mono, and DotGNU's Portable.NET (abbreviated pnet). They differed in goals, licenses, and methods: Portable.NET was a GNU project concerned with, among other things, limiting the ability of Microsoft to impose vendor lock-in via its proprietary .NET implementation and software patents. As a GNU project, it used the GPL for its runtime and compiler, and the GPL with a linking exception for its standard library, pnetlib. Mono, on the other hand, used a mix of many copyleft and permissive licenses: X11 for the standard library, GPL for the compiler (later dual-licensed to add an X11 option), and LGPL for the runtime, with GPL and LGPL code also offered "under commercial terms for when the GPL and the LGPL are not suitable". In 2016 after its acquisition by Microsoft, the runtime was relicensed to use the Expat (MIT) license.

But perhaps most importantly to us, while Mono opted to write its C# compiler, mcs, in... C#, Portable.NET's runtime and C# compiler were both written in C. Portable.NET along with the entire DotGNU project (except for LibJIT) was decommissioned in 2012, but the source is still available, and it still works fine (with a few modifications for compatibility with newer versions of its dependencies). In September of 2022, Adam Faiz submitted patches to package pnet and pnetlib, along with one of their dependencies named treecc. These packages were based on the last release of Portable.NET, version 0.8.0, released in 2007. I initially used these packages as the basis for my bootstrap efforts, and even managed to get mono-1.2.6 built using them, but later discovered that using a more recent version from git made it much easier. For example, while pnet-0.8.0 can do pointer arithmetic inside unsafe code blocks, it doesn't support the += or -= operators specifically, which requires lots of patching (after all, who would use x = x + y when you could do x += y?). There are many other similar improvements in the git version, so for this patch series I've decided to go with pnet-git.

The start

After building mono-1.2.6, I tried a few later versions, and the third or fourth one would always fail with errors about missing methods. It turns out that the reason for this is that, contrary to what their marketing suggests, C# and Java are not "write once, run everywhere". This is because their compilers rely on the details of the libraries that the program will be run with at compile-time. This is used, for example, to do overload resolution. Suppose, for example, that a certain implementation of the == operator is present in version 1.0 of a library, and then in version 2.0 of a library a more specific implementation is introduced. Now code that is compiled against version 2.0 may instead automatically reference the more-specific implementation, as is in accordance with the rules of C#. But when it is run with version 1.0, it will fail because that implementation doesn't exist. In my case, for some reason the initial mcs and core libraries being built to compile the rest of Mono were being compiled against a 2.0 library and then run with a 1.0 library. It turns out that this was because mcs uses mono's code for producing assemblies (.NET dlls and exes), and mono decides which version to put in an assembly it writes based on "which runtime version" is being used, and that version is decided at startup based on... the version that was put in the assembly it is running. So for example, mono-1.9.1 would produce 2.0 assemblies because mono-1.2.6 produced 2.0 assemblies because pnet produced 2.0 assemblies. So I modified Mono's runtime in mono-1.9.1 to allow for this version to be overridden via environment variable, and set it to v1.1.4322, and things went a lot more smoothly after that.

From there on it was mostly the usual trial-and-error process of identifying where things had bitrotted. I made sure to unvendor libgc wherever possible, though eventually by mono-4.9.0 they explicitly dropped support in their configure script for using any libgc other than what was bundled, so at that point I switched to using their homebrewed sgen garbage collector.

A concerning development

Once I got to mono-2.11.4, though, things took a turn for the interesting: Mono started using git submodules, and the (recursive? #t) clones were all failing. It turns out that this is because their submodules reference github.com using the git:// protocol.

This is notable for a few reasons.

First, GitHub dropped support for the git:// protocol in 2021, so recursive clones won't work now. This means I have to explicitly list out every submodule, its commit, and its sha256 hash, for every Mono version until they switched to using http or https. mono-2.11.4 has only 4 submodules, but that doesn't last for long: by mono-4.9.0 it has 14 submodules. A significant portion of these patches is just listing these submodules and their hashes. It's a bit annoying.

The more concerning reason, though, is why GitHub dropped support for the git:// protocol: it is unencrypted and unauthenticated. This is mitigated somewhat by the use of sha-1 hashes to identify commits in the referenced submodules, putting a significant computational burden on anyone who would try to alter what was fetched corresponding to a given submodule. Significantly more risky, though, is the process of updating submodules that use git:// URLs. It is quite unlikely that a developer is going to independently clone one of the submodules over https, navigate to a desirable commit, copy the sha-1 hash, and manually update the submodule reference's commit. They're far more likely to run cd submodule; git pull; cd ..; git add submodule; git commit ... or an equivalent.

Of course, any changes a network man-in-the-middle might try to make here would still be reflected in the commit history, so even if a developer did that, they or any of their fellow committers could spot anything strange or malicious and point it out. Also, the changes couldn't be propagated to others trying to pull them who weren't on a network path containing the MITM because the potentially-malicious commit wouldn't be present in the real submodule's repository. So the transparency of git clearly showing changes to text files, combined with the fact that surely no git hosting platform would just allow arbitrary entities to make whatever commits they want accessible under any arbitrary repository URL, rather mitigate this security issue.

This usage of git:// URLs lasted all the way until September 28, 2021, when GitHub's removal of support for it forced the developers to change them to https.

Meanwhile, in reality

On November 28, 2016, Mono added a submodule named roslyn-binaries. Unsurprisingly, it included binary blobs for Microsoft's Roslyn compiler (which I believe had been open-sourced shortly prior). From here on, Mono's build system would default to using these binaries for building on little-endian systems (though another compiler could be specified with the --with-csc configure flag). I happen to know that it is extremely unlikely that many Mono developers used this configure flag. I know this because the 5.0 series is an absolute pain in the neck to build from source, because they consistently depend on new C# features before they implement them.

To go on a brief tangent: does anyone remember back when youtube-dl was temporarily taken down from GitHub due to the RIAA's DMCA request? Many were unhappy about that. One such unhappy person made news when they made the full contents of youtube-dl's repository available to access through the DMCA request repository. It turns out that there are many actions that one can take on GitHub that will make arbitrary commits available under arbitrary repository URLs.

So, in reality, for the span of time from November 28, 2016 to September 28, 2021, anybody sitting on the network path between GitHub and any Mono developer updating the roslyn-binaries submodule could decide on any arbitrary new commit to be used. Of course, merely inspecting the diff for the commit will reveal nothing of use, because the contents are binary blobs. And not only are these blobs those of a compiler, they are the blobs of a compiler that is sure to be used to compile another compiler, which will then be redistributed as an opaque, non-bootstrappable binary blob to be used for compiling other compilers.

You would be hard-pressed to find a more fertile breeding ground for Ken Thompson / Trusting Trust attacks. If every agent of the NSA (and whatever other agencies, including those of other countries, had access to the appropriate network traffic) somehow failed to capitalize on 6 years of opportunity to compromise an entire software ecosystem using only a basic MITM of unencrypted traffic, they deserve to be sacked. Whether such an attack actually occurred or not, this is a case study in carelessness and why bootstrappability is so important; discovering all this made me quite worried about having used a Mono version built from blobs previously, and has convinced me that, as time-wasting and tedious as this project has been, it is nevertheless probably an important one.

Another note on roslyn-binaries

If you're going to write a self-hosting compiler, the least you can do is keep it self-hosting. Deciding to write a self-hosting compiler is a valid choice, of course, with its own merits and demerits, but there is something bitterly poetic about Mono starting out requiring specifically Microsoft's C# compiler in order to build (Mono did its initial bootstrapping using Microsoft's proprietary csc), achieving independence through self-hosting, being acquired by Microsoft, and thereafter coming crawling back to Microsoft's C# compiler once more before eventually dying.

The funny thing is that it's not even necessary. The dependencies on new C# features are all in Mono's standard library (which increasingly borrowed code from Microsoft's corefx library), not in Mono's compiler.

More binary submodules?

Even before roslyn-binaries, there was binary-reference-assemblies, which contained prebuilt "reference" blobs for the various versions of the standard libraries. These exist, I assume, precisely because of the library incompatibility problems regarding overloading that I mentioned earlier. While later versions of Mono included sources and a build system for producing these reference binaries, mono-4.9.0 and earlier did not. Mono's build system still demanded something to install, though, so I told it to use the real standard library of the input Mono version. When I did get to a Mono version that at least claimed to support regenerating the reference binaries, I found that it didn't work with mcs due to differences in which libraries had to be referenced, so I had to patch it to add a bunch of references determined through trial and error.

The xunit-binaries submodule was also added sometime before mono-5.1.0. This dependency makes it impossible to run the full test suite without binary blobs. Presumably for this reason, Debian elects to only run tests within the mono/mini/ and mono/tests/ subdirectories. For my part, I've disabled all tests except for those of mono-6.12.0, the final version, limited to the two aforementioned subdirectories. This is because it would take extra time for the builds, because several of the tests depend on binary blobs bundled into the Mono repository itself (which my thorough cleaning of all dlls and exes from the sources removes), because a large chunk of the tests depend on binary blobs in xunit-binaries in later versions, and because "expect some test failures" is part of the Mono documentation and I don't have the time to figure out for the Mono developers every reason why each of 17 versions of their test suite is broken.

The long march through the 5.0s

The 5.0 series was when Microsoft acquired Mono, and it shows. You'll notice I needed to introduce pre- packages for various versions because in several cases a tagged release could not build the following tagged release. For that matter, they couldn't build the pre- package either, but it at least took fewer patches to get them working. The reason for this is that Mono added a dependency on Microsoft's corefx library source code, and it usually started using C# features well before mcs was able to compile them. Because of this, despite taking 8 versions to get from 1.2.6 to 4.9.0, it took another 8 versions to get through the 5.0 series, and 5 of them required nontrivial patching to massage the source into a form compilable by mcs.

The final stretch

Eventually I realized that the dependencies on new features were all coming from corefx, not from Mono's compiler. Consequently, the only reason for this particular bootstrap-hostile ordering of builds is that it happened to be the order the Mono devs committed things. So I just cherry-picked every commit I could find touching mcs/mcs (magit was quite useful for this) and applied it to 5.10.0 to produce what is essentially the 6.12.0 compiler, then used it to jump straight to building 6.12.0.

Use of this technique earlier on in the bootstrap process may be of interest to anyone looking to shorten the chain of packages.

The finishing touches

My initial goal was to package dotnet, and I had tried to progress toward that from mono-4.9.0 for a period, but with no success. During that time, though, I did encounter a bug in Mono's xbuild condition parser, which I wrote a patch for, and included in mono-6.12.0.

I also discovered that xbuild would wrongly complain about missing references even when the proper assemblies were in MONO_PATH or MONO_GAC_PREFIX, because xbuild would erroneously only consider the path /gnu/store/...mono-6.12.0/lib/mono/gac when looking for global assembly caches, completely ignoring MONO_GAC_PREFIX. So I wrote a patch to fix that, and included it in mono-6.12.0.

Having witnessed how much nicer it is to package things that use rpath / runpath than things that use environment variables (like python) and therefore require constant wrapping of executables and use of propagated-inputs, I devised a patch that would extend Mono's per-assembly config files to support a <runpath> element. For example, if you have a file /tmp/dir2/test2.exe, and there is also a file /tmp/dir2/test2.exe.config, and its contents are

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <runpath path="/tmp/dir1"/>
</configuration>

and it references test1.dll, it will first look for it at /tmp/dir1/test1.dll. Note that, of course, test1.dll still needs to be accessible to the compiler at compile-time through MONO_PATH or an explicitly-specified path passed on the mcs command line.

It is my hope that this feature will be of use to anybody interested in developing a build system.

Future work

Mono had several difficult points in bootstrapping and packaging, but at the end of the day it still met the basic description of a software package: well-defined environment-supplied inputs and sources, a user-supplied install prefix, and files installed under that prefix.

The dotnet world is an entirely different beast. The first step of most build systems I have encountered from that realm is downloading an entire toolchain, among other dependencies, as a binary blob. They heavily depend on the exact packages they specify being available exactly where they say to install them. There is no "install", there are no "install directories" to my knowledge. A build that doesn't contact nuget.org is an aberration. I am at a loss how to build these things, much less package them. I badly need help.

There are also some portability issues with the current bootstrap path. While Portable.NET can fall back to an interpreter written in C where LibJIT isn't supported, old versions of Mono have no such capability. Strictly speaking, there is some bitrotted code for an interpreter that used to work, but has stopped working by mono-1.2.6. It was left unmaintained until it was eventually removed in 2014, only to be revived in 2017. This poses a dilemma for anybody wanting to bootstrap Mono on a platform that wasn't supported by mono-1.2.6's JIT compiler. There are a number of possible ways to try resolving this, ranging from backporting the new interpreter, to fixing up the old one for every version prior to the new interpreter, to forward-porting the old compiler and class libraries to the new interpreter, etc.

The most interesting option, though, in my opinion, would be to port mcs to Portable.NET. This would achieve the intended portability, while also allowing individual builds to be much faster since we're only building mcs, not the runtime and class library each time. It would also allow us to make much bigger version jumps, since, as we discovered earlier, many of the new C# feature dependencies in Mono come from the class library rather than the compiler. Such a shortened bootstrap could also make the bootstrap path more appealing for other distributions to use instead of binaries.

Closing thoughts

"You wish now that our places had been exchanged. That I had died, and DotGNU had lived?"

"... Yes. I wish that."

Maintenance of Mono was recently transferred over to WineHQ. With that announcement this statement was placed at https://www.mono-project.com:

"We want to recognize that the Mono Project was the first .NET implementation on Android, iOS, Linux, and other operating systems. The Mono Project was a trailblazer for the .NET platform across many operating systems. It helped make cross-platform .NET a reality and enabled .NET in many new places and we appreciate the work of those who came before us."

I would like to clarify that, according to Miguel de Icaza himself, DotGNU "started working on the system about the same time". According to this DotGNU newsletter, Portable.NET began "in January 2001". While it's unclear exactly when Portable.NET reached various milestones, and the significance of the various milestones varies somewhat (for example, Mono probably does not care that Portable.NET also includes a Java and C compiler), I think that there is cause to dispute the claim that Mono was "the first" .NET implementation on Linux.

On a related note, if we haven't looked at the possibility of using Portable.NET in the Java bootstrap process, it may be worth visiting at some point.

Thank you to the DotGNU project, for the .NET implementation that made this bootstrap possible, Adam Faiz, for the initial packaging of it that let me jump straight in, the Mono project, for... Mono, and you, for your time.