OCaml Weekly News

Over the past year we in Robur have been working towards easing deployment of reproducible mirage applications. The work has been funded by the Eurepean Union under the Next Generation Internet (NGI Pointer) initiative. The result is available as a website.

The overall goal is to push MirageOS into production in a trustworthy way. We worked on reproducible builds for MirageOS - with the infrastructure being reproducible itself. A handful of core packages are hosted there (described below in this article), next to several ready-to-use MirageOS unikernels - ranging from authoritative DNS servers (secondary, let's encrypt DNS solver), DNS-and-DHCP service (similar to dnsmasq), TLS reverse proxy, Unipi - a web server that delivers content from a git repository, DNS resolver, CalDAV server, and of course your own MirageOS unikernel.

Reproducible builds are crucial for supply chain security - everyone can reproduce the exact same binary (by using the same sources and environment), without reproducible builds we would not publish binaries.

Reproducible builds are also great for fleet management: by inspecting the hash of the binary that is executed, we can figure out which versions of which packages are in the unikernel - and suggest updates if newer builds are available or if a used packages has a security flaw in that version – albatross-client-local update my-unikernel is everything needed for an update.

In the following, we'll explain in more detail two scenarios: how to deploy MirageOS unikernels using the infrastructure we provide, how to bootstrap and run the infrastructure for yourself. Afterwards we briefly describe how to reproduce a package, and what are our core packages and their relationships.

MirageOS is an operating system, developed in OCaml, which produces unikernels. A unikernel serves a single purpose and is a single process, i.e. only has the really needed dependencies. For example, an OpenVPN endpoint does neither include persistent storage (block device, file system) nor user management. MirageOS unikernels are developed in OCaml, a statically typed and type-safe programming language - which avoids common pitfalls from the grounds up (spatial and temporal memory safety issues).

Robur is a collective that develops MirageOS and OCaml software with open source license. It was started in 2017, and is part of the non-profit company center for the cultivation of technology. We received funding from several projects (prototypefund, NGI pointer), donations, and some commercial contracts.

To run a MirageOS unikernel on your laptop or computer with virtualization extensions (VT-x - KVM/BHyve), you can first install solo5-hvt as a package (take which fits your distribution), and albatross.

There is no configuration needed, you should start the albatross_console and the albatross_daemon service (via systemctl daemon-reload ; systemctl start albatross_daemon on Linnux or service albatross_daemon start on FreeBSD). Executing albatross-client-local info ~ should return success (exit code 0) and no running unikernel. You may need to be in the albatross group, or change the permissions of the Unix domain socket (~vmmd.sock in /run/albatross/util/ on Linux, /var/run/albatross/util/ on FreeBSD).

The fundamental tools for building in a reproducible way are orb and builder. On some distributions we provide binary packages (orb, builder) that you can use. On other distributions you'll need to bootstrap them from source:

• To build in a reproducible way, we developed orb, which is written in OCaml. It is an opam package available at https://github.com/roburio/orb (installation via opam pin add orb https://github.com/roburio/orb.git) - once you have OCaml and opam installed.
• To build builder, opam install builder is all you need to do. opam install builder-web will install the latest version of builder-web.

From a build on https://builds.robur.coop, select an operating system and distribution that has been used for a build. Go to the specific build, and download the "system-packages" file – these are the exact versions of host system packages that were used during the build. Make sure they're installed (version variance may lead to non-reproducibility - orb and builder are not needed for a manual rebuild).

Download the build-environment file, which contains all environment variables that were set during the build. Set these, and only these, in your shell.

Install opam (at least in version 2.1). Then, download the opam-switch file - which includes all opam files and dependencies (including the OCaml compiler). Execute opam switch import opam-switch --switch reproduced-unikernel which will create a fresh opam switch where it will install the unikernel. This will be located in ~opam switch prefix~/bin/unikernel.hvt.

We have enhancements and more features planned in the future. At the same time we are looking for feedback of the reproducible build and unikernel deployment system (with a security perspective, with a devops perspective, etc.). We are also keen to collaborate and would take new people on board.

• Improving the web UI on https://builds.robur.coop/. If you're interested, please get in touch, we have funding available.
• Supporting more distributions: tell us your favourite distribution and how to build a package, then we can integrate that into our reproducible builds infrastructure.
• Supporting spt - the sadboxed process tender - to run unikernels without a hypervisor.
• Data analytics: which system packages updates or opam package releases result in variance of the binaries - did the release of an opam package increase or decrease the overall build times?
• Functional and performance tests of the unikernels: for each different build, conduct basic functional testing, and performance test - to graph in the ouput. Also includes data analytics: did the release of an opam package increase or decrease the performance of unikernels?
• Whole system performance analysis with memory profiling, and how to integrate this into a running unikernel.
• MirageOS 4.0 support.
• Metrics and logging collection and dynamic adjustment of metrics and log levels.
• DNS resolver unikernel, still missing DNSSec support.

Interested? Get in touch with us via eMail to team at robur dot coop.

Knowing that winter is coming, I tied some loose ends in preparation for release 4.14, including more deprecation warnings #10675, proper termination of signal handling #10726, and increasing the native stack size limit when the operating system allows #10736. The latter should mitigate the problem of “Stack Overflow” crashing non-tail-recursive code for large inputs that hit operating-system restrictions.

I also worked on reimplementing the Random standard library module using more modern pseudo-random number generation (PRNG) algorithms. In RFC#28, Gabriel Scherer proposed to change the random-number generation algorithm of the standard library Random module to be "splittable", to offer better behavior in a Multicore world. ("Splitting" a random-number generator state gives two separate states that supposedly produce independent streams of random numbers; few RNG algorithms support splitting, and its theory is not well-understood.)

My first proposal was based on the Xoshiro256++ PRNG, which is fast and statistically strong: #10701. However, Xoshiro does not support full splitting, only a limited form called "jumping", and the discussion showed that jumping was not enough. Then a miracle happened: at exactly the same time (OOPSLA conference in october 2021), Steele and Vigna proposed LXM, a family of PRNGs that have all the nice properties of Xoshiro and support full splitting. I promptly reimplemented the Random module using LXM #10742, and I find the result very nice. I hope this implementation will be selected to replace the existing Random module.

Gabriel Scherer (@gasche) finished working on the TMC (Tail modulo constructor) PR (#9760) in time for the glaciation deadline, thanks to a well-placed full-day meeting with Pierre Chambart (@chambart), who had done the last review of the work. They managed to get something that we both liked, and the feature is now merged upstream.

Note that this is the continuation of the TRMC work started by Frédéric Bour (@let-def) in #181 in May 2015 (also with major contributions from Basile Clément (@Elarnon)); this merge closed one of the longest-open development threads for the OCaml compiler.

One may now write:

let[@tail_mod_cons] rec map f = function
| [] -> []
| x::xs -> f x :: (map[@tailcall]) f xs

and get an efficient tail-recursive definition of map.

A section of the manual is in progress to describe the feature: #10740.

(On the other hand, there was no progress on the constructor-unboxing work, which will have to wait for 5.0.)

As part of RFC#15: Fast native toplevel using JIT, there was a batch of small changes on native-code emission and linking, and on the native toplevel proposed by @NathanRebours and David @dra27: #10690, #10714, #10715.

Ulysse Gérard, Thomas Refis and Leo White proposed a new program analysis within the OCaml compiler, designed to help external tools understand the structure of implementation files (implementations of OCaml modules), in particular to implement the "locate definition" function – which is non-trivial in presence of include, open, etc.

The result of their analysis is a "shape" describing the items (values, types, etc.) of a module in an easy-to-process yet richly-structured form.

Florian Angeletti (@Octachron) allowed to merge this PR thanks to his excellent review work, running against the Glaciation deadline.

(The authors of the PR initially wanted to add new kinds of compilation artifacts for OCaml compilation units to store shape information in .cms and .cmsi files, instead of the too-large .cmt files. People were grumpy about it, so this part was left out for now.)

In #10710 support for UTF-X decoding and validation was added by Daniel Bünzli (@dbuenzli), a long-standing missing feature of the standard library. The API was carefully designed to avoid allocations and exceptions while providing an easy-to-use decoding interface.

The type Seq.t of on-demand (but non-memoized) sequences of values was contributed by Simon Cruanes (@c-cube) in 2017, with only a minimal set of function, and increased slowly since. A large import of >40 functions was completed just in time before the glacation by François Potter (@fpottier) and Simon, thanks to reviews by @gasche, @dbuenzli and many others. This is work that started in February 2020 thanks to issue #9312 from Yawar Amin.

Behold:

val is_empty : 'a t -> bool
val uncons : 'a t -> ('a * 'a t) option
val length : 'a t -> int
val iter : ('a -> unit) -> 'a t -> unit
val fold_left : ('a -> 'b -> 'a) -> 'a -> 'b t -> 'a
val iteri : (int -> 'a -> unit) -> 'a t -> unit
val fold_lefti : (int -> 'b -> 'a -> 'b) -> 'b -> 'a t -> 'b
val for_all : ('a -> bool) -> 'a t -> bool
val exists : ('a -> bool) -> 'a t -> bool
val find : ('a -> bool) -> 'a t -> 'a option
val find_map : ('a -> 'b option) -> 'a t -> 'b option
val iter2 : ('a -> 'b -> unit) -> 'a t -> 'b t -> unit
val fold_left2 : ('a -> 'b -> 'c -> 'a) -> 'a -> 'b t -> 'c t -> 'a
val for_all2 : ('a -> 'b -> bool) -> 'a t -> 'b t -> bool
val exists2 : ('a -> 'b -> bool) -> 'a t -> 'b t -> bool
val equal : ('a -> 'b -> bool) -> 'a t -> 'b t -> bool
val compare : ('a -> 'b -> int) -> 'a t -> 'b t -> int
val init : int -> (int -> 'a) -> 'a t
val unfold : ('b -> ('a * 'b) option) -> 'b -> 'a t
val repeat : 'a -> 'a t
val forever : (unit -> 'a) -> 'a t
val cycle : 'a t -> 'a t
val iterate : ('a -> 'a) -> 'a -> 'a t
val mapi : (int -> 'a -> 'b) -> 'a t -> 'b t
val scan : ('b -> 'a -> 'b) -> 'b -> 'a t -> 'b t
val take : int -> 'a t -> 'a t
val drop : int -> 'a t -> 'a t
val take_while : ('a -> bool) -> 'a t -> 'a t
val drop_while : ('a -> bool) -> 'a t -> 'a t
val group : ('a -> 'a -> bool) -> 'a t -> 'a t t
val memoize : 'a t -> 'a t
val once : 'a t -> 'a t
val transpose : 'a t t -> 'a t t
val append : 'a t -> 'a t -> 'a t
val zip : 'a t -> 'b t -> ('a * 'b) t
val map2 : ('a -> 'b -> 'c) -> 'a t -> 'b t -> 'c t
val interleave : 'a t -> 'a t -> 'a t
val sorted_merge : ('a -> 'a -> int) -> 'a t -> 'a t -> 'a t
val product : 'a t -> 'b t -> ('a * 'b) t
val map_product : ('a -> 'b -> 'c) -> 'a t -> 'b t -> 'c t
val unzip : ('a * 'b) t -> 'a t * 'b t
val split : ('a * 'b) t -> 'a t * 'b t
val partition_map : ('a -> ('b, 'c) Either.t) -> 'a t -> 'b t * 'c t
val partition : ('a -> bool) -> 'a t -> 'a t * 'a t
val of_dispenser : (unit -> 'a option) -> 'a t
val to_dispenser : 'a t -> (unit -> 'a option)
val ints : int -> int t

Dong An (@kirisky) finished a left-open PR from Anukriti Kumar (#9398, #10666) to complete the documentation of the OCAMLRUNPARAM variable.

Dong An also improved the README description of which C compiler should be available on MacOS or Windows to build the compiler codebase: #10685.

Thanks to Wiktor Kuchta, the ocaml toplevel now shows a tip at startup about the #help directive to get help: #10527. (Wiktor is not really a "new" contributor anymore, with many nice contributions over the last few months.)

While we are at it, a PR from @sonologico, proposed in May 2020, was merged just a few months ago (#9621). It changes the internal build system for the ocamldebug debugger to avoid module-name clashes when linking user-defined printing code. Most of the delay came from maintainers arguing over which of the twelve name-conflict-avoidance hacks^Wfeatures should be used.

When registering a user an attestation certificate is optionally provided. The attestation certificate is a X509 certificate signing the public key sent during registration and contains information such as the maker and model(family) of the authenticator. At the moment only fido-u2f and and none are supported.

The work was sponsored by skolem.tech

• Increase the allowed artifact cache size from 5GB to 10GB.

https://github.com/ocaml/setup-ocaml/releases/tag/v2.0.0-beta9

• The next version (0.3.0) includes at least one breaking change (upgrade of ocamlformat from 0.8.0 to 0.9.0)

builds any project easily with just one command:

opam build

The command will setup a local switch and install all the required dependencies, then build the project (as described in the opam files). So no matter your build system, calling opam build should be enough.

opam-test does the same thing as opam-build but runs the tests on top of it. Just call:

opam test

to get started.

opam-test also circumvents issues with cyclic test dependencies in opam (where the tests require a package that needs the library it is trying to test). Such cyclic dependency is present in packages such as odoc or base. See https://github.com/ocaml/opam/issues/4594

If your package is in such a case, opam-test allows you to use the post variable to make that work. For instance:

In project A:

run-test: ["dune" "build "-p" name "-j" jobs] {post}
depends: [
  "B" {with-test & post}
]

In project B:

depends: [
  "A"
]

Without the {post} variable, opam alone would not be able to run the tests as the tests are run before the installation of the package and this would lead to a cyclic dependency. opam-test allows you to split the installation from the tests and thus makes that work.

To install, simply call

opam update
opam install opam-build opam-test

If the package do not exist, they might not have been merged in opam-repository yet (watch for https://github.com/ocaml/opam-repository/pull/20085). In the meantime, instead you can install it directly from the source repository:

opam pin add git+https://github.com/kit-ty-kate/opam-build

Again, these plugins are highly experimental and are looking for use-cases. If you think this can be useful, feel free to send some feedbacks here or on the bugtracker: https://github.com/kit-ty-kate/opam-build/issues