OCaml Weekly News

Previous Week Up Next Week


Here is the latest OCaml Weekly News, for the week of February 02 to 09, 2021.

Table of Contents

data-encoding.0.3: performances and streaming

Raphaël Proust announced

Guess what?! Version 0.3 of data-encoding is now available!

Data-encoding is a library for defining encodings, which can then be used to serialise and deserialise values to-and-from a binary or JSON representation.

let e : (string * int list) t =
  obj2 (req "name" string) (list uint8)
let v : (string * int list) =
  ("low", [0;1;2;3;4])
let s : string = Binary.to_string_exn e v
let j : Ezjson.t = Json.construct e v

Version 0.3 of the library is now available on opam. The code is distributed under MIT license and hosted on Gitlab: https://gitlab.com/nomadic-labs/data-encoding/. The documentation is available online: https://nomadic-labs.gitlab.io/data-encoding/data-encoding/Data_encoding/index.html

In addition to numerous miscellaneous improvements, this version brings two major changes.

  1. Support for streamed JSON serialisation.

    JSON serialisation for large values can be expensive. And in some context, prohibitively so. One such context is cooperative concurrency where serialising very large JSON values can block other on-going tasks.

    Data-encoding provides the necessary functions to serialise values as sequences of strings which can be consumed with appropriate yielding.

    let rec write seq = match seq () with
      | Seq.Nil ->
      | Seq.Cons (chunk, seq) ->
          Lwt_io.write oc chunk >>= fun () ->
          Lwt.pause () >>= fun () ->
          write seq
    let j = Json.construct_seq e v in
    let s = Json.string_seq_of_jsonm_lexeme_seq ~chunk_size_hint:512 j in
    write s
  2. Performance improvements.

    The serialisation and deserialisation of some encodings has been optimised a lot. On some encodings the performances have gotten competitive with Marshal.

    More details below.

According to the micro-benchmark below data-encoding.0.3 gets close to Marshal performances on the serialising and deserialising of Micheline values (S-EXP-like values used to represent smart-contracts on the Tezos blockchain).


The results are printed below. Notice the speed up:

  • for serialising it progressed from a 13.40× slow-down over Marshal to a 1.01× slow-down,
  • for deserialising it progressed from a 18.72× slow-down over Marshal to a 1.02× slow-down.

data-encoding 0.2

Estimated testing time 20s (2 benchmarks x 10s). Change using '-quota'.

Name Time R^2 Time/Run 95ci mWd/Run mjWd/Run Prom/Run Percentage Speedup
marshal_encode 1.00 102.89us -0.05us +0.06us 5.74kw 0.11w 0.11w 7.46% 1.00
binary_encode 1.00 1_378.70us -7.47us +8.73us 953.58kw 1_489.94w 1_489.94w 100.00% 13.40

Estimated testing time 20s (2 benchmarks x 10s). Change using '-quota'.

Name Time R^2 Time/Run 95ci mWd/Run mjWd/Run Prom/Run Percentage Speedup
marshal_decode 1.00 73.58us -0.62us +0.59us 7.59kw 0.16w 0.16w 5.34% 1.00
binary_decode 1.00 1_377.68us -4.26us +4.35us 951.99kw 1_464.25w 1_464.25w 100.00% 18.72

data-encoding 0.3

Name Time R^2 Time/Run 95ci mWd/Run mjWd/Run Prom/Run Percentage Speedup
marshal_encode 1.00 102.05us -0.15us +0.18us 5.74kw 0.11w 0.11w 99.40% 1.00
binary_encode 1.00 102.67us -0.39us +0.44us 37.55kw 2.55w 2.55w 100.00% 1.01

Estimated testing time 20s (2 benchmarks x 10s). Change using '-quota'.

Name Time R^2 Time/Run 95ci mWd/Run mjWd/Run Prom/Run Percentage Speedup
marshal_decode 1.00 72.21us -0.29us +0.33us 7.59kw 0.16w 0.16w 97.57% 1.00
binary_decode 1.00 74.00us -0.29us +0.31us 36.48kw 2.89w 2.89w 100.00% 1.02

Do not hesitate to open an issue on the project's issue tracker to let us know about encodings that are still too slow.

2020 at OCamlPro

OCamlPro announced

We've published a short review of our 2020 activities. It's available on our blog!

A short history of ReScript (BuckleScript)

Hongbo Zhang told

It takes time to write such a post for a non-native speaker like me, but I appreciate what the OCaml gives me and believe I am doing good for the community, so I decide to write a short history of ReScript and explain the motivations behind this project.

I like OCaml, that’s why I decided to go to PlClub@UPenn for further study after undergraduate (2011). I wrote lots of tutorials around OCaml and used to be one of the maintainers of camlp4, I was even honored to be listed as core maintaines of OCaml once.

However, the reality is brutal that it’s very difficult to find a decent job using OCaml if you are not interested in Finance or get unlucky with a Jane Street interview. Here a decent job, I mean you get paid as good as those Silicon Valley companies while using your favorite programming language.

JSOO started growing mature back in 2013, it is the first compiler which gets bootstrapped in the browser, truly impressive. However, even for OCaml experts like me, it is really hard to get started with JSOO. It is a black box to me that I can not read the generated JS code under the hood. If your code works under the bytecode but stops working in the browser. You have to do a random guess to do the debugging. These are the impressions I have back then.

Another story happened on my side is that I started to have lots of experience in JavaScript in daily jobs and love it. It’s the most beautiful dynamic language I have used – you have a decent IDE everywhere as long as it ships a browser. So the idea comes to my mind that why not make a compiler which compiles OCaml to idiomatic JS for the convenience of JS users?

The beauty of this idea is that OCaml will share the same runtime as JS so that interop is really easy and you can use your favorite language in your daily jobs. I also shared my ideas with JSOO devs: discuss: compiling rawlambda output to javascript · Issue #338 · ocsigen/js_of_ocaml (github.com)

The project was originally named OCamlScript but it has a name conflict with an existing package so that it is renamed into BuckleScript.

At a similar time, ReasonML syntax was released, it is a JS friendly syntax for people to take advantage of OCaml. The combination of ReasonML and BuckleScript is very natural. To avoid confusion to users, we call it ReasonML or projects under the ReasonML umbrella – but the heavy lifting is mostly done by BuckleScript.

The development of ReasonML syntax has slowed down since 2018. What worries me is that the remaining commits keep adding stuff that we don’t want. To maintain the compatibility with OCaml for each release, the ReasonML syntax keeps a snapshot for all versions of OCaml AST, the code gets bloated quickly that the parser is even larger than the compiler itself – this is hard to believe, it means that all my hard work to squeeze the performance will be wasted in the parser.

I wish the very few resources could be put into bug fixes instead. We communicated and realized that the top priority for ReasonML syntax is the compatibility for OCaml ecosystem while our top priority is providing the best dev experience for JS users.

It is open source projects and people can have different visions so that we decide a peaceful separation.

At a similar time, Iwan – one of the major contributors of ReasonML syntax, shows us a POC of a handwritten parser which supports the whole language. This is something I want but do not have time to do – a hand written parser for the best error message and better dev experience.

The original goal is to have the hand written parser 100% compatible with ReasonML syntax so that we can finish the transition, but it ends up being slightly different.

The current status of ReScript

Despite the changes of the syntax, the core gets improved day by day. It has over 11,000 commits in the last few years.

We are going to make a release of version 9, we believe it is currently the fastest compiler on the JS market which generates the most readable JS code and yields the smallest JS output size.

I do wish it could be a useful language for people who like OCaml but can use it in their daily JS work. It is a fierce competition in such a market. We have to compete with languages of huge successes: TypeScript. That’s why given a small team with very few resources, we are very focused and don't want to get distracted by anything else.

Multicore OCaml: Dec 2020 / Jan 2021

Anil Madhavapeddy announced

Welcome to a double helping of the multicore monthlies, with December 2020 and January 2021 bundled together (the team collectively collapsed into the end of year break for a well deserved rest). We encourage you to review all the previous monthly updates for 2020 which have been compiled by @shakthimaan, @kayceesrk, and me.

Looking back over 2020, we achieved a number of major milestones towards upstreaming multicore OCaml. The major highlights include the implementation of the eventlog tracing system to make debugging complex parallelism practical, the enormous rebasing of from OCaml 4.06 to 4.11, a chapter on parallel programming, the publication of "Retrofitting Parallelism onto OCaml" at ICFP 2020, the production use of the Sandmark benchmark, and the implementation of system threading integration. While all this was happening in the multicore code trees, the upstreaming efforts into mainline OCaml also went into full gear, with @xavierleroy leading the efforts from the core team to ensure that the right pieces went into various releases of OCaml with the same extensive code review as any other features get.

The end of 2020 saw enhancements and updates to the ecosystem libraries, with more tooling becoming available. In particular, we would like to thank:

  • @mattpallissard for getting merlin and dot-merlin-reader working with Multicore OCaml 4.10. This makes programming using OCaml Platform tools like the VSCode plugin much more pleasant.
  • @eduardorfs for testing the no-effect-syntax Multicore OCaml branch with a ReasonML project.

@kayceesrk also gave a couple of public talks online:

We're really grateful to the OCaml core developers for giving this effort so much of their time and focus in 2020! We're working on a broader plan for 2021's exciting multicore roadmap which will be included in the next monthly after a core OCaml developer's meeting ratifies it soon. The broad strategy remains consistent: putting pieces of functionality steadily into each upcoming OCaml release so that each can be reviewed and tested in isolation, ahead of the OCaml 5.0 release which will include domains parallelism.

With OCaml 4.12 out in beta, our January has mainly been spent tackling some of the big pieces needed for OCaml 4.13. In particular, the safe points PR has seen a big update (and corresponding performance improvements), and we have been working on the design and implementation of Domain-Local Allocation Buffers (DLAB). We've also started the process of figuring out how to merge the awesome sequential best-fit allocator with our multicore major GC, to get the best of both worlds in OCaml 5.0. The multicore IO stack has also restarted development, with focus on Linux's new io_uring kernel interface before retrofitting the old stalwart epoll and kqueue interfaces.

Tooling-wise, the multicore Merlin support began in December is now merged, thanks to @mattpallissard and @eduardorfs. We continue to work on the enhancements for Sandmark 2.0 benchmarking suite for an upcoming alpha release – @shakthimaan gave an online seminar about these improvements to the multicore team which has been recorded and will be available in the next monthly for anyone interested in contributing to our benchmarking efforts.

As with previous reports, the Multicore OCaml updates are listed first for the month of December 2020 and then January 2021. The upstream OCaml ongoing work is finally mentioned for your reference after the multicore-tree specific pieces..

December 2020

Multicore OCaml
  • Ecosystem
    • ocaml-multicore/lockfree#6 Current status and potential improvements

      An RFC that lists the current status of the lockfree library, and possible performance improvements for the Kcas dependency, test suite and benchmarks.

    • ocaml-multicore/lockfree#7 Setup travis CI build

      A .travis.yml file, similar to the one in https://github.com/ocaml-multicore/domainslib/ needs to be created for the CI build system.

    • ocaml-multicore/effects-examples#20 Add WebServer example

      An open task to add the httpaf based webserver implementation to the effects-examples repository.

    • ocaml-multicore/effects-examples#21 Investigate CI failure

      The CI build fails on MacOS with a time out, but, it runs fine on Linux. An on-going investigation is pending.

    • ocaml-multicore/multicore-opam#39 Multicore Merlin

      Thanks to @mattpallissard (Matt Pallissard) and @eduardorfs (Eduardo Rafael) for testing merlin and dot-merlin-reader, and to get it working with Multicore OCaml 4.10! The same has been tested with VSCode and Atom, and a screenshot of the UI is shown below.


  • API
    • ocaml-multicore/ocaml-multicore#448 Reintroduce caml_stat_accessors in the C API

      The caml_stat_minor_words, caml_stat_promoted_words, caml_allocated_words caml_stat_minor_collections fields are not exposed in Multicore OCaml. This is a discussion to address possible solutions for the same.

    • ocaml-multicore/ocaml-multicore#459 Replace caml_root API with global roots

      A work-in-progress to convert variables of type caml_root to value, and to register them as global root or generational global root, in order to remove the caml_root API entirely.

  • Sundries
    • ocaml-multicore/ocaml-multicore#450 "rogue" systhreads and domain termination

      An RFC to discuss on the semantics of domain termination for non-empty thread chaining. In Multicore OCaml, a domain termination does not mean the end of a program, and slot reuse adds complexity to the implementation.

    • ocaml-multicore/ocaml-multicore#451 Note for OCaml 5.0: Get rid of compatibility.h

      OCaml Multicore removed modify and initialize from compatibility.h, and this is a tracking issue to remove compatibility.h for OCaml 5.0.

    • ocaml-multicore/ocaml-multicore#458 no-effect-syntax: Remove effects from typedtree

      The PR removes the the effect syntax use from typedtree.ml, and enables external applications that use the AST to work with domains-only Multicore OCaml.

    • ocaml-multicore/ocaml-multicore#461 Remove stw/leader_collision events from eventlog

      A patch to make viewing and analyzing the logs better by removing the stw/leader_collision log messages.

  • ocaml-multicore/effects-examples#23 Migrate to dune

    The build scripts were using OCamlbuild, and they have been ported to now use dune.

  • ocaml-multicore/ocaml-multicore#402 Split handle_gc_interrupt into handling remote and polling sections

    The PR includes the addition of caml_poll_gc_work that contains the polling of GC work done in caml_handle_gc_interrupt. This facilitates handling of interrupts recursively without introducing new state.

  • ocaml-multicore/ocaml-multicore#439 Systhread lifecycle work

    The improvement fixes a race condition in caml_thread_scan_roots when two domains are initializing, and rework has been done for improving general resource handling and freeing of descriptors and stacks.

  • ocaml-multicore/ocaml-multicore#446 Collect GC stats at the end of minor collection

    The GC statistics is collected at the end of a minor collection, and the double buffering of GC sampled statistics has been removed. The change does not have an impact on the existing benchmark runs as observed against stock OCaml from the following illustration:


  • ocaml-multicore/ocaml-multicore#454 Respect ASM_CFI_SUPPORTED flag in amd64

    The CFI directives in amd64.S are now guarded by ASM_CFI_SUPPORTED, and thus compilation with --disable-cfi will now provide a clean build.

  • ocaml-multicore/ocaml-multicore#455 No blocking section on fork

    A patch to handle the case when a rogue thread attempts to take over the thread masterlock and to prevent a child thread from moving to an invalid state. Dune can now be used safely with Multicore OCaml.

  • ocaml-bench/rungen#1 Fix compiler warnings and errors for clean build

    The patch provides minor fixes for a clean build of rungen with dune to be used with Sandmark 2.0.

  • ocaml-bench/orun#2 Fix compiler warnings and errors for clean build

    The unused variables and functions have been removed to remove all the warnings and errors produced when building orun with dune.

  • ocaml-bench/sandmark#198 Noise in Sandmark

    An RFC to measure the noise between multiple execution runs of the benchmarks to better understand the performance with various hardware configuration settings, and with ASLR turned on and off.

  • ocaml-bench/sandmark#200 Global roots microbenchmark

    The patch includes globroots_seq.ml, globroots_sp.ml, and globroots_mp.ml that adds microbenchmarks to measure the efficiency of global root scanning.

  • We are continuing to integrate the existing Sandmark benchmark test suite with a Sandmark 2.0 native dune build environment for use with opam compiler switch environment. The existing benchmarks have been ported to the same to use their respective dune files. The orun and rungen packages now live in separate GitHub repositories.
  • ocaml-bench/sandmark#196 Filter benchmarks based on tag

    The benchmarks can now be filtered based on tags instead of custom target .json files. You can now build the benchmarks using the following commands:

    $ TAG='"run_in_ci"' make run_config_filtered.json
    $ RUN_CONFIG_JSON=run_config_filtered.json make ocaml-versions/4.10.0+multicore.bench
  • ocaml-bench/sandmark#201 Fix compiler version in CI

    A minor update in .drone.yml to use ocaml-versions/4.10.0+multicore.bench in the CI for 4.10.0+multicore+serial.

  • ocaml/ocaml#9876 Do not cache young_limit in a processor register

    This PR for the removal of young_limit caching in a register for ARM64, PowerPC and RISC-V ports hardware is currently under review.

January 2021

Multicore OCaml
  • ocaml-multicore/ocaml-multicore#464 Replace Field_imm with Field

    The patch replaces the Field immediate use with Field from the concurrent minor collector.

  • ocaml-multicore/ocaml-multicore#468 Finalisers causing segfault with multiple domains

    An on-going test case where Finalisers cause segmentation faults with multiple domains.

  • The design and implementation of Domain-Local Allocation Buffers (DLAB) is underway, and the relevant notes on the same are available in the following DLAB Wiki.
  • Ecosystem
    • ocaml-bench/rungen#1 Fix compiler warnings and errors for clean build

      Minor fixes for a clean build of rungen with dune to be used with Sandmark 2.0.

    • ocaml-bench/orun#2 Fix compiler warnings and errors for clean build

      A patch to remove unused variables and functions without any warnings and errors when building orun with dune.

    • ocaml-bench/rungen#2 Added meta files for dune-release lint

      The dune-release lint checks for rungen now pass with the inclusion of CHANGES, LICENSE and updates to rungen.opam files.

    • ocaml-bench/orun#3 Add meta files for dune-release lint

      The CHANGES, LICENSE, README.md and orun.opam files have been added to prepare the sources for an opam.ocaml.org release.

    • ocaml-multicore/multicore-opam#39 Multicore Merlin

      Thanks to @mattpallissard (Matt Pallissard) and @eduardorfs (Eduardo Rafael) for testing merlin and dot-merlin-reader, and to get it working with Multicore OCaml 4.10! The changes work fine with VSCode and Atom. The corresponding PR#40 is now merged.

    • ocaml-multicore/ocaml-multicore#45 Merlin and OCaml-LSP installation instructions

      The README.md file has been updated to include installation instructions to use Merlin and OCaml LSP Server.

  • Sundries
    • ocaml-multicore/ocaml-multicore#458 no-effect-syntax: Remove effects from typedtree

      The PR enables external applications that use the AST to work with domains-only Multicore OCaml, and removes the effect syntax use from typedtree.ml.

    • ocaml-multicore/ocaml-multicore#461 Remove stw/leader_collision events from eventlog

      The stw/leader_collision log messages have been cleaned up to make it easier to view and analyze the logs.

    • ocaml-multicore/ocaml-multicore#462 Move from Travis to GitHub Actions

      The continuous integration builds are now updated to use GitHub Actions instead of Travis CI, in order to be similar to that of upstream CI.

    • ocaml-multicore/ocaml-multicore#463 Minor GC: Restrict global roots scanning to one domain

      The live domains scan all the global roots during a minor collection, and the patch restricts the global root scanning to just one domain. The sequential and parallel macro benchmark results are given below:



    • ocaml-multicore/ocaml-multicore#467 Disable the pruning of the mark stack

      A PR to disable the mark stack overflow for a concurrency bug that occurs when remarking a pool in another domain when that domain also does allocations.

  • ocaml-bench/sandmark#202 Add bench clean target in the Makefile

    A benchclean target has been added to the Makefile to only remove _build and _results. The _opam folder is retained with the required packages and dependencies installed, so that the benchmarks can be quickly re-built and executed.

  • ocaml-bench/sandmark#203 Implement ITER support

    The use of ITER has been correctly implemented with multiple instances of the benchmarks being built, and to repeat the executions of the benchmarks. This helps to take averages from multiple runs for metrics. For example, using ITER=2 produces two .summary.bench files as shown below:

    $ ls _build/
      4.10.0+multicore_1  4.10.0+multicore_2  log
    $ ls _results/
      4.10.0+multicore_1.orun.summary.bench  4.10.0+multicore_2.orun.summary.bench
  • ocaml-bench/sandmark#204 Adding layers.ml as a benchmark to Sandmark

    Th inclusion of Irmin layers benchmark and its dependencies into Sandmark. This is a work-in-progress.

  • We are continuing the enhancements for Sandmark 2.0 that uses a native dune to build and execute the benchmarks, and also port and test with the current Sandmark configuration files. The orun and rungen packages have been moved to their respective repositories. The use of a meta header entry to the .summary.bench file, ITER support, and package override features have been implemented.
  • ocaml-bench/sandmark#200 Global roots microbenchmark

    The implementation of globroots_seq.ml, globroots_sp.ml, and globroots_mp.ml to measure the efficiency of global root scanning has been added to the microbenchmarks.

  • ocaml/ocaml#10039 Safepoints

    An update to the draft Safepoints implementation that uses the prologue eliding algorithm and is now rebased to trunk.The runtime benchmark results on sherwood (an AMD EPYC 7702) and thunderx (a Cavium ThunderX CN8890) are shown below:



  • ocaml/ocaml#9876 Do not cache young_limit in a processor register

    The PR removes the caching of young_limit in a register for ARM64, PowerPC and RISC-V ports hardware.

Our thanks to all the OCaml users and developers in the community for their continued support and contribution to the project, and we look forward to working with you in 2021!


  • API: Application Programming Interface
  • ARM: Advanced RISC Machine
  • ASLR: Address Space Layout Randomization
  • AST: Abstract Syntax Tree
  • CFI: Call Frame Information
  • CI: Continuous Integration
  • GC: Garbage Collector
  • ICFP: International Conference on Functional Programming
  • JSON: JavaScript Object Notation
  • OPAM: OCaml Package Manager
  • PR: Pull Request
  • RFC: Request For Comments
  • RISC-V: Reduced Instruction Set Computing - V
  • UI: User Interface

Bikal Lem asked and Anil Madhavapeddy replied

Does this mean ocaml 5.0 will have a concurrency baked right in to the default ocaml? If so, this is an excellent news also how does this fit in to the mirage stack? i.e. will ocaml 5.0 concurrency work with mirage too?

OCaml 5.0 is slated to have domains-only parallelism and no fibres, but stay tuned for more news on our plans for fibres shortly. The Mirage prototype is looking very good so far, but I'm not quite ready to talk about it in more detail until it's a bit more baked :slight_smile:

Simon Cruanes and Anurag Soni asked, and Anil Madhavapeddy replied

Simon Cruanes:

Will there be only one runtime in 5.0? As in, multicore will be the default, not an opt-in switch as happened with flambda1?

The intention is to have a single runtime, yes. A huge amount of the work on the multicore project to date (see Retrofitting Parallelism onto OCaml) has been focussed on reducing the performance hit to sequential code when running on a parallel-capable runtime. If we end up with multiple runtimes in OCaml 5.0 after all this work, you'll find me sobbing quietly in a corner of the developer meeting.

Anurag Soni:

This is exciting to hear. Is the project public? I’ve spent some time over the past couple of months working on wrappers around the various polling interfaces (I started with kqueue and select ), and I’ve been working on io-uring bindings as well. After reading this months’s multicore report I think i’d rather work on testing/contributing to an effort maintained by the multicore team (There are gaps in my ctypes/ocaml runtime knowledge) instead of duplicating some effort

I've just pushed the non-multicore bits of the uring bindings over to https://github.com/ocaml-multicore/ocaml-uring. I'm still iterating on them, as there's a very specific design that minimises heap pressure that we're aiming towards. So not quite ready for contributions yet, but should be soon. The bindings there should be perfectly usable for pre-OCaml 5.0 use once that design settles (and in fact, I'll probably add it to Lwt in order to give us experience using io_uring in production ahead of any default use in multicore OCaml).

I've got to say though – so far, io_uring has absolutely exceeded my expectations. I never want to look at epoll again now. And a side worry is that the BSDs are steadily getting left behind in terms of post-POSIX IO standards. Between io_uring in Linux, Grand Central Dispatch in macOS, and iocp in Windows, we're spoilt for async choice in the "mainstream" operating systems.

EduardoRFS then added

if anyone is playing with multicore, I made this Lwt engine which is capable of running things like opium partially in parallel, 2x gains when running with 4 domains, it doesn't scale a lot, but 2x is quite nice.

Just drop this in a file and setup Lwt with it.


Resto 0.6 and 0.6.1

Raphaël Proust announced

On behalf of Nomadic Labs, it is my pleasure to announce the release of resto v0.6 and v0.6.1. Resto is set of packages to build REST RPC directories and to serve and call the built directories. It is split into different packages (resto, resto-directory, resto-cohttp-server, resto-cohttp-client, etc.) so as to minimise dependencies (e.g., you can call a directory without pulling in the server dependencies).

Both versions share the same code (see change log below), but the licensing on the code had been mixed up. This is the reason for the two releases: the version 0.6 is distributed under LGPL with linking exception and the version 0.6.1 is distributed under MIT. Future releases will be under MIT only.

The library can be installed from opam. The code is hosted on Gitlab: https://gitlab.com/nomadic-labs/resto The documentation is available online: https://nomadic-labs.gitlab.io/resto/

Other than the above packaging/licence fixes, the new versions:

  • Fix a URL percent-decoding bug in the server
  • Provide access-control capability to selectively allow/forbid some services
  • Provide a self-serving client that acts as its own server
  • Provide support for chunk transfer encoding for large values
  • Improves the logging and the logging levels

First release of paf, a simple MirageOS layer for HTTP/AF

Calascibetta Romain announced

I'm glad to announce the first release of paf, a simple MirageOS layer for http/af with a TLS support. Two simple unikernels exist into the distribution to show how to use paf with MirageOS.

$ cd unikernel/client
$ mirage configure
$ make depends
$ mirage build
$ ./minipaf -u https://discuss.ocaml.org/

Unikernel, abstraction, HTTP & TLS

About MirageOS, the main problem about the building of an unikernel is the constraint about the full abstraction. Indeed, the same piece of code should work for UNIX (as a simple executable) and for Solo5 (to be able to virtualize the operating system with KVM or Xen). MirageOS did the choice to use functors to abstract underlying pieces of the operating system such as the TCP/IP stack, the DNS resolver or the TLS stack.

HTTP is a good example as a mix of several stacks where the choice of them depends on:

  • static values such as the target (UNIX or Solo5)
  • dynamic values such as http or https (depends on the given Uri.t)

paf is a little library which focus on the possibility to use http/af on MirageOS in the easiest way as a server and a client for any targets. Static choices are handled by functoria which does the resolution about the implementation of the underlying TCP/IP according to your target:

  • for Unix, we will use the host's TCP/IP stack
  • for Solo5, we will use the mirage-tcpip implementation

Dynamic choices are handled by mimic which invokes the TLS stack (with ocaml-tls) or not - depending on the given Uri.t.

Let's encrypt challenge

With all of that and some side improvements about ocaml-tls/ca-certs and letsencrypt, we are able to provide a simple unikernel which does the let's encrypt challenge at the boot and initiate then an HTTP (with TLS) server (and your valid own certificate).

The distribution provides a simple unikernel which does this job if the user wants to provide an HTTPS service (with rock, a sub-library of opium):

$ cd unikernel/server
$ mirage configure
$ make depends
$ mirage build
$ ./minipaf --production=false --hostname www.x25519.net --https

CoHTTP layer (client part)

For a better use, paf provides a not fully-implemented compatible layer with CoHTTP (with another definition of the ctx, the Mimic.ctx).

Production ready

Even if the release still is a beta, I used such implementation for a long time about my unikernels (which are down but for some others reasons…) and I did not get any errors (for ~ 1 year). The distribution comes with a test which stress-test the implementation with parallel and concurrent processes.

I hope that will help people to play a bit with MirageOS!

I'm building a mailing list platform with Ocsigen

Tom announced

I've been working on a mailing list platform for the past few months in my spare time, and I thought I would share it here since I'm building the project using Ocsigen.

The project is still in early stages, but I hope the community finds this interesting.



If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.