OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of June 16 to 23, 2020.

Table of Contents

Alt-Ergo-free 2.2.0 release

Albin Coquereau announced

As discussed with our users and both our academic and industrial partners in the Alt-Ergo Users' Club, we are pleased to announce the new free release of Alt-Ergo. Alt-Ergo 2.2.0 is now under the Apache license. The alt-ergo-free package has been updated in the opam repository to this version as well.

You can find the source archive, changes and more on the Alt-Ergo website. The main changes are the new SAT solver (from Alt-Ergo 2.1.0) and the experimental support for the SMT-lib standard (introduced in Alt-Ergo 2.2.0)

Ocaml-migrate-parsetree 2.0.0

Jérémie Dimino announced

We are preparing a 2.0.0 release of ocaml-migrate-parsetree. This new major release will simplify the project to its core purpose, which is to provide migration functions between the various versions of the parsetree types. Everything else, such as the driver or the per-version copy of various modules from the compiler distribution will go away. This will have an impact for projects that are currently using these functionalities and this post describes what we think such projects should do and how we plan to help.

The motivation for this change is to simplify ocaml-migrate-parsetree and make it easier to maintain. The core functionality it provides is valuable and worth maintaining, however right now the project is way too big and difficult to maintain. Going through compiler upgrades has been unpleasant and difficult. And because of this, bugs of the segfault kind have accidentally been introduced and detected much later.

Once we have dropped everything that doesn't fit in the purpose of the project, what will remain will be pretty simple, coherent and easy to maintain.

What is the replacement?

At the moment, ocaml-migrate-parsetree offers one way of writing ppx rewriters. For instance, one might write the following to create a ppx rewriter with ocaml-migrate-parsetree:

open Migrate_parsetree
open Ast_404

let structure = ...
let signature = ...

let () =
  Driver.register ~name:"ppx_cstruct" Versions.ocaml_404
    (fun _config _cookies -> {default_mapper with structure; signature})

Following the simplification, the above code will break for at least two reasons:

  • the Migrate_parsetree.Driver will be gone
  • the default_mapper and mapper type coming from Ast_404.Ast_mapper will be gone

As a result, ocaml-migrate-parsetree itself will no longer be enough to write ppx rewriters.

We recommend that ppx rewriters using the ocaml-migrate-parsetree API switch to ppxlib. The purpose of ppxlib is to provide a good API for ppx rewriters with in particular a simple and good composition model, so it seems like a natural choice. Since ppxlib will take a more central place in the ecosystem, we are currently addressing the more controversial aspects of this library. For instance, we are dropping the dependency on Base.

Organising the transition

The following ticket is the meta issue tracking the preparation of the 2.0.0 release:

https://github.com/ocaml-ppx/ocaml-migrate-parsetree/issues/101

To make this transition smoother, we are planning to help with the porting effort. We have prepared a list of ppx rewriters to port to ppxlib. This list is not exhaustive and in particular excludes ppx rewriters that have no reverse dependencies in opam in order to keep this work manageable for us. In any case, help porting projects to ppxlib would be very much appreciated.

If you would like to help porting a project, please mention on the corresponding ticket from the list that you are looking at porting this project so that we know someone is working on it. If you would like to port a project that is not on this list, you should feel free to open a new ticket and mention that you are porting this project. In this case, please make sure to add the label port-to-ppxlib to the issue.

Once we have successfully converted a few projects, we will proceed with the release of ocaml-migrate-parsetree 2.0.0 and a compatible ppxlib release. At this point, most existing reverse dependencies of ocaml-migrate-parsetree will be marked as incompatible with ocaml-migrate-parsetree 2.0.0 and we will continue porting existing projects to ppxlib to gradually make the world compatible with ocaml-migrate-parsetree 2.0.0.

Consequences for other ppx libraries

  • ppx_deriving

    The master of ppx_deriving is now using ppxlib, so there shouldn't be much to do to make ppx_deriving compatible with ocaml-migrate-parsetree 2.0.0.

  • ppx_tools_versioned

    ppx_tools_versioned will no longer have a reason to exist in this new state of the world and so will stop being updated.

Nicolás Ojeda Bär asked and Jérémie Dimino replied

what is the story with dune? Right now, dune “just works” with ppx rewriters based on OMP, what will happen after the announced evolutions?

OMP won't provide a driver anymore. To write a ppx rewriter that is compatible with Dune, you will have to use ppxlib. You could also write your own driver, thought I wouldn't recommend doing that; we are trying to reduce the number of APIs rather that increase it :)

Will the planned updates to ppxlib (getting rid of base, etc.) be done in time for the release of OMP 2.0.0?

Yes. @ceastlund started working on dropping the Base dependency and since it will be the same group of people doing the ppxlib and omp changes, we will make sure that there is a new version of ppxlib available on time. We plan to do things in this order:

  • prepare OMP2 and an updated ppxlib
  • add upper bounds to all OMP reverse dependencies in the opam repository
  • release OMP2
  • release an updated ppxlib shortly after OMP2 is merged in the opam repository

Would it be possible to make a minimal version of ppxlib/OMP for those that are willing to write a ppx rewriter against compiler-libs directly and do not need/want the rest of the library, but still want their rewriters to be compatible with dune?

Technically yes. That said, I wouldn't recommend going down this route. Once most of the dependencies of ppxlib have been dropped and OMP has been simplified, the resulting OMP+ppxlib combo will be pretty lean. If there are aspects of ppxlib that would prevent people from switching to it, we are happy to consider those.

Once we have finished our work on simplifying the ppx ecosystem, I also expect that the bare ppx API in the compiler will disappear, so I would definitely not recommend to use it.

Could you say a few words on how the evolution of ppxlib is related to ppx 1?

Sure. The ppx project is part of this work. Our plan was to move to a world like this:

  • the compiler offers a minimal stable Astlib library, using a dynamic AST
  • the ppx project is written only against Astlib and not the rest of compiler-libs
  • ppx has a stable API
  • ppx rewriters are written against the ppx API

In this world, existing ppx rewriters are compatible with trunk at all times. Moreover, when a new compiler is released:

  • existing released ppx rewriters are immediately compatible with the new compiler
  • one can use existing ppx rewriters and the new features of the language

in other words, you can in theory release a ppx rewriters once and never touch it again. The trade-off is that it uses a technology that is quite complex and users don't have direct access to the parsetree types.

However, we don't expect everyone to switch to this API at once. So we will need a transition period where the existing APIs are implemented as shims on top of the new ppx API. The more ppx APIs we have to consider and the more difficult this work will be for us. So one can see this work of simplifying OMP and ppxlib as preparatory work.

Now, all of the above was our original plan. We are now considering an alternative plan that would lead to a technically simpler system that is easier to reason about. The new plan would lead to the following world:

  • the compiler offers a minimal stable Astlib library, using the OMP technology
  • ppxlib is written only against Astlib and not the rest of compiler-libs
  • ppx rewriters are written against ppxlib

In this world, existing ppx rewriters are compatible with trunk at all times. Moreover, when a new compiler is released:

  • existing released ppx rewriters are immediately compatible with the new compiler
  • one cannot use existing ppx rewriters and the new features of the language until:
    • ppxlib has been updated to use the new parsetree types
    • the subset of ppx rewriters that are broken by this change have been updated as well

This new plan requires updating potentially many packages after each new compiler release. We know that from a technical point of view this is relatively easy. In fact, we do that all the time at Jane Street: when we bump the version of the AST, one single person sit downs and updates all the ppx rewriters at once, which doesn't take very long. In the open-source world, this was historically impossible, though now that most OCaml projects are using dune this is becoming more realistic; we can create one big dune workspace with all the ppx rewriters and update them all at once. That said, there is a social component to this approach since the various packages are owned by various groups of people and even if one person does all the work, we still need various groups of people to take action to merge PRs and make new releases. So we plan to try this idea with one or two compiler release. In fact, @NathanReb is experimenting with this idea as we speak for the 4.10 upgrade of ppxlib. If this all works out well, we'll continue with this new plan. Otherwise, we'll go back to the original plan.

Anton Kochkov added

There is also ppx_deriving_protobuf #28 for porting to ppxlib.

Free culture OCaml book progress

Daniil Baturin announced

You may (or may not) remember that I've been working on a free culture OCaml book, under "Unreal world OCaml" working title.

Zeroth, it's now named "OCaml from the ground up". I hope that title tells what I want the book to be. "Unreal world OCaml" clearly wasn't a title it could use forever. I briefly considered naming it "Elements of OCaml", but the Bourbaki reference is a bit too obscure and it sounds strange (not that their "Elements of Mathematic" doesn't sound strange to begin with…).

First, it has a proper home now at https://ocamlbook.org/ The old https://ocaml-book.baturin.org URL redirects to it now. It's hosted on Netlify for better availability. From my quick research, Netlify doesn't do anything obviously evil, but if I missed something, let me know.

Second, there's now CI setup, so the live version is going to be in sync with the source. Examples in "(triple backtick)ocaml" blocks are automatically typechecked (with ocamlc -stop-after typing). These fail the build if they fail to typecheck. Intentionally invalid examples should be in "(triple backtick)invalid-ocaml" blocks instead—those are run through syntax highlighting but excluded from type checking.

Content-wise, I've made some improvements to the Records and References chapter. I'll keep posting progress here in the thread if no one objects.

OCaml 4.11.0, third alpha release with RISC-V backend

octachron announced

A new alpha version of OCaml 4.11.0 has been published.

Exceptionally, this third alpha release integrates a new exciting feature: a native-code backend for RISC-V. Since this new backend cannot affect negatively other architecture, the feature has been backported to the 4.11 branch.

Beyond this important change, the full change list compared to the second alpha is rather small:

  • new feature 9440: Add RISC-V RV64G native-code backend. (Nicolás Ojeda Bär, review by Xavier Leroy and Gabriel Scherer)
  • additional fixes 8920, 9238, 9239, 9254, 9458: New API for statistical memory profiling in Memprof.Gc. The new version does no longer use ephemerons and allows registering callbacks for promotion and deallocation of memory blocks. The new API no longer gives the block tags to the allocation callback.
  • 7520, 9547: Odd behaviour of refutation cases with polymorphic variants (Jacques Garrigue, report by Leo White, reviews by Gabriel Scherer and Leo)
  • 7741, 9645: Failure to report escaping type variable (Jacques Garrigue, report by Gabriel Radanne, review by Gabriel Scherer)
  • 9623, 9642: fix typing environments in Typedecl.transl_with_constraint (Gabriel Scherer, review by Jacques Garrigue and Leo White, report by Hugo Heuzard)
  • 9610: manual, C FFI: naked pointers are deprecated, detail the forward-compatible options for handling out-of-heap pointers. (Xavier Leroy, review by Mark Shinwell, David Allsopp and Florian Angeletti)
  • 9618: clarify the Format documentation on the margin and maximum indentation limit (Florian Angeletti, review by Josh Berdine)

We are expecting this third alpha release to be the last alpha release.

The compiler can be installed as an OPAM switch with one of the following commands

opam switch create ocaml-variants.4.11.0+alpha3 --repositories=default,beta=git+https://github.com/ocaml/ocaml-beta-repository.git

or

opam switch create ocaml-variants.4.11.0+alpha3+<VARIANT> --repositories=default,beta=git+https://github.com/ocaml/ocaml-beta-repository.git

where <VARIANT> is replaced with one of these: afl, flambda, fp, fp+flambda

The source code for the alpha is also available at these addresses:

https://github.com/ocaml/ocaml/archive/4.11.0+alpha3.tar.gz
https://caml.inria.fr/pub/distrib/ocaml-4.11/ocaml-4.11.0+alpha3.tar.gz

If you find any bugs, please report them here: https://github.com/ocaml/ocaml/issues

Roll your own bubblewrap sandboxing scripts in OCaml

Darren announced

I recently used a small bit of OCaml code to help generate bash scripts for sandboxing programs via bubblewrap and have found the suite to be fairly useful and saved a fair bit of typing.

The repo: https://github.com/darrenldl/sandboxing

Note: This is not a well packaged project right now, and has no versioning. I suggest forking the repo if you actually plan on using it. Might do proper packaging work at some point, however.

Syscall filter handling

Seccomp BPF compilation (for syscall filter) can also be handled via the suite in OCaml's side. (If you've seen this project over at Discord before, then this part is the new addtion that was previously absent)

  • Context: right now bubblewrap only allows specification of syscall filtering via a compiled BPF file, this is handled by the suite as follows: OCaml code --generates--> C code --generates--> BPF filter

Firefox example

Some code for firefox is as follows

  • OCaml side

    Firefox profile specification in bw-script-gen/src/profiles.ml

    let make_firefox_profile ~(suffix : string option) : profile =
      let name = match suffix with None -> "firefox" | Some s -> "firefox-" ^ s in
      {
        name;
        cmd = "/usr/lib/firefox/firefox --ProfileManager";
        home_jail_dir = Some name;
        syscall_blacklist = default_syscall_blacklist;
        args =
          usr_share_common
          @ usr_lib_lib64_bin_common
          @ etc_common
          @ proc_dev_common
          @ tmp_run_common
          @ sound_common
          @ wayland_common
          @ dconf_common
          @ dbus_common
          @ set_up_jail_home ~tmp:false ~name
          @ [
            Bind ("$HOME/.mozilla", Some "/home/jail/.mozilla");
            Bind ("$HOME/.cache/mozilla", Some "/home/jail/.cache/mozilla");
            Unsetenv "DBUS_SESSION_BUS_ADDRESS";
            Setenv ("SHELL", "/bin/false");
            Setenv ("USER", "nobody");
            Setenv ("LOGNAME", "nobody");
            Setenv ("MOZ_ENABLE_WAYLAND", "1");
            Hostname "jail";
            Unshare_user;
            Unshare_pid;
            Unshare_uts;
            Unshare_ipc;
            Unshare_cgroup;
            New_session;
          ];
      }
    
  • C code

    The generated C code for compiling the BPF file can be seen here: https://github.com/darrenldl/sandboxing/blob/master/seccomp-bpf/firefox.c

  • Bash script

    The generated bash script:

    #!/usr/bin/env bash
    
    set -euxo pipefail
    
    gcc "$(dirname $0)"/../seccomp-bpf/firefox.c -lseccomp -o "$(dirname $0)"/../seccomp-bpf/firefox.exe
    "$(dirname $0)"/../seccomp-bpf/firefox.exe
    mv firefox_seccomp_filter.bpf "$(dirname $0)"/../seccomp-bpf
    
    mkdir -p "$HOME/jails/firefox"
    mkdir -p "$HOME/jails/firefox/Downloads"
    
    bwrap \
      --ro-bind "/usr/share/X11" "/usr/share/X11" \
      --ro-bind "/usr/share/icons" "/usr/share/icons" \
      --ro-bind-try "/usr/share/fontconfig" "/usr/share/fontconfig" \
      --ro-bind "/usr/share/fonts" "/usr/share/fonts" \
      --ro-bind "/usr/share/mime" "/usr/share/mime" \
      --ro-bind "/usr/share/ca-certificates" "/usr/share/ca-certificates" \
      --ro-bind "/usr/share/glib-2.0" "/usr/share/glib-2.0" \
      --ro-bind "/usr/lib" "/usr/lib" \
      --ro-bind "/usr/lib64" "/usr/lib64" \
      --tmpfs "/usr/lib/modules" \
      --tmpfs "/usr/lib/systemd" \
      --symlink "/usr/lib" "/lib" \
      --symlink "/usr/lib64" "/lib64" \
      --ro-bind "/usr/bin" "/usr/bin" \
      --symlink "/usr/bin" "/bin" \
      --symlink "/usr/bin" "/sbin" \
      --ro-bind "/etc/fonts" "/etc/fonts" \
      --ro-bind "/etc/machine-id" "/etc/machine-id" \
      --ro-bind "/etc/resolv.conf" "/etc/resolv.conf" \
      --proc "/proc" \
      --dev "/dev" \
      --tmpfs "/tmp" \
      --tmpfs "/run" \
      --ro-bind-try "/usr/share/gst-plugins-bad" "/usr/share/gst-plugins-bad" \
      --ro-bind-try "/usr/share/gst-plugins-base" "/usr/share/gst-plugins-base" \
      --ro-bind-try "/usr/share/gstreamer-1.0" "/usr/share/gstreamer-1.0" \
      --ro-bind "/run/user/$UID/pulse" "/run/user/$UID/pulse" \
      --ro-bind "/run/user/$UID/wayland-0" "/run/user/$UID/wayland-0" \
      --setenv "QT_QPA_PLATFORM" "wayland" \
      --bind "/run/user/$UID/dconf" "/run/user/$UID/dconf" \
      --ro-bind "/run/user/$UID/bus" "/run/user/$UID/bus" \
      --bind "$HOME/jails/firefox" "/home/jail" \
      --setenv "HOME" "/home/jail" \
      --bind "$HOME/.mozilla" "/home/jail/.mozilla" \
      --bind "$HOME/.cache/mozilla" "/home/jail/.cache/mozilla" \
      --unsetenv "DBUS_SESSION_BUS_ADDRESS" \
      --setenv "SHELL" "/bin/false" \
      --setenv "USER" "nobody" \
      --setenv "LOGNAME" "nobody" \
      --setenv "MOZ_ENABLE_WAYLAND" "1" \
      --hostname "jail" \
      --unshare-user \
      --unshare-pid \
      --unshare-uts \
      --unshare-ipc \
      --unshare-cgroup \
      --new-session \
      --seccomp 10 10<"$(dirname $0)"/../seccomp-bpf/firefox_seccomp_filter.bpf \
      /usr/lib/firefox/firefox --ProfileManager
    

Second release of Sek

François Pottier announced

We are pleased to announce the second release of Sek, an OCaml library that offers an efficient implementation of sequences.

The library offers both ephemeral (mutable) sequences and persistent (immutable) sequences, and offers constant-time conversions between these flavors.

It supports all of the standard operations on stacks, queues, deques (e.g. push, pop at either end), catenable sequences (concat, split), and random access sequences (get, set).

The new release makes a large number of additions to the library's API. The most significant one is the introduction of very efficient iterators over ephemeral and persistent sequences. A full list of changes appears below.

To install the new version of Sek, just type

opam update && opam install sek.20200618

The API documentation is online:

http://cambium.inria.fr/~fpottier/sek/doc/sek/Sek/

Feedback is welcome!

Arthur Charguéraud and François Pottier

2020/06/18

  • The time complexity of every operation is now documented as part of its specification.
  • New submodules E.Iter and P.Iter, which offer very efficient iterators over ephemeral and persistent sequences.
  • New functions take, drop, sub, iter_segments, to_seq, of_list_segment, of_list, of_seq_segment, of_seq, find, find_opt, find_map, for_all, exists, mem, memq, map, mapi, rev, zip, unzip, filter, filter_map, partition, flatten, flatten_map, iter2, iter2_segments, fold_left2, fold_right2, map2, for_all2, exists2, equal, compare, sort, stable_sort, uniq, merge, in ephemeral and persistent flavors.
  • New functions E.fill and E.blit.
  • New function other of type side -> side.
  • New function opposite of type direction -> direction.
  • New functor SupplyDefault, which allows supplying a default element once and for all so as to obtain a simpler API. Unfortunately, this requires choosing a fixed type of elements at the same time.
  • New submodules Emulated.Array and Emulated.List, which can be used as drop-in replacements for OCaml's standard Array and List modules.
  • New submodule Segment, which offers a few facilities for iterating over array segments.
  • Breaking change: the default behavior of E.copy is now to produce a disjoint sequence in time O(n). The previous behavior, which exploits sharing and produces a result in time O(K), is obtained by invoking E.copy ~mode:`Share. The two copying modes have the same observable behavior; they differ only in their performance characteristics.
  • Breaking change: the submodules Queue and Stack are renamed Emulated.Queue and Emulated.Stack. This makes it easier to avoid unintended shadowing of Stdlib.Queue, Stdlib.Stack, etc. It is now safe to use open Sek.
  • Breaking change: the functor Make now takes just one structure as a parameter, instead of several structures. This is more pleasant and should make future evolution easier.

Tools or pointers for control flow analysis of OCaml code?

Anton Kochkov announced

Just a heads up - there is a work going on adding OCaml to GitHub Semantic, see the corresponding PRs:

Once these are merged, it will be possible to start adding Semantic support itself.

A new build system API in Dune: please send thoughts, ideas and wishes

Jérémie Dimino announced

Do you have ideas about what a good build API should look like? Are there functionalities you always wished a build system API would give you?

If yes, please write your thoughts, ideas or wishes on the following github ticket: https://github.com/ocaml/dune/issues/3571

To give a bit more context, the Dune team is preparing to design a new build system API. This API is what comes in between the core build system engine and the build rules explaining how to go from .ml files to a final executable. The main motivation for doing this work now is to help migrating from Jenga to Dune inside Jane Street. However, eventually we would like to open this API to users, so that everyone can write their own set of build rules outside of Dune. For this reason, we'd like to open the discussion to the community.

Other OCaml News

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.