OCaml Weekly News

opam-grep is an opam plugin that greps through the sources of all opam packages.

To install it, simply call:

$ opam install opam-grep

Then to use it, simply call:

$ opam grep "your regexp"

Side note: currently opam-grep will cache the sources in your cache directory (~/.cache/opam-grep), so a few GB of available disk space is most likely required.

I hope this can help core compiler and community library devs alike, to know which part of their software is used in the wild.

Over the past previous OCaml releases the release-readiness team tried its best to make most packages, and especially core OCaml packages (dune, merlin, ppxlib, …) available day one so people can have the smoothest transition possible to the new compiler release.

Today I'm announcing that merlin (arguably the trickiest core tool to port to new releases, and the most indispensable) is now available to install for early adopters of OCaml 4.13.

To install the latest OCaml 4.13 beta together with Merlin simply call:

$ opam switch create 4.13 --formula='"ocaml-base-compiler" {>= "4.13" & < "4.14"}'
$ opam install merlin

The package is still an early preview. If you encounter any problems while using merlin, please feel free to report it directly in https://github.com/ocaml/merlin/pull/1381

All of the OCaml core packages (with the exception of ocaml-lsp, hopefully soon) should be available out-of-the-box. If some packages aren't available yet, opam-alpha-repository should have you covered:

$ opam remote add alpha git://github.com/kit-ty-kate/opam-alpha-repository.git

If you have any questions or remarks feel free to ask them here.

This summer the OCaml community rebooted its participation in Outreachy "…a diversity initiative that provides paid, remote internships to people subject to systemic bias and impacted by underrepresentation in the technical industry where they are living".

We had the pleasure of working with three talented interns over 13 weeks this summer on projects relating to v3.ocaml.org. Each intern has summarised their experience in a blog post that we'd like to share with you.

A quick reminder, if you are interested in mentoring for the OCaml community in the next round, please read this post and apply with your project. The project submission deadline is Sept. 23, 2021 at 4pm UTC.

I am pleased to announce v4.* series release of http-cookie. The v4 release contains many new enhancements to make working with cookies in OCaml web applications a little bit easier.

v4.*.* release series features the following additions and changes:

1. Add pretty printers - pp, pp_date_time and pp_same_site. These can be used during debugging and in a OCaml toplevel session
2. Flatten out Same_site module
3. Add of_cookie to parse Cookie header.
4. Add of_set_cookie to parse Set-Cookie header. This can be used by http client applications
5. Rename of_cookie_header to of_cookie, to_cookie_header_value to to_cookie, to_set_cookie_header_value to to_set_cookie to enhance API ergonomics
6. Add validation for cookie domain value which consists of - host name, IPv4 and IPv6 - to conform to their respective RFC specs - Domain name RFC(RFC 1034), RFC 1123 and draft IPv4 and IPv6 RFCs.
7. Improve documentation

• Documentation
• Github repo

on behalf of the Dune team I'm pleased to announce the release of Dune 2.9.1.

This is a minor release, fixing an important problem with opam file generation when (lang dune 2.9) was set, and a few other minor fixes; please find the changelog below:

• Don't use subst --root in Opam files (https://github.com/ocaml/dune#4806, @MisterDA)
• Fix compilation on Haiku (https://github.com/ocaml/dune#4885, @Sylvain78)
• Allow depending on ocamldoc library when ocamlfind is not installed. (https://github.com/ocaml/dune#4811, fixes https://github.com/ocaml/dune#4809, @nojb)
• Fix (enabled_if ...) for installed libraries (https://github.com/ocaml/dune#4824, fixes https://github.com/ocaml/dune#4821, @dra27)
• Create more future-proof opam files using --promote-install-files=false (https://github.com/ocaml/dune#4860, @bobot)

On behalf of the MDX maintainers I'd like to announce the availability of MDX 1.11.0. This is mostly a maintenance release but it prepares for a 2.0 release soon. The most important change in this regard is that it will output deprecation warnings when OCaml toplevel lines are not ;; terminated.

As an example this will trigger a deprecation warning to be printed on stderr:

# List.map (fun x -> x + 1) [1; 2; 3]
- : int list = [2; 3; 4

Note that this will only print a warning, it will still process these lines correctly and output code just like before. To avoid the warning, make sure to add terminating ;;:

# List.map (fun x -> x + 1) [1; 2; 3];;
- : int list = [2; 3; 4

To ease migration, MDX 2.0 will change behaviour and insert ;; into the output so users will be able to use e.g. dune promote to get their source files updated with ;;. In MDX 3.0 reading toplevel phrases without the trailing ;; will be finally unsupported and the deprecation warning will be removed.

Happy documenting!

The release of OCaml 4.13.0 is imminent. As a last test that everything is in order, we are publishing a first release candidate for OCaml 4.13.0.

Compared to the last beta release, this release candidate includes one small fix for flambda, another for the compiler-libs, one regression fix when compiling C files with ocamlc or ocamlopt, and a handful of configuration and build system improvements (in particular on Windows). See the changelog below for more details.

If you find any bugs, please report them here:

https://github.com/ocaml/ocaml/issues

The release of OCaml 4.13.0 is currently planned for mid-September. An assembler generation bug was discovered this morning, we might release another release candidate next week once this bug is fixed.

Happy hacking, Florian Angeletti for the OCaml team.

Vincent and I are pleased to announce a new release of ProVerif, version 2.03. ProVerif is a automatic security protocol verifier that relies on the symbolic model of cryptography.

ProVerif is available at http://proverif.inria.fr It is also available via opam (see installation instructions in Section 1.4 of the manual http://proverif.inria.fr/manual.pdf )

This release contains many important changes since version 2.01 (version 2.02 was not announced here because of a pending anonymous submission), mainly implemented by Vincent:

• Change of license for Windows binaries: Windows binaries are now distributed under GPL, like the source (because ProVerif now integrates a modified version of an OCaml module under LGPL). If this change of license prevents you from using ProVerif, please email me.
• Major speedups for large examples.
• Several extensions, including temporal variables in correspondence queries (see Section 4.3.1 of the manual, paragraph "Temporal correspondences").
• Bug fixes.

On behalf of the ocaml-lsp team, I'm pleased to announce version 1.8.0. This release contains some quality of life bug fixes, better diagnostics locations, and a few new code actions. Happy hacking.

The full change log is rendered below for your convenience:

I have a technical question. I notice that the code base uses dune fibers for concurrency. What benefits do fibers have over, say, Lwt/Async?

I wanted to write about more that in more detail at some point. I can give a quick preview here I guess. The gist of it is that fibers are much more lightweight and provide simpler error handling semantics. You do lose out on choice (Deferred.choose and friends) though.

Like deferred/lwt, a fiber is also a monadic value that represents asynchronous computation. In deferred/lwt, binding on the computation waits for it to finish, and then stores the result in a ref. Subsequent binds will reuse the saved value. A fiber stores nothing and will always re-run the computation from scratch to produce the value. For example, this will print foo twice.

let foo = Fiber.of_thunk (fun () -> Fiber.return (print_endline "foo"))

let bar =
  let* () = foo in
  let* () = bar in
  Fiber.return ()

To save the value computed, one must opt-in and create an Ivar manually. What's the advantage of this approach? Well, bind becomes a lot simpler (and cheaper). I'll even paste the implementation to prove how simple it is:

let ( >>= ) t f k = t (fun x -> f x k)

Lwt's and Async's bind are far more complex (and slow).

Moving on to error handling. Lwt has the disaster that is Lwt.async. Enough said. There's Lwt.dont_wait now, but it's still fishy to me that yet another error handling channel is needed in addition to Lwt.catch. With Async, things are cleaner with monitors, but IMO, a completely separate concept for error handling is overkill.

To explain how fiber does error handling, I'll first explain the other (aside from sequencing with bind) composition primitive for fibers. A fiber may spawn sub fibers that will be executed concurrently. The simplest primitive to do is that is the fork:

(* the main concurrency primitive *)
val fork_and_join : (unit -> 'a Fiber.t) -> (unit -> 'b Fiber.t) -> ('a * 'b) Fiber.t

Before a fiber can finish and produce a value, it must wait for all of its sub fibers to terminate. If a fiber raises an exception instead of producing a value, the exception will be passed to the fiber's error handler. If there' no error handler set, the exception will be passed to the parent's fiber error handler (if the error handler itself raises, the error will always be passed to the parent's error handler).

The rules above are exhaustive and leave very little room for ambiguity. In practice, I find it very convenient to write robust error handling code this way. For example with lsp, it's easy to install a handler in one place for a single request and know that errors will not leak outside of it. It is also easy to make sure that no spawned computation is left "dangling" after a request is served.

There's also some practical benefits that made me choose Fiber:

• Depending on async/lwt would impose version constraints for end users on these key libraries. In contrast, fiber is a single module that is trivial to vendor and is invisible to users.
• Fiber contains no C and hence very portable. This can be handy if one wants to get ocamllsp working with an experimental compiler for example.

There's downsides as well of course. Fiber is quite minimal and therefore comes without a scheduler. I managed to throw something together by mostly copying dune's scheduler, but it's not something I'd recommend to anybody. If fiber ever becomes public, we'd need something like libuv to provide us with a serious and reusable scheduler.

That's all for now. It would be much more interesting to write about how dune itself uses fibers, but that is better written by other people from the team.

Abstraction over polymorphic types is something that one comes across from time to time, especially when implementing operations generic over collections, or embedding typed DSLs (particularly in tagless-final style). One immediately learns that OCaml does not have higher-kinded type variables (type variables that range over type constructors like list and array); does not permit type constructors appearing by themselves, without being applied to the right number of arguments; and restricts polymorphic types in other ways (e.g., not allowing them in type constraints of package types). One soon learns that the module system (using functors) is a way to realize higher-kinded polymorphism in OCaml. It takes longer to learn it is not the only way, however.

The following web page is written to speed up this learning. It collects what I have learned and rediscovered on this topic, arranged into a single story and with new explanations and attributions (and occasionally, new variations).

http://okmij.org/ftp/ML/higher-kind-poly.html

One interesting discovery is that some problems that ostensibly require higher-kinded polymorphism really do not.

The article comes with the complete accompanying code.

The direct links (for search results) to the videos of the talks from the OCaml Workshop at ICFP 2021 are listed below:

• GopCaml A structural Editor for OCaml (https://www.youtube.com/watch?v=KipRuiLXYEo)
• OCaml and Python getting the best of both worlds (https://www.youtube.com/watch?v=ZseAgG9kmMw)
• Adapting the OCaml ecosystem for Multicore OCaml (https://www.youtube.com/watch?v=MfjMpznU4nM)
• Parafuzz coverage guided Property Fuzzing for Multicore OCaml programs (https://www.youtube.com/watch?v=GZsUoSaIpIs)
• Wibbily Wobbly Timey Camly (https://www.youtube.com/watch?v=3Jpla-vWLR4)
• Leveraging Formal Specifications to Generate Fuzzing Suites (https://www.youtube.com/watch?v=-6fpLF_n7cA)
• Probabilistic resource limits using StatMemprof (https://www.youtube.com/watch?v=9OES0qe_OvE)
• Continuous Benchmarking for OCaml Projects (https://www.youtube.com/watch?v=folKaWN7jy4)
• A Multiverse of Glorious Documentation (https://www.youtube.com/watch?v=qUMbbKwagaU)
• Digodoc and Docs (https://www.youtube.com/watch?v=gODavKhyd2I)
• Keynote - Xavier Leroy: 25 years of OCaml (https://www.youtube.com/watch?v=PGJdHUqgX7o)
• Experiences with Effects (https://www.youtube.com/watch?v=k3oQwpyXmpo)
• opam-bin Binary Packages with Opam (https://www.youtube.com/watch?v=vadoOkgEs6w)
• Love a readable language interpreted by blockchain (https://www.youtube.com/watch?v=um55QNwz8o4)
• From 2n+1 to n (https://www.youtube.com/watch?v=3xRGeJSslPE)
• Property based testing for OCaml through Coq (https://www.youtube.com/watch?v=w-dTTlVwVgY)
• Safe protocol updates via propositional logic (https://www.youtube.com/watch?v=ckN1YREM3Gk)
• Binary Analysis Program (BAP) (https://www.youtube.com/watch?v=S9XPI1fHWqM)
• Semgrep a fast lightweight polyglot static analysis tool to find bugs (https://www.youtube.com/watch?v=q7kuvyAOk78)

I am pleased to announce the second preview release of the Diskuv OCaml (“DKML”) distribution. Diskuv OCaml is an OCaml distribution focused on a) secure, cross-platform software development and b) ease of use for language learners and professional developers.

Thanks to everybody who tried out the first preview! The first preview was geared towards Windows users and distributed Microsoft compiled packages. The Microsoft compiler is critical for first-class Windows support: many popular C libraries like 'libuv' support the Microsoft compiler as Tier 1 but relegate the GCC compiler to an unsupported third-class citizen.

This second preview has the same Windows users in mind, with several new features and improvements. There are too many changes to list here, but some notable ones are:

• (Critical security fix) The MSYS2 installer is now downloaded over HTTPS and its SHA256 checksum is validated.
• Removed auto-installation of Cygwin and removed the 16GB auto-download of two Docker images. This cuts down the install time in half (except for some people; see sharp edges below), and should reduce the frequency of installation failures caused by limited or flaky networks.
• Autodetection and use of existing Visual Studio instances on your machine if the installations are compatible with DKML.
• Clarified that students are covered under free personal licenses in a prominent location in the main READMEs.
• The ctypes, mirage-crypto and feather packages are available for testing in a highly unstable form. That means support may be deprecated in a future release, but today they all build + pass tests with the Microsoft compiler.

There are also some sharp edges in this preview. The major ones are:

• On some machines (especially Docker containers or VMs) one of the installation commands can peg your CPU for up to 2 hours while making incredibly slow progress.
• There are backwards incompatible changes. See the release notes linked below.

The Version 0.2.0 release notes has instructions for upgrading and also a detailed list of changes. Instructions for installing from scratch and updated documentation are still at the Diskuv OCaml documentation site.

What's Next?

Barring some critical security updates, the next preview will focus on Linux and macOS parity, and the removal of the sharp edges. Will post another "[ANN]" and cross-link on Twitter when that preview is ready.

In the meantime, I would appreciate feedback!