OCaml Weekly News

Guess what?! Version 0.3 of data-encoding is now available!

Data-encoding is a library for defining encodings, which can then be used to serialise and deserialise values to-and-from a binary or JSON representation.

let e : (string * int list) t =
  obj2 (req "name" string) (list uint8)
let v : (string * int list) =
  ("low", [0;1;2;3;4])
let s : string = Binary.to_string_exn e v
let j : Ezjson.t = Json.construct e v

Version 0.3 of the library is now available on opam. The code is distributed under MIT license and hosted on Gitlab: https://gitlab.com/nomadic-labs/data-encoding/. The documentation is available online: https://nomadic-labs.gitlab.io/data-encoding/data-encoding/Data_encoding/index.html

In addition to numerous miscellaneous improvements, this version brings two major changes.

1. Support for streamed JSON serialisation.

   JSON serialisation for large values can be expensive. And in some context, prohibitively so. One such context is cooperative concurrency where serialising very large JSON values can block other on-going tasks.

   Data-encoding provides the necessary functions to serialise values as sequences of strings which can be consumed with appropriate yielding.

   let rec write seq = match seq () with
     | Seq.Nil ->
         Lwt.return_unit
     | Seq.Cons (chunk, seq) ->
         Lwt_io.write oc chunk >>= fun () ->
         Lwt.pause () >>= fun () ->
         write seq
   in
   let j = Json.construct_seq e v in
   let s = Json.string_seq_of_jsonm_lexeme_seq ~chunk_size_hint:512 j in
   write s
   

2. Performance improvements.

   The serialisation and deserialisation of some encodings has been optimised a lot. On some encodings the performances have gotten competitive with Marshal.

   More details below.

According to the micro-benchmark below data-encoding.0.3 gets close to Marshal performances on the serialising and deserialising of Micheline values (S-EXP-like values used to represent smart-contracts on the Tezos blockchain).

https://gitlab.com/nomadic-labs/data-encoding/-/snippets/2060331

The results are printed below. Notice the speed up:

• for serialising it progressed from a 13.40× slow-down over Marshal to a 1.01× slow-down,
• for deserialising it progressed from a 18.72× slow-down over Marshal to a 1.02× slow-down.

We've published a short review of our 2020 activities. It's available on our blog!

It takes time to write such a post for a non-native speaker like me, but I appreciate what the OCaml gives me and believe I am doing good for the community, so I decide to write a short history of ReScript and explain the motivations behind this project.

I like OCaml, that’s why I decided to go to PlClub@UPenn for further study after undergraduate (2011). I wrote lots of tutorials around OCaml and used to be one of the maintainers of camlp4, I was even honored to be listed as core maintaines of OCaml once.

However, the reality is brutal that it’s very difficult to find a decent job using OCaml if you are not interested in Finance or get unlucky with a Jane Street interview. Here a decent job, I mean you get paid as good as those Silicon Valley companies while using your favorite programming language.

JSOO started growing mature back in 2013, it is the first compiler which gets bootstrapped in the browser, truly impressive. However, even for OCaml experts like me, it is really hard to get started with JSOO. It is a black box to me that I can not read the generated JS code under the hood. If your code works under the bytecode but stops working in the browser. You have to do a random guess to do the debugging. These are the impressions I have back then.

Another story happened on my side is that I started to have lots of experience in JavaScript in daily jobs and love it. It’s the most beautiful dynamic language I have used – you have a decent IDE everywhere as long as it ships a browser. So the idea comes to my mind that why not make a compiler which compiles OCaml to idiomatic JS for the convenience of JS users?

The beauty of this idea is that OCaml will share the same runtime as JS so that interop is really easy and you can use your favorite language in your daily jobs. I also shared my ideas with JSOO devs: discuss: compiling rawlambda output to javascript · Issue #338 · ocsigen/js_of_ocaml (github.com)

The project was originally named OCamlScript but it has a name conflict with an existing package so that it is renamed into BuckleScript.

At a similar time, ReasonML syntax was released, it is a JS friendly syntax for people to take advantage of OCaml. The combination of ReasonML and BuckleScript is very natural. To avoid confusion to users, we call it ReasonML or projects under the ReasonML umbrella – but the heavy lifting is mostly done by BuckleScript.

The development of ReasonML syntax has slowed down since 2018. What worries me is that the remaining commits keep adding stuff that we don’t want. To maintain the compatibility with OCaml for each release, the ReasonML syntax keeps a snapshot for all versions of OCaml AST, the code gets bloated quickly that the parser is even larger than the compiler itself – this is hard to believe, it means that all my hard work to squeeze the performance will be wasted in the parser.

I wish the very few resources could be put into bug fixes instead. We communicated and realized that the top priority for ReasonML syntax is the compatibility for OCaml ecosystem while our top priority is providing the best dev experience for JS users.

It is open source projects and people can have different visions so that we decide a peaceful separation.

At a similar time, Iwan – one of the major contributors of ReasonML syntax, shows us a POC of a handwritten parser which supports the whole language. This is something I want but do not have time to do – a hand written parser for the best error message and better dev experience.

The original goal is to have the hand written parser 100% compatible with ReasonML syntax so that we can finish the transition, but it ends up being slightly different.

Welcome to a double helping of the multicore monthlies, with December 2020 and January 2021 bundled together (the team collectively collapsed into the end of year break for a well deserved rest). We encourage you to review all the previous monthly updates for 2020 which have been compiled by @shakthimaan, @kayceesrk, and me.

Looking back over 2020, we achieved a number of major milestones towards upstreaming multicore OCaml. The major highlights include the implementation of the eventlog tracing system to make debugging complex parallelism practical, the enormous rebasing of from OCaml 4.06 to 4.11, a chapter on parallel programming, the publication of "Retrofitting Parallelism onto OCaml" at ICFP 2020, the production use of the Sandmark benchmark, and the implementation of system threading integration. While all this was happening in the multicore code trees, the upstreaming efforts into mainline OCaml also went into full gear, with @xavierleroy leading the efforts from the core team to ensure that the right pieces went into various releases of OCaml with the same extensive code review as any other features get.

The end of 2020 saw enhancements and updates to the ecosystem libraries, with more tooling becoming available. In particular, we would like to thank:

• @mattpallissard for getting merlin and dot-merlin-reader working with Multicore OCaml 4.10. This makes programming using OCaml Platform tools like the VSCode plugin much more pleasant.
• @eduardorfs for testing the no-effect-syntax Multicore OCaml branch with a ReasonML project.

@kayceesrk also gave a couple of public talks online:

• Multicore OCaml - What's coming in 2021, hosted by Nomadic Labs.
• Effect handlers in Multicore OCaml. NUS PLV Research Seminar.

We're really grateful to the OCaml core developers for giving this effort so much of their time and focus in 2020! We're working on a broader plan for 2021's exciting multicore roadmap which will be included in the next monthly after a core OCaml developer's meeting ratifies it soon. The broad strategy remains consistent: putting pieces of functionality steadily into each upcoming OCaml release so that each can be reviewed and tested in isolation, ahead of the OCaml 5.0 release which will include domains parallelism.

With OCaml 4.12 out in beta, our January has mainly been spent tackling some of the big pieces needed for OCaml 4.13. In particular, the safe points PR has seen a big update (and corresponding performance improvements), and we have been working on the design and implementation of Domain-Local Allocation Buffers (DLAB). We've also started the process of figuring out how to merge the awesome sequential best-fit allocator with our multicore major GC, to get the best of both worlds in OCaml 5.0. The multicore IO stack has also restarted development, with focus on Linux's new io_uring kernel interface before retrofitting the old stalwart epoll and kqueue interfaces.

Tooling-wise, the multicore Merlin support began in December is now merged, thanks to @mattpallissard and @eduardorfs. We continue to work on the enhancements for Sandmark 2.0 benchmarking suite for an upcoming alpha release – @shakthimaan gave an online seminar about these improvements to the multicore team which has been recorded and will be available in the next monthly for anyone interested in contributing to our benchmarking efforts.

As with previous reports, the Multicore OCaml updates are listed first for the month of December 2020 and then January 2021. The upstream OCaml ongoing work is finally mentioned for your reference after the multicore-tree specific pieces..

Does this mean ocaml 5.0 will have a concurrency baked right in to the default ocaml? If so, this is an excellent news also how does this fit in to the mirage stack? i.e. will ocaml 5.0 concurrency work with mirage too?

OCaml 5.0 is slated to have domains-only parallelism and no fibres, but stay tuned for more news on our plans for fibres shortly. The Mirage prototype is looking very good so far, but I'm not quite ready to talk about it in more detail until it's a bit more baked :slight_smile:

Simon Cruanes:

Will there be only one runtime in 5.0? As in, multicore will be the default, not an opt-in switch as happened with flambda1?

The intention is to have a single runtime, yes. A huge amount of the work on the multicore project to date (see Retrofitting Parallelism onto OCaml) has been focussed on reducing the performance hit to sequential code when running on a parallel-capable runtime. If we end up with multiple runtimes in OCaml 5.0 after all this work, you'll find me sobbing quietly in a corner of the developer meeting.

Anurag Soni:

This is exciting to hear. Is the project public? I’ve spent some time over the past couple of months working on wrappers around the various polling interfaces (I started with kqueue and select ), and I’ve been working on io-uring bindings as well. After reading this months’s multicore report I think i’d rather work on testing/contributing to an effort maintained by the multicore team (There are gaps in my ctypes/ocaml runtime knowledge) instead of duplicating some effort

I've just pushed the non-multicore bits of the uring bindings over to https://github.com/ocaml-multicore/ocaml-uring. I'm still iterating on them, as there's a very specific design that minimises heap pressure that we're aiming towards. So not quite ready for contributions yet, but should be soon. The bindings there should be perfectly usable for pre-OCaml 5.0 use once that design settles (and in fact, I'll probably add it to Lwt in order to give us experience using io_uring in production ahead of any default use in multicore OCaml).

I've got to say though – so far, io_uring has absolutely exceeded my expectations. I never want to look at epoll again now. And a side worry is that the BSDs are steadily getting left behind in terms of post-POSIX IO standards. Between io_uring in Linux, Grand Central Dispatch in macOS, and iocp in Windows, we're spoilt for async choice in the "mainstream" operating systems.

if anyone is playing with multicore, I made this Lwt engine which is capable of running things like opium partially in parallel, 2x gains when running with 4 domains, it doesn't scale a lot, but 2x is quite nice.

Just drop this in a file and setup Lwt with it.

https://gist.github.com/EduardoRFS/0000a193a5a7b964651ef532b412ce5f

On behalf of Nomadic Labs, it is my pleasure to announce the release of resto v0.6 and v0.6.1. Resto is set of packages to build REST RPC directories and to serve and call the built directories. It is split into different packages (resto, resto-directory, resto-cohttp-server, resto-cohttp-client, etc.) so as to minimise dependencies (e.g., you can call a directory without pulling in the server dependencies).

Both versions share the same code (see change log below), but the licensing on the code had been mixed up. This is the reason for the two releases: the version 0.6 is distributed under LGPL with linking exception and the version 0.6.1 is distributed under MIT. Future releases will be under MIT only.

The library can be installed from opam. The code is hosted on Gitlab: https://gitlab.com/nomadic-labs/resto The documentation is available online: https://nomadic-labs.gitlab.io/resto/

Other than the above packaging/licence fixes, the new versions:

• Fix a URL percent-decoding bug in the server
• Provide access-control capability to selectively allow/forbid some services
• Provide a self-serving client that acts as its own server
• Provide support for chunk transfer encoding for large values
• Improves the logging and the logging levels

I'm glad to announce the first release of paf, a simple MirageOS layer for http/af with a TLS support. Two simple unikernels exist into the distribution to show how to use paf with MirageOS.

$ cd unikernel/client
$ mirage configure
$ make depends
$ mirage build
$ ./minipaf -u https://discuss.ocaml.org/

I've been working on a mailing list platform for the past few months in my spare time, and I thought I would share it here since I'm building the project using Ocsigen.

The project is still in early stages, but I hope the community finds this interesting.

postheat.com