OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of July 28 to August 11, 2020.

Table of Contents

Is Tezos the only well known blockchain written in Ocaml?

subarashiku asked and Davide Gessa replied

Is Tezos the only well known blockchain written in Ocaml?

there is also:

caml_qiskit 0.1.0: quantum computing for Ocaml

Davide Gessa announced

Hi developers, today I released a (yet) tiny OCaml binding for the quantum computing sdk of IBM, named qiskit. Qiskit is a python library, so I used pyml for wrapping python classes and modules over OCaml (that was very easy, pyml worked like a charm).

At the moment the library allows to:

  • create a quantum circuit using all the standard gates and draw it
  • simulate the quantum circuit execution locally
  • send the circuit to IBM quantum computer for real hardware execution

The source code of this is available here: https://github.com/dakk/caml_qiskit , and you will be able to install it using opam where its name is qiskit; the library needs python3 + qiskit + matplotlib + numpy to work.

I wrote a medium post showing a basic example: https://medium.com/@dakk/ocaml-and-quantum-computing-fcf4b60d3159

d2dd5543922e19e547929e9fd97f661836e4c01b.png

Chet Murthy then said

Hi, you might be interested in this code: https://github.com/chetmurthy/qc-ocaml
tutorial: https://github.com/chetmurthy/qc-ocaml/blob/master/docs/Tutorial.asciidoc

I wrote it a good while ago, but IIRC I was able to create a circuit, compile to a "circuit", submit a job, monitor its progress, recover results.

I didn't get around to interfacing with IBM's simulator.

If you find the code useful, I'm happy to re-license under GPL.

Multicore OCaml: July 2020

Anil Madhavapeddy announced

Welcome to the July 2020 Multicore OCaml report! This update, along with the previous updates, has been compiled by @shakthimaan, @kayceesrk and myself. There are a number of advances both in upstream OCaml as well as our multicore trees.

Multicore OCaml

  • Thread compatibility via Domain Execution Contexts

    TL;DR: once #381 is merged, dune will work with multicore OCaml.

    As I noted last month, not having a Thread module that is backwards compatible with traditional OCaml's is a big blocker for ecosystem compatibility. This can be a little confusing at first glance – why does Multicore OCaml need non-parallel threading support? The answer lies in the relationship between concurrency and parallelism in multicore OCaml. Concurrency is how we partition multiple computations such that they run in overlapping time periods, and parallelism is how we run them on separate cores simultaneously to gain greater performance. A number of packages (most notably, Dune) currently use the Thread module to conveniently gain concurrency while writing straight-line code without using monadic abstractions. These uses do not require parallelism, but are very difficult to rewrite to not use thread-based concurrency.

    Therefore, multicore OCaml also needs a way to provide a reasonably performant version of Thread. The first solution we attempted (started by @jhw and continued by @engil in #342) mapped a Thread to a multicore Domain, but scaled poorly for a larger number of threads since we may have a far greater number of concurrency contexts (Thread instances) than we have CPUs available (Domain instances). This lead to a bit of brainstorming (#357) to figure out a solution that would work for applications like Dune or the XenServer stack that are heavy Thread users.

    Our solution introduces a concept that we have dubbed Domain Execution Contexts in #381, which allows us to map multiple system threads to OCaml domains. Once that PR is reviewed and merged into the multicore OCaml branches, it will unlock many more ecosystem packages, as the Dune build system will compile unmodified. The last "big" remaining blocker for wider opam testing after this is then ocaml-migrate-parsetree, which requires a small patch to support the effect keyword syntax that is present in the multicore OCaml trees.

  • Domain Local Storage

    Domain Local Storage (DLS) (#372) is a simple way to attach OCaml values privately to a domain. A good example of speedup when using DLS is shown in a PR to the LU decomposition benchmark. In this case, the benchmark needs a lot of random numbers, and initialising them in parallel locally to the domain is a win.

    Another example is the parallel implementation of an evolutionary algorithm (originally suggested by @per_kristian_lehre in #336) which speeds up nicely in #151 (for those who want to check the baseline, there is a sequential version in #155 that you can look up in the Sandmark web interface).

  • Parallel Programming with Multicore OCaml (document)

    A tutorial on Parallel Programming with Multicore OCaml has been made available. It provides an introduction to Multicore OCaml and explains the concepts of Domains, Domainslib, and Channels. Profiling of OCaml code using perf and Eventlog are also illustrated with examples.

    This draft was shared on Reddit as well as on HackerNews, so you'll find more chatter about it there.

  • Coq benchmarks

    The Sandmark benchmarking suite for OCaml has been successfully updated to use dune.2.6.0 and builds for Multicore OCaml 4.10.0. With this major upgrade, we have also been able to include Coq and its dependencies. We are working on adding more regression Coq benchmarks to the test suite.

Upstream OCaml

The upstream OCaml trees have seen a flurry of activity in the 4.12.0dev trees with changes to prepare for multicore OCaml. The biggest one is the (to quote @xavierleroy) fabled page-less compactor in ocaml/ocaml#9728. This followed on from last month's work (#9698) to eliminate the use of the page table when the compiler is built with the "no-naked-pointers" option, and clears the path for the parallel multicore OCaml runtime to be integrated in a future release of OCaml.

One of the other changes we hope to get into OCaml 4.12 is the alignment of the use of garbage collector colours when marking and sweeping. The #9756 changes make the upstream runtime use the same scheme we described in the Retrofitting Parallelism onto OCaml ICFP paper, with a few extra improvements that you can read about in the PR review comments.

If you are curious about the full set of changes, you can see all the multicore prerequisite issues that have been closed to date upstream.

Detailed Updates

As with the previous updates, the Multicore OCaml updates are first listed, which are then followed by the enhancements to the Sandmark benchmarking project. The upstream OCaml ongoing and completed updates are finally mentioned for your reference.

  • Multicore OCaml
    • Ongoing
      • ocaml-multicore/ocaml-multicore#342 Implementing the threads library with Domains

        This is an on-going effort to rebase @jhwoodyatt's implementation of the Thread library for Domains.

      • ocaml-multicore/ocaml-multicore#357 Implementation of systhreads with pthreads

        A Domain Execution Context (DEC) is being introduced in this implementation as a concurrency abstraction for implementing systhreads with pthreads.

      • ocaml-multicore/ocaml-multicore#374 Force major slice on minor collection

        A blocked thread in a domain may not progress the major GC when servicing the minor collector through handle_interrupt, and hence we need to have a minor collection to schedule a major collection slice.

    • Completed
      • Domain-Local State
      • Removal of vestiges in Concurrent Minor GC
        • ocaml-multicore/ocaml-multicore#370 Remove Cloadmut and lloadmut

          The Cloadmut and Iloadmut implementation and usage have been cleaned up with this patch. This simplifies the code and brings it closer to stock OCaml.

        • ocaml-multicore/ocaml-multicore#371 Domain interrupt cleanup

          In runtime/domain.c the struct interruptor* sender has been removed. The domain RPC functions have been grouped together in domain.h, and consistent naming of definitions have been applied.

      • Code Cleanup
      • Sundries
        • ocaml-multicore/ocaml-multicore#366 Add event to record idle domains

          The domain/idle_wait and domain/send_interrupt events are added to track domains that are idling. An eventlog screenshot with this effect is shown below:

          a7028229dbf95c975c65faecc99417268024520a_2_1380x596.png

        • ocaml-multicore/ocaml-multicore#369 Split caml_urge_major_slice into caml_request_minor_gc and caml_request_major_slice

          The caml_urge_major_slices is split into caml_request_minor_gc and caml_request_major_slice. This reduces the total number of minor garbage collections as observed in the following illustration:

          66705c292bab15146adedf8d9f870484a6a87e87_2_1380x406.png

        • ocaml-multicore/ocaml-multicore#373 Fix the opam pin command in case the current directory name has spaces

          Use the -k path command-line argument with opam pin to handle directory names that have whitespaces.

        • ocaml-multicore/ocaml-multicore#375 Only lock the global freelist to adopt pools if needed

          The lock acquire and release on allocation is removed when there are no global pools requiring adoption.

        • ocaml-multicore/ocaml-multicore#377 Group env vars for run in travis CI

          The OCAMLRUNPARAM parameter is defined as part of the environment variable with the USE_RUNTIME=d command.

        • ocaml/dune#3608 Upstream Multicore dune bootstrap patch

          The patch is used to build dune using the secondary compiler approach for ocaml/dune#3548.

  • Benchmarking
    • Ongoing
      • ocaml-bench/sandmark#107 Add Coq benchmarks

        The upgrade of Sandmark to use dune.2.6.0 for Multicore OCaml 4.10.0 has allowed us to install Coq and its dependencies. We are currently working on adding more Coq regression benchmarks to Sandmark.

      • ocaml-bench/sandmark#122 Measurements of code size

        The code size of a benchmark is one measurement that is required for flambda branch, and we are exploring adding the same to the Sandmark bench runs.

      • ocaml-bench/sandmark#142 [RFC] How should a user configure a sandmark run?

        We are gathering user feedback and suggestions on how you would like to configure benchmarking for Sandmark. Please share your thoughts and comments in this discussion.

      • ocaml-bench/sandmark#150 Coq files that work

        Addition of more Coq files for benchmarking in Sandmark.

    • Completed
      • Dune 2.6.0 Upgrade
        • ocaml-bench/sandmark#131 Update decompress benchmarks

          The decompress benchmarks were updated by @dinosaure to use the latest decompress.1.1.0 for dune.2.6.0.

        • ocaml-bench/sandmark#132 Update dependency packages to use dune.2.6.0 and Multicore OCaml 4.10.0

          Sandmark has now been updated to use dune.2.6.0 and Multicore OCaml 4.10.0 with an upgrade of over 30 dependency packages. You can test the same using:

          $ opam install dune.2.6.0
          $ make ocaml-versions/4.10.0+multicore.bench
          
      • Coq Benchmarks
      • Continuous Integration
        • ocaml-bench/sandmark#136 Use BUILD_ONLY in .drone.yml

          The .drone.yml file has been updated to use a BUILD_ONLY environment variable to just install the dependencies and not execute the benchmarks for the CI.

        • ocaml-bench/sandmark#147 Add support to associate tags with benchmarks

          The macro_bench and run_in_ci tags have been introduced to associate with the benchmarks. The benchmarks tagged as run_in_ci will be executed as part of the Sandmark CI.

      • Sundries
        • ocaml-bench/sandmark#124 User configurable paramwrapper added to Makefile

          The --cpu-list can now be specified as a PARAMWRAPPER environment variable for running the parallel benchmarks.

        • ocaml-bench/sandmark#134 Include more info on README

          The README has been updated to include documentation to reflect the latest changes in Sandmark.

        • ocaml-bench/sandmark#141 Enrich the variants with additional options

          The ocaml-versions/* files now use a JSON file format which allow you to specify the ocaml-base-compiler source URL, configure options and OCAMLRUNPARAMS. An example is provided below:

          {
          "url" : "https://github.com/ocaml-multicore/ocaml-multicore/archive/parallel_minor_gc.tar.gz",
          "configure" : "-q",
          "runparams" : "v=0x400"
          }
          
        • ocaml-bench/sandmark#146 Update trunk from 4.11.0 to 4.12.0

          Sandmark now uses the latest stock OCaml 4.12.0 as trunk in ocaml-versions/.

        • ocaml-bench/sandmark#148 Install python3-pip and intervaltree for clean CI build

          The .drone.yml file has been updated to install python3-pip and intervaltree software packages to avoid errors when the Makefile is invoked.

  • OCaml
    • Ongoing
      • ocaml/ocaml#9722 EINTR-based signals, again

        The patch provides a new implementation to solve locking and signal-handling issues.

      • ocaml/ocaml#9756 Garbage collector colours change

        The PR removes the gray colour in the garbage collector (GC) colour scheme in order to use it with the Multicore OCaml major collector.

    • Completed
      • ocaml/dune#3576 In OCaml 4.12.0, empty archives no longer generate .a files

        A native archive will never be generated for an empty library, and this fixes the compatibility with OCaml 4.12.0 when dealing with empty archives.

      • ocaml/ocaml#9541 Add manual page for the instrumented runtime

        The manual/manual/cmds/instrumented-runtime.etex document has been updated based on review comments and has been merged to stock OCaml.

      • ocaml/ocaml#9728 Simplified compaction without page table

        A self-describing closure representation is used to simplify the compactor, and to get rid of the page table.

      We would like to thank all the OCaml developers and users in the community for their continued support, code reviews, documentation and contributions to the multicore OCaml project.

  • Acronyms
    • CI: Continuous Integration
    • DEC: Domain Execution Context
    • GC: Garbage Collector
    • OPAM: OCaml Package Manager
    • PR: Pull Request
    • RFC: Request for Comments
    • RPC: Remote Procedure Call

ANN: benchpress 0.1

Simon Cruanes announced

I'm glad to announce that benchpress is now available in 0.1.

Benchpress is a test/benchmarking tool designed for automated theorem provers (including SMT solvers and first-order provers), both to run a bunch of provers, and to display their results. Results are stored in sqlite files, one per run. The sister package benchpress-server contains a daemon that provides a web interface to examine the results; a read-only live instance can be seen here.

These are early days for benchpress but it could be of interest to the sub-community of logicians here :slightly_smiling_face:

Announcing a new maintainer for Lwt

Anton Bachin announced

I am pleased to announce that Raphaël Proust (@raphael-proust) is taking over as maintainer of Lwt.

Raphaël Proust is a long-time Lwt contributor. Outside the repo, he has released several libraries using Lwt, and has written a very helpful Introduction to it — among other output. He is currently working at Nomadic Labs.

containers 3.0

Simon Cruanes announced

I'm happy to announce, on behalf of containers' contributors, the release of containers 3.0. API documentation can be found here.

Containers is a BSD-licensed standard library extension [1] that aims at being lightweight, convenient, modular, portable, and only pay for what you use; that includes compatibility with OCaml >= 4.03. This is the second major update in 7 years of existence, following semantic versioning.

Release 3.0 was the opportunity to clean up some inconsistencies (in printers, among others), to focus on the standard Seq.t type, and to split some sub-libraries into their own packages (containers-thread and containers-data respectively). The hope is that the new version is more consistent, lightweight, and pleasant to use.

I want to thank all contributors for their hard work, and in particular Fardale.

[1]: as in, containers extends the stdlib and does not intend to replace it.

Overview of breaking changes

(this is extracted from the readme's migration section)

  1. The biggest change is that some sub-libraries have been either turned into their own packages (containers-thread, containers-data), deleted (containers.iter),or merged elsewhere (containers.sexp). This means that if use these libraries you will have to edit your dune~/~_oasis~/~opam files.
    • if you use containers.sexp (i.e. the CCSexp module), it now lives in containers itself.
    • if you used anything in containers.data, you need to depend on the containers-data package now.
  2. Another large change is the removal (at last!) of functions deprecated in 2.8, related to the spread of Seq.t as the standard iterator type. Functions like CCVector.of_seq now operate on this standard Seq.t type, and old-time iteration based on iter is now named of_iter, to_iter, etc.

    Here you need to change you code, possibly using search and replace. Thankfully, the typechecker should guide you.

  3. Array_slice and String.Sub have been removed to simplify the code and String more lightweight. There is no replacement at the moment. Please tell us if you need this to be turned into a sub-library.
  4. Renaming of some functions into more explicit/clear names. Examples:
    • CCVector.shrink is now CCVector.truncate
    • CCVector.remove is now CCVector.remove_unordered, to be contrasted with the new CCVector.remove_and_shift.
    • CCPair.map_fst and map_snd now transform a tuple into another tuple by modify the first (resp. second) element.
  5. All the collection pretty-printers now take their separator/start/stop optional arguments as unit printer (i.e. Format.formatter -> unit -> unit functions) rather than strings. This gives the caller better control over the formatting of lists, arrays, queues, tables, etc.
  6. Removal of many deprecated functions.

New packages: js_of_ocaml-webidl and js_of_ocaml-webgpu

Misha Aizatulin announced

Announcing two packages. One generates jsoo bindings from webidl definitions. The other package is a specific application: a complete generated API for WebGPU .

Project page: https://github.com/tari3x/webgpu
Documentation: https://tari3x.github.io/webgpu

opam-bin: binary packages for OPAM, beta release

Fabrice Le Fessant announced

I am happy to announce the first public release of opam-bin, a framework to create, use and share binary packages with opam:

https://ocamlpro.github.io/opam-bin

With opam-bin, you can :

  • build binary packages while installing their source counterpart with opam
  • automatically reuse previously created binary packages instead of compiling them again
  • export and share your binary packages as part of opam repositories for other users/computers to use

opam-bin is a framework in 3 parts :

This is the first public release, it should be considered as a "beta release", though we have tested it a lot in the last days.

opam-bin is a collaborative work between OCamlPro and Origin Labs.

Fabrice Le Fessant later added

Little update: the binary repository for OCaml 4.10.0 has been expanded to contain a set of 1800+ binary packages:

https://www.origin-labs.com/opam-bin/debian10.4-amd64/4.10.0

New packages: plist-xml 0.1 and plist-xml-lwt 0.1

dosaylazy announced

I would like to announce my first two OPAM packages, plist-xml and plist-xml-lwt. These two packages build upon the markup and markup-lwt libraries to read and write plist files expressed as XML.

I originally wrote this code to process the TextMate grammar files found in https://github.com/github/linguist/tree/master/vendor. I hope that other people can find good use of these two libraries. Please report any bugs as well as any inconveniences in the API.

Call to testers (OCaml 4.11.0, release candidate)

Kate announced

The release of OCaml version 4.11.0 is imminent. We have created a release candidate that you can test. For users of the beta releases, this release candidate is exactly the same as the last beta release except for a minor fix for the #show directive in the toplevel.

— :sparkles: The opam ecosystem ---

The opam ecosystem is in good shape for this release candidate. For maximum compatibility, you can use the following opam-alpha-repository:

$ opam repo add alpha https://github.com/kit-ty-kate/opam-alpha-repository.git

during your tests. This repository integrate not-yet-upstreamed or unreleased fixes to various packages. With this repository all except a handful of packages are compatible with this release candidate.

Using opam-alpha-repository:

including Merlin :tada: – work, except for ocaml-lsp-server (which is not officially released yet and only relevant for Windows users).

  • All packages except 7 over the 2134 available packages in opam-repository that were compatible

with OCaml 4.10 are also compatible with OCaml 4.11.

— :checkered_flag: Call to testers ---

Given that the release is very close and the state of the community packages stable enough, we would like to encourage people to give it a try before the release, especially for those on non-x86_64 architectures. I've personally been using OCaml 4.11 almost exclusively for the past 3 months and did not really encountered any issues on my x86_64 machine with a fairly regular setup.

Give it a try! Happy testing! :milky_way:

Anil Madhavapeddy then said

And a huge thank you to @kit-ty-kate who has done an extraordinary amount of behind-the-scenes work for this release. This is the first time we're experimenting with the idea of an "OCaml Readiness Team" of the maintainers of the various core tools, and she has superbly coordinated those projects along with the wider opam-repository (along with a large amount of direct compatibility fixing).

Due to this effort, for the first time since we started disaggregating tools out of the core OCaml distribution starting about 8 years ago, the forthcoming releases of OCaml should come with tooling that is ready for that release from day 1. Props to all the maintainers for your hard work – I'll be talking about how this is working more in the forthcoming OCaml Workshop in a few weeks.

OCaml 4.10.1, first release candidate

Kate announced

The release of OCaml version 4.10.1 is imminent. @octachron and the rest of the compiler dev team have created a release candidate that you can test.

The source code is available at these addresses:

https://github.com/ocaml/ocaml/archive/4.10.1+rc1.tar.gz
https://caml.inria.fr/pub/distrib/ocaml-4.10/ocaml-4.10.1+rc1.tar.gz

The compiler can also be installed as an OPAM switch with one of the following commands.

opam switch create ocaml-variants.4.10.1+rc1 --repositories=default,beta=git://github.com/ocaml/ocaml-beta-repository.git

We want to know about any bugs. Please report them here: https://github.com/ocaml/ocaml/issues

If every goes well, the full release should follow next week.

— OCaml 4.10.1 changes ---

This version contains a collection of configuration, build systems and runtime fixes:

  • Runtime system:
    • #9344, #9368: Disable exception backtraces in bytecode programs built with "-output-complete-exe". At the moment, such programs do not embed debug information and exception backtraces where causing them to crash. (Jérémie Dimino, review by Nicolás Ojeda Bär)
  • Build system:
    • #9531: fix support for the BFD library on FreeBSD (Hannes Mehnert, review by Gabriel Scherer and David Allsopp)
  • Bug fixes:
    • #9068, #9437: ocamlopt -output-complete-obj failure on FreeBSD 12 (Xavier Leroy, report by Hannes Mehnert, review by Sébastien Hinderer)
    • #9165: Add missing -function-sections and -O3 flags in Makefiles. (Greta Yorsh, review by David Allsopp)
    • #9495: fix a bug where bytecode binaries compiled with -output-complete-exe would not execute at_exit hooks at program termination (in particular, output channels would not be flushed). (Nicolás Ojeda Bär, review by David Allsopp)
    • #9714, #9724: Use the C++ alignas keyword when compiling in C++ in MSVC. Fixes a bug with MSVC C++ 2015 onwards. (Antonin Décimo, review by David Allsopp and Xavier Leroy)
    • #9736, #9749: Compaction must start in a heap where all free blocks are blue, which was not the case with the best-fit allocator. (Damien Doligez, report and review by Leo White)
  • Tools:
    • #9552: restore ocamloptp build and installation (Florian Angeletti, review by David Allsopp and Xavier Leroy)

OCamlformat 0.15.0

Guillaume Petiot announced

On behalf of the development team, I’d like to announce the release of ocamlformat version 0.15.0 :tada:.

Here are the main highlights of this release:

Support for OCaml 4.11

This means both that it compiles and runs using this version, but also that it can format 4.11-specific language features (quoted extensions: {%foo|...|}).

Compatibility with base and stdio v0.14.0

This compatibility has been restored since ocamlformat-0.14.3 but wasn't publicly announced, ocamlformat is now compatible from base v0.12.0 to v0.14.0 included.

Bugfixes

Many bugs and unoptimal formatting were also fixed in this release:

  • Do not break inline elements such as {i blah} in docstrings
  • Distinguish hash-getter from hash-comparison infix operators. Operators of the form #**# or #**. where ** can be 0 or more operator chars are considered getter operators and are not surrounded by spaces, as opposed to regular infix operators
  • Type constraint on return type of functions is now always printed before the function body
  • Restore previous functionality for pre-post extension points
  • Fix extra break before function body of a fun Indent further args of anonymous functions
  • Do not clear the emacs *compilation* buffer on successful reformat
  • Fix disabling with attributes on OCaml < 4.08
  • Preserve unwrapped comments by not adding artificial breaks when wrap-comments=false and ocp-indent-compat=true are set to avoid interfering with ocp-indent indentation
  • Break long literal strings at the margin
  • Break after a multiline argument in an argument list
  • Remove unnecessary parens around object
  • Fix placement of comments on constants
  • Do not escape arguments of some Odoc tags. The characters []{} must not be escaped in the arguments of @raise, @author, @version and others.
  • Fix missing open line between multi-line let-binding with poly-typexpr
  • Remove trailing space after expression when followed by an attribute and break before attributes attached to multi-line phrases
  • Do not add a space to minimal comments (* *), (** *) and (*$ *)
  • Fix attributes position in labelled arguments type
  • Add missing parens around type annotation in anonymous function
  • Fix alignment of 'then' keyword in parenthesised expression
  • Recognise eliom file extensions

A note for new users

We encourage you to try ocamlformat, that can be installed from opam directly ( opam install ocamlformat ), but please remember that it is still beta software. We added a FAQ for new users that should help you decide if ocamlformat is the right choice for you.

ocamlnet-4.1.8

Gerd Stolpmann announced

there is now ocamlnet-4.1.8 available:

  • compatibility with upcoming OCaml-4.11

See the project page for download, documentation, a detailed changelog, and the mailing list: http://projects.camlcity.org/projects/ocamlnet.html

The repository is at

https://gitlab.com/gerdstolpmann/lib-ocamlnet3/

opam follows soon.

Other OCaml News

From the ocamlcore planet blog

Here are links from many OCaml blogs aggregated at OCaml Planet.

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.