OCaml Weekly News

I would like to announce the release of Caqti 1.2.0. Caqti is an abstraction layer over RDBMS client libraries, including MariaDB, PostgreSQL, and Sqlite3. It uses asynchronous calls where available to provide monadic concurrency, implements connection pooling, provides encoding and decoding of parameters and results, and aids in formulating query strings usable across databases.

I am pleased to see outside contributions for this release; the changes since 1.1.0 are:

• Add a signature for the populate function, and add basic implementations for all drivers (GPR#28).
• Add stream for result extraction (GPR#22 Nathan Rebours).
• Use the postgres driver when postgres is specified as the URL scheme (GPR#25 James Owen).
• Documentation improvements (GPR#20 Nathan Rebours, etc.).
• Reimplemented partly ineffective prepare-cache for PostgreSQL.
• Backwards incompatible changes to the driver API.
• Backwards incompatible changes to modules marked internal but exposed due to being split across packages.
• Fix forward compatibility past OCaml 4.08 as announced by deprecations.

About the first point, it should be mentioned that the new populate function has no optimized implementation yet. It will map to COPY-FROM for PostgreSQL.

Project page: https://github.com/paurkedal/ocaml-caqti/
API documentation: http://paurkedal.github.io/ocaml-caqti/index.html

It is our pleasure the announce the first release of ROTOR: a tool for refactoring multi-file OCaml codebases. Currently, it just performs renaming of value bindings, but the intention is for ROTOR to support a full range of refactoring tasks, and to be the go-to refactoring tool for the OCaml community.

ROTOR has been developed at the University of Kent, with the support of Jane Street, as part of a research project on trustworthy refactoring.

It is available now on OPAM:

opam install rotor

This release is an alpha version, and so we'd love you all to try it out and give us your feedback. Since it is still a prototype, we are certainly not making any claims that it works perfectly in all cases, but we think it is at a stage where it can start to be useful to the community, and where it will be useful for us to seek community feedback.

Please raise any issues on the gitlab page. Please also email us to report on your experiences.

Currently, ROTOR is invoked from the command-line:

rotor rename [options] <from-id> <to-id>

ROTOR can determine the files that comprise your codebase using .merlin files, and outputs a diff patch which can be applied to effect the renaming. For the future, we are looking at integrating within IDEs, which will make it easier to use.

An « experimental » FileRotating Handler was added in version 0.6.2 of Easy_logging. Behaviour is about the same as the python one, except that rotating parameters (max_kbytes and backup_count) shouldn't be 0.

In the OCaml library ecosystem, there are two important pieces of software that are being relied on for managing our code:

• ocamlfind, lead by Gerd Stolpmann has been the standard for managing OCaml libraries for many years. It provides a convention for storing libraries on the filesystem, and a META file format for describing the properties of libraries. There is a CLI tool that interprets META files and makes it easy to embed within build systems.
• dune, lead by @diml, is an specialised build system for OCaml that aims to be fast and expressive. Dune's functionality complements but doesn't fully replace ocamlfind – it can interpret META files as installed by ocamlfind, but it also now has newer features such as virtual libraries which cannot be expressed by ocamlfind. Dune installs a dune-package file with every library that contains its full metadata, and translates the compatible subset into the META format as well.

This has lead to some incompatibilities in recent months, since independent build systems now need to handle two separate formats (META and dune-package). Therefore, the authors of both systems have gotten together to make the following changes to improve the consistency of our community:

• ocamlfind has moved to github.com/ocaml/ocamlfind from its former home at GitLab. Gerd will work on migrating links and merge requests from the old repository to the new home. This also belatedly recognises ocamlfind as a very important piece of the OCaml Platform and ecosystem, by hosting it at the ocaml/ GitHub repository alongside the compiler, opam, dune and the ocaml.org website.
• There is a new ocaml/ocaml-library-standard repository where we will work on a documented standard for both dune and ocamlfind to use for the purposes of clean interoperability. This is just starting up this week, and so we will give Gerd and @diml some months to establish the common ground between the projects. If anyone has specific asks for features that are not covered by ocamlfind and dune at the moment from a packaging perspective, then this is a good time to get in touch. Note that new feature requests may not make the first version of the spec – we are keen to ensure that the existing status is reflected initially to solve the immediate problem, and so the focus is on documenting the existing interactions between the OCaml compiler, ocamlfind, and dune.

It's really good news.

Opam is obviously another key OCaml tool. I should admit that I'm not 100% clear about how the packages are exactly handled at the lowest level, especially when coming to the cooperation between dune, ocamlfind and opam. See https://discuss.ocaml.org/t/connection-between-libraries-in-opam-dune-and-findlib/2536 Especially about the "1-1 mapping between opam packages and findlib libraries enforced by dune to make the difference (between packages and libraries) manageable ".

Obviously, the ideal solution would be to get only one package&libraries file for each Ocaml program in order to feed opam and dune with reliable information (for package and build management, respectively).

Can you tell us if the ongoing effort you present will solve that? Or if another effort should be made, and give us hints about how to contribute?

dune itself has features these days to generate opam files from dune metadata – see ocaml-github for an example project that uses this functionality (or of course, dune itself). This all greatly helps with reducing the amount of boilerplate we have to maintain, and is an incremental step on the way to reducing the number of namespaces we have without breaking backwards compatibility.

Note that new feature requests may not make the first version of the spec – we are keen to ensure that the existing status is reflected initially to solve the immediate problem, and so the focus is on documenting the existing interactions between the OCaml compiler, ocamlfind, and dune.

But please bear the above in mind. We'll first be working on getting ocamlfind and dune to play well together, and then look at further improvements after that. Each step is quite delicate and needs to not break any existing software.

In parallel, is there any current discussion about future requirements regarding an “integrated package&build OCaml solution”, that could feed a kind of candidate roadmap? Do you have an idea of a next must-have requirement in that field? (I’m not talking about doing it now, just planning smth)

We're planning this in the context of adding namespaces to the compiler and tools, but this is not yet ready. We'll likely run it through an OCaml core developers meeting sometime next year and then post about concrete plans for integration then.

As the author of a few released (odig and omod) eco-system tools and a few yet unreleased ones (e.g. brzo and b0caml) I have given the subject quite a bit of thought and have been compiling OCaml software without additional metadata or ocamlfind for quite some time now without any problem by relying on the fact that OCaml compilation objects are largely self-describing.

While these techniques work, they can certainly be made more efficient and reliable by redefining a bit how OCaml libraries get installed and adding a tiny bit of new metadata to library archives.

I'm largely convinced that we do not need an additional metadata format for the simple task of using libraries and that this problem should be solved upstream – since it then improves the usability globally from using the bare compilers to the toplevel.

I have gathered my thought about this and a possible simple plan for action in this gist. The proposed scheme has the advantage of putting the eco-system on a good conceptual and naming diet with each of the names involved appearing at the file system level which is good for usability (e.g. which library name should I specify to use this module can be answered by looking up the file system).

Dear OCaml community,

It is my pleasure to publicize the OCaml Software Foundation, a non-profit foundation dedicated to grow and strengthen the OCaml community. It is funded by industrial and academic sponsors who wish to contribute back to the community, and its main responsibilty is to spend those funds on actions that benefit the community as a whole.

The Foundation was created by the efforts of Michel Mauny⁰, launched in June 2018; I became its director in January 2019, working with an Executive Board containing Xavier Leroy, Damien Doligez, Yann Regis-Gianas and myself, taking ideas, advice and feedback from the excellent people representing our sponsors (Currently: ahrefs, Jane Street, OCamlPro, Tezos, Bloomberg, Lexifi and SimCorp).

Our website¹ lists a few of our recent actions, such as developing the Learn-OCaml teaching platform, funding Outreachy internships on OCaml, and funding video capture for OCaml Meetups. We have been asking around, whenever we meet people from the community, on how they would recommend using money to improve the community. We have been proposing ideas, working on suggestions from our sponsors and the general community; we are not very good at communication, we do more than we talk about. Of course, as a small organization led by volunteers working part-time, things can take a long time to get going – please be patient when you interact with us.

Our current objective is to reach a funding level of $200K per year, to be split between actions directed at "teaching OCaml"² and general OCaml actions. Once split among several important action areas, this is unfortunately not enough to employ someone full-time in a stable manner, but it can fund more actions that are more limited in time and scope. For example, I have been getting in touch with the maintainers of important bricks of the OCaml ecosystem, wondering about whether it's possible to solidify the ecosystem by supporting their maintenance work where needed.

That's it! I would be happy to answer questions and receive suggestions about the Foundation, in this thread or privately (by email at gabriel.scherer@gmail.com). If your company or institution is interested in a new way to give back to the OCaml community, please consider becoming a sponsor; we will do our best to spend this money in the general interest of the community.

—

⁰: Michel Mauny was one of the earliest adopters of Caml as a programming language; he did his PhD thesis on the Categorical Abstract Machine, which was the basis for the very first Caml implementation – before Caml Light and then OCaml.

¹: at the time I'm posting this, it is not a nice website; I built it in the last few weeks, and web design is not my forte.

²: Yann Regis-Gianas has been spear-heading efforts to turn the Learn-OCaml platform, originally developed by OCamlPro for the OCaml MOOC, into a versatile teaching platform with an open corpus of automatically-graded OCaml exercises. We are interested in supporting and promoting all forms of OCaml teaching, whether they use this technical platform or not (for example, Jupyter notebooks are also used for OCaml teaching).

I've recently released the first version of Protocell which offers yet another way of generating OCaml code from .proto files.

Feature highlights:

• Full support for all proto3 primitive and user-defined types.
• Supports imports and generates one .ml file per .proto file.
• Automagically supplies code for imports of Google's "well-known types" when needed.
• Concise yet comprehensive test framework that cross-checks serializations with those of protoc itself.
• Fully bootstrapped: Protocell uses Protocell-generated code to interact with protoc.
• Lean on dependencies, especially when it comes to the runtime library.
• Supports OCaml compiler versions 4.04.1 and above.
• Decent proto2 support.
• Can generate and parse of protoc's text format (mostly for testing and debugging purposes).

More information and example code can be found at the project's homepage.

I'm still just a newbie when it comes to OCaml, its tooling and ecosystem. I warmly welcome any sort of input or feedback.

On behalf of Nomadic Labs, I'm pleased to announce the release of multiple libraries used in the cryptography section of the Tezos project. All three libraries are available through opam.

blake2 provides the fast and secure hash function blake2b (optimised for 64 bits architecture). You can find more details on the original library on https://blake2.net/ and the bindings on https://gitlab.com/nomadic-labs/ocaml-blake2

uecc (read “micro-ecc”) implements eliptic curve primitives (specifically ECDH and ECDSA). You can find the original library on https://github.com/kmackay/micro-ecc and the bindings on https://gitlab.com/nomadic-labs/ocaml-uecc

hacl implements multiple cryptographic algorithms. The original library, HACL*, developed in F*, can be found on https://github.com/project-everest/hacl-star, and the bindings on https://gitlab.com/nomadic-labs/ocaml-hacl

These releases are part of a broader effort to release in-house bindings and libraries and upstream changes to vendored libraries. Look out for our past and future announces.