OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of January 05 to 12, 2021.

Table of Contents

Marshal determinism and stability

Continuing this thread, David Allsopp said

A couple of notes on Marshal, which I don't think have been covered

  • Although the guarantee is only between identical versions of OCaml, the implementation actually goes to considerable lengths to maintain backwards compatibility (so a value written by older OCaml remains readable in newer OCaml). Our own testsuite, for example, indirectly includes a test which unmarshals a 3.12.1 value. I don't know exactly how far back the support goes.
  • As it happens, the change which affected Unison in 4.08 was the first breaking change to Marshal since either 4.00 or 4.01. The fact that it doesn't break often (and that the two code paths - at least at present - are small) meant I have suggested a few months back that we could in future add an additional flag in the style of Compat_32 to allow values to be written in a way which should be readable on older versions of OCaml. Indeed, it's small enough that flags could be added for the changes in 4.08 (PR#1683) and in 4.11 (PR#8791).
  • Neither point undermines using alternative formats either for network serialisation or persistent storage, for the many reasons discussed above!

Sedlex + Menhir parser for both tty and file parsing

Bernard Sufrin announced

I am a great fan of Menhir, and have used it in several private language projects, using the ulexing scanner generator to provide Unicode-capable scanners.

Alarmed by the obsolescence of ulexing, and needing a utf8-capable scanner in a hurry I decided to (teach myself to) use Sedlex. On the whole the experience was very satisfactory, and I found it straightforward to produce a variant of the sedlexing library which supports buffers with variable chunk sizes, thereby enabling efficient lexing on channels connected to files as well as immediately responsive lexing on channels connected to terminals.

I also wanted to teach myself how to use the error-reporting, incremental, interfaces to Menhir-generated parsers. In the hope that it might be useful to others facing the same learning task, or the problem of adapting Sedlex for efficient interactive use, I have placed the example mock-S-Expression parser that resulted from this excursion in:

Git Repository: github.com/sufrin/InteractiveSedlexMenhirExample

First release of awa-ssh

Hannes Mehnert announced

I'm happy to announce that awa-ssh (https://github.com/mirage/awa-ssh) has just been merged into opam-repository. It is a pure OCaml implementation of the ssh (secure shell, https://en.wikipedia.org/wiki/SSH_(Secure_Shell)) protocol.

This is the initial release, please report issues you encounter.

It was initially developed by Christiano Haesbaert in 2016, and revived mid-2019 by myself and in 2020 it was migrated to the MirageOS organization on GitHub for further development and maintenance.

Both client and server code are present in the library (pure code in the main awa package), though the awa-lwt package implements only a server, and the awa-mirage package implements only a client. Tests and examples are in the test subdirectory.

The MirageOS client has been successfully used to clone git repositories (on private servers, on GitHub, etc.). It supports apart from RSA keys also ED25519 keys (and key exchanges).

Introducing Feather: A lightweight shell-scripting library for OCaml

Charles announced

I wrote a shell scripting library called Feather. I like idea of writing bash-like code quickly, later being able to intersperse OCaml to add more typeful components as needed. It's kind of like Shexp but without the monadic interface and with Async support. (Feather_async)

There's a tutorial and some examples in the link above but here's a quick taste:

open Feather

let lines = find "." ~name:"*.ml"
  |. tr "/" "\n"
  |. map_lines ~f:String.capitalize
  |. sort
  |. process "uniq" [ "-c" ]
  |. process "sort" [ "-n" ]
  |. tail 4
  |> collect_lines
in
String.concat ~sep:", " lines |> print_endline

Let me know if you have any feedback! And feel free to file bug reports here. Hope it ends up being useful, entertaining, or both!

postdoc researcher and research engineer positions for CHERI and Arm verification

Peter Sewell announced

We are looking for postdoctoral researchers and postdoctoral or postgraduate research engineers to help develop semantics and verification to improve the foundations and security of mainstream computer systems, for CHERI and Arm system software verification, at the University of Cambridge. OCaml expertise to help develop verification tools will be especially welcome. Closing date 13 January 2021 - see the advert http://www.jobs.cam.ac.uk/job/28012/.

First ocb (OCaml Badgen) release

zapashcanon announced

A few days ago, I released ocb. It's a library and a command-line tool to generate SVG badges.

To get started quickly:

ocb --label Hello --color green --style flat --labelcolor white --status Goodbye

Will gives this result: SVG example.

My first use case was To.ml where I'm using bisect_ppx to generate and deploy a coverage report. I wanted to display the coverage percentage in the README and tried existing tools but wasn't fully satisfied as they didn't work or were failing randomly. Now, I'm generating the badge directly in a GitHub action.

The project was inspired by badgen. I still have to add support for icons and to improve the documentation but it's usable.

Release of OCaml-Git v3.0 and co

Ulugbek Abdullaev announced

We, the ocaml-git team, are happy to announce a new major release of ocaml-git v3.0 and related libraries.

Release Notes

OCaml-Git v3.0

OCaml-Git is a library that implements git format and protocol implementation in pure OCaml. The library is used by libraries such as irmin, a git-like distributed database, or pasteur, a MirageOS unikernel-based snippet storage service.

Changes

The main goal behind this major release was to get better compatibility with various platforms, including [~MirageOS~](mirage.io), 32-bit platforms, and js_of_ocaml. In order to achieve that, we broke down ocaml-git into several components, which are represented as sub-libraries. We will describe some of those components later in this post.

Along with better support for various platforms, ocaml-git 3.0 also comes with SSH support for fetch/push and various bug fixes.

The rest of the changes are mostly internal and pave a way for interesting features such as a full-blown git garbage collector and wire protocol v2 (announcment and spec).

References:

In the new version of ocaml-git, we try to have better separation of concerns by breaking some of the ocaml-git components into sub-libraries, which do not contain git-specific logic and can be reused for other purposes.

Carton

Git uses PACK files to store old git objects such as commits and transfer objects over wire using git's wire protocols (git-nss library mentioned below implements v1 of the protocol; v2 implementation is in progress).

Carton is a library to work with PACK files. The library does not contain git-specific code, so one can easily reuse the library and PACK format for non-git objects. One can see how ocaml-git uses carton for its purposes here.

References:

  • PR that introduces carton
Git-NSS (Not So Smart)

When one wants to synchronize with a remote repository using git, they need to use git fetch/push. Communication and synchronization/negotiation is defined by git wire protocol, which has two versions: older version 1 and newer leaner version 2. The protocols are defined for four wire transports: HTTP(S), SSH, and git:// (TCP).

Not-So-Smart library is a library that allows for such synchronization based on the git wire protocols but without git-specific code, meaning that files being fetched do not need to be git objects or that there is no assumptions on the "repository" that one is synchronizing with. So, as well as carton, the library aims to be reusable for other purposes.

This release features support for SSH using awa-ssh by @hannesm (see the release), support for partial-clone (of various depth), and memory consumption fixes for unikernels.

Note 1: The library's name "Not so smart" is a play on the git's "smart" protocol, a part of wire protocol v1 over HTTP(S) transport.

Note 2: only client side logic is implemented for wire protocols. The server-side is planned but not yet implemented. One can use git as the server for now.

Mimic

Mimic is a small reimplementation of conduit, a library that helps to abstract over a transport protocol such as HTTP(S) or SSH. In other words, the code using mimic can deal not with different types that represent an HTTP or SSH connection, but just deal, e.g., read from or write to, with a flow value, which hides protocol-specific details under its hood.

There are several independent libraries that were upgraded along with ocaml-git 3.0.

Duff v0.3

Duff is a library that implements git's libXdiff (xdiff algorithm) in OCaml. PACK files use a binary diff algorithm, xdiff, to compress binary data. More on the project page and release notes for ocaml-git 2.0.

Changes

This release fixes the support for 32-bit architecture platforms.

Encore v0.7

Encore is a library that can create an encoder/decoder based on the format given. It also ensures isomorphism by construction.

Changes

Extensive changes to the API. See the project page.

Decompress v1.2.0

Decompress is an OCaml implementation of certain decompression algorithms such as Zlib, Gzip, etc.

Changes

ocaml-git 3.0 uses new version of decompress with extensive performance improvements documented in Tarides's blog API changes and performance improvements.

We'd be happy to get your feedback or questions! :-)

Other OCaml News

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.