OCaml Weekly News
Hello
Here is the latest OCaml Weekly News, for the week of January 05 to 12, 2021.
Table of Contents
- Marshal determinism and stability
- Sedlex + Menhir parser for both tty and file parsing
- First release of awa-ssh
- Introducing Feather: A lightweight shell-scripting library for OCaml
- postdoc researcher and research engineer positions for CHERI and Arm verification
- First ocb (OCaml Badgen) release
- Release of OCaml-Git v3.0 and co
- Other OCaml News
- Old CWN
Marshal determinism and stability
Continuing this thread, David Allsopp said
A couple of notes on Marshal
, which I don't think have been covered
- Although the guarantee is only between identical versions of OCaml, the implementation actually goes to considerable lengths to maintain backwards compatibility (so a value written by older OCaml remains readable in newer OCaml). Our own testsuite, for example, indirectly includes a test which unmarshals a 3.12.1 value. I don't know exactly how far back the support goes.
- As it happens, the change which affected Unison in 4.08 was the first breaking change to Marshal since either 4.00 or 4.01. The fact that it doesn't break often (and that the two code paths - at least at present - are small) meant I have suggested a few months back that we could in future add an additional flag in the style of
Compat_32
to allow values to be written in a way which should be readable on older versions of OCaml. Indeed, it's small enough that flags could be added for the changes in 4.08 (PR#1683) and in 4.11 (PR#8791). - Neither point undermines using alternative formats either for network serialisation or persistent storage, for the many reasons discussed above!
Sedlex + Menhir parser for both tty and file parsing
Bernard Sufrin announced
I am a great fan of Menhir, and have used it in several private language projects, using the ulexing scanner generator to provide Unicode-capable scanners.
Alarmed by the obsolescence of ulexing, and needing a utf8-capable scanner in a hurry I decided to (teach myself to) use Sedlex. On the whole the experience was very satisfactory, and I found it straightforward to produce a variant of the sedlexing library which supports buffers with variable chunk sizes, thereby enabling efficient lexing on channels connected to files as well as immediately responsive lexing on channels connected to terminals.
I also wanted to teach myself how to use the error-reporting, incremental, interfaces to Menhir-generated parsers. In the hope that it might be useful to others facing the same learning task, or the problem of adapting Sedlex for efficient interactive use, I have placed the example mock-S-Expression parser that resulted from this excursion in:
Git Repository: github.com/sufrin/InteractiveSedlexMenhirExample
First release of awa-ssh
Hannes Mehnert announced
I'm happy to announce that awa-ssh
(https://github.com/mirage/awa-ssh) has just been merged into
opam-repository. It is a pure OCaml implementation of the ssh (secure shell,
https://en.wikipedia.org/wiki/SSH_(Secure_Shell)) protocol.
This is the initial release, please report issues you encounter.
It was initially developed by Christiano Haesbaert in 2016, and revived mid-2019 by myself and in 2020 it was migrated to the MirageOS organization on GitHub for further development and maintenance.
Both client and server code are present in the library (pure code in the main awa package), though the awa-lwt package implements only a server, and the awa-mirage package implements only a client. Tests and examples are in the test subdirectory.
The MirageOS client has been successfully used to clone git repositories (on private servers, on GitHub, etc.). It supports apart from RSA keys also ED25519 keys (and key exchanges).
Introducing Feather: A lightweight shell-scripting library for OCaml
Charles announced
I wrote a shell scripting library called Feather. I like idea of writing bash-like code quickly, later being able to intersperse OCaml to add more typeful components as needed. It's kind of like Shexp but without the monadic interface and with Async support. (Feather_async)
There's a tutorial and some examples in the link above but here's a quick taste:
open Feather let lines = find "." ~name:"*.ml" |. tr "/" "\n" |. map_lines ~f:String.capitalize |. sort |. process "uniq" [ "-c" ] |. process "sort" [ "-n" ] |. tail 4 |> collect_lines in String.concat ~sep:", " lines |> print_endline
Let me know if you have any feedback! And feel free to file bug reports here. Hope it ends up being useful, entertaining, or both!
postdoc researcher and research engineer positions for CHERI and Arm verification
Peter Sewell announced
We are looking for postdoctoral researchers and postdoctoral or postgraduate research engineers to help develop semantics and verification to improve the foundations and security of mainstream computer systems, for CHERI and Arm system software verification, at the University of Cambridge. OCaml expertise to help develop verification tools will be especially welcome. Closing date 13 January 2021 - see the advert http://www.jobs.cam.ac.uk/job/28012/.
First ocb (OCaml Badgen) release
zapashcanon announced
A few days ago, I released ocb. It's a library and a command-line tool to generate SVG badges.
To get started quickly:
ocb --label Hello --color green --style flat --labelcolor white --status Goodbye
Will gives this result: SVG example.
My first use case was To.ml where I'm using bisect_ppx to generate and deploy a coverage report. I wanted to display the coverage percentage in the README and tried existing tools but wasn't fully satisfied as they didn't work or were failing randomly. Now, I'm generating the badge directly in a GitHub action.
The project was inspired by badgen. I still have to add support for icons and to improve the documentation but it's usable.
Release of OCaml-Git v3.0 and co
Ulugbek Abdullaev announced
We, the ocaml-git
team, are happy to announce a new major
release of ocaml-git v3.0
and related libraries.
Release Notes
OCaml-Git v3.0
OCaml-Git is a library that implements git
format and
protocol implementation in pure OCaml. The library is used by libraries such as
irmin
, a git-like distributed database, or
pasteur
, a MirageOS unikernel-based snippet storage service.
Changes
The main goal behind this major release was to get better compatibility with various platforms,
including [~MirageOS~](mirage.io), 32-bit platforms, and js_of_ocaml
. In order to achieve that, we
broke down ocaml-git
into several components, which are represented as sub-libraries. We will
describe some of those components later in this post.
Along with better support for various platforms, ocaml-git 3.0
also comes with SSH support for
fetch/push
and various bug fixes.
The rest of the changes are mostly internal and pave a way for interesting features such as a
full-blown git
garbage collector and wire protocol v2
(announcment and
spec).
References:
- Full changes list
- PR that introduced the major rewrite of
ocaml-git
—
In the new version of ocaml-git
, we try to have better separation of concerns by breaking some of the
ocaml-git
components into sub-libraries, which do not contain git
-specific logic and can be reused
for other purposes.
Carton
Git uses PACK files
to store old git objects such as commits and transfer objects over wire using git's wire protocols
(git-nss
library mentioned below implements
v1 of the protocol;
v2 implementation is
in progress).
Carton is a library to work with PACK
files. The library does not contain git-specific code, so one can easily reuse the library and PACK
format for non-git objects. One can see how ocaml-git
uses carton
for its purposes
here.
References:
- PR that introduces
carton
Git-NSS (Not So Smart)
When one wants to synchronize with a remote repository using git, they need to use git fetch/push
.
Communication and synchronization/negotiation is defined by git wire protocol, which has two
versions: older version 1 and newer leaner version 2. The protocols are defined for four wire
transports: HTTP(S), SSH, and git://
(TCP).
Not-So-Smart
library is a library
that allows for such synchronization based on the git wire protocols but without git-specific code,
meaning that files being fetched do not need to be git objects or that there is no assumptions on the
"repository" that one is synchronizing with. So, as well as carton
, the library aims to be reusable
for other purposes.
This release features support for SSH using awa-ssh by @hannesm
(see the release), support for
partial-clone (of various depth
), and memory consumption
fixes for unikernels.
Note 1: The library's name "Not so smart" is a play on the git's "smart" protocol, a part of wire protocol v1 over HTTP(S) transport.
Note 2: only client side logic is implemented for wire protocols. The server-side is planned but not
yet implemented. One can use git
as the server for now.
Mimic
Mimic is a small reimplementation of
conduit
, a library that helps to abstract over a transport
protocol such as HTTP(S) or SSH. In other words, the code using mimic
can deal not with different
types that represent an HTTP or SSH connection, but just deal, e.g., read from or write to, with a
flow
value, which hides protocol-specific details under its hood.
—
There are several independent libraries that were upgraded along with ocaml-git 3.0
.
Duff v0.3
Duff is a library that implements git's
libXdiff
(xdiff
algorithm) in OCaml. PACK files use a
binary diff algorithm, xdiff
, to compress binary data. More on the project
page and release
notes for ocaml-git 2.0
.
Changes
This release fixes the support for 32-bit architecture platforms.
Encore v0.7
Encore is a library that can create an encoder/decoder based on the format given. It also ensures isomorphism by construction.
Changes
Extensive changes to the API. See the project page.
Decompress v1.2.0
Decompress is an OCaml implementation of certain
decompression algorithms such as Zlib
, Gzip
, etc.
Changes
ocaml-git 3.0
uses new version of decompress
with extensive performance improvements documented in
Tarides's blog API changes
and performance
improvements.
We'd be happy to get your feedback or questions! :-)
Other OCaml News
From the ocamlcore planet blog
Here are links from many OCaml blogs aggregated at OCaml Planet.
Old CWN
If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.
If you also wish to receive it every week by mail, you may subscribe online.