OCaml Weekly News

I am looking for an idiomatic implementation of a Lexer + Recursive Descent Parser not making use of ocamllex, ocamlyacc or menhir. The kind you would write during the first chapters of Crafting Interpreters [0][1] in OCaml.

This Markdown parser [2] by @dbuenzli is a great example of what I am looking for. I'd be happy if you can recommend similar resources.

There are many OCaml repos for Lox interpreters but it's hard to assess quality. And the readme often says "I am doing this to learn OCaml", which doesn't inspire confidence.

As a broader note, it would be nice to have (community vetted) OCaml translations of well-known learning material using mainstream languages. But I'll raise the topic in another thread later.

Thanks in advance.

• [0] https://craftinginterpreters.com/scanning.html
• [1] https://craftinginterpreters.com/parsing-expressions.html
• [2] https://github.com/dbuenzli/cmarkit/blob/af8930c307957a546ea833bbdabda94a2fa60b4b/src/cmarkit.ml#L879

You might be interested in the book Compiling to Assembly from Scratch. There is a port to OCaml. It suggests the use of parser combinators.

Parser combinators is the same manual recursive descent method, but in a functional way. You can either use an existing library or you can write your own.

odoc's parser is half of what you're asking for. It uses ocamllex for the lexical scan, because it's very simple and convenient to do it that way, but the syntax is then analyzed by a hand-written recursive descent parser, in large part because that's easier for the doc language.

An example of a non-ocamllex and non-ocamlyacc parser is Markup.ml (tokenizer, parser). But this isn't a traditional recursive descent parser. Rather, it's a pretty huge hand-written state machine in continuation-passing style, almost completely implementing the corresponding huge state machine specified in HTML5. But it's the kind of code that fits well the topics of an impure FP class, especially since it has mutable cells for its continuations, that it uses to mimic effects.

Couldn’t let it rest, so I’m (already) announcing version 2 of it — now a much improved, practically feature-complete reimplementation of both Boulder Dash 1 & 2.

Version 2 was an excuse for me to mess around with the OCaml bindings to popular graphics engines, and as a result, it now comes with 3 backends to choose from:

1. the homely bare OCaml Graphics library (https://github.com/ocaml/graphics),
2. the TSDL binding to the SDL2 API (https://github.com/dbuenzli/tsdl),
3. the binding to the Raylib engine (https://github.com/tjammer/raylib-ocaml).

The list is in order of increasingly better user experience, for the price of a potentially harder build experience. In theory, all versions should run on Windows, Mac, and Linux, though I was too lazy to test all combinations, and I (or my opam) had trouble installing some of the dependencies on some of the systems.

Features:

• Faithful original physics, graphics, animations, sound, and music
• Authentic scrolling mechanics combined with dynamic resizing
• All 40 levels and 5 difficulties of Boulder Dash 1 & 2
• Pause-and-go mode for relaxed playing

Relative to the previous release, version 2 adds the following niceties:

• Support for SDL and Raylib engines, which allow all of the following
• Original sound effects and music
• Original level color schemes
• Full screen mode
• Faster graphics
• Dynamic graphics scaling adjustment
• Gamepad/joystick support as well as more precise keyboard controls
• Boulder Dash 2 levels and decoder

Almost looks like a real game now. One from the 80s anyways. :)

I’m pleased to announce the release of js_of_ocaml 5.9.0. It should soon be available in opam.

Js_of_ocaml is a compiler from OCaml bytecode to JavaScript. It makes it possible to run pure OCaml programs in JavaScript environment like browsers and Node.js.

Most significant changes:

• Support for OCaml 5.3
• Speedup sourcemap generation and improve sourcemap accuracy.
• Prepare the merge of the wasm_of_ocaml fork back into the js_of_ocaml repo.
• JS backtraces are really expansive. They now need to be explicitly requested with OCAMLRUNPARM=b=1. This speeds up programs linked with libraries enabling backtraces programmatically using Printexc.record_backtrace true.

See the Changelog for other changes.

We are stoked to announce the 2.3.0 release of liquidsoap, a general-purpose scripting language written in OCaml with specialized operators to build media streams.

The release is available on github: https://github.com/savonet/liquidsoap/releases/tag/v2.3.0

During this release cycle, we have rewritten huge chunk of the application's internal, including a new media streaming abstraction and clock system.

As an OCaml application, liquidsoap's scope and complexity has greatly expanded in the next years.

Some of the most challenging areas for us at this point are memory usage (and, incidentally, CPU usage).

Although OCaml's garbage collection is a very powerful tool, in the context of very short lived streaming cycles (typically 0.02s) with potentially quite large memory allocations (typically video images), controlling the timing of memory allocations and release is becoming more and more critical.

We are also aware of the work done by Jane St on adding a local call stack. This could be an avenue to explore as well but:

1. Some of our content has to be stored in the long-term heap
2. We want to work with an official OCaml compiler for obvious long-term maintenance concerns.

Nonetheless, we are thrilled to be part of a community whose array of tools (building, packaging, debugging, etc) and libraries has expanded so well along with a vibrant compiler development team.

In the future, we wish to explore more of the new OCaml concurrency features. This might require that we revisit the way we handle short-term memory first.

It's my pleasure to announce the first release of the bytesrw library:

Bytesrw extends the OCaml Bytes module with composable, memory efficient, byte stream readers and writers compatible with effect-based concurrency.

Except for byte slice life-times, these abstractions intentionally separate away ressource management and the specifics of reading and writing bytes.

Bytesrw distributed under the ISC license. It has no dependencies.

Optional support for compressed and hashed bytes depend, at your wish, on the C zlib, libzstd, blake3, libmd, xxhash and libraries.

The only reason I was longing for OCaml algebraic effects was so that I could avoid using them: when you write codecs on byte streams it should not be a concern where your bytes are coming from or headed to. The bytesrw library provides structures to abstract this. Additionally it establishes a buffer ownership discipline that enables byte streams to (de)compose while remaining memory efficient.

I do not expect the library to change much and it has been used. But it's new and practice may call for adjustments. Do not hesitate to get in touch if you run into problems or see obvious defects or improvements. I do expect the library will add more convenience (e.g. for processing lines and UTF) and more optional stream formats over time.

• Homepage: https://erratique.ch/software/bytesrw
• Docs: https://erratique.ch/software/bytesrw/doc or odig doc bytesrw
• Install: opam install bytesrw conf-zlib conf-zstd conf-libblake3 conf-libmd conf-xxhash (opam PR)

This first release was made possible thanks to a grant from the OCaml Software Foundation. I also thank my donors for their support.

[ANN] dream-html 3.7.0

Happy to announce the addition of a helper module for typed form decoding functionality. See the docs here: https://yawaramin.github.io/dream-html/dream-html/Dream_html/Form/index.html

An example:

type user = { name : string; age : int option }

open Dream_html.Form

let user_form =
  let+ name = required string "name"
  and+ age = optional int "age" in
  { name; age }

let dream_form = ["age", "42"; "name", "Bob"]
let user_result = validate user_form dream_form
(* => Ok { name = "Bob"; age = Some 42 } *)

let error_result = validate user_form ["age", "none"]
(* => Error [("age", "error.expected.int"); ("name", "error.required")] *)

Astute readers may observe that this provides some convenience functionality beyond what Dream itself offers; to validate the above form and get a complete set of field validation errors using only Dream you would do something like:

let user_result = match dream_form with
  | ["age", age; "name", name] ->
    (match int_of_string age with
    | age -> Ok { name; age = Some age }
    | exception Failure _ -> Error ["age", "error.expected.int"])
  | ["name", name] -> Ok { name; age = None }
  | ["age", age] ->
    (match int_of_string age with
    | age -> Error ["name", "error.required"]
    | exception Failure _ -> Error ["age", "error.expected.int"; "name", "error.required"])
  | _ -> Error ["name", "error.required"]

And this is a form with only two fields. You can imagine how convoluted the logic would be for more complex forms. Of course, you might just decide to use List.assoc_opt and build up the validation errors, but even that can get tricky. So if you are making heavy use of HTML forms, a helper module that takes care of all these validation details can be very useful. Enjoy!

One month after the release of the first beta for OCaml 5.3.0, we are releasing a second and hopefully last beta release for OCaml 5.3.0 .

The most notable changes for this second beta are probably a handful of type system bugfixes. In particular, those fixes revert a change of behaviour in the first beta when pattern matching GADTs with non-injective type parameters.

We also have a C++ header compatibility fix and the restoration of some configuration variable in Makefiles for the sake of backward compatibility.

Overall, the release is converging and we are expecting to have a first release candidate around the middle of December. The progresses on stabilising the ecosystem are tracked on the opam readiness for 5.3.0 meta-issue.

Meanwhile, the second beta release of OCaml 5.3.0 is here to help you update your software and libraries ahead of the release (see below for the installation instructions).

The full release is expected before the end of December.

If you find any bugs, please report them on OCaml's issue tracker.

If you are interested in full list of features and bug fixes of the new OCaml version, the updated change log for OCaml 5.3.0 is available on GitHub.

Happy hacking, Florian Angeletti for the OCaml team.

I am pleased to announce a new release of Monolith, a library that helps perform strong automated testing of OCaml libraries.

opam update
opam install monolith.20241126

The changes are as follows:

• The documentation of the specification combinators has been re-organized in sections and subsections. Finding the desired combinator should now be easier.
• The new combinator naive_array offers limited support for arrays.
• The new combinator naive_seq offers limited support for sequences (that is, for the type constructor Seq.t).
• The new combinator pair is a synonym for ( *** ).
• The new combinator triple offers support for triples.
• The new combinator either offers support for the type constructor Either.t.
• The new combinators iter, foldr, foldl, iteri, foldri, foldli offer support for iteration functions.
• An unintentional and undocumented limitation has been fixed: so far, uses of the combinator map_into would work only at the root of the specification or in the right-hand side of an arrow ^>. It should now be possible to use map_into under other combinators that expect a deconstructible specification, such as ^!> (in the right-hand side), ( *** ), option, result, list, etc. This improvement affects not only map_into, but also a number of other combinators that are defined in terms of map_into.
• Monolith now requires OCaml 4.12 or later.
• In Makefile.monolith,
  • the default switch is changed from 4.11.1 to 4.14.2; this can be overridden by defining SWITCH;
  • make test automatically disables the MacOS crash reporter;
  • the use of ulimit -s is abandoned.

It's my pleasure to announce the first release of the jsont libary:

Jsont is an OCaml library for declarative JSON data manipulation. It provides:

• Combinators for describing JSON data using the OCaml values of your choice. The descriptions can be used by generic functions to decode, encode, query and update JSON data without having to construct a generic JSON representation.
• A JSON codec with optional text location tracking and layout preservation. The codec is compatible with effect-based concurrency.

The descriptions are independent from the codec and can be used by third-party processors or codecs.

Jsont is distributed under the ISC license. It has no dependencies. The codec is optional and depends on the bytesrw library. The JavaScript support is optional and depends on the brr library.

The library has been used in practice but it's new so a few adjustments may be needed and more convenience combinators added.

The library also enables quite a few things that I did not have the time to explore like schema description generation from descriptions, quasi-streaming JSON transformations, description generation from dynamic type representations, etc. Lots of this can be done outside the core library, do not hesitate to get in touch if you use the library and find interesting applications or pesking limitations.

• Homepage: https://erratique.ch/software/jsont
• Docs: https://erratique.ch/software/jsont/doc (or odig doc jsont)
• Install: opam install jsont bytesrw

This first release was made possible thanks to a grant from the OCaml Software Foundation. I also thank my donors for their support.

Best,

Daniel

P.S. I think that the technique used by the library, which I dubbed finally tagged is interesting in itself. You can read a paper about it here along with a smaller, self-contained, implementation of what the library does.

I like Lwt. It's a fantastic library, but how does it work? I spent a few days studying its source code.

Finally, inspired by the implementation of Lwt and the CS3110 chapter, 8.7. Promises. I wrote a maximally stupid tiny-async-lib library.

Maybe you may be interested in this naive implementation.

Eliom 11.1 has been released recently. This minor release brings compatibility with Web Assembly and the upcoming version of js_of_ocaml. Ocsigen Toolkit and Ocsigen Start have been updated as well.

Stay tuned for further announcements concerning client-server Web and mobile apps in Ocaml with Web Assembly!

Links:

• Ocsigen
• Eliom
• Documentation
• Github

I figured people might be bored of British pub names by now so I did another thing: a generator for titles of table-top role-playing games.

$ opam install areas-and-adversaries
...
$ areas-and-adversaries
Woods & Wizards

The code is on Gitlab: https://gitlab.com/raphael-proust/areas-and-adversaries

It was a good excuse to experiment with non-dune build systems (to scope things out). I went for a plain Makefile in the end which works well.

I also wanted to figure out a better way to embed data in an executable. I ended up wondering about moving as much of the processing as possible into the build phase. What I ended up with is a small program which prints a compilation unit (.ml) which has mostly array literals. Still have some open questions on that, any input welcome:

• Should I have used meta-ocaml to print the code? The data/munch.ml would probably be more readable, but the build probably less.
• How could I generate this kind of processed-data code for data-structures which don't have a literal (maps, sets, hash tables, etc.)? How can I minimise the initialisation cost of the program for such situations?

I'm please to announce a new release of the mariadb package, which provides client bindings for MariaDB and MySQL. See the full release notes below.

This is also to announce that I have taken over maintenance of the project. Currently I am the sole maintainer (and I usually use PostgreSQL for my own deployments), so if someone has the time en interest to contribute, let me know. The main focus from my side is to keep the project up to date and stable, rather than making major changes.

Release notes:

• Added Stmt.start_txn (#59 by Corentin Leruth).
• Added Res.insert_id as binding for mysql_stmt_insert_id (#58 by Corentin Leruth).
• Updated to support recent OCaml versions (#45 by @kit-ty-kate).
• Fixed too-early retrieval of statement metadata (#41 by Albert Peschar).
• Fixed decoding bug for the integer type (#54 by Raman Varabets, tested by #61 by Corentin Leruth).
• Fixed a memory leaks related to result metadata (#39 by Albert Peschar).
• The build system is now dune and dune-configurator (#52 by Petter A. Urkedal) and some of the examples have been converted to a test suite (#60 by Petter A. Urkedal).
• The project has been transferred to ocaml-community with Petter A. Urkedal as the new maintainer.

It is my please to announce the proposed package archiving policy in the name of the opam-repository maintainers.

I'm pleased to announce the release of capnp-rpc 2.0, an OCaml implementation of the Cap'n Proto RPC specification.

If you haven't used the library before, please see the documentation and tutorial. Cap'n Proto RPC aims to provide secure, efficient, typed communications between multiple parties.

The main change in this version is the switch from Lwt to Eio for concurrency. The echo benchmark is about 40% faster than before. This isn't because Eio is actually that much faster than Lwt, but more because it has better profiling support so spotting problems was easier. See https://roscidus.com/blog/blog/2024/07/22/performance/ for an example:

There is a capnp-rpc-lwt compatibility package that provides the old Lwt API using the new Eio version, allowing libraries using the old API to be used in applications using the new code, without having to update everything at once.

To migrate to the new version (checking everything still works after each step):

1. First, update to capnp-rpc 1.2.4 (this ensures you are using the newer mirage-crypto API, to get that migration out of the way first).
2. Switch your application (that sets up the networking) to capnp-rpc-unix 2.0.
3. Migrate client and server code away from capnp-rpc-lwt when convenient.

For more detailed instructions, see the changelog.

Here's an example of the changes needed in the solver-service project:

• Update to capnp-rpc-unix 2.0
• Remove Capnp_rpc_lwt

Here are links from many OCaml blogs aggregated at the ocaml.org blog.

• Irmin on MirageOS: Introducing the Notafs File System