OCaml Weekly News

The release of OCaml 4.12.0 is approaching. We have released a second alpha version to help fellow hackers join us early in our bug hunting and opam ecosystem fixing fun.

Beyond the usual bug fixes this new alpha version removes the type system change that restricted the propagation of type information between branches of a "match". The newly introduced warning was more troublesome than expected, the feature has been thus postponed to 4.13 .

The base compiler can be installed as an opam switch with the following commands

opam update
opam switch create 4.12.0~alpha2
--repositories=default,beta=git+https://github.com/ocaml/ocaml-beta-repository.git

If you want to tweak the configuration of the compiler, you can pick configuration options with

opam update
opam switch create <switch_name> --packages=ocaml-variants.4.12.0~alpha2+options,<option_list>
--repositories=default,beta=git+https://github.com/ocaml/ocaml-beta-repository.git

where <option_list> is a comma separated list of ocaml-option-* packages. For instance, for a flambda and afl enabled switch:

opam switch create 4.12.0~alpha2+flambda+afl
--packages=ocaml-variants.4.12.0~alpha2+options,ocaml-option-flambda,ocaml-option-afl
--repositories=default,beta=git+https://github.com/ocaml/ocaml-beta-repository.git

All available options can be listed with "opam search ocaml-option".

The source code for the alpha is also available at these addresses:

• https://github.com/ocaml/ocaml/archive/4.12.0-alpha2.tar.gz
• https://caml.inria.fr/pub/distrib/ocaml-4.12/ocaml-4.12.0~alpha2.tar.gz

If you want to test this version, it is advised to install the alpha opam repository

https://github.com/kit-ty-kate/opam-alpha-repository

with

opam repo add alpha git://github.com/kit-ty-kate/opam-alpha-repository.git

This alpha repository contains various packages patched with fixes in the process of being upstreamed. Once the repository installed, these patched packages will take precedence over the non-patched version.

If you find any bugs, please report them here: https://github.com/ocaml/ocaml/issues

I am please to announce the new releases of two opam packages: ez_subst and ez_cmdliner. We use both of them as dependencies of drom (and use drom to manage them).

• ez_subst is a simple library to perform string replacements in strings. It can be seen as a replacement for Printf when you are lost with too many %s in one format, or a replacement for Buffer.add_substitute when you want a more control. Replacements are chosen by functions, and can be separately specified using optional arguments `brace (for ${var}), `paren (for $(var)), `bracket (for $[var]) and `var (for $alphanum). Separator $ can be changed, and notation can be symmetric (%{x}%).

  https://ocamlpro.github.io/ez_subst
  https://ocamlpro.github.io/ez_subst/doc/ez_subst/Ez_subst/V1/EZ_SUBST/index.html

  For example:

  open Ez_subst.V1 (* versionned interface *)
  
  let s = EZ_SUBST.string ~brace:(fun ctxt n -> string_of_int
     (ctxt + int_of_string n)) ~ctxt:3 "${4} ${5}"
  
  let s = EZ_SUBST.string ~sep:'!'  ~paren:(fun () s ->
     String.uppercase s) ~ctxt:() "!(abc) !(def)"
  
  let s = EZ_SUBST.string ~sym:true ~sep:'%' ~brace:(fun ctxt_ s
     -> ctxt ^ " " ^ s) ~ctxt:"Hello" "%{John}% %{Sandy}%"
  
  let s = EZ_SUBST.string_from_list ~default:"unknown" [ "name",
     "Doe"; "surname", "John" ] "${name} $(surname) is missing"
  

• ez_cmdliner is a simple layer over cmdliner to provide an interface à la Arg module. It provides support for a one-command and sub-commands modes. It also provides a ReST generator to document sub-commands and integrate the documentation in a Sphinx documentation (to use with drom for example).

  https://ocamlpro.github.io/ez_cmdliner

  For example:

  open Ezcmd.V2
  let cmd_new =  EZCMD.sub "new"   (* for `drom new` *)
   ~args: [
    [ "dir" ],  Arg.String (fun s -> dir := Some s),
       EZCMD.info ~docv:"DIRECTORY"
        "Dir where package sources are stored (src by default)";
    [ "library" ],  Arg.Unit (fun () -> skeleton := Some "library"),
       EZCMD.info "Project contains only a library";
    [ "i"; "inplace" ], Arg.Set inplace, (* for `-i` or `--inplace` *)
        EZCMD.info "Create project in the the current directory";
    [],    Arg.Anon (0, fun name -> project_name := Some name),
        EZCMD.info ~docv:"PROJECT" "Name of the project"
   ]
   ~doc:"Create a new project"
   (fun () ->
      action ~name:!project_name ~skeleton:!skeleton ~dir:!dir
        ~inplace:!inplace ~args)
   ~man: [
     `S "DESCRIPTION";
     `Blocks [
       `P "This command performs the following actions:";
     ]
  ]
  let () = EZCMD.main_with_subcommands ~name:"drom" ~version:"0.1.0"
     ~doc:"Create and manage an OCaml project" ~man:[] ~argv [ cmd_new ]
     ~common_args
  

Both packages are now available in opam repository.

I would like to announce a new release of Menhir, the LR(1) parser generator for OCaml. The most prominent new features are intended to improve the comfort of the machinery that allows producing custom syntax error messages: a demo of this machinery has been added, new library functions have been added so as to make it easier to use, and the commands that deal with .messages files have been improved. An excerpt of the changelog appears below.

opam update
opam upgrade menhir

Happy parsing!

It is my pleasure to announce the release of http-multipart-formdata v1.0.0. As the name suggests, the library implements functionality to allow HTTP file uploads and form processing. Tangentially, it implements the standard RFC 7578 - Returning Values from Forms: multipart/form-data which is the standard browsers use to send form data to a web server.

I developed this library as part of my endeavour to create ocaml web applications.

It is also an example of the parser construction library reparse which I also released a few days ago.

• http-multipart-formdata
• API docs

Welcome to the November 2020 Multicore OCaml report! This update along with the previous updates have been compiled by @shakthimaan, @kayceesrk, and @avsm.

Multicore OCaml: Since the support for systhreads has been merged last month, many more ecosystem packages compile. We have been doing bulk builds (using a specialised opam-health-check instance) against the opam repository in order to chase down the last of the lingering build bugs. Most of the breakage is around packages using C stubs related to the garbage collector, although we did find a few actual multicore bugs (related to the thread machinery when using dynlink). The details are under "ecosystem" below. We also spent a lot of time on optimising the stack discipline in the multicore compiler, as part of writing a draft paper on the effect system (more details on that later).

Upstream OCaml: The 4.12.0alpha2 release is now out, featuring the dynamic naked pointer checker to help make your code only used external pointers that are boxed. Please do run your codebase on it to help prepare. For OCaml 4.13 (currently the trunk) branch, we had a full OCaml developers meeting where we decided on the worklist for what we're going to submit upstream. The major effort is on GC safe points and not caching the minor heap pointer, after which the runtime domains support has all the necessary prerequisites upstream. Both of those PRs are highly performance sensitive, so there is a lot of poring over graphs going on (notwithstanding the irrepressible @stedolan offering a massive driveby optimisation).

Sandmark Benchmarking: The lockfree and Graph500 benchmarks have been added and updated to Sandmark respectively, and we continue to work on the tooling aspects. Benchmarking tests are also being done on AMD, ARM and PowerPC hardware to study the performance of the compiler. With reference to stock OCaml, the safepoints PR has now landed for review.

As with previous updates, the Multicore OCaml tasks are listed first, which are then followed by the progress on the Sandmark benchmarking test suite. Finally, the upstream OCaml related work is mentioned for your reference.

But then what’s the point of Seq ?

A bit of a spoiler for an upcoming release of a few of our libraries at Nomadic Labs…

We had a bug report: calls to some RPCs exposed by some of our binaries would occasionally cause some lag. One of the root causes of the issue was JSON serialisation. The original serialisation scheme was intended for a limited range of uses (especially, small sizes) but then it was used outside of this intended range and some relatively big values were serialised and pushed down the RPC stack.

To circumvent this, we are about to release

• a “json lexeme sequence” backend for our serialiser library: construct_seq : 'a encoding -> 'a -> json_lexeme Seq.t where json_lexeme = Jsonm.lexeme = [ `Null | `Bool of bool | … | `As | `Ae | `Os | `Oe ]
• a json lexeme sequence to string sequence converter.

For this second part, we actually have three different converters intended for slightly different uses. They have different granularity, they have different allocation profiles, and they make slightly different assumption most notably about concurrency:

• string_seq_of_json_lexeme_seq : chunk_size_hint:int -> json_lexeme Seq.t -> string Seq.t which uses one (1) internal buffer of size chunk_size_hint. Consuming one element of the resulting sequence causes several json lexemes to be consumed and printed onto the internal buffer until it is full. When this happens, a snapshot (copy) of the buffer is delivered in the Cons cell. So for chunk-size-hint of, say, 1Ko, the sequence translator uses roughly 1Ko of memory and emits 1Ko chunks of memory that the consumer is responsible for.
• small_string_seq_of_json_lexeme_seq : json_lexeme Seq.t -> string Seq.t which translates each of the lexeme as a single string. It's a little bit more than a simple Seq.map because it needs to insert separators and escape strings. It mostly returns statically allocated strings so there are no big allocations at all.
• blit_instructions_seq_of_jsonm_lexeme_seq : buffer: bytes -> json_lexeme Seq.t -> (bytes * int * int) Seq.t which works somewhat similarly to the first one but uses buffer instead of allocating its own. And it returns a seq of (source, offset, length) which are intended to be blitted onto whatever the consumer wants to propagates the data too. This barely allocates at all (it currently does allocate relatively big chunks when escaping strings, but we have planned to improve this in the future. (The sequence returns a source to blit; this source is physically equal to buffer most of the time but not always; specifically, for large strings that are present within the json data, the sequence just points to them as a source.)

Note that the description above is a simplification: there is a bit more to it than that. Also note that all this is still Work In Progress. Check out https://gitlab.com/nomadic-labs/json-data-encoding/-/merge_requests/5 (the value to json lexeme sequence code) and https://gitlab.com/nomadic-labs/data-encoding/-/merge_requests/19 (the json lexeme sequence to string sequence code).

This is the debug adapter protocol library extract from ocamlearlybird. Include types generated from specification and a DAP prioritized JSON RPC implementation. It's useful to implement debug adapter in OCaml.

Debug adapter protocol

• Project page: https://github.com/hackwaly/ocaml-dap
• Documentation: https://hackwaly.github.io/ocaml-dap/

when doing static web sites, feedback is an issue. form2xml helps you keep the server stupid, but still makes form-data feedback possible.

Just dump the form posts, rsync and merge them into your client-side, unix pipe, toolchain.

form2xml bridges the tooling-gap between http and xml/xslt with utmost primitivity in the making. I chose simplicity over compliance because form2xml isn't intended to run server-side or unattended. I'm aware of excellent prior art, but hesitated to add build dependencies for now and rather see if it proves useful as is.

https://mro.name/form2xml

I have just released a maintenance release of http-multipart-formadata. This is a maintenace release to address a reported issue.

#10 Fix equality

We have a changelog since this release.

By the way, I'm preparing to publish v2 of setup-ocaml. It has a cache feature that is entirely independent of GitHub, so you don't have to worry about cache limit per repository, and you don't have to spend nearly 10 minutes on setup.

https://github.com/avsm/setup-ocaml/blob/master/CHANGELOG.md

https://github.com/avsm/setup-ocaml/releases/tag/v1.1.4

We are happy to announce the first public release of Memthol, a visualizer and analyzer for memory profiling data generated from OCaml programs, thanks to the work of Adrien Champion and Vincent Laviron.

Memthol is a visualizer and analyzer for program profiling. It works on memory dumps containing information about the size and (de)allocation date of part of the allocations performed by some execution of a program. For information regarding building memthol, features, browser compatibility… refer to the memthol github repository. Please note that Memthol, as a side project, is a work in progress that remains in beta status for now.

Last week, @BikalGurung blogged On Effectiveness of Exceptions in OCaml, in part as a follow-up to his announcement of his parser combinator library reparse, which eschews Result-based error handling in favor of exceptions. I've long preferred using Result (and its equivalents in other languages), and my experience so far with OCaml is that that preference is shared by many in the community and by authors of key libraries, but I was happy to consider a new counterpoint.

Doing so prompted me to consider my rationale(s) more than I had previously, and do some additional reading and research, all of which ended up further cementing my pro-Result bias. What follows are counterpoints to Bikal's two most consequential arguments (in my opinion), and some elaboration beyond. Many thanks to Bikal for his posting his experience report!

Just chiming in to note that there has been an interesting discussion on this topic two years ago: https://discuss.ocaml.org/t/specific-reason-for-not-embracing-the-use-of-exceptions-for-error-propagation/1666/40

It's also interesting to note that that discussion also ended up talking about typed effects. As I understand it, they would indeed subsume checked exceptions, and I'm quite excited about them.

Cristiano Calcagno has been doing some pretty interesting work on this: https://github.com/reason-association/reanalyze/blob/72712393459d7e132c78e0700abffc5fc4cd09b8/EXCEPTION.md

Let me quote the central concept from there:

The exception analysis is designed to keep track statically of the exceptions that might be raised at runtime. It works by issuing warnings and recognizing annotations. Warnings are issued whenever an exception is raised and not immediately caught. Annotations are used to push warnings from he local point where the exception is raised, to the outside context: callers of the current function. Nested functions need to be annotated separately.

I'm going to address the general issue of "programming with monads", and not specifically the result monad, b/c I think it's just an instance of the general phenomenon.

TL;DR In 1992, when someone told me about "programming with monads", I replied that I already programmed with monads: I used the "SML Monad". And this LtU post seems to me to be pithily succinct (http://lambda-the-ultimate.org/node/5504 )

(1) when we talk about program correctness, we mean two things: reasoning about programs, and type-safety. I'll address each in turn below.

(2) All monadic transformations of which I am aware (exceptions, state, control, I/O) are direct equivalents to the "standard semantics" for such language-features, e.g. as described in Michael J.C. Gordon's book The Denotational Description of Programming Languages. Programming with monads is programming with some combinators and macros, on the right-hand-side of the denotational interpreters in that book.

(3) "reasoning about programs" has historically meant "equational reasoning", and IIUC, Felleisen&Sabry's work (and follow-on works) proved pretty conclusively that anything you can prove about the right-hand-side of the denotational semantics interpeter defiition, you can "pull back" to equational reasoning with extra rules, on the left-hand-side of the DS interpreter.

(4) "type safety":

If only we had a way to know statically which exceptions can escape any functions

There was a cottage industry of "effect type systems" to capture/reason-about exceptions, state, maybe other things, decades ago. They were judged too cumbersome for programmers to use, and hence died-out. >10yr ago there was a caml-light (OCaml?) variant that checked exceptions in function-types; it didn't catch on. Look at Java, where some exceptions are "checked" and others are not: some exceptions, it's just too cumbersome to track in the type system. And so either your "result" monad only captures some of the exceptions, or it's going to be wildly cumbersome.

(5) Monads are less-efficient than direct-style, memory-wise. For me, the moment in 1992 when I (an avowed SML/NJ bigot) became convinced of the superiority of caml-light (notwithstanding 2.5x slower on average) was when I realized that it was -so- much less memory-intensive. Because it didn't allocate stack-frames on the heap, and closures started out on the stack and only moved to the heap on-demand. Henry Baker made the observation >20yr ago that the stack is a form of nursery. Writing in monadic style is sacrificing this obviously performance advantage. In the era of multicore, arguments made back then about memory, can be recast as arguments about the cache today, since (as hardware designers put it) "memory is at infinity" today.

P.S. And yet, I use monads sometimes, too. Rarely. But for instance, it's a good model for (e.g.) writing a type-checker that wants to type-check a list of expressions (no unification, hence no side-effects) and not stop at the firs type-error, but rather gather together errors from all the expressions in the list, and produce an error with all of them combined. So the type-checker at the top of each member of the list catches any raised exception, stores it in an accumulator, and goes on to the rest of the list; at the end of the list, if the accumulator is empty, it returns the list of result-types; otherwise it raises an exception containing list of errors stored in the accumulator.

It's rare, and if the Result monad didn't exist, I'd hack something together, but …. it's literally the only use I can think of, that wasn't driven by a library (e.g. bos) using the Result monad itself (and my needing to use that library).

And this efficiency is the real reason that exception-based backtraces are better: IIUC, OCaml exceptions are really cheap because they don't materialize that backtrace until demanded. It means that you have to be careful what code you put between the "try-with" and the demand for the backtrace, but it's efficient. Materializing the backtrace for every exception raised would be ….. pretty horrendously inefficient, and yet that's what you have to do if you use the result monad.

I wrote two blog posts on my experience using result awhile ago, linked below. Much of it still holds. Many of the pain points others have mentioned do exist, but in my judgement, given the current state of Ocaml, results are strictly better (at the very least at the API boundary, assuming you can convince yourself no exceptions escape it) than exceptions. I also believe that the reasonable error values are necessary. For example, I know some APIs like some variation of ('a, string) result which, IMO, is not a great API as I end up comparing strings and hoping the string value is actually part of the API and not some rando value tossed in there. Double for when meaningful aspects of the error are encoded in the string and I have to decode it to decide what to do.

For my own things I do require that all errors are convertible to a string so I can just show them to the user, this is especially important for development and debugging, IME. This is one of the few places where I do wish we had something like type classes so I could do something like:

foo ()
>>= function
| Ok () -> yadda
| Error err -> show err

YMMV

http://functional-orbitz.blogspot.com/2013/01/experiences-using-resultt-vs-exceptions.html

http://functional-orbitz.blogspot.com/2013/01/introduction-to-resultt-vs-exceptions.html

Hi, I am new to OCaml and in fact, I'm not a even a programmer (although I did study CS at Cornell back in the 80's and learned functional programming in a language called scheme.) I am thinking about developing financial wellness web applications with the underlying computations in OCaml, but with the user interface in something else - like java script. How would a java script website make a call to an Ocaml program (or function)? Or put another way, can I publish a financial model built in OCaml so that a web (or mobile) application could call it - passing arguments to the function and receiving back the result of evaluating the functional expression? My apologies if this is not asked correctly or if it is a very basic question, but I am not sure I have the right terminology to ask the question properly! Help is appreciated!

You can, there are multiple ways to achieve what you want. If I understand correctly you want to build a web server in OCaml and a web app front end. Here's a list of what different projects used: https://discuss.ocaml.org/t/your-production-web-stack-in-2020/6691/11

Edit: To elaborate because I realized I didn't answer your question, typically you'd have a frontend which uses something like rest or graphql to fetch data from your server. There's a lot to unpack here. I'm sure you'll be able to pull it off but if you're not comfortable with programming make sure that you approach the problem gradually and make sure to avoid analysis paralysis.

Hi, a couple of thoughts here. As @wokalski said, you will need to set up a server application, and a web frontend. I don't know much about your background but, my guess is you would like to avoid complexity and keep things simple. Personally here's what I would recommend:

1. Write a simple command-line application, in the style of a Unix filter, in OCaml that takes 'requests' in the form of plain text on standard input and prints its calculation result to standard output. E.g., to take an input of add 2 2 and output 4, it could work like this:

   $ echo 'add 2 2' | my_calculator.exe
   4
   

2. Next, use websocketd to wrap your calculator tool and serve it over WebSocket, which is a standard Web technology that allows clients to continuously talk to servers (2-way communication). So, clients could send a plain text command like add 2 2 (note, exactly the same as you would have on the command line), and get back a response 4.
3. Finally, write a web application (just some HTML and JavaScript) that connects to the WebSocket server from step (2) and sends and receives messages. Here is an example of that: https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_client_applications

The reason I am recommending this strategy, is to let you start small and simple, and skip over much of the complexity of dealing with modern web application development. You can focus on writing your calculator as a simple command-line tool, and 'outsource' the server part to a specialized tool.

Final thought: if you are working on a financial wellness tool, you almost certainly need decimal arithmetic (as opposed to binary arithmetic from OCaml's built-in float type). You will want to use a decimal package like https://github.com/janestreet/bigdecimal , or (disclaimer: mine) https://opam.ocaml.org/packages/decimal/ .

Good luck!

@peter I think I'm doing something similar – a simple web tool for geographic calculations from character sequences called geohash to gps coordinate pairs and vice versa. Here is it: https://demo.mro.name/geohash.cgi/u154. You'll find the source there, too.

Key is, I scale towards n=1, need no state.

The backend is <200 LOC to handle all the http stuff (there isn't much, no auth, no state, no cookies) and another ~100 LOC for the actual computation.

1 dependency, no 'modern' web toolkit, no client libs/frameworks, no concurrency

We just released a first version of a package providing OCaml bindings to the wasmtime WebAssembly runtime. The package is available on opam and can be found in this GitHub repo. It can be used to run .wasm modules in an OCaml process, including modules making system calls through WASI. For now, the package only provides a low-level api closely matching the Rust implementation. We intend to provide a higher level api on top of this. The GitHub repo contains various examples in the tests directory which reproduce some examples from the main wasmtime repo. Feedback/issue reports are very welcome!

On behalf of Nomadic Labs, I'm happy to announce the first release of Lwt-exit, a small opinionated library to cleanly handle exits and signals in applications that use Lwt.

The library is available through opam: opam install lwt-exit, hosted on gitlab: https://gitlab.com/nomadic-labs/lwt-exit, distributed under the MIT license: https://gitlab.com/nomadic-labs/lwt-exit/-/blob/master/LICENSE and the documentation is available online: https://nomadic-labs.gitlab.io/lwt-exit/

This library is used in the Tezos codebase to clean up system resources (flush buffered writes, cleanly close p2p connections, etc.) during exits. It is also used to attach signal handlers (both for interactive use via Ctrl+C and for daemonisation via systemctl).