OCaml Weekly News

I’ve started learning OCaml recently and one issue that I struggle with is finding a workflow that gives me quick feedback about changes I’ve made (I’m looking to reduce compile/run cycles). To give you some context - most of the time I’m programming in Lisps and Ruby, where it’s relatively easy to modify a running application. With all Lists my workflow is:

• I make some change (e.g. I update some function)
• I compile the changed bit of code (e.g. a function)
• I immediately run this with the help of the REPL
• Based on the results I got I go to the beginning or move on to the next things that needs doing

That’s known as interactive programming in some circles and I totally love it. With Ruby I can’t do the same, but at least with web apps there’s some degree of code reloading that allows you modify a running app.

With OCaml, however, I’m not sure how to get the quick feedback, as it seems I need to constantly compile and run stuff. I’m trying to work like with Lisps - I keep a toplevel open alongside my source code and occasionally send some code there, but the toplevel is limited in many way (you’re just dumping text there at the global level). `dune utop` helps to some extent, but it can’t reload changes. I’ve heard that some people were saying they didn’t even use a toplevel, so I’m wondering if I’m missing something. I’d be curious to hear any tips and tricks from your OCaml workflows.

Have a look at expectation testing with ppx_expect. That’s what I use to get this kind of quick feedback (and you can keep the results as a test).

Instead of sending code to a REPL, you write your test in an expect block and run dune runtest. Anything you print out in the expect gets captured, and dune shows you the output as a diff.

That said, I’ve never had good luck with REPLs, so I might be missing something about that workflow.

Any test can be exploratory if they’re expect tests, imho :slightly_smiling_face: . The core requirement is to have printers for your types (e.g. ppx_deriving.show), so you can just create values and use

Format.printf "my thing is now %a@." pp_thing my_thing;;

and see the output in dune. There’s a few annoyances (e.g. when it crashes you need a bit of elbow grease to get the backtrace) but for the most part it’s a nice workflow. I don’t apply it to everything, though.

W.r.t to development cycle in OCaml, I find myself working in 3 distinct ways depending on at which point of a project I am at:

• small experiments - sometimes I need a particular function that isn’t provided by the stdlib, or some other bespoke well-defined function. In these cases, I write the function directly in the source code - using merlin’s typing information etc. to develop as usual. Once I’ve done that, just to check the behaviour is as I would expect, I copy over the definition to utop - if the function depends on any larger state from my project, I write a dummy module to mock these parts.
• exploring a stateful system - sometimes, I’m interacting with some kind of stateful existing system that can be hard to run in utop (for example, it might be a pain to get the library working in bytecode) - e.g. gtk. In these cases, I setup a small executable module that performs the minimum required to setup the initial state, and do my iterative experiments by recompiling after each change. Because the module is small, and the OCaml compiler/dune is fast, this leads to a quick iterative editing experience.
• working on a well-known codebase - in all other cases, I’m working on a codebase that I either know fairly well, or has sufficient documentation and typing-discipline to make it easy to understand what’s going on. In these cases, I can rely on the types and documentation to make changes/add new functions, and if needed, use tests to document expected behaviours.

As I’m speaking to an Emacs celebrity @bbatsov (thank you for your great packages!), I’d be remiss if I didn’t mention my own little experiment in this direction - interactive-utop-mode:

https://gitlab.com/-/snippets/2170216

It’s not quite the same level of interactivity as lisp, granted, but with judicious use of forking, you can emulate a slightly more interactive repl experience.

Disclaimer: I don’t actually use this that much myself, it’s just more of a proof of concept to show how it works.

Big +1 to expect tests for interactive programming. I wrote a bit about this in an old blog post:

https://blog.janestreet.com/repeatable-exploratory-programming/

As to how to make things yet better:

On the ppx_expect, this could mean dropping dependencies.

At present, I think ppx_expect itself depends only on a couple other ppx’s, and Base and Stdio, so I think it’s already reasonably light, and the libraries in question are all portable.

From my perspective, a more important direction is to improve the editor integration. Our internal expect test workflow is delightful, and quite a bit better than what’s available publicly. When an error pops up from the build, you hit a key to see the diff, hit another key to accept the diff if you want to. It would be great if we could get something similar to that in vscode.

Indeed, I posted an issue about this here:

https://github.com/ocamllabs/vscode-ocaml-platform/issues/226

Dear (aspiring) MirageOS hacker,

it is my please to announce that there will be the 11th MirageOS retreat in early October in Mirleft, southern Morocco – yes, this time at the seaside. Please find more details, including writeups of earlier retreats at http://retreat.mirage.io

Everyone is welcome, be kind to each other. There won’t be much Internet connectivity – but there’s plenty of beach, discussions, impromptu talks, and a local network mainly constructed by MirageOS unikernels.

If you have questions, don’t hesitate to ask them here in this thread, or contact me directly via eMail “my first name” at mehnert.org.

I have recently compiled our whole codebase with js_of_ocaml, producing a PDF processing library with can be used from JavaScript programs on the server, or within the browser.

I started with no knowledge of JavaScript or its ecosystem, and the js_of_ocaml examples don’t touch on this particular use case, so I thought it would be useful to write up a quick experience report. I’m still pretty naive on the topic, so do take it with a pinch of salt. Corrections welcome.

Perhaps you could make your favourite OCaml library accessible from JavaScript?

Source code: Coherentpdf.js.

There is also a nice document called “Format Unraveled”, if you want to learn more about Format. It can be found here

Hi everyone! I am pleased to announce a brand new package for the OCaml ecosystem: Guile-ocaml

Guile-ocaml is a Free Software library that provides high-level OCaml bindings to the FFI interface for GNU Guile Scheme. The aim of these bindings are to provide an easy way for OCaml developers to extend their OCaml applications with GNU Guile scheme scripting capabilities, providing simple combinators to translate terms and send queries between the two languages.

You can find the documentation here - alongside a fairly comprehensive documentation of all the bindings, it provides a step by step guide on implementing the classical GNU Guile based turtle drawing program, except using OCaml’s graphics module.

Here’s a sneak preview from that guide of what passing an OCaml function to GNU Guile looks like using these bindings:

let move_by n =
  if not @@ Guile.Number.is_integer n then
    failwith "expected numeric arg";
  let n = Guile.Number.int_from_raw n in
  let x, y =
    let cur_pos = Graphics.current_point () in
    move n cur_pos !direction in
  if !pen_down then
    Graphics.lineto x y;
  Graphics.moveto x y;
  Guile.eol

let () = ignore @@ Guile.Functions.register_fun1 "move-by" move_by

Also, I’d like to give extra thanks to opam-repository’s tireless maintainers! The package was in limbo for a while because I couldn’t work out why conf-guile was failing to build on certain distributions, but thankfully @mseri and @kit-ty-kate were able to fix the issue!

Eio provides an effects-based direct-style IO stack for OCaml 5.0. It aims to be easy to use, secure, well documented, and fast. It consists of a generic cross-platform API, plus optimised backends for different platforms. It can be used instead of (or alongside) Lwt and Async.

This morning I spent a bit of time writing down some of my initial observations, experiences and thoughts about OCaml. Hopefully they’ll be useful to other newcomers to the language.

TLDR; I can’t say that my initial experience with OCaml was mind-blowing, but it was definitely pleasant and with time the language grows on me. I miss the simplicity and uniformity of Clojure (still my favorite programming language by far) and Lisps in general or some of Haskell’s goodies (e.g. typeclasses, Hackage and ghci), but I feel OCaml strikes a very good balance between functional programming, pragmatism and performance. It’s a fun language to work with!

I’d be really curious to hear your own thoughts on the subject and to receive feedback about any mistakes I might have made due to not being familiar enough with OCaml and its ecosystem.

Nice blog post!

I’d say the development tooling for OCaml is pretty decent, although I wouldn’t go as far as saying it’s great.

Finally, a shameless plug, if you’re more familiar with lisp development tooling, have you tried gopcaml-mode? It provides paredit style structural editing support for OCaml:

I’ve had a nice use-case of GADTs recently in ocaml-srt that I thought I would share.

On the C side, the library implements an API a la syscall with named options and polymorphic types:

SRT_API       int srt_getsockopt   (SRTSOCKET u, int level /*ignored*/, SRT_SOCKOPT optname, void* optval, int*
optlen);
SRT_API       int srt_setsockopt   (SRTSOCKET u, int level /*ignored*/, SRT_SOCKOPT optname, const void* optval, int
optlen);

And the socket options are listed in an enum:

typedef enum SRT_SOCKOPT {

   SRTO_MSS = 0,             // the Maximum Transfer Unit
   SRTO_SNDSYN = 1,          // if sending is blocking
   SRTO_RCVSYN = 2,          // if receiving is blocking
   SRTO_ISN = 3,             // Initial Sequence Number (valid only after srt_connect or srt_accept-ed sockets)
   SRTO_FC = 4,              // Flight flag size (window size)
   SRTO_SNDBUF = 5,          // maximum buffer in sending queue
   SRTO_RCVBUF = 6,          // UDT receiving buffer size
...

On the OCaml side, it is then pretty natural to also want a polymorphic API;

type 'a socket_opt

val messageapi : bool socket_opt
val payloadsize : int socket_opt
val rcvsyn : bool socket_opt
val sndsyn : bool socket_opt

val getsockflag : socket -> 'a socket_opt -> 'a
val setsockflag : socket -> 'a socket_opt -> 'a -> unit

This can be achieved in a cool type-safe way and without writing any C by combining ctypes and GADTs!

First, we export the socket option enums as OCaml polymorphic variants using the ctypes API:

  type socket_opt =
    [ `Messageapi
    | `Payloadsize
    | `Rcvsyn
    | `Sndsyn]

  let socket_opt : socket_opt typ =
    enum "SRT_SOCKOPT"
      [
        (`Messageapi, constant "SRTO_MESSAGEAPI" int64_t);
        (`Payloadsize, constant "SRTO_PAYLOADSIZE" int64_t);
        (`Rcvsyn, constant "SRTO_RCVSYN" int64_t);
        (`Sndsyn, constant "SRTO_SNDSYN" int64_t)
      ]

Next, on our public OCaml API side, we use regular variant types with a type annotation to use for GADTs pattern matching and match then with the polymorphic variants returned by ctypes:

type _ socket_opt =
  | Messageapi : bool socket_opt
  | Payloadsize : int socket_opt
  | Rcvsyn : bool socket_opt
  | Sndsyn : bool socket_opt

let messageapi = Messageapi
let payloadsize = Payloadsize
let rcvsyn = Rcvsyn
let sndsyn = Sndsyn

let srt_socket_opt_of_socket_opt (type a) : a socket_opt -> Srt.socket_opt =
  function
  | Messageapi -> `Messageapi
  | Payloadsize -> `Payloadsize
  | Rcvsyn -> `Rcvsyn
  | Sndsyn -> `Sndsyn

ctypes also. gives us bindings to the get~/~set functions from the C library:

  let setsockflag =
    foreign "srt_setsockflag"
      (int @-> socket_opt @-> ptr void @-> int @-> returning int)

  let getsockflag =
    foreign "srt_getsockflag"
      (int @-> socket_opt @-> ptr void @-> ptr int @-> returning int)

Now, we’re ready to implement our get~/~set polymorphic functions:

let getsockflag : type a. socket -> a socket_opt -> a =
 fun sock opt ->
  let arg = allocate int 0 in
  let arglen = allocate int (sizeof int) in
  ignore
    (check_err
       (getsockflag sock
          (srt_socket_opt_of_socket_opt opt)
          (to_voidp arg) arglen));
  let to_bool () = !@arg <> 0 in
  let to_int () = !@arg in
  match opt with
    | Rcvsyn -> to_bool ()
    | Sndsyn -> to_bool ()
    | Messageapi -> to_bool ()
    | Payloadsize -> to_int ()

let setsockflag : type a. socket -> a socket_opt -> a -> unit =
 fun sock opt v ->
  let f t v = to_voidp (allocate t v) in
  let of_bool v =
    let v = if v then 1 else 0 in
    (f int v, sizeof int)
  in
  let of_int v = (f int v, sizeof int) in
  let arg, arglen =
    match opt with
      | Rcvsyn -> of_bool v
      | Sndsyn -> of_bool v
      | Messageapi -> of_bool v
      | Payloadsize -> of_int v

⚠️ Question ⚠️

I wasn’t able to join cases of the same type, for instance this:

  match opt with
    | Rcvsyn
    | Sndsyn
    | Messageapi -> to_bool ()
    | Payloadsize -> to_int ()

Is this a theoretical or implementation limitation?

It’s an implementation limitation. There are some details at https://github.com/ocaml/ocaml/issues/5736

Slightly related to the problem of sharing constants between the C side and the OCaml side but much less sophisticated:

For my small epoll library polly I am creating a function in C using a macro that returns the constant:

/* Make all constants available to clients by exporting them via a
 * function. This avoids having to hard code them on the client side.
 * */

#define CONSTANT(name) \
  CAMLprim value caml_polly_ ## name(value unit) { return Val_int(name); }

CONSTANT(EPOLLIN);
CONSTANT(EPOLLPRI);
CONSTANT(EPOLLOUT);

In the bindings I have one function per constant and call it once to obtain the constant; the actual value of the constant is not part of the OCaml code.

  type t = int

  external polly_IN : unit -> t = "caml_polly_EPOLLIN"
  external polly_PRI : unit -> t = "caml_polly_EPOLLPRI"
  external polly_OUT : unit -> t = "caml_polly_EPOLLOUT"

  let inp = polly_IN ()
  let pri = polly_PRI ()
  let out = polly_OUT ()

There have been some significant changes since this last release (Versions 0.4.0 and 0.5.0 are available on opam):

• Buffered reader has a new utility method that allows reading lines
• Shuttle now supports file descriptors that don’t support nonblocking I/O. For blocking I/O Shuttle uses async’s support for running syscalls in its thread pool
• Buffered reader’s api has been simplified to remove read_one_chunk_at_a_time in favor of a more familiar read, read_line etc. refill operation is supported to perform a read syscall to fill a channel’s buffer, and Input_channel.view can be used to get a view into the channel’s underlying buffer.
• Supports v0.15 series of the janestreet libraries
• Buffered reader now uses an auto-growing buffer instead of a fixed size buffer than notified users that the internal buffer is full and no progress can be made unless some content is consumed. This should allow starting with a smaller buffer without needing to worry about implementing some client side buffering to hold unconsumed data. Channels allow configuring an upper bound on the internal buffer length, if a buffer grows beyond that an exception is raised.
• Buffered writer’s support a richer flush interface. Flush operations report errors encountered while attempting to write any pending bytes. This results in a flush operation that returns a deferred that will resolve at some point with either a success or an error, instead of the older flush operation that would return a deferred that never resolved if there was an error during a write syscall.

Does either ML or OCaML have coinductive data types ? And if so could you please point me at the/some appropriate documentation on them.

ML and OCaml have algebraic data types, which are recursive (that is, more general than inductive and co-inductive types). Algebraic data type definitions are not subject to a positivity restriction, and algebraic data types can be constructed and deconstructed by recursive functions, which are not subject to a termination check.

If you want to see a typical example of a “co-inductive” data structure encoded in OCaml, I suggest to have a look at “sequences”, which can be described as potentially infinite lists:

https://v2.ocaml.org/api/Seq.html