OCaml Weekly News

First of all, let's define what is programming in large and what is in small. The truth is that there is no definite answer, you have to develop some taste to understand where you should just stick with plain ADT (PADT) or where to unpack the heavy artillery of GADT and final tagless styles. There is, however, a rule of thumb that I have developed and which might be useful to you as well.

So you're designing a large project, possibly with API, that will help lots of developers with different backgrounds and skills. And as always it should be delivered next week, well at least the minimal viable product. The upper management is stressing you but you still don't want to mess things up and be cursed by the generations of software developers that will use your product.

One of the killer features of OCaml, and the main reason why I have switched to it a long time ago and use it since then, is that it ideally fits the above-described use case. It fully supports the programming in large style (like Java and Ada) but enables at the same easy prototyping and delivering MVPs the next day your manager asks (like Python). And if you have young students in your team, their enthusiasm will not leave the boundaries of the module abstraction. And if you are the manager you have excellent language (the language of mli files) that you can use to convey your design ideas to other team members and even to non-technical personal.

So let's do some prototyping and design, using the language of signatures. First of all, we shall decide which type should be designed for change and which should be regular data types. E.g., we probably don't want to have a string data type with pluggable implementation. But we definitely know that our figures will change and more will be added, possibly by 3rd-party developers.

We finally decide that Point and Canvas will be the regular types (for Canvas we might regret our design decision). For prototyping, we will write the module type of our project. At the same time, we should also write documentation for each module and its items (functions, types, classes, etc). Writing documentation is an important design procedure. If you can't describe in plain English what a module is doing then probably it is a wrong abstraction. But now, for the sake of brevity and lack of time we will skip this important step and unroll the whole design right away.

module type Project = sig

  module Point : sig
    type t
    val create : int -> int -> t
    val x : t -> int
    val y : t -> int
    val show : t -> string
  end

  module Canvas : sig
    type t
    val empty : t
    val text : t -> Point.t -> string -> unit
  end

  module Widget : sig
    type t
    val create : ('a -> Canvas.t -> unit) -> 'a -> t
    val draw : t -> Canvas.t -> unit
  end

  module type Figure = sig
    type t
    val widget : t -> Widget.t
  end

  module Rectangle : sig
    include Figure
    val create : Point.t -> Point.t -> t
  end

  module Circle : sig
    include Figure
    val create : Point.t -> int -> t
  end
end

Let's discuss it a bit. The Canvas and Point types are pretty obvious, in fact this design just assumes that they are already provided by the third-party libraries, so that we can focus on our figures.

Now the Widget types. Following the dependency inversion principle, we decided to make our rendering layer independent of the particular implementation of the things that will populate it. Therefore we created an abstraction of a drawable entity with a rather weak definition of the abstraction, i.e., a drawable is anything that implements ('a -> Canvas.t -> unit) method. We will later muse on how we will extend this abstraction without breaking good relationships with colleagues.

Another point of view on Widget is that it defines a Drawable type class and that Widget.create is defining an instance of that class, so we could even choose a different naming for that, e.g.,

module Draw : sig
  type t
  val instance : ('a -> Canvas.t -> unit) -> 'a -> t
  val render : t -> Canvas.t -> unit
end

The particular choice depends on the mindset of your team, but I bet that the Widget abstraction would fit better more people.

But let's go back to our figures. So far we only decided that a figure is any type t that defines val widget : t -> Widget.t. We can view this from the type classes standpoint as that the figure class is an instance of the widget class. Or we can invoke Curry-Howard isomorphism and notice that val widget : t -> Widget.t is a theorem that states that every figure is a widget.

The next should be trivial with this signatures, so let's move forward and develop MVP,

module Prototype : Project = struct

  module Point = struct
    type t = {x : int; y : int}
    let create x y = {x; y}
    let x {x} = x
    let y {y} = y
    let show {x; y} = "(" ^ string_of_int x ^ ", " ^ string_of_int y ^ ")"
  end

  module Canvas = struct
    type t = unit
    let empty = ()
    let text _canvas _position = print_endline
  end

  module Widget = struct
    type t = Widget : {
        draw : 'obj -> Canvas.t -> unit;
        self : 'obj;
      } -> t

    let create draw self = Widget {self; draw}
    let draw (Widget {draw; self}) canvas = draw self canvas
  end

  module type Figure = sig
    type t
    val widget : t -> Widget.t
  end

  module Rectangle = struct
    type t = {ll : Point.t; ur : Point.t}
    let create ll ur = {ll; ur}
    let widget = Widget.create @@ fun {ll} canvas ->
      Canvas.text canvas ll "rectangle"
  end

  module Circle = struct
    type t = {p : Point.t; r : int}
    let create p r = {p; r}
    let widget = Widget.create @@ fun {p; r} canvas ->
        Canvas.text canvas p "circle"
  end
end

Et voila, we can show it to our boss and have some coffee. But let's look into the implementation details to learn some new tricks. We decided to encode widget as an existential data type, no surprises here as abstract types have existential type and our widget is an abstract type. What is existential you might ask (even after reading the paper), well in OCaml it is a GADT that captures one or more type variables, e.g., here we have 'obj type variable that is not bound (quantified) on the type level, but is left hidden inside the type. You can think of existential as closures on the type level.

This approach enables us to have widgets of any types and, moreover, develop widgets totally independently and even load them as plugins without having to recompile our main project.

Of course, using GADT as encoding for abstract type is not the only choice. It even has its drawbacks, like we can't serialize/deserialize them directly (though probably we shouldn't) it has some small overhead.

There are other options that are feasible. For example, for a widget type, it is quite logical to stick with the featherweight design pattern and represent it as an integer, and store the table of methods in the external hash table.

We might also need more than one method to implement the widget class, which we can pack as modules and store in the existential or in an external hash table. Using module types enables us gradual upgrade of our interfaces and build hierarchies of widgets if we need.

When we will develop more abstract types (type classes), like the Geometry class that will calculate area and bounding rectangles for our figures, we might notice some commonalities in the implementation. We might even choose to implement dynamic typing so that we can have the common representation for all abstract types and even type casting operators, e.g., this is where we might end up several years later.

module Widget : sig
  type 'cls t

  module type S = sig
    type t
    ...
  end

  type 'a widget = (module S with type t = 'a)

  val create : 'a widget -> 'a -> 'a cls t

  val forget : 'a t -> unit t
  val refine : 'a Class.t -> unit t -> 'a cls t option
  val figure : 'a t -> 'a
end

We're now using a module type to define the abstraction of widget, we probably even have a full hierarchy of module types to give the widget implementors more freedom and to preserve backward compatibility and good relationships. We also keep the original type in the widget so that we can recover it back using the figure function. yes, we resisted this design decision, because it is in fact downcasting, but our clients insisted on it. And yes, we implemented dynamic types, so that we can upcast all widgets to the base class unit Widget.t using forget, but we can still recover the original type (downcast) with refine, which is, obviously, a non-total function.

In BAP, we ended up having all this features as we represent complex data types (machine instructions and expressions). We represent instructions as lightweight integers with all related information stored in the knowledge base. We use dynamic typing together with final tagless style to build our programs and ensure their well-formedness, and we use our typeclass approach a lot, to enable serialization, inspection, and ordering (we use domains for all our data type). You can read more about our knowledge base and even peek into the implementation of it. And we have a large library of signatures that define our abstract types, such as semantics, values, and programs. In the end, our design allows us to extend our abstractions without breaking backward compatibility and to add new operations or new representations without even having to rebuild the library or the main executable. But this is a completely different story that doesn't really fit into this post.

We can easily see that we our design makes it easy to add new behaviors and even extend the existing one. It also provisions for DRY as we can write generic algorithms for widgets that are totally independent of the underlying implementation. We have a place to grow and an option to completely overhaul our inner representation without breaking any existing code. For example, we can switch from a fat GADT representation of a widget to a featherweight pattern and nobody will notice anything (except, hopefully) improved performance. With that said, I have to conclude as it already took too much time. I am ready for the questions if you have any.

• Stop setting switch jobs variable on Windows (OPAMJOBS is sufficient).

• With Sébastien, David, and Gabriel's help, I have finally merged the change needed to integrate odoc in our documentation pipeline. Currently, this is hidden behind a configuration switch (or specific Makefile's target). The user experience is still a bit rough, in particular it requires an trunk-updated version of odoc. Fortunately, the number of users right now is most probably of only one. My current plan is to see how well the maintenance goes during this release cycle before maybe switching to odoc for the 4.13.0 version of the manual.
• I have been discussing with David about how much time and effort we should spend on testing the manual. (My opinion is that testing only the PR that alters the manual's source file is essentially fine.) David has been testing more thorough configuration however but that requires some more tuning to avoid sending scary emails to innocent passersby.

GC safepoints continues to be the focus of the OCaml 4.13 release development for multicore. While it might seem quiet with only one PR being worked on, you can also look at the compiler fork where an intrepid team of adventurous compiler backend hackers have been refining the design. You can also find more details of ongoing upstream work in the first core compiler development newsletter. To quote @xavierleroy from there, "it’s a nontrivial change involving a new static analysis and a number of tweaks in every code emitter, but things are starting to look good here.".

The switch to using OCaml 4.12 has now completed, and all of the development PRs are now working against that version. We've put a lot of focus into establishing whether or not Domain Local Allocation Buffers (ocaml-multicore#508) should go into the initial 5.0 patches or not.

What are DLABs? When testing multicore on larger core counts (up to 128), we observed that there was a lot of early promotion of values from the minor GCs (which are per-domain). DLABs were introduced in order to encourage domains to have more values that remained heap-local, and this should have increased our scalability. But computers being computers, we noticed the opposite effect – although the number of early promotions dropped with DLABs active, the overall performance was either flat or even lower! We're still working on profiling to figure out the root cause – modern architectures have complex non-uniform and hierarchical memory and cache topologies that interact in unexpected ways. Stay tuned to next month's monthly about the decision, or follow ocaml-multicore#508 directly!

Aside from this, the test suite coverage for the Multicore OCaml project has had significant improvement, and we continue to add more and more tests to the project. Please do continue with your contribution of parallel benchmarks. With respect to benchmarking, we have been able to build the Sandmark-2.0 benchmarks with the current-bench continuous benchmarking framework, which provides a GitHub frontend and PostgreSQL database to store the results. Some other projects such as Dune have also started also using current-bench, which is nice to see – it would be great to establish it on the core OCaml project once it is a bit more mature.

We are also rolling out a multicore-specific CI that can do differential testing against opam packages (for example, to help isolate if something is a multicore-specific failure or a general compilation error on upstream OCaml). We're pushing this live at the moment, and it means that we are in a position to begin accepting projects that might benefit from multicore. If you do have a project on opam that would benefit from being tested with multicore OCaml, and if it compiles on 4.12, then please do get in touch. We're initially folding in codebases we're familiar with, but we need a diversity of sources to get good coverage. The only thing we'll need is a responsive contact within the project that can work with us on the integration. We'll start reporting on project statuses if we get a good response to this call.

As always, we begin with the Multicore OCaml ongoing and completed tasks. This is followed by the Sandmark benchmarking project updates and the relevant Multicore OCaml feature requests in the current-bench project. Finally, upstream OCaml work is mentioned for your reference.

• AMD: Advanced Micro Devices
• CI: Continuous Integration
• CTF: Common Trace Format
• DLAB: Domain Local Allocation Buffer
• DWARF: Debugging With Attributed Record Formats
• ETL: Extract Transform Load
• GC: Garbage Collector
• OPAM: OCaml Package Manager
• PR: Pull Request
• UI: User Interface
• URL: Uniform Resource Locator
• ZMQ: ZEROMQ

This graph shows the "contributor cohorts" for the OCaml compiler over time. For example, the big dark-red bar that shows up in 2015 represents the "2015 cohort", the number of long-term contributors to the OCaml compiler that did their first contribution in 2015. The dark-red bar in similar position in each following year represents the contributors from the 2015 cohort that are still active on that year. The bar shrinks over time, as some members of this cohort stop contributors. Short-term contributors (all their contributions fall within a 90-days period) are shown as the "Brief" bars at the top.

The main thing we see on this graph is that moving the compiler development on Github in 2015 increased sharply the number of contributors, which has remained relatively stable since (there is an "expert pool" that is stable in size), with a large fraction of occasional contributors each year.

(Note: stability of contributor numbers is fine for the compiler, which is not meant to keep growing in size and complexity. We hope most contributors go to other parts of the OCaml ecosystem.)

This graph shows the number of commits from the contributors of each cohort. We see for example that the 1995 contributor, namely Xavier, has remained relatively active throughout the compiler development, with a marked uptick in 2020 (possibly related to the Multicore upstreaming effort). Today most of the commit volume seems to come from community members that started contributing right after the Github transition, after 2015-2016.

It's interesting to compare these two charts: we see that the 2015 cohort has shrunk in size in 2020 (by half), but that they contributed much more in 2020 than in 2015: over time, the remaining contributors from this cohort grew in confidence/expertise/interest and are now contributing more (several of them became core maintainers, for example).

I then ran the same visualization tool on all OCaml git repositories listed in the public opam-repository. This is a very-large subset of all open source software implemented in OCaml. But it does not represent well the "industrial" codebases that some industiral OCaml users are working on – even when the code is open-source, it may be packaged and distributed separately.

This graph shows the number of contributors, in yearly cohorts. We can see that the number of contributors has been growing each year, plateauing in 2018.

Note: there is a measurement artefact that makes the last column smaller than the previous ones: some of the "short-term" contributors in 2020 will later become longer-term contributor by contributing again in 2021, so they be added to the long-term cohort of 2020. This artefact may suffice to explain the small decrease in long-term contributors in 2020.

This graph shows the volume of commits. Here we don't see a plateau; there is in fact a small decrease in 2018, and further growth in 2019 and 2020. Another aspect I find striking is the stability of commit volume in each cohort. For example, the 2014 cohort seems to have contributed roughly as many commits during all years 2016-2020. Given the reduction in the number of contributors in this commit, this is again explained by fewer contributors gradually increasing their contribution volume.

Some industrial OCaml codebases are included in the public opam repository, but a large part is not.

This visualization aggregates project data assuming that they follow "standard" git development practices. The data is imperfact, it may be skewed by tool-generated commits. For example, some of the Jane Street software packaged on opam uses git repository mirrors that are updated automatically by usually a single committer, in a way that does not reflect their true development activity. (Thanks to @yminsky for catching that.)

Another threat to validity is that some authors commit in different projects using different names, so they may be counted as separate contributors instead. (Inside a project, one may use a .mailmap file to merge contributor identities, but afaik there is no support in git or fornalder for overlaying an extra .mailmap file that would work across repositories.)

If you wish to study the dataset to see if the overall conclusions are endangered by such anomalies, please feel free to replay the data-collection steps. You can either manually inspect git repositories, or play with the SQLite database generated by fornalder.

I found this analysis interesting. Here would be my conclusion so far:

• The OCaml community gets a regular influx of new contributors.
• Some of our contributors stay for a long period, and they contribute more and more over time.
• We observe a plateau-ing numbers of new contributors on the years 2018-2020 (and the pandemic is probably not going to improve the figure for 2021), but the volume of commits keeps growing.

It is difficult to draw definitive conclusions from these visualizations, especially as we don't have them for many other communities to compare to. Compared to the Gnome trends shown in the original blog post ( https://hpjansson.org/blag/2020/12/16/on-the-graying-of-gnome/ ), I would say that we are doing "better" than the Gnome ecosystem (in terms of attracting new contributors).

My personal view for now is that OCaml remains a more niche language than "mainstream" contenders (we don't see an exponential growth here that would change the status), but that its contributor flow is healthy.

You can find a curated log of my analysis process in logs.md; this should contain enough information for you to reproduce the result, and it could easily be adapted to other software communities.

I uploaded all the small-enough data of my run in this repository, in particular the list of URLs I tried to clone – some of them failed. Not included: the cloned git repositories, and the databases build by fornalder to store its analysis data.

npm-ocaml contributors

npm-ocaml commits

(If you wonder what's the long trail between 2003 and 2009: this is the development of `bs-sedlex`, which goes back to an old OCaml-only prototype by Alain Frisch in 2003. There is also a version of OCaml packaged on npm, but I removed it from the analysis as it was adding noise and was mostly not-esy-specific contributions.)

Note that we are talking about ~4K commits here, which remains fairly small compared to the ~120K commits for opam-repository packages on the last year. When I tried to merge both sets together this didn't make much of a difference compared to just-opam numbers.

Maybe someone should redo the analysis with "reason/rescript" tags in addition, to measure the contribution volume there. I sticked with packages that self-identify as "ocaml" for now.

Here are graphs for batteries-included.

Cohorts, per number of contributors:

Cohorts, per volume of commits:

What we see, I think, is that Batteries has been fairly quiet since 2015, which probably corresponds to entering some kind of "maintenance mode". There is still a reasonable diversity of contributors, with many one-shot contributors (which I assume corresponds to user that are mostly happy silently using the library, and come to add a function or fix a bug once in a while).

Looking at the volume of commits: the strong decrease of the gray bar in 2018 corresponds, I think, to when I stopped contributing actively, and you took over as a contributor. It looks like I was effectively the last of the early-day contributors still active. The purple "2013" cohort is interesting, and I went to look at the data: it's you (François Berenger) and Simon @c-cube Cruanes. Simon contributed a lot on a short period, and then went off to create the very nice Containers library that would move faster. You stuck and are now the most active contributor (and maintainers).

Contributors:

Commits:

Containers is mostly a one-person library with Simon doing most of the work. There were many new contributors in 2017 and 2018 (most of them brief), and the strong show of purple year 2018 in today's commit volume is mostly due to the enigmatic Fardale.

I think that fornalder is more useful to study large repositories (or set of repositories) that have been going for many years. For a single project, especially if they are relatively small or young, git shortlog -n -s (over the whole log or --since 2018, etc.) tells you mostly the same thing.

• Timestamp and date time handling with platform independent time zone support
  • Subset of the IANA time zone database is built into this library
• Supports Gregorian calendar date, ISO week date, and ISO ordinal date
• Supports nanosecond precision
• ISO8601 parsing and RFC3339 printing

This is a much more polished repackaging of the date time components from Timere. The separation and restructuring came from the growing size of the date time components, and very nice and extensive feedback on UX from @gasche at issue #25 and other issues branching from it (many thanks!).

And as usual, many thanks to @Drup for his advice.