OCaml Weekly News

This is the first release of DKML in winget, the (new) Windows Package Manager.

Windows Package Manager winget command-line tool is bundled with Windows 11 and modern versions of Windows 10 by default as the App Installer. If you are running an earlier version of Windows and the App Installer is not installed, you can get App Installer from the Microsoft Store. If it’s already installed, make sure it is updated with the latest version.

Once you have winget, uninstall any existing DKML using the uninstall link (see the main announcement) and then type the following for the 90-minute install:

winget install Diskuv.OCaml

You will then be able to start hacking in a new PowerShell window at Install is done! What next?

How to check if you are using setup-dkml? You have a ci/setup-dkml folder.

The new version of setup-dkml CI goes along with the DKML 1.1.0 release (yes, it would be nice if the version numbers were in sync). You will get CI for Windows MSVC with OCaml 4.14.0 (among other things) when you upgrade.

You should check ci/setup-dkml/*.md to see if there is any upgrade documentation. If not, you can upgrade with:

opam update
opam upgrade dkml-workflows
opam exec -- generate-setup-dkml-scaffold
opam exec -- dune build '@gen-dkml' --auto-promote

The first thing I found notable (and reassuring) in the 2022 survey results is what wasn’t mentioned in the user responses: instability with the core system. We began 2022 drunk with relief from the multicore PR being merged, and began the year-long 5.0.0 release process. In one of the core developer meetings early in the year, I requested that the 5.0.0 release would be restricted to just the multicore runtime and effects features, since so much had changed there that we would have our hands full. This was promptly ignored and followed by a surge of PRs removing lots of legacy cruft that had built up over the course of the 4.x series. A recipe for release management disaster!

I’m glad to say that I was wrong, and (as you’ll see from the infrastructure report below) that the collective 5.0.0 release effort has been one of the most impressive I’ve witnessed. The core team signalled deprecations clearly, and the various tooling teams (such as opam and dune) in the ecosystem performed lots of differential builds and incremental releases to remove vestigial deprecated fragments that now broke builds. This was then followed by an engaged community releasing their various dependencies to the mainline opam repository, all in good time for the 5.0 release candidates to be cut and be usable.

We are, of course, still really early in the lifetime of the OCaml 5.x series, and some serious breakages may yet lurk and only be discovered as our users migrate. But we are at a point now where experimentation, prototyping and migration can be done in a controlled way across both OCaml 4.x and 5.x, so let’s pat ourselves on the backs for a moment about a job well done before moving on. I’ll ninja-edit this post if next year’s user survey is full of complaints about OCaml 5 ;-)

The 2020 user surveys made it crystal clear that we needed to improve the state of art with documentation and OCaml. Accordingly, we started work in 2021 on a new site, previewed it in early 2022 and launched in April 2022.

This new website preserved older links (by redirecting them to an archived v2.ocaml.org) and provided a brand new centralised documentation site with package search and incremental rebuilds to ensure new packages get up there in a timely fashion. This was a complex task behind the scenes, since it requires ongoing bulk package builds of every version of every package in the opam repository, with consistent cross-linking.

It’s still by no means perfect, with some work needed on rendering glitches, missing sections, and the overall information architecture of how to present so much information to a range of users (beginner to advanced), but it’s a really solid foundation to work from (unlike the previous website, which was really showing its age). This year’s user survey continues to emphasise the need for advancing documentation, so I hope to see more contributions to the new website to ensure the content continues to improve. And of course, look to your own published libraries and ensure that your odoc markup is as good as you can make it.

The other hot topic in 2022 was for us to figure out how to integrate better with modern, stateless workflows for code development. This is a complex topic for tool maintainers to work on, since we must also preserve backwards compatibility with existing workflows (witness the very high percentage of users in this year’s survey that use the opam cli as their primary mechanism of interaction). We also had a decent idea of who the various sorts of users are from discussion in 2019 with application developers, library authors and OS maintainers.

There have been a number of prototypes built this year to experiment with new workflows, but most of the effort from core maintainers has gone into the earlier priorities (releasing OCaml 5, especially). The prevalence of dune (>91% in the 2022 survey) means that one simple stateless workflow is to have all the source code available in one monorepo, and perform a dune build. This works because dune can scan all the dune files in the repo and build them in one pass. In this workflow, opam can be optionally used to assemble the source code (see the opam-monorepo) plugin, but our Nix-loving friends also have their own alternative mechanisms (1, 2, 3). While some projects like MirageOS and Real World OCaml are using these workflows, they are still maturing. Now that OCaml 5.0 is out and the new website is live, I hope to see more production quality workflows emerging in 2023.

There is also some concern that dune shouldn’t be a hard requirement on any workflow. This requirement has been successfully preserved to date, but is getting increasingly difficult to reconcile with the demand for a more opinionated, beginner-friendly workflow. With opam, our architectural answer to this is to separate the opam file format from the opam CLI, and make it easier to interpret opam repositories via external tools. The same discipline should work for dune files with alternative build systems.

How do we – as a community – figure out what will work for new workflows? That’s my segway into what I want to hear your thoughts on for 2023: how to incrementally improve community communication.

• discuss.ocaml.org (Discussion forum): I setup this forum back in 2018 in response to a user request, and it has been a successful experiment. The number of OCaml users who report interacting via this forum has increased percentage-wise from the 2020 to 2022 results. I am also pleased that a small moderation team has been sufficient to deal with spam. Traffic-wise, we have had to upgrade the hosting capacity several times in the past 5 years as demand rises, and there has been a mild surge in page-views towards the end of 2022 with the release of OCaml 5.0.0.

  

  I am still regretful that I had to sunset the mailing list service, but it will hopefully be back in 2023, especially if we find a volunteer to help configure and maintain it.

• watch.ocaml.org (Video sharing): The Peertube-based service began in the 2020 lockdowns when we shifted our workshops online. Since then, it has been a resilient and useful resource to host videos about OCaml related topics. There are some advantages to hosting our own videos: they can be permanently archived and linked to, and we can integrate well with other “Fediverse”-based services (more on this later).

  What I’d like to see in 2023 is more content being backfilled onto this site, so that we can have all the videos from the last few decades of OCaml conferences and workshops in one place! To that end, we will promote the service to non-beta status soon. It is a little tricky to use Peertube with multiple users (still involves password sharing), so we’re figuring it out as we go along and before finishing the promotion to production status. Please do leave comments and thoughts on that issue or here.

• opam.ocaml.org (Package management): The surveys confirm that opam remains the dominant mechanism of installing and accessing the OCaml package ecosystem. In addition to regular releases of the opam tool itself, the backend infrastructure has been upgraded significantly so that the package archive should be more available and secure, and easier to mirror onto global CDNs in 2023 and also integrate with other software security supply chain software.

  Alongside serving the package archives themselves, we maintain a significant multi-architecture cluster of machines that perform the bulk builds to ensure the health and integrity of the opam repository (the curated package database). These machines comprise of x86, ARM, PowerPC and RISC-V machines (yes, we did finally get rackmounted RISC-V boards this year!). The machines are variously hosted at the Cambridge Computer Laboratory, Inria, Scaleway and Equinix, and we are sunsetting our use of AWS for cost reasons. Individual machines have been generously funded by IBM, Tarides, Jane Street, the Works on ARM program.

  The software driving this cluster continues to grow, with support for Windows and macOS builds going live in 2022. These are not yet hooked up to the live opam-repository-ci, but will hopefully be back in 2023. This marks our migration away from hosted CI services such as Travis and AppVeyor, and the backing infrastructure is open source and possible to deploy for yourself (e.g. in an industrial context).

• www.ocaml.org (Website and Docs): I’ve talked extensively about the new website earlier, but I would like to emphasise the importance of receiving external contributions to the content of the website. The repository is open for PRs, and it has been a little quiet in the latter half of 2022 outside of a small maintainer team. If you’d like to get involved, then please feel free to open an issue and discussing your plans, or signal any blockers you encountered. Gabriel has already noted bottlenecks in the core OCaml distribution, and a similar story is playing out in the wider ocaml.org ecosystem. We need your contributions.

• git.ocaml.org (not launched): a service that we have considered since 2019 and did not launch is a git mirror, or an alternative way of procuring OCaml ecosystem source code than from GitHub. There have been a small but steady stream of requests for this, with several motivations: availability (GitHub is a central point of failure), security (replicating ecosystem git branches is sound and secure practise), and privacy. The core OCaml team is firmly committed to GitHub at present, but launching a read-only mirror is in scope for 2023 if a maintainer is willing to step forward and survey available solutions (ideally not as heavy-weight as GitLab) for mirroring scripts. Once we have a robust read-only git mirror, we can begin to consider how to accept patch contributions (particularly to the opam repository) via email or other mechanisms, but no promises until we reach the first read-only milestone.

  I’d really like to hear from industrial users who have stronger requirements for secure software supply chains here as well. I participated in a White House summit on software security earlier in the year, and it is clear that this is going to be an important topic for OCaml to keep up with in 2023, especially with our role in the formal verification ecosystem.