OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of January 03 to 10, 2023.

Table of Contents

OCaml.org: recapping 2022 and queries on the Fediverse

Deep in this thread, Anil Madhavapeddy replied to several questions

Energy Usage

were environmental issues taken into account (eg. energy consumption)?

I punted on this in my recap as I hadn’t had a chance to catch up with @patricoferris about it, but since it’s a very important topic let’s start talking about it incrementally now instead of waiting for that!

In the redesign for the new site, we explicitly removed third-party trackers and took advantage of the spare screen space (usually reserved for a privacy policy, now no longer needed) and put in a OCaml.org carbon footprint statement as a placeholder until we obtained more specific data.

Later in 2022, @patricoferris investigated how we could do better in terms of power monitoring, and is developing a suite of OCaml tools that will hopefully be useful to the wider community as well:

  • Variorium collects hardware-level power counter information, for accurate monitoring
  • Carbon Intensity is a tool to integrate with country-level APIs for where energy is primarily coming from, in the absence of more specific information from the datacentre provider.
  • Clarke combines all this into a convenient Prometheus monitoring, for centralised analysis.

These are all still unreleased, and I’ve opened a tracking issue about the deployment of these into the ocaml.org cluster. If anyone would like to help out (particularly around finding more accurate APIs for carbon intensity) then feel free to open issues/PRs on those various repositories.

Some services, such as restoring inbox.ocaml.org are a little blocked on this topic as I’m reluctant to provision more long running virtual machines without thinking through more efficient alternatives that can consolidate services (e.g. have just one SMTP endpoint instead of multiple). My apologies to @xavierleroy and @nojb for the delay, as they have both done a bunch of work towards restoring it already, and I’ll do my best to catch up this month.

Privacy

I’m wondering if private messages on this forum are actually ‘private’, or are they readable by the admins

The only digital communications mechanism that we’re using that features end-to-end encryption is Matrix. That implies that, as a general rule, that most of the alternatives such as Slack, IRC, Discord and Mastodon, do allow their respective admins to read your messages. Discourse (the software powering this forum) has explicit support for admins to monitor private messages for online safety reasons, although to my knowledge this facility has never needed to be used for this deployment.

If you want a reasonably usable mechanism for private messages, then Matrix is the way to go, including for encrypted group channels, and all the other services are one security breach away from going public.

As for the discussion about openness, I’m personally not really a believer in being radically transparent when doing open-source work. I find it really difficult to focus on a topic when in the public eye, and instead prefer to work on it with my immediate collaborators and then have an open discussion about it. What I really miss is the ability to promote information that results from the private discussions into a more open forum – all these recaps and newsletters are entirely written from scratch, and the inefficiency means that it’s a huge amount of effort to get right. It’s easy to put the time in with full papers since there is a reward structure (for academics, anyway) in place via the conference circuit, but less so for other mediums. A project I’m going to return to sometime this year is Bushel, where I’ve been prototyping a communications format suitable for iterative promotion and integration with data scrapers.

Source mirroring

I know I’m no maintainer and this is no survey but please consider sourcehut ! It’s lean, easy to maintain, and very email-friendly! It could even double as the new home for ocaml-list.

To be clear, there’s no special ’maintainer bit’ or survey required to give your feedback – a maintainer is just someone who puts the time in to help out with a particular area. For example, we got brilliantly helpful external feedback for the opam archive migration here just a few days ago.

I do like SourceHut a lot, but we’d ideally self-host it, and that’s quite a bit of work due to its microservice architecture. It should be possible to strip down the services (remove the autobuilders and bug trackers) for a read-only mirror, and so a good way to contribute would be assemble a Docker compose file with such an installation and demonstrate how it might work with a sample set of Git repositories to mirror. If you (or anyone else reading this) wants to have a go, feel free to create an issue on https://github.com/ocaml/infrastructure/issues with your prototype.

Anil Madhavapeddy later replied to more questions

The “Books” page (OCaml Books ) is often out of date and needs more maintenance.

I updated the RWO entry and added Michael Clarkson’s brilliant book from Cornell’s CS3110. We finished importing in the last of the OCaml Workshop videos yesterday so they should be up-to-date from 2012->2022 now when ocaml/ocaml.org#112 is merged.

The papers could use significant backfilling (there have been a lot of OCaml related papers in the last 20 SIGPLAN conferences and affiliated workshops). @octachron suggested BibTeX import for that which should make it more practical.

In general, more hands make light work of this data maintenance. There’s a good CONTRIBUTING.md for the ocaml.org site, so please do all of you look at it and see if you can help improve the job board, or success stories, or papers/books/anything else in there.

  • Having online documentation for OCaml packages is a huge progress. Yet, a Google search for a package name (e.g. “ocaml cryptokit”) generally lands on an opam.ocaml.org page (e.g. https://opam.ocaml.org/packages/cryptokit/) that does NOT point to the documentation, instead of the ocaml.org page (e.g. https://ocaml.org/p/cryptokit/1.18) that does link to the documentation. Why do we have two different pages with different contents for the same package? Could we merge them in one page that contains everything there is to know about that package?

Tracking issue is now up at ocaml/infrastructure#26. It covers a few low hanging fruit things we can do straight away that’ll improve the situation.

I agree that OCaml’s share of the computer carbon footprint is negligible. What then is the goal of pursuing this direction? Is there a demand among government users for these features? Is there commercial demand for this niche where OCaml could fill a role? Or is it because it’s a research area that is currently underserved?

We are doing it because it is the right thing to do at every level when we are in the middle of a climate crisis. At an individual and organisational level, we all need to reduce our emissions footprints by avoiding wasteful consumption. As computer scientists, we need to develop tools to help society reduce our emissions footprints. As purchasers of computers and materials, we need to apply pressure on our vendors to reduce their emissions footprints and ensure they have responsible supply chains that adhere to good recycling and zero deforestation commitments. As consumers of cloud computing resources, we need to ensure they are minimal in their energy footprints and use of renewable power. As architects of distributed systems infrastructures, we need to ensure we engineer facilities to accurately record our emissions. As members of the global open source community, we need to set an example of best practises that may be replicated more widely.

We learn by doing, and I am very disappointed that the responses here so far include no enthusiasm or encouragement to the interesting OCaml libraries by @patricoferris that I linked to. All of those are reusable more widely, and have obvious applications in reliable energy monitoring (in e.g. embedded systems) in the real world.

And for those unmoved by the burning planet argument, there’s an even easier one. The best way to make computers go fast is to do less work, and that’s what all the disciplined tracking and reproducible infrastructure forming around ocaml.org does: makes it all go faster. Weren’t a bunch of you complaining that the opam package database sometimes took a day to update? Well, it’s rebuilding entire documentation universes in that time now, and package updates should take minutes once we finish the opam2web migration :slight_smile:

Anil Madhavapeddy answered yet more questions

I think that the new website has too much empty space. On the old website, information is a lot more compact. The new website has a lot of padding, which feel gratuitous to me.

To follow on from @tmattio’s comments, I also raised the same query during the design process. As far as I can figure, most modern websites are designed for mobile usage – a huge percentage of users now come in via tablet/mobiles, and so responsive design is really important. That explains the medium/small layouts, but not why there is so much whitespace in the wider screen layouts. It’s really obvious when comparing an opam.ocaml.org package description with the equivalent on ocaml.org/p. The next iteration of design is beginning now, so please do help out with the survey and your thoughts that @sabine just posted about, and we can get those fixed.

I’ve been running a Peertube instance with thousands of users for years and it never involved password sharing. I’m quite surprised to hear this…

The problem is pretty simple: PeerTube doesn’t support shared video channels, so one user has to own them. In our case, the intrepid bactrian returns for all the OCaml Workshop videos.

If we do create separate users, then it also looks weird. For example, @patricoferris uploaded OCaml Workshop 2022 videos under his own account, and now when you reference them from Mastodon it looks like you’re referring to @patrickferris since it drops the domain portion by default. See here for a ’toot’ that is owned by the bot, and here one that is harder to distinguish.

And then… I did indeed create you an oups@watch.ocaml.org account in April, but you haven’t uploaded anything since. What do we do if someone else in the future wants to take over OUPS videos and get them online? You need to account share. Most of the ActivityPub services like Mastodon and PeerTube are quite user-centric at the moment, and not well suited to shared publishing. But it works well enough with a little coordination amongst ourselves and some trust. Concrete suggestions for improvement welcome.

It’s probably better with a link to the room . :wink:

Even better with issues to improve the ocaml.org website ;-) Now at ocaml/ocaml.org#767 and ocaml/ocaml.org#768 for your events suggestion.

It’s nice to hear about all the existing/future decentralized services. In the other hand, IIUC, all these services hosting/maintainance seem to be handled by a single person and I feel it may goes against the idea and benefits of decentralized services.

Using decentralisation has got nothing to do with one hosted instance. By using these protocols, the information related to OCaml can be replicated across multiple sites and reconstructed if one service goes down. For example, my personal crank.recoil.org instance “follows” and mirrors the videos on watch.ocaml.org, as do around 50 other PeerTube instances. So the ocaml.org domain is most valuable as a namespace, where it can aggregate and publish information that is actually generated elsewhere. In an ideal world, the ACM SIGPLAN team would publish their videos on PeerTube as well as YouTube, and ocaml.org would be a bookmarking/mirroring service.

While this is the theory, in practise the ActivityPub protocol is very URL-centric and so makes it hard to recover from federated domains disappearing. You can read more about this in an excellent undergraduate project last year by Gediminas Lelešius on Improving the Resilience of ActivityPub Services.

As for your point about hosting being centralised; I’m not the only maintainer. There are around a dozen maintainers spread across the ocurrent and opam/dune/ocaml orgs that keep everything ticking along. Some of the core machines do indeed only have a couple of people with access, but this is for obvious security reasons. No service has only 1 person with access, so we have a reasonable “bus factor”.

As @jbeckford observed in another thread, we are in definite need of more maintainers throughout the OCaml ecosystem. For infrastructure, the best way to get involved is by help to scope out technologies (like Mobilizon or SourceHut, in this thread), or by contributing to the software stacks behind it (like the various CIs listed in the ocurrent site). My personal hope is that someone will start building complete ActivityPub bindings in OCaml so we can start having some Fediverse fun in our own language :slight_smile:

shuttle v0.3.1 released

Anurag Soni announced

There’s a new version (0.8.1) of the library published to opam. Some highlights:

Shuttle

  • Buffered reader’s support timeouts for the refill operation. If a file descriptor is currently not ready for a read sys call, the reader will wait for the user-defined upper bound time span and raise an exception indicating a timeout was reached if the underlying file-descriptor isn’t ready for IO within that duration.
  • The buffered reader and writer optionally accept an Async Time_source. They default to wall clock, but this input can be used to provide alternate time sources that match the Time_source interface.

Shuttle_ssl

  • Client and Server SSL connections forward the SSL connection context to the user provided callbacks. This can be useful to access the session certificates for ensuring they meet the user’s expectations.

Shuttle_http

This is a new companion library that implements the HTTP/1.1 server codec. There is more work needed to ensure we cover the HTTP 1.1 spec properly, but this initial release includes a usable implementation of the most common use-cases. The library ships with a hand-rolled parser with decent performance and good test coverage (Thanks to ocaml-afl and afl-fuzz I found some bugs I otherwise wouldn’t have!!), and simple service definitions that can support HTTP keep-alive, fixed length and chunked bodies, streaming support to help work with large bodies, request pipelining and centralized error handling for any unhandled exception in the user handler or the I/O runtime. Most of the implementation of http_async directly within the shuttle project. Future work for the HTTP runtime will happen within shuttle, and http_async will focus on providing a more opinionated higher level interface for writing async http services.

Some additional features that aren’t necessarily related to the HTTP spec include timeout support for request header parsing (this helps shut down clients that are too slow), and a public api that provides enough control to allow scheduling resources cleanup for streaming response bodies (Resources backing a stream will be shutdown when the socket connection closes, either for EOF, user-action, or any unhandled error while the runtime performs i/o).

  • Examples
    open Shuttle_http
    
    let hello_world (request : Request.t) =
      return (Response.create ~body:(Body.string "Hello World") `Ok)
    ;;
    
    open Shuttle_http
    
    let my_service (ctx : Server.t) (request : Request.t) =
      let%map reader =
        (* This an example to show a stream that works with an external resource. *)
        Reader.open_file "<some file path>"
      in
      (* Create a pipe from the reader that we will use as a streaming response body. *)
      let reader_pipe = Reader.pipe reader in
      (* Register a callback that's called when the server handler is closed that will also
         close the pipe, in-turn closing the Reader and the underlying file descriptor. This
         is useful in scenarios where the connection is interrupted before the response body
         is fully exhausted and helps avoid resource leaks. This example is demonstrating how
         to do this manually using the server handler. Creating a response via
         [Server.respond_stream] will automatically register the action to close a stream on
         Server connection closing. *)
      upon (Server.closed ctx) (fun () -> Pipe.close_read reader_pipe);
      let response_stream = Body.of_pipe `Chunked reader_pipe in
      Response.create ~body:response_stream `Ok
    ;;
    
    (* The same service written using the utility method for creating streaming responses. *)
    let my_service (ctx : Server.t) (request : Request.t) =
      let%map reader = Reader.open_file "<some file path>" in
      let reader_pipe = Reader.pipe reader in
      return (Server.respond_stream ctx (Body.Stream.of_pipe `Chunked reader_pipe))
    ;;
    

    The library is available via opam. If you try the library and experience any issues, or have further questions, please report an issue on the Github Issue tracker.

Blog post: Testing in Seqes

Raphaël Proust announced

I have written a blog post about the test suite which covers this library.

TL;DR:

  • I use GADTs to describe the library API and generate QCheck tests.
  • I manually used mutaml for mutation-testing on the test suite.

https://raphael-proust.gitlab.io/code/testing-seqes.html

The big takeaway for me is how useful mutation testing is. It gives some confidence in the test coverage not just in terms of what paths are visited but also that modifications are indeed caught by the tests. If you haven’t used mutaml I’d recommend giving it a go. If you have I’d be interested in ideas for integrating it better with some common build tools (dune mostly).

(Also, version 0.2 of Seqes is now available.)

ocplib-simplex v0.5 released

OCamlPro announced

With a bit of delay, we are happy to announce that a new version of ocplib-simplex, v0.5, was released on last November.

ocplib-simplex implements the Simplex Algorithm, and is used for example in the Alt-Ergo SMT solver.

This new version aims at making the interface more generic, so that it can easily be used in other projects.

For more information, see our post on OCamlPro’s blog

Looking for Participants for User Survey on ocaml.org Package Documentation and Learn Area

Sabine Schmaltz announced

the ocaml.org maintenance team is working diligently to bring you desperately needed UI/UX improvements. Starting right now, we are working with a UI/UX designer (hi Claire!) to make sure that the changes we make improve your experience on the site while also looking nice!

We’re running a (short) user survey via Google forms on the package documentation and learn areas with two goals:

  1. to get a broad overview of what the most pressing UI/UX issues to fix are, and
  2. to recruit some participants for interviewing them in a 20 minute video chat with Claire (optional)

To participate, please fill out the survey at https://docs.google.com/forms/d/e/1FAIpQLScRGByWGyqawjTJHssJ5jfSoh6kIayuFmc12E0gLOUb82OJ1A/viewform?usp=sf_link (any feedback is useful, you do not need to be available for interview to participate)

Claire is an experienced UI/UX designer, but not a developer, so having the opportunity to get to know the community while also learning what problems you are experiencing on ocaml.org is very useful to bring her up to speed.

It is likely that there will not be enough time to interview everyone who offers to help, but if you get invited to an interview with Claire, do not hesitate to show her what is, according to your experiences, wrong or broken with the package documentation and learn area on ocaml.org.

Release of ocaml-sf/learn-ocaml:0.14.1

Erik Martin-Dorel announced

We are very pleased to announce the latest stable release of Learn-OCaml, version 0.14.1.

Many thanks to all users who reported bugs and feature requests!

A comprehensive list of the fixes and enhancements offered by this release is available in the Release Notes, and the corresponding opam-repository pull request is pending.

If you are using learn-ocaml.0.14.0, we strongly recommend that you update your version to 0.14.1 to benefit from these latest fixes.

In particular, this release removes the automatic dialog “A more recent answer exists on the server. Do you want to fetch the new version?” that was showing up after 180“, but happened to be flawed: Instead, the exercises view now contains a 3-fold Reload button that allows end users to retrieve the Template, the latest draft, or the latest graded version:

209889453-72c43707-1829-4b36-bcba-db158234eac5.png

https://github.com/ocaml-sf/learn-ocaml/pull/506#issuecomment-1367003486

If need be, feel free to open issues in the Learn-OCaml bug tracker or the learn-ocaml.el bug tracker, or post in this thread to share comments.

Happy OCaml learning and teaching!

The relationship between call/cc, shift/reset, and effect handlers

François Pottier announced

Advertisement!: Paulo de Vilhena’s PhD dissertation, whose main theme is formal reasoning about programs that involve effect handlers, contains chapters about the connections between callcc/throw, shift/reset, and effect handlers (including encodings between these combinators and proofs that these encodings give rise to the expected reasoning rules).

It also contains a specification and a proof of correctness for invert, a function that uses effect handlers to turn an iter function into a lazy sequence.

10 years of OUPS, meetup january 2023

zapashcanon announced

The next OUPS meetup will take place on Thursday, 19th of January 2022. It will start at 7pm at the 4 place Jussieu, 75005 Paris.

This will be an opportunity to celebrate the 10th anniversary of OUPS meetups! :tada:

:warning: :trumpet: It won’t be in the usual room but in the Herpin amphitheater in the Esclangon building. :trumpet: :warning:

Please, register on meetup as soon as possible to let us know how many pizza we should order.

For more details, you may check the OUPS’ website .

This month will feature the following talks :

Analyse statique par interprétation abstraite de programmes fonctionnels manipulant des types algébriques récursifs – Milla Valnet

Afin de prévenir les erreurs de programmation, des analyseurs statiques ont été développés pour de nombreux langages ; cependant, aucun analyseur mature ne cible l’analyse de valeurs pour un langage fonctionnel à la ML. On décrira ici un analyseur statique de valeurs par interprétation abstraite pour un langage fonctionnel typé du premier ordre, approche sûre et automatique pour garantir l’absence d’erreurs à l’exécution. Cette approche nous permet d’analyser des fonctions récursives manipulant des types algébriques récursifs et d’inférer dans un domaine abstrait leur relation entrée-sortie. Une implémentation est en cours sur la plateforme MOPSA et analyse avec succès de courts programmes de quelques lignes. Ce travail ouvre ainsi la voie vers une analyse de valeurs précise et relationnelle basée sur l’interprétation abstraite pour les langages fonctionnels d’ordre supérieur à la ML.

YOcaml - Xavier Van de Woestyne

YOCaml is a static blog generator that was developed to test the Preface library. It is relatively impragmatic, partially poorly documented, and has no effective caching strategy. However, its genesis is fun (and tries to use tools not very present in the world of OCaml) and some external contributions have made it quite explicitly coupling with MirageOS. In this presentation, I invite you to discover its general operation, some design choices and a brief tutorial on how to use it.

ocaml-lsp-server 1.15.0-4.14 and 1.15.0-5.0

Ulugbek Abdullaev announced

On behalf of the ocaml-lsp team, I’m pleased to announce ocaml-lsp-server versions 1.15.0-4.14 and 1.15.0-5.0 for OCaml 4.14 and 5.0, respectively. This is the first time we release ocaml-lsp supporting two compiler versions. We look forward to your feedback.

(the versions will be available on OPAM soon)

Full change log:

  • Features
    • Enable semantic highlighting support by default (#933)
    • Support connecting over pipes and socket. Pipes on Windows aren’t yet supported (#946)

      More about communication channels in LSP specification.

    • Re-enable ocamlformat-rpc for formatting code snippets (but not files and not on Windows) (#920, #939)

      One needs to have installed either ocamlformat package version > 0.21.0 or, otherwise, ocamlformat-rpc package. Note that previously ocamlformat-rpc came in a standalone OPAM package, but since ocamlformat version > 0.21.0, it comes within ocamlformat package.

    • Add custom ocamllsp/hoverExtended request (#561)
    • Support utf-8 position encoding clients (#919)

      More about position encoding in LSP specification.

    • Show unwrapped module alias types on hovering over module names. This is due to upgrading to merlin 4.7 and using merlin’s verbosity=smart by default (#942)
  • Fixes
    • Respect the client’s completion item resolve and preSelect capabilities (#925, #936)
    • Disable polling for dune’s watch mode on Windows and OCaml 4.14.0 (#935)
    • Fix semantic highlighting of “long identifiers,” e.g., Foo.Bar.x (#932)
    • Fix syncing of document contents:
      • For ranges that span an entire line (#927)
      • Previously, whole line edits would incorrectly eat the newline characters (#971)

OCamlCC: Bazel-enabled OCaml Toolchain (new alpha release)

Gregg Reynolds announced

Hi folks. A new alpha version of OCamlCC, the Bazel-enabled version of the OCaml repository, is now available.

This version includes many major changes and improvements. Highlights:

  • All the compilers are buildable under the usual names: ocamlc.byte, ocamlopt.byte, etc. This includes the flambda compilers, whose names use “optx” instead of “opt” (e.g. ocamloptx.optx), and the profiling compilers ocamlcp.byte, etc.
  • Test support. The makefiles use a custom tool, ocamltest, to run tests. This tool is essentially a mini build engine. Since we already have an excellent build engine in Bazel, we do not need ocamltest - all of its functionality is provided by Bazel and a few custom Bazel test rules. A small subset of the tests in testsuite (about 50) have been converted to use Bazel. Bazel’s testing capabilities are very powerful and flexible; for example it is easy to run any test individually, to create custom test suites, and to select tests to run based on tags - e.g run only of the inline expect tests in some set of test directories, or only tests involving integers.
  • The tools (ocamldep, ocamlobjinfo) can be built and run under Bazel’s control, which means arguments to the tools can be expressed as Bazel target labels. This saves the user the problem of finding filesytem paths for the arguments; e.g. you can pass //bytecomp:Bytegen to the ocamlcmt runner, which will automatically configure it to be built with -bin-annot, and will find the correct path for the resulting .cmt file and pass it to the tool. Build labels can also be used with ocamldep; the runner will find the source corresponding file and pass it to the tool.
  • Dependencies are fine-grained. For example, targets that depend on submodules of the standard library depend on them directly; they do not depend on the stdlib archive file.
  • In general, signatures (.mli files) depend only on other signatures, and with a few exceptions “modules” (.ml files) depend only on other modules. All of these dependencies are explicitly expressed, and they are quasi-typed: if you list a module as a dependency of a signature, Bazel will complain.
  • By default, compiler builds do not use archived libraries (e.g. stdlib.cmx?a); this can be configured by a command-line switch.

This version also includes a lot of internal features relevant to maintainers. For example much of the logic involving configuration and preprocessing has been reworked to use mustache templates. This is not strictly-speaking a Bazel thing, it’s more of an experiment in improving and simplifying this logic. It also has implications for portability - the mustache tooling is written in portable C, and it replaces non-portable scripts.

I’ve added quite a bit of documentation, although as you might expect it is neither complete nor polished. I’ve included a good deal of information about how Bazel does certain things.

I’ve tested it on Mac an Linux. It does not require any special configuration, so it should be pretty easy to get started with it.

If you need help: I monitor this list and the OCaml discord server, and I’ve set up an OBazl discord server. You can also file an issue.

PRs are welcome, but since this is still in flux, please file an issue first to make sure your idea is still relevant and needed.

Other OCaml News

Old CWN

If you happen to miss a CWN, you can send me a message and I’ll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.