OCaml Weekly News

This is my first public ocaml package. Please let me know if you have any feedback or advice on how I can improve it.

https://github.com/travisstaloch/combinaml

Its meant to be similar to angstrom except more easily customizable and with a few api differences here and there which i found useful.

I was using angstrom to parse peg grammars and ran into issues i could only solve by adding a tokenization step. This seemed messy so I decided to write this library and found a solution which uses an `until` parser like this:

let definition =
  lift2 pair (ident_str <* leftarrow)
    (until (ident_str <* leftarrow) expression)

let grammar =
  spacing *> many1 definition <* is_end_of_input >>| fun defs -> Grammar defs

It was just merged into ocaml/opam-repository a couple of hours ago and doesn’t seem to show up in opam search yet. I’m sure its a caching issue and will be there soon.

Its up on opam now: https://opam.ocaml.org/packages/combinaml/

We’ve been working on an overall outline for “an ideal state” of the Learn area. I took the current Work-in-progress, and updated the Miro board at https://miro.com/app/board/uXjVM4HlreI=/?share_link_id=39374874841. For many of the topics, we now propose tentative titles of documents relating to them. Some topics are still in need of titles.

Here’s an image of the state at the time I’m making this post:

Note: I have to re-read the previous answers on this thread to make sure this captures the additional topics that have been brought up.

Now… what would be very interesting to me:

1. Among these proposed documents, which ones are the most important to you?
2. Which ones do you see as least important?
3. Is there something you feel is missing? What would be the title of the missing document and where does it belong?

Hi everyone, I’m working on implementing the changes for the new Learn Area and I’m currently looking at the books area. It made me wonder… do you know of any excellent new books that are not yet on https://ocaml.org/books?

Wang and Zhao (the team behind Owl) have an advanced book: https://link.springer.com/book/10.1007/978-1-4842-8853-5

https://caml.inria.fr/pub/docs/oreilly-book/html/index.html is a bit dated but excellent.

The Little MLer is a fantastic introduction to functional programming and, in particular, thinking recursively. It’s in SML but includes a page on translating the examples to OCaml.

I also can’t recommend highly enough The Functional Approach to Programming by Cousineau and Mauny, which uses Caml (no object system yet!). But it clarified many concepts that I hadn’t previously grokked. One of the huge advantages of OCaml is its stability and I think all of the examples still worked.

Maybe not so relevant for an OCaml books page but I also read the classic ML books when I was learning OCaml and they were a HUGE help: Paulson’s ML for the Working Programmer is phenomenal and I also recall Elements of ML Programming and Introduction to Programming using SML being helpful. SML and OCaml are close enough that both concepts and syntax readily transfer from the former to the latter.

Github link This is just a minor release of a pandemic project that I never announced on discuss.ocaml.org. This library is “Complete”. There are no known bugs and no known missing features.

The T-Digest has become fairly well known in the last few years, but in short:

• it’s a lossy data structure that allows the user to (very) accurately approximate percentiles and p_ranks without having to keep the entire sorted dataset in one place.
• the user can combine multiple T-Digests just by concatenating them, and this can be done in the database itself!
• both querying and insertion are blazing fast

facebook/infer has been using it for a few years, and I know it’s also used in a few closed-source projects elsewhere.

All comments and feedback is welcome! I hope this library proves useful to the OCaml ecosystem as a whole.

The Future of OCaml page on OCamlverse needs some love, considering that an issue on GitHub from 2021 is still relevant. Although I have just posted what I am aware of on the GitHub issue, some of you should know even more. The page is the top result when you search “Future of OCaml” on Google, so leaving the page outdated will affect the impression of the language negatively. Shall we improve it?

A new release of OCaml-RDF is available: https://www.good-eris.net/ocaml-rdf/posts/ocaml-rdf-0.15.0.html

This release includes new modules:

• Rdf.Activitystreams defining the activitystreams/activitypub vocabularies,
• Rdf.Nq to read and writes N-quads format.

A new package, rdf_json_ld implements part of the JSON-LD API: context processing, expansion, deserialization to RDF and serialization to Json-ld. Note that serialization produces a flat Json-ld rather than implementing the algorithm from the recommandation. It’s worth noting that the specification seems to have been written by javascript developers with little notion of typing. Moreover, JSON-LD format is far more complicated (and undoubtedly more energy-consuming) than simpler formats such as XML, Turtle or N-quads. I therefore advise against its use (but several activitypub servers seem to communicate only with this format…).

Packages rdf, rdf_ppx, rdf_json_ld, rdf_mysql, rdf_postgresql, and rdf_lwt are avalable in opam.

Welcome to the June 2023 edition of the OCaml.org newsletter! As with the previous update, this has been compiled by @sabine and @tmattio.

The OCaml.org newsletter provides an overview of changes on the OCaml.org website and gives you a glimpse into what has been going on behind the scenes. You can find a list of previous issues here.

Our goal is to make OCaml.org the best resource for anyone who wants to get started and be productive in OCaml. We couldn’t do it without all the amazing OCaml community members who help us review, revise, and create better OCaml documentation. Your feedback enables us to better prioritise our work and make progress towards our goal. Thank you!

We present the work we’ve been doing this month in three sections:

• Learn Area: We’re working towards making OCaml.org a great resource to learn OCaml and discover its ecosystem. This month, we continued working on the wireframes and designs of the new Learn area. We also focused on writing the new documentation with a couple of tutorials on Dune and S-Expressions.
• Governance Page: The OCaml Platform team is working towards making the decision-making processes and ongoing development more transparent and community-driven (including the work on the OCaml Platform roadmap). To support the initiative, we’re working on a governance page that lists the teams and maintainers of the OCaml organisation.
• General Improvements: As usual, we also worked on general maintenance and improvements and we’ve highlighted some of them in this newsletter.

This is an excellent opportunity to thank Ahrefs for their support as they provide a free account for OCaml.org!

The team has been using it numerous times to improve the site’s quality, including broken links, error pages, bad HTML formatting, etc. It’s been invaluable to identify how we can improve the site and if you’ve noticed improvements in the search engine ranking of the site (for instance https://ocaml.org/p/base/v0.15.0 now ranks higher than https://opam.ocaml.org/packages/base/ for me, it wasn’t the case a few months ago), it’s in large part thanks to Ahrefs who gave us the tools to improve SEO.

The goal of the broken link checker is to have something that can be integrated more easily into ocaml.org’s CI. It allows us to have the workflow on display in the last PR: https://github.com/ocaml/ocaml.org/pull/1354

We (I and my co-conspirator Ricardo Rodríguez) are organizing a new edition of our erstwhile one-day hybrid workshop dedicated to the OCaml programming language and its industrial users. We will have talks and discussions with members of the OCaml development team, as well as companies using the language to solve complex and interesting problems.

Once again, the objective is to present a broad picture of the OCaml ecosystem and, more widely, of functional programming (and related techniques) as a viable and powerful choice for building correct and reliable computer systems. This is done as part of the second edition of an OCaml summer school, hosted this year again by the University of Zaragoza, and (again) generously sponsored by the OCaml Software Foundation.

Participation is free and open to everyone. We will stream the workshop on Zoom (https://us06web.zoom.us/j/89373710207?pwd=ZGZuSnBFWEhSc2UzNnpSbWF4d0hzZz09, passcode: 247844) and Twitch. Here’s the preliminary schedule, additional information and updates can be found on our website.

20 July 2023, all times CEST (UTC+2)

Morning – Language session

• 09:00-09:30: Carmen Lazo and José Merseguer (University of Zaragoza) – Welcome reception
• 09:30-10:30: Florian Angeletti (Inria) – The OCaml project and ecosystem
• 10:30-11:00: Coffee break
• 11:00-12:00: OCaml developers – Round table and Q&A

Afternoon – Industry session

• 14:30-14:55: Vincent Balat (Tarides / Be Sport) – Building functional systems / Social network for sports
• 14:55-15:20: Javier Chávarri (Ahrefs) – Petabyte-scale web crawler
• 15:20-15:45: Raphaël Proust (Nomadic Labs) – Tezos blockchain development
• 15:45-16:10: Chris Casinghino (Jane Street) – Large-scale quantitative trading
• 16:10-16:30: Coffee break
• 16:30-17:30: Industrial users – Round table and Q&A

We look forward to seeing you there. Feel free to join, participate and distribute!

A few days ago I posted about the T-Digest library. Today I’m back with another small algorithmic library:

-> Github link

You’re probably familiar with the Levenshtein distance: the number of single character edits (additions, deletions, replacements) between 2 strings.

Calculating the Levenshtein distance is famously more expensive than a simple equality check.

Instead of calculating the distance, this library instead returns whether 2 values are within D distance of each other (bool). There has been substantial development on the topic of Levenshtein automata in the last decade. See the “Fast String Correction with Levenshtein-Automata” paper by Klaus Schulz and Stoyan Mihov.

Using the graph construction technique from the paper, plus a few ideas from this article and several additional optimizations of my own, this library can answer the question (“are these 2 values within D edits of each other”) in the 1-10µs range, scaling linearly with the length of the values.

• a Functor is provided to enable comparisons across any arbitrary types
• string comparisons are provided (functorized) out of the box
• reuse the same automaton across all comparisons with the same max_edits, regardless of the type of the values being compared
• max_edits must be between 0 and 3 (inclusively) due to the astronomical scaling factor during graph building
• most comparisons take under 5 µs, depending on the length of the values

It’s fast enough that it can be used instead of String.equal for some tasks and/or on large datasets.

All comments and feedback are welcome! I hope this library proves useful to the OCaml ecosystem as a whole. I’ll be back in a few days with a special algorithmic library to complete this little trilogy.

Hello :wave: The `ppxlib` July dev meeting is tomorrow Tue, July 18th, at 6pm CET. Here’s what’s on our agenda so far:

• OMP:
  • Do we “stop maintaining it” or do we add OCaml 5.1 support?
• Ppxlib - OCaml trunk compact:
  • Currently, there’s no compatibility due to an ocaml-compiler-libs build problem. Who’s affected?
• Ppxlib’s general maintenance:
  • OCaml 5.1 support: The bug fix around generative functor applications is being worked on.
  • We’re not in a hurry to bump the AST this time.
  • A few pending reviews on Ppxlib. What’s the best strategy for reviews / reacting to non-urgent issues now that we’re in “minimal maintenance mode”?
  • Is there anything else that will come up before September?
• OCaml workshop 2023:
  • Recap on why our talk proposal on Ppxlib has been rejected.
• Outreachy internship on Ppxlib:
  • How is it going? :heart:

We’re always happy to add things, if anyone is interested in anything else.

Thank you @gasche for your interest and input!

Expanding on what @rgrinberg and @rjbou mentioned earlier, there are no intentions to phase out the `opam` client. As a matter of fact, the Dune team has currently forked opam and is patching the opam libraries, with the ultimate goal of merging them back upstream once the libraries exposing the necessary APIs for Dune package management appear to be stable. You can see some of this work in progress in pull requests like #5568, #5508, #5498, #5496, #5452, and so on.

As for the opam repository, you understood correctly that there is absolutely no plan to deprecate it, or even to make large changes to it in the context of Dune package management. The goal is for Dune package management to be 100% compatible with the opam repository.

To expand on a slightly divergent path and talk about the role of the OCaml Platform: the Platform essentially mirrors the state of the world. For the opam client to become deprecated in the Platform, it would need to become the de-facto reality first. While it might be that the opam client (as for any other tool) enters a maintenance mode and eventually becomes deprecated, that seems unlikely for now, given the number of users who are relying on the opam client. That being said, if and when that happens, the Platform’s role will be to make sure that there is a smooth transition path for users, and that’s something that will require careful planning and discussions. All of which is entirely out of scope for the initial release of package management in Dune.

On a different note, following a discussion with @dra27, opam switches are architected around findlib/ocamlfind. Dune package management presents an alternative solution to achieve the same result. As you point out, it’s not meant to be reusable between workflows: the opam packages Dune compiles for your project are intended for Dune’s internal use during its build, not for external use with the shell. This could be viewed as a parallel to how opam builds switches for opam exec (with eval $(opam env) serving as a convenient shortcut). So, you can think of Dune package management as performing a similar function but specifically for dune exec.

Now, there is the question of how to make sure this doesn’t create confusion and hurt adoption. But that’s not something that’s specific to Dune and opam. In fact, if anything else, it makes the Platform more cohesive: odoc, ocamlformat, merlin, utop and mdx are all tools that work well independently, but with which you don’t need to interact if you use Dune. Dune has grown as the frontend of the Platfrom and the integration with opam is another step in this direction, not something very new if we look at what’s being done with the other tools. And this is the best of both worlds: as a power user, you’re free to use each tool independently and you’re not locked in, but as a newcomer or even as a power user who’s happy with the default experience, you can just use Dune.

Reading the opam package management RFC gives me the impression that the situation with the `opam` client is in fact rather different from the situation with other tools, because Dune is in the process of reimplementing large parts of this logic internally, instead of delegating to opam, because you want a tighter integration into Dune internals than a pure-delegation model allows. As far as I know, dune is not reimplementing logic from merlin or odoc or ocamlformat.

I tried to make uninformed guesses at which part of the opam client responsibilities Dune would reuse and replace in my post above. My best guess as to what part you would reuse in the long term (of the opam client responsibilities) is “parsing and programmatic understanding of opam files”. Are there other important ones that I missed?

Another consequence of this design is that the new features which are planned, and are indeed quite nice, will be specific to projects that use dune for package management. The plan is for Dune to provide, for example, good support for incremental rebuilding (when dependencies change), caching (of package artifacts across independent projects), a nice local-switch-first command-line UI with lockfile integration by default, but also editor support (building package dependencies from the IDE directly). None of those features are planned for people using the opam client – if I understand correctly. Some of those features (in particular incremental rebuilding) are clearly in the ballpark of a build system and reimplementing within Dune makes a lot of sense. But for some others, for example the latter three in my list, adding them to the opam client would also have been a possible approach, but you chose to work within Dune instead.

This is also the root of my question on whether the long-term strategy is to keep two tools/codebases alive, or just one. For ocamlformat or odoc, it wouldn’t make sense to ask whether odoc will disappear once dune gets first-class documentation support. For the opam client and package management, it does.

This is also the root of my question on whether the long-term strategy is to keep two tools/codebases alive, or just one.

My longer term view has always been that we should focus on having well-specified file formats that our tools use, and let many domain specific tools that operate over that file metadata bloom. The reason for this is that files that are checked into a project have a habit of sticking around for the long-term (or forever, if you consider historical releases), whereas tools naturally evolve and perish.

The only thing necessary to publish something “into the OCaml community” (that is, something that shows up on a package search on the website) is a tarball with an opam file in it. This opam file specifies interdependencies and a build plan. We have, as of just now, 28296 of these checked into the central opam repository. The vast majority of those packages can be downloaded, extracted, and an installation plan generated simply by looking at the local opam file in the tarball and the central collection of them that represent potential dependencies (the opam repository).

Over the years, we’ve had many build tools spring up: OCamlMakefile, omake, ocamlbuild, oasis, b0, ninja, and jbuilder/dune. What makes dune so interesting from a long-term perspective is that the checked in dune file is also separately versioned, so that it should (with a sufficiently good specification) be possible to analyse the build logic of a repository just by examining it. With most of the other build systems, you needed to run an executable to get a build plan (notably with ocamlbuild, and even with oasis running over ocamlbuild), which tightly couples it to a particular tool. That’s why I’ve been so resistant to the idea of publishing opam packages which do not include a generated opam file (even if its autogenerated from dune), since you then lose the property of simply being able to examine a published artefact to determine how to build it.

What other file formats do we have in common platform tools? We used to have .merlin files, but they’re autogenerated now from a dune build plan in most cases. There are .ocamlformat files, mostly in a k/v format. Generally speaking, we’ve been pretty good at promoting and exposing metadata in an opam or dune file and not having too much of a proliferation of other files.

:wave: deer OCaml aficionados,

Moonpool 0.3 was just released on opam. Moonpool is a new concurrency library for OCaml >= 4.08, with support for OCaml 5 from the get-go. It started out with a thread pool (possibly distributed on multiple domains to be able to use multiple cores) along with a future/promise module.

This release comes with a set of new features on top of pool+futures:

• a small 'a Lock.t abstraction to protect a resource with a lock in RAII-style
• a type of unbounded channels (which are fairly naive in implementation)
• improvements to Pool such as Pool.run_wait_block: (unit -> 'a) -> 'a that runs a whole computation on the pool, and waits for its result (or re-raises)
• add Fut.await (only on OCaml 5)
• add support for domain-local-await if installed
• a Fork_join module for, well, fork-join parallelism, including parallel for and parallel List.map~/~Array.map. These computations can be nested and “feel” like writing code in a direct style. This relies on effects and is only available on OCaml 5.

On behalf of the BINSEC team, I am glad to announce that version 0.8.0 now lives in Opam.

As a short introduction, BINSEC is an open-source program analyzer developed at CEA List to help improve software security at the binary level. It has been successfully applied in a number of security-related contexts, such as vulnerability finding, (malware) deobfuscation, decompilation, formal verification of assembly code or even binary-level formal verification.

More information can be found on the website, including publications, tutorials or contacts, but also the description of this release as well as previous ones.

there’s a new version of the “File Manipulation” tutorial on

https://staging.ocaml.org/docs/file-manipulation

For comparison: the old version of this tutorial is here https://ocaml.org/docs/file-manipulation.

https://github.com/ocaml/ocaml.org/pull/1400

Thanks for taking a look and giving feedback and suggestions for revising this! :)

I’m happy to announce the release of Mutaml 0.1, a mutation testing tool for OCaml: https://github.com/jmid/mutaml

Mutaml attempts to make small random changes your code, e.g., turning e+1 into e to see if the off-by-one change is caught by your test suite. By finding examples of uncaught wrong behaviour, it can thereby reveal limitations of a test suite and indirectly suggest improvements.

Overall 0.1 is considered an initial working prototype.

@raphael-proust previously blogged about how he used it while developing Seqes.

Acknowledgements

Mutaml was developed with support from the OCaml Software Foundation. While developing it, I also benefitted from studying the nice source code of @antron’s bisect_ppx.

Welcome to the third instalment of the OCaml Platform newsletter!

This edition brings the latest improvements made in June to enhance the OCaml developer experience with the OCaml Platform. As in the previous updates, the newsletter features the development workflow currently being explored or enhanced.

The month’s standout highlight is undoubtedly the first alpha release of opam 2.2! Years in the making (opam 2.1 was released almost two years ago), the significance of the hard work put in by the opam team can’t be overstated. Much appreciation goes out to the opam team (Raja Boujbel, David Allsopp, Kate Deplaix, Louis Gesbert, in a united OCamlPro/Tarides collaboration), and especially to Raja Boujbel for diligently pushing the work to completion in order to achieve this alpha. The announcement holds more details, and we encourage you to provide feedback on the Discuss post.

• Releases
• Building Packages
  • Dune Exploring Package Management in Dune
  • opam Native Support for Windows in opam 2.2
  • Dune Improving Dune’s Documentation
  • Dune New dune show command
• Generating Documentation
  • odoc Add Search Capabilities to odoc
• Editing and Refactoring Code
  • Merlin Support for Project-Wide References in Merlin
  • Merlin Improving Merlin’s Performance
  • OCaml LSP Upstreaming OCaml LSP’s Fork of Merlin
  • OCaml LSP Extract code actions
  • OCaml LSP Support for Inlay Hints
• Formatting Code
  • OCamlFormat Closing the Gap Between OCamlFormat and ocp-indent