OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of August 06 to 13, 2019.

Table of Contents

OCaml release 4.08.1

Florian Angeletti announced

We have the pleasure of celebrating the birthday of Alexander Fleming by announcing the release of OCaml version 4.08.1.

This is a bug-fix release, which fixes compilation failures in presence of "-pack" (breaking nocrypto), and dynlinking failures (breaking Frama-C) (see also the list of changes below).

It is (or soon will be) available as a set of OPAM switches, and as a source download here:
https://caml.inria.fr/pub/distrib/ocaml-$BRANCH/

Happy hacking,

— Florian Angeletti, for the OCaml team.

OCaml 4.08.1:

  • Bug fixes:
    • #7887: ensure frame table is 8-aligned on ARM64 and PPC64 (Xavier Leroy, report by Mark Hayden, review by Mark Shinwell and Gabriel Scherer)
    • #8751: fix bug that could result in misaligned data section when compiling to native-code on amd64. (observed with the mingw64 compiler) (Nicolás Ojeda Bär, review by David Allsopp)
    • #8769, #8770: Fix assertion failure with -pack (Leo White, review by Gabriel Scherer, report by Fabian @copy)
    • #8816, #8818: fix loading of packed modules with Dynlink (regression in #2176). (Leo White, report by Andre Maroneze, review by Gabriel Scherer)
    • #8830: configure script: fix tool prefix detection and Debian's armhf detection (Stéphane Glondu, review by David Allsopp)
    • #8843, #8841: fix use of off_t on 32-bit systems. (Stephen Dolan, report by Richard Jones, review by Xavier Leroy)

First release of lwt-pipeline

Raphaël Proust announced

On behalf of Nomadic Labs, I'm pleased to announce the first release of lwt-pipeline. Lwt-pipeline started as an internal library for the Tezos project to order-preservingly process lists of items with some Lwt.t and Pervasives.result thrown in.

The library is available through opam: opam install lwt-pipeline. The project is hosted at https://gitlab.com/nomadic-labs/lwt_pipeline. The code is available under MIT license.

We'll release further versions of this library as well as other internal libraries and tools. Consider this release a small first step and watch out for other, bigger releases.

Halfway-usable Lua-ML

Daniil Baturin announced

Lua-ML is a pure OCaml, heavily functorized implementation of Lua 2.5. It's possible to extend it with your own libraries or replace its own libraries.

Originally it was a part of the now defunct C– project, then Christian Lindig rescued it from it and packaged as a standalone library, but he didn't have much time to work on it. Recently he passed the project to me, so I'm now its maintainer, for better or worse.

Thanks to a patch by @Drup it can now be installed as an OPAM package, and I've made an effort to remove the dependency on lipsum and eliminate the fork of an old Hashtbl from its source tree.

From my quick tests it appears to work as expected, but since it never had real users, it needs as much testing as it can get. And a test suite. And features of Lua newer than the bronze age 2.5. And code clean-up. But the first steps towards reviving it have been made at least.

Perry E. Metzger asked

Are there any neat things you can do with it? Like can Lua code call into OCaml code?

Daniil Baturin replied

@perry I should have given more information in the announce indeed.

Yes, you can expose any OCaml values as Lua userdata and any functions as fallbacks. Here is my quick test of exposing lambdasoup to Lua:

utop # let st = I.mk () ;;
val st : I.state =
  {I.Value.globals = <abstr>; fallbacks = <abstr>; callstack = [];
   currentloc = None; startup = <abstr>}

utop # I.register_globals ["soup", lua_of_soup (Html.SoupNode (Soup.parse "<p>foo</p> <p>bar</p>"))] st;;
- : unit = ()

utop # I.dostring st "x = HTML.select(\"p\", soup); k, v = next(x); while k do print(v); k, v = next(x, k) end";;
<p>foo</p>
<p>bar</p>
2
- : I.value list = [I.Value.LuaValueBase.String "executed without errors"]

The module for that took quite a bit of yak shaving to get right, and there may be a better way around the phantom types than that artificial sum type, but here it is: https://gitlab.com/snippets/1883747

Daniel Bünzli said

I don't know if it's still accurate but this paper specifically talks about this.

Marek Kubica asked

It is cool to see more Lua implementations, though I am a bit surprised that the version is 2.5 and not 5.1, the version that the community has seemingly decided to ossify on. How hard or easy would it be to update it to a newer version?

Christian Lindig replied

I would expect this to be a lot of work. The motivation for providing Lua as an extension language is spelled out in the README: to make your OCaml application more flexible. The motivation was not to consume all the existing Lua code - which indeed would require a Lua 5.x implementation. I believe Lua 2.5 still serves the original purpose well enough.

If one wanted Lua 5.x integration, it would be easier to embed a Lua 5.x C implementation into OCaml at the cost of a more complicated interface.

Daniil Baturin then said

Lack of for loops is quite annoying to modern Lua users, so I'm planning to add it. Other than that, I also think for the intended use case, compatibility with Lua libraries isn't much of a concern.

What makes it worse is that after Hashtbl fork removal, the next() function is O(n), so any iteration is quadratic, and I see no straightforward way to memoize it. Probably not that much of an issue for small data on modern machines, but it's far from nice. A built-in for that calls Hashtbl.iter directly would alleviate that problem.

soupault: a static website generator based on HTML rewriting

Daniil Baturin announced

I've made a 1.2 release, now with Lua plugin support thanks to Lua-ML: https://baturin.org/projects/soupault/#plugins

OpenAPI Generator now supports OCaml client generator

William Cheng announced

UPDATE: The new OCaml client generator has been included in the OpenAPI Generator v4.1.0 release: https://twitter.com/oas_generator/status/1160000504455319553

The end of Camlp4

Jérémie Dimino announced

Dear community,

As you may know, the Camlp4 project has been relatively inactive for the past few years. With the help of @gasche, @dra27 and a few other contributors, we have been keeping the project alive so that existing OCaml projects using Camlp4 could continue to build with new OCaml compilers.

At this point, we expect that the most used features of Camlp4 are covered by ppx, and in particular all the effort is focused towards the latter.

We feel like the time has come to officially abandon Camlp4. In order to help distributions and other package managers deal with this fact, we will soon release a 4.08 compatible version of Camlp4. This will in particular help getting OCaml 4.08 in Debian. However, this will be the very last release of Camlp4. After that, the project will be abandonned and will no longer receive updates.

Of course, anyone interested in taking over the project is very welcome to do so. Please get in touch if you are interested and we will happily arrange for the transfer of ownership.

Migrating away from Camlp4

There are a lot of code bases out there still using Camlp4. If you need to bring an old project using Camlp4 to the modern age, I recommend reading this blog post which describes in details how the huge Jane Street code base was migrated from Camlp4 to ppx. In particular, the camlp4-to-ppx tool mentioned in the blog post is availabe on github.

Stream parsers

One of the pain point of migrating away from Camlp4 is the stream parser syntax. In the past, the OCaml compiler used to have a syntax for stream parsers. When Camlp4 was merged into OCaml, the stream parser syntax moved from the OCaml parser to Camlp4.

Nowadays, Camlp4 is the only tool that understands the stream parser syntax. Stream parsers are not considered to be a great design and we encourage users to explore alternatives such as lazy lists or parser combinator libraries.

However, may you need to upgrade a large code base using Camlp4 and stream parsers, it would be quite easy to develop a ppx syntax for stream parsers and automatically upgrade the syntax using camlp4-to-ppx. If you would like to explore this possibility and need pointers, do not hesitate to get in touch!

Ivan Gotovchits said

There is one more dependency, which is pretty popular and even serves as an entry point to the language for many students. It's the LLVM OCaml Kaleidoscope set of tutorials, which heavily relies on the stream syntax. I also believe that there are some courses in the wild that use the stream syntax.

Also, what about camlp5 is it also no longer supported?

Jérémie Dimino replied

AFAIK, camlp5 is still maintained by its original author.

Regarding the stream syntax, we discussed it on the OCaml developer mailing list and nobody think streams and stream parsers are a great design. Given that, it seems better not to use them in tutorials and courses. Do you know who maintains the LLVM OCaml Kaleidoscope? Maybe we could reach out to them to let them know about this.

Anton Kochkov said

Lately I prefer writing parsers with Anstrom. It offers a lot for writing easy to understand, still performant parsers: https://github.com/inhabitedtype/angstrom

Anton Kochkov also said

I exported the published packages list that still rely on the camlp4 (I removed all pa_* packages as obsolete, and some of the packages that dropped camlp4 already, but not yet released, like coq or cocinelle):

So it makes sense to migrate the packages either to ppx, or to menhir/ocamllex/angstrom/whatever. Or, in the worst case - to camlp5. And if the project is dead - set the upper limit of the OCaml version to 4.08 in the opam file.

gasche said

Keeping Camlp4 compatible with upcoming OCaml releases is not actually that much work. If you – or someone else – are interested in having a look, I personally think that it's not more than a day of work per release of OCaml. (In contrast, fixing bugs within Camlp4 requires more expertise and can be painful, but that's not required for keeping existing project compatible.)

The problem we have, collectively, is a that a fairly small set of people ended up maintaing a large portion of the building blocks of the OCaml ecosystem (Camlp4, ppx stuff, etc.), while mostly not using them themselves. It's okay for a while, because it helps other people so we do it out of good will, but it gets old, especially if you have several "less than a day of work" projects to update on a regular basis. (In the specific case of Camlp4, Jérémie was maintaining it because Jane Street used it, but they moved away in 2015 and it's been fan service since.)

If people outside that small set stepped up to maintain Camlp4, we wouldn't need to make noise around end of life, deprecating projects that use it, the future of stream parsers, etc. But for now they don't. (I'm not exactly sure why.) I think that some of the actions discussed here or elsewhere are fruitful (using lex+menhir in the LLVM tutorial: excellent idea), and some are probably a waste of effort compared to just keeping Camlp4 on life support, but it's a bit hard to tell which is which.

Ivan Gotovchits announced

My initial plan was not to volunteer myself and take it unless nobody will take it before the 4.09 release. But I suspect, that anyway it will be me :) So just to close this discussion, I will take the burden to keep camlp4 alive for 2 or more next releases. Then, we will probably just archive it. My github handle is ivg.

Jérémie Dimino replied

Initially, I imagined that someone might want to step up because they like the technology itself or because they need it for their day to day development. In this case it would have made sense to transfer the project to them. In the end, we are only talking about keeping Camlp4 alive for longer, so we can leave it where it is for now.

Before we go ahead, there is one aspect that hasn't been discussed here; what is the exact value of keeping Camlp4 alive? When modernising an older project, it's unlikely that Camlp4 will be the only blocker. There are many other things that have changed in the past few years. Strings are now immutable for instance. The dependencies of a project might have changed as well. There are many reasons other than Camlp4 why bringing a project up to date might require additional work.

All in all, I'm wondering if doing new releases of Camlp4 will really help anything, or at least it seems to me that we should wait for feedback from the 4.08 upgrades in the various systems. For instance, now that we have a 4.08 Camlp4 in opam, how many packages using Camlp4 are compatible with 4.08 out of the box?

On the other hand, keeping the project alive just to keep it alive for an unbounded period of time can have a negative impact. It's less incentive to migrate away from it and we might miss out cases that we would have discovered by really stopping Camlp4. Delaying the 4.08 release of Camlp4 was a good example of how it triggered discussions and a few projects to migrate to ppx.

What do you think?

Ivan Gotovchits said

I would suggest to look at the problem from a very different perspective. From a user perspective. I'm sorry for a long narration, I didn't have time to explain this in a more concise manner.

User story #1

Consider an industrial user who has developed a solution in OCaml and is planning to deliver it for a few years. During the initial stages of the software product lifecycle, the project manager benefited from abundance of coders and the project was moving up-to-date with the OCaml infrastructure. Then the project was released, majority (if not all) coders left, and the project entered the support stage. At this stage users still expect bugs to be fixed, but the project manager no longer has the man-power to move the project as fast as the OCaml infrastructure moves.

Since the project was heavily using external dependencies (to minimize the cost of development), it occurs that a few of the bugs, discovered at the later stages of the project lifetime, actually belong to other projects, such as Batteries or Core. Those bugs were fixed, of course, in the upstream, but it requires updating to the newest version of OCaml (which no longer provides camlp4, and all the libraries are newer, so the effort is much higher than a manager can afford).

So from the perspective of this project manager, OCaml moves too fast and is not really caring about its users. What the project manager wants? The support cycle of at least 5 years, where the end-of-life announcement of a feature is issued no less than three years before its end of life.

User story #2

In this story we have a PhD student, actually two of them. One has written an excellent paper and even submitted an artifact with the paper, so that everyone can compile and reuse it, and compare with own results. The paper was submitted and accepted, and even got a reward. And the student moved forward and even graduated and now has lots of other stuff to do.

Another student, a couple of years later is developing a competitive solution and they want to at least repeat the results of the older paper using the provided artifact. Unfortunately, since the OCaml community moves so fast and doesn't provide long term support, the artifact is no longer installable. Since the younger student is very new to OCaml, they hasn't enough capability to revive the project, so they leave it very frustrated by the existing work (heavily suspecting that the results were forged), they are also very frustrated with OCaml (since it doesn't work, in their opinion) and although they initially thought to learn it, the abandon it in favor of an inferior but a much more stable language. They end up spending more time on implementation, getting inferior results and as a result, both the OCaml and the science community suffers.

Conclusions

The following, as well as anything else that I say, is my personal opinion. Right now, OCaml moves really fast, which is good, but we are burning the bridges behind us, leaving those don't match up with our speed behind without any option to catch up again. Deprecation of Pervasives, dropping Bigarray.*.map_file, dropping camlp4, making strings immutable, stopping opam 1.x repositories, switching to Dune everywhere, adding/removing compiler plugins, camlp5->camp4->ppx_driver->ppxlib story, and many more changes which are definitely good for the overall health of OCaml still hurt those who do not have enough manpower to adapt to those changes.

And although as an active OCaml programmer I'm happy with all those changes, as an active project manager and research scientist I'm a little bit frustrated. The half-life period of an average OCaml project is about half an year. It basically means, that most of the projects bit rot in a year or so. And together with those projects the knowledge and other assets invested into this project, decay as well. Which actually renders OCaml less usable for real world project.

Suggestions

I understand the burden associated with the support of older versions of any non-trivial project. But at least, we should try not to break things that we can leave intact. And we also need a much more explicit deprecation procedure, with proper announcements of the end of life (which do not coincide with the actual end-of-life of the feature, as it happened with camlp4, where it was deprecated and killed basically at the same time).

Besides, another good example, is the map_file deprecation, which was deprecated for some time with no clear end-of-life date and no clear mitigation path, until it suddenly was removed (yes the mitigation in the form of mirage/mmap library was added but without any announcements or references to it, or updated documentation (which still mentions a non-existed bigarray library) and, most importantly, all this mitigation was added not before, but at the same time as the PR which removes it was accepted). All these stories do not contribute well to the stability story of OCaml, to a story where OCaml is a serious industrial language, not a playground for academia.

On the OCaml list, Richard W.M. Jones said

We killed off camlp4 in Fedora 31:

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/G2JBIWB423ECYGBXZ3QCPR7NQ66XGXTU/

ocaml-gettext was the one which affected most packages, but I have ported it to use Ast from compiler-libs.

Another dependency I didn't expect was LablGL. What happened to that? A number of games depend on it, and it depends on camlp4 in what seems to be a fairly fundamental way.

Stéphane Glondu replied

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.