OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of September 17 to 24, 2019.

Table of Contents

Interesting OCaml Articles

Archive: https://discuss.ocaml.org/t/interesting-ocaml-articles/1867/54

Ryan Slade announced

I thought this might be of interest since AFAIK the backend is implemented in OCaml: https://darklang.com/

How does one print any type?

Archive: https://discuss.ocaml.org/t/how-does-one-print-any-type/4362/7

Continuing this thread, rixed said

As an additional piece of information, Batteries has to be mentioned. Batteries is a superset of the standard library which, amongst other things, offers composable printers for every types, so that it is often not necessary to devise custom printers for your own types.

For instance, if you had this type: 

type t = (string, (float * int) list) Hashtbl.t

then a printer for this would be: 

let print output_chan value =
  Hashtbl.print String.print (List.print (Tuple2 Float.print Int.print)) output_chan value

Basically the kind of code that the ppx_deriving extension would generate, with the advantage that you can more easily substitute part of it with some custom printers of your own.

Marek Kubica also said

If you just want composable printers, there is the Fmt library by @dbuenzli.

OCamlPro cheat-sheets updates

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2019-09/msg00014.html

Thomas Blanc announced

It is our pleasure to announce the release of our up-to-date cheat-sheets for the OCaml language and standard library. Our original cheat-sheets were published in 2011, and as you know there has been quite a few changes in the language, library and ecosystem overall. See http://www.ocamlpro.com/2019/09/13/updated-cheat-sheets-ocaml-language-and-ocaml-standard-library/ for details.

Links:

Comments and contributions on Github are welcome. Special thanks to Hannes Mehnert for his lightning fast bug-reporting and Fourchaux for his keen spelling eye.

As you can see, our Github repository contains not-yet-updated cheat-sheets. Stay tuned for coming updates! Obviously, our opam cheat-sheet is already in writing by the greatest opam experts you can find. We also wish to make sheets for Lwt, Core, Dune, ppx - any suggestions are welcome.

Happy hacking!

Thomas Blanc
OCamlPRO SAS
PS: if you're curious about what we've been doing lately, check out our blogpotst:
http://www.ocamlpro.com/2019/08/30/ocamlpros-compiler-team-work-update/
http://www.ocamlpro.com/2019/07/11/opam-2-0-5-release/
http://www.ocamlpro.com/2019/07/09/alt-ergo-participation-to-the-smt-comp-2019/

OCaml 4.09.0 released

Archive: https://discuss.ocaml.org/t/ocaml-4-09-0-released/4384/1

octachron announced

The OCaml team have the pleasure of celebrating the birth of James Waddell Alexander II by announcing the release of OCaml version 4.09.0 .

Some of the highlights of this new 4.09 release are:

  • New optimisations, in particular for affine functions in matches, for instance 

    type t = A | B | C
let affine = function
  | A -> 4
  | B -> 3
  | C -> 2
  • The graphics library was moved out of the compiler distribution.
  • The vmthread library was removed.
  • Support for compiler plugins was removed.
  • Many bug fixes.

The full list of change can be found in the changelog below.

It is available as a set of OPAM switches, and as a source download here:

https://caml.inria.fr/pub/distrib/ocaml-4.09/

Happy hacking.

Changes

This is the changelog.

  • Runtime system:
    • (breaking change) #1725, #2279: Deprecate Obj.set_tag and Obj.truncate (Stephen Dolan, review by Gabriel Scherer, Damien Doligez and Xavier Leroy)
    • (breaking change) #2240: Constify "identifier" in struct custom_operations (Cedric Cellier, review by Xavier Leroy)
    • (breaking change) #2293: Constify "caml_named_value" (Stephen Dolan, review by Xavier Leroy)
    • #8787, #8788: avoid integer overflow in caml_output_value_to_bytes (Jeremy Yallop, report by Marcello Seri)
    • #2075, #7729: rename _T macro used to support Unicode in the (Windows) runtime in order to avoid compiler warning (Nicolás Ojeda Bär, review by Gabriel Scherer and David Allsopp)
    • #2250: Remove extra integer sign-extension in compare functions (Stefan Muenzel, review by Xavier Leroy)
    • #8607: Remove obsolete macros for pre-2002 MSVC support (Stephen Dolan, review by Nicolás Ojeda Bär and David Allsopp)
    • #8656: Fix a bug in [caml_modify_generational_global_root] (Jacques-Henri Jourdan, review by Gabriel Scherer)
  • Standard library:
    • #2262: take precision (.<n>) and flags ('+' and ' ') into account in printf %F (Pierre Roux, review by Gabriel Scherer)
    • #6148, #8596: optimize some buffer operations (Damien Doligez, reports by John Whitington and Alain Frisch, review by Jeremy Yallop and Gabriel Scherer)
  • Other libraries:
    • (breaking change) #2318: Delete the graphics library. This library is now available as a separate "graphics" package in opam. Its new home is: https://github.com/ocaml/graphics (Jérémie Dimino, review by Nicolas Ojeda Bar, Xavier Leroy and Sébastien Hinderer)
    • (breaking change) #2289: Delete the vmthreads library. This library was deprecated in 4.08.0. (Jérémie Dimino)
    • #2112: Fix Thread.yield unfairness with busy threads yielding to each other. (Andrew Hunter, review by Jacques-Henri Jourdan, Spiros Eliopoulos, Stephen Weeks, & Mark Shinwell)
    • #7903, #2306: Make Thread.delay interruptible by signals again (Xavier Leroy, review by Jacques-Henri Jourdan and Edwin Török)
    • #2248: Unix alloc_sockaddr: Fix read of uninitialized memory for an unbound Unix socket. Add support for receiving abstract (Linux) socket paths. (Tim Cuthbertson, review by Sébastien Hinderer and Jérémie Dimino)
  • Compiler user-interface and warnings:
    • (breaking change) #2276: Remove support for compiler plugins and hooks (also adds [Dynlink.unsafe_get_global_value]) (Mark Shinwell, Xavier Clerc, review by Nicolás Ojeda Bär, Florian Angeletti, David Allsopp and Xavier Leroy)
    • #2301: Hint on type error on int literal (Jules Aguillon, review by Nicolás Ojeda Bär , Florian Angeletti, Gabriel Scherer and Armaël Guéneau)
    • (breaking change) #2314: Remove support for gprof profiling. (Mark Shinwell, review by Xavier Clerc and Stephen Dolan)
    • #2190: fix pretty printing (using Pprintast) of "lazy …" patterns and "fun (type t) -> …" expressions. (Nicolás Ojeda Bär, review by Gabriel Scherer)
    • #2277: Use newtype names as type variable names The inferred type of (fun (type t) (x : t) -> x) is now printed as ('t -> 't) rather than ('a -> 'a). (Matthew Ryan)
    • #2309: New options -with-runtime and -without-runtime in ocamlopt/ocamlc that control the inclusion of the runtime system in the generated program. (Lucas Pluvinage, review by Daniel Bünzli, Damien Doligez, David Allsopp and Florian Angeletti)
    • #3819, #8546 more explanations and tests for illegal permutation (Florian Angeletti, review by Gabriel Scherer)
    • #8537: fix the -runtime-variant option for bytecode (Damien Doligez, review by David Allsopp)
    • #8541: Correctly print multi-lines locations (Louis Roché, review by Gabriel Scherer)
    • #8579: Better error message for private constructors of an extensible variant type (Guillaume Bury, review by many fine eyes)
  • Code generation and optimizations:
    • #2278: Remove native code generation support for 32-bit Intel macOS, iOS and other Darwin targets. (Mark Shinwell, review by Nicolas Ojeda Bar and Xavier Leroy)
    • #8547: Optimize matches that are an affine function of the input. (Stefan Muenzel, review by Alain Frisch, Gabriel Scherer)
    • #1904, #7931: Add FreeBSD/aarch64 support (Greg V, review by Sébastien Hinderer, Stephen Dolan, Damien Doligez and Xavier Leroy)
    • #8507: Shorten symbol names of anonymous functions in Flambda mode (the directory portions are now hidden) (Mark Shinwell, review by Nicolás Ojeda Bär)
    • #8681, #8699, #8712: Fix code generation with nested let rec of functions. (Stephen Dolan, Leo White, Gabriel Scherer and Pierre Chambart, review by Gabriel Scherer, reports by Alexey Solovyev and Jonathan French)
  • Manual and documentation:
    • #7584, #8538: Document .cmt* files in the "overview" of ocaml{c,opt} (Oxana Kostikova, rewiew by Florian Angeletti)
    • #8757: Rename Pervasives to Stdlib in core library documentation. (Ian Zimmerman, review by David Allsopp)
    • #8515: manual, precise constraints on reexported types (Florian Angeletti, review by Gabriel Scherer)
  • Tools:
    • #2221: ocamldep will now correctly allow a .ml file in an include directory that appears first in the search order to shadow a .mli appearing in a later include directory. (Nicolás Ojeda Bär, review by Florian Angeletti)
  • Internal/compiler-libs changes:
    • #1579: Add a separate types for clambda primitives (Pierre Chambart, review by Vincent Laviron and Mark Shinwell)
    • #1965: remove loop constructors in Cmm and Mach (Vincent Laviron)
    • #1973: fix compilation of catches with multiple handlers (Vincent Laviron)
    • #2228, #8545: refactoring the handling of .cmi files by moving the logic from Env to a new module Persistent_env (Gabriel Scherer, review by Jérémie Dimino and Thomas Refis)
    • #2229: Env: remove prefix_idents cache (Thomas Refis, review by Frédéric Bour and Gabriel Scherer)
    • #2237, #8582: Reorder linearisation of Trywith to avoid a call instruction (Vincent Laviron and Greta Yorsh, additional review by Mark Shinwell; fix in #8582 by Mark Shinwell, Xavier Leroy and Anil Madhavapeddy)
    • #2265: Add bytecomp/opcodes.mli (Mark Shinwell, review by Nicolas Ojeda Bar)
    • #2268: Improve packing mechanism used for building compilerlibs modules into the Dynlink libraries (Mark Shinwell, Stephen Dolan, review by David Allsopp)
    • #2280: Don't make more Clambda constants after starting Cmmgen (Mark Shinwell, review by Vincent Laviron)
    • #2281: Move some middle-end files around (Mark Shinwell, review by Pierre Chambart and Vincent Laviron)
    • #2283: Add [is_prefix] and [find_and_chop_longest_common_prefix] to [Misc.Stdlib.List] (Mark Shinwell, review by Alain Frisch and Stephen Dolan)
    • #2284: Add various utility functions to [Misc] and remove functions from [Misc.Stdlib.Option] that are now in [Stdlib.Option] (Mark Shinwell, review by Thomas Refis)
    • #2286: Functorise [Consistbl] (Mark Shinwell, review by Gabriel Radanne)
    • #2291: Add [Compute_ranges] pass (Mark Shinwell, review by Vincent Laviron)
    • #2292: Add [Proc.frame_required] and [Proc.prologue_required]. Move tail recursion label creation to [Linearize]. Correctly position [Lprologue] relative to [Iname_for_debugger] operations. (Mark Shinwell, review by Vincent Laviron)
    • #2308: More debugging information on [Cmm] terms (Mark Shinwell, review by Stephen Dolan)
    • #7878, #8542: Replaced TypedtreeIter with tast_iterator (Isaac "Izzy" Avram, review by Gabriel Scherer and Nicolás Ojeda Bär)
    • #8598: Replace "not is_nonexpansive" by "maybe_expansive". (Thomas Refis, review by David Allsopp, Florian Angeletti, Gabriel Radanne, Gabriel Scherer and Xavier Leroy)
  • Compiler distribution build system:
    • #2267: merge generation of header programs, also fixing parallel build on Cygwin. (David Allsopp, review by Sébastien Hinderer)
    • #8514: Use boot/ocamlc.opt for building, if available. (Stephen Dolan, review by Gabriel Scherer)
  • Bug fixes:
    • #8864, #8865: Fix native compilation of left shift by (word_size - 1) (Vincent Laviron, report by Murilo Giacometti Rocha, review by Xavier Leroy)
    • #2296: Fix parsing of hexadecimal floats with underscores in the exponent. (Hugo Heuzard and Xavier Leroy, review by Gabriel Scherer)
    • #8800: Fix soundness bug in extension constructor inclusion (Leo White, review by Jacques Garrigue)
    • #8848: Fix x86 stack probe CFI information in caml_c_call and caml_call_gc (Tom Kelly, review by Xavier Leroy)
    • #7156, #8594: make top level use custom printers if they are available (Andrew Litteken, report by Martin Jambon, review by Nicolás Ojeda Bär, Thomas Refis, Armaël Guéneau, Gabriel Scherer, David Allsopp)
    • #3249: ocamlmklib should reject .cmxa files (Xavier Leroy)
    • #7937, #2287: fix uncaught Unify exception when looking for type declaration (Florian Angeletti, review by Jacques Garrigue)
    • #8610, #8613: toplevel printing, consistent deduplicated name for types (Florian Angeletti, review by Thomas Refis and Gabriel Scherer, reported by Xavier Clerc)
    • #8635, #8636: Fix a bad side-effect of the -allow-approx option of ocamldep. It used to turn some errors into successes (Jérémie Dimino)
    • #8701, #8725: Variance of constrained parameters causes principality issues (Jacques Garrigue, report by Leo White, review by Gabriel Scherer)
    • #8777(partial): fix position information in some polymorphic variant error messages about missing tags (Florian Angeletti, review by Thomas Refis)
    • #8779, more cautious variance computation to avoid missing cmis (Florian Angeletti, report by Antonio Nuno Monteiro, review by Leo White)
    • #8810: Env.lookup_module: don't allow creating loops (Thomas Refis, report by Leo White, review by Jacques Garrigue)
    • #8862, #8871: subst: preserve scopes (Thomas Refis, report by Leo White, review by Jacques Garrigue)
    • #8921, #8924: Fix stack overflow with Flambda (Vincent Laviron, review by Pierre Chambart and Leo White, report by Aleksandr Kuzmenko)
    • #8944: Fix "open struct .. end" on clambda backend (Thomas Refis, review by Leo White, report by Damon Wang and Mark Shinwell)

OCaml Training 5-6 and 7-8 November 2019 at OCamlPro

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2019-09/msg00020.html

Muriel SHAN SEI FAN announced

In our endeavour to encourage professional programmers to understand and use OCaml, OCamlPro will be giving 2 training sessions, in French, in our Paris offices:

  1. OCaml Beginner course for professional programmers (5-6 Nov)
  2. OCaml Expertise (7-8 Nov).

The "Expert" OCaml course is for already experienced OCaml programmers to better understand advanced type system possibilities (objects, GADTs), discover GC internals, write "compiler-optimizable" code. These sessions are also an opportunity to come discuss with our OPAM & Flambda lead developers and core contributors in Paris. Training in English can also be organized, on-demand. Do not hesitate pass the word.

Links:

Cheers, Muriel
PS: Note that this complements the excellent OCaml MOOC from Université Paris-Diderot and the learn-OCaml platform of the OCaml Software Foundation.

Arch Linux installer written in OCaml

Archive: https://discuss.ocaml.org/t/arch-linux-installer-written-in-ocaml/4388/1

Darren announced

I'm sharing the WIP project on the off chance it's interesting/useful to someone.

What

This installer written in OCaml allows installing Arch Linux with optional disk encryption (where all key files placements, boot parameters adjustment etc are adjusted for you). The installer aims to be somewhat smart and hassle free to use.

Right now

  • It recognises whehter your Arch Linux live CD is running in BIOS or EFI mode, and adjusts prompts/partitioning accordingly (not technically difficult at all, but just so you know it does that)
  • Handles optional disk encryption, with encrypted boot (haven't added code for toggling boot partition encryption yet)
  • Allows two disk layout schemes atm
    1. use a single disk, installer does partitioning for you
    2. you tell installer partitions for /, /boot and /efi if in EFO mode, and installer use those info instead

WIP

  • Installation of SaltStack files (for package management) and other helper scripts
    • Section below gives more context on these things

TODO

  • Resumable installation (add logging etc)
  • Static config file support (so installation is fully automated)
  • Add disk layout scheme where it's system partition + USB key
  • Whatever is missing from the original script

Context

This installer is essentially a rewrite of the install script I've been using for a while.

I especially needed an semi-automated solution to installing Arch Linux as I was using a encrypted USB key + encrypted system partition setup, where manual installation is error prone and time consuming.

The install script was thus born, written in bash, accompanied by salt files for package management, and various helper scripts for easier handling of the USB key in day-to-day operations.

This is all fine and dandy until I wanted to have different disk layout, and pick whether I want encryption or not. Bash script is not terribly good for these somewhat decision/logic heavy tasks for obvious reasons.

And so rewrite in OCaml began, and stuck with OCaml even after considering various other choices (e.g. elvish, xonsh, fish, python, racket).

Intention

The code aims to facilitate customisation (make a copy and adjust things your way) by providing a sane starting point, but does not aim to provide a proper "library" type of code base due to the somewhat volatile nature of Arch, and assumptions made may not be directly portable to other distros.

Navigation

  • scripts/
    • setup.sh is the original bash installer script, which this installer is based on
    • remaining files are helpers, and misc scripts
  • saltstack/
    • SaltStack files for package management
  • installer_ml/ contains the OCaml installer code
    • src/
      • installer.ml somewhat show cases the intended high level use, the install workflow is specified here
      • disk_layout.ml handles formatting, LUKS setup, mounting, unmounting of partitions etc
      • task_book.ml facilitates high level task registration, used in installer.ml
      • remainders are helper code largely

TL;DR

I wrote a Arch installer in OCaml, thought it's pretty neat.

gasche asked and Darren replied

> Sounds very cool!

Thanks!

> Do you think there is potential for adoption within the Arch community?

Yes, but definitely not widespread, there are two main reasons.

For people who like to build their system programmatically

The language choice is very niche to begin with - OCaml is definitely not a popular choice (if is one to begin with) for system administration.

For initial bootstrapping, most other langs are better choices: rich stdlib is desirable for the initial phase, good handling of interprocess communication, easy access to regex (maybe not as efficient efficient), easy text file manipulation (a lot of formats supported in stdlib or ecosystem etc).

So overall Python, Ruby, Racket are better for this phase. Especially since there are system administration frameworks in Python (e.g. Fabric) and in Ruby (the name escaped me). I stuck with OCaml since my brain can't process more than a hundred lines of code in dynamically typed languages.

So the targeted audience is quite limited - one that knows OCaml, the distro's native installer doesn't work well enough and thus interested in building their own installer, and does not already have their favourite installer.

Arch overall promotes customisation

So most people have their own installers. This is especially true when Arch deprecated the official installer some years ago.

So sticking to mine as is probably doesn't make too much sense to most people. The intention was definitely allowing other people to use it as a template for making their own OCaml installer to begin with anyway.

> What are other people using, and could you convince them to migrate to your installer (and help maintain it)?

Some of the reasons mentioned in above section carry over.

Due to Arch's philosophy on customisation, actively convincing others to use my installer would be close to convincing someone to use your favourite editor and abandon theirs.

In any case, I am the primary beneficiary I had in mind, so lack of adoption would not prevent me from maintaining it. Being able to reinstall your Linux willy-nilly with full disk encryption, and giving you a fresh but essentially identical copy of your current install (via package management + dotfiles management), that's gold.

EDIT: One extra reason is maybe just to demonstrate writing one in OCaml is feasible, as it wasn't quite obvious what the scope of work was when I first started writing.

Anton Kochkov said

You might want to check also the 0install which is also was ported to OCaml from Python, see their GitHub repository. Though judging by their repositories activity, they seem abandoned OCaml version and thrown all resources to the C#/.NET version in 0install-dotnet.

Thomas Leonard replied

> Though judging by their repositories activity, they seem abandoned OCaml version and thrown all resources to the C#/.NET version

Well, this is the problem with OCaml. When 0install was written in Python, I did frequent bug-fix releases. Now it's in OCaml, there's no need to do that. e.g. there were 8 Python releases in 2012 vs 1 OCaml release last year. I prefer it that way, but it's bad for the project's metrics.

There is no directing of resources. Bastian Eicher did the C# version before and is still doing it now (though more of it is now hosted in the main github org and we just merged the two sets of documentation to make http://docs.0install.net/).

Online programming playground

Archive: https://discuss.ocaml.org/t/online-programming-playground/4391/1

Vishal Rana announced

I wanted to shared https://code.labstack.com/ocaml, an online OCaml programming playground. I hope you will find it useful.

A look back on OCaml 4.*

Archive: https://discuss.ocaml.org/t/a-look-back-on-ocaml-4/4397/1

Thomas Blanc announced

To celebrate the 0th anniversary of the 4.09.0 release, OCamlPro decided to look back at how far the language and standard library got since 3.12.

If you want to feel old and nostalgic, or if you want see what your elders had to put up with before, feel free to check out our blog.

Multicore prerequisite patches appearing in released OCaml compilers now

Archive: https://discuss.ocaml.org/t/multicore-prerequisite-patches-appearing-in-released-ocaml-compilers-now/4408/1

Anil Madhavapeddy announced

The release of OCaml 4.09.0 is particularly significant for us at OCaml Labs, as it represents a phase shift in our development efforts towards integrating multicore parallelism into the language. For the past few years, we have been implementing multicore as a branch based off released versions of the compiler. We finished rebasing it to OCaml 4.06.1 in April and since then have been working on upstreaming a series of incremental changes to OCaml itself.

OCaml 4.09.0 is the first such release in which multicore patches are appearing in released versions of the compiler. This is not the full multicore feature set, but rather the prerequisites to introducing changes required towards introducing parallelism into the runtime. You can now expect to see a regular set of incremental changes towards multicore in every release of OCaml as we ramp up our upstreaming efforts.

One decision we have taken recently is to spend our time on upstreaming changes in favour of further rebases to more recently versions of the compiler. If someone does have a pressing need for a rebase to OCaml 4.08 or 4.09, then please get in touch with me – but bear in mind that it's a significant amount of work and so will need to be justified with a usecase.

In the meanwhile, here's a summary of what some of those patches are, and what to expect in future releases:

4.09.0

  • In the upcoming multicore GC, object headers (tags and lengths) are immutable due to multiple threads scanning the heap simultaneously; any mutations could violate heap invariants in another thread and cause corruption. Therefore, Obj.truncate (#2279) and Obj.set_tag (#1725) have now been deprecated, and all uses removed from the standard library.
  • Values can be passed from OCaml to C by registering them under a known name using the Callback.register function. They can later be retrieved from C using caml_named_value, which returns a value* that can then later be dereferenced. OCaml 4.09.0 modifies the C return type to const value* to indicate that the C code cannot use the pointer that is returned to mutate the value that is registered (#2293). The ability to mutate a value using the raw pointer returned by caml_named_value is incompatible with the upcoming multicore GC, and rarely (never?) used in existing single-core OCaml code.

Ongoing for 4.10.0~dev

This is the subsequent release that is branching imminently now that OCaml 4.09.0 has been released.

  • Variables that are global in the OCaml runtime need to be duplicated per-domain in multicore, since each parallel thread of execution maintains its own table of domain local variables. OCaml 4.09.0 moves all such global C variables into a "domain state" table (#8713). While the change does not introduce any API changes, it significantly alters code generation by reserving a register that was previously used as the exception pointer in every CPU backend for quickly accessing the domain state table. It is therefore a syntactically heavy change, but shouldn't modify the semantics of your code. If you do notice any oddnesses when testing OCaml 4.10~dev when it is released as a beta, please do report a reproduction case upstream.
  • (bonus change) While emerging from deep in a rabbit hole from fixing thread stack overflow detection and reentrant marshalling by ensuring that allocation functions do not trigger OCaml callbacks when invoked from C, it was discovered that major GC hooks could also interact with the GC heap. This is now forbidden (#8711) in OCaml 4.09.0. There was no code found in the wild that did not already conform to this restriction, and it is generally safer this way for the multicore GC as well.

Ongoing for 4.11.0~dev

As 4.10 is about to be branched, we are working away on the following next set of features to push upstream into OCaml 4.11:

  • Better safe points (#187)
  • Tracing and deprecating the instrumented runtime
  • Converging on the representation of closures in bytecode and native code.
  • Modifying GC colors to suit multicore.

As always, these chunks of ongoing work are subject to change as the technical design process is quite iterative and dependent on benchmarking results, but are hopefully useful for you to know!

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.

Alan Schmitt