OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of June 12 to 19, 2018.

Table of Contents


New tool to generate the CWN

Archive: https://framagit.org/brab/cwn

The editor announced:

Thanks to the hard work of Enzo Crance, we now have an OCaml-based tool to generate the OCaml weekly news. We now use Org mode for the conversion to html and text. Please don't hesitate to contact me for any bug report or suggestion.


You like parmap? You like parany?

Archive: https://discuss.ocaml.org/t/you-like-parmap-you-like-parany/2131/1

UnixJunkie announced:

So, here comes parpair: https://github.com/UnixJunkie/parpair

The ultimate step in minimalist parallel programming.

Of course, your function needs to be pretty heavy in order for the computation to be accelerated.


opam-check-all: Are your packages broken for the upcoming OCaml releases?

Archive: https://discuss.ocaml.org/t/ann-opam-check-all-are-your-packages-broken-for-the-upcoming-ocaml-releases/1875/28

Continuing this discussion, Kate announced:

The service now has a proper domain name: http://check.ocamllabs.io/


OCaml job opportunity at Citrix in Cambridge UK

Archive: https://discuss.ocaml.org/t/ocaml-job-opportunity-at-citrix-in-cambridge-uk/2132/1

Christian Lindig announced:

The Citrix office in Cambridge develops the XenServer hypervisor which is open source and builds on Xen. The toolstack part of the product, which orchestrates the VMs, is developed in OCaml (see GitHub) and the team is looking for an engineer with strong affinity to functional programming in the Unix environment. Feel free to send me an email at christian.lindig@citrix.com with any questions. Below is the official job description.

https://jobs.citrix.com/job/CITRA005811517/Software-Engineer-Functional-Programming


Omod 0.0.1

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2018-06/msg00020.html

Daniel Bünzli announced:

I'd like to announce omod which provides data-driven module toplevel loaders.

Like ocamlfind it allows to load compilation objects and their dependencies in the toplevel. However omod works without additional metadata files and with top level (compilation unit) module names rather than ocamlfind package or archive names.

This means you only need to know the name of the module you want to load rather than having to remember in which ocamlfind package or archive it lives.

It should work well in ocaml, ocamlnat[0] and utop. To get started type:

opam install omod

and follow the [tutorial][1]. More information on how omod came to be is given below [2].

omod is ISC licensed and can be found at http://erratique.ch/software/omod

Best,

Daniel

P.S. Currently omod doesn't work with opam system compilers.

[0] One issue with ocamlnat is that some libraries lack their cmxs files. E.g. the lack of threads.cmxs prevents the Core module from loading. But other ones like Irmin, Containers or Base do however load in ocamlnat via omod.

[1]: http://erratique.ch/software/omod/doc/Omod.html#tutorial

[2] This is basically a dependency-less rewrite of odig's toplevel loaders which will be removed from odig. It solves the following problems they had:

  1. Too many dependencies. They brought too many dependencies in the toplevel including compiler-libs which are problematic in ocamlnat​. omod has no dependency, so using it limits toplevel scope pollution to the Omod module only which loads fine in ocamlnat.
  2. Resolution for multiple load sequences. odig had no support for choosing between multiple load solutions (e.g. for library variants). omod has both a variant specification syntax and an interactive mode to resolve them.
  3. Slow. They were too slow on large libraries as resolution was done in the (bytecode) toplevel itself, omod's performance (and API) should be improved for scripting however it is reasonably fast for your toplevel interactive usage.
  4. Outdated information in long running toplevel sessions. They provided no mecanism to update the package information whenever the package install base changed.

To solve 1., 3. and 4. the Omod API invokes the omod binary with the load sub-command (whose textual output you can check when you suspect there's a problem) to find out the load sequences.


Tools for logic programming in OCaml

Archive: https://discuss.ocaml.org/t/tools-for-logic-programming-in-ocaml/2126/4

Continuing this thread from last week, th3rac said:

Also https://github.com/c-cube/datalog, a Datalog implementation


opam-ci: first release of a tool to check the health of your packages

Archive: https://discuss.ocaml.org/t/ann-opam-ci-first-release-of-a-tool-to-check-the-health-of-your-packages/2146/1

Anil Madhavapeddy announced:

Cross posted from https://github.com/ocaml/infrastructure/wiki/Using-the-opam-ci-tool

Have you submitted a new opam package after testing it on your desktop, and then wondered if it builds on OpenSUSE or CentOS, or on an ARM or PowerPC architecture, or against a different version of the OCaml compiler, or if older version continues to work after a few years of being published? But you don't have the resources to check all these interactions manually, especially as the opam package database now contains thousands of revisions of OCaml source code. The new opam ci plugin comes to the rescue! The remainder of this post describes its uses, how to triage issues and then fix them.

opam-ci provides an interface to the opam2 continuous integration cluster, which regularly rebuilds the full package repository across a variety of OCaml compiler versions, operating system distributions and CPU architectures. These builds are done regularly in remote infrastructure and the results are pushed to a metadata repository where they are fetched by a CLI client to let you query the status of your packages.

  • opam ci status shows a dashboard of the build results across this matrix. Packages can be filtered by maintainer substrings or tag names in the opam package description, so you see only those relevant to you.
  • opam ci logs will show you the build errors so you can fix them. It also generates a Dockerfile of the precise build to reproduce the environment locally for you.

To get started, try these commands with the maintainer argument replaced with your own information or tags:

# show all the failing MirageOS packages
opam ci status -m org:mirage | less -R
# show all the packages maintained by anil@recoil.org
opam ci status -m anil@recoil.org --filter=all | less -R
# show all the packages failing on the latest RC of the OCaml compiler
opam ci status --filter=variants:rc | less -R
# display all failure logs for the mirage-xen package
opam ci logs mirage-xen

The status view shows a panel of icons that represent different combinations of ways to build opam packages. From left to right, these are:

  • Compiler: The circled numbers represent OCaml compiler versions (a circled 6 is OCaml 4.06, a circled 7 is 4.07, and so on).
  • Distro: The square letters indicate different OS distributions. D is Debian, F is Fedora, A is Alpine, U is Ubuntu and S is OpenSUSE.
  • CPU Architecture: The small circled letters represent different CPU architectures. x represents x86_64, a is arm64 and p is PowerPC64LE.

Some compiler variants are also tested to track down specific problems, shown by the icons to the far right of the display. (see [below](#fix-specific-issues-around-ocaml-features) for more information on these).

  • safe-string: The ss icon is for 'safe-string' failures, which would happen in OCaml 4.06 due to the switch to immutable strings.
  • flambda: The fl icon is for packages that fail to compile with the flambda variant of the compiler.
  • release-candidate: The flag icon is for packages that fail to compile with the latest release candidate of OCaml; this is useful to figure out how much of the ecosystem works with a soon-to-be-released compiler.

The colours indicate the result of the build: white indicates the package was not built due to constraints, green is a successful build, yellow indicates the build was skipped due to a dependency failure, red is a direct build failure of that package, and magenta and blue indicate package metadata errors such as a failure of the solver to find a solution or the package sources being unavailable. (note: colors are currently mandatory but this needs to be improved)

Uses

Check your own package builds

See all failing builds by specifying the maintainer:

opam ci status -m anil@recoil.org | less -R

See all builds, including successes:

opam ci status -m anil@recoil.org -f all | less -R

Check on a project's libraries

You can specify additional -m fields, which match based on the maintainer: or tags: field in the opam metadata. For example, the MirageOS uses org:mirage in its tags to group libraries:

opam ci status -m org:mirage | less -R

Package you have forgotten about

Sometimes you constrain a package due to an incompatibility with a newer version of OCaml, but then forget to release a new version.

You can query for "lagging" packages whose latest version is incompatible with the latest release of OCaml due to constraints:

opam ci status -f lagging | less -R

Fix specific issues around OCaml features

Newer releases of OCaml come with some backwards incompatible changes. You can find packages with some of those particular problems.

  • Migration to safe-string

    OCaml shifted from mutable to immutable strings by default in OCaml 4.06, but there is a variant of the compiler with the old behaviour. You can list packages which break with the new immutable default, but that work with the older setting.

    opam ci status -f variants:ss | less -R
    
  • Flambda inliner compilation

    There is an experimental inliner available with the flambda configure time variant of the compiler. A few packages fail to compile with the new option, so you can list those explicitly to determine what's wrong:

    opam ci status -f variants:fl | less -R
    
  • Testing on trunk / release candidate OCaml

    When there are release candidates for OCaml, it is helpful to test packages on those pre-release versions. You can find packages that compile successfully on a stable release but fail on the bleeding edge compiler:

    opam ci status -f variants:rc | less -R
    

    Note that this shows actual build failures. You can use the "lagging" filter to find packages that have been constrained to prevent them from being compiled on the latest version entirely, which is useful to figure out what needs porting.

    opam ci status -f lagging | less -R
    

Community service

You can find all unmaintained packages that might seem some assistance since their maintainer fields are blank:

opam ci status -f orphaned | less -R

Identifying issues

Once you have identified a package that is failing, you can inspect the build logs to figure out how to fix the issue.

Firstly, use opam ci logs to find out what's wrong. For example, I might run this on the xen-gnt-unix package.

$ opam ci logs xen-gnt-unix
xen-gnt-unix: multiple build failures found with different configuration parameters.
Please refine the command to select exactly one of the following:
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=ubuntu-18.04
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=fedora-28
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=debian-9
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06+flambda --arch=amd64 --distro=debian-9
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06+default-unsafe-string --arch=amd64 --distro=debian-9
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.05 --arch=amd64 --distro=debian-9
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.04 --arch=amd64 --distro=debian-9
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=alpine-3.7
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=ppc64le --distro=debian-9
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=arm64 --distro=debian-9
  opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=opensuse-42.3
  opam-ci logs xen-gnt-unix.3.0.1 --compiler=4.06 --arch=ppc64le --distro=debian-9
  opam-ci logs xen-gnt-unix.3.0.1 --compiler=4.06 --arch=amd64 --distro=opensuse-42.3

If just one failure is found, then the build logs are shown for that failure. If there is more than one failure, the output will give you a more precise command line to enter to select just one of the failures, as shown above.

Just pick the first one, and the output shows us the abbreviated failure log, and some metadata such as which git revision of the [opam-repository (https://github.com/ocaml/opam-repository) the package was built against.

opam-ci logs xen-gnt-unix.3.0.0 --compiler=4.06 --arch=amd64 --distro=ubuntu-18.04
====> xen-gnt-unix.3.0.0 4.06 Ubuntu 18.04 amd64 (exit code 31) (opam-repository 8425e617):
<snip>
### output ###
# File "/home/opam/.opam/4.06/lib/io-page/META", line 1, characters 0-0:
# Error: Library "io-page-unix" not found.
# -> required by library "io-page.unix" in /home/opam/.opam/4.06/lib/io-page
# Hint: try: jbuilder external-lib-deps --missing -p xen-gnt-unix @install

This then lets you hopefully see a path to fixing the issue.

Fixing issues

The problems you find can range from a few root causes. If you think of more, please add them here by updating the wiki page or posting to the discussion forum.

  • Incorrect opam package constraints

    If a package used to build but subsequently starts failing, it is probably because some dependencies have changed their interfaces. You can use opam package constraints to fix this by identifying which the offending dependency is from the build failure, and modifying your package to only select

    Feature: #5 tracks including dependency information into the opam-ci metadata directly.

    A package occasionally also becomes uninstallable due to dependency constraints resulting in an impossible situation. This is normally caught by the opam repository maintainers, but mistakes slip through. In this case, you'll need to figure out the constraints that let your package install.

    Feature: #7 tracks including opam lint output in opam-ci status to make finding these problems easier.

  • OCaml compiler version

    OCaml now releases on a 6-8 month release schedule, and the march of progress occasionally breaks existing code. In this case, maintainers can release a new version of their package that works, but older releases are still tracked in opam and should be constrained to prevent them being selected by the new compiler.

    One example is the migration to safe-string, which broke a large number of packages out of the box in OCaml 4.06.0. In this case, you might see an issue like this:

    $ opam-ci logs syslog.1.4 --compiler=4.06 --arch=amd64 --distro=debian-9
    # File "syslog.ml", line 196, characters 50-53:
    # Error: This expression has type bytes but an expression was expected of type
    # string
    

    In this case, you need to prevent this package from being installed on OCaml 4.06.0 or higher, and release a new version of the package with the functionality fixed. The opam1 constraint looks like:

    available: [ ocaml-version < "4.06.0" ]
    
  • Operating System portability

    A very common situation is that you test your package on your local desktop, but cannot try it on the huge number of Linux distributions out there. Thanks to the magic of Docker containers, the opam-ci shows build information on many distributions: Ubuntu, Debian, CentOS, Alpine, OpenSUSE and OracleLinux across several versions (the full list is here).

    To pick on an example of zstd, the opam ci status zstd shows that it builds on Debian and Ubuntu, but fails on CentOS and Alpine. Inspecting the opam file reveals why:

    $ opam show zstd --raw
    <snip>
    depexts: [
      ["libzstd-dev"] {os-distribution = "debian"}
      ["libzstd-dev"] {os-distribution = "ubuntu"}
    ]
    

    The depexts field is driven by opam-depext which understands a large number of operating systems. You can fix this by including the packages for Alpine and CentOS and submitting the fixes. The opam-repository CI will test your submitted fixes against the matrix of operating systems and verify if the fix worked or not, or you can use Docker locally via the [OCaml containers](Containers) before submitting the fix.

  • CPU portability

    OCaml runs on a variety of CPU architectures, but most of us develop on x86. The bulk builds also run regularly on arm64 and ppc64le, which exposes portability bugs in C bindings quite often.

    The AFL testing system fails on ARM for example:

    opam-ci logs afl.2.52b --compiler=4.06 --arch=arm64 --distro=debian-9
     ### output ###
     # [*] Checking for the ability to compile x86 code...
     # /tmp/cctyEiSt.s: Assembler messages:
     # /tmp/cctyEiSt.s:10: Error: unknown mnemonic `xorb' -- `xorb %al,%al'
     #
     # Oops, looks like your compiler can't generate x86 code.
     #
     # Don't panic! You can use the LLVM or QEMU mode, but see docs/INSTALL first.
     # (To ignore this error, set AFL_NO_X86=1 and try again.)
    

    In this case, you can fix the portability issue in your package and release a new version. To prevent the older (already released) packages from being selected by the opam solver on that architecture, you can add a cpu constraint in the available: field. An example is the yaml.0.2.0 package that was broken on PowerPC and fixed in yaml.0.2.1:

    $ opam show yaml.0.2.0 --raw
    <snip>
    available: arch != "ppc64"
    

Contributions

We would love to see more contributions to opam-ci and the associated infrastructure. The most obvious way you can get involved is by looking at the build failures and submitting fixes to the opam-repository to help us maintain the rapidly growing package database.

If you would like to work on the CLI tool itself, then feel free to get started and look at the issue list and the contributions guidelines to get started, and/or post on the OCaml discussion forum. You can also add more triaging tips to this wiki to help other users.


Does OCaml ecosystem have CSS preprocessors?

Archive: https://discuss.ocaml.org/t/does-ocaml-ecosystem-have-css-preprocessors/2139/3

Volodymyr Melnyk asked and Nadeem Bitar replied:

Does OCaml's ecosystem have CSS preprocessors except CCSS (cause I want to write CSS in OCaml)?

I've used bs-css to write CSS in OCaml.

It allows you to write something like:

open CSS
let card = style
[
  display flexBox;
  flexDirection column;
  boxShadow ~y:(px 3) ~blur:(px 5) (rgba 0 0 0 0 0.3);
]

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.