OCaml Weekly News

It is my pleasure to announce the first release of the package curve-sampling which provides adaptive sampling of functions (and more generally parametric curves) so they can be represented graphically with a minimal number of evaluations. Curve_sampling takes the viewpoint that the evaluation function may be costly, so one specifies the number of evaluations one wants. For example, a representation of x ↦ x sin(1/x) with 227 evaluations gives (rendered with Gnuplot but Curve_sampling is independent of a rendering engine):

The chosen evaluation points are displayed on the following graph:

As a comparison, here is the graph produced by Mathematica for the same function (which uses a different procedure which does not allow to limit the number of evaluations, only the recursion depth, here 5, yielding to the evaluation at 227 points):

With 389 evaluations (recursion depth of 6 for Mathematica), the graphs become: and

Enjoy the library and, if you have functions that are not sampled well with the current algorithm, do not hesitate to fill an issue.

is there a way to feed the sampled points to matplotlib, for example?

There is Curve_sampling.to_file that saves the data to a file as lines of x y with blank lines when the curve needs to be interrupted. It can certainly be read from Python. Another possibility would be to use py to import the data without using a file — there is Curve_sampling.to_list to get it from the sampling but additional exports may be added (a Seq export is available on OCaml ≥ 4.07). If you try that latter route, let me know.

Second article is here: https://medium.com/@sanderspies/the-road-to-webassembly-gc-for-ocaml-bd44dc7f9a9d (slightly rushed).

Last Thursday at re:Invent, AWS announced the ability to run custom runtimes in AWS Lambda (blog post).

Today I have the pleasure of announcing an OCaml custom runtime for AWS Lambda that, while in its early stages, already works quite well.

I haven't made a release yet, but if you wanna try it out you can head over to the GitHub repo and ping me directly if you have any questions.

Thanks all for this input. While working on the intrinsics and details of the tools, it's easy to lose the big picture, and the very important point of view of the newcomers.

So first, as the main developper of opam 2.0, I'd like to say that we have been putting a lot of work in it, and a large part of the effort was to improve for convenience, and the many use-cases that weren't supported before — including, but not limited to, reproducible build environments, and project-local sandboxes, a.k.a. switches.

The documentation, however, is severly lacking at the moment on all these new features, and the preferred and simplest ways to accomplish all the basic tasks. For all asking about the detailed formats, we have a fairly complete manual [1], and the API should be fairly well documented [2], but indeed, it's way too detailed to be the first documentation you would get to.

Let me assure you however that everything is slowly getting into place for an easier approach for everyone. I'll go below through the "typical workflow" you propose, checking what is here or not, but let's first focus on the users rather than the developers:

• installing ocaml: indeed, if not easily available for your system, the easiest is to install opam, then just run opam init. (Yes, we should be explicit in the doc that this is the command that will install the compiler)
• installing a given package, and assuming opam is installed and initialised:
  • if in the repository, just one opam install command should be all you need
  • otherwise, if your source is available somewhere and contains an opam package definition file, opam pin URL is everything you need (URL pointing to an archive, or a git repository, etc.). We could merge this use-case into opam install for better discoverability.
  • if not and/or you want to build the project manually from a clone, the support has been much improved in opam 2, so that you can for example document specific pinned dependencies, or a "locked" development state (see opam-lock [3] to do that automatically). Then e.g. opam switch create . --locked will recreate a local switch with the exact same development configuration, and install the project in it. opam install . --locked also works, if you don't want the local sandbox.
• it has been mentionned already in this thread, but the opam bundle plugin can make distribution easier by including the whole OCaml + opam + package environment in a single, self-building self-extracting archive. At the cost of rebuilding everything on installation. A new release is pending [4]. Yes, it's yet another tool, but with its straight-forward interface and everyting explained in its 100-lines, included manpage, I find the criticism uncalled for. Not a silver bullet by any means, but it fits some use-cases.

As for using wrapping Makefiles, they are nice for simple build-system calls, and I like them if only to document the entry-point, but shouldn't IMHO mess with the packaging system. Note also that the main purpose of opam files is actually to document the building commands of any project, taking into account all OS specificities, and in an easily understandable format. I personally find that having clear and simple build instructions arout the top of the README is enough.

Once properly documented from the opam side (huh), I expect project maintainers to be able to put simple setup + installation instructions at the top of their READMEs, so that users who don't care about OCaml or opam just need to copy-paste them to get the environment setup and the project compiled. As far as I understand it now, this is where the problem really stands. To avoid having to look anything up or learning about exotic build system, this is the best compromise IMHO.

I'd also like to point out that this is not specific to OCaml, and I believe all language package managers / build systems suffer from these issues: I for one struggle every time I have to use something building with NPM, and they don't generally provide Makefiles. Of course, with a tool as popular as NPM, the problem is less visible because you have to go through it anyway. So we do need to improve documentation and simplify basic workflows as much as possible, but expecting people to work with OCaml without learning any of the tooling is unrealistic (unless they stick to an online IDE or e.g. Learn-OCaml, and even that is tooling in some form).

Let me now go through your "typical workflow":

> cd some-ocaml-proj
> opam install # Switches compiler if necessary and installs and locally
> caches package dependencies

You can do this with opam switch create . Since "if necessary" is pretty subjective, just run opam install . if you prefer to share the environment with other projects.

> opam build

opam install .

> opam run # Automatically builds if necessary

there is no package←→executable bijection, so I don't see how this would work? (same as for OS-level packages) see below, but this might be dune exec <command>

> opam test # Ditto

indeed here we enter the domain where the separation between build system and packaging system can hurt. You can run opam install . --with-test, but probably want dune runtest instead.

> opam package # Ditto; –upload option can immediately upload to opam

at this point you must already have a package definition available ? Or do you mean creating a release archive ? If your source is hosted on Github, you only need to push a tag and run opam publish (you otherwise need to provide an URL for the source archive and that's it).

> opam doc # Builds documentation with ocamldoc or whatever
> opam login -u user -p password

I am not sure what you have in mind here. opam publish will go through Oauth authentification with Github for submitting your new package.

As one last note, let me mention that we are right now discussing:

• better integration of opam and dune
• integration of system dependency handling ("depexts") into opam

Hope this helps, feedback and questions welcome.

Louis Gesbert — OCamlPro

[1] https://opam.ocaml.org/doc/Manual.html
[2] https://opam.ocaml.org/doc/api/index.html
[3] https://github.com/AltGr/opam-lock
[4] https://github.com/ocaml/opam-repository/pull/13064

> I have a related request. I am not a trusting person. I do not like "curl | sudo sh" type installation methods.

You're not the only one :) Some notes on opam's security model:

• opam 2.0 uses, by default bubblewrap [1] on Linux and sandbox-exec on OSX to ensure that package scripts:
  • don't make any network access
  • don't interact with other processes
  • don't write outside of their build dir, /tmp, and (in the case of install) the switch prefix (excl. opam files)
• this is done using simple wrapper scripts [2] and some default hooks configuration in ~/.opam/config, so if you know about built-in sandboxing engines for other OSes, it is fairly easy to experiment with them, and a contribution would be very welcome.
• while I expect this to be reasonably secure, it's intended first and foremost to avoid dramatic errors, not to protect against malicious repositories
• package scripts are protected but not any use made by the users of the programs or libraries that were installed through opam. In other words, building should be safe, but there is no guarantee about what the result of the build will do: that is not restrained by opam in any way
• the effort to provide end-to-end package signatures in the repository [3] is still ongoing. Cheers to Hannes Mehnert for the awesome work he has already done here. Most of the work should be done, but then we need to integrate all that, and there is a lot of work on the tooling so that it won't add to much burden on users and repository maintainers (this commonly results in most disabling the security features, which is as good has having no security features to begin with).
• we do advertise curl | sh on the installation page as the easiest entry point, but the script is quite trivial and only uses root to copy to your prefix; it's very easy to fetch the binary by hand from Github if you prefer not to run it, and of course, you can also build from source using the bootstrap scripts.

> If a package has 'rm -rf $BUILD/', or equivalent ocaml code, are its ill > effects confined when BUILD is unset?

yes, that's the whole point of the sandboxing that was introduced in 2.0

> Can the build process grab screenshots from the background?

not sure. Probably not on Linux since we use a different process space, but maybe on OSX. In anycase, since network access is blocked in both cases, that wouldn't do much harm.

> One reason I like make is, if the Makefile is simple you know what it's going to do.

I would object that opam package definition files (opam or foo.opam) should be at least as straightforward to read even if you have never seen the syntax, are less error-prone, and are generally much shorter. Just look for the "build:" and "install:" parts. But I agree you need to know first to look at them, and since they are generally an indirection to some build-system (make, dune, topkg…), you would just start digging…

> Also, the xkcd on standards seems relevant: https://xkcd.com/927/

We have actually been converging as of late, though.

Best, Louis Gesbert — OCamlPro

[1] https://github.com/projectatomic/bubblewrap
[2] https://github.com/ocaml/opam/blob/master/src/state/shellscripts/bwrap.sh and https://github.com/ocaml/opam/blob/master/src/state/shellscripts/sandbox_exec.sh
[3] https://github.com/hannesm/conex

Shameless plug, I tried to document basic opam usage for people who have experience with npm. It turns ok it can also be an introduction to total newcomers. Hope it can help some people. https://khady.info/opam-npm.html

I have the pleasure to announce that containers 2.4 was just released and has been merged into opam. Highlights include more labelled modules, migration to dune and opam2, a few new combinators, and some bugfixes. Thanks to all the contributors and issue reporters!

Full release here.

We're looking for a new lead maintainer for yojson, ideally with the support of their employer who can use the opportunity to give back to the community and attract talent. Yojson is a direct dependency of 89 opam packages.

I created yojson in 2010 to replace json-wheel as part of an effort to improve the performance of serialization code derived from type definitions. This was done at the time by json-static, a camlp4-based syntax extension. Specifically, yojson exports efficient functions for parsing and printing json elements directly without going through an AST, unlike its predecessor. This allows a type-driven code generator like json-static's replacement atdgen to produce efficient parsing and printing code. In addition to this, yojson also provides a json AST like json-wheel used to. This AST can be manipulated directly and widely used by people who haven't found out about atdgen or haven't figured out how to set it up. It can also be genuinely useful as a last resort for handling untypable json data where needed.

Yojson is stable and requires light maintenance, typically less than one day per month. It's up to the new maintainers to decide where they want to take the project in terms of new features, documentation, and community involvement.

Some ICFP 2018 OCaml videos are up:

Winning on Windows: https://www.youtube.com/watch?v=1DAuSSljLFI
Wall (vector graphics): https://www.youtube.com/watch?v=bQB8kBkHxjk
Safely mixing OCaml and Rust: https://www.youtube.com/watch?v=UXfcENNM_ts
Merlin: A language server for OCaml: https://www.youtube.com/watch?v=VjLL9We1Fxc

The next OUPS meetup will take place on Tuesday, December 11, 7pm at IRILL on the Jussieu campus. As usual, we will have a few talks, followed by pizzas and drinks.

The talks will be the following:

• Pierre Chambart, FLambda, https://caml.inria.fr/pub/docs/manual-ocaml/flambda.html
• Romain Calascibetta, OCaml-git, https://github.com/dinosaure/ocaml-git.

Please do note that we are always in demand of talk proposals for future meetups.

To register, or for more information, go here: https://www.meetup.com/ocaml-paris/events/256921398

Registration is required! Access is not guaranteed after 7pm if you're not registered. (It also helps us to order the right amount of food.)

Access map:
IRILL - Université Pierre et Marie Curie (Paris VI)
Barre 15-16 1er étage
4 Place Jussieu
75005 Paris
https://www.irill.org/pages/access.html

The dune team is pleased to announce the release of Dune 1.6.0. This release does not contain any notable features or bug fixes that need to be specifically pointed out. Nevertheless, I expect that this will be a quality of life improvement to power users who have experienced dune's sharper corners.

Happy hacking!

Here is a sneak peek at some potential future features of the Ocaml compiler, discussed by their implementers in these Github Pull Requests.

• Use autoconf to generate the compiler's configuration script
• Optimize some local functions
• deprecate the mutability of Gc.control record fields
• Decide unboxing of let-bound expressions based on their Cmm translation + unbox across static handlers
• Avoid page table lookup in Hashtbl.hash with no-naked-pointers
• Document ocaml.local attribute on functions
• Provide let operators in the standard library
• Windows - Stdlib: Format function always uses LF newlines
• Add clz and popcnt intrinsics

Here are links from many OCaml blogs aggregated at OCaml Planet.

• Site Reliability Engineer (SRE II)
• Software Engineer II (Programming Languages)
• Software Engineer (Front-End Engineer II)
• Frama-C 18.0 (Argon) is out. Download ithere.