OCaml Weekly News

Hello

Here is the latest OCaml Weekly News, for the week of July 06 to 13, 2021.

slug 1.0.0 - URL-safe slug generator
Developer Experience Engineer at Jane Street
Hardcaml MIPS CPU Learning Project and Blog
OCaml for Windows installation confusion
OCamlFormat config file auto-completion support in VSCode
Multicore OCaml: June 2021
memprof-limits (first official release): Memory limits, allocation limits, and thread cancellation, with interrupt-safe resources
Bitwuzla 1.0.0 (SMT solver for AUFBVFP)
Old CWN

slug 1.0.0 - URL-safe slug generator

Archive: https://discuss.ocaml.org/t/ann-slug-1-0-0-url-safe-slug-generator/8107/1

Khoa Nguyen announced

I want to introduce my first library ever released on opam. Link to Github repository: https://github.com/thangngoc89/ocaml-slug

ocaml-slug

Url safe slug generator for OCaml

A URL slug is the part of a URL or link that comes after the domain extension. In websites the keyword used for your URL slug can be used to SEO optimize the URL by showing Google the structure of your site and the contents of the page in question.

This library turns title into URL-safe slug with support for non-latin characters.

This library uses algorithm and data from node-slugify

Installation

opam install slug

If you use esy

esy add @opam/slug

Usage

OCaml syntax:

Slug.slugify "my string";;
- : string = "my-string"

(* Custom separator *)
Slug.slugify ?sep: "_" "my string";;
- : string = "my_string"

(* Retain uppercase *)
Slug.slugify ?lowercase: false "My String";;
- : string = "My-String"

(* Use locale *)
let with_vi = Slug.(Charmap.mk_charmap [Slug_data.base; Slug_data.vi]);;
- : Charmap.t = <abstr>
Slug.slugify ?charmap: with_vi "Đ";;
- : string = "d"
Slug.slugify "Đ";;
- : string = "dj"

(* Custom characters map *)
let custom_map = Slug.(Charmap.mk_charmap [Slug_data.base; [ ("M", "z"); ("m", "z") ]]);;
val custom_map : Charmap.t = <abstr>

Slug.slugify ?charmap: custom_map "Mm";;
- : string = "zz"

ReasonML syntax

Slug.slugify("my string");
- : string = "my-string"

/* Custom separator */
Slug.slugify(~sep = "_", "my string");
- : string = "my_string"

/* Retain uppercase */
Slug.slugify(~lowercase = false, "My String");
- : string = "My-String"

/* Use locale */
let with_vi = Slug.(Charmap.mk_charmap([Slug_data.base, Slug_data.vi]));
let with_vi : Charmap.t = <abstr>
Slug.slugify(~charmap = with_vi, "Đ");
- : string = "d"
Slug.slugify("Đ");
- : string = "dj"

/* Custom characters map */
let custom_map = Slug.(Charmap.mk_charmap([Slug_data.base, [ ("M", "z"), ("m", "z") ]]));;
let custom_map : Charmap.t = <abstr>

Slug.slugify(~charmap = custom_map, "Mm");
- : string = "zz"

Notice

Please don't send PR to update data/*. They are auto-generated from upstream library.

Please send PRs about new locales to node-slugify.

If you really need it, you can use a custom_map instead.

License

MIT. Data are downloaded from node-slugify

Developer Experience Engineer at Jane Street

Archive: https://discuss.ocaml.org/t/job-developer-experience-engineer-at-jane-street/8113/1

Yaron Minsky announced

You can read more about the job here:

https://blog.janestreet.com/looking-for-a-developer-experience-engineer-index/

The role is for someone to act as an advocate for our internal developers, helping move information in both directions: teaching people how to make most use of the tools, and figuring out what our developers need, and helping set priorities for our dev-tools and compiler teams.

Please share this with anyone you think might be a good fit for the role!

Hardcaml MIPS CPU Learning Project and Blog

Archive: https://discuss.ocaml.org/t/hardcaml-mips-cpu-learning-project-and-blog/8088/9

Clément asked and Alexander (Sasha) Skvortsov announced

👍 Does your website have an RSS feed?

It does now! https://ceramichacker.com/rss. Also https://ceramichacker.com/atom for Atom users.

On a side note, we've published two more posts:

More coming soon!

OCaml for Windows installation confusion

Archive: https://discuss.ocaml.org/t/ocaml-for-windows-installation-confusion/8105/17

Deep in this thread, Nicolás Ojeda Bär said

At LexiFi we have been writing OCaml under Windows for a long time and are quite experienced with it. I know of at least one other large industrial player that uses OCaml on Windows as their main environment cc @keleshev.

If I had to summarize the situation of OCaml on Windows I would say that once you get past the installation of the dev environment (C toolchain + Cygwin), the experience is quite similar to Linux. The compiler, standard library, dune and merlin, all work flawlessly on Windows. OPAM is not 100% ready yet on Windows (we don't use it) but it has never been easier to work "monorepo" style using dune.

For beginners the main stumbling block is setting up the dev environment (C toolchain + Cygwin). On Linux installing a C toolchain is not needed because the compiler is installed by default, but this is not the case on Windows. As for Cygwin, it is strictly speaking only necessary when building the compiler itself. Once the compiler is installed you are free to never use Cygwin again…

So if you want a Rust-style exeperience for beginners you need to figure out how to provide a point-and-click installer that does the following things:

installs the C toolchain,
installs Cygwin and builds & installs the OCaml compiler using it
installs merlin and dune

At this point you will be left with a very capable environment for writing OCaml programs using common editors such as VS Code.

Regarding 1. above (the C toolchain), there are two main options. If you want a completely freestanding C toolchain you must use the native MSVC compiler command-line tools, which you can get from https://docs.microsoft.com/en-us/cpp/build/building-on-the-command-line (link appears down at the moment).

If you are willing to use Cygwin for your day-to-day developing (as opposed to only using it when building the compiler), then you have a second option which is to use the mingw64 compiler. This is a compiler that produces native Windows binaries, but the advantage is that it is essentially gcc so you get an user experience which you may be used to from Linux. As it runs under Cygwin, this option may appeal to those that prefer to insulate themselves as much as possible from Windows specificities.

My guess is that the recommanded windows way is WSL2.

It depends what you mean by "on Windows". If all you want is to develop "on" a Windows machine, yes by all means WSL is a good option. But the binaries you produce are Linux binaries, so they won't run on Windows outside of the WSL environment.

Thank you. However if the repository won’t get any more updates from next month, that seems like a showstopper for OCaml usage on Windows to me.

Only if you insist on using OPAM for your development. It is perfectly feasible to develop on Windows "monorepo" style without using OPAM. The experience may not be as pleasent as it would be on Unix and depending on how much you rely on external libraries it may be more or less convenient, but it works quite well in general.

OCamlFormat config file auto-completion support in VSCode

Archive: https://discuss.ocaml.org/t/ocamlformat-config-file-auto-completion-support-in-vscode/8118/1

тars announced

Hi, I made a tiny vscode extension which provides auto-completion for .ocamlformat file. (I posted this on discord but let me post it here too.)

…BTW I'm a complete beginner so this is written in TS. If I become comfortable with OCaml and its ecosystems, I'd like to re-write this in OCaml. But as of now, I have no clue. :upside_down_face:

Cheers.

https://github.com/tars0x9752/ocamlformat-auto-completion-vscode

тars then added

Here's how it works.

Max Lantas replied

Wow, this is great! Nice job!

I’d like to re-write this in OCaml.

If you decide to try this, you can look VSCode OCaml Platform source code to see how we wrote the OCaml extension in OCaml code. The basic idea is that it uses the Js_of_ocaml compiler to compile OCaml code to Javascript with bindings to the VSCode API.

I glanced over your source code and it looks like it would translate very well to OCaml code. Feel free to reach out if you'd like help.

Multicore OCaml: June 2021

Archive: https://discuss.ocaml.org/t/multicore-ocaml-june-2021/8134/1

Anil Madhavapeddy announced

Welcome to the June 2021 Multicore OCaml monthly report! This month's update along with the previous update's have been compiled by @avsm, @ctk21, @kayceesrk and @shakthimaan.

Our overall goal remains on track for generating a preview tree for OCaml 5.0 multicore domains-only parallelism over the summer.

Ecosystem compatibility for 4.12.0+domains

In May's update, I noted that our focus was now on adapting the ecosystem to work well with multicore, and I'm pleased to report that this is progressing very well.

The 4.12.0+domains multicore compiler variant has been merged into mainline opam-repo, so you can now opam switch 4.12.0+domains directly. The base-domains package is also available to mark your opam project as requiring the Domains module, so you can even publish your early multicore-capable libraries to the mainline opam repository now.
The OCaml standard library was made safe for parallel use by multiple domains (wiki, issue, fixes); and in particularly the Format and Random modules. These modules were the main sources of incompatibilities we found when running existing OCaml code with multiple domains.
The Domain module has had its interface slimmed with the removal of critical_section, wait, notify which has allowed significant runtime simplification. The GC C-API interface is now implemented and this means that Jane Street's Base, Core, and Async now compile on 4.12+domains without modifications; for example opam install patdiff works out of the box on a 4.12+domains switch!
Domainslib 0.3.0 has been released which incorporates multiple improvements including the work-stealing deques for task distribution. The performance of reading domain local variables has also been improved with a primitive and a O(1) lookup. The chapter on Parallel Programming in Multicore OCaml has been updated to reflect the latest developments with Domainslib.

This means that big application stacks should now compile pretty well with 4.12.0+domains (applications like the Tezos node and patdiff exercise a lot of the dependency trees in opam). If you do find incompatibilities, please do report them on the repository.

4.12.0+domains+effects

Most of our focus has been on getting the domains-only trees (for OCaml 5.0) up to speed, but we have been progressing the direct-style effects-based IO stack as well.

The uring bindings to Linux Io_uring are now available on opam-repository, so you can try it out on sequential OCaml too. A good mini-project would be to add a uring backend to the existing Async or Lwt engines, if anyone wants to try a substantial contribution.
The eio library is fairly usable now, for both filesystem and networking. We've submitted a talk to the OCaml workshop to dive into the innards of it in more detail, so stay tuned for that in the coming months if accepted. The main changes here have been performance improvements, and the HTTP stack is fairy competitive with (e.g.) rust-hyper.

We will soon also have a variant of this tree that removes the custom effect syntax and implements the fibres (the runtime piece) as Obj functions. This will further improve ecosystem compatibility and allow us to build direct-style OCaml libraries that use fibres internally to provide concurrency, but without exposing any use of effects in their interfaces.

Benchmarking and performance

We are always keen to get more benchmarks that exercise multicore features; if you want to try multicore out and help write benchmarks there are some suggestions on the wiki. We've got a private server which runs a Sandmark nightly benchmark pipeline with Jupyter notebooks, which we can give access to anyone who submits benchmarks. We continue to test integration of Sandmark with current-bench for better integration with GitHub PRs.

As always, the Multicore OCaml ongoing and completed tasks are listed first, which are then followed by updates from the ecosystem and their associated libraries. The Sandmark benchmarking and nightly build efforts are then mentioned. Finally, the status of the upstream OCaml Safepoints PR is provided for your reference.

Multicore OCaml

Ongoing

ocaml-multicore/ocaml-multicore#573 Backport trunk safepoints PR to multicore

A work-in-progress to backport the Safepoints PR from ocaml/ocaml to Multicore OCaml.
ocaml-multicore/ocaml-multicore#584 Modernise signal handling

A patch to bring the Multicore OCaml signals implementation closer to upstream OCaml.
ocaml-multicore/ocaml-multicore#598 Do not deliver signals to threads that have blocked them

A draft PR to not deliver signals to threads that are in a blocked state. The without-systhreads case needs to be handled.
ocaml-multicore/ocaml-multicore#600 Expose a few more GC variables in headers

The caml_young_start, caml_young_limit and caml_minor_heap_wsz variables have been defined in the runtime.
ocaml-multicore/ocaml-multicore#601 Domain better participants

The iterations 0(Max_domains) from STW signalling and 0(n_running_domains) from domain creation have now been removed.
ocaml-multicore/ocaml-multicore#603 Systhreads tick thread

An initial draft PR for porting the tick thread to Multicore OCaml.

Completed

Enhancements

ocaml-multicore/ocaml-multicore#552 Add a force_instrumented_runtime option to configure

The configure script now accepts a new --enable-force-instrumented-runtime option to facilitate use of the instrumented runtime on linker invocations to obtain event logs.
ocaml-multicore/ocaml-multicore#558 Refactor Domain.{spawn/join} to use no critical sections

The critical sections in Domain.{spawn/join} and the use of Domain.wait have been removed.
ocaml-multicore/ocaml-multicore#561 Slim down Domain.Sync: remove wait, notify, critical_section

A breaking change in Domain.Sync that removes critical_section, notify, wait, wait_for, and wait_until. This is to remove the need for domain-to-domain messaging in the runtime.

ocaml-multicore/ocaml-multicore#576 Including Git hash in runtime

A Git hash is now printed in the runtime as shown below:

$ ./boot/ocamlrun -version
The OCaml runtime, version 4.12.0+multicore
Built with git hash 'ae3fb4bb6' on branch 'runtime_version' with tag '<tag unavailable>'

ocaml-multicore/ocaml-multicore#579 Primitive for fetching DLS root

A new primitive has been implemented for fetching DLS, and is now a single mov instruction on amd64.

Upstream

ocaml-multicore/ocaml-multicore#555 runtime: CAML_TRACE_VERSION is now set to a Multicore specific value

A CAML_TRACE_VERSION is defined to distinguish between Multicore OCaml and trunk for the runtime.
ocaml-multicore/ocaml-multicore#581 Move our usage of inline to Caml_inline

We now use Caml_inline for all the C inlining in the runtime to align with upstream OCaml.
ocaml-multicore/ocaml-multicore#589 Reintroduce adjust_gc_speed

The caml_adjust_gc_speed function from trunk has been reintroduced to the Multicore OCaml runtime.
ocaml-multicore/ocaml-multicore#590 runtime: stub caml_stat_* interfaces in gc_ctrl

The creation of caml_stat_* stub functions in gc_ctrl.h to introduce a compatibility layer for GC stat utilities that are available in trunk.

Fixes

ocaml-multicore/ocaml-multicore#562 Import fixes to the minor heap allocation code from DLABs

The multiplication factor of two used for minor heap allocation has been removed, and the Minor_heap_max limit from config.h is no longer converted to a byte size for Multicore OCaml.
ocaml-multicore/ocaml-multicore#593 Fix two issues with ephemerons

A patch to simplify ephemeron handover during termination.
ocaml-multicore/ocaml-multicore#594 Fix finaliser handover issue

The caml_finish_major_cycle is used leading to the major GC phase Phase_sweep_and_mark_main for the correct handoff of finalisers.
ocaml-multicore/ocaml-multicore#596 systhreads: do st_thread_id after initializing the thread descriptor

The thread ID was set even before initializing the thread descriptor, and this PR fixes the order.
ocaml-multicore/ocaml-multicore#604 Fix unguarded caml_skiplist_empty in caml_scan_global_young_roots

The PR introduces a caml_iterate_global_roots function and fixes a locking bug with global roots.

Cleanups

ocaml-multicore/ocaml-multicore#567 Simplify some of the minor_gc code

The not_alone variable has been cleaned up with a simplification to the minor_gc.c code.
ocaml-multicore/ocaml-multicore#580 Remove struct domain

The caml_domain_state is now the single source of domain information with the removal of struct domain. struct dom_internal is no longer leaking across the runtime.
ocaml-multicore/ocaml-multicore#583 Removing interrupt queues

The locking of struct_interruptor when receiving interrupts and the use of struct interrupt have been removed, simplifying the implementation of domains.

Sundries

ocaml-multicore/ocaml-multicore#582 Make global state domain-local in Random, Hashtbl and Filename

The Domain-Local is now set as the default state in Random, Hashtbl and Filename.
ocaml-multicore/ocaml-multicore#586 Make the state in Format domain-local

The default state in Format is now set to Domain-Local.
ocaml-multicore/ocaml-multicore#595 Implement caml_alloc_dependent_memory and caml_free_dependent_memory

Dependent memory are the blocks of heap memory that depend on the GC (and finalizers) for deallocation. The caml_alloc_dependent_memory and caml_free_dependent_memory have been added to runtime/memory.c.

Ecosystem

Ongoing

ocaml-multicore/eventlog-tools#3 Use ocaml/setup-ocaml@v2

An update to .github/workflows/main.yml to build for ocaml/setup-ocaml@v2.
ocaml-multicore/parallel-programming-in-multicore-ocaml#7 Add a section on Domain-Local Storage

The README.md file now includes a section on Domain-Local Storage.
ocaml-multicore/eio#26 Grand Central Dispatch Backend

The implemention of the Grand Central Dispatch (GCD) backend for Eio is a work-in-progress.
ocaml-multicore/domainslib#34 Fix initial value accounting in parallel_for_reduce

A patch to fix the initial value in parallel_for_reduce as it was being accounted for multiple times.
ocaml-multicore/domainslib#36 Switch to default Random module

The library has been updated to use the default Random module as it stores its state in Domain-Local Storage which can be called from multiple domains. The Sandmark results are given below:
ocaml-multicore/multicore-opam#56 Base-effects depends strictly on 4.12

A query on the use of strict 4.12.0 lower bound for OCaml in base-effects.base/opam.
ocsigen/lwt#860 Lwt_domain: An interfacet to Multicore parallelism

The Lwt_domain module has been ported to domainslib Task pool for performing computations to CPU cores using Multicore OCaml's Domains. A few benchmark results obtained on an Intel Xeon Gold 5120 processor with 24 isolated cores is shown below:

Completed

Ocaml-Uring

The ocaml-uring repository contains bindings to io_uring for OCaml.

ocaml-multicore/ocaml-uring#21 Add accept call

The accept call has been added to uring along with the inclusion of the unix library as a dependency.
ocaml-multicore/ocaml-uring#22 Add support for cancellation

A cancel method is added to request jobs for cancellation. The queuing operations and tests have also been updated.
ocaml-multicore/ocaml-uring#24 Sort out cast

The Int_val has been changed to Long_val to remove the need for sign extension instruction on 64-bit platforms.
ocaml-multicore/ocaml-uring#25 Fix test_cancel

A with_uring function is added with a queue_depth argument to handle tests for cancellation.
ocaml-multicore/ocaml-uring#26 Add openat2

The openat2 method has been added giving access to all the Linux open and resolve flags.
ocaml-multicore/ocaml-uring#27 Fine-tune C flags for better performance

The CFLAGS have been updated for performance improvements. The following results are observed for the noop benchmark:
```
Before: noop   10000  │        1174227.1170 ns/run│
After:  noop   10000  │         920622.5802 ns/run│
```
ocaml-multicore/ocaml-uring#28 Don't allow freeing the ring while it is in use

The ring is added to a global set on creation and is cleaned up on exit. Also, invalid cancellation requests are checked before allocating a slot.
ocaml-multicore/ocaml-uring#29 Replace iovec with cstruct and clean up the C stubs

The readv and writev now accept a list of Cstructs which allow access to sub-ranges of bigarrays, and to work with multiple buffers. The handling of OOM errors has also been improved.
ocaml-multicore/ocaml-uring#30 Fix remaining TODOs in API

The read and write methods have been renamed to read_fixed and write_fixed respectively. The Region.to_cstruct has been added as an alternative to creating a sub-bigarray. An exception is now raised if the user requests for a larger size chunk.
ocaml-multicore/ocaml-uring#31 Use caml_enter_blocking_section when waiting

The caml_enter_blocking_section and caml_leave_blocking_section are used when waiting, which allows other threads to execute and the GC can run in the case of Multicore OCaml.
ocaml-multicore/ocaml-uring#32 Compile uring using the C flags from OCaml

Use the OCaml C flags when building uring, and remove the unused dune file.
ocaml-multicore/ocaml-uring#33 Prepare release

The CHANGES.md, README.md, dune-project and uring.opam files have been updated to prepare for a release.
ocaml-multicore/ocaml-uring#34 Convert liburing to subtree

We now use a subtree instead of a submodule so that the ocaml-uring can be submitted to the opam-repository.

Parallel Programming in Multicore OCaml

ocaml-multicore/parallel-programming-in-multicore-ocaml#5 num_domains to num_additional_domains

The documentation and code examples have been updated to now use num_additional_domains instead of num_domains.
ocaml-multicore/parallel-programming-in-multicore-ocaml#6 Update latest information about compiler versions

The compiler versions in the README.md have been updated to use 4.12 and its variants.
ocaml-multicore/parallel-programming-in-multicore-ocaml#8 Nudge people to the default chunk_size setting

The recommendation is to use the default chunk_size when using parallel_for, especially when the number of domains gets larger.
ocaml-multicore/parallel-programming-in-multicore-ocaml#9 Eventlog section updates

The eventlog-tools library can now be used for parsing trace files since Multicore OCaml includes CTF tracing support from trunk. The relevant information has been updated in the README.md file.

Eio

The eio library provides an effects-based parallel IO stack for Multicore OCaml.

Additions
- ocaml-multicore/eio#41 Add eio.mli file
  
  A lib_eio/eio.mli file containing modules for Generic, Flow, Network, and Stdenv have been added to the repository.
- ocaml-multicore/eio#45 Add basic domain manager
  
  The PR allows you to run a CPU-intensive task on another domain, and adds a mutex to traceln to avoid overlapping output.
- ocaml-multicore/eio#46 Add Eio.Time and allow cancelling sleeps
  
  Use psq instead of bheap library to allow cancellations. The Eio.Time module has been added to lib_eio/eio.ml.
- ocaml-multicore/eio#53 Add Switch.sub_opt
  
  A new Switch.sub_opt implementation has been added to allow running a function with a new switch. Also, Switch.sub has been modified so that it is not a named argument.
- ocaml-multicore/eio#54 Initial FS abstraction
  
  A module Dir has been added to allow file system abstraction along with the ability to create files and directories. On Linux, it uses openat2 and RESOLVE_BENEATH.
- ocaml-multicore/eio#56 Add with_open_in, with_open_out and with_open_dir helpers
  
  The Eio.Dir module now contains a with_open_in, with_open_out and with_open_dir helper functions.
- ocaml-multicore/eio#58 Add Eio_linux.{readv, writev}
  
  The Eio_linux.{readv, writev} functions have been added to lib_eio_linux/eio_linux.ml which uses the new OCaml-Uring API.
- ocaml-multicore/eio#59 Add Eio_linux.noop and a simple benchmark
  
  A Eio_linux.noop implementation has been added for benchmarking Uring dispatch.
- ocaml-multicore/eio#61 Add generic Enter effect to simplify scheduler
  
  A Enter effect has been introduced to simplify the scheduler operations, and this does not have much effect on the noop benchmark as illustrated below:
Improvements
- ocaml-multicore/eio#38 Rename Flow.write to Flow.copy
  
  The code and documentation have been updated to rename Flow.write to Flow.copy for better clarity.
- ocaml-multicore/eio#36 Use uring for accept
  
  The enqueue_accept function now uses Uring.accept along with the effect Accept.
- ocaml-multicore/eio#37 Performance improvements
  
  Optimisation for Eunix.free and process completed events with Uring.peek for better performance results.
- ocaml-multicore/eio#48 Simplify Suspend operation
  
  The Suspend effect has been simplified by replacing the older Await and Yield effects with the code from Eio.
- ocaml-multicore/eio#52 Split Linux support out to eio_linux library
  
  eunix now has common code that is shared by different backends, and eio_linux provides a Linux io-uring backend. The tests and the documentation have been updated to reflect the change.
- ocaml-multicore/eio#57 Reraise exceptions with backtraces
  
  Added support to store a reference to a backtrace when a switch catches an exception. This is useful when you want to reraise the exception later.
- ocaml-multicore/eio#60 Simplify handling of completions
  
  The PR adds Job and Job_no_cancel in type io_job along with additional Log.debug messages.
Cleanups
- ocaml-multicore/eio#42 Merge fibreslib into eio
  
  The Fibreslib code is now merged with eio. You will now need to open Eio.Std instead of opening Fibreslib.
- ocaml-multicore/eio#47 Clean up the network API
  
  The network APIs have been updated with few changes such as renaming bind to listen, replacing Unix.shutdown_command with our own type in Eio API, and replacing Unix.sockaddr with a custom type.
- ocaml-multicore/eio#49 Remove Eio.Private.Waiters and Eio.Private.Switch
  
  The Eio.Private.Waiters and Eio.Private.Switch modules have been removed, and waiting is now handled using the Eio library.
- ocaml-multicore/eio#55 Some API and README cleanups
  
  The PR has multiple cleanups and documentation changes. The README.md has been modified to use Eio.Flow.shutdown instead of Eio.Flow.close, and a Time section has been added. The Eio.Network module has been changed to Eio.Net. The Time.now and Time.sleep_until methods have been added to lib_eio/eio.ml.
Documentation
- ocaml-multicore/eio#43 Add design note about determinism
  
  The README.md documentation has been updated with few design notes on Determinism.
- ocaml-multicore/eio#50 README improvements
  
  Updated README.md and added doc/prelude.ml for use with MDX.

Handling Cancellation

ocaml-multicore/eio#39 Allow cancelling accept operations

The PR now supports cancelling the server accept and read operations.
ocaml-multicore/eio#40 Support cancelling the remaining Uring operations

The cancellation request of connect, wait_readable and await_writable Uring operations is now supported.
ocaml-multicore/eio#44 Fix read-cancel test

The ENOENT value has been correctly fixed to use -2, and the documentation for cancelling the read request has been updated.
ocaml-multicore/eio#51 Getting EALREADY from cancel is not an error

Handle EALREADY case in lib_eunix/eunix.ml where an operation got cancelled while in progress.

Sundries

ocaml-multicore/eventlog-tools#2 Add a pausetimes tool

A eventlog_pausetimes tool has been added to eventlog-tools that takes a directory of eventlog files and computes the mean, max pause times, as well as the distribution up to the 99.9th percentiles. For example:
```
ocaml-eventlog-pausetimes /home/engil/dev/ocaml-multicore/trace3/caml-426094-* name
{
  "name": "name",
  "mean_latency": 718617,
  "max_latency": 33839379,
  "distr_latency": [191,250,707,16886,55829,105386,249272,552640,1325621,13312993,26227671]
}
```
ocaml-multicore/kcas#9 Backoff with cpu_relax

The Domain.Sync.{critical_section, wait_for} have now been replaced with Domain.Sync.cpu_relax, which matches the implementation with lockfree.
ocaml-multicore/retro-httpaf-bench#10 Add Eio benchmark

The Eio benchmark has now been added to the retro-httpaf-bench GitHub repository.
ocaml-multicore/retro-httpaf-bench#11 Do a recursive checkout in the CI build

The build_image.yml workflow has been updated to perform a recursive checkout of the submodules for the CI build.
domainslib#29 Task stealing with Chase Lev deques

The task-stealing Chase Lev deques for scheduling tasks across domains is now merged, and shows promising results on machines with 128 CPU cores.
ocaml-multicore/multicore-opam#55 Add 0.3.0 release of domainslib

The opam file for domainslib.0.3.0 has been added to the multicore-opam repository.

Benchmarking

Ongoing

ocaml-bench/sandmark-nightly#1 Cannot alter comparison input values

The Timestamp and Variant fields in the dropdown option in the parallel_nightly.ipynb notebook get reset when recomputing the whole workbook.
ocaml-bench/sandmark#230 Build for 4.13.0+trunk with dune.2.8.1

The ocaml-migrate-parsetree.2.2.0 and ppxlib.0.22.2 packages are now available for 4.13.0+trunk, and we are currently porting the Irmin Layers benchmark in Sandmark from using Irmin 2.4 to 2.6.
ocaml-bench/sandmark#231 View results for a set of benchmarks in the nightly notebooks

A feature request to filter the list of benchmarks when using the Sandmark Jupyter notebooks.
ocaml-bench/sandmark#233 Update pausetimes_multicore to fit with the latest Multicore changes

The pausetimes are now updated for both the 4.12.0 upstream and 4.12.0 Multicore branches to use the new Common Trace Format (CTF). The generated graphs for both the sequential and parallel pausetime results are illustrated below:
ocaml-bench/sandmark#235 Update selected benchmarks as a set for baseline benchmark

The baseline benchmark for comparison should only be one from the user selected benchmarks in the Jupyter notebooks.
ocaml-bench/sandmark#236 Implement pausetimes support in sandmark_nightly

The sequential and parallel pausetimes graph results need to be implemented in the Sandmark nightly Jupyter notebooks. The results are similar to the Figures 10 and 12 produced in the Retrofitting Parallelism ont OCaml, ICFP 2020 paper.
ocaml-bench/sandmark#237 Run sandmark_nightly on a larger machine

The testing of Sandmark nightly sequential and parallel benchmark runs have been done on a 24-core machine, and we would like to deploy the same on a 64+ core machine to benefit from the recent improvements to Domainslib.
ocaml-bench/sandmark#241 Switch to default Random module

An on-going discussion on whether to switch to using Random.State for the sequential Minilight, global roots micro-benchmarks and Evolutionary Algorithm.

Completed

ocaml-bench/sandmark#232 num_domains -> num_additional_domains

The benchmarks have been updated to now use num_additional_domains, to be consistent with the naming in Domainslib.
ocaml-bench/sandmark#239 Port grammatrix to Task pool

The Multicore Grammatrix benchmark has now been ported to use Domainslib Task pool. The time and speedup graphs are given below:

OCaml

Ongoing

ocaml/ocaml#10039 Safepoints

The PR is currently being testing and evaluated for both ARM64 and PowerPC architectures, in particular, the branch relaxations applied to Ipoll instructions.

Our thanks to all the OCaml users, developers and contributors in the community for their continued support to the project. Stay safe!

memprof-limits (first official release): Memory limits, allocation limits, and thread cancellation, with interrupt-safe resources

Archive: https://discuss.ocaml.org/t/ann-memprof-limits-first-official-release-memory-limits-allocation-limits-and-thread-cancellation-with-interrupt-safe-resources/8135/1

Guillaume Munch-Maccagnoni announced

Dear OCamlers, I am satisfied to announce the first official release of the package memprof-limits (v0.2.0).

From the user guide (which has detailed examples and use-cases):

This package lets you interrupt tasks in a thread-safe and resource-safe way, when a resource limit is reached or a cancellation token is set. A task is an isolated piece of computation running within a thread.

Global memory limits interrupt a task when the major heap exceeds a certain size. Allocation limits interrupt a task when a certain number of words have been allocated, a portable mesure of time elapsed and quantity of work done. Token limits interrupt an allocating task when an arbitrary token is set.

It is available on opam for OCaml ≥ 4.12.

The name comes from the fact that it uses OCaml's Memprof statistical profiler engine (to perform limit checks frequently-enough, related to the allocation rate of the program, and at the same time rarely-enough to not affect performance).

It comes with an implementation of masking that lets you define interrupt-safe resources that are guaranteed to be cleaned-up, and with them ensure that your program remains in a valid state after being interrupted.

To learn more about it, I recommend again the user guide.

Example

You can try it with OCaml 4.12 on the following example (opam install memprof-limits -> utop -> #require "memprof-limits";;):

A worker task allocates 300M words, only 3k of which live simultaneously. This comes close to an allocation limit of 330M words set by the monitor. The probability that the worker is interrupted is less than 10^-50, and thus the computation successfully completes, in about a second.

(* worker *)
let f () =
  let rec alloc n x =
    if n = 0 then x else alloc (n-1) (()::x)
  in
  (* allocate 300'000 kw *)
  for i = 0 to 100_000 do ignore (alloc 1_000 []) done

(* monitor *)
let g () =
  match Memprof_limits.limit_allocations ~limit:330_000L f with
  | Ok ((), n) -> Printf.printf "success (est. alloc. %#Lu kw)\n" n
  | Error _ -> print_endline "out of fuel"

(* main *)
let () =
  Memprof_limits.start_memprof_limits () ;
  g ()

Changes

Major changes to last year's experimental release:

The internal state is now protected against asynchronous exceptions arising from memprof callbacks (especially the own interrupt of memprof-limits), so it is no longer buggy (“experimental”).
And now it works under bytecode too.
Added token limits: interrupts that can be triggered at a distance.
Detailed guide and reference manuals, now generated by odoc.
Features to program with interrupt- and resource-safety in mind.
API revamp.

Bitwuzla 1.0.0 (SMT solver for AUFBVFP)

Archive: https://discuss.ocaml.org/t/ann-bitwuzla-1-0-0-smt-solver-for-aufbvfp/8138/1

Frédéric Recoules announced

On behalf of the bitwuzla team I am pleased to announce the release of bitwuzla ocaml binding (https://github.com/bitwuzla/ocaml-bitwuzla).

Bitwuzla (https://bitwuzla.github.io/) is a Satisfiability Modulo Theories (SMT) solver for the theories of fixed-size bit-vectors, floating-point arithmetic, arrays and uninterpreted functions and their combinations.

This ocaml binding comes in two flavors:

a straight low-level C binding Bitwuzla_c (for the brave and the unaware) – opam depext -i bitwuzla-c;
a type-safier OCaml API Bitwuzla – opam depext -i bitwuzla.

(Additionally, you can build bitwuzla standalone executable with opam depext -i bitwuzla-bin.)

Documentation and examples are available online (https://bitwuzla.github.io/docs/ocaml/).

Feel free to give it a try,

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.

Alan Schmitt

OCaml Weekly News

Table of Contents

slug 1.0.0 - URL-safe slug generator

Khoa Nguyen announced

ocaml-slug

Installation

Usage

Notice

License

Developer Experience Engineer at Jane Street

Yaron Minsky announced

Hardcaml MIPS CPU Learning Project and Blog

Clément asked and Alexander (Sasha) Skvortsov announced

OCaml for Windows installation confusion

Deep in this thread, Nicolás Ojeda Bär said

OCamlFormat config file auto-completion support in VSCode

тars announced

тars then added

Max Lantas replied

Multicore OCaml: June 2021

Anil Madhavapeddy announced

Ecosystem compatibility for 4.12.0+domains

4.12.0+domains+effects

Benchmarking and performance

Multicore OCaml

Ongoing

Completed

Enhancements

Upstream

Fixes

Cleanups

Sundries

Ecosystem

Ongoing

Completed

Ocaml-Uring

Parallel Programming in Multicore OCaml

Eio

Handling Cancellation

Sundries

Benchmarking

Ongoing

Completed

OCaml

Ongoing

memprof-limits (first official release): Memory limits, allocation limits, and thread cancellation, with interrupt-safe resources

Guillaume Munch-Maccagnoni announced

Example

Changes

Links

Bitwuzla 1.0.0 (SMT solver for AUFBVFP)

Frédéric Recoules announced

Old CWN