OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of October 27 to November 03, 2020.

Table of Contents

Brr 0.0.1, a toolkit for programming browsers

Archive: https://discuss.ocaml.org/t/ann-brr-0-0-1-a-toolkit-for-programming-browsers/6608/9

Continuing this thread, Daniel Bünzli said

One thing I forgot, is that there is a todomvc example in the repo, see todomvc.{html,ml} in this directory.

It doesn't use the UI toolkit you mentioned, just the basic reactive DOM support provided by Brr_note and Brr_note_kit. But you can see how quickly you get reusable and composable components like bool_editor and string_editor.

The program structure in that example is quite similar to the one I had in the drawing app. You define a purely functional, non reactive data model, actions over the data model, create small UI fragments that renders parts of your data model and generate actions events for it, gradually glue them together using note combinators and finally define a fixed point signal that holds the data model as massaged by the actions events of your UI (as mentioned I'd like to replace fix points by direct let rec and a lazy infinitesimal delay combinator).

There are a few pitfalls like you should avoid retaining parts of your data model in the UI otherwise you could get outdated data come back in your model (makes for very fun and spooky bugs though). Identity in the data model is also a bit tricky, it seems in todomvc I used ==. That didn't work in the drawing app where my surfaces had properties that could be updated but they could also be linked toghether (that window belongs to that wall etc.) so I needed stable identifiers for which I introduced a little abstraction to identify values and define relations between them.

One thing I remember fondly when doing the drawing app is that I would still get the odd interaction glitches you get when coding direct mouse manipulation interactions (surface definition/selection/move/transform) however thanks to the ability to denotationally reason and act (left leaning E.select) on the simultaneity of events, they were easy to understand and fix in an explicit way (that is via a defining expression).

Also if you get into Note the denotational semantics notation is not yet explained there, refer to the one of react it's the same.

Yoann Padioleau asked and Daniel Bünzli replied

How hard would it be to build on top of Brr_note something like an Elm Architecture-style toolkit? I know there's a TEA-Bucklescript library, but I'd rather use something relying on dune/jsoo.

I've read somewhere else that you were a bit skeptical about the advantage of MVU (movel-view-update) over MVC, but I personnaly find the counter UI example in ELM at https://guide.elm-lang.org/architecture/buttons.html far simpler than the corresponding one in Brr at https://github.com/barko/brr-eg/blob/master/counter/counter.ml

I don't know. I didn't look into MVU too much, but to me it's largely a remix of MVC – despite what its proponents try to tell you. Since we now live in an age of software adverstising it's a bit hard to get frank assessments.

As far as I'm concerned the compositionality story of MVU doesn't look great. Basically it enforces state machines on you, and composing state machines is a bit meh. In FRP state machines become signals (via S.accum) which are highly composable entities with fine granularity (and bonus point, a well defined denotational semantics for equational reasoning).

If you are looking for MVU I think you can simply jump on LexiFI's vdom. But when I see how you get to compose two models in that paradigm, I'm not convinced.

There’s no need for those E.select. The UI is IMHO more declarative in ELM.

That example could be rewritten (I didn't write the examples in this repo) to be more like the ELM one in it's declarations.

But I think the ELM example is also more rigid. You may not like that E.select on this toy example, but you may get to enjoy it you when you start composing larger systems from smaller components.

Yaron Minsky then said

You might be interested in Bonsai! At some level, you can think of it as a library for building composable state machines. It uses Incremental as its engine for incrementalizing the computation of views, with a virtual-dom implementation underneath.

https://github.com/janestreet/bonsai

It's the primary tool we use for building UIs inside of Jane Street.

In some ways, Bonsai is like Elm, but it has its own interesting ideas. Some of the concepts are borrowed from this paper:

https://www.cl.cam.ac.uk/~jdy22/papers/the-arrow-calculus.pdf

though I won't pretend to understand this paper myself!

Bonsai doesn't yet have enough public-facing documentation, and really the bleeding edge version on github is considerably better and more usable than the one released into opam. But there's at least one public-facing UI that's built with it, if you want a real-world example.

https://blog.janestreet.com/finding-memory-leaks-with-memtrace/

Yoann Padioleau replied

Thx for the links!

The memtrace viewer example is pretty cool, but Bonsai looks far more complicated than ELM. If you look at the counter example (the hello world of UI), here: https://github.com/janestreet/bonsai/blob/master/examples/counters/lib/bonsai_web_counters_example.ml

and you compare it to the one in ocaml-vdom (thx @dbuenzli for the link) at https://github.com/LexiFi/ocaml-vdom/blob/master/examples/counters/counters.ml

there's a huge difference in simplicity.

Ty Overby then said

Hi Aryx, I wrote the Bonsai example that you linked, and it certainly isn't the most concise, but that's because it was built for a tutorial on building small components (one counter is a single component), how to use more advanced combinators (Bonsai.assoc), and how to move data from one component to another (the add_counter_component into the associated counters component.) I think it's a great example of the power of structuring an UI as a DAG rather than a tree, but it definitely doesn't make for the most concise code!

In the example, the comments that look like "CODE_EXCERPT_BEGIN" are actually preprocessor definitions that are used in the (honestly, kinda out of date) tutorial here. A bonsai app that wasn't written for such a tutorial would look more like this.

New release of Monolith (20201026)

Archive: https://discuss.ocaml.org/t/ann-new-release-of-monolith-20201026/6667/1

François Pottier announced

It is my pleasure to announce a major new release of Monolith. 

opam update && opam install monolith

Monolith offers facilities for testing an OCaml library (for instance, a data structure implementation) by comparing it against a reference implementation. It can be used to perform either random testing or fuzz testing. Fuzz testing relies on the external tool afl-fuzz.

More information on Monolith is available here and in the draft paper Strong Automated Testing of OCaml Libraries.

MirageOS 3.9.0 released

Archive: https://discuss.ocaml.org/t/ann-mirageos-3-9-0-released/6668/1

Martin Lucina announced

We are pleased to announce the release of MirageOS 3.9.0.

Our last release announcement was for MirageOS 3.6.0, so we will also cover changes since 3.7.x and 3.8.x in this announcement.

New features:

  • The Xen backend has been re-written from scratch to be based on Solo5, and now supports PVHv2 on Xen 4.10 or higher, and QubesOS 4.0.
  • As part of this re-write, the existing Mini-OS based implementation has been retired, and all non-UNIX backends now use a unified OCaml runtime based on ocaml-freestanding.
  • OCaml runtime settings settable via the OCAMLRUNPARAM environment variable are now exposed as unikernel boot parameters. For details, refer to #1180.

Security posture improvements:

  • With the move to a unified Solo5 and ocaml-freestanding base MirageOS unikernels on Xen gain several notable improvements to their overall security posture such as SSP for all C code, W^X, and malloc heap canaries. For details, refer to the mirage-xen 6.0.0 release announcement.

API breaking changes:

  • Several Xen-specific APIs have been removed or replaced, unikernels using these may need to be updated. For details, refer to the mirage-xen 6.0.0 release announcement.

Other notable changes:

  • Mirage_runtime provides event loop enter and exit hook registration (#1010).
  • All MirageOS backends now behave similarly on a successful exit of the unikernel: they call exit with the return value 0, thus at_exit handlers are now executed (#1011).
  • The unix backend used a toplevel exception handler, which has been removed. All backends now behave equally with respect to exceptions.
  • Please note that the Mirage_net.listen function still installs an exception handler, which will be removed in a future release. The out of memory exception is no longer caught by Mirage_net.listen (#1036).
  • To reduce the number of OPAM packages, the mirage-*-lwt packages are now deprecated. Mirage_net (and others) now use Lwt.t directly, and their buffer type is Cstruct.t (#1004).
  • OPAM files generated by mirage configure now include opam build and installation instructions, and also an URL to the Git origin (#1022).

Known issues:

  • mirage configure fails if the unikernel is under version control and no origin remote is present (#1188).
  • The Xen backend has issues with event delivery if built with an Alpine Linux GCC toolchain. As a work-around, please use a Fedora or Debian based toolchain.

Acknowledgements:

  • Thanks to Roger Pau Monné, Andrew Cooper and other core Xen developers for help with understanding the specifics of how Xen PVHv2 works, and how to write an implementation from scratch.
  • Thanks to Marek Marczykowski-Górecki for help with the QubesOS specifics, and for forward-porting some missing parts of PVHv2 to QubesOS version of Xen.
  • Thanks to @palainp on Github for help with testing on QubesOS.

An AST typing problem

Archive: https://discuss.ocaml.org/t/an-ast-typing-problem/3677/8

Chet Murthy announced

This note discusses the beginnings of an OCaml attribute-grammar evaluator generator. You can find this code on github at camlp5/pa_ppx_ag.

All of this code is implemented using camlp5 and the pa_ppx suite of PPX rewriters.

Caveat: this code is less than a week old, so it's changing fast. In the unlkely event that anybody out there is actually interested in using this code, I'm happy to help in any way I can. But just be aware that it's changing -really- fast.

Attribute Grammars for the multipass AST analysis problem

A year-and-a-half ago, the OP "An AST Typing Problem" (https://discuss.ocaml.org/t/an-ast-typing-problem/3677) raised the problem of how to deal with ASTs, in the presence of multiple passes of program-analysis, each of which will want to hang various bits of data off nodes. The author of the OP pointed also at a couple of posts on Lambda-the-Ultimate (LtU), discussing related problems.

The author notes:

There’s a lot of passes, many of which depend on the previous ones, each one making some slight change to the AST which might or might not result in having to walk through the whole AST to catch all occurrences of that particular node. Clearly you’ll want to encode semantic errors in the types, so each pass ends up having its own unique AST, each depending on the previous one. To change a single node deep in the AST I have to write about a hundred lines of types and mapping functions’ worth of boilerplate. Any change in the lower levels of the AST bubbles up to the higher ones, and refactoring becomes a nightmare.

I've been thinking about this problem ever since, and at the time, had suggested that while it seemed like attribute-grammars might be a workable solution, they were a pretty heavy hammer. It doesn't help (of course) that there exist no attribute-grammar evaluator generators, for OCaml. Also, at least in the LtU threads, there was discussion of modifying the AST, and having the analyses automatically be updated for the modified AST. Obviously this would require an incremental re-attribution algorithm: more complexity and again, something that isn't implemented for OCaml.

But imagine that there existed an attribute-grammar evaluator generator for OCaml. So for a simple language of expressions, with an assignment-operator, we could write an evaluator as an attribute-grammar. Imagine that you could write an ast like this (test1_ast.ml): 

type expr =
    INT of int
  | BINOP of binop * expr * expr
  | UNOP of unop * expr
  | REF of string
  | ASSIGN of string * expr
  | SEQ of expr * expr
and unop = UPLUS | UMINUS
and binop = PLUS | MINUS | STAR | SLASH | PERCENT
and prog = expr

and then (having elsewhere written parser/pretty-printer) declare attributes on those types (test1_variants.ml): 

module Attributed = struct
  [%%import: Test1_ast.expr]
  [@@deriving attributed {
    attributed_module_name = AT
  ; normal_module_name = OK
  ; attributes = {
      expr = {
        inh_env = [%typ: (string * int) list]
      ; syn_env = [%typ: (string * int) list]
      ; value_ = [%typ: int]
      }
    ; prog = {
        value_ = [%typ: int]
      }
    ; binop = {
        oper = [%typ: int -> int -> int]
      }
    ; unop = {
        oper = [%typ: int -> int]
      }
    }
  }]
end

and then declare attribute equations (test1_ag.ml): 

module REC = struct
[%%import: Test1_variants.Attributed.AT.expr]
  [@@deriving ag {
    module_name = AG
  ; storage_mode = Records
  ; axiom = prog
  ; attributes = {
      expr = {
        inh_env = [%typ: (string * int) list]
      ; syn_env = [%typ: (string * int) list]
      ; value_ = [%typ: int]
      }
    ; prog = {
        value_ = [%typ: int]
      }
    ; binop = {
        oper = [%typ: int -> int -> int]
      }
    ; unop = {
        oper = [%typ: int -> int]
      }
    }
  ; attribution = {
      expr__INT = (
        [%nterm 0].syn_env := [%nterm 0].inh_env ;
        [%nterm 0].value_ := [%prim 1].intval
      )
    ; expr__BINOP = (
        [%nterm expr.(1)].inh_env := [%nterm expr].inh_env ;
        [%nterm expr.(2)].inh_env := [%nterm expr.(1)].syn_env ;
        [%nterm expr].syn_env := [%nterm expr.(2)].syn_env ;
        [%nterm expr].value_ := [%nterm binop.(1)].oper [%nterm expr.(1)].value_ [%nterm
expr.(2)].value_
      )
    ; expr__UNOP = (
        [%nterm expr.(1)].inh_env := [%nterm expr].inh_env ;
        [%nterm expr].syn_env := [%nterm expr.(1)].syn_env ;
        [%nterm expr].value_ := [%nterm unop.(1)].oper [%nterm expr.(1)].value_
      )
    ; expr__REF = (
        [%nterm 0].syn_env := [%nterm 0].inh_env ;
        [%nterm 0].value_ := List.assoc [%prim 1].stringval [%nterm 0].inh_env
      )
    ; expr__ASSIGN = (
        [%nterm 0].syn_env := ([%prim 1].stringval, [%nterm expr.(1)].value_) :: [%nterm
expr.(1)].syn_env ;
        [%nterm expr.(1)].inh_env := [%nterm 0].inh_env ;
        [%nterm 0].value_ := [%nterm expr.(1)].value_
      )
    ; expr__SEQ = (
        [%nterm 1].inh_env := [%nterm 0].inh_env ;
        [%nterm 2].inh_env := [%nterm 1].syn_env ;
        [%nterm 0].syn_env := [%nterm 2].syn_env ;
        [%nterm 0].value_ := [%nterm 2].value_
      )
    ; prog = (
        [%nterm 1].inh_env := [] ;
        [%nterm 0].value_ := [%nterm 1].value_ ;
        assert True
      )
    ; unop__UPLUS = (
        [%nterm unop].oper := fun x -> x
      )
    ; unop__UMINUS = (
        [%nterm unop].oper := fun x -> (- x)
      )
    ; binop__PLUS = (
        [%nterm binop].oper := (+)
      )
    ; binop__MINUS = (
        [%nterm binop].oper := (-)
      )
    ; binop__STAR = (
        [%nterm binop].oper := fun a b -> a*b
      )
    ; binop__SLASH = (
        [%nterm binop].oper := (/)
      )
    ; binop__PERCENT = (
        [%nterm binop].oper := (mod)
      )
    }
  }]
end

and then, turning a crank, you would get an evaluator: 

let test_records ctxt =
  assert_equal 3 ({| x := 1 ; x ; y := 2 ; x + y |} |> pa_prog_attributed |> REC.AG.evaluate)
; assert_equal 0 ({| x := 1 ; y := 2 ; x / y |} |> pa_prog_attributed |> REC.AG.evaluate)

where pa_prog_attributed is a parser that parses the surface syntax into an AST, which has empty slots for all attributes, and REC.AG.evaluate evaluates attributes in its argument AST, and then returns a tuple of all the synthesized attributes of the root node.

Retaining familiar surface syntax for pattern-matching and constructing ASTs

Now, we don't want to give up easy pattern-matching and construction of the AST, just because the AST has attributes strewn throughout it. But we don't have to: with Camlp5's "quotations", once we define a surface syntax parser for the basic AST (unadorned with attributes – viz. test1_ast.ml), we can use that to bootstrap ourselves to a surface syntax parser for expressions and patterns over that AST, and then in a similar manner we can get them for the AST adorned with attributes.

This has already been done for hashconsed ASTs, and ASTs with built-in unique-IDs, and and doing it for "attributed ASTs" isn't any harder. Those examples can be found in the github project camlp5/pa_ppx_q_ast.

Limitations

There are still limitations.

  1. The current code only implements topological-order evaluation. That is, it builds the entire dependency-graph, topologically-sorts it, and then evaluates attributes. This is …. suboptimal, when we well know that almost all interesting AGs are already in the class of ordered attribute-grammars (OAGs). I plan to implement the OAG evaluation strategy next.

  2. Traditionally AGs are defined over "productions" which are sequences of nonterminals and terminals. This doesn't correspond to the way we define OCaml constructor data-types. So instead of a constructor like 

    type expr =
  ... | Call of name * arg_list
and arg_list = NoArgs | SomeArgs of expr * arg_list

    we might want to use ~ 'a list~ 

    type expr =
  ... | Call of name * expr list

    Problem is: defining attribute-equations for (effectively) an array of nodes, is not part of the standard lingo of AGs. But I believe we can invent new syntax and make this succinct.

  3. Storage optimization. A naive implementation of AGs can store all attributes ever computed, at all the nodes in the AST. This can use a lot of memory. But there are well-known techniques to discard attributes once they'll never more be needed in the rest of the attribute-evaluation, and I plan to implement these techniques.

There's an entire literature on things like remote-references in attribute grammars, aggregates, and other things, all of which can probably be usefully employed.

Conclusion

I think that attribute-grammars could be a useful way to structure complex multipass program-analysis, just as they used to do back in the good ol' days.

Maybe worth a look-see!

erlang 0.0.14, a toolkit to manipulate Erlang sources

Archive: https://discuss.ocaml.org/t/ann-erlang-0-0-14-a-toolkit-to-manipulate-erlang-sources/6694/1

ostera announced

Hej, hope you're staying safe :raised_hands:

I'm excited to share with you the first release of erlang.

tl;dr: parser/lexer/ast/printer for Erlang

Description

erlang is a toolkit for manipulating Standard Erlang and Core Erlang sources and their abstract syntax trees according to the Erlang specifications.

Version 0.0.14 provides:

  • A lexer/parser written in Menhir for Standard Erlang
  • ASTs for Core Erlang and Standard Erlang
  • An AST helper module for constructing Standard Erlang programs
  • A printer for the Standard Erlang AST (of highly volatile prettiness)
  • Support to turn ASTs to S-expressions
  • erldump, a binary tool for reading Erlang sources and printing their concrete syntax trees as S-expressions.

It is distributed under Apache-2.0 license, depends on Menhir and Cmdliner, and it is being developed as part of the Caramel project.

I started writing erlang to let Caramel do an entirely symbolic compilation from the OCaml typedtree that would still allow for other passes/checks to be made cleanly. It's come with a decent number of tests, and it can parse some OTP modules with small modifications.

There's a few outstanding issues regarding the parsing for the next release, but it should be a starting point for anyone wanting to read sources and do something with them. I plan on cover these issues in the rest of the year, but as with all open source, it may take longer.

I'd like to add a few other things, like an AST invariants module to check that ASTs are actually valid Erlang programs, and transformations more suitable for static analyses of the sources.

My thanks go to @antron, @c-cube, @Drup, @rgrinberg, and @mseri for helping me get around the OCaml compiler, Menhir, and eventually to get this version split from Caramel and released independently. Also a shoutout to the Js_of_ocaml project that served as a starting point for the parser/lexer work here.

If you can give me some feedback on the design and implementation, I'd very much like to hear your thoughts :slight_smile:

For those of you hoping to start using it, do not let it crash.

opam-bin.1.0.0: binary packages for opam

Archive: https://discuss.ocaml.org/t/ann-opam-bin-1-0-0-binary-packages-for-opam/6696/1

Fabrice Le Fessant announced

I am happy to announce the first stable release of opam-bin, version 1.0.0, a framework to CREATE, USE and SHARE binary relocatable packages with opam, to speed-up installation of packages. It is easily installable from opam-repository, and available on Github:

https://ocamlpro.github.io/opam-bin

With opam-bin, you can :

  • build binary packages while installing their source counterpart with opam
  • automatically reuse previously created binary packages instead of compiling them again
  • export and share your binary packages as part of opam repositories for other users/computers to use

opam-bin is a framework in 3 parts :

This is the first stable release:

  • Specific support has been added in the current master branch of opam to make working with this version more convenient, by printing pre- and post- installation messages. Yet, it will still work with previous version of opam, but with no output on the terminal when calling opam.
  • The sharing option can be enabled to share files with hard-links between switches, making the creation of new local switches almost costless in time and disk space.

opam-bin is a collaborative work between OCamlPro and Origin Labs.

opam-bin is particularly useful if you create many local switches, as they become unexpensive. Tools like Drom (an OCaml project scaffolder, https://ocamlpro.github.io/drom) can take advantage of that to provide a cargo-like experience.

Interesting OCaml Articles

Archive: https://discuss.ocaml.org/t/interesting-ocaml-articles/1867/63

Ryan Slade announced

Anyone who's been following this blog probably saw this coming:

https://blog.darklang.com/leaving-ocaml/

It's an interesting read and hopefully can be used as constructive criticism in order to improve the state of the OCaml ecosystem.

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.

Alan Schmitt