Previous week Up Next week

Hello

Here is the latest OCaml Weekly News, for the week of May 17 to 24, 2016.

  1. RPC for OCaml?
  2. Simple library to manipulate automata?
  3. rresult 0.4.0 // fmt 0.8.0 // logs 0.6.0
  4. Fpath 0.7.0
  5. More frequent pushes to Jane Street github repositories + janestreet/opam-repository
  6. opam-cross-windows
  7. Next OUPS meetup, 7th of June 2016
  8. Bos 0.1.0
  9. Topkg 0.7.0
  10. TyXML 4.0
  11. Ocaml Github Pull Requests
  12. Other OCaml News

RPC for OCaml?

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00151.html

rixed asked:
I'm thinking about implementing a library for doing RPC with OCaml, with large
scale environments in mind (à la Stubby but with better type checking of
course). I'm wondering what are the related libs I should make myself familiar
with before starting. I've seen a few interesting things for serialization
(piqi come to mind), some interesting event engines (LWT, Core), some protocol
implementations suitable for transport but no HTTP2, nothing to interface with
monitoring subsystems or TSDBs, nothing related to load balancing, routing,
DDoS detection, etc, some crypto, an interesting TLS implementation from
MirageOs, no OAuth or similar.

What other related projects should I look at?

Also, if anyone would be interested in contributing ideas, experience or code
please let me know.
      
Yaron Minsky suggested:
Async-RPC is perhaps worth looking at, though I agree it doesn't give
you much of what you want --- certainly, we don't to RPC over HTTP, we
do it over bog-standard TCP, and the protocol is very much
OCaml-specific, being based on bin-io.  That said, it might be useful
to look at for inspiration, in particular for how versioning is
handled in Versioned_rpc.  We do also have some kerberos support in
there as well, though I'm not sure that's in the open source release.
      
Jon Ludlam then said:
Mirage has a simple RPC generator that fits the same hole as Async-RPC,
which we use quite heavily in XenServer. It's camlp4 dependent right now,
which I'm keen to fix in the near future.

    http://github.com/mirage/ocaml-rpc 

I'll certainly be casting a careful eye over the Async-RPC versioning
support as our current versioning story is somewhat primitive.
      
Chet Murthy suggested:
Hi, I'm in the middle of building a nontrivial WAN-distributed system in Ocaml,
and have built a few distributed systems as part of research, in the past. I'd
like to urge you to try to use and improve one of the existing multiplatform RPC
systems that exist -- specifically Thrift or Protobuf3.

My last system used Thrift, and consisted in a Chubby clone, storage servers and
clients, and RPC over TCP and RoCEE. Some parts were in C++ for efficiency, and
some in Ocaml (for programmer-efficiency). It mattered to me, that I could get
really, really efficient (microseconds counted) RPC implementations for C++,
that were wireline compatible with Ocaml (so I could convert an Ocaml prototype
into a C++ program for speed).

The system I'm working on right now will require (in addition to Ocaml) Java,
Javascript, and Golang compatibility for the RPC substrate.

I think it's also important that you have stub-compilers and "standard" RPC
transports for Ocaml -- you don't want to be rolling your own stubs&skeletons
that you have to update as you change your IDL, when all the other languages'
stubs/skeletons get automatically generated.

If I had my druthers, I'd go with protobuf3. I've used protobuf2 at my previous
employer, and it was quite performant. But ocaml isn't supported. Someday, when
I have time, I'll fix that. But right now, that isn't high-priority.

Secondly, I don't know what the story is, on thrift-vs-protobuf3 performance.
Maybe it's great, GREAT! But it might not be, and there are at least reasons why
it could be worse. Specificaly protobuf3 was designed for use on the open
internet, not in more-controlled settings. These are issues that really involve
the transports, not the upper-layers of the RPC stack, but still, it's something
to be checked-out and verified carefully.

OK: so my own decision was: "use Thrift for now, but be ready to switch to
protobuf3 once it gets ocaml support (e.g., I write it) and I verify performance
parity".
      
Maxime Ransan then said:
Regarding protobuf you can use [1] if you want to generate OCaml from `.proto`
file or [2] if you want to use ppx extension to serialize OCaml types to
protobuf. 

Regaring [1], protobuf3 format is not fully supported but it could be improve.
Feel free to raise issues. 

[1] https://github.com/mransan/ocaml-protoc 
[2] https://github.com/whitequark/ppx_deriving_protobuf/
      

Simple library to manipulate automata?

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00177.html

Continuing this thread, David MENTRÉ said:
Privately somebody pointed me to FSM:
   http://udel.edu/~heinz/software/index.html#fsm

Thanks for all the links, I'll look at them.
      

rresult 0.4.0 // fmt 0.8.0 // logs 0.6.0

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00181.html

Daniel Bünzli announced:
A few updates: 

* rresult 0.4.0 // Result value combinators for OCaml
Homepage: http://erratique.ch/software/rresult
Release notes: https://github.com/dbuenzli/rresult/blob/v0.4.0/CHANGES.md

* fmt 0.8.0 // OCaml Format pretty-printer combinators
Homepage: http://erratique.ch/software/fmt
Release notes: https://github.com/dbuenzli/fmt/blob/v0.8.0/CHANGES.md

* logs 0.6.0 // Logging infrastructure for OCaml
Homepage: http://erratique.ch/software/logs
Release notes: https://github.com/dbuenzli/logs/blob/v0.6.0/CHANGES.md
      

Fpath 0.7.0

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00183.html

Daniel Bünzli announced:
It is my pleasure to announce the first release of Fpath, described as:

```
Fpath is an OCaml module for handling file system paths on POSIX and
Windows operating systems. Fpath processes paths without accessing
the file system and is independent from any system library.

Fpath depends on Astring and is distributed under the ISC license.
```

Homepage: http://erratique.ch/software/fpath
API docs: http://erratique.ch/software/fpath/doc/Fpath

Fpath claims to behave reasonably on Windows, in the sense that it is aware of
drives, UNC paths, etc. This is just a claim, Windows user are encouraged to
shout on the issue tracker if they feel some things are wrong or could be
improved.

Many thanks to David Sheets, whose insightful review and comments contributed to
make Fpath a much saner and usable library than I would have produced in the
first place; remaining misdesigns are mine.
      

More frequent pushes to Jane Street github repositories + janestreet/opam-repository

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00184.html

Jeremie Dimino announced:
We are currently trying to improve our workflow for releasing our code ​on github
​and following up on external contributions.

As a first step in this direction, w​e reworked​ our internal process for
​exporting​ the public release and we are now able to push to github more often.
​So going forward, instead of pushing one huge commit before every ​major release​
in the main opam repository​, we will push smaller commits more frequently.

This has several consequences:

- it will be easier to follow the latest changes
- accepted pull requests will appear in the git repository earlier
- you can complain if we break your code before the next release in opam...

​Initially the commit messages will still be meaningless and the changelogs might
be slightly lagging behind given that producing them still requires a fair
amount of work from a human.​ However, we are planning to improve this over time.

​Note that we'll continue to provide support for the latest version that is
released in opam.​

## ​janestreet/opam-repository

To help trying the development version of our libraries, we've setup an opam
repository that follows their latest development version [1]. So in order to try
out the latest version of core, you can do:

$ opam repo add git://github.com/janestreet/opam-repository.git
$ opam install core

​Pinning individual packages with `opam pin add --dev` might work as well, but
there is no guarantee about it​.

## 113.43+70

To start the new process, we just pushed the latest version of our code on
github. The version is 113.43+70 and includes a fair amount of changes.

[1] https://github.com/janestreet/opam-repository
      

opam-cross-windows

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00185.html

Deep in this thread, Mauricio Fernández announced:
FYI I have packaged several libs for opam-cross-windows and will soon submit
them for inclusion in the main repos, see

    https://github.com/mfp/opam-cross-windows-repos/tree/master/packages
      

Next OUPS meetup, 7th of June 2016

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00186.html

Continuing the thread from last week, Louis Roché announced:
Updated informations:

The meetup will take place at Mozilla Paris. The address is 16 Bis Boulevard
Montmartre Paris 75009. The access is controlled. You won't be allowed to join
the meetup if you are not registered.

The definitive list of talks:

- Danny Willems : Bindings OCaml à Cordova grâce à js_of_ocaml et gen_js_api
- Frédéric Bour : sturgeon — A toolkit for communicating with Emacs from OCaml
- Corentin De Souza : dead_code_analyzer — un détecteur de code mort pour OCaml

To register, or for more information, go here:

http://www.meetup.com/fr-FR/ocaml-paris/events/231068590/
      

Bos 0.1.0

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00190.html

Daniel Bünzli announced:
I'd like to announce the first release of bos:

```
Bos provides support for basic and robust interaction with the operating system
in OCaml. It has functions to access the process environment, parse command line
arguments, interact with the file system and run command line programs.

Bos works equally well on POSIX and Windows operating systems.

Bos depends on Rresult, Astring, Fmt, Fpath, Logs and the OCaml Unix library. 
It is distributed under the ISC license.
```

Homepage: http://erratique.ch/software/bos
API docs: http://erratique.ch/software/bos/doc/

Bos can be seen as improved Sys module for programs and scripts that have light
OS interaction requirements. Inspired by the excellent work of scsh [0], it
seems this path has been pursued more than once in OCaml for example with cash
[1] or shcaml [2].

Bos however has a slightly different take on this. It does not try to recover
the shell's terseness or processing model but rather tries to overcome the
shell's brittleness in face of errors, error reporting and insane quoting
conventions. The difference between a script and a program is an artificial one
and bos tries to encourage you to write programs that do not fail obscurely and
evolve gracefully from a quick, small one (a.k.a "script") to a complex one
(a.k.a. "program") while keeping good usability for the end user who is the one
who eventually gets processing failures in the face.

This first version number is intentionally low, as I don't see bos as fully
finished at the moment. Some interfaces could be tweaked and other added in the
future (feedback and discussion on the issue tracker is welcome). I did however
already write a few programs/scripts that make use of most of the features
provided by the library, so the low number should not be interpreted as "alpha"
quality software. What is provided here should work as described and if it
doesn't, scream on the issue tracker.

Best, 

Daniel


[0] https://scsh.net/
[1] http://pauillac.inria.fr/cash/
[2] http://users.eecs.northwestern.edu/~jesse/code/shcaml/
      
SP asked and Daniel Bünzli replied:
> What does this mean for Cmdliner? Does it overlap?
>
> [Cmdliner]: http://erratique.ch/software/cmdliner

The answer is in the documentation preamble of the OS.Arg module:  

http://erratique.ch/software/bos/doc/Bos.OS.Arg.html
      
Martin DeMello asked and Daniel Bünzli replied:
> Do you think this would be a good route towards getting ocamlbuild working
> seamlessly on windows?

Well bos does not do any kind of magic to work well on windows except avoiding
what needs to be avoided, namely fork(2). Contrary to what it's name suggests
the Unix library is not a bad OS abstraction layer and works quite well on
windows; even more so with the work David Allsopp put into 4.03.

At the moment, but it may change in the future, bos is only a more lighter and
simpler way to work with the Unix library correctly. Unix programming can be
tricky and quite verbose, e.g. to setup program pipelines. Also note that bos'
simpler interface may be not be suitable for all programs, i.e. bos doesn't
replace Unix.

IIRC (Gabriel may want to comment) to get ocamlbuild working seamlessly on
windows one needs to get rid of fork, a few pointless shellouts and make people
stop use Sh command specs in their plugins. So I don't think bos necessarily
helps here, using Unix conservatively in ocamlbuild should do.
      
Gabriel Scherer then said:
There are some parts of the ocamlbuild internals that have their own
implementations of file path handling and quoting/escaping, and that
are a bit too arcane to my taste. (I tend to not change them because
they mostly work well, but they come back to bite us once in a while.)
When I got the announcement for Bos, I indeed thought that it could be
a nice idea to have a look. (I probably won't do it myself in the
short term, my development time is very constrained right now; plus
the fact that I cannot test on Windows myself makes me wary of
changing those parts of the codebase.)
      
Adrien Nader added:
It's not really fork but Unix.open_process*.

The signatures and documentation sum up the issue:
  val open_process_in : string -> Pervasives.in_channel
  [...] The command is interpreted by the shell /bin/sh (cf. system).

As such, the process command-line needs to be properly quoted. I'm not
going to attempt to get the whole logic right so I'll only give keywords
for things that can be involved: /bin/sh, $SHELL, quoting for posix
shells, quoting for windows command-line (entirely different and 200%
ad-hoc over the course of 20 years), cmd.exe, %COMSPEC% (and probably
more things).

I see the use of a single string as the main issue because it makes
supporting the quoting for windows process invocations very difficult
(the entry point on windows gets a single string with holds every
process argument: argv in main() is actually the result of parsing...).

When spawning with a string array or string list, one isn't usually
concerned about shell escapes: only special characters such as !, ':' or
'&' might matter but the last really do not make sense in this context..

This makes it possible to quote each argument separately before merging
everything (with quoting) and call the process (NB: there are also
_exec* functions in the Microsoft CRT that could help).
I consider moving away from single-string arguments a prerequisite for
proper and sane handling of this. One annoyance for ocamlbuild is that
it exposes that same API to its plugins.

Support for wide-character functions is another change needed for better
Windows support in the OCaml ecosystem. If this has to be done with API
changes or new APIs, maybe it will be the right time to also remove
support (or not offer them at all in the new APIs) for functions that
use a single string to spawn functions.

Back to the topic, as Gabriel said, there is a number of useful
functions in ocamlbuild (see My_Unix for instance iirc) which work
fairly well and are therefore risky changes (that's actually pretty
funny logic). Nevertheless, I don't think they should stay there.
      

Topkg 0.7.0

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00191.html

Daniel Bünzli announced:
Hello (last one today, promised),

I'd like to announce the first release of topkg. For those who know topkg, it is
no longer an embedded script but a library that you add as a build dependency to
your package. It now also comes with an optional tool that helps you to manage
your package and make swift and correct releases.

Bla bla bla:
```
Topkg is a packager for distributing OCaml software. It provides an API to
describe the files a package installs in a given build configuration and to
specify information about the package's distribution creation and publication
procedures.

The optional topkg-care package provides the `topkg` command line tool which
helps with various aspects of a package's life cycle: creating and linting a
distribution, releasing it on the WWW, publish its documentation, add it to the
OCaml OPAM repository, etc.

Topkg is distributed under the ISC license and has **no** dependencies. This is
what your packages will need as a *build* dependency.

Topkg-care is distributed under the ISC license and depends on fmt, logs, bos,
cmdliner and opam-lib.
```

Install: opam install topkg topkg-care
Homepage: http://erratique.ch/software/topkg
API docs & manual: http://erratique.ch/software/topkg/doc/Topkg.html#basics
See also `topkg help release` and `topkg help`

A few things worth noting:

* Topkg pretends to work on Windows (provided your build system does) but early
users may need to give a hand, please do report issues if things go wrong.

* Topkg pretends to be able to work with hg source repositories, the same as for
Windows applies.

* Topkg should be able to work with other build systems than ocamlbuild. But a
few things (e.g. documentation support or dependency linting) do at the moment
rely on ocamlbuild being used. It's not my priority but I'm not against
improving it to make it easier to use with other build systems.

* Topkg only supports OCaml from 4.01 on and hence will also impose this limit
on your packages.

* Topkg has toy, proof of concept, support to publish documentation and
distributions via GitHub's pages and release system. I'm personally not using
this so do not hesitate to report broken or suboptimal behaviour. Better support
— especially gh issue interaction for 'topkg issue' and better authentication
would be nice, but is left in the hands of a motivated github user. Someone
should develop and distribute a good github topkg delegate (e.g. using
ocaml-github) that github users can install; see 'toy-github-topkg-delegate
--help' and 'topkg help delegate'; the source of the toy delegate [1] can be
used as a blueprint. Do not hesitate to get in touch for more information and/or
design discussion.

Topkg may not be the final word in OCaml building & packaging, however,
depending on your needs, topkg's release workflow (see 'topkg help release') may
come as a significant improvement over the current status quo.

I have been using a similar but more ad-hoc system with my packages for a few
years now since I realized that oasis was getting in my way to scale modularity.
Except for the maintainability problem of having an embedded script which is now
gone with this new incarnation of topkg, it has already been shown to work quite
well in practice for me and a limited number of other persons, in a large range
of software install scenarios, YMMV.

Best,

Daniel

P.S. If you are on github and start to use `topkg` you can mention me
(@dbuenzli) in the inaugural PR or commit, I'm interested to see how it is used
and misused.

[1] https://github.com/dbuenzli/topkg/blob/v0.7.0/src-bin/toy_github_delegate.ml
      

TyXML 4.0

Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00193.html

Drup announced:
It is with great pleasure that we are announcing the release of TyXML 4.0.0. The
major features of this new release are a new PPX syntax extension that allows to
use the standard HTML syntax and an improved user experience for both old and
new TyXML users.

TyXML is a library for building statically correct HTML5 and SVG documents. It
provides a set of combinators which use the OCaml type system to ensure the
validity of the generated document. TyXML’s combinators can be used to build
textual HTML and SVG, but also DOM trees or reactive interfaces, using Eliom and
Js_of_ocaml.

A presentation of the new release is available on the ocsigen blog and the
changelog is available on github.

Happy HTML and SVG hacking!
      

Ocaml Github Pull Requests

Gabriel Scherer compiled this list:
Here is a sneak peek at some potential future features of the Ocaml
compiler, discussed by their implementers in these Github Pull Requests.

Spacetime: a new memory profiler
https://github.com/ocaml/ocaml/pull/585
      

Other OCaml News

From the ocamlcore planet blog:
Here are links from many OCaml blogs aggregated at OCaml Planet,
http://ocaml.org/community/planet/.

ppx_core: context-free rewriters for better semantics and faster compilation
 https://blogs.janestreet.com/ppx_core-context-free-rewriters-for-better-semantic-and-faster-compilation/

Thirteenth OCaml compiler hacking evening at Pembroke College
 http://ocamllabs.github.com/compiler-hacking/2016/05/20/spring-compiler-hacking

TyXML 4.0.0
 https://ocsigen.github.io/blog/2016/05/20/tyxml4/
      

Old cwn

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.


Alan Schmitt