Hello
Here is the latest OCaml Weekly News, for the week of May 17 to 24, 2016.
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00151.html
rixed asked:I'm thinking about implementing a library for doing RPC with OCaml, with large scale environments in mind (à la Stubby but with better type checking of course). I'm wondering what are the related libs I should make myself familiar with before starting. I've seen a few interesting things for serialization (piqi come to mind), some interesting event engines (LWT, Core), some protocol implementations suitable for transport but no HTTP2, nothing to interface with monitoring subsystems or TSDBs, nothing related to load balancing, routing, DDoS detection, etc, some crypto, an interesting TLS implementation from MirageOs, no OAuth or similar. What other related projects should I look at? Also, if anyone would be interested in contributing ideas, experience or code please let me know.Yaron Minsky suggested:
Async-RPC is perhaps worth looking at, though I agree it doesn't give you much of what you want --- certainly, we don't to RPC over HTTP, we do it over bog-standard TCP, and the protocol is very much OCaml-specific, being based on bin-io. That said, it might be useful to look at for inspiration, in particular for how versioning is handled in Versioned_rpc. We do also have some kerberos support in there as well, though I'm not sure that's in the open source release.Jon Ludlam then said:
Mirage has a simple RPC generator that fits the same hole as Async-RPC, which we use quite heavily in XenServer. It's camlp4 dependent right now, which I'm keen to fix in the near future. http://github.com/mirage/ocaml-rpc I'll certainly be casting a careful eye over the Async-RPC versioning support as our current versioning story is somewhat primitive.Chet Murthy suggested:
Hi, I'm in the middle of building a nontrivial WAN-distributed system in Ocaml, and have built a few distributed systems as part of research, in the past. I'd like to urge you to try to use and improve one of the existing multiplatform RPC systems that exist -- specifically Thrift or Protobuf3. My last system used Thrift, and consisted in a Chubby clone, storage servers and clients, and RPC over TCP and RoCEE. Some parts were in C++ for efficiency, and some in Ocaml (for programmer-efficiency). It mattered to me, that I could get really, really efficient (microseconds counted) RPC implementations for C++, that were wireline compatible with Ocaml (so I could convert an Ocaml prototype into a C++ program for speed). The system I'm working on right now will require (in addition to Ocaml) Java, Javascript, and Golang compatibility for the RPC substrate. I think it's also important that you have stub-compilers and "standard" RPC transports for Ocaml -- you don't want to be rolling your own stubs&skeletons that you have to update as you change your IDL, when all the other languages' stubs/skeletons get automatically generated. If I had my druthers, I'd go with protobuf3. I've used protobuf2 at my previous employer, and it was quite performant. But ocaml isn't supported. Someday, when I have time, I'll fix that. But right now, that isn't high-priority. Secondly, I don't know what the story is, on thrift-vs-protobuf3 performance. Maybe it's great, GREAT! But it might not be, and there are at least reasons why it could be worse. Specificaly protobuf3 was designed for use on the open internet, not in more-controlled settings. These are issues that really involve the transports, not the upper-layers of the RPC stack, but still, it's something to be checked-out and verified carefully. OK: so my own decision was: "use Thrift for now, but be ready to switch to protobuf3 once it gets ocaml support (e.g., I write it) and I verify performance parity".Maxime Ransan then said:
Regarding protobuf you can use [1] if you want to generate OCaml from `.proto` file or [2] if you want to use ppx extension to serialize OCaml types to protobuf. Regaring [1], protobuf3 format is not fully supported but it could be improve. Feel free to raise issues. [1] https://github.com/mransan/ocaml-protoc [2] https://github.com/whitequark/ppx_deriving_protobuf/
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00177.html
Continuing this thread, David MENTRÉ said:Privately somebody pointed me to FSM: http://udel.edu/~heinz/software/index.html#fsm Thanks for all the links, I'll look at them.
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00181.html
Daniel Bünzli announced:A few updates: * rresult 0.4.0 // Result value combinators for OCaml Homepage: http://erratique.ch/software/rresult Release notes: https://github.com/dbuenzli/rresult/blob/v0.4.0/CHANGES.md * fmt 0.8.0 // OCaml Format pretty-printer combinators Homepage: http://erratique.ch/software/fmt Release notes: https://github.com/dbuenzli/fmt/blob/v0.8.0/CHANGES.md * logs 0.6.0 // Logging infrastructure for OCaml Homepage: http://erratique.ch/software/logs Release notes: https://github.com/dbuenzli/logs/blob/v0.6.0/CHANGES.md
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00183.html
Daniel Bünzli announced:It is my pleasure to announce the first release of Fpath, described as: ``` Fpath is an OCaml module for handling file system paths on POSIX and Windows operating systems. Fpath processes paths without accessing the file system and is independent from any system library. Fpath depends on Astring and is distributed under the ISC license. ``` Homepage: http://erratique.ch/software/fpath API docs: http://erratique.ch/software/fpath/doc/Fpath Fpath claims to behave reasonably on Windows, in the sense that it is aware of drives, UNC paths, etc. This is just a claim, Windows user are encouraged to shout on the issue tracker if they feel some things are wrong or could be improved. Many thanks to David Sheets, whose insightful review and comments contributed to make Fpath a much saner and usable library than I would have produced in the first place; remaining misdesigns are mine.
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00184.html
Jeremie Dimino announced:We are currently trying to improve our workflow for releasing our code on github and following up on external contributions. As a first step in this direction, we reworked our internal process for exporting the public release and we are now able to push to github more often. So going forward, instead of pushing one huge commit before every major release in the main opam repository, we will push smaller commits more frequently. This has several consequences: - it will be easier to follow the latest changes - accepted pull requests will appear in the git repository earlier - you can complain if we break your code before the next release in opam... Initially the commit messages will still be meaningless and the changelogs might be slightly lagging behind given that producing them still requires a fair amount of work from a human. However, we are planning to improve this over time. Note that we'll continue to provide support for the latest version that is released in opam. ## janestreet/opam-repository To help trying the development version of our libraries, we've setup an opam repository that follows their latest development version [1]. So in order to try out the latest version of core, you can do: $ opam repo add git://github.com/janestreet/opam-repository.git $ opam install core Pinning individual packages with `opam pin add --dev` might work as well, but there is no guarantee about it. ## 113.43+70 To start the new process, we just pushed the latest version of our code on github. The version is 113.43+70 and includes a fair amount of changes. [1] https://github.com/janestreet/opam-repository
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00185.html
Deep in this thread, Mauricio Fernández announced:FYI I have packaged several libs for opam-cross-windows and will soon submit them for inclusion in the main repos, see https://github.com/mfp/opam-cross-windows-repos/tree/master/packages
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00186.html
Continuing the thread from last week, Louis Roché announced:Updated informations: The meetup will take place at Mozilla Paris. The address is 16 Bis Boulevard Montmartre Paris 75009. The access is controlled. You won't be allowed to join the meetup if you are not registered. The definitive list of talks: - Danny Willems : Bindings OCaml à Cordova grâce à js_of_ocaml et gen_js_api - Frédéric Bour : sturgeon — A toolkit for communicating with Emacs from OCaml - Corentin De Souza : dead_code_analyzer — un détecteur de code mort pour OCaml To register, or for more information, go here: http://www.meetup.com/fr-FR/ocaml-paris/events/231068590/
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00190.html
Daniel Bünzli announced:I'd like to announce the first release of bos: ``` Bos provides support for basic and robust interaction with the operating system in OCaml. It has functions to access the process environment, parse command line arguments, interact with the file system and run command line programs. Bos works equally well on POSIX and Windows operating systems. Bos depends on Rresult, Astring, Fmt, Fpath, Logs and the OCaml Unix library. It is distributed under the ISC license. ``` Homepage: http://erratique.ch/software/bos API docs: http://erratique.ch/software/bos/doc/ Bos can be seen as improved Sys module for programs and scripts that have light OS interaction requirements. Inspired by the excellent work of scsh [0], it seems this path has been pursued more than once in OCaml for example with cash [1] or shcaml [2]. Bos however has a slightly different take on this. It does not try to recover the shell's terseness or processing model but rather tries to overcome the shell's brittleness in face of errors, error reporting and insane quoting conventions. The difference between a script and a program is an artificial one and bos tries to encourage you to write programs that do not fail obscurely and evolve gracefully from a quick, small one (a.k.a "script") to a complex one (a.k.a. "program") while keeping good usability for the end user who is the one who eventually gets processing failures in the face. This first version number is intentionally low, as I don't see bos as fully finished at the moment. Some interfaces could be tweaked and other added in the future (feedback and discussion on the issue tracker is welcome). I did however already write a few programs/scripts that make use of most of the features provided by the library, so the low number should not be interpreted as "alpha" quality software. What is provided here should work as described and if it doesn't, scream on the issue tracker. Best, Daniel [0] https://scsh.net/ [1] http://pauillac.inria.fr/cash/ [2] http://users.eecs.northwestern.edu/~jesse/code/shcaml/SP asked and Daniel Bünzli replied:
> What does this mean for Cmdliner? Does it overlap? > > [Cmdliner]: http://erratique.ch/software/cmdliner The answer is in the documentation preamble of the OS.Arg module: http://erratique.ch/software/bos/doc/Bos.OS.Arg.htmlMartin DeMello asked and Daniel Bünzli replied:
> Do you think this would be a good route towards getting ocamlbuild working > seamlessly on windows? Well bos does not do any kind of magic to work well on windows except avoiding what needs to be avoided, namely fork(2). Contrary to what it's name suggests the Unix library is not a bad OS abstraction layer and works quite well on windows; even more so with the work David Allsopp put into 4.03. At the moment, but it may change in the future, bos is only a more lighter and simpler way to work with the Unix library correctly. Unix programming can be tricky and quite verbose, e.g. to setup program pipelines. Also note that bos' simpler interface may be not be suitable for all programs, i.e. bos doesn't replace Unix. IIRC (Gabriel may want to comment) to get ocamlbuild working seamlessly on windows one needs to get rid of fork, a few pointless shellouts and make people stop use Sh command specs in their plugins. So I don't think bos necessarily helps here, using Unix conservatively in ocamlbuild should do.Gabriel Scherer then said:
There are some parts of the ocamlbuild internals that have their own implementations of file path handling and quoting/escaping, and that are a bit too arcane to my taste. (I tend to not change them because they mostly work well, but they come back to bite us once in a while.) When I got the announcement for Bos, I indeed thought that it could be a nice idea to have a look. (I probably won't do it myself in the short term, my development time is very constrained right now; plus the fact that I cannot test on Windows myself makes me wary of changing those parts of the codebase.)Adrien Nader added:
It's not really fork but Unix.open_process*. The signatures and documentation sum up the issue: val open_process_in : string -> Pervasives.in_channel [...] The command is interpreted by the shell /bin/sh (cf. system). As such, the process command-line needs to be properly quoted. I'm not going to attempt to get the whole logic right so I'll only give keywords for things that can be involved: /bin/sh, $SHELL, quoting for posix shells, quoting for windows command-line (entirely different and 200% ad-hoc over the course of 20 years), cmd.exe, %COMSPEC% (and probably more things). I see the use of a single string as the main issue because it makes supporting the quoting for windows process invocations very difficult (the entry point on windows gets a single string with holds every process argument: argv in main() is actually the result of parsing...). When spawning with a string array or string list, one isn't usually concerned about shell escapes: only special characters such as !, ':' or '&' might matter but the last really do not make sense in this context.. This makes it possible to quote each argument separately before merging everything (with quoting) and call the process (NB: there are also _exec* functions in the Microsoft CRT that could help). I consider moving away from single-string arguments a prerequisite for proper and sane handling of this. One annoyance for ocamlbuild is that it exposes that same API to its plugins. Support for wide-character functions is another change needed for better Windows support in the OCaml ecosystem. If this has to be done with API changes or new APIs, maybe it will be the right time to also remove support (or not offer them at all in the new APIs) for functions that use a single string to spawn functions. Back to the topic, as Gabriel said, there is a number of useful functions in ocamlbuild (see My_Unix for instance iirc) which work fairly well and are therefore risky changes (that's actually pretty funny logic). Nevertheless, I don't think they should stay there.
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00191.html
Daniel Bünzli announced:Hello (last one today, promised), I'd like to announce the first release of topkg. For those who know topkg, it is no longer an embedded script but a library that you add as a build dependency to your package. It now also comes with an optional tool that helps you to manage your package and make swift and correct releases. Bla bla bla: ``` Topkg is a packager for distributing OCaml software. It provides an API to describe the files a package installs in a given build configuration and to specify information about the package's distribution creation and publication procedures. The optional topkg-care package provides the `topkg` command line tool which helps with various aspects of a package's life cycle: creating and linting a distribution, releasing it on the WWW, publish its documentation, add it to the OCaml OPAM repository, etc. Topkg is distributed under the ISC license and has **no** dependencies. This is what your packages will need as a *build* dependency. Topkg-care is distributed under the ISC license and depends on fmt, logs, bos, cmdliner and opam-lib. ``` Install: opam install topkg topkg-care Homepage: http://erratique.ch/software/topkg API docs & manual: http://erratique.ch/software/topkg/doc/Topkg.html#basics See also `topkg help release` and `topkg help` A few things worth noting: * Topkg pretends to work on Windows (provided your build system does) but early users may need to give a hand, please do report issues if things go wrong. * Topkg pretends to be able to work with hg source repositories, the same as for Windows applies. * Topkg should be able to work with other build systems than ocamlbuild. But a few things (e.g. documentation support or dependency linting) do at the moment rely on ocamlbuild being used. It's not my priority but I'm not against improving it to make it easier to use with other build systems. * Topkg only supports OCaml from 4.01 on and hence will also impose this limit on your packages. * Topkg has toy, proof of concept, support to publish documentation and distributions via GitHub's pages and release system. I'm personally not using this so do not hesitate to report broken or suboptimal behaviour. Better support — especially gh issue interaction for 'topkg issue' and better authentication would be nice, but is left in the hands of a motivated github user. Someone should develop and distribute a good github topkg delegate (e.g. using ocaml-github) that github users can install; see 'toy-github-topkg-delegate --help' and 'topkg help delegate'; the source of the toy delegate [1] can be used as a blueprint. Do not hesitate to get in touch for more information and/or design discussion. Topkg may not be the final word in OCaml building & packaging, however, depending on your needs, topkg's release workflow (see 'topkg help release') may come as a significant improvement over the current status quo. I have been using a similar but more ad-hoc system with my packages for a few years now since I realized that oasis was getting in my way to scale modularity. Except for the maintainability problem of having an embedded script which is now gone with this new incarnation of topkg, it has already been shown to work quite well in practice for me and a limited number of other persons, in a large range of software install scenarios, YMMV. Best, Daniel P.S. If you are on github and start to use `topkg` you can mention me (@dbuenzli) in the inaugural PR or commit, I'm interested to see how it is used and misused. [1] https://github.com/dbuenzli/topkg/blob/v0.7.0/src-bin/toy_github_delegate.ml
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2016-05/msg00193.html
Drup announced:It is with great pleasure that we are announcing the release of TyXML 4.0.0. The major features of this new release are a new PPX syntax extension that allows to use the standard HTML syntax and an improved user experience for both old and new TyXML users. TyXML is a library for building statically correct HTML5 and SVG documents. It provides a set of combinators which use the OCaml type system to ensure the validity of the generated document. TyXML’s combinators can be used to build textual HTML and SVG, but also DOM trees or reactive interfaces, using Eliom and Js_of_ocaml. A presentation of the new release is available on the ocsigen blog and the changelog is available on github. Happy HTML and SVG hacking!
Here is a sneak peek at some potential future features of the Ocaml compiler, discussed by their implementers in these Github Pull Requests. Spacetime: a new memory profiler https://github.com/ocaml/ocaml/pull/585
Here are links from many OCaml blogs aggregated at OCaml Planet, http://ocaml.org/community/planet/. ppx_core: context-free rewriters for better semantics and faster compilation https://blogs.janestreet.com/ppx_core-context-free-rewriters-for-better-semantic-and-faster-compilation/ Thirteenth OCaml compiler hacking evening at Pembroke College http://ocamllabs.github.com/compiler-hacking/2016/05/20/spring-compiler-hacking TyXML 4.0.0 https://ocsigen.github.io/blog/2016/05/20/tyxml4/
If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.
If you also wish to receive it every week by mail, you may subscribe online.