OCaml Weekly News

Previous Week Up Next Week

Hello

Here is the latest OCaml Weekly News, for the week of April 02 to 09, 2019.

Table of Contents

Easy_logging 0.2

Sapristi announced

Version 0.4 is out (or should be soon) :

  • printf style logging is now the default
  • simplifed configuration in case of multiple loggers (and closer to the python module) :
    • the loggers form a tree (based on their name, dots indicating descendence)
    • log items are passed to the handlers of a logger's ancestors (so that few handlers need initialisation) (possible to override)
    • loggers inherit the level of their ancestor if not set explicitely
  • an additional package easy_logging_yojson provides initialisation of loggers from the json format (with ppx_deriving_yojson), so that configuration can be fully done outside the code.

OCaml Users and Developers Meeting 2019

David Allsopp announced

I'm pleased to annouce that the 8th ICFP co-located OCaml Users and Developers Workshop will be held in Berlin on August 23rd.

The workshop is an informal occasion consisting of ~20 minute talks and a poster session on topics ranging from compiler developments, new libraries and applications through to experience reports on industrial/experimental uses of OCaml.

The submission site for talk proposals is now open and full details of the call may be found on the ICFP 2019 website. The deadline for proposals is May 17 Anywhere on Earth.

Firewall-tree - demo using MirageOS is available, with overview of current progress

Darren announced

I'm very pleased to announce that firewall-tree library has reached a presentable stage and a demo which uses firewall-tree in MirageOS is available.

Demo

The demo code is available here.

The code itself is quite thoroughly commented, the README in the folder contains build and run instructions and also illustrations of the tree in graphical form, so I won't dive into the details of the demo here.

In short, the demo shows how to separate of ICMP trafffic and TCP traffic. For TCP, filter some TCP traffic, then load balance the remaining TCP connections across multiple destination IPv4 addresses. For ICMP ping requests, only reply to every other one.

The tree itself is only a couple of lines, but there is quite a bit setup code for wrapping MirageOS stuff into a module and types usable by firewall-tree. However, it should be possible to package the setup code into a library supporting MirageOS as a firewall-tree backend, so in future the usage of the library should be much smoother and less cluttered than the demo.

I'd like to note that the library right now is still very rough at more complex constructs, and the connection tracker, which is used inside the translators, works fine for a very typical TCP connection, but fails to handle TCP retransmissions, or TCP PDUs with RST and PSH flags.

Library overview

The library requires a "base", which you can think of as an environment implementation (see here for module type signature), which it then extends into a full firewall module. The resulting module provides the following functions/constructs

The package provides another set of functors which take the resulted tree module as parameter and derive commonly used constructs. See demo for example use.

Moving onward

Still a lot of things to be done, following lists a few of the items

  • TCP connection tracking needs to be fixed
  • Implement more translators, make them more modular (e.g. swappable algorithms)
  • Couple translators to form NAT modules
  • Benchmark
    • I am hoping this library doesn't slow down MirageOS too much for equivalent functionalities, but I don't have concrete benchmarks

Final thoughts

The progress so far has been quite time consuming to implement, with a lot of redesigns and rewrites in-between. I am hoping the current design is sane, but I feel I've invested too much time in last couple of weeks to get a clear view of things. So I would absolutely appreciate any feedback, whether it's on API design, the architecture of the code or any other aspects.

I had some experiences with networking and firewall, but as far as implementing a network stack goes, I have zero experience, so I'd also appreciate help on components which demand such expertise as well.

Hannes Mehnert asked and Darren replied

Thanks for reading through!

> I’ve not been reading through all of your code, but you seem to have implemented 80% of a TCP stack and also mention that reset handling (what about simultaneous open, (IP) fragmentation, and retransmissions?) is missing.

  • Simultaneous open
    • Uh…uh oh, I didn't realise simultaneous open is a thing
    • This might require redesign of the TCP connection tracker as it right now assumes only one side is the initiator, and the other side must be the responder
  • IP fragmentation
    • Don't think this matters too much unless the library moves toward supporting deep packet inspection?
    • But yeah this may be a problem if the library needs to support moving packets from a layer 2 segment of higher MTU to one with lower MTU? But that should be handled by the IP layer (of MirageOS) in this case
    • In any case I don't know if it should be in the scope of the library
  • Retransmissions
    • From firewall perspective I think it only needs to remember seeing the PDU at some point and just allow duplicates to be sent through up to some threshold, which means collecting the hashes should suffice I hope

Ideally the library is network implementation agnostic, but I can't tell if that's a reachable goal exactly considering all these intricate things.

> Did you try to use mirage-tcpip instead, which already implements some of the missing pieces (at least reset handling AFAIR)?

Right now the demo itself uses Tcp.Tcp_packet from tcpip to handle serialisation and deserialisation, but not flow or any connection establishing components.

I can't tell if flow is suitable for firewall, since that means the unikernel needs to add 65535 listeners to cover all ports? I might be being silly here or misunderstood something, please correct me if I am.

> But given the complexity of TCP, I’m not sure how many partial TCP implementations in OCaml are sensible

W.r.t. stack implementation, I think I might have picked the wrong wording. For the library atm, RST handling entails some adjustments in the state machine (which is not too much work, but need a lot of debugging probably), and retransmission entails recording PDUs (through hashes most likely) and allow pass through if duplicates are seen (up to N times etc). Both of which are much less work than a full stack implementation.

A firewall does need to work on a slightly different perspective compared to a client, however, as when a firewall sees a PDU, it's not always the case that either client will see it, as opposed to being a client and receiving something means you know for sure you've received it. So the other adjustment required is the state machine of connection tracker needs to tolerate "partial" state transition in some way (so just look ahead one state but don't commit or whatever).

So overall it's more of a partial stack implementation powerful enough to observe and forward things and that's the end of the story.

Thanks for the link to the paper btw! I'm interested in formal verification stuff in general and currently researching on protocol verification topics at uni, so I'll definitely give it a thorough read later.

routes: path based routing for web applications

Anurag Soni announced

I have just published an initial release for a path based routing library. The library has minimal dependencies and isn't tied to any particular HTTP or UI framework.

Example usage can be like below:

module Request = struct
  type t
  ...
end

module Response = struct
  type t
  ...
end

let get_user state (id: int) =
  (* Request handlers can define their own routes too *)
  let open Routes in
  let routes = [ int64 </> boolean </> ==> (fun _ i b -> ... ) ] in
  match_with_state ~state routes with
  | None -> ...
  | Some response -> ...

let search_user _ (name: string) (city : string) =
  ...

let routes =
  let open Routes in
  [ empty ==> idx (* matches the index route "/" *)
  ; method' `GET </> s "user" </> int </> empty ==> get_user (* matches "/user/<int>" *)
  ; method' `GET </> s "user" </> str </> str ==> search_user (* missing empty so it matches "/user/<str>/<str>/*" *)
  ]

match Routes.match' routes ~req ~target:"/some/url" ~meth:`GET =
| None -> (* No route matched. Alternative could be to provide default routes *)
| Some r -> (* Match found. Do something further with handler response *)

Git repository: https://github.com/anuragsoni/routes

Example using Httpaf: https://github.com/anuragsoni/routes/blob/7fc2ecb0b3a2e46b26254efe54f1147d99d91995/example/main.ml

This is not published on opam yet so it can be pinned locally via: opam pin add routes git+https://github.com/anuragsoni/routes.git

Jean Michel asked and Anurag Soni replied

> Have you looked at dispatch?

> What are the advantages/differences of path?

Yes i did look at dispatch. One difference from dispatch is that the URL's path parameters are extract in a way that assigns them the types defined by the user. If such a type coercion isn't possible the route won't match.

Later on, Anurag Soni added

A small update to the library.

  • Updated the internal representation of a route so the same source is used for both scanning and printing routes.
  • Added a sprintf like function to format routes.
  • Route matching is now strict by default. ex: s "user" </> str will just match /user/<string> and not /user/<string>/*
  • Following from the previous point, nested routing has been removed for now.
utop # let route = method' None (s "foo" </> int </> str </> bool);;
- : val route : (int -> string -> bool -> unit -> '_weak1, '_weak1) route =
  Route (None, S ("foo", S ("/", Int (S ("/", Str (S ("/", Bool End)))))))

utop # sprintf route;;
- : int -> string -> bool -> unit -> string = <fun>

utop # (sprintf route) 12 "bar" false ();;
- : string = "foo/12/bar/false"

Other OCaml News

From the ocamlcore planet blog

Here are links from many OCaml blogs aggregated at OCaml Planet.

Old CWN

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.