OCaml Weekly News
Hello
Here is the latest OCaml Weekly News, for the week of April 02 to 09, 2019.
Table of Contents
Easy_logging 0.2
Sapristi announced
Version 0.4 is out (or should be soon) :
- printf style logging is now the default
- simplifed configuration in case of multiple loggers (and closer to the python module) :
- the loggers form a tree (based on their name, dots indicating descendence)
- log items are passed to the handlers of a logger's ancestors (so that few handlers need initialisation) (possible to override)
- loggers inherit the level of their ancestor if not set explicitely
- an additional package
easy_logging_yojson
provides initialisation of loggers from the json format (withppx_deriving_yojson
), so that configuration can be fully done outside the code.
OCaml Users and Developers Meeting 2019
David Allsopp announced
I'm pleased to annouce that the 8th ICFP co-located OCaml Users and Developers Workshop will be held in Berlin on August 23rd.
The workshop is an informal occasion consisting of ~20 minute talks and a poster session on topics ranging from compiler developments, new libraries and applications through to experience reports on industrial/experimental uses of OCaml.
The submission site for talk proposals is now open and full details of the call may be found on the ICFP 2019 website. The deadline for proposals is May 17 Anywhere on Earth.
Firewall-tree - demo using MirageOS is available, with overview of current progress
Darren announced
I'm very pleased to announce that firewall-tree library has reached a presentable stage and a demo which uses firewall-tree in MirageOS is available.
Demo
The demo code is available here.
The code itself is quite thoroughly commented, the README in the folder contains build and run instructions and also illustrations of the tree in graphical form, so I won't dive into the details of the demo here.
In short, the demo shows how to separate of ICMP trafffic and TCP traffic. For TCP, filter some TCP traffic, then load balance the remaining TCP connections across multiple destination IPv4 addresses. For ICMP ping requests, only reply to every other one.
The tree itself is only a couple of lines, but there is quite a bit setup code for wrapping MirageOS stuff into a module and types usable by firewall-tree. However, it should be possible to package the setup code into a library supporting MirageOS as a firewall-tree backend, so in future the usage of the library should be much smoother and less cluttered than the demo.
I'd like to note that the library right now is still very rough at more complex constructs, and the connection tracker, which is used inside the translators, works fine for a very typical TCP connection, but fails to handle TCP retransmissions, or TCP PDUs with RST and PSH flags.
Library overview
The library requires a "base", which you can think of as an environment implementation (see here for module type signature), which it then extends into a full firewall module. The resulting module provides the following functions/constructs
- Hierarchical PDU definition
- PDU decapsulation
- PDU mapping (can also map inner components)
- PDU replacement (can also replace inner components)
- PDU pretty printing (see the demo for sample output)
- Memory bounded lookup table with evict timed out entries function
- Routing logic unit only partial implementaiton is in place currently
- Connection tracker (bidirectional, so you can use it without much thinking)
- Largely functional, still a long way to go before becoming robust enough
- Predicates for building filters/selectors
- Translators for building load balancers or proxy
- These were essentially a more general form of NAT translators when I wrote them, so hopefully they (or variant of them) can be used to build a full blown NAT at some point
The package provides another set of functors which take the resulted tree module as parameter and derive commonly used constructs. See demo for example use.
Moving onward
Still a lot of things to be done, following lists a few of the items
- TCP connection tracking needs to be fixed
- Implement more translators, make them more modular (e.g. swappable algorithms)
- Couple translators to form NAT modules
- Benchmark
- I am hoping this library doesn't slow down MirageOS too much for equivalent functionalities, but I don't have concrete benchmarks
Final thoughts
The progress so far has been quite time consuming to implement, with a lot of redesigns and rewrites in-between. I am hoping the current design is sane, but I feel I've invested too much time in last couple of weeks to get a clear view of things. So I would absolutely appreciate any feedback, whether it's on API design, the architecture of the code or any other aspects.
I had some experiences with networking and firewall, but as far as implementing a network stack goes, I have zero experience, so I'd also appreciate help on components which demand such expertise as well.
Hannes Mehnert asked and Darren replied
Thanks for reading through!
> I’ve not been reading through all of your code, but you seem to have implemented 80% of a TCP stack and also mention that reset handling (what about simultaneous open, (IP) fragmentation, and retransmissions?) is missing.
- Simultaneous open
- Uh…uh oh, I didn't realise simultaneous open is a thing
- This might require redesign of the TCP connection tracker as it right now assumes only one side is the initiator, and the other side must be the responder
- IP fragmentation
- Don't think this matters too much unless the library moves toward supporting deep packet inspection?
- But yeah this may be a problem if the library needs to support moving packets from a layer 2 segment of higher MTU to one with lower MTU? But that should be handled by the IP layer (of MirageOS) in this case
- In any case I don't know if it should be in the scope of the library
- Retransmissions
- From firewall perspective I think it only needs to remember seeing the PDU at some point and just allow duplicates to be sent through up to some threshold, which means collecting the hashes should suffice I hope
Ideally the library is network implementation agnostic, but I can't tell if that's a reachable goal exactly considering all these intricate things.
> Did you try to use mirage-tcpip instead, which already implements some of the missing pieces (at least reset handling AFAIR)?
Right now the demo itself uses Tcp.Tcp_packet
from tcpip to handle serialisation and deserialisation, but not flow or any connection establishing components.
I can't tell if flow is suitable for firewall, since that means the unikernel needs to add 65535 listeners to cover all ports? I might be being silly here or misunderstood something, please correct me if I am.
> But given the complexity of TCP, I’m not sure how many partial TCP implementations in OCaml are sensible
W.r.t. stack implementation, I think I might have picked the wrong wording. For the library atm, RST handling entails some adjustments in the state machine (which is not too much work, but need a lot of debugging probably), and retransmission entails recording PDUs (through hashes most likely) and allow pass through if duplicates are seen (up to N times etc). Both of which are much less work than a full stack implementation.
A firewall does need to work on a slightly different perspective compared to a client, however, as when a firewall sees a PDU, it's not always the case that either client will see it, as opposed to being a client and receiving something means you know for sure you've received it. So the other adjustment required is the state machine of connection tracker needs to tolerate "partial" state transition in some way (so just look ahead one state but don't commit or whatever).
So overall it's more of a partial stack implementation powerful enough to observe and forward things and that's the end of the story.
Thanks for the link to the paper btw! I'm interested in formal verification stuff in general and currently researching on protocol verification topics at uni, so I'll definitely give it a thorough read later.
routes: path based routing for web applications
Anurag Soni announced
I have just published an initial release for a path based routing library. The library has minimal dependencies and isn't tied to any particular HTTP or UI framework.
Example usage can be like below:
module Request = struct type t ... end module Response = struct type t ... end let get_user state (id: int) = (* Request handlers can define their own routes too *) let open Routes in let routes = [ int64 </> boolean </> ==> (fun _ i b -> ... ) ] in match_with_state ~state routes with | None -> ... | Some response -> ... let search_user _ (name: string) (city : string) = ... let routes = let open Routes in [ empty ==> idx (* matches the index route "/" *) ; method' `GET </> s "user" </> int </> empty ==> get_user (* matches "/user/<int>" *) ; method' `GET </> s "user" </> str </> str ==> search_user (* missing empty so it matches "/user/<str>/<str>/*" *) ] match Routes.match' routes ~req ~target:"/some/url" ~meth:`GET = | None -> (* No route matched. Alternative could be to provide default routes *) | Some r -> (* Match found. Do something further with handler response *)
Git repository: https://github.com/anuragsoni/routes
Example using Httpaf: https://github.com/anuragsoni/routes/blob/7fc2ecb0b3a2e46b26254efe54f1147d99d91995/example/main.ml
This is not published on opam
yet so it can be pinned locally via:
opam pin add routes git+https://github.com/anuragsoni/routes.git
Jean Michel asked and Anurag Soni replied
> Have you looked at dispatch?
> What are the advantages/differences of path?
Yes i did look at dispatch. One difference from dispatch is that the URL's path parameters are extract in a way that assigns them the types defined by the user. If such a type coercion isn't possible the route won't match.
Later on, Anurag Soni added
A small update to the library.
- Updated the internal representation of a route so the same source is used for both scanning and printing routes.
- Added a
sprintf
like function to format routes. - Route matching is now strict by default. ex:
s "user" </> str
will just match/user/<string>
and not/user/<string>/*
- Following from the previous point, nested routing has been removed for now.
utop # let route = method' None (s "foo" </> int </> str </> bool);; - : val route : (int -> string -> bool -> unit -> '_weak1, '_weak1) route = Route (None, S ("foo", S ("/", Int (S ("/", Str (S ("/", Bool End))))))) utop # sprintf route;; - : int -> string -> bool -> unit -> string = <fun> utop # (sprintf route) 12 "bar" false ();; - : string = "foo/12/bar/false"
Other OCaml News
From the ocamlcore planet blog
Here are links from many OCaml blogs aggregated at OCaml Planet.
Old CWN
If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.
If you also wish to receive it every week by mail, you may subscribe online.