OCaml Weekly News
Hello
Here is the latest OCaml Weekly News, for the week of May 25 to June 01, 2021.
Table of Contents
Dream — a simple, yet feature-complete Web framework
Anton Bachin announced
I am pleased to announce Dream, a very easy-to-use Web framework with high performance, secure defaults, and thorough documentation!
It is available now from opam, with opam install dream.
Dream offers:
- WebSockets and GraphQL.
- A template syntax, which you can see in the image above.
- Trivial HTTPS and HTTP/2 support, allowing simple deployments without a proxy.
- Sessions with pluggable back ends.
- Easy secure cookies and CSRF-safe forms.
…and more, yet Dream sticks to a simple programming model:
- Web apps are just bare functions from requests to responses.
- Middlewares are just higher-order wrapper functions.
- Routes tell the router which of these functions to call.
Indeed, for those who like algebra, there is a certain structure to Dream. However, that's not the point of this post!
Dream is meant to be very easy to understand. It sticks to base types, introducing only a few types of its own, and uses existing languages, such as HTML for templates, and URLs for routes. Dream itself is one module in one opam package, which lives in a monorepo. The docs are on one page.
Dream is loosely coupled. Even though Dream offers many defaults, it is unopinionated, and you can quickly configure or replace anything. For example, it is easy to use TyXML for templates, and Dream happily supports such usage with examples.
Security-sensitive features, such as cookies, are arranged so that simple and obvious usage is automatically secure. Wherever security still depends on the Dream app, the docs highlight it. Dream has selected a modern cipher as a default, supports key rotation, and offers suggestions for other purposes, such as password hashing. It implements and abstracts away all of the OWASP security guidelines that are relevant to its level.
Dream is designed for full internationalization. It has a centralized error handler that intercepts even lower-level HTTP errors, so that you can decorate them with your app's own error template, and leak no hardcoded strings. Dream's URL encoders favor internationalized (UTF-8) URIs, and the router accepts them.
Finally, Dream is designed for a wide range of applications, including with or without a proxy, standalone or embedded in larger binaries, and with external static assets or assets compiled in.
Documentation
Dream is very extensively documented. See…
- Examples, the first several of which make up a tutorial. Each example is a complete project.
- The online playground, which features many of the examples, and is itself a Dream app!
- The API docs.
In particular, see
- Deployment examples for Heroku, Digital Ocean with Docker, and Digital Ocean with systemd, all of which include GitHub Actions scripts and instructions.
- Full-stack examples with js_of_ocaml, ReScript, and Melange.
- Examples in Reason syntax.
- Development watching and live reloading.
Contributing
Dream has already received several very helpful
contributions, and more are very welcome! See
CONTRIBUTING.md. I must also acknowledge all
the people working on Dream's
dependecies and
prior art. In particular, Dream relies heavily on the HTTP and
WebSocket servers
primarily by Spiros Eliopoulos (@seliopou) and Antonio Nuno Monteiro (@anmonteiro).
Apart from accepting code, docs, and examples, Dream will happily link to:
- Blogs and articles, as different people learn best from different presentations.
- "Downstream" libraries to use with Dream.
For example, Thibaut Mattio (@tmattio) is working on dream-livereload, a live-reloading middleware for Dream, similar to the example, which he also contributed! Once dream-livereload is slightly more mature, Dream will link to it from its README.
There is also dream-serve, a live-reloading static site server based on Dream and libuv, which was used to develop the docs.
Roadmap
Dream is currently in an alpha state. It is thought (by me) to be internally quite stable. However, there will probably be various API tweaks before release 1.0.0.
My current, rough plan is to release several alphas of Dream over six months or so. The releases will address:
- Flow control for very large responses, and getting the "advanced" part of the I/O API to be as close to zero-copy and non-allocating as possible (or reasonable).
- Remaining (optional) security enhancements, such as a default content security policy.
- Remaining session improvements, such as re-keying.
- Friction in handling of JSON, database access, etc. This is not properly part of or due to Dream, but it should be addressed for a better Web development experience.
- Multicore and effects support.
That's all. Let's bring OCaml to the Web! Happy Web programming!
Anton Bachin then added
For readers who saw the repo during the earlier "leak," the main updates are:
- A large number of new examples, including deployment.
- The playground, which runs the examples, and itself served as a test.
- An esy-based quick start script.
There have also been very many smaller changes to the code, API, and the rest of the docs, but the above changes are the biggest "chunks." The rest is too much to detail :)
Ivan Gotovchits asked and Anton Bachin replied
I was always wondering how does the source code that uses templates work with OCaml tooling, in particular with merlin, ocp-indent, ocaml-format, tuareg and other editor modes?
It doesn't work well in practice with anything other than syntax highlighting. Note that you control the syntax mode
with the extension. If your template is mostly HTML, you can name it foo.eml.html.
The intent is that the templates should contain mostly HTML in a large project, and most of them would be in their
own template/ subdirectory. OCaml tooling wouldn't be needed for these mostly-HTML files. For a still-small, but
real example of this, see the Playground's
client.eml.html.
The one-file .ml projects with templates, where tooling is a problem, are mostly good for the very first steps of
getting started, and examples.
There is also an issue about this in the repo, #55 " how to apply ocamlformat".
Note that, as in the announcement text, you can use Dream with other templaters, including TyXML, which has an HTML PPX. In addition, if you are using Reason, you can use TyXML JSX. Either of these options interacts well with tooling, as far as I know.
I didn't make TyXML the default because it considerably increases the Dream learning curve for getting basic tasks done. However, Dream still supports the choice of using TyXML with examples and links.
Ocaml developer at Routine, Paris, Remote OK
mefyl announced
Routine (https://routine.co) is looking for an OCaml developer.
Routine is a personal productivity assistant. The technological revolves heavily around OCaml which represents 90% of the codebase, the remaining 10% being the UI in Typescript and Vue.js. We target both the browser and desktop through electron, using Js_of_ocaml.
While the product is "just" a web app, our technological and academic background leads us to use designs that, I think, can pique the interest of seasoned Ocaml developer. Amongst other things :
- Type-driven programming based on ppx derivers that produces typescript declaration for frontend bindings, JSON schema to expose and consume external REST APIs (Google, Notion, …), automatic SQL bindings, etc.
- Angstrom based parsing for the interactive console with highlighting and completion.
- Incremental based state updates to refresh minimal subsets of the app.
- Highly concurrent implementation through Lwt, exception-free design.
We use state of the art CI/CD and development processes. We plan on distributing open sources packages of these utilities (type-driven system, Google API bindings, Notion API bindings, …). Future exciting subjects could be extending Angstrom with manual rollback to implement generic completions or binding Vue in OCaml directly using melange or rescript to achieve rock solid typing down to the very frontend code (highly prospective teases, don't quote me on this yet :).
The company is very much a startup, having just completed YC batch W21 and closed its first round of investment. Salary is up to market standard depending on the profile, plus usual options package, to be discussed.
While we expect great OCaml and general computer science proficiency, we're open to most levels of experience. Thoroughness and a love for well rounded, robust and beautiful software design is a must have - but that comes bundled with OCaml love, right ?
Do not hesitate to reach out for any question here, at quentin.hocquet@routine.co or refer this to someone who may be interested.
Thanks for your time and happy camel riding !
Feather 0.2.0
Charles announced
I'm happy to announce feather version 0.2.0! Feather is a minimal library for bash-like scripting and process execution. (github, opam)
This release fixes some bugs and adds three new functions
val and_ : cmd -> cmd -> cmd— chain two commands, short circuiting if the first fails, akin to bash's&&operator.val or_ : cmd -> cmd -> cmd— chain two commands, short circuiting if the first succeeds, akin to bash's||operator.val sequence : cmd -> cmd -> cmd— chain two commands regardless of exit status.
We include two new operators &&. and ||. which correspond to and_ and or_ respectively. They'll be found in
the Feather.Infix module, which has been renamed from Feather.File_redirection_infix.
Many thanks to new contributors @Firobe @juxd and @tmarti2 for making this release possible!
BAP 2.3.0 Release
Ivan Gotovchits announced
We're proud to release the next stable version of Carnegie Mellon University Binary Analysis Platform (BAP). The full list of changes can be found on the release page but the most interesting new features are highlighted below.
The Primus Lisp Frontend
Now BAP is able to understand not only binary programs but sources written in Primus Lisp. In case if you don't know, Primus Lisp is our DSL for writing analysis and library stubs (e.g., to specify semantics of missing library functions). Now, it is possible to reify Primus Lisp programs into static representation. For example, we can translate the following Lisp program
;; file demo.lisp
(defun example1 (x)
(set R0 1)
(set R1 2)
(set R3 (+ R1 R2 (* R1 R2 3)))
(memory-write R4 (+ R3 R1))
(if (> R0 (* R0 R0))
(exec-addr 0xDEADBEEF)
(set R0 (* R0 R2 R3))))
into the BIL (BAP Instruction Language) AST and then pretty print it,
$ bap show --primus-lisp-load=demo --target=armv7+le -obap:bil example1
example1:
"{
R0 := 1
R1 := 2
R3 := R1 + R2 + R1 * R2 * 3
mem := mem with [R4] <- low:8[R3 + R1]
#1 := R0 * R0 < R0
if (#1) {
jmp 0xDEADBEEF
}
else {
R0 := R0 * R2 * R3
}
}"
This new feature not only allows us to reify our Lisp stubs into static form but also enables the main killer feature. It is now possible to specify the semantics of machine instructions in Primus Lisp. This feature enables rapid development and experimentation with CPU semantics. And this brings us to the next new feature.
New Target: RISC-V
The first application of the Primus Lisp Frontend was writing the RISC-V semantics. It took me only one day to write
the semantic of the minimal subset of RISC-V instruction. Well, partially it is because RISCV-V is truly RISC,
like the add instruction just adds,
(defun ADDI (dst rm rn) (set$ dst (+ rm rn)))
New Target: ARMv8 (Aarch64)
The next target that we tried was Aarch64, the 64-bit ARM architecture. It was a little bit harder but still definitely more readable than the official ARM semantics.
Adds namespaces (packages) to Primus Lisp
Since now we have much more code in Primus Lisp we found ourselves struggling with name clashes. The Primus Lisp program model is a set of mututally recursive overloaded definitions, so naming things is crucial for us. Therefore we implemented namespaces (which are, following Common Lisp trandition, named packages). We ended up in a very Common Lisp look and fill but without inheriting CL problems, like the dependency on the order of inclusion and package redefinitions, and so on. Given our model, and that Primus Lisp features type inference and Haskell-style type classes for overloading, it wasn't that easy to implement :)
Adds the bap dependencies command
The command outputs program dependencies such as libraries and symbols. The information is collected recursively with various output options, including dependency graph, YAML, JSON, and SEXP.
Much like nm~+~ldd on steroids and cross-platform (works on PE/ELF/COFF, and on binaries that are not native to the
host). So it could be quite useful even if you're not doing program analysis, but just want to solve a nasty missing
library feature or figure our what programs use what libraries, e.g.,
$ bap dependencies `which ping` --recursive --ldconfig -ograph | graph-easy --as boxart
┌────────────────┐
│ libresolv.so.2 │ ──────────────────────────────────┐
└────────────────┘ │
▲ │
│ │
│ │
┌──────────────┐ ┌──────────────────────────┐ ┌────────────────┐ │
│ libidn.so.11 │ ◀── │ ping │ ──▶ │ libnettle.so.6 │ │
└──────────────┘ └──────────────────────────┘ └────────────────┘ │
│ │ │ │ │
│ │ │ │ │
│ ▼ │ │ │
│ ┌────────────────┐ │ │ │
│ │ libcap.so.2 │ │ │ │
│ └────────────────┘ │ │ │
│ │ │ │ │
│ │ │ │ │
│ ▼ ▼ │ │
│ ┌──────────────────────────┐ │ │
└────────────────▶ │ libc.so.6 │ ◀─────┘ │
└──────────────────────────┘ │
│ ▲ │
│ └───────────────────────────┘
▼
┌────────────────┐
│ ld-linux.so.2 │
└────────────────┘
What's Next?
We are working on decompilation and integrating with Ghidra, so in 2.4.0 you should expect that bap will output C code for binaries. But it is not all, we're even working into turning BAP into a program analysis framework that enables analysis of source code programs. And even crazier, we're working on adding compilation capabilities to BAP, i.e., an ability to compile/recompile the input sources. So soon BAP will outlive its name, or we will need to find a new interpretation for the BAP acronym, something like the Best Analysis Platform ;)
We also plan to make BAP more available for non-seasoned OCaml users and want to push bap into mainstream Linux distributions and overall lower the entrance barrier. Of course, with the end goal to lure users into installing opam))
Questions and Suggestions
Please, do not hesitate to ask questions and provide your suggestions and, ideally, join our community. Even if you don't plan to work on binary analysis, BAP offers lots of opportunities for writing your toy programs for learning the language, or maybe even student projects.
Building Ahrefs codebase with Melange
Javier Chávarri announced
At Ahrefs, we make extensive use of OCaml and ReScript —previously known as BuckleScript. So we have been following the latest developments in the ReScript ecosystem with great interest.
A few months ago, António Monteiro released
Melange, a fork of ReScript with an emphasis of keeping compatibility with
OCaml ecosystem. One of the key features of Melange is that it uses OCaml 4.12, with all the upsides that that
entails (ppxlib, let syntax, better errors, …). Besides that, Melange has been modeled recently as just a
compiler-libs library, so it can be integrated with other OCaml
code in a single opam switch.
We decided to give Melange a try recently at Ahrefs, and shared the results of this experiment in a blog post:
https://tech.ahrefs.com/building-ahrefs-codebase-with-melange-9f881f6d022b
We are currently looking into how a deeper integration with Dune would look like. If your team or company has tried Melange, or is interested on doing so, we would be very interested to hear your use cases and share experiences.
Lwt 5.4.1
Raphaël Proust announced
We are glad to announce the release of version 5.4.1 of Lwt: a bugfix-only release.
https://github.com/ocsigen/lwt/releases/tag/5.4.1
You can update to this version in opam:
opam update opam upgrade lwt
Thanks to the contributors for finding and fixing the bugs, leading to this release. Check out the release notes (link above) for a full list.
Other OCaml News
From the ocamlcore planet blog
Here are links from many OCaml blogs aggregated at OCaml Planet.
- Beta release of Frama-C 23.0~rc1 (Vanadium)
- Building Ahrefs codebase with Melange
- Computing an integer using a Grothendieck topos
- ReScript 9.1
- Tutorial: Format Module of OCaml
- Tarides project SCoP is selected as one of the brightest Data Portability projects in Europe!
- Alt-Ergo Users’ Club Annual Meeting (2021)
Old CWN
If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.
If you also wish to receive it every week by mail, you may subscribe online.