Hello
Here is the latest OCaml Weekly News, for the week of December 31, 2013 to January 07, 2014.
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2014-01/msg00002.html
Jeremie Dimino announced:
I am pleased to announce the 109.58.00 release of the Core suite.
Starting from 109.55.00, core libraries contain inline benchmark. It
is now possible to run them:
echo 'let () = Inline_benchmarks.Runner.main ~libname:"core_kernel"' > bench.ml
ocamlfind ocamlopt -thread -linkpkg -linkall -package str,core_kernel,core_bench.inline_benchmarks bench.ml -o bench
./bench -benchmarks-runner
Replace core_kernel by the library you want to run benchmarks
for. Currently only core and core_kernel contain inline benchmark.
In this release the following packages were upgraded:
- async
- async_extra
- async_inotify
- async_kernel
- async_parallel
- async_unix
- core
- core_bench
- core_extended
- core_kernel
- jenga
- sexplib
Files and documentation for this release are available on our
website and all packages are in opam:
https://ocaml.janestreet.com/ocaml-core/109.58.00/individual/
https://ocaml.janestreet.com/ocaml-core/109.58.00/doc/
Here is list of changes for this version:
# 109.58.00
## async_extra
- Changed `Cpu_usage` to use `Core.Percent` instead of `float` where
appropriate.
- Made `Bus.unsubscribe` check that the subscriber is subscribed to
the given bus.
- Made `Log.t` support `with sexp_of`.
- Fixed `Tcp.on_port 0` to return the port actually being listened on,
like `Tcp.on_port_chosen_by_os`.
Previously, a serverlistening on `Tcp.on_port 0` would have its
`Tcp.Server.listening_on` as `0`, which of course is not the port
the server is listening on.
## async_kernel
- Renamed the `Async_core` library as `Async_kernel`, to parallel
`Core_kernel`.
Someday `Async_core` will depend only on `Core_kernel`, but not yet.
- Added a thread-safe queue of "external actions" that is checked
after each job.
- Fixed a race condition in `Clock.Event.abort`.
Here is the race condition:
* `Clock.Event.at` adds an alarm, its value is a job (let's call it
job1) with this run function:
```ocaml
let fire () `
t :` Happened;
Ivar.fill ready `Happened;
```
* later a job (let's call it job2) aborting the clock event is queued
in the async scheduler
* in the same cycle, the `Timing_wheel.advance_clock` fires the
alarm and job1 scheduled
* at this point:
+ job1 and job2 are still pending
+ the alarm was removed so it is invalid
+ the clock event is still in the state `Waiting`
* job2 is executed before job1: the clock event is still in the
`Waiting` state, so the abort tries to remove the alarm from the
timing wheel: CRASH
The bugfix is for `Clock.Event.abort` to check if the alarm has
already been removed from the timing wheel and if so, don't remove
it again.
- Changed `Monitor.try_with` when run with `~rest:\`Ignore`, the
default, so that the created monitor is detached from the monitor
tree.
The detached monitor has no parent, rather than being a child of the
current monitor. This will eliminate recently observed space leaks
in `Sequencer_table` and `Throttle`, like:
```ocaml
let leak () =
let seq = Throttle.Sequencer.create () in
let rec loop n =
Throttle.enqueue seq (fun () ->
loop (n + 1);
Deferred.unit
)
|> don't_wait_for
in
loop 0
```
- Changed Async's scheduler to pool jobs rather than heap allocate
them, decreasing the cost of a job by 30-40%.
Changed the main scheduler queue of jobs to be an `Obj_array.t` that
is essentially a specialized `Flat_queue` (the specialization was
necessary for speed).
Also, cleaned up the scheduler run-job loop.
With these changes, the cost of a simple job decreases significantly
(30-40%), across a range of live data sizes. Here are the
nanoseconds-per-job numbers for a microbenchmark with the old and
new approaches.
```
| num live jobs | old ns/job | new ns/job |
|---------------+------------+------------|
| 1 | 74 | 53 |
| 2 | 75 | 47 |
| 4 | 76 | 41 |
| 8 | 63 | 39 |
| 16 | 62 | 38 |
| 32 | 61 | 37 |
| 64 | 61 | 37 |
| 128 | 60 | 37 |
| 256 | 60 | 38 |
| 512 | 60 | 38 |
| 1024 | 60 | 39 |
| 2048 | 61 | 40 |
| 4096 | 67 | 41 |
| 8192 | 65 | 45 |
| 16384 | 75 | 56 |
| 32768 | 115 | 67 |
| 65536 | 171 | 108 |
| 131072 | 255 | 158 |
| 262144 | 191 | 130 |
| 524288 | 216 | 139 |
| 1048576 | 238 | 152 |
```
See async/bench/nanos\_per\_job.ml for the benchmark.
- Removed `debug_space_leaks` from Async's internals. It hadn't been
used in years.
## async_unix
- Improved fairness of the async scheduler with respect to external
threads, including I/O done in external threads.
The change is to add a thread-safe queue of "external actions" that
is checked after each job.
Previously, when a job given to `In_thread.run` finished,
`In_thread.run` would take the async lock, fill the result ivar and
run a cycle. The problem is that in some situations, due to poor OS
scheduling, the helper thread never had a chance to grab the lock.
Now, `In_thread.run` tries to take the lock:
- if it can it does as before
- if it can't it enqueues a thunk in the external actions queue and
wakes up the scheduler
With this change, the helper thread doing an `In_thread.run` will
always quickly finish once the work is done, and the async scheduler
will fill in the result ivar as soon as the current job finishes.
- Fixed `Epoll_file_descr_watcher.invariant` to deal with the timerfd,
which has the edge-triggered flag set.
- Added `Writer.write_gen`, a generic functor for blitting directly to
a writer's buffer.
## core
- Added `Debug.should_print_backtrace : bool ref`, to control whether
`Debug.am*` functions print backtraces.
- Added to `Float` inline benchmarks.
- Moved all of the `Gc` module into `Core_kernel`.
Part of the `Gc` module used to be in `Core` because it used
threads. But it doesn't use threads anymore, so can be all in
`Core_kernel`.
- Improved `Iobuf` support for copying to/from strings and bigstrings.
The new modules are `Iobuf.{Consume,Peek}.To_{bigstring,string}`.
They match a `Blit`-like interface. We don't yet implement the
`Blit` interface in all applicable contexts, but do use `Blit.Make`
and expose some of the operations we want in the forms we expect
them to take under a `Blit` interface.
- Added `Linux_ext.Timerfd.to_file_descr`.
- Added to `Time.next_multiple` an optional argument to control
whether the inequality with `after` is strict.
- Added `Time.Zone.local`, a lazily cached `Time.Zone.machine_zone ()`.
This is the first stage in a plan to make explicit timezones more
pervasive. First, they are made more convenient, by replacing the
relatively wordy `Time.Zone.machine_zone ()` with `Time.Zone.local`.
This involves making the underlying timezone type lazy.
The next stage will be to remove `machine_zone` and use
`Time.Zone.local` everywhere instead. Then (it is hoped) instead of
`of_local_whatever`, we just say e.g. `of_date_ofday
Time.Zone.local` and currently-implicitly-local functions will be
able to switch over to explicit timezones without becoming too
verbose.
- Added `Timing_wheel.Alarm.null`.
- Made `Unix.File_descr.t` have `with sexp`.
Closes janestreet/async_unix#3
- Fixed OpenBSD compilation failures in C stubs.
- Fixed `Lock_file.is_locked` to require read permission, not write
permission, on the lock file.
- Added to `Unix.mcast_join` an optional `?source:Inet_addr.t` argument.
From pull-request on bitbucket:
https://bitbucket.org/janestreet/core/pull-request/1/receive-source-specific-multicast/diff
## core_bench
- Added support for saving inline benchmark measurements to tabular
files for easy loading into Octave.
## core_extended
- Cleaned up the `Stats_reporting` module
## core_kernel
- Moved all of the `Gc` module into `Core_kernel`.
Part of the `Gc` module used to be in `Core` because it used
threads. But it doesn't use threads anymore, so can be all in
`Core_kernel`.
- Made `Stable.Map` and `Set` have `with compare`.
- Added `String.rev`.
Closes janestreet/core#16
We will not add `String.rev_inplace`, as we do not want to encourage
mutation of strings.
- Made `Univ_map.Key` equivalent to `Type_equal.Id`.
- Added `Univ.view`, which exposes `Univ.t` as an existential, `type t
= T : 'a Id.t * 'a -> t`.
Exposing the existential makes it possible to, for example, use
`Univ_map.set` to construct a `Univ_map.t`from a list of `Univ.t`s.
This representation is currently the same as the underlying
representation, but to make changes to the underlying representation
easier, it has been put in a module `Univ.View`.
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2014-01/msg00003.html
Martin DeMello asked and Gerd Stolpmann replied:
> Does OCaml have any equivalent to ruby's StringIO, to wrap an
> in_channel around a string and then treat it as if it were a file?
With the standard library this is not possible.
When I need such a feature, I normally use OCamlnet's channel objects:
http://projects.camlcity.org/projects/dl/ocamlnet-3.7.3/doc/html-main/Netchannels_tut.html
http://projects.camlcity.org/projects/dl/ocamlnet-3.7.3/doc/html-main/Netchannels.html
http://www.ocaml-programming.de/rec/IO-Classes.html
Hezekiah M. Carty also suggested:
Batteries provides something similar with its BatIO.input_string
function. It returns the Batteries equivalent to an in_channel.
John Whitington also suggested:
Here's another example of wrapping IO around strings, bigarrays, etc as
a record of functions:
https://github.com/johnwhitington/camlpdf/blob/master/pdfio.ml
Here's a type:
type input =
{pos_in : unit -> pos;
seek_in : pos -> unit;
input_char : unit -> char option;
input_byte : unit -> int;
in_channel_length : pos;
set_offset : pos -> unit;
caml_channel : in_channel option;
source : string}
It's good to add the 'source' field - which is an arbitrary string for
debugging purposes. Of course, if the input came from a file, we would
use the file name.
The caml_channel field optionally stores a channel this input was
derived from. So, we can still drill down and use fast functions from
the standard library such as Pervasives.really_input.
(The set_offset field is a nasty PDF-related hack: a PDF file can have
garbage bytes preceding the header, and PDF absolute byte offsets are
offsets from the header, so every read and write might have to be
offset)
Functions in records as an abstraction don't seem to be a speed problem.
The use of option-returning functions like input_char above rather than
exception-raising ones or minus-one-as-exception functions like
input_byte above is big, though, if the data processed is large enough,
due to garbage collection.
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2014-01/msg00008.html
Daniel Bünzli announced:
I made thin, low-level, ocaml-ctypes bindings to OpenGL 3.{2,3},
4.{0,1,2,3,4} and OpenGL ES {2,3}. They are automatically generated
from the offical XML representation of the OpenGL Registry [1].
These bindings are neither tested nor used and interfaces are still
subject to change. Testers are welcome; report any problems you may
encounter on github's infrastructure. Have a look at the readme to
understand how these libraries are structured:
https://github.com/dbuenzli/tgls/blob/master/README.md#supported-opengl-versions
The documentation of the bindings is here:
http://erratique.ch/software/tgls/doc/
To install (only tested on osx with OpenGL 3.2 and linux or raspberry
pi with OpenGL ES 2 so far) you'll need `curl` and `xmlm` installed on
your system. This is only for this pre-release, actual distribution will
have the generated bindings. Then:
opam repo add erratique-u http://erratique.ch/software/opam/unreleased
opam update && opam install xmlm tgls
Sample code (needs tsdl, `opam install tsdl`), is in the directory
`opam config var tgls:doc`.
If you want to hack the source first have a read at:
https://github.com/dbuenzli/tgls/blob/master/DEVEL.md
then:
git clone http://github.com/dbuenzli/tgls
cd tgls && ./build support && ./build trigl3.native && ./trigl3.native
A few notes:
* This time the cryptic acronym is [T]hin bindings to Open[GL] librarie[S].
No need to waste everybody's time with alternate name suggestions:
it won't change.
* ocaml-ctypes was used as this was the shortest path to get the
work done, using it should also solve the problem of getting the
OpenGL function pointers. However the support for reading and making
sense of the OpenGL registry being done, regular OCaml stubs could
also be generated in the future if ctypes turns out to be a
bottleneck.
* There's one OCaml library per major OpenGL release (which correspond
to GPU hardware generations). Such a library has all the entry
points and enumerants for supporting all the core (and only core)
profiles of each minor releases of the major release. If you are
programming with let's say 4.3 context it's your duty not to use a
4.4 entry point/enumerant.
* I tried to include the C OpenGL libraries link flags in the OCaml ones
so that OCamler's can just use the package with ocamlfind to
build. Please report anything that may be missing. The current logic
is as follows. On Darwin for tgl{3,4} we link against the OpenGL
framework and nothing is done for tgles{2,3}. On linux pkg-config is
used when packages can be found: gl for tgl{3,4}, glesv2 for tgles2
and glesv3 (does that exist ?) for tgles3. Raspberry pi is like
linux but tgles2 links to the pi's GLESv2 library located in
/opt/vc/lib (would be nice from them to use pkg-config).
* A single type represents all GLenums. Types for subsets could
be devised to enhance the signature/documentation of functions. This
was not done for now since it seems the data from the registry is
not clean/up to date (according to the registry's manual). However,
the documentation links to the entry point's documentation and
indicates the name of argument variables which should help to invoke
the functions correctly (could have automatically derived labelled
arguments but I thought that doing so systematically would result in
a cumbersome API).
* The binding is very raw and can be unsafe. It could
be made safer by either:
1) Making more manual work, checking some bounds, automatically
deriving some arguments for sizes. Doing that automatically in
general is non trivial: for OpenGL 4.4 there are ~300 out of ~550
entry points with pointer arguments, the registry has an
attribute that should describe how the length of the pointer is
to be determined but according to the registry's manual its
syntax is poorely defined.
2) Devising higher-level APIs that ensure type safety.
I tend to lean in favour of 2) OpenGL being so low-level and
tiresome (rendering a colored triangle in "modern" OpenGL is a few
hunderds of loc) that you'll anyway build an abstraction
(a.k.a. rendering engine) on top of it.
* Binding generation is almost entirely automatic and type directed.
At the moment for OpenGL 4.4 there are only 7 out of 543 entry
points that are manually devised. One thing that did require manual
work though, was to enrich the registry information to indicate
whenever null pointers can be specified for pointer arguments to map
them to option types as the registry does unfortunately not specify that.
* There's no direct for support compatibility profiles or extensions.
Nowadays it's easy enough to use directly ocaml-ctypes (extensions
usually add only a few entry points and/or enumerants). Alternatively
tgls' generation infrastructure could be used as the extensions are also
described in the registry but that may be overkill (some steps have been
done in that direction but it's not a priority right now for me).
I hope tgls can be a good starting point to get up to date OCaml
OpenGL bindings as work for supporting new versions should be minimal
provided they update the XML registry (which they should do since they
use it to generate the C headers). A first release will be done in a
few months, after the bindings will have been used a little bit by
myself. Meanwhile your feedback is welcome.
Best,
Daniel
[1] http://www.opengl.org/registry/#specfiles
Archive: https://sympa.inria.fr/sympa/arc/caml-list/2014-01/msg00014.html
Jakob Lichtenberg asked and Jeremy Yallop replied:
> How can I look at source code change for:
>
> PR#5312: command-line arguments @reponsefile auto-expansion
> feature
>
> In a web browser?
You can find it here:
https://github.com/ocaml/ocaml/commit/8c9ef9f24
(Searching for the PR number by running 'git log --grep=5312' in a
clone of the OCaml repository returns the revision number, 8c9ef9f24.)
Thanks to Alp Mestan, we now include in the OCaml Weekly News the links to the
recent posts from the ocamlcore planet blog at http://planet.ocaml.org/.
OCaml: the bugs so far:
http://roscidus.com/blog/blog/2014/01/07/ocaml-the-bugs-so-far/
When QuickCheck Fails Me:
http://www.mega-nerd.com/erikd/Blog/CodeHacking/Haskell/quickcheck_fail.html
Presenting Decks:
http://openmirage.org/blog/decks-n-drums
Fedora 21 has a working OCaml ARM64:
http://rwmj.wordpress.com/2013/12/31/fedora-21-has-a-working-ocaml-arm64/
If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.
If you also wish to receive it every week by mail, you may subscribe online.