Previous week Up Next week

Hello

Here is the latest Caml Weekly News, for the week of November 03 to 10, 2009.

  1. open-sourcing of the XAPI toolstack
  2. Ocal IDE: Vote for OcalIDE
  3. compiling C library wrapper
  4. Camlight version 0.80
  5. MetaOcaml and high-performance
  6. Modules
  7. Other Caml News

open-sourcing of the XAPI toolstack

Archive: http://groups.google.com/group/fa.caml/browse_thread/thread/b19555a7e09f063c#

Thomas Gazagnaire announced:
We are pleased to announce the open-sourcing of the XAPI toolstack, written in
OCaml, as used in the Citrix XenServer product line. The XAPI toolstack is
licensed under the LGPL v2.1 with a special static linking exception.

**context**

Xen [1] is an open-source type 1 hypervisor, providing the ability to run
multiple operating systems, called virtual machines, concurrently on a single
physical processor. Type 1 (or native, bare-metal) hypervisors are software
systems that run directly on host's hardware. They have been very popular
architecture since the CP/CMS, developed at IBM in the 1960s. Citrix XenServer
[2] is an enterprise-class, cloud-proven server virtualization platform that
delivers the critical features of live migration and centralized multi-server
management for Xen based virtual machine. XenServer is an open and powerful
server virtualization solution that radically reduces datacenter costs by
transforming static and complex datacenter environments into more dynamic,
easy to manage IT service delivery centers.

**xapi**

The xapi toolstack, which provides the core Citrix Xenserver's
functionalities, is a set of libraries and programs written in OCaml. It is
approximately 200k lines of code developed from early 2006. Within Citrix,
more than 40 people have already contributed to its source code.

What the XAPI toolstack has to manage covers a large range, from low-level
(interface with kernel and hypervisor) to high-level (distributed data/locking
management). It also need to have good performance because it may be used in
an environment of thousands of virtual machines per cluster and has to be able
to run on systems with limited resources (embedded editions of Citrix
XenServer are distributed on an USB stick and run in highly customized OS).
Finally, as a major enterprise product, the XAPI toolstack evolves constantly,
due to customer/business requests.

The OCaml language has been particularly good so far to help us facing all
this challenges, and we can't think of a better language to use :-) We really
appreciate that the compiler tools are stable, that the compiled code is fast
and small and that static-type inference makes our software robust to changes.

**links**

The released code is the current development version ("trunk") and is under
active development. The repositories are here:

http://xenbits.xen.org/xapi/xen-dist-ocaml.hg - scripts and Makefiles to build
some external library dependencies
http://xenbits.xen.org/xapi/xen-api-libs.hg - internal libraries (e.g. for
talking to xenstore)
http://xenbits.xen.org/xapi/xen-api.hg - the toolstack itself

The easiest way to build the code is in a specially-prepared VM, instructions
are here:

http://xenbits.xen.org/xapi/install.html

From now on, all development will take place on the
xen-api@lists.xensource.com mailing list. In the future we plan to move
relevant content such as documentation, designs, roadmaps etc. onto the main
xen.org wiki.

Comments, criticism and contributions are welcome!

--
The XAPI toolstack team

[1] http://xen.org
[2] http://www.citrix.com/English/ps2/products/feature.asp?contentID=1686939
			

Ocal IDE: Vote for OcalIDE

Archive: http://groups.google.com/group/fa.caml/browse_thread/thread/6b0998fcce38e25f#

Hugo Ferreira announced:
If you like OcaIDE, please vote for it to show your support:
 http://marketplace.eclipse.org/node/599
			

compiling C library wrapper

Archive: http://groups.google.com/group/fa.caml/browse_thread/thread/5ace8ddf0af6701a#

Aaron Bohannon asked and David Allsopp replied:
> I am quite confused by the whole process of compiling and installing
> wrappers for C libraries.  It seems like I can get things to work OK
> without really knowing what I'm doing if everything is put and built
> in a single directory.  The hard part seems to be putting the right
> files in the right places and getting the path arguments correct.
> Then things stop working, and I have to really understand what's going
> on, but the manual doesn't explain this part of the process in any
> detail.

Have a look at ocamlmklib in the manual - if it's available on your platform
then it nicely hides away a lot of this.

> Let's say I have a library "/opt/local/lib/libfoo.a" for which I want
> to build and use a (native-code) OCaml wrapper.  Here are the steps as
> I understand them:
> 
> 1) Write the file "foo_stubs.c" and compile it to get "foo_stubs.o".
> 
> 2) Build the library "libfoo_stubs.a" by running
> 
> ar rc libfoo_stubs.a foo_stubs.o
> 
> 3) Copy "libfoo_stubs.a" to its permanent location, let's say,
> "/usr/local/lib/ocaml/stubs/libfoo_stubs.a".
> 
> 4) Write "foo.mli" and use it to build "foo.cmi"

Steps 1-4 are fine.

> 5) Write "foo.ml" and use it to build "foo.cmxa" by running
>
> ocamlopt -a -o foo.cmxa foo.ml
>   -ccopt -L/opt/local/lib -cclib -lfoo
>   -ccopt -L/usr/local/lib/ocaml/stubs -cclib -lfoo_stubs

This command will also build foo.cmx, foo.a and foo.o

> 6) Copy "foo.cmi" and "foo.cmxa" to their permanent location, let's
> say "/usr/local/lib/ocaml/foo/"

You should also copy foo.a to this directory which should fix the problem.
foo.cmxa contains information required by OCaml but the actual code is in
foo.a (as it's been natively compiled). Similarly, .cmx files contain
information which ocamlopt needs but the actual code is in .o (or .obj)
files. For native code, it's a code idea to copy the .cmx files too as it
allows ocamlopt to do some inlining (I think that's right...)

> 7) Write my file "bar.ml" that needs to use the library, and compile
> it by running
>
> ocamlopt -I /usr/local/lib/ocaml/foo -o bar foo.cmxa bar.ml

This should now work without error.
			
Aaron Bohannon then said and Richard Jones replied:
> Ah!  Yes, that's exactly the part I didn't understand and everything
> works fine now.  Thank you!
>
> I can't say I completely understand the byte-code case, but I don't
> have an urgent need to.
>
> ocamlmklib seems fine, but if it's just a matter of saving keystrokes,
> "make" does a pretty good job of that (as long as I know what to tell
> "make" to do).

For the install step, use 'ocamlfind install'.

Grab the latest source tarball for one of our projects,
eg. libvirt-ocaml, to see how we do it:

http://libvirt.org/sources/ocaml/
or: http://libguestfs.org/download/
			

Camlight version 0.80

Archive: http://groups.google.com/group/fa.caml/browse_thread/thread/418b6809e94ea8bd#

François Boisson announced:
Depuis quelque temps je maintiens des paquets camllight pour debian et ubuntu.

Ces paquets se trouvent sur

deb http://boisson.homeip.net/debian lenny divers
(lenny peut être remplacé par woody, sarge (version 0.75) ou etch)

deb http://boisson.homeip.net/ubuntu karmic divers

karmic peut être remplacé par breezy, dapper edgy, feisty, gutsy, hardy,
intrepid, jaunty.

Ces paquets existent pour les architectures i386 et amd64 et vont des versions
0.75 à 0.80.

La version 0.80 existent pour etch, lenny, hardy, intrepid, jaunty et karmic.

Les sources permettant de faire ces paquets (assez «crades», c'était un de mes
premiers paquets), se trouvent sur

deb-src http://boisson.homeip.net/source/   ./

Par ailleurs, il y avait un bug dans la librairie graphique entrainant un
segfault sur une architecture 64 bits si on faisait

#open "graphics";;
open_graph "";;
(text_size "CAML");;

cela entrainait une erreur de segmentation..

L'origine était un transtypage sauvage du à une décalration manquante de la
fonction alloc_tuple. J'ai réglé le problème et ai fait des paquets corrigés
pour lenny, karmic et hardy. (paquet camllight_0.80-1_amd64.deb).
			

MetaOcaml and high-performance

Archive: http://groups.google.com/group/fa.caml/browse_thread/thread/1b613030e6a9be54#

Jon Harrop said and Oleg replied:
> I did some experiments with MetaOCaml. Firstly, it is x86 only and not x64
> which means poor floating-point performance, many times slower than HLVM's.
> The language itself is also very restrictive, e.g. you cannot generate
> pattern matches dynamically so you cannot leverage the pattern match
> compiler, which is a huge loss. In essence, effective use of MetaOCaml
> requires writing in continuation passing style which OCaml and, therefore,
> MetaOCaml do not handle well (closure invocations are slow in OCaml and CPS
> is not optimized back into anything sane). So I do not consider MetaOCaml to
> be a viable option for performance users.

A few clarifications seems to be in order. First of all, the original
poster asked about _offshoring_ with MetaOCaml. When the generated
code is run with offshoring, a C of Fortran file is created, which can
be compiled with your favorite compiler and dynamically linked back
into the running OCaml program. Alternatively, you can use the
generated C/Fortran code as it is, as a part of a C/Fortran project. We
did exactly the latter in our FFT project: we used MetaOCaml to create C
files for FFT kernels, and plugged the files into the FFTW benchmarking
framework, which is pure C. It worked as expected.

Because offshoring produces a portable C or Fortran code file, you can
use the code on 32 or 64-bit platform. The reason the native MetaOCaml
without offshoring does not work on amd64 is because at that time
OCaml didn't emit PIC code for amd64. So, dynamic linking was
impossible. That problem has long been fixed in later versions of
OCaml. Offshoring is a good way to get around it, thanks Jan Kybic.

Offshoring could just as well produce Verilog or LLVM code. Alas,
we didn't get around to exploring that idea.

Regarding continuation-passing style (CPS): if your code generatOR
needs let-insertion, then the generatOR _may_ need to be encoded in
CPS. The generatED code is _not_ in CPS; it is in the `conventional',
so-called `direct style'. Even if we assume that CPS code is
difficult to compile efficiently (which is not certain: in CPS, all
`important' function calls are tail-calls, and OCaml is very good with
tail-calls), that difficulty affects only the code generator rather
than the generated code. Most of the time it is the generated code
that is performance-critical.

One may say that writing the generator in CPS is a bother. I have
heard such objections. Please see our PEPM2009 paper about a way to
address it
       http://okmij.org/ftp/Computation/Generative.html#circle-shift
and write generators in the conventional style. Please see the example
of writing a flexible Gaussian Elimination code, paying no penalty for
abstractions.

Fortunately, some people have considered MetaOCaml to be a viable
option for performance users and have reported good results. For
example,

       Tuning MetaOCaml Programs for High Performance
       Diploma Thesis of Tobias Langhammer.
       http://www.infosun.fmi.uni-passau.de/cl/arbeiten/Langhammer.pdf

Here is a good quotation from the Introduction:

``This thesis proposes MetaOCaml for enriching the domain of high-performance
computing by multi-staged programming. MetaOCaml extends the OCaml
language.
...
   Benchmarks for all presented implementations confirm that the
execution time can be reduced significantly by high-level
optimizations. Some MetaOCaml programs even run as fast as respective
C implementations. Furthermore, in situations where optimizations in
pure MetaOCaml are limited, computation hotspots can be explicitly or
implicitly exported to C. This combination of high-level and low-level
techniques allows optimizations which cannot be obtained in pure C
without enormous effort.''
			

Modules

Archive: http://groups.google.com/group/fa.caml/browse_thread/thread/a81606be6805729e#

Wouter Swierstra asked and Pierre-Evariste Dagand replied:
> I've been learning OCaml, coming from a Haskell background.
> While I'm familiar with functional programming, I am occasionally puzzled by
> errors relating to the module system.

That's normal, everyone does, at some point. Especially when you start
having Parameterized Modules (called Functors in ML) all over the
place.

> I was wondering if there was a
> comprehensive overview/tutorial of OCaml's module system.

For a comprehensive source of information, there is the OCaml manual:

http://caml.inria.fr/pub/docs/manual-ocaml/manual004.html

For a tutorial on modules, I would recommend Chapter 14 of "Developing
Applications With Objective Caml":

http://caml.inria.fr/pub/docs/oreilly-book/html/index.html
			
blue storm then added:
I'd also add :
- the "formal" part of the manual (Part II) description of module
types : http://caml.inria.fr/pub/docs/manual-ocaml/manual018.html . In
particular the "Type specifications" section is quite informative
imho.
- the module-related chapter of the "UUU book" :
http://caml.inria.fr/pub/docs/u3-ocaml/ocaml-modules.html

If you're interested in theory, you can find an introduction to the ML
Module system theory in the Pierce's book "Advanced topic in Types and
Programming Languages" [1], and a some more advanced papers in the "A
Few Paper On Caml" section of this website :
http://caml.inria.fr/about/papers.en.html

[1] http://books.google.com/books?id=A5ic1MPTvVsC&pg=PA293&dq=pierce+module+ML#v=onepage&q=pierce%20module%20ML&f=false
			

Other Caml News

From the ocamlcore planet blog:
Thanks to Alp Mestan, we now include in the Caml Weekly News the links to the
recent posts from the ocamlcore planet blog at http://planet.ocamlcore.org/.

Core Gems: Time:
  http://ocaml.janestcapital.com/?q=node/73

Reflecting on One-Liners:
  http://alaska-kamtchatka.blogspot.com/2009/11/reflecting-on-one-liners.html

Dyn alpha01:
  http://caml.inria.fr/cgi-bin/hump.cgi?contrib=714

Xapi toolstack:
  http://caml.inria.fr/cgi-bin/hump.cgi?contrib=713

The next server, please!:
  http://blog.camlcity.org/blog/ocamlnet3_ha.html

ocamlbuild-plus:
  http://forge.ocamlcore.org/projects/ocamlbuildplus/
      

Old cwn

If you happen to miss a CWN, you can send me a message and I'll mail it to you, or go take a look at the archive or the RSS feed of the archives.

If you also wish to receive it every week by mail, you may subscribe online.


Alan Schmitt