OCaml Weekly News

When using or writing PPX rewriters, it's pretty bloody difficult to dagnose errors, especially when using multiple PPX rewriters. I've been writing my own using camlp5, and both when trying to figure out what existing rewriters do (so I could reverse-engineer them) and debugging my own, it was pretty painful. ocamlc -dsource really isn't enough. So with not-ocamlfind, you can take any ocamlfind ocamlc invocation, viz.

ocamlfind ocamlc -package ppx_deriving.show simple_show.ml

and replace ocamlfind ocamlc with not-ocamlfind preprocess:

not-ocamlfind preprocess -package ppx_deriving.show simple_show.ml

and get the preprocessed output (along with a list of the commands used to produce that output), so you can run it yourself. In addition, there's an executable papr_official.exe installed in the not-ocamlfind package-directory, that you can invoke to convert text to binary-AST and back. Notice how the steps are i. convert text to binary AST; ii. run PPX rewriter; iii. convert binary AST to text. Steps i and iii are of course not executed by ocamlc: they're there so we can run this outside of ocamlc and see the result.

ppx_execute: ocamlfind not-ocamlfind/papr_official.exe -binary-output -impl simple_show.ml /tmp/simple_showa44366
ppx_execute: /home/chetsky/Hack/Ocaml/GENERIC/4.10.0/lib/ppx_deriving/./ppx_deriving package:ppx_deriving.show
/tmp/simple_showa44366 /tmp/simple_show2225a0
format output file: ocamlfind not-ocamlfind/papr_official.exe -binary-input -impl /tmp/simple_show2225a0

N.B. If you use this with camlp5, you need to add a "pr_o.cmo" (or the equivalent" in order to get human-readable output.

That's all: just a way to more transparently debug PPX rewriter invocations.

When building moderate-to-large projects, none of the build-tools are very suitable: a. Makefiles quickly grow unwieldy, esp. when .cmo files from one directory are used as inputs in another; it gets difficult to ensure minimal-work rebuilds ("change file f1 in directory a, and everything downstream has to get rebuilt") while still actually rebuilding what does need to be rebuilt. b. ocamlbuild was …. impenetrable, and completely inimical to C/C++. c. ditto Oasis. I used oasis a lot, and it was always difficult. d. dune is pretty opaque too.

Of all these tools, Makefiles come closest, except for the complete lack of "composability". So let's fix that!

a. Each subdirectory of a large build, as its last step, installs a package into a "local install" repository, and other subdirectories used those packages instead of directly referring to files from other directories b. In the "depend" section of each subdirectory's Makefile, it lists the package-directories it depends upon, thus:

EXTERNAL := $(shell $(OCAMLFIND) query -predicates byte $(PACKAGES))
$(CMO): $(EXTERNAL)

So all the .cmo files in the directory get rebuilt, every time any of the package-directories used in this build are changed. c. In that last step, where the subdirectory's build installs an ocamlfind package, the problem is that that install will update last-modified times. So we need a new command:

$(NOT_OCAMLFIND) reinstall-if-diff pa_ppx_expect_test -destdir $(DESTDIR)/lib META $(TARGET) $(TARGET:.cma=.a)
$(TARGET:.cma=.cmxa) $(wildcard *.cmt*)

reinstall-if-diff compares the files-to-be-installed, with the files that are already in the package directory, and if there are any differences, invokes ocamlfind remove followed by ocamlfind install; if the files are identical, nothing is done. And we need a "local-install" target:

local-install::
	$(MAKE) DESTDIR=$(WD)/$(TOP)/local-install/ install

With these changes, a large project's toplevel Makefile can simply say:

all:
	set -e; for i in $(SYSDIRS) $(TESTDIRS); do cd $$i; $(MAKE) all; cd ..; done

and we can be assured that no superfluous build-steps will be run.

There is one little problem left, which I haven't cleaned-up, mostly because it isn't that troublesome: if the directories in that list $(SYSDIRS) aren't in topological order, then "make all" might fail. But this is, I suspect, an easily solvable problem. Just not high on the list of priorities right now.

All of the little snippets in this post are taken from the pa_ppx project: https://github.com/chetmurthy/pa_ppx

and the not-ocamlfind project is available both from opam and on github: https://github.com/chetmurthy/not-ocamlfind

• Zed_edit:
  • Set_pos action
  • Insert_str action

and notable changes:

• LTerm_read_line and LTerm_vi:
  • vi visual mode
  • register support

• compatible with OCaml 4.11
• switch to the new parser exposed since 4.11
• Vi edit mode: register support

Below is an illustration, that I split and store parts of a phrase into two registers(a and b) within the visual mode and then assemble them together.

• Updated Bin_prot functors Binable.Of_sexpable and Binable.Of_stringable to take UUIDs.
• Changed Base and Core partition* functions from using [ `Fst of 'a | `Snd of 'b ] to using ('a, 'b) Either.t.
• Replaced Date functions add_weekdays and add_business_days with functions that clarify the behavior w.r.t. weekends.
• Added syntax while%bind, analogous to if%bind.
• Added to the Async.Log.create function a required unit argument, and an optional argument to specify a Synchronous_time_source.t.
• Made Core_kernel deprecate the Unix module, recommending Core.Unix, with Caml_unix as a fallback.
• Changed Core_kernel.failwiths (aka Error.failwiths) to require its ~here argument.
• Added ppx_let syntax let%mapn and let%bindn, for n-ary map and bind.
• Renamed Expect_test_helpers as Expect_test_helpers_async, and renamed Expect_test_helpers_kernel as Expect_test_helpers_core. Removed the inclusion of Expect_test_helpers_core in Expect_test_helpers_async.
• Increased Async's default maximum number of open file descriptors from 2^15 to 2^16.
• Moved most of Core.Iobuf to a standalone library, Iobuf, that depends only on Core_kernel, and can be used in javascript. Moved the rest of Core.Iobuf into a standalone library, Iobuf_unix, that depends on Core.

• accessor (https://github.com/janestreet/accessor): A library that makes it nicer to work with nested functional data structures.
• accessor_async (https://github.com/janestreet/accessor_async): Accessors for Async types, for use with the Accessor library.
• accessor_base (https://github.com/janestreet/accessor_base): Accessors for Base types, for use with the Accessor library.
• accessor_core (https://github.com/janestreet/accessor_core): Accessors for Core types, for use with the Accessor library.
• expect_test_helpers_async (https://github.com/janestreet/expect_test_helpers_async): Async helpers for writing expectation tests.
• expect_test_helpers_core (https://github.com/janestreet/expect_test_helpers_core): Helpers for writing expectation tests.
• higher_kinded (https://github.com/janestreet/higher_kinded): A library with an encoding of higher kinded types in OCaml.
• incr_dom_interactive (https://github.com/janestreet/incr_dom_interactive): A monad for composing chains of interactive UI elements.
• incr_dom_sexp_form (https://github.com/janestreet/incr_dom_sexp_form): A library for building forms that allow the user to edit complicated types.
• ppx_accessor (https://github.com/janestreet/ppx_accessor): [@@deriving] plugin to generate accessors for use with the Accessor libraries.
• ppx_fixed_literal (https://github.com/janestreet/ppx_fixed_literal): Simpler notation for fixed point literals.
• ppx_log (https://github.com/janestreet/ppx_log): Ppx_sexp_message-like extension nodes for lazily rendering log messages.
• ppx_string (https://github.com/janestreet/ppx_string): ppx extension for string interpolation.
• re2_stable (https://github.com/janestreet/re2_stable): Re2_stable adds an incomplete but stable serialization of Re2.
• vcaml (https://github.com/janestreet/vcaml): OCaml bindings for the Neovim API.

Async:

• Deleted functions that were deprecated in 2018 and earlier.
• Deprecated the Deferred.choice type alias, in favor of Deferred.Choice.t.
• Removed Pipe.init, which has been deprecated since 2016.
• Deprecated Throttle.Deferred, which was a pointless alias for Deferred.

Core:

• Removed the Iobuf_debug module.

Core_kernel:

• Deprecated the Bug exception.
• Made Core_kernel deprecate the Unix module, recommending Core.Unix, with Caml_unix as a fallback.

Core:

• Moved most of Core.Iobuf to a standalone library, Iobuf, that depends only on Core_kernel, and can be used in javascript. Moved the rest of Core.Iobuf into a standalone library, Iobuf_unix, that depends on Core.
• Moved the Bigstring module out of Core to a standalone library, Bigstring_unix.
• Moved the Linux_ext module out of Core to a standalone library.

Core_kernel:

• Renamed Fqueue functions to be consistent with Queue and Fdeque.
• Moved Bus to a standalone library.

Async:

• Deleted functions that were deprecated in 2018 and earlier.

Async_kernel:

• Added Pipe.folding_filter_map' function.
• Added Synchronous_time_source.next_alarm_fires_at function.
• Added Synchronous_time_source.Event.Option module, an immediate option with support for optional syntax.
• Added Pipe.concat_pipe function.
• Added Scheduler.run_every_cycle_end, analogous to run_every_cycle_start.
• Exposed Scheduler.Expert.last_cycle_num_jobs function.
• Added Async_kernel_scheduler.Expert functions run_every_cycle_{start,end} and deprecated set_on_{start,end}_of_cycle.
• Added Async_kernel_scheduler.last_cycle_time accessor function.
• Deprecated the Deferred.choice type alias, in favor of Deferred.Choice.t.
• Exposed Synchronous_time_source function Expert.max_alarm_time_in_min_timing_wheel_interval.
• Optimized Time_source.advance_by_alarms and Scheduler.yield_until_no_jobs_remain to reduce the per-alarm cost from 660ns to ~140ns. Added to ~yield_until_no_jobs_remain an optional argument, ?may_return_immediately:bool to opt in to the faster behavior.
• Fixed an issue with Pipe.upstream_flushed, so that its result does not become determined if an exception happens while processing some of the elements.
• Removed Pipe.init, which has been deprecated since 2016.
• Deprecated Throttle.Deferred, which was a pointless alias for Deferred.

Async_rpc_kernel:

• Renamed Pipe_rpc.Direct_stream_writer.Group.flushed as flushed_or_closed, and changed it so that its result becomes determined if the individual pipes are flushed or closed.

Async_rpc:

• Added Rpc.Connection.serve_inet function, for synchronously creating a TCP IP server.

Async_unix:

• Added Tcp.Server.create_sock_inet function, which is synchronous and serves only inet addresses.
• Added Tcp.Server.create_inet function, for synchronously creating a TCP IP server.
• In Unix functions that create a File_descr.t supplied ~close_on_exec:true rather than calling set_close_on_exec.
• Changed blocking Sexp.save* functions from @@deprecated to @@alert.
• Made Async_unix shadow Core_kernel.eprint_s, which can block.
• Added to the Log.create function a required unit argument, and an optional argument to specify a Synchronous_time_source.t.
• Made Writer.flushed_* functions consistently flush the synchronous output channel if one is set.
• Fixed a bug in Writer error handling so that it calls stopped_permanently when the writer is not allowed to write due to file-descriptor flags.
• Added to the Log module the ability to set a transform function applied to each message that is logged.
• Added Unix support for flock, paralleling the existing support for lockf, and extended the with_file function to support both mechanisms. Changed the lockf functions' lock names from Read | Write to Shared | Exclusive.
• Added Writer functions flushed_or_failed_*, which, unlike Writer.flushed, return a result when the underlying writer fails.
• Fixed a race condition in Shutdown.don't_finish_before.
• Added to Writer.with_file an optional ?syscall argument.
• Improved Fd error messages for syscall* functions and with_file_descr_deferred_exn.
• Improved Writer.with_file_atomic's error message when f closes t.
• Increased Async's default maximum number of open file descriptors from 2^15 to 2^16.
• Fixed a race in Writer.with_file_atomic and the save* functions that use it that could cause the file permissions to be incorrect.
• Fix a bug in Shutdown.shutdown_on_unhandled_exn where a process would never shutdown if Debug.log raised (e.g. if stderr was a broken pipe).
• Added to Sys.when_file_changes an on_exn optional argument.
• Added Process.send_signal function that is safe against pid reuse.
• Added Unix.waitpid_prompt function, a version of waitpid that guarantees that it's determined in the same async job where the wait system call is done.

Base:

• Fixed a bug in Int.round_nearest, which could overflow.
• Added Hashtbl.Merge_into_action.t type.
• Eliminated the Base_boot library, lib/base/boot, by including its ppx expansions directly in the Base library, lib/base/src.
• Changed partition* functions from using [ `Fst of 'a | `Snd of 'b ] to using ('a, 'b) Either.t.
• Added Hashtbl functions find_and_call1 and find_and_call2, aimed at avoiding allocation.
• Improved the Gc module's deprecation message to recommend using Caml.Gc.
• Added List.is_suffix function.
• Added Map.combine_errors function.
• Added Result functions to_either and of_either, and deprecated ok_fst.
• Added Char.Caseless submodule, analogous to String.Caseless.
• Added String.Caseless functions is_substring* and substr*, for case-insensitive substring matching.
• Added the Nothing module, moved from Core_kernel.
• Added Ref function sets_temporarily, a generalization of set_temporarily to multiple refs.
• Changed Float.min and Float.max to never return Float.nan, and instead to always return one of the arguments. This improves performance by avoiding allocation.
• Added Option.try_with_join.
• Added Int_intf.S functions clz and ctz, for leading and trailing zeros in the binary representation of an int.
• Added Nothing module (moved from Core_kernel), which defines type t = |, the uninhabited type.
• Dropped support for OCaml < 4.07.
• Added Source_code_position.of_pos, which allows you to create Source_code_position.t~s from the OCaml built-in ~Caml.__POS__ special value.
• Added Sequence functions to convert between Sequence.t and Caml.Seq.t.
• Added integer byte-swap functions, bswap*.
• Added Set.are_disjoint function.
• Added String.Search_pattern accessor functions pattern and case_sensitive.
• Added String functions chop_{suffix,prefix}_if_exists.
• Added Comparable functors Infix and Polymorphic_compare, which implement the comparisons in their eponymous signatures, and require only compare as input.
• Added Blit.S1_distinct interface, which allows the src and dst types to differ.
• Added Uniform_array.iteri function.
• Added Map.validatei function.

Bin_prot:

• Updated functors Binable.Of_sexpable and Binable.Of_stringable to take UUIDs.

Core:

• For Unix functions that create a File_descr.t, added an optional argument, ?close_on_exec:bool, for atomic creation of close-on-exec file descriptors.
• Added Unix function flock_blocking.
• Improved Unix.create_process_env's error message when it is called with ~prog_search_path:[].
• Removed the Iobuf_debug module.
• Added Iobuf functions copy and clone.
• Moved most of Core.Iobuf to a standalone library, Iobuf, that depends only on Core_kernel, and can be used in javascript. Moved the rest of Core.Iobuf into a standalone library, Iobuf_unix, that depends on Core.
• Added Unix.Env submodule, with type Unix.Env.t = Unix.env, and exposed functions for manipulating Env.t.
• Added Filename function open_temp_file_fd, which is like open_temp_file but returns a file descriptor rather than an out channel.
• Made Time_ns.{Span,Ofday}.Option match Quickcheck.S.
• Moved the Bigstring module out of Core to a standalone library, Bigstring_unix.
• Moved the Linux_ext module out of Core to a standalone library.

Core_kernel:

• Added Bus.Callback_arity.Arity5.
• Generalized Univ_map to support a user-supplied key type.
• Added Gc.For_testing submodule, with functions for measuring allocation, previously in Expect_test_helpers_kernel.
• Renamed Fqueue functions to be consistent with Queue and Fdeque.
• Moved Bus to a standalone library.
• Added to Option.Stable.V1.t @@deriving equal.
• Added to Time_ns.Stable.V1 Map and Set modules.
• Replaced Date functions add_weekdays and add_business_days with functions that clarify the behavior w.r.t. weekends.
• Fixed a bug in Command.run that caused incorrect calls in let%test_unit to mistakenly succeed.
• Added Command.Auto_complete submodule, naming the type of Arg_type.create's ~complete argument.
• Added Date.Option submodule, Immediate_option.S.
• Added submodule Percent.Option : Immediate_option.S_without_immediate.
• Changed Command.Param.choose_one to disallow parameters which don't have any CLI flags.
• Changed Blang.or_ and Blang.and_ to make trees that short-circuit more quickly by traversing fewer Blang.t nodes at eval.
• Added to Command.run an optional argument, ?when_parsing_succeeds:(unit -> unit), which runs before the program's main thunk.
• Improved Command.Param.choose_one's error messages.
• Deprecated the Bug exception.
• Made Core_kernel deprecate the Unix module, recommending Core.Unix, with Caml_unix as a fallback.
• Added Gc function allocated_words : unit -> int.
• Changed Core_kernel.failwiths (aka Error.failwiths) to require its ~here argument.
• Changed Command argument parsing so that -- is only interpreted as a prefix of exactly --, an in particular not a prefix of --help.
• Extended String.Stable.V1 to match Stringable.S.
• Added Hash_queue function lookup_and_remove.
• Add Set support for writing Set.V1.M(Elt).t in stable types.
• Added Byte_units.arg_type.
• Extracted from Command the Shape-related code into a new file, command_shape.ml.
• Made Date.Option match Comparable.S_plain and Quickcheckable.S.
• Added Quickcheckable.Of_quickcheckable* functors.
• Added Time_ns.Ofday.every function.
• Made [@@deriving quickcheck] work with Base-style maps and and sets, e.g. type t = int Map.M(String).t [@@deriving quickcheck].
• Added to Command support for validating a command line, via the function Command_test_helpers.validate_command_line.
• Added Filename.to_absolute_exn function.
• Updated gc_stubs.c to work better with OCaml 4.10.
• Exposed Quickcheck.Shrinker.filter* functions.
• Exposed the Univ_map.Data module type.

Expect_test_helpers:

• Generalized print_and_check_container_sexps so that it applies to stable container modules.
• Fixed the handling of quickcheck shrinkers to print only the final failing value, rather than every value that was tried and failed.
• Added Async analogs of Ref.set_temporarily and sets_temporarily: set_temporarily_async, sets_temporarily_async.
• Renamed Expect_test_helpers as Expect_test_helpers_async, and renamed Expect_test_helpers_kernel as Expect_test_helpers_core. Removed the inclusion of Expect_test_helpers_core in Expect_test_helpers_async.

Expect_test_helpers_async:

• Removed Expect_test_helpers_async.Expect_test_config.

Expect_test_helpers_base:

• Added functions replace and replace_s, for rewriting strings and sexps in output.
• Added function hide_temp_files_in_string, for hiding temp file names in expect-test output.
• Changed require_sets_are_equal to take a module matching Comparator.S rather than a fragment of Set.S, which better fits with Base idioms.

Incremental:

• Added function node_value, to get the current value of a node, even if it is stale or invalid.
• Added support for syntaxes let%mapn and let%bindn.
• Added Incremental.Var.replace function, analogous to Ref.replace.

Linux_ext:

• Added function peer_credentials, for using the SO_PEERCRED socket option.

Stdio:

• Optimized In_channel.input_all to eliminate an unnecessary copy between buffers.

Timezone:

• Added Timezone support in javascript.

• API: Application Programming Interface
• CTF: Common Trace Format
• DWARF: Debugging With Attributed Record Formats
• GC: Garbage Collector
• GUI: Graphical User Interface
• LU: Lower-Upper
• OPAM: OCaml Package Manager
• PIP: Pip Installs Python
• PR: Pull Request
• RFC: Request for Comments
• UI: User Interface