Havoc's Blog

Update: see also this post on JSON-like config formats.

My latest work project was a quick side-track to unify the config file handling for Akka 2.0 and Play 2.0. The result is on GitHub and feels pretty well-baked. Patches have now landed in both Akka and Play, thanks to Patrik and Peter.

I can’t make this project seem glamorous. It was a code cleanup that reinvented one of the most-reinvented wheels around. But I thought I’d introduce this iteration of the wheel for those who might encounter it or want to adopt it in their own project.

The situation with Akka 1.2 and Play 1.2 was:

• Akka 1.2 used a custom syntax that was JSON-like semantically, but prettier to human-edit. It supported features such as including one file in another.
• Play 1.2 used a Java properties file that was run through Play’s template engine, and supported some other custom stuff such as substituting environment variables (the syntax looked like ${HOME}).

Akka’s format looked like this:

actor {
    timeout = 5
    serialize-messages = off
}

While Play was like this:

application.name=yabe
db=${DATABASE_URL}
date.format=yyyy-MM-dd

With the new 2.0 setup, both Akka and Play support your choice of three formats: JSON, Java properties, or a new one called “Human-Optimized Config Object Notation“. You can mix and match; if you have multiple files in different formats, their contents are combined.

HOCON has a flexible syntax that can be JSON (it’s a superset), or look much like Akka’s previous file format, or look much like a Java properties file. As a result, some existing Akka and Play config files will parse with no changes; others will require minor changes.

A single configuration file for the whole app

Play 1.2 has a single configuration file; everything you might want to set up is done in application.conf. We wanted to keep a single configuration, even as Play 2.0 adds a dependency on Akka.

With the new setup, once Play moves to Akka 2.0, you should be able to set Akka settings in your Play application.conf. If other libraries in your app also use the config lib, you should be able to set their settings from this single config file, as well.

To make this happen, apps and libraries have to follow some simple conventions. A configuration is represented by an object called a Config, and after loading a Config applications should provide it to all of their libraries. Libraries should have a way to accept a Config object to use, as shown in this example.

Applications can avoid having to pass a Config instance around by using the default Config instance; to make this possible, all libraries should use the same default, obtained from ConfigFactory.load(). This default loads application.conf, application.json, and application.properties from the classpath, along with any resources called reference.conf.

For a given app, either everyone gets a Config passed down from the app and uses that, or everyone defaults to the same “standard” Config.

Keeping useful features from the existing formats

Akka allowed you to split up the config into multiple files assembled through include statements, and the new format does too.

Play allowed you to grab settings such as ${DATABASE_URL} from the system environment, and the new format does too.

In the spirit of those two features, the new format also allows ${} references within the config, which enables “inheritance” and otherwise avoids cut-and-paste; there are some examples in the README.

Migration path

Some existing Akka and Play config files will parse unchanged in the new format. Handling of special characters, escaping, and whitespace does differ, however, and you could encounter those differences. To migrate from an existing Play application.conf, you can use one of two strategies:

1. Rename the file to application.properties, which will make the escaping rules more like the old format. However, you won’t be able to use environment variable substitution, it’s just a plain vanilla properties file.
2. Add quoting and escaping. If you get parse errors, add JSON-style double quotes around the strings causing the problem.

Akka is similar; if you have parse errors, you might need quoting and escaping to avoid them. The error messages should be clear: if they are not, let me know.

There’s a section in the HOCON spec (search for “Note on Java properties”, near the end) with a list of ways the new format differs from a Java properties file.

Override config at deploy time

After compiling your app, you may want to modify a configuration at deploy time. This can be done in several ways:

• With environment variables if you refer to them using ${DATABASE_URL} syntax in your config.
• System properties override the config, by default. Set -Dfoo.bar=42 on the command line and it will replace foo.bar in the app’s config.
• Force an alternative config to load using the system properties config.file, config.resource, or config.url. (Only works on apps using the default ConfigFactory.load(), or apps that independently implement support for these properties.)

A more machine-friendly syntax

To generate the previous Play or Akka formats, you would need custom escaping code. Now you can just generate JSON or a properties file, using any existing library that supports those standard formats.

Implemented in Java

The config library is implemented in Java. This allows Java libraries to “join” the config lib way of doing things. In general the Typesafe stack (including Play and Akka) has both Java and Scala APIs, and in this case it seemed most appropriate to implement in Java and wrap in Scala.

That said, I haven’t implemented a Scala wrapper, since it seems barely necessary; the API is small and not complicated. You can easily create an implicit enhancement of Config with any convenience methods you would like to have. While the API is a Java API, it does some things in a Scala-inspired way: most notably the objects are all immutable.

The implementation is much larger and more complex than it would have been if it were implemented in Scala. But I endured the Java pain for you.

Conventional way of managing defaults

By convention, libraries using the config lib should ship a file called reference.conf in their jar. The config lib loads all resources with that name into ConfigFactory.defaultReference() which is used in turn by ConfigFactory.load(). This approach to loading defaults allows all libraries to contribute defaults, without any kind of runtime registration which would create ordering problems.

(By the way: all of these conventions can be bypassed; there are also methods to just parse an arbitrary file, URL, or classpath resource.)

Well-defined merge semantics

The HOCON spec defines semantics for merging two config objects. Merging happens for duplicate keys in the same config file, or when combining multiple config files.

The API exports the merge operation as a method called withFallback(). You can combine any two config objects like this:

val merged = config.withFallback(otherConfig)

And you can combine multiple config objects with chained invocations of withFallback(), for example:

val merged = configs.reduce(_.withFallback(_))

withFallback() is associative and config objects are immutable so the potentially-parallel reduce() should work fine.

Retrieving settings

This is straightforward:

val foobar = config.getInt("foo.bar")

The getters such as getInt() throw an exception if the setting is missing or has the wrong type. Typically you have a reference.conf in your jar, which should ensure that all settings are present. There’s also a method checkValid() you can use to sanity-check a config against the reference config up front and fail early (this is nicer for users).

Each Config object conceptually represents a one-level map of paths to non-null values, but also has an underlying JSON-parse-tree-style representation available via the root() method. root() gives you a ConfigObject which corresponds pretty exactly to a JSON object, including null values and nested child object values. Config and ConfigObject are alternative views on the same data.

Any subtree of a config is just as good as the root; handy if you want multiple separately-configurable instances of something.

Configuration as data or code

I know many people are experimenting with configuration as Scala code. For this cleanup, we kept configuration as data and implemented the library in plain Java. My impression is that often a machine-manipulable-as-data layer ends up useful, even though there’s a code layer also. (See your .emacs file after using M-x customize, for example, it inserts a “do not edit this” section; or, SBT’s equivalent of that section is the .sbt file format.) But we did not think about this too hard here, just kept things similar to the way they were in 1.2, while improving the implementation.

Have fun

Not a whole lot else to it. Please let me know if you have any trouble.