Havoc's Blog

Update: see also this post on JSON-like config formats.

My latest work project was a quick side-track to unify the config file handling for Akka 2.0 and Play 2.0. The result is on GitHub and feels pretty well-baked. Patches have now landed in both Akka and Play, thanks to Patrik and Peter.

I can’t make this project seem glamorous. It was a code cleanup that reinvented one of the most-reinvented wheels around. But I thought I’d introduce this iteration of the wheel for those who might encounter it or want to adopt it in their own project.

The situation with Akka 1.2 and Play 1.2 was:

• Akka 1.2 used a custom syntax that was JSON-like semantically, but prettier to human-edit. It supported features such as including one file in another.
• Play 1.2 used a Java properties file that was run through Play’s template engine, and supported some other custom stuff such as substituting environment variables (the syntax looked like ${HOME}).

Akka’s format looked like this:

actor {
    timeout = 5
    serialize-messages = off
}

While Play was like this:

application.name=yabe
db=${DATABASE_URL}
date.format=yyyy-MM-dd

With the new 2.0 setup, both Akka and Play support your choice of three formats: JSON, Java properties, or a new one called “Human-Optimized Config Object Notation“. You can mix and match; if you have multiple files in different formats, their contents are combined.

HOCON has a flexible syntax that can be JSON (it’s a superset), or look much like Akka’s previous file format, or look much like a Java properties file. As a result, some existing Akka and Play config files will parse with no changes; others will require minor changes.

A single configuration file for the whole app

Play 1.2 has a single configuration file; everything you might want to set up is done in application.conf. We wanted to keep a single configuration, even as Play 2.0 adds a dependency on Akka.

With the new setup, once Play moves to Akka 2.0, you should be able to set Akka settings in your Play application.conf. If other libraries in your app also use the config lib, you should be able to set their settings from this single config file, as well.

To make this happen, apps and libraries have to follow some simple conventions. A configuration is represented by an object called a Config, and after loading a Config applications should provide it to all of their libraries. Libraries should have a way to accept a Config object to use, as shown in this example.

Applications can avoid having to pass a Config instance around by using the default Config instance; to make this possible, all libraries should use the same default, obtained from ConfigFactory.load(). This default loads application.conf, application.json, and application.properties from the classpath, along with any resources called reference.conf.

For a given app, either everyone gets a Config passed down from the app and uses that, or everyone defaults to the same “standard” Config.

Keeping useful features from the existing formats

Akka allowed you to split up the config into multiple files assembled through include statements, and the new format does too.

Play allowed you to grab settings such as ${DATABASE_URL} from the system environment, and the new format does too.

In the spirit of those two features, the new format also allows ${} references within the config, which enables “inheritance” and otherwise avoids cut-and-paste; there are some examples in the README.

Migration path

Some existing Akka and Play config files will parse unchanged in the new format. Handling of special characters, escaping, and whitespace does differ, however, and you could encounter those differences. To migrate from an existing Play application.conf, you can use one of two strategies:

1. Rename the file to application.properties, which will make the escaping rules more like the old format. However, you won’t be able to use environment variable substitution, it’s just a plain vanilla properties file.
2. Add quoting and escaping. If you get parse errors, add JSON-style double quotes around the strings causing the problem.

Akka is similar; if you have parse errors, you might need quoting and escaping to avoid them. The error messages should be clear: if they are not, let me know.

There’s a section in the HOCON spec (search for “Note on Java properties”, near the end) with a list of ways the new format differs from a Java properties file.

Override config at deploy time

After compiling your app, you may want to modify a configuration at deploy time. This can be done in several ways:

• With environment variables if you refer to them using ${DATABASE_URL} syntax in your config.
• System properties override the config, by default. Set -Dfoo.bar=42 on the command line and it will replace foo.bar in the app’s config.
• Force an alternative config to load using the system properties config.file, config.resource, or config.url. (Only works on apps using the default ConfigFactory.load(), or apps that independently implement support for these properties.)

A more machine-friendly syntax

To generate the previous Play or Akka formats, you would need custom escaping code. Now you can just generate JSON or a properties file, using any existing library that supports those standard formats.

Implemented in Java

The config library is implemented in Java. This allows Java libraries to “join” the config lib way of doing things. In general the Typesafe stack (including Play and Akka) has both Java and Scala APIs, and in this case it seemed most appropriate to implement in Java and wrap in Scala.

That said, I haven’t implemented a Scala wrapper, since it seems barely necessary; the API is small and not complicated. You can easily create an implicit enhancement of Config with any convenience methods you would like to have. While the API is a Java API, it does some things in a Scala-inspired way: most notably the objects are all immutable.

The implementation is much larger and more complex than it would have been if it were implemented in Scala. But I endured the Java pain for you.

Conventional way of managing defaults

By convention, libraries using the config lib should ship a file called reference.conf in their jar. The config lib loads all resources with that name into ConfigFactory.defaultReference() which is used in turn by ConfigFactory.load(). This approach to loading defaults allows all libraries to contribute defaults, without any kind of runtime registration which would create ordering problems.

(By the way: all of these conventions can be bypassed; there are also methods to just parse an arbitrary file, URL, or classpath resource.)

Well-defined merge semantics

The HOCON spec defines semantics for merging two config objects. Merging happens for duplicate keys in the same config file, or when combining multiple config files.

The API exports the merge operation as a method called withFallback(). You can combine any two config objects like this:

val merged = config.withFallback(otherConfig)

And you can combine multiple config objects with chained invocations of withFallback(), for example:

val merged = configs.reduce(_.withFallback(_))

withFallback() is associative and config objects are immutable so the potentially-parallel reduce() should work fine.

Retrieving settings

This is straightforward:

val foobar = config.getInt("foo.bar")

The getters such as getInt() throw an exception if the setting is missing or has the wrong type. Typically you have a reference.conf in your jar, which should ensure that all settings are present. There’s also a method checkValid() you can use to sanity-check a config against the reference config up front and fail early (this is nicer for users).

Each Config object conceptually represents a one-level map of paths to non-null values, but also has an underlying JSON-parse-tree-style representation available via the root() method. root() gives you a ConfigObject which corresponds pretty exactly to a JSON object, including null values and nested child object values. Config and ConfigObject are alternative views on the same data.

Any subtree of a config is just as good as the root; handy if you want multiple separately-configurable instances of something.

Configuration as data or code

I know many people are experimenting with configuration as Scala code. For this cleanup, we kept configuration as data and implemented the library in plain Java. My impression is that often a machine-manipulable-as-data layer ends up useful, even though there’s a code layer also. (See your .emacs file after using M-x customize, for example, it inserts a “do not edit this” section; or, SBT’s equivalent of that section is the .sbt file format.) But we did not think about this too hard here, just kept things similar to the way they were in 1.2, while improving the implementation.

Have fun

Not a whole lot else to it. Please let me know if you have any trouble.

TL;DR

Modern application development platforms are addressing the related issues of globally-coordinated task dispatch and nonblocking IO.

Here’s my definition of the problem, an argument for why it matters, and some suggestions for specific standard library features to add to Scala in particular.

The same ideas apply to any application development platform, though. It’s rapidly becoming mandatory for a competitive platform to offer an answer here.

Definitions

Let’s define a blocking task to be anything that ties up a thread or process but does not use CPU cycles. The most common ways to block are on IO channels and locks.

A busy loop is not a blocking operation in this sense; it takes up a thread, but it’s using the CPU, not “wasting” the thread.

By “task” I mean any piece of executable code. A task is blocking if it spends part of its time waiting, or nonblocking if it needs the CPU the whole time.

Dispatch just means scheduling the task on a thread and executing it.

Dispatch for nonblocking tasks, in an ideal world

For nonblocking tasks (which are CPU-bound), the goal is to use 100% of all CPU cores. There are two ways to lose:

• Fail to use all the cores (not enough threads or processes).
• Too many threads for the number of cores (inefficient and wastes memory).

The ideal solution is a fixed thread or process pool with a number of threads related to the number of cores. This fixed pool must be global to the app and used for all nonblocking tasks. If you have five libraries in your app and they each create a thread per CPU core, you’re losing, even though each library’s approach makes sense in isolation.

When the fixed number of threads are all in use, tasks should be queued up for later dispatch.

Dispatch for blocking tasks, in an ideal world

Blocking tasks pose some competing concerns; the trouble with blocking tasks is that these concerns are hard to balance.

• Memory: each blocking task ties up a thread, which adds overhead (the thread) to the task. A super-tiny http parser gets you nowhere if you accompany each one with a thread.
• Deadlocks: blocking tasks are often waiting for another blocking task. Limiting the number of threads can easily create deadlocks.
• Tasks outstanding: with IO, it is desirable to send lots of requests at once or have lots of sockets open at once. (With CPU-bound tasks, the opposite is true.)

The ideal solution (if you must block) is an “as huge as memory allows” thread/process pool.

If you run blocking tasks on a bounded pool, you could have deadlocks, and you would not maximize tasks outstanding. Still, as memory pressure arrives, it would be better to start making some tasks wait than it would be to exhaust memory. Apps inevitably become pathological when memory is exhausted (either you have swap and performance goes to hell, or you don’t have swap and an out-of-memory exception breaks the app). But as long as memory is available, it’s better to add threads to the pool than it is to queue tasks.

An automatic solution to this problem might require special help from the JVM and/or the OS. You’d want to have an idea about whether it’s reasonable to create another thread, in light of each thread’s memory usage, the amount of memory free, and whether you can GC to recover memory.

In practice, you have to do some amount of manual tuning and configuration to get a thread pool setup that works in practice for a particular deployment. Maybe setting a large, but fixed, thread pool size that happens to keep your app using about the right amount of memory.

Different tasks, different pools

It’s broken to dispatch nonblocking tasks to an unbounded (or large) pool, and broken to dispatch blocking tasks to a small bounded pool. I can’t think of a nice way to handle both kinds of task with the same pool.

Possible conclusion: a dispatch API should permit the application to treat the two differently.

Physical resource coordination requires global state

We’ve all been taught to avoid global variables, global state, and singletons. They cause a lot of trouble, for sure.

Assuming your app runs on a single machine, you have N CPUs on your computer – period. You can’t create a new “CPUs context” with your own CPUs every time you need more CPUs. You have N megabytes of RAM – period. You can’t create a new “RAM context” with its own RAM.

These resources are shared among all threads and processes. Thus you need global, coordinated task dispatch.

Nonblocking IO creates another global resource

With nonblocking IO APIs, such as poll(), you can have multiple IO operations outstanding, using only one thread or process.

However, to use poll() or equivalent you have a new problem: every IO operation (on the same thread) must be coordinated so that the file descriptors end up in a single call to poll(). The system for coordinating this is called an “event loop” or “main loop.”

In an API such as libev or GMainLoop, applications can say “I need to wake up in 3 seconds” or “I need to know about activity on this socket handle,” and the library aggregates all such requests into a single invocation of poll(). The single poll() puts the thread to sleep until one of the requests is ready.

Nonblocking IO requires a globally-coordinated “managed poll()” — also known as an event loop. Otherwise you’re back to needing threads.

How Java sucks at this

In brief:

1. No global task dispatcher to coordinate CPU and memory usage.
2. The APIs are mostly blocking.
3. The nonblocking APIs in nio have limited utility because there’s no global event loop.

1. No global dispatcher

Java has all sorts of nice executors allowing you to dispatch tasks in many different ways.

But for average apps doing average things, we need two global singleton executors, not a zillion ways to create our own.

An average app needs the executor for nonblocking CPU-bound tasks, so that executor can coordinate CPU-limited tasks. And it needs the executor for blocking tasks, so that executor can coordinate memory-limited tasks.

In the JVM ecosystem, you start using a library for X and a library for Y, and each one starts up some tasks. Because there’s no global executor, each one creates its own. All those per-library executors are probably great by themselves, but running them together sucks. You may never create a thread by hand, but when you run your app there are 100 threads from a dozen different libraries.

2. Blocking APIs

With the standard Java APIs, many things are hard or impossible to do without tying up a thread waiting on IO or waiting on a lock. If you want to open a URL, there’s URL.openStream() right there in the standard library, but if you want to open a URL without blocking you’ll end up with a far more involved external dependency (such as AsyncHttpClient).

Just to kick you while you’re down, many of the external dependencies you might use for nonblocking IO will create at least one dedicated thread, if not a whole thread pool. You’ll need to figure out how to configure it.

3. No event loop

Low-level nonblocking APIs in the spirit of poll() are not enough. Even if every library or code module uses poll() to multiplex IO channels, each library or code module needs its own thread in which to invoke poll().

In Java, a facility as simple as Timer has to spawn its own threads. On platforms with an event loop, such as node.js, or browsers, or many UI toolkits, you tell the platform how long to wait, and it ensures that a single, central poll() has the right timeout to wake up and notify you. Timer needs a thread (or two) because there’s no centralized event loop.

The impact in practice

If you just use Java and Scala APIs naively in the most convenient way, you end up with a whole lot of threads. Then you have to start tracking down thread pools inside of all your libraries, sharing them when possible, and tuning their settings to match the size of your hardware and your actual production load. Or just buy a single piece of hardware more powerful than you’ll ever need and allow the code to be inefficient (not a rare strategy).

I recently wrote a demo app called Web Words, and even though it’s not complex, it shows off this problem well. Separately, the libraries it uses (such as Akka, AsyncHttpClient, Scala parallel collections, RabbitMQ) are well-behaved. Together, there are too many threads, resulting in far more memory usage than should be required, and inefficient parallelization of the CPU-bound work.

This is a whole category of developer tedium that should not exist. It’s an accident of broken, legacy platform design.

The node.js solution

node.js has a simple solution: don’t have any APIs that block. Implement all nonblocking APIs on top of a singleton, standard event loop. Run one process per CPU. Done.

Dispatch of blocking tasks is inherently hard, so node.js makes it impossible to implement a blocking task and avoids the problem.

This would fail without the global singleton event loop. If node.js provided poll() instead of an event loop, poll() would be a blocking API, and any task using it would take over the node.js process.

People often say that “no threads” is the secret to node.js; my take is that first the global singleton event loop enables all APIs to be nonblocking; second the lack of blocking APIs removes the need for threads. The global singleton event loop is the “root cause” which unlocks the big picture.

(The snarky among you may be thinking, “if you like node.js so much, why don’t you marry it?”; node.js is great, but I love a lot of things about Scala too. Type safety goes without saying, and I’ll also take actors over callbacks, and lots more. Comparing one aspect of the two platforms here.)

Event loop as a third-party library: not a solution

You can get event loop libraries for lots of languages. This is in no way equivalent to a standard, default global singleton event loop.

First: the whole point of an event loop is to share a single call to poll() among all parts of the application. If the event loop library doesn’t give you a singleton, then every code module has to create its own loop with its own poll().

Second: if the event loop singleton is not in the standard library, then the platform’s standard library can’t use it. Which means the standard library either has no IO facilities, or it has only broken, blocking IO facilities.

Solving it in Scala

This could be solved on the Java level also, and maybe people are planning to do so — I hope so.

In the meantime, if you’ve read this far, you can probably guess what I’d propose for Scala.

Blocking tasks are inherently somewhat intractable, but they are also a legacy fact of life on the JVM. My suggested philosophy: design APIs assuming nonblocking tasks, but give people tools to manage blocking tasks as best we can.

The critical missing pieces (the first three here) should not be a lot of work in a strictly technical sense; it’s more a matter of getting the right people interested in understanding the problem and powering through the logistics of landing a patch.

1. Two global singleton thread pools

The Scala standard library should manage one thread pool intended for CPU-bound nonblocking tasks, and one intended for blocking tasks.

• The simplest API would let you submit a function to be executed in one of the pools.
• Another useful API would create an ExecutorService proxy that used the pools to obtain threads. ExecutorService.shutdown() and ExecutorService.awaitTermination() would have to work correctly: wait for tasks submitted through the proxy to complete, for example. But shutting down the proxy should not interfere with the underlying global thread pool. The proxy would be provided to legacy Java APIs that allow you set a custom ExecutorService.

Built-in Scala features such as actors and parallel collections should make use of these thread pools, of course.

2. A global default event loop instance

The goal is to allow various code modules to add/remove channels to be watched for IO events, and to set the timeout on poll() (or equivalent) to the earliest timeout requested by any code module.

The event loop can be very simple; remember, it’s just a way to build up a poll() invocation that “takes requests” from multiple code modules. More complexity can be assembled on top.

3. A standard Future trait

The standard Future in Scala doesn’t quite have what people need, so there’s a proliferation of third-party solutions, most of them similar in spirit. There’s even a wrapper around all the flavors of Future, called sff4s.

(Note: all of these Future are nonblocking, while Java’s Future requires you to block on it eventually.)

A standard Future is essential because it’s the “interoperability point” for nonblocking APIs. Without a good Future in the standard library, there can’t be good nonblocking APIs in the standard library itself. And without a standard Future, third-party nonblocking APIs don’t play nicely together.

4. Bonus Points and Optional Elaborations

await

In my opinion, the C# await operator with async keyword is the Right Thing. Microsoft understands that great async support has become a required language feature. At my last company, C. Scott Ananian built a similar continuation-passing-style feature on JavaScript generators and it worked great.

Scala has the shift/reset primitives for continuation-passing style, but it isn’t clear to me whether they could be used to build something like await, and if they could, there’s nothing in the standard library yet that does so.

await depends on a standard Future because it’s an operator on Future. In Scala, await would probably not be a language feature, it would be a library function that operated on Future. (Assuming the shift/reset language feature is sufficient to implement that.)

Nonblocking streams

Many, many Java APIs let you pass in or obtain an InputStream or an OutputStream. Like all blocking APIs, these are problematic. But there isn’t a great alternative to use when designing an API; you’d have to invent something custom. The alternative should exist, and be standard. The standard library should have a nonblocking version of URL.openStream(), too.

Actors

If you code exclusively with actors and write only nonblocking code inside your actors, there’s never a need to mess with dispatchers, futures, or event loops. In that sense, actors solve all the problems I’m describing here. However: to code exclusively with actors, you need libraries that implement actor-based alternatives to all blocking APIs. And those libraries in turn need to be implemented somehow. It can’t be actors all the way down. There’s also the matter of using existing, non-actor APIs.

Ideally, Akka, and associated actor-based APIs, would build on a standard task dispatch and event loop facility. Akka already builds on regular Java executors; it doesn’t need anything super-fancy.

An issue with actors today is that they’re allowed to block. My idea: there should be a way to mark a blocking actor as such. It would then be dispatched to the unbounded executor intended for blocking tasks. By default, Akka could dispatch to the bounded executor intended for nonblocking tasks. People who want to block in their actors could either mark them blocking so Akka knows, or (brokenly) configure their nonblocking task executor to be unbounded. If you do everything right (no blocking), it would work by default. Remember the theory “design for nonblocking by default, give people tools to damage-control blocking.”

JDBC and similar

Peter Hausel pointed out to me recently that JDBC only comes in blocking form. Now there’s a fun problem… There are quite a few libraries with similar issues, along with people trying to solve them such as Brendan McAdams‘s Hammersmith effort for MongoDB.

What’s the big deal?

Developers on platforms with no good solution here are used to the status quo. When I first came to server-side development from client-side UI toolkit development, the lack of an event loop blew my mind. I’m sure I thought something impolitic like How can anybody use anything so freaking broken??? In practice of course it turns out to be a manageable issue (obviously lots of stuff works great running on the JVM), but it could be better.

In my opinion, with node.js, C# adding async and await to the language, and so on, good answers in this area are now mandatory. People are going to know there’s a better way and complain when they get stuck using a platform that works the old way.

A high-quality application development platform has to address global task dispatch coordination, have an event loop, and offer nice nonblocking APIs.

Related posts…

Some old posts on related topics.

• Guidelines for sync and async callbacks in nonblocking APIs.
• I wrote a node.js-like code experiment that shared one event loop among many threads and JavaScript global objects, there’s some discussion of using JS yield like C# await in that post.
• I claimed offhand above that out-of-memory exceptions always break apps, here’s why I think that.

Often when our minds turn to design, we think first of cosmetics. With all the talk of Apple these days, it’s easy to think they’re successful because of shiny aluminum or invisible screw holes.

But this is a side issue. The Steve Jobs quote that gets to the heart of product development might be: So why the fuck doesn’t it do that?

If you’re trying to make a successful tech product, 90% of the battle is that it works at all.

“Works” means: the intended customers really use it for its intended purpose.

Plain old bugs (deviations from the design specification) can keep a product from working, if they mean people don’t use the product. But bugs can be irrelevant.

It’s easy to fail even without bugs:

• Using the product has too many steps or a single “too hard” step. People give up.
• The product solves a non-problem or the wrong problem. Nobody will use it.

It doesn’t matter why people don’t or can’t use the product. If they don’t use it, it does not work.

Startups based on making it work

Plenty of startups were successful because they broke the “it has to work” barrier. A few famous examples:

• Dropbox
• Sonos
• Flip

How many “file sync” products existed before Dropbox? It must have been thousands. But it’s easy to create a nonworking file sync product. Too many steps, too much complexity, or too many bugs: any of these could mean “file sync” didn’t work.

Can you imagine pitching these companies to potential investors? “Yeah, thousands of people have failed to do our idea. But we’re going to finally do it right.” Tough sell.

Working vs. well-designed

Dropbox, Sonos, and Flip are working products people seem to agree were well-designed. But some successful products have pretty bad cosmetics:

• despite its eventual failure, MySpace became popular initially because it met a need and it worked
• craigslist
• eBay
• git got the big picture right, but every small UI design decision or “cosmetic” issue seemed to be wrong
• HTML/CSS/JavaScript (now I’m just trolling, sorry)

“Working” is a little bit different from “well-designed.” Sometimes working can involve “worse is better,” perhaps. “Working” has to do with solving a problem, not (necessarily) doing so in an elegant or beautiful way.

“It just works” vs. solving the problem

The phrase “it just works” comes up a lot, and that’s almost enough. “Just working” seems to mean there aren’t many manual steps to use the product.

But you still have to solve the problem. I remember the iPod because it solved my music problem: its use of a hard drive meant I could get all my music on there, and listen to all my music. That’s what I wanted to do. Other players didn’t work for me because they didn’t hold all my music, which created a problem (deciding what to put on there) rather than solving one (getting rid of all those CDs). To this day, I find the iTunes app hard to use (bordering on terrible), but it works.

Easy to use is not quite the same as working, though perhaps it’s a helpful step.

QA that asks “does it work?”

At Red Hat, we used to use a sophisticated QA technique called the “yellow pad.” To yellow pad a product, you get one of those yellow legal pads. You need a fresh setup just like the one the customer will have (say, a computer your software has never been installed on). Then you install and try to use your software, writing down on the yellow pad anything that fails or looks embarrassing or nobody would ever understand.

Plenty of “finished” products will fail miserably.

QA teams and developers get tunnel vision. It’s easy to go for months with nobody on the team taking a fresh look at the product, step-by-step.

Once you can pass your own “yellow pad” criticism, or in parallel if you like, you can jump to hallway usability testing, maybe still using a yellow pad: watch someone else try to use the product for the first time, and again take notes. Fix that stuff.

The point is this: over and over and over, iteratively as you fix stuff, you need to try the product by walking through it, not by looking at features in isolation. Feature lists and requirements documents are death. Step-by-step stories are life.

I’m sure you can see the resonance with agile software development and lean startup methodology here, but you need not buy into a complex method or theoretical framework.

Yellow pads won’t help you solve the right problem, but they’ll get you past a lot of details that can sabotage you even if you’re solving the right problem.

Focus

A good way to kill a product: start adding extra features when it doesn’t work. First it needs to work. Focus on that. Then you can elaborate.

People who can’t tell if it works

Many people argue that non-technical CEOs can’t lead a technology company. One reason, I believe, is that many non-technical CEOs can’t tell whether a product works, or can’t understand which technical barriers to workingness are surmountable and which are fundamental.

Often, IT departments can’t tell if software works; they produce lists of requirements, but “it works” may not be one of them. Remember, “working” means “people will really use it in practice,” not “it can be made to do something if you have enough patience.”

Interaction design is a branch of design with an emphasis on step-by-step, and I find that designers with this background understand that it has to work. Many (not all) designers with other backgrounds may have an emphasis on the appearance of static snapshots, rather than the flow through the steps; and I’ve found that some of those designers don’t know whether a design will work.

Software developers are good at recognizing some ways that products don’t work, but as frequently noted, many of us overestimate the general public’s tolerance for complexity.

It might take a few years

Some kinds of product (notably, many web apps) can go from concept to launch in a few months or less, but whole categories cannot. Operating systems; anything involving hardware; non-casual video games such as World of Warcraft. In my experience, which spans a few projects anyway, complicated tech products tend to take a year or two to be “done” and around three years to work.

Sometimes I wonder if the “secret sauce” at Apple is no more than understanding this. Other hardware manufacturers have structural problems investing enough time in a single product. While I have no idea how long Apple spent developing the iPhone, I’m willing to bet it was at least three years. That’s more than enough time for most big companies to cancel a project.

Video game companies seem a little more open to investing the time required than other kinds of software companies, with major games spending several years in development. Video games don’t have the luxury of launching a “1.0” and then fixing it, I guess.

The first milestone that matters

If you’re building something, celebrate on the day that the product works.

You may have a long way to go yet (does anyone know your product exists?), but you’ve already succeeded where others failed.

Here are two guidelines for designing APIs that use callbacks, to add to my inadvertent collection of posts about minor API design points. I’ve run into the “sync vs. async” callback issue many times in different places; it’s a real issue that burns both API designers and API users.

Most recently, this came up for me while working on Hammersmith, a callback-based Scala API for MongoDB. I think it’s a somewhat new consideration for a lot of people writing JVM code, because traditionally the JVM uses blocking APIs and threads. For me, it’s a familiar consideration from writing client-side code based on an event loop.

Definitions

• A synchronous callback is invoked before a function returns, that is, while the API receiving the callback remains on the stack. An example might be: list.foreach(callback); when foreach() returns, you would expect that the callback had been invoked on each element.
• An asynchronous or deferred callback is invoked after a function returns, or at least on another thread’s stack. Mechanisms for deferral include threads and main loops (other names include event loops, dispatchers, executors). Asynchronous callbacks are popular with IO-related APIs, such as socket.connect(callback); you would expect that when connect() returns, the callback may not have been called, since it’s waiting for the connection to complete.

Guidelines

Two rules that I use, based on past experience:

• A given callback should be either always sync or always async, as a documented part of the API contract.
• An async callback should be invoked by a main loop or central dispatch mechanism directly, i.e. there should not be unnecessary frames on the callback-invoking thread’s stack, especially if those frames might hold locks.

How are sync and async callbacks different?

Sync and async callbacks raise different issues for both the app developer and the library implementation.

Synchronous callbacks:

• Are invoked in the original thread, so do not create thread-safety concerns by themselves.
• In languages like C/C++, may access data stored on the stack such as local variables.
• In any language, they may access data tied to the current thread, such as thread-local variables. For example many Java web frameworks create thread-local variables for the current transaction or request.
• May be able to assume that certain application state is unchanged, for example assume that objects exist, timers have not fired, IO has not occurred, or whatever state the structure of a program involves.

Asynchronous callbacks:

• May be invoked on another thread (for thread-based deferral mechanisms), so apps must synchronize any resources the callback accesses.
• Cannot touch anything tied to the original stack or thread, such as local variables or thread-local data.
• If the original thread held locks, the callback will be invoked outside them.
• Must assume that other threads or events could have modified the application’s state.

Neither type of callback is “better”; both have uses. Consider:

list.foreach(callback)

in most cases, you’d be pretty surprised if that callback were deferred and did nothing on the current thread!

But:

socket.connect(callback)

would be totally pointless if it never deferred the callback; why have a callback at all?

These two cases show why a given callback should be defined as either sync or async; they are not interchangeable, and don’t have the same purpose.

Choose sync or async, but not both

Not uncommonly, it may be possible to invoke a callback immediately in some situations (say, data is already available) while the callback needs to be deferred in others (the socket isn’t ready yet). The tempting thing is to invoke the callback synchronously when possible, and otherwise defer it. Not a good idea.

Because sync and async callbacks have different rules, they create different bugs. It’s very typical that the test suite only triggers the callback asynchronously, but then some less-common case in production runs it synchronously and breaks. (Or vice versa.)

Requiring application developers to plan for and test both sync and async cases is just too hard, and it’s simple to solve in the library: If the callback must be deferred in any situation, always defer it.

Example case: GIO

There’s a great concrete example of this issue in the documentation for GSimpleAsyncResult in the GIO library, scroll down to the Description section and look at the example about baking a cake asynchronously. (GSimpleAsyncResult is equivalent to what some frameworks call a future or promise.) There are two methods provided by this library, a complete_in_idle() which defers callback invocation to an “idle handler” (just an immediately-dispatched one-shot main loop event), and plain complete() which invokes the callback synchronously. The documentation suggests using complete_in_idle() unless you know you’re already in a deferred callback with no locks held (i.e. if you’re just chaining from one deferred callback to another, there’s no need to defer again).

GSimpleAsyncResult is used in turn to implement IO APIs such as g_file_read_async(), and developers can assume the callbacks used in those APIs are deferred.

GIO works this way and documents it at length because the developers building it had been burned before.

Synchronized resources should defer all callbacks they invoke

Really, the rule is that a library should drop all its locks before invoking an application callback. But the simplest way to drop all locks is to make the callback async, thereby deferring it until the stack unwinds back to the main loop, or running it on another thread’s stack.

This is important because applications can’t be expected to avoid touching your API inside the callback. If you hold locks and the app touches your API while you do, the app will deadlock. (Or if you use recursive locks, you’ll have a scary correctness problem instead.)

Rather than deferring the callback to a main loop or thread, the synchronized resource could try to drop all its locks; but that can be very painful because the lock might be well up in the stack, and you end up having to make each method on the stack return the callback, passing the callback all the way back up the stack to the outermost lock holder who then drops the lock and invokes the callback. Ugh.

Example case: Hammersmith without Akka

In Hammersmith as originally written, the following pseudocode would deadlock:

connection.query({ cursor => /* iterate cursor here, touching connection again */ })

Iterating the cursor will go back through the MongoDB connection. The query callback was invoked from code in the connection object… which held the connection lock. Not going to work, but this is natural and convenient code for an application developer to write. If the library doesn’t defer the callback, the app developer has to defer it themselves. Most app developers will get this wrong at first, and once they catch on and fix it, their code will be cluttered by some deferral mechanism.

Hammersmith inherited this problem from Netty, which it uses for its connections; Netty does not try to defer callbacks (I can understand the decision since there isn’t an obvious default/standard/normal/efficient way to defer callbacks in Java).

My first fix for this was to add a thread pool just to run app callbacks. Unfortunately, the recommended thread pool classes that come with Netty don’t solve the deadlock problem, so I had to fix that. (Any thread pool that solves deadlock problems has to have an unbounded size and no resource limits…)

In the end it works, but imagine what happens if callback-based APIs become popular and every jar you use with a callback in its API has to have its own thread pool. Kind of sucks. That’s probably why Netty punts on the issue. Too hard to make policy decisions about this in a low-level networking library.

Example case: Akka actors

Partly to find a better solution, next I ported Hammersmith to the Akka framework. Akka implements the Actor model. Actors are based on messages rather than callbacks, and in general messages must be deferred. In fact, Akka goes out of its way to force you to use an ActorRef to communicate with an actor, where all messages to the actor ref go through a dispatcher (event loop). Say you have two actors communicating, they will “call back” to each other using the ! or “send message” method:

actorOne ! Request("Hello")
// then in actorOne
sender ! Reply("World")

These messages are dispatched through the event loop. I was expecting my deadlock problems to be over in this model, but I found a little gotcha – the same issue all over again, invoking application callbacks with a lock held. This time it was the lock on an actor while the actor is processing a message.

Akka actors can receive messages from either another actor or from a Future, and Akka wraps the sender in an object called Channel. The ! method is in the interface to Channel. Sending to an actor with ! will always defer the message to the dispatcher, but sending to a future will not; as a result, the ! method on Channel does not define sync vs. async in its API contract.

This becomes an issue because part of the “point” of the actor model is that an actor runs in only one thread at a time; actors are locked while they’re handling a message and can’t be re-entered to handle a second message. Thus, making a synchronous call out from an actor is dangerous; there’s a lock held on the actor, and if the synchronous call tries to use the actor again inside the callback, it will deadlock.

I wrapped MongoDB connections in an actor, and immediately had exactly the same deadlock I’d had with Netty, where a callback from a query would try to touch the connection again to iterate a cursor. The query callback came from invoking the ! method on a future. The ! method on Channel breaks my first guideline (it doesn’t define sync vs. async in the API contract), but I was expecting it to be always async; as a result, I accidentally broke my second guideline and invoked a callback with a lock held.

If it were me, I would probably put deferral in the API contract for Channel.! to fix this; however, as Akka is currently written, if you’re implementing an actor that sends replies, and the application’s handler for your reply may want to call back and use the actor again, you must manually defer sending the reply. I stumbled on this approach, though there may be better ones:

private def asyncSend(channel: AkkaChannel[Any], message: Any) = {
    Future(channel ! message, self.timeout)(self.dispatcher)
}

An unfortunate aspect of this solution is that it double-defers replies to actors, in order to defer replies to futures once.

The good news about Akka is that at least it has this solution – there’s a dispatcher to use! While with plain Netty, I had to use a dedicated thread pool.

Akka gives an answer to “how do I defer callbacks,” but it does require special-casing futures in this way to be sure they’re really deferred.

(UPDATE: Akka team is already working on this, here’s the ticket.)

Conclusion

While I found one little gotcha in Akka, the situation is much worse on the JVM without Akka because there isn’t a dispatcher to use.

Callback-based APIs really work best if you have an event loop, because it’s so important to be able to defer callback invocation.

That’s why callbacks work pretty well in client-side JavaScript and in node.js, and in UI toolkits such as GTK+. But if you start coding a callback-based API on the JVM, there’s no default answer for this critical building block. You’ll have to go pick some sort of event loop library (Akka works great), or reinvent the equivalent, or use a bloated thread-pools-everywhere approach.

Since callback-based APIs are so trendy these days… if you’re going to write one, I’d think about this topic up front.

In a post a couple months ago I argued that a given financial effect on an individual could be called a “tax increase plus a tax credit for doing XYZ,” or a “tax penalty for not doing XYZ,” and that the two are identical in terms of how many dollars all parties involved end up with. Based on that, I was wondering why a tax penalty for not buying insurance would be legally different from the existing, longstanding tax credits for buying insurance.

Two of the judges in the recent Sixth Circuit opinion upholding the law addressed this issue, though the court as a whole declined to rule on taxing power grounds (because they upheld the law on commerce clause grounds anyway).

First reaction: wow, the Sixth Circuit reads my blog and answers my questions! Nice!  (Note to the thick: kidding.)

I was arguing (as a non-lawyer) that the government shouldn’t have the power to do something, or lack that power, based purely on what label they stick on it. Surely if the individual mandate were unconstitutional, the government could not fix the constitutional problem with a rewording that would have no practical effect. Should abridging free speech or skipping due process become OK as long as we use the right words to describe them?

(A small twist that might merit a footnote: due to a special rule in the health care bill, the IRS isn’t allowed to enforce the individual mandate’s penalty as strongly as they can enforce falsely claiming a credit. That’s a small practical difference between the penalty and a credit, but one that makes the penalty a weaker infringement on individual rights than the existing credits.)

In the Sixth Circuit opinion, the two judges addressing the tax issue write (see page 29 for the relevant stuff):

…it is easy to envision a system of national health care, including one with a minimum-essential-coverage provision, permissibly premised on the taxing power. Congress might have raised taxes on everyone in an amount equivalent to the current penalty, then offered credits to those with minimum essential insurance. Or it might have imposed a lower tax rate on people with health insurance than those without it.

That is, they agree with me that the tax increase plus the credit would have been the same thing as the penalty, economically speaking. (“Economically speaking” = the same parties end up with the same number of dollars in the same situations.)

But they go on and say it matters what you call it. In other words, they argue the taxing power does not include tax penalties for XYZ, but does include tax credits for not-XYZ.

I thought that was an absurdity proving either 1) the penalty is allowed or 2) the credits are not allowed, depending on your political bent. They embraced the position I found absurd. Lawyers! (eye roll) (Sorry, lawyer friends.)

I see a big gap between those who are all for, or up in arms about, the individual mandate (everyone cites political-philosophy-oriented arguments), and what the two judges are arguing here. Those I’ve heard arguing about political philosophy would be against both penalty and credit, or for both penalty and credit. I don’t know of a philosophical argument where the labeling makes the difference.

Relabeling something doesn’t change what rights an individual has, or what rights a state has, in practice. The distinction, if any, is legalistic.

I’m imagining the founding fathers: “Call it a credit rather than a penalty, or give me death!”

I’m blown away that a law with such huge practical effects — some will say positive, some will say negative, not the point — can be in limbo over this. Neither advocates nor opponents of reform would consider the wording of the bill “what’s at issue,” but it could be what the courts base a decision on. Congress, break out your thesaurus next time.

(As with the previous post, spare us the generic debate about health care reform in the comments, we’ve all heard it before. I haven’t heard much discussion of this very specific legal topic though, insights are welcome on that.)

I contribute to money.stackexchange.com sometimes; the site doesn’t get a ton of traffic yet, sadly. Some of the stuff I’ve written over there is as good (or bad) as my blog posts over here, so I thought I’d link to some highlights.

• Where to start with personal finance: my take was, don’t start with investing. Reading questions on the site, you quickly see that “getting serious about my finances” means “learn about investing” in many people’s minds. But I argue that’s all wrong.
• How much should I save for retirement: I like rules of thumb rather than expecting to get a precise answer from a complex calculation. I’m not a fan of “retirement calculators,” for example those that your mutual fund company may offer.
• A first post and a second post about the trouble with investing in precious metals.
• Here are some ideas on what you could do instead of investing in precious metals, and a guess at the lowest-risk investment you might make (“risk” = long term loss, not volatility).
• Bonds and why you’d want them in your portfolio – why not go with all-stocks? When do bonds lose money?
• When is debt a good thing? I pay off my mortgage as slowly as possible, here’s why.
• How should you benchmark an investment – time vs. dollar weighting, and a caveat that maybe a market benchmark isn’t the right comparison.
• Why term life insurance is better — there’s a big-picture obvious answer: you can’t afford a large enough benefit if you’re buying whole life.

It’s not a bad Q&A site on personal finance, if you browse around. Though it needs more people asking and answering.

Despite the title, Selfish Reasons to Have More Kids, the “why you should have more kids” part feels tacked-on. The interesting part of the book reviews twin and adoption studies, making the case that parenting style doesn’t matter much in the long run.

The book’s argument in brief is:

• in twin studies and adoption studies, most variation in how adults turn out can’t be explained by what their parents did. “Normal” variation in first-world parenting (i.e. excluding abuse, malnutrition, etc.) does not affect how people turn out in adulthood very much.
• parenting affects how kids act while they are kids, but once people move out of the house they bounce back to their “natural state.” (“As an adult, if I want a cookie, I have a cookie, okay?” – Seinfeld)
• one long-run thing parents can strongly affect is whether their kids have fond memories of them and enjoy having them around.
• parents should be more relaxed and invest less time in “kid improvement,” more time in everyone enjoying themselves even if it’s by watching TV.
• discipline is mostly for the benefit of the parents (“keep kids from being annoying around us”) not for the benefit of the kids (“build character”).
• (the conclusion in the title) since parenting can and should be less unpleasant, people should consider having more kids than they originally planned (i.e. the cost/benefit analysis is better than they were thinking).

Can you tell the author is an economist?

I get annoyed by “nature vs. nurture” popular science writing, and have a couple thoughts along those lines about this book.

Thank you for skipping the just-so stories

There’s a mandatory paragraph in nature vs. nurture articles where someone speculates on the just-so story. Something along the lines of “such-and-such behavior evolved so that women could get their mates to bond with them and hang around to care for children,” or whatever. Bryan Caplan 100% skips the silly evolutionary speculation. Thank you!

Cultural and genetic factors

However, I did find the book a little quick to jump to the genes. In “Appendix to Chapter 2,” Caplan explains that twin and adoption studies try to estimate three variables:

1. fraction of variance explained by heredity (similarity of twins raised apart)
2. fraction of variance explained by shared family environment (similarity of biologically unrelated children raised together)
3. fraction of variance explained by non-shared family environment (the rest of variance)

Caplan (to his credit) goes to some lengths to point out that the twin and adoptions studies are virtually all from first-world countries and typical homes in those countries. For example, one of the studies was done in Sweden, likely even more homogeneous than the ones done in the United States.

Obvious point, I’m sure Caplan would agree: when something correlates with genes, that doesn’t mean “there’s a gene for it.” For example, attractive people have higher incomes; attractiveness is mostly genetic. So one way income can be genetic has to do with your genes for appearance. Most people would find it misleading to say there’s a gene for high income, even though income correlates with various genes. This is a pretty simple example, but it can be much, much more complicated. Look at this diagram for how genes might get involved in autism, for example.

An outcome such as income often involves genes interacting with culture – say, standards of appearance. The ideal genes for appearance change over time and around the world.

But income isn’t just affected by appearance. Who-knows-how-many genes get involved: countless genes affect appearance, personality, intelligence, and then all of those factors in turn affect your income… in ways that depend on your culture and environment. Make it more complicated: there are also genes that affect how we react to certain appearances or personalities. If standards of attractiveness have a genetic component, then you’d expect that there are also genetic variations in what people find attractive, and then cultural variations layered on that.

Genes and culture also interact with what I think of as tradeoffs or “physical properties of the world.” All I mean here is that you can’t combine behaviors and traits arbitrarily, they tend to come in clusters that make sense together or work well together. This seems true to me for both personalities and for cultures. If you slice-and-dice your time or your concepts in one way, then you didn’t slice-and-dice them another way. Some ways of thinking or doing work better than others, some are more compatible with each other, etc. There’s a source of universals here that need not point to genes.

Finally, culture, like genes, gets passed on between generations. A very simple example: if you had a culture that was fundamentally anti-natalist, it would not last very long. And in fact most cultures are very enthusiastic about having children. This is a cultural universal (other than short-lived sub-groups), but just its universality doesn’t connect it to genes; it could be cultural rather than genetic evolution. Humans inherit so many ways of thinking and doing, and so much background knowledge, through interaction with other people, rather than through DNA.

Getting back around to the book. If you do a twin study in Sweden, then you might find that twins raised apart have similar outcomes. But it’s important to recognize that there aren’t (necessarily) genes for those outcomes; there are genes that cause the twins to have (likely unknown) traits which somehow result in those outcomes, in Sweden.

A couple thoughts:

• This probably doesn’t matter so much for the book’s practical conclusions. Parents can’t change the culture surrounding their children any more than they can change their kids’ genes.
• But from a wider perspective, it sure would be interesting — and perhaps useful at times — to know the mechanism for hereditary outcomes. i.e. the causality and not only the correlation.

When Caplan says “it’s genetic” I would say that’s true in that there appears to be a Rube Goldberg chain of causality, such that somehow certain genes are resulting in certain outcomes. However, my feeling is that the mechanism matters.

There’s a difference between a gene for unattractiveness/low-IQ/bad-personality (or sex, or race, for that matter) resulting in low income because the world (including culture) works against high income for people with those traits, and saying that “income is genetic.” Does it really feel accurate to say that “such-and-such percent of variation in income is explained by genetics” here, without mentioning the intermediate traits that are genetic, which in turn affect income?

I’m not sure Caplan would really disagree, but I do think the “nature vs. nurture” genre, including this book, glosses over a lot of complexity that kinda matters.

(The mistake is inherent in the phrase “nature vs. nurture”; if you start spelling out the mechanisms, it’s pretty clear that the two interact and the real question is how specifically in this case, right?)

Bottom line, when talking about the “fraction of variance explained by heredity” I’d add the footnote in this physical and cultural environment, because the background reality, cultural and otherwise, shared among families in the study — or even shared among all families on Earth — has a lot to do with which genetic traits matter and exactly how they matter.

More hours spent parenting

Caplan talks a bit about how people spend a lot more time parenting these days than they used to, and mostly blames increased parental guilt (people thinking they have to “improve” their children).

I’d wonder about other trends, such as the decline of extended family ties and the tendency for people to move across the country. If you’ve had a child, you may have noticed that they’re designed to be raised by more than two people. (The fact that some single parents raise them alone blows my mind.)

Missing from the book, I thought, was research into how other social/demographic trends were affecting parenting. More geographic mobility, more tendency toward social isolation, more tendency to need two incomes to achieve a “middle class” lifestyle, etc. – I have no idea which trends are most important, but surely some of these factors go into parenting decisions.

Caplan kind of implies that changes in parenting are almost all “by choice,” due to beliefs about the importance of parenting, and I’m not sure I buy it. It seems plausible to me that changes in parenting could be mostly by choice, but also plausible that they could be mostly due to socioeconomic trends.

Tangents

The book has several semi-related side trails, which may be interesting:

• some discussion of safety statistics, probably familiar from Free-Range Kids
• some discussion of the decades-old Julian Simon vs. Paul Ehrlich “will population outstrip resources” debate
• editorial in favor of reproductive technology
• thoughts for grandparents or future grandparents on how to end up with more grandchildren

Child services risk

On the Free-Range Kids tangent, I always wonder about “child services risk,” the risk that some nosy neighbor gets incensed and creates a whole traumatic drama with child services. To me this risk seems plausible.  In the book, Caplan says they let their 7-year-old stay home alone; which seems fine to me for the right 7-year-old in the right context, but I’m pretty sure a lot of state governments say it’s not fine.

I’d sort of like some “child services screwing up your child’s life” statistics to go along with the stats on abductions and drownings. Should we worry about that?

It’s one thing to say you don’t care what other people think, but when people can turn you in and just the fact of being reported creates a nightmare, I can understand why one wouldn’t want to appear unconventional.

Overall

I found the review of twin and adoption research interesting (and practical, for parents). Can’t hurt to remember that your kids are mostly going to do whatever they want anyway, once they move out of the house, and that chances are they’ll do a bunch of the same stuff their parents did. Caplan follows through nicely with implications for discipline, family harmony, and so on.

Some of the rest of the book wasn’t as interesting to me; I’ve heard the Free-Range Kids and Simon vs. Ehrlich stuff for example many times before. But you can skim these bits if you like.

Buy on Amazon – I get money if you do!

My Dad died from cancer one year ago, today I’d like to write something about him.

Dad liked animals, (certain) children, the woods, hunting, fine guns and knives, books, history, sharing his knowledge, strong coffee, and arguing about politics.

He grew up partly at my grandparents’ summer camp in Michigan called Chippewa Ranch, and partly on their cattle ranch in Georgia.

With brother Kenny

Keith On Poco

Dad’s favorite book, The Old Man and the Boy, is almost a blueprint for how he wanted to live and what kind of father he wanted to be. Robert Ruark’s epigraph in that book says “Anyone who reads this book is bound to realize that I had a real fine time as a kid,” and we spent our childhood weekends doing all sorts of things other kids weren’t allowed to do. In my copy of the book when I was 15, Dad wrote “This is all you really need to know, all you have to do is ‘do'”.

Havoc hunting

With my sister

I inherited a lot more of Dad’s reading-a-book-constantly side than his outdoor adventure side, but a little of both rubbed off.

Dad signed up for Vietnam, and while he never talked about it much, I’m guessing in some ways it was the last time he mostly enjoyed his day job.

His closest friend summarized his military career:

Diverted in 1968 from assignment to 5th SFGA to the Americal Division he was a LRRP Platoon Commander for his first tour. He extended in country to serve with the II Corps Mike Force in their separate 4th Battalion in Kontum. He chose to command a Rhade company in preference to available staff positions and was wounded severely during the Joint Mike Force Operation at Dak Seang in 1970 sufficient to require medevac to Japan with a severe leg wound from taking a grenade at about 4 feet while leading an assault on an NVA position. He was awarded three Silver Stars if the third one ever caught up with him–he certainly never searched it out.

with a dog in Vietnam

explaining something in Vietnam

Dad wasn’t one to define himself by military glory days, though. I think the adventure in Vietnam was just one more adventure, preceded by others, and he continued throughout his life.

One of the themes running through Dad’s life was his dislike for convention, and people who were too conventional in his eyes. He wasn’t afraid to name his son Havoc, for example. He loved revolutionaries and adventurers of all stripes, right-wing or left-wing. His military dogtags list his religion as “animist.” As we were growing up, he had nothing to say about religion one way or the other; he felt we ought to figure it out for ourselves. That was another of his parenting philosophies, he wasn’t going to tell us what to think. The fastest way to earn Dad’s contempt was to have an opinion just because other people had it, or to have an ignorant opinion because you hadn’t read enough books.

Another quick way to earn contempt was to be unprepared or incompetent. We had to have enough equipment at all times; I still have a basement full of equipment and a house full of books. Some old friends may remember laughing about my pile of assorted axes and hatchets. Dad could never remember for sure whether I had enough, including the several necessary varieties, so he’d send another one along every so often.

Whenever we got into some activity, whether cycling or leatherworking or hunting or backpacking, we’d end up with several times more equipment for that activity than we could ever use, as Dad tried everything out to be sure we had what worked best. We’d also have a complete library of books on the topic. And we got into a lot of activities.

Dad loved anything he thought was neat, which included most animals. We had a lot of crazy pets, from a squirrel to a 500-pound wild hog. As I’m looking through old photos, he’s always hanging out with a dog.

With a hunting hound

With another hound

It turned out that he more or less killed himself with cigarettes. He’d always rationalized bad habits saying he didn’t want to get old and dependent anyway, but in the end I think he’d rather have lived to see his grandchildren grow up. He died at home with family and friends, and was only confined to bed for his last day or two.

When he died my own son was six months old, and I stood outside the house where I grew up and hugged my son for all I was worth. Whenever I start to think about my son knowing my Dad, learning some of the things I learned as a kid, that’s what brings on the tears. I wish we’d had some adventures with the three of us.

I know my son and I will have some adventures anyhow, and I’ll think about Dad every time, and tell my son what advice Grandpa would have had, as best I can remember it.

At Horace Kephart's grave in Bryson City, September 2009

————–

I happened to stumble on this Yale Law Journal Online article yesterday. I hadn’t realized what the court cases about the individual mandate were (at least in part) arguing.

Credits and penalties

The individual mandate is structured as a tax penalty (i.e. an extra tax hike) if you don’t have health insurance. Those of us who have lived in Massachusetts know what it might look like, since it’s modeled on “Romneycare.”

The thing about a tax penalty, as far as I can see, is that it’s economically identical to a tax credit plus a tax hike. That is, say I’m the government and I want people who do not buy child care to pay $500 more tax than people who do buy child care. I can either increase taxes across the board $500 and then offer a $500 credit if you buy childcare; or I can increase taxes by a $500 penalty if you don’t buy childcare. In either case, if you don’t buy childcare, you pay $500 more than before, and if you do buy it, you pay the same as before.

Certainly the credit and the penalty are a different “spin” – I’m sure people have a different reaction if the tax forms say “you must do this or pay a penalty” vs. “you can do this to get a credit.” There’s a psychological difference. But if people were completely rational and didn’t look at the wording, the fact is that it doesn’t matter to their pocketbook what the tax forms call the rule. The rule is simply “you pay less if you buy X and more if you don’t.”

The tax code is already full of credits for buying stuff. Well-known ones include child care and the temporary first-time homebuyer credit. And… you can even deduct health insurance costs already.

The new law’s penalty for not having insurance is identical to raising taxes by the penalty, and then allowing anyone who has insurance an additional credit, on top of the existing deduction, equal to the penalty.

The penalty means you pay less on taxes, partially offsetting the cost of insurance, if you buy insurance. That’s all it means.

In the same sense, the tax code already requires you to buy insurance (if self-employed anyway). The new penalty increases the incentive somewhat, but there’s a tax incentive to buy insurance today.

(I realize there are complicating elements to how this works — phase-outs, credits vs. deductions, refundable vs. nonrefundable credits, etc. — but I don’t think they matter for this discussion.)

Some implications

• There’s a claim in these cases that the government has never required people to actively buy a certain product. However, at least the enforcement of this requirement, i.e. a tax savings if you do buy, is precisely equivalent to all the credits and deductions you can already get for buying various things. Go into TurboTax and look at the credits and deductions available. You are “required” to buy all of that in exactly the same sense that you are required to buy health insurance under the new law — at least as far as enforcement goes. The punishment is the same, you pay higher taxes if you don’t buy.
• In fact there’s a popular argument “can the government make you buy GM cars?” — and yes, there have been tax credits for buying certain kinds of car (hybrid, electric, whatever). Which means you pay more (you are penalized) if you don’t buy those cars. The government can, under current law, punish you through taxation if you don’t buy the right car.
• Because there’s not an economic difference between the penalty and a hike+credit, and people don’t want to argue all the existing credits are unconstitutional, the legal argument in these court cases seems to be that it’s unconstitutional because Congress called it a penalty and not a tax. “No calling it a penalty when it’s a tax!” is some kind of grammar-nerd point, not something that should inspire throwing crates of tea into the harbor…
• The Yale Law Journal Online article makes two points on this, first that the statute does call it a tax in many places, and second that Supreme Court precedent since the 1860s is that it doesn’t matter whether it’s called a tax or not, when judging constitutionality.

Laurence Tribe’s argument

Laurence Tribe predicted an 8-1 vote to uphold the individual mandate. He says:

There is every reason to believe that a strong, nonpartisan majority of justices will do their constitutional duty, set aside how they might have voted had they been members of Congress and treat this constitutional challenge for what it is — a political objection in legal garb.

He feels that the law is so clearly constitutional according to precedent, on both interstate commerce and tax power grounds, that the Court will have no coherent way to strike it down.

I see his point, based on the power to tax. Interstate commerce may be a fuzzier issue, I don’t know. But the government only has to have the power on one ground. If it’s constitutional using the power to tax, it’s constitutional.

Discussion

There’s no need to post comments about whether the government should have a taxation power in the Constitution, or whether the health care law is a good idea, or any generic debate like that.

In this post I wanted to raise the issue of whether a tax incentive to buy insurance is constitutional following existing precedent, and whether it can be legally distinguished from other tax incentives (whether framed as credit or penalty) that involve buying particular goods and services. I don’t see where the distinction can be made. Anyone have any good theories?

I am not a lawyer, if you are one, please add your thoughts!

Update July 2011

The Sixth Circuit discussed this topic in their ruling, here’s a new post on it.

Update June 2012

The deciding vote from John Roberts was based on this same tax power argument.

As you may have noticed, this blog is a big old grab-bag of random topics. In that spirit, here are some products I enjoyed lately, that people may not have heard of. In several cases these products are from small companies and I think they deserve a mention for their good work.

I’m going to affiliate-link the stuff, because why not, so be aware of that if it bothers you.

GeekDesk

The computer geeks reading my blog have probably seen this, but for the rest of you, I’d recommend it if you do any kind of desk work.

GeekDesk lets you use a standing desk part of the day without committing to standing up always. It has a little motor so you can quickly raise or lower the desktop.

(Why a standing desk? Sitting down all day is really bad for you, even if you exercise daily.)

I’ve found that I almost always stand, now that I’m used to it. But it’s nice to have the option to sit.

I have the GeekDesk Mini which is still large, about 3 laptops wide by 2 deep.

When standing, a laptop screen is too low for me and requires hunching over, so I had to get a monitor with a stand, and then I had to pile some books under the stand. With the monitor, I can stand up straight and look at the screen directly.

You can also buy only the motorized legs and put your own top on the GeekDesk if you have a nice piece of wood in mind.

The desk has decent cable management, but I also screwed a power strip to the bottom of the desktop so only one power cord goes to the floor.

Quakehold

Quakehold is a removable museum putty that makes things stay put.

As the name suggests, one use of it is to keep stuff on shelves during an earthquake, and I’m guessing those of you who live in an earthquake zone already know about it. I didn’t know about it.

It’s useful in a duct-tape kind of way. Some examples in our house:

• making our lamps harder for kids to knock over
• keeping a power strip on the bottom of my GeekDesk from slipping off its mounting screws
• sticking down a diaper changing apparatus to keep it from sliding around
• keeping our toddler from sliding an anti-toddler fence across the room

In most cases you could also use duct tape, I suppose, but the putty is easier to remove without damaging surfaces, and avoids looking too There, I Fixed It.

SimpliSafe

SimpliSafe is an alarm system we installed in our house a few weeks ago, and it’s a Boston startup, for those of you in Boston wanting to support local companies.

I’m very impressed with the product, but boy was it hard to discover. I just Googled “alarm system” for example, and they aren’t in the ads and aren’t in the first 8 pages of organic results. (If you’re an SEO consultant you might want to get in touch.)

SimpliSafe uses completely wireless (battery powered) sensors that stick to your wall with 3M Command. When you get the system it’s preloaded with the codes for your sensors, so there’s no pairing process. All you do is pull the plastic tab blocking the battery from each sensor, stick it to the wall or put it on a shelf, and then plug a USB stick they provide into your computer. On the computer there’s a simple setup process to tell the monitoring service who to call and so on. After setting up, you put the USB stick in the base station to transfer the settings, and that’s it.

It takes about half an hour to install and set up. Maybe an hour if you’re the kind to read the (clear and excellent) instructions.

Here’s the comparison:

• ADT: you have to talk to a salesperson on commission. They are selling a 3-year contract that auto-renews if you don’t cancel in time, and it costs almost $50/month if you get cellular monitoring. The up-front equipment can be expensive (they have free or cheap packages, but those don’t include what you probably need).
• SimpliSafe: you order online and self-install in half an hour. There’s no contract, and cellular monitoring is $15/month. No need for a land line. The up-front equipment is reasonably-priced.

As a middle ground, I guess there’s a whole community out there of people who roll their own alarm and home automation systems, and it looks possible to get a lot cheaper than ADT that way as well. However, it looked way too time consuming for me. I think you can also switch your ADT equipment over to a cheaper monitoring service, at least after your 3 year contract expires if you catch it prior to autorenew.

SimpliSafe‘s industrial and interaction design are great. The web UI is simple, and everything is pre-configured as much as possible. (For example, the base unit already knows about your sensors when you get it.) They really thought through the whole experience.

The product is marketed for apartments (because there’s no permanent installation), but it seems to be fine for our house. If you live in a large enough place or have metal walls, it may not work for you.

Other possible downsides:

• I’m guessing the system uses ZigBee or something similar, but they don’t say, and they don’t claim to support any sensors they don’t sell. Basically it isn’t an open system, you have to buy components from them.
• They only have basic sensors right now, for example no fire alarm or glass break detector yet, though they say they will in the future.
• There’s no “home automation” stuff, it’s purely a burglar alarm.

By the way, in researching this two other interesting companies I saw using low-power wireless were VueZone and AlertMe (UK only). I have not tried either one, but they seem to be similar in spirit to SimpliSafe (i.e. low-power wireless technology with decent industrial design).

SLS-Free Toothpaste

This product won’t be relevant to everyone, but if it is and you don’t know about it, you might thank me.

At the risk of too much information, I used to get canker sores, anytime I bit my lip or flew on an airplane or just had bad luck. These would make me miserable and grumpy for days, every other week or so. (Yeah, more grumpy than usual, haha.) While not life-threatening, it was unpleasant.

Now I use some stuff called Biotene which has been essentially a miracle cure. Instead of being in pain on a regular basis, I never have a problem. It isn’t some placebo effect “maybe it’s a bit better” kind of thing, it’s a change from “have constant chronic problem for years” to “never have the problem at all.” If I travel or something and don’t use the miracle toothpaste, I can get a canker sore again, but on resuming the toothpaste it will clear up.

Biotene claims to have magic enzymes. I don’t know if the enzymes do anything, or if it’s primarily the SLS-freeness that works. You may have luck with other toothpastes as well. Anyway, I pay for my overpriced toothpaste and it is worth every penny. Most drugstores, Target, etc. carry it.

GoGo Babyz Travelmate car seat wheels

The biggest problem here is the name, GoGo Babyz Travelmate. Bad name.

If you take your baby or toddler on a plane, this eliminates the need for a stroller, giving you one less thing to check and one more free hand. If you’ve taken a baby or toddler on a plane, you understand the value of that.

The gadget adds roller-bag wheels and handle to your car seat, so you can push or pull your kid like a roller bag. Comical but it works.

We gate-checked the GoGo Babyz, but I think you could get it in the overhead bin especially if you pop the wheels off (which is pretty easy).