Havoc's Blog

Twitter (and mirroring it to
Facebook) seem to be
replacing this blog. I like what Valerie Aurora writes on her home page:

For those of you born after 1980, a “homepage” is an ancient form of
social presence on the web which has been superceded by more
structured social networks like MySpace and Facebook. One of the main
problems with a homepage was that it is easy to forget to update
it. My current homepage is mostly an archive of the past and a
collection of pointers to my current active Internet presence. Hope
you enjoyed your history lesson!

Awesome.

Because impending parenthood and working for a startup aren’t
enough stress for one year, we’ve decided to relocate. (This has been
the plan for several years, but stuff kept coming up. It seemed a bit
now-or-never at this point.)

If you’ve seen the movie Away We Go, we did a
trip like that in 2007 and again this year. The movie is about two
people exactly our age, traveling around deciding where to live while
expecting a kid. (Aside: the main characters in the movie made sense
to me, but critics described them as unbelievable personalities. It
seems movie critics don’t know a lot of extreme introvert
couples. Maybe we should get out more.)

Last year I got a lot out of a book called Who’s
Your City, which is a kind of flip side to The
World is Flat. Who’s Your City sounds like a “decide where to
live” book but in fact it’s about economic geography; Richard Florida
argues that cities are essential to knowledge work, and that economic
growth in recent decades has been due to knowledge work in regional
centers (finance in New York, tech in Silicon Valley, sales in
Chicago, etc.) … the geographic clustering allows people to network,
jump jobs, launch startups, exchange ideas, and in general get things
moving. This book claims the world is becoming less flat,
with a widening economic and cultural gap between urban knowledge
workers and everyone else. You can understand a lot about people’s gut
reactions for or against Barack Obama from this one.

Relevance to our move: economically, it’s near-insane for software
developers in the US to live outside Silicon Valley, with a very short
list of second-tier options (Boston, Boulder, RDU, Austin, maybe a
couple others). And I love startup-type work environments, groups of
smart people, and all that tech industry goodness. Amy’s nursing
specialty creates a similar situation; it’s only found in large
hospitals.

Aside from work, though, these places don’t match us very well. (Work
is not the only factor.)

It turns out that litl‘s parent company
has a (small) office in Asheville, NC, a city on our short list for
many years – it’s near family, pretty, not too large, not too boring,
affordable, lots of outdoor stuff to do. We enjoyed North Carolina
when we lived in Chapel Hill.

The downside of Asheville is that it’s not a tech industry kind of
place. That’s the risky (insane?) bit of the move, but I’m hopeful it
will work out. Oddly enough, in practice it’s almost more convenient
to Boston than where we live now … instead of being two hours from
litl’s Boston office, I can be five or fifteen minutes from the office
in Asheville, complete with a fancy video link to the team in Boston
and London. Going into the Boston office two days in a row was already
8 total hours of commuting, flying to Boston for a two-day trip isn’t
notably worse.

This will continue our North-South ping-pong, for me Georgia, Chicago,
Chapel Hill, Boston, Asheville. Amy was there too except for the
Georgia part.

Long story not short, we’ll be in Asheville in a few weeks, but I’ll
still be in Boston often for work. And we’ll see how it goes.

(This post was originally found at http://log.ometer.com/2009-07.html#24)

Haven’t had time to blog in months …

• Have been working on litl during nearly
  all available hours… it took some time (doesn’t it always?) but we
  have a great team assembled and it’s a ton of fun to focus on writing
  the code rather than hiring, and see the rapid progress a complete
  team is able to make. (If you want to pile on, we aren’t urgently
  hiring, but we probably will be at some point especially for the right
  person, so we’d love to see your resume.)
• There is nothing better than getting high-quality work done with a
  high-quality group of people.
• Clutter is rapidly
  approaching 1.0 … spent some time this weekend patching
  the
  remaining issue that was driving me crazy, hoping to
  get it in under the wire. Emmanuele may be owed beers.
• It’s gratifying that the new default home screen of Facebook looks a lot
  like Mugshot.org,
  a site some of us came up with at Red Hat.
  We coded Mugshot’s personal-lifestream-thingy before Facebook’s news feed came
  out and before FriendFeed came out. Not saying either site copied
  us, but it’s still nice to know at least our idea was good (even
  though there were lots of reasons we weren’t the ones to get anywhere
  with it).
• It used to be that the cool kids claimed to have listened to
  R.E.M. before Document
  (these are cool kids 20 years ago, I guess). I was listening to
  mutual fund manager John Hussman before
  he showed up on fivethirtyeight.com
  and in the Wall
  Street Journal. Am I a cool kid? Ha.
• New
  York Times tries to explain why people get distracted by small(er)
  things like millions in AIG bonuses, while missing large things, like
  billions to bail out AIG’s
  bondholders. Possible Occam’s razor answer: most people don’t know
  what a bond is, but they know what a bonus is.
• Some spammer selling Hewlett-Packard products took over my Twitter
  account “hp”, and now Twitter seems to have deleted it – and allowed
  someone else to grab the account. I emailed Twitter a while back
  asking them to fix it but all I got was an autoresponse saying “we’ve
  taken two weeks and still haven’t gotten back to you, let us know if
  you still need help.” I rarely twittered anything anyway, but
  … lame.
• Funniest thing today was a random snark on Ayn
  Rand that Miguel
  reposted.
• Why is it reasonable to pretend that socialist or libertarian
  principles can guide a decision between 35% and 39.6% tax rates?
  Sure, 5% vs. 80% is an ideological debate. But 35% vs. 39.6% is a
  pragmatic debate. The vast majority of American voters agree on the
  main government spending items – social security, medicare, defense –
  that eat up most of our tax revenues. Controversial spending items
  are too small to affect the tax rate much. And the vast majority of
  Americans are opposed to 50-60% tax rates. We’re going to stay right
  about where we are because big tax hikes and big spending cuts are
  both third rails. Here’s
  some historical perspective; tax rates are much more
  stable over the last few decades than they have been over the last century.
• Amy discovered Cake
  Wrecks (scroll past Sunday, they have non-wrecks on
  Sunday). Awesome.

(This post was originally found at http://log.ometer.com/2009-03.html#22)

For GNOME
summit attendees we’re buying some coffee, dessert, and snacks on
Saturday afternoon from a local French
restaurant. litl’s founder John Chuang plans to drop by as well to
meet people and learn.

This isn’t a talk or a big announcement (litl will still be in stealth
mode post-summit) – the idea is to say thanks to the developers behind
GNOME and introduce ourselves.

The litl devel team will be around all weekend, doing some hacking. As
you may have
guessed we have quite a bit of code written in JavaScript, with a
C/C++ core. We’ve invested a few weeks splitting the
JavaScript/gobject-introspection bindings out of our codebase and
rebasing to newer gobject-introspection. By Saturday we hope to have
it compiling and posted, so we can hack on combining our work with Alex’s
gscript API among other things. We might also sign up for a
technical talk about the bindings if people are interested in learning
more.

(This post was originally found at http://log.ometer.com/2008-10.html#9)

In an effort to outdo other software developers in nerdiness, I read a
blog called Accrued
Interest, subtitle “Come for the analysis and research on the
U.S. Bond market. Stay for the geeky Star Wars references.”

Accrued Interest has a nice
explanation of how the credit crisis affects everyone.

Here’s
another
good one from the New York Times.

(This post was originally found at http://log.ometer.com/2008-09.html#30)

I’ve been wondering for a while why nobody contrasts Obama’s health
care proposal with some of the past Democratic proposals. It’s a
market-based, incremental proposal, that makes use of existing
programs and existing insurance companies.

Today Obama’s campaign has a new ad making this point, so I guess I’m on their
wavelength. After my What happened to John
McCain? post, the next day Obama ripped that
off, too, for a negative ad.

Finally fixing health care would make a huge difference for quite a
few people I know. It may be the single most consequential thing the
government could do.

If you watched the debates, see factcheck.org‘s
coverage. They ding both candidates on multiple claims.

(This post was originally found at http://log.ometer.com/2008-09.html#29)

Worst
stock plunge in 20 years.

Here’s a relevant
post from last year, when the stock plunge was much less dramatic.

Lessons for the next bull market (yes there will be one eventually):

• Low interest rates and high liquidity do not mean stock prices will
  stay up. Those things are indicators of insufficient risk premiums,
  not indicators of permanent demand for assets.
• If companies have 2x their long-run average profit margins, something
  unsustainable is going on. Abnormal profits don’t last forever in
  (more or less) free market systems, due to competition and the
  business cycle. Anytime you read an article that mentions a P/E
  ratio, without discussing normalized margins, run away in fear. (The
  inverse goes as we most likely head into recession. P/E
  ratios will become misleadingly high, while in the bull market they
  were misleadingly low.)

(This post was originally found at http://log.ometer.com/2008-09.html#29.2)

Nice, factual, 1-chart explanation of the competing tax proposals. (Thanks Luis.)

Barack Obama’s policy proposals.

(This post was originally found at http://log.ometer.com/2008-09.html#14)

McCain was always the respectable Republican, who opposed
tax cuts without spending cuts, did not deny environmental
threats, had a reasonable approach to immigration, and was against the
scorched-earth Rove-style politics. Somehow he has reversed himself on
all of these things while running for president. It’s not
even clear he still opposes torture (see this
vote,
for example).

I don’t consider flip-flopping in itself to be bad; the famous
Keynes
quote is, “When the facts change, I change my mind. What do you
do, sir?”

But the facts haven’t changed, McCain has. Read this
interview with Time magazine – cringe-inducing. I just feel bad
for him.

McCain 2008 is so unlike McCain 2000, it’s very hard to understand
what’s going on. I’m not willing to write this one off as “politics”
because it feels unusual and strange.

On top of McCain’s personality-ectomy, since just before the
convention, his campaign has kicked into a series of
frivolous distractions. The New York Times has an
article and an
op-ed about it, but if you’re a New York Times hater, factcheck.org is excellent and calls
both candidates on their inaccuracies. factcheck.org doesn’t try to
make a subjective judgment about which candidate distorts the most,
but read the last couple months of fact checks on each of them and see
what you think.

On at least two substantive policy matters, taxes and health care, the
candidates are failing to have a real discussion – McCain has been
repeating misleading talking points instead of engaging with and
debating the actual proposals.

And Sarah Palin – in her first interview, she says this with a
straight face:

What insight into Russian actions, particularly in the last couple of weeks, does the proximity of the state give you?

They’re our next-door neighbors. And you can actually see Russia
from land here in Alaska. From an island in Alaska.

Come on. She’s applying for a serious job.

While a hollywood
actor mocking creationism is not going to win over any swing
voters, I have to agree with the “Disney movie” part of the sentiment.

To combat this, before announcing Palin’s nomination, the McCain
campaign seems to have invented a narrative that “they” are attacking
her family, attacking her because she’s a woman, blah blah. Even
though Obama did
the opposite and condemned attacking her family, McCain stuck to
the talking points. And though it took his campaign a couple weeks,
they found this “lipstick on a pig” comment they could use as
“evidence” for the talking points they’d already been pushing. Obama responded
very well – emphasizing how ridiculous it is to play
games given the weight of the substantive issues at stake.

If your attention span is too short to watch Obama’s serious responses, Jon
Stewart takes apart the pundits on Palin in a more humorous way.

It makes me sad. When McCain won the nomination, though I was not
going to vote for him, I was happy that we would have two people
running with competence and integrity well above the politician
average, and well above most of the others in the primaries. It felt
somewhat “can’t lose” – the upgrade from Bush would be significant
either way. McCain 2008 is still better than Bush (it’s not a high
bar). But “new McCain” has been very disappointing compared to McCain
2000.

Mike
Evans reposted a nice case for
Obama from Marc Andreessen. Unfortunately, where the majority of
young people and “knowledge workers” see competence, a lot of swing
voters seem to see something negative.

There are serious problems facing the United States, especially the US
economy, and the economic problems ripple through national security
and other issues. Our economic problems have some straightforward
answers: simplify the tax code, make it a bit more progressive so we
have a solid base of consumers to buy stuff, reduce lost productivity
and inflation caused by a broken approach to health care, and – the
big one – address the redistribution of wealth and power from the US
to other countries created by our oil dependency.

One of the presidential candidates wants to talk about credible
solutions to these problems, and one of them wants to talk about
distractions.

(This post was originally found at http://log.ometer.com/2008-09.html#13)

“If synchronous IO becomes a problem, it can be made asynchronous later.” Tempting to imagine that some operations on local files are “fast enough” to implement with synchronous IO. “Premature optimization is the root of all evil,” right?

Wishful thinking. Async vs. sync IO is not a performance issue (in fact synchronous IO can be faster). It is a structural or qualitative issue, a design issue. Sync IO is guilty until proven innocent.

My new rule is: any code that will be blocking during a user-interactive main loop must run in a known-short time. The task must be known short, not indefinite-but-often-short.

Local file access could be short, but is not known short. Another common culprit: synchronous D-Bus calls.

I’ve seen so many rewrites from synchronous to asynchronous IO over the years – it is almost 100% likely that anything blocking the main loop eventually comes to be seen as a bug. There’s near-zero chance it’s really OK to use those nice, easy-to-program blocking D-Bus calls, or that nice, simple g_file_get_contents(). Even the harmless looking g_file_test() can bite.

These APIs exist only to tempt you. They are the dark path.

(Note: by “asynchronous” I mean “the main loop is not blocking, for example because IO is in its own thread” – I don’t mean special AIO system calls.)

Here’s the core issue: the UI’s main loop needs to wake up at 30-60 frames per second to do animations and repaints, and it should respond to user input with similar speed. It needs to do this consistently (think real-time-like), not “on average” – if there are big outliers, like an occasional quarter-second delay in frame rate, the app will feel sluggish.

“Local” IO can be slower than you think; the firefox fsync problem shows one extreme case (the whole kernel gets bogged down, not just the IO operation), but think about network file shares, or large file sizes, or systems under load. Even mild manifestations of these issues are visible in decreased user responsiveness and animation smoothness.

It’s very common for supposedly fast IO operations to end up batched together, creating a long delay at once; for example, calling stat() on everything in a directory, or queuing a bunch of idle handlers that all happen to kick in at once.

Another Firefox example: try uploading a bunch of photos to Flickr in one go. Ouch. The files are local, but … there are a lot of them and they are big. Pretty sure Firefox uses synchronous IO here.

Enough “this local IO should be fast enough” laziness scattered around an app creates distributed bloat that’s hard to pin down and doesn’t show up very well in profiles because it depends on external circumstances.

Not only is it hard to find later, synchronous code is hard to fix – it means rewriting.

Bottom line: don’t write this code in the first place. Using async APIs is not premature optimization, it’s correct design.

(This post was originally found at http://log.ometer.com/2008-09.html#7)

Chris mentions
JavaScript embedding using SpiderMonkey.

When GNOME first started out, there was much talk of using Guile as an
embedded language. The concept was similar to the architecture of
Emacs, Firefox, or all the
games using Lua (such as World of Warcraft).

I would define an “embedded” language as one that doesn’t come
with a platform. It’s small and easy to glue together with an app
written in C or C++. It uses the same platform as the app. If the
platform has an introspection capability like XPCOM or
GObject-Introspection, then the platform can be exported to the
embedded language in an automated way.

At some point, and I was one of the people to blame for this, we
changed the focus of the Linux language discussion to grand unified
schemes involving component technologies and virtual machines. The
result is that too much desktop stuff is still written in C.

A “C/C++ core plus embeddable language” architecture for an
application has many pragmatic advantages.

• Higher layers of code such as UI code are so much easier in a
  dynamic “scripting” language.
• Proven to be practical and scalable for huge apps such as Firefox and
  World of Warcraft, even with dead-simple languages such as JavaScript
  and Lua.
• No platform silos. In a GNOME app, the embeddable language uses
  GIO, GTK+, GStreamer, Clutter, or whatever you are using – it does not
  have its own http stack and everything else.
• You can choose to write each part of the app in the easiest language;
  some stuff is easier in C/C++.
• Sandboxable / well-modularized. The interpreter is generally an object
  that can be destroyed; no heavy language runtime or VM takes over the
  entire app.
• Embeddable languages don’t seem to create a huge jump in memory
  footprint, for whatever reason, so they are still sensible for
  “swarm of many processes” environments like GNOME and KDE that
  multiply any per-process overhead by a significant factor.

(There are advantages to the heavier full-platform languages also, but
I’m guessing everyone knows them already so I won’t make a list.)

JavaScript is a great choice for an embedded language. I did not
realize at first that SpiderMonkey has no dependencies; it is not tied
up with the rest of the Mozilla platform (XPCOM, etc.), it’s completely
standalone. And most programmers understand at least some JavaScript
already.

There are some problems right now, for example there’s no standard
packaging and .pc file for jsapi.h/libmozjs.so, so it’s hard to make
an app build on multiple distributions unless you suck in an internal
copy of the library.

It’s necessary to add a couple things to JavaScript before it’s
truly usable; most notably, some kind of module system. I’m not sure
SpiderMonkey itself should have that though – it starts to
define a platform and make the language “not embedded”. Maybe modules
are introduced by higher layers (in fact xulrunner adds
a
module system if you use the whole xulrunner platform).

I don’t mean this as a SpiderMonkey advocacy post, though we’re using
it successfully at work. Other options are the WebKit JavaScript
engine, Guile, Lua, Tcl, etc.

My main point: the “C/C++ core plus embedded scripting” architecture
is a very smart way to write an app, and doesn’t have some of the real
downsides of heavier platforms.

(This post was originally found at http://log.ometer.com/2008-08.html#25)