Havoc's Blog

Most of my career has been in commercial companies related to open source. I learned to code just out of college at a financial company using Linux. I was at Red Hat from just before the IPO when we were selling T-shirts as a business model, until just before the company joined the S&P500. I worked at litl making a Linux-based device, and now I’m doing odd jobs at Typesafe.

I’ve seen a lot of open source project evolution and I’ve seen a lot of open-source-related commercial companies come and go.

Here are some observations that seem relevant to the Scala world.

Open source vs. open project

In some cases, one company “is” the project; all the important contributors are from the company, and they don’t let outsiders contribute effectively. There are lots of ways to block outsiders: closed infrastructure such as bug trackers, key decisions made in private meetings, taking forever to accept patches, lagged source releases, whatever. This is “one-way-ware,” it’s open source but not an open project.

In other cases, a project is bigger than the company. As noted in this article, Red Hat has a mission statement “To be the catalyst in communities of partners and customers and contributors building better technology the open-source way” where the key word is “catalyst.” And when Linux Weekly News runs their periodic analysis of Linux kernel contributions, Red Hat is a large but not predominant contributor. This is despite a freaking army of kernel developers at Red Hat. Red Hat has many hundreds of developers, while most open source startups probably have a dozen or two.

In a really successful project, any one company will be doing only a fraction of the work, and this will remain true even for a billion-dollar company. As a project grows, an associated company will grow too; but other companies will appear, more hobbyists and customers will also contribute, etc.  The project will remain larger than any one company.

(In projects I’ve been a part of, this has gone in “waves”; sometimes a company will hire a bunch of the contributors and become more dominant for a time, but this rarely lasts, because new contributors are always appearing.)

Project direction and priorities

Commercial companies will tend to do a somewhat random grab-bag of idiosyncratic paying-customer-driven tasks, plus maybe some strategic projects here and there. The nature of open projects is that most work is pretty grab-bag; because it’s a bunch of people scratching their own itches, or hiring others to scratch a certain itch.

In the Scala community for example, some work is coming from the researchers at EPFL, and (as I understand it) their itch is to write a paper or thesis.  Given dictatorial powers over Scala, one could say “we don’t want any of that work” but one could never say “EPFL people will work on fixing bugs” because they have to do something suitable for publication. Similarly, if you’re building an app on Scala, maybe you are willing to work on a patch to fix some scalability issue you are encountering, but you’re unlikely to stop and work on bugs you aren’t experiencing, or on a new language feature.

An open project and its community are the sum of individual people doing what they care about. It’s flat-out wrong to think that any healthy open project is a pool of developers who can be assigned priorities that “make sense” globally. There’s no product manager. The community priorities are simply the union of all community-member priorities.

It’s true that contributors can band together, sometimes forming a company, and help push things in a certain direction. But it’s more like these bands of contributors are rowing harder on one side of the boat; they aren’t keeping the other side of the boat from rowing, or forcing people on the other side of the boat to change sides.

Commercial diversity

My experience is that most “heavy lifting” and perhaps the bulk of the work overall in big open projects tends to come  from commercial interests; partly people using the technology who send in patches, partly companies that do support or consulting around the technology, and partly companies that have some strategic need for the technology (for example Intel needs Linux to run on its hardware).

There’s generally a fair bit of research activity, student activity, and hobbyist activity as well, but commercial activity is a big part of what gets done.

However, the commercial activity tends to be from a variety of commercial entities, not from just one. There are several major “Linux companies,” then all the companies that use Linux in some way (from IBM to Google to Wall Street), not to mention all the small consulting shops. This isn’t unique to Linux. I’ve also been heavily involved in the GNOME Project, where the commercial landscape has changed a lot over the years, but it’s always been a multi-company landscape.

The Scala community will be diverse as long as it’s growing

With the above in mind, here’s a personal observation, as a recent member of the Scala community: some people have the wrong idea about how the community is likely to play out.

I’ve seen a number of comments that pretty much assume that anything that happens in the Scala world is going to come from Typesafe, or that Typesafe can set community priorities, etc.

From what I can tell, this is currently untrue; there are a lot more contributors in the ecosystem, both individuals and companies. And in my opinion, it’s likely to remain untrue. If the technology is successful, there will be a never-ending stream of new contributors, including researchers, hobbyists, companies building apps on the technology, and companies offering support and consulting. Empirically, this is what happens in successful open projects.

I’ve seen other comments that assume the research aspect of the Scala community will always drive the project, swamping us in perpetual innovation. From what I can tell, this is also currently untrue, and likely to remain untrue.

Some open communities do get taken over by narrow interests. This can kill a community, or it can happen to a dead community because only one narrow interest cares anymore. But the current Scala ecosystem trend is that it’s growing: more contributors, more different priorities, more stuff people are working on.

How to handle it

Embrace growth, embrace more contributors, embrace diversity.

The downside is that more contributors means more priorities and thus more conflicts.

When priorities conflict, the community will have to work it out. My advice is to get people together in-person and tackle conflicts in good faith, but head-on. Find a solution. In-person meetings are critical. If you have a strong opinion about Scala ecosystem priorities, you must make a point of attending conferences or otherwise building personal relationships with other contributors.

Never negotiate truly hard issues via email.

As the community grows and new contributors appear, there will be growing pains figuring out how to work together. All projects that get big have to sort out these issues. There will be drama; it’s best taken as evidence that people are passionate.

Structural solutions will appear. For example, in the Linux world, the “enterprise Linux” branches are a structural solution allowing the community to roll forward while offering customers a usable, stable target. Red Hat’s Fedora vs. Enterprise Linux split is a structural solution to separate its open project from its customer-driven product. In GNOME, the time-based release was a structural solution that addressed endless fights about when to release. Most large projects end up explicitly spelling out some kind of governance model, and there are many different models out there.

Whatever the details, the role of Typesafe — and every other contributor, commercial or not — will be to discuss and work on their priorities. And the overall community priorities will include, but not be limited to, what any one contributor decides to do. That’s the whole reason to use an open project rather than a closed one — you have the opportunity, should you need it, to contribute your own priorities.

When talking about an open project, it can be valuable (and factually accurate) to think “we” rather than “they.”

(Hopefully-unnecessary note: this is my personal opinion, not speaking for anyone else, and I am not a central figure in the Scala community. If I got it wrong then let me know in the comments.)