Havoc's Blog

When I end up explaining the same thing over and over, I try to get around to writing it down. So, pretty often I talk to someone who doesn’t track the open source world on a daily basis, and doesn’t understand the idea behind Red Hat Enterprise Linux: it’s a subscription, but to what?

Red Hat has several products sold via subscription. The OS, originally known as Red Hat Linux Advanced Server, is now known as Red Hat Enterprise Linux (with four variants: AS, ES, WS, and Red Hat Desktop). We’re using the same subscription approach to Red Hat applications on top of the OS, such as Cluster Suite and Application Server.

The way I think of it is: anyone can download the software from the Internet. Just go to kernel.org or SourceForge or wherever and grab the bits. What the subscription buys is all the work Red Hat adds on top of the bits. But what work is that? What do people here do all day?

Some of it is illustrated by Red Hat Enterprise Linux Update 3, which just came out. This includes:

• New features, such as exec-shield
• Security fixes
• Bug fixes
• New hardware support, so the OS runs out-of-the-box on the latest stuff you can get from hardware vendors
• Each change is documented anddocumented again and tested by QA
• Changes are done while preserving ABI compatibility, API compatibility, config file formats, and program semantics; in other words, rather than say wholesale upgrade to a new upstream version, the changes are carefully backported, or the new version carefully evaluated
• When the changes are made, the same change is made on “HEAD” or upstream, to be included in future versions

In addition to the rolled-up mass updates, we have out-of-band single errata (for time sensitive security issues), and “hot fixes” delivered to specific customers.

These updates are a ton of work, because they don’t involve just downloading the latest version of everything; the fixes have to be backported and tested. It’s the difference between saying “to fix this security flaw, go from Apache 1.x to 2.0” and “to fix this security flaw, install this small patch to Apache 1.x.” We do this ongoing work for five to seven years, so if we release a new version annually, we’re building up to seven of these updates simultaneously. Needless to say, this isn’t trivial.

When I say “backporting” I’m contrasting it with two other approaches. One is the wholesale upgrade, and the other is permanent forking. Take the Enterprise Linux 3 kernel for example, which has many of the key features of the 2.6 kernel. Lots of patches. But when we moved to the 2.6 kernel, we dropped nearly all these patches without creating regressions in functionality. This way we meet customer demands quickly, but don’t create a long term problem for those same customers. (As an aside, time-based releases as in GNOME mean we only do small backports in updates, rather than large backports in the initial release as seen with the kernel…)

What else does Red Hat do all day?

• Keep the same version of the software running on seven CPU architectures (again, we do backports, we don’t upgrade to whatever works)
• A QA and release engineering process on all seven architectures
• Develop new features, tracking a list of customer and partner requests and prioritizing features based on those requests
• Recruiting and certifying ISVs, and modifying the OS to make their software work well
• Certifying hardware platforms and devices
• Standards-compliance and other certifications and benchmarks
• Technical support, in-person, on the phone, via web and email
• Documentation
• Security errata, including tracking, fixing, and documenting each issue
• Upstream maintenance duties for lots of open source software
• Inclusion (on a separate CD) of some proprietary software packages that require licensing
• Bandwidth to deliver updates and fixes (this costs a lot more than you might think)
• A lengthy beta program, where we test the software for several months and collect partner, ISV, and customer feedback
• Guarantee from a company that all of the above will happen

Every one of these things involves full-time Red Hat employees and costs money to provide. You could call the full list “support,” but I tend to avoid that term since it makes people think of calling someone on the phone for help. That’s part of it, but a supported, enterprise-class OS involves a lot more.

Another way to think of this can be found in this table contrasting Enterprise Linux and Fedora. Still, the delta between Enterprise Linux and the Fedora Project doesn’t show the full value of the subscription, because we do give a lot of our work away in the Fedora Core releases. Fedora contains our new feature development in real time, while that development is done. The integration work to convert hundreds of upstream packages into a working, compiled OS happens in a Fedora context as well. Even if you use Debian instead of Fedora Core, we’re always feeding a stream of features and fixes directly into upstream projects, and anyone can benefit from those.

It absolutely makes sense that we give away what we do, because what we’re giving away is the same kind of thing we receive from others: enhancements to the source code. We benefit from countless code contributions from around the world, and in turn everyone benefits from our code contributions.

I think the subscription model captures the original idea of an open source business very well. If you want just the software, here it is. If you want the software as a supported, vendor-backed product with everything that implies, we have that too. If you buy a subscription you benefit from the work that the company does; otherwise you don’t. Most importantly, if you can support yourself, or if you want to hire another company that can support you, go for it. Fortunately for Red Hat, many people let us do it for them.

Once it’s explained, most people seem pretty happy that our subscription approach is sensible and in the spirit of open source. (Unless they confused free speech and free beer from the start.) When someone remains unhappy, it often has to do with the exact product profiles and pricing. Taking the operating system for example, we slice-and-dice it into four versions (Desktop, WS, ES, and AS) and strip out the tech support for an Academic version. This is oversimplifying; we also have site licenses, volume pricing, and so on and so forth. I don’t pretend to be an expert on product pricing, but will just point out that it should be above the cost of producing the product, reflect what people are willing to pay, and is ideally kept a little bit simpler than plane ticket pricing. Hardly an exact science, and something that evolves over time.

One final aspect of the subscription I haven’t mentioned. The subscription is to the product line, not to a specific version. So if you buy an Enterprise Linux ES subscription, you can run 2.1, 3, or any future version using that subscription. We have no financial reason to force you to upgrade. The only time you’re required to upgrade is at the end of the product support lifetime of 5-7 years; even then, you can of course keep using the bits, if you can find someone to do security errata for you, or if the computer is not connected to the Internet.

(This post was originally found at http://log.ometer.com/2004-09.html#5)