Software Engines

Please read the next two paragraphs of "marketing hyperbole":

We have developed an engine that delivers blogging, web site hosting, instant messaging, file sharing, revision control, email distribution, and many interesting combinations of all those things (uploading files to your web site with built-in revision control, posting notes to your blog with automatic email notifications, etc).

Our engine is also blazingly fast and has very high capacity for simultaneous users. It's not written in a slow interpreted language like Perl or PHP or even Java. It's written in ANSI C. It's small and has a low memory footprint.

I believe all that hype, and would like to convince you of it too. In fact I'd like to compare our engine to our competitors in a way that does not reflect my own bias, but is more like a benchmark. But I'm having a lot of trouble doing so....

There are no metrics for software engines. No concept of horsepower or displacement or torque or miles per gallon like automotive engines. No way to measure gigahertz or dual processors like hardware engines. You're just expected to evaluate a server on marketing materials and feature claims.

When hybrid cars were introduced, they could be compared to conventional cars in a way that people could understand. The tradeoff was lower horsepower in return for much better mileage.

We've been looking hard to try to find a benchmarking service that will help us calibrate our software engine against competition like Apache and Movable Type to back up our claims to speed and capacity. We can't find any benchmarking facilities at all so far. If you Google for "web load testing" you find hundreds of "tools" that let you test your web-built software application to see if you can handle 20 users or not. I've downloaded and tried to build a half dozen open source tools. They're terrible. They don't even compile on FreeBSD or OS X. The ones that do, don't work very well. Some of these tools boast that they can simulate "up to 256 simultaneous users!" That's nothing. How do we simulate up to 250,000 simultaneous users?

The tools, and the load-testing industry, and the press/analyst measurement and metrics, well, they're just not there.

Why is this? Web servers and software engines in general are becoming the underpinnings of modern society. Everything is running on a server somewhere nowadays. It seems important to be able to evaluate and benchmark these engines in the same ways that we can evaluate most other kinds of engines.