Release Criteria

A complaint that's become increasingly common with new games is that they don't feel "finished". Look around the forums of almost any newly released game and you'll find a post to the effect of "I paid for a complete game, not a beta!" Reviews of "has a lot of potential, but feels like a work in progress" are not uncommon, even for high-profile, big-budget games. While I think part of the blame for this trend lies with publishers who push games out the door too soon and developers who don't spend enough time on testing, it's also an inevitable result of increasing complexity, both in terms of games and of hardware. It's simply impossible for a game to run perfectly on the nearly limitless combinations of hardware and software that make up the PC landscape. Game companies have to make some sort of compromise in order to release a game in a timely manner at a price people are willing to pay. An open source game, on the other hand, is always considered a work in progress. Short release cycles and small iterations foster the feeling that the end product is a continuous process of improvement rather than a monolithic single product. There are bugs, of course, but user tend to be more tolerant of them: they know that bugs are (at least in theory) in their power to fix, without being hostage to the original developer for changes to the code.

Metropolis falls somewhere in between these two cases. The game itself, being open source, will be improving continuously; In a sense it will never be "finished". But the development model we're pursuing, which asks users to fund the game by "buying" shares, also demands that there be a definite release, where a final product is delivered to the "buyers". The question I want to address today is, how do we determine what constitutes an acceptable point for an official release?

Of course *some* companies can afford keep their products in beta forever...

Before addressing the kind of criteria we might want to use, let me reiterate why an official release is necessary. First of all, it's a natural corollary to the way funding for the project is being organized. The budget is collected prior to the start of development; with only a set amount of financing, it's not possible to keep working on the game indefinitely (or "until it's ready," as many open source projects do.) Second, since we are asking for pledges on a basis analagous to a more traditional purchase, members will expect to see a completed game on a basis analagous to a traditional game release. Lastly, as I outlined in a previous article, membership status in the project is based in part on whether the game has been officially released; Without a definite end point to the official development cycle, we will have no way of determining when the terms of membership have been fulfilled.

So, when is the game finished? We are somewhat fortunate in that there is at least one immutable answer to this question: if the budget runs out completely, the game will be "finished" at least far as "official" development is concerned. So why not just use this as our measure? Work until the money runs out? There are a couple reasons why this really isn't a desirable outcome. Software development isn't simply a matter of more lines of code = more finished; a particular feature being two-thirds complete isn't any better, for the end user, than it being one-third complete. Our goal should be to have a smoothly working game, even one with fewer bells and whistles, rather than one with a multitude of half-completed features. To make that happen, we'll need to have a larger budget than we expect to need. Why? To accomodate:

• Cost overruns: Writing code isn't an exact science, and estimating how long it will take even less so; It's almost inevitable that some things will take longer than the original estimate, and thus cost more; That will have to be reflected in the budget to avoid being stuck with something half finished. That also means, however, that if things go smoothly there might be extra money left over.
• Testing: Testing is vital to the development process, but there's no way to know how much will be necessary until the testers actually dig in and start finding bugs. Again, this is something we don't want to skimp on, but it also means potentially having some of the budget unused if things go well.
• Post-release Patching: This is probably the most relevant one of all; Inevitably, some bugs won't manifest themselves until the software is in the wild being abused by the masses of end users. It's expected that game developers provide patches for any critical bugs or issues that emerge post-release, and Metropolis will be no different; the last thing we want to do is dump the 1.0 version on the community and say, "Well, we're done!" But working on patches after the release means setting aside money to do that, so "work until there's no money left" becomes impractical as a strategy for determining when we're done.

As we can see, just using the budget as our measuring stick would be a disaster. What other measure of completion can we use? A commercial studio ultimately makes the decision based on economic considerations: the best time to release is as soon as the game is polished enough to generate good reviews and sales; Further testing or improvement beyond that point only subtracts from profits; In other words, the less money the studio can get away with spending, the better. Thus publishers will push to release games as soon as they can be considered even remotely finished.

At least in most cases.

For Metropolis, our goals are slightly different. We do, of course, have a budget, and thus can't ignore economic reality; But our goal is not to make the game as cheaply as possible. Money left over in the budget is not profit, it's simply wasted (well, we can use it to start working on expansions, but that's a different issue.) Our objective, instead, should be to use every dollar we collect as efficiently as possible for improving the game. To put it another way: We want to make the game as good as possible with the money we have.

To figure out when we've reached that point of optimal completion, we have to turn to the final judges of whether the game is "good" or not: the community of project members. Ultimately, it's the players who must decide whether or not the game is finished. So how do we gauge something as nebulous as "player satisfaction"? At first blush, it seems a simple vote would do the trick: the dev team posts a release candidate, the members try it out, and if enough vote "finished" the game is considered released.

But there are potential problems with this approach. Most glaringly, it doesn't establish any metric for members to decide what "finished" is. Some members might interpret "finished" as "mostly playable" and vote yes on that basis, while some might swing completely in the other direction and consider "finished" to mean "completely free from bugs, with all the features the community asked for included," and thus continue to vote no until the game is spotlessly perfect. Without specific metrics by which to reflect a reasonable balance of polish and budget reality, it's entirely likely that a straightforward "complete" vote might never happen (leaving us to fall back on the worst-case "out-of-money" scenario I described above.)

The solution I would suggest is to pull back a level and look for the community's input not only for simple confirmation, but for a deeper analysis and selection of the criteria themselves. In effect, members would both ask and answer their own questions. It might not be immediately obvious why this is an improvement over simply asking for a vote; but the difference is rather profound. By asking members to enumerate what is most important to them in judging the game complete, and sharing those criteria with others, we can encourage a broader consensus in the community and make a game that more accurately meets expectations.

Here's how this might work on a practical basis: We create a forum in which users can post objective questions about different elements of the release candidate. Examples might be "How many of the features pitched by the development team have been implemented?" or "How many bugs have you encountered?" The community could then vote these questions up or down (in a format similar to Stack Exchange, for instance.) The most popular questions would then form the survey used to determine whether the game is ready for release.

While this approach might be a little cumbersome, I believe it can really help the community to reach a general agreement about when our game is done. As always, I'd love to hear everyone's thoughts on whether you think this is a good approach, or how we can improve it!