Saturday, 10 November 2007

Release Criteria: Aligning official documentation with reality

First of all: Thank you for your positive feedback on the MySQL 5.1 Errata Sheet!

While I never doubted that publishing the 5.1 Errata Sheet was the right thing to do, I had expected a more mixed feedback. It turned out the feedback was very grateful. So thank you for your encouragement

  • Kevin Burton for commenting in Jay's blog
    "This was a big win guys. Good work.We were going to deploy 5.1.x in out or production slave configurations just to test the reliability but this give us a lot more confidence in this release."

  • Baron Schwartz for commenting in my blog
    "Kaj, this is VERY helpful. I just discovered the character set issue with mysqldump myself last week.This is a big step forward in quality, from the user’s point of view. Now it’s much easier to see the state of things."

  • Peter Zaitsev for explaining some background in the blog entry "Making bugs public - good job MySQL"

The Errata Sheet had somewhat of a difficult birth process, as our pompously titled "Release Philosophy" page optimistically refers to the ideal situation of "no known critical bugs". If we follow that policy, there is no need for an errata sheet, is there? The Errata Sheet should by definition be empty.

But it isn't.

And it wouldn't have been, for the last four years, had we just had sufficient awareness of critical bugs, and a policy of publishing them.

The only logical conclusion from our Reality Check was that wishful thinking must stop, and we need to do the right thing under the circumstances: Get the best possible quality releases out with the given resources, and set the expectations of our customers and users according to what we can and will deliver.

And hence we appointed a cross-team working group, the "Release Criteria Committee", which is defining both external and internal processes and policies when it comes to

  • releasing new releases in general

  • moving from one maturity stage to another

  • dealing with bugs of all kinds

  • aligning MySQL release policy documentation with reality

So let's first look at the circumstances:

  • Resources are finite: There is only so much time in a day, and there are only so many developers around to fix bugs.

  • Timely releases have value to our users: Earlier access to bug fixes. Earlier access to new features. Earlier testing of both, i.e. faster convergence. Reduced guesswork about when the next release will be.

  • Zero critical bugs is impossible: Sure, zero bugs is desirable. And at times, we internally glorify MySQL's past, with "production level alpha releases". Those were the days, but those days also had less complex and less interwoven features, and fewer developers to manage.

As such, it is MySQL's responsibility to create a release policy that is aligned with the needs of the customers and the user base. This will involve a number of difficult judgement calls. However, we do think that we've got a lot to learn from our customers and users, when it comes to defining the best possible release criteria. So I am inviting your input.

To structure your thinking and the work, let me briefly explain some basics:

1. We have four maturity levels: Alpha, Beta, RC and GA.
1a. Alpha is for testing and development only. Features are added, and
syntax may change. (*)
1b. Beta means feature-freeze. We fix bugs. (*)
1c. RC or Release Candidate needs solid criteria. RC has some industry definitions, and is the final stage prior to GA.
1d. GA or General Availability means the version is for production use. The name GA is somewhat silly for Open Source products, as even alpha releases are available for the general public.
(*) These are my own simplifications, and our work group may refine them.

2. Bugs are described by their Severity, Defect Classification, Impact, Risk To Fix, and Effort. They may have even more dimensions.
2a. Severity describes the damage from the original user (bug reporter) point of view. The most severe bugs are about crashes, security problems and corruption, without any known workaround. The least severe refer to unpleasant but documented behaviour, i.e. feature requests. From the severity of the individual original reporter, we derive the Defect Classification, which attempts at being a neutral classification across our user base.
2b. Impact refers to how many users may be affected. They range from
the entire user base, all the way to corner cases that affect only users of a particular feature on one platform.
2c. Risk To Fix and Effort refer to the difficulty of fixing the bug. A bug fix may destabilise the code (i.e. introduce more problems than it fixes), requiring postponing the fix to a later version. And the time for fixing the bug may vary (influencing the complexity of reviews).

Some of my personal thinking on the release policy is as follows:

(i) Maturity changes need quality criteria.

  • Progressing from Alpha to Beta is not useful if most testers find the same elementary critical bugs; basic usability should exist

  • Progressing from Beta to RC needs for the impact of critical bugs to be very low

  • Progressing from RC to GA needs the impact of critical bugs to be even lower

  • Hence, I advocate quality based (not time based) release criteria for maturity changes

(ii) New releases within the same maturity level need different quality criteria (from above).

  • Regression bugs must be avoided (what worked in release n must still work in release n+1) (**)

  • Beyond that, we mainly need for the quality of release n+1 to be higher than that of release n. In practice, if you look at today's (10 Nov 2007) version of the Errata Sheet, you will see a lot of bugs annotated with "Already fixed in 5.1.23", while others have "Target fix: 5.1.23". "Let's not wait until we can cure all illnesses, before handing out the medicine we already have!" is my logic.

  • Hence, I advocate mostly time based (seldom quality based) release criteria for new releases within the same maturity level

  • (**) By "regression bugs" I mean unintentionally changed behaviour, i.e. pure bugs. There may be situations where behaviour has to be consciously changed to enable the fixing of a bug.

(iii) Simplicity rules.

  • The definitions of the dimensions that describe bugs must be as simple as possible.

  • The rules triggering maturity changes must be as simple as possible.

  • It should be easy at any point in time for us to evaluate whether we should change a maturity level (as opposed to being a negotiation between self-proclaimed advocates of quality and timeliness, respectively)

(iv) Timeliness has a value, and RC criteria may differ from GA criteria.

  • Some claim that by definition, an RC must fulfill all GA criteria except the exposure to wide user base testing

  • Others (like me) think that it's in the best interest of the user base to converge to a GA quicker, through marginally relaxed RC criteria, whereby we can fix the remaining less-critical bugs in parallel with the RC getting exposed to massive parallel testing by the user base

When we (the members of our Release Criteria Committee) think we're done, we'll upgrade our Release Policy (and, while we're at it, no longer pompously describe it as a Philosophy). We're working at a fairly hectic pace, and expect to be done within weeks rather than months.

In the meantime, your input is welcome!

1 comment:

  1. I personally appreciate very much the new errata sheet. It really is a big help. And while I wouldn't presume to give specific advice I very much thing you all are on the right track.

    You are right in that times and the MySQL product has changed. It is not realistically possible to have releases that are "bug-free". As users of the MySQL we desire/demand new features but adding those same features brings more complexity to the system.

    Anyways, as long as there is a clear set of "what bugs affect this release" (ie.. the errata sheet) it is a lot easier to decide when to move to a release.

    Thanks for the hard work!!