1. Start with a Test Suite

Remember the goal is to go fast with confidence. If you're just going fast, you're simply speeding, and will soon become another statistic.

Okay, end of car analogy. However, you need to build confidence, which is what the test suite is for. Start there. You don't need to over-do it, but be smart. Start by ensuring it's easy to add tests, and then write relatively comprehensive tests for the critical flows in your application. What these are is entirely up to you - if you don't know them, maybe you had better start there :).

You should also set up a CI server, and get the tests running on every commit. If nothing else, this should enhance your existing development process.

2. Rig up Continuous Deployment to Staging

Staging, test, UAT, whatever you call these environments that aren't production - get continuous deployment working at least that far. These environments can be broken with relative inconsequence, so you can work with confidence initially. Over time, you should find their uptime improves, especially as your test suite begins to show its strength.

In particular, in this step, your aim is to put the test suite in the way of deployment. It should be impossible to deploy without a test run having completed successfully on the code being deployed.

Note that the presentation suggests that you employ a "Commit triggers tests, tests passing triggers deployment" tactic. This is a simplification. In larger teams, it may make more sense to work more like this: "tests are run every 15 minutes, with success resulting in deployment". This allows many commits to be made in one window (common for large teams), with the knowledge that anything you commit will take at worst 30 minutes to make it to production (assuming the tests pass).

3. The Management Gambit

You will know you've done the first two steps when the tech team thinks "this is working nicely all the way to staging - why not to production?". There's one final obstacle in your way - the fear of everyone outside the tech team.

On some level, management/clients are going to have to get comfortable with the idea of code being deployed without their approval, and often even without their knowledge. You can use presentations like mine to help here. Remember, if you can win over just one person, they can help sell the idea to everyone else.

At a base level, they most likely will fear the lack of control they will have. To combat this, tell them that you'll implement the system, but they have the right to demand change freezes - periods of time where you won't be allowed to deploy. They probably already have this right, but re-affirm it anyway.

This is false security for them. Martyn has a sign on his desk, which reads "Change Freeze (n): A time in which more changes are made to production than usual". When there are urgent problems, change freezes be damned - changes are made, and we all know it. The next time they have a change freeze, they'll ask for a change, and instead of flailing about you'll be able to make it with a smile. They'll never have another freeze again.

I also mentioned during the talk a USB rocket launcher for retaliating against people who break the build. Enjoy!

Continuous Deployment - Going Fast With Confidence

On the CD project, the complete lack of effort required to deploy changes has been a huge timesaver. I have never felt the need to wait before deploying one change, even if I was about to work on another. Little fixes, in other words, made it to production very quickly, pleasing my customers far more than assurances of "it'll be fixed next week" would have.

I think the best moment was when I noticed a user trying and failing to complete a wizard due to a bug. I fixed the bug and deployed - allowing them, on their sixth try (and probably to their complete surprise), to complete it. If ever there was a moment where I appreciated the value of a robust, quick deployment process, this was it.

Furthermore, this experience highlights one of the key benefits of CD. I could have hacked a fix on production - but it was easier to use the CD process, which included a full test suite run. There's simply no way any other process could have provided the same speed with the same level of assurance - hacking on production would have been the only faster option, and it would have been wildly dangerous. [1]

Fortnightly Deployment - The Lie

On the fortnightly deploy project on the other hand, we encountered all the same issues that I'm so tired of.

We'd do a release, then for the next two weeks, some fixes would be marked as so urgent that we had to do a deployment of just that fix, immediately. We'd made sure deployment was as close to a one-command process as possible. However, the process of patching and testing the stable branch was an annoying break in rhythm, given that we were doing most development on trunk [2].

This was actually a point raised by Andy Chilton at the talk today. It seems that many project teams realise that there are some fixes that just have to make it out fast, and as a result they build a separate "hotpatch" channel to accomodate them.

In my view this is madness, no matter how well tended the "hotpatch" process is. Do your hotpatches go through the test suite? They certainly should! And why create a "fast path", and then forbid its use in ways that would delight your customers?

But I think my biggest objection is this: why have two processes when you could just have one? We coders know the evil that lies in needless duplication and complexity - which is exactly what a "hotpatch" system is. Duplication and complexity.

The whole idea of having a separate deployment process exposes the "fortnightly" claim as a lie anyway. Who can honestly claim they deploy every fortnight, if they're hotpatching? [3]

Objections to CD

Perhaps the strongest objection that came up was that clients wouldn't tolerate the possibility of things breaking without them being aware of it. To me, this objection has a slight air of childishness about it - I'd give it more credit if clients ever bothered to hire a world-class QA team, but they never do, and they miss bugs slipping into production all the time even with their checking. I think there's just our old friend, the "Cover Your Ass" policy, at work here.

Besides, nothing about CD precludes the possibility that they can still have a QA team checking things - with the able assistance of feature flags that limit features under development to just them. And I'd contend that the QA team would be just as delighted as the client themselves when told a bug they found half an hour ago was not only fixed on production and ready for them to check again, but that a test had been written to make sure it never happens again.

Having said all of this, Brenda Wallace made the point that it all depends on the client, regardless of how good the idea sounds. Some simply won't change from what they know, and at the end of the day it's their project. Perhaps this is why CD is doing so well in the tech startup world - it's the startups themselves who are the clients [4].

Try It For Yourself - I'll Help

All up, it was a great discussion, and it seemed like many there could at least see how CD could be better. If you count yourself among their number, I encourage you to try it out on the next project you do, and see how you go. I'm more than happy to chat with you about it and share experiences if you do, so feel free to contact me if you want to discuss anything about it.

As an aside, I do intend to continue my Web App Performance series, I've just been focused on other things recently. Apart from business, I've joined the Standby Task Force and am developing scripts to automatically deploy an Ushahidi within a few minutes of a disaster occuring. More on that in a future post.