Continuous Deployment Isn’t Easy – Yet
When CI is not enough
I have been looking for some guidance on best practices for continuous deployment for projects at work. It seems like the answer has always been to set up a CI (continuous integration) server. And while that is great advice there is still so much more in terms of people and process that needs to happen to make the CI server useful to anyone but a stat loving development junkie.
What are the successful consultancies saying about it?
The video at of Sam Newman, a Thoughtworks consultant, at http://www.infoq.com/presentations/From-Development-To-Production is an amazing example of both presentation skill and content. The involvement of different groups and people within an organization and some of the tools that can be used are shown and discussed. There is a huge amount of ideas that can be gleamed from the video but there is a disturbing idea – that the people and organization might have to change in order to make the process work. In most of the examples the organization is a shining beacon of productive people who want to take responsibility and get things done. The reality I have seen at most organizations is that the developers spend most of their time filling out paperwork and everyone is in CYA mode. That doesn’t bode well for continuous integration. As the presentation shows, there is no standard way to do this and there is no drop in tool that will do it all for you. It takes some thinking and risk taking to get the tools you need built and it takes solid developers that can produce functional code independently. Unfortunately, most organizations do not have two decent developers to rub together so their efforts at continuous integration seem to flicker and then peter out after they install CruiseControl.
A couple of notes from the presentation that hit home:
- to store the configuration for projects for dev/qa/pro in your SCM (source control manager) with you project code
- validate config values in the project code on startup and fail hard and fast if there are any issues
What do successful startups think?
The Facebook founder, Robert Johnson, in a video presentation (http://www.infoq.com/presentations/Facebook-Moving-Fast-at-Scale) provides some insight into how they are organized to go from concept to production implementation in a day. He spoke mainly about the architectural pieces at Facebook and how they overcame some scaling issues. Facebook has spent some time streamlining their implementation process so that the implementation is extremely tailored to their application. In their case they only really are ever deploying one codebase which greatly simplifies their releases. If I only had to deal with one idiosyncratic project deployment I am pretty sure I would get as good at it as they have. One thing he mentioned did ring in my ears:
The people who have responsibility should have control.
How can you tell if your organization has a problem with this? If you are looking to solve a project implementation issue and all you hear is “that is something that XYZ handles” from your developer then they don’t have the control or communication channels they need to effect change. It might seem like a vanity or control issue but the developer has a vested interest in making sure the deployment is smooth and painless since it reflects on them. They also have the tools to make deployments more automated.
One thing that wasn’t brought up until the question portion of the presentation was that Facebook has teams of people whose job it is to clean up after the initial development. These full time refactorers go through and scrub clean the initial mess created by the breakneck development pace. He even seemed to even hesitate to answer how they prevented a big ball of mud (http://www.laputan.org/mud/) despite their intense dev/deploy schedule. After all, isn’t a PHP app a big ball of mud with only a single code file (“unfair man. i love php. it’s not the language it’s the programmer!” yeah, whatever hippie. take a shower.)
How can these lessons find their way to where I work?
First things first. I’ll have to show at least the first presentation to the other developers involved in the continuous integration project. The other thing is to make sure that the right leadership is in place. We need to do a decent analysis of the pieces in the deployment pipeline and the tools that we will need. From there we can decide to buy or build the tools we need and start getting support from other departments. There are still huge people and process issues like the ONE FINAL CHECK-IN that needs to turn into an incremental feature based check-in that happens at least once a day, but overall it is doable and the presentations gave me a solid enough framework to take back and pitch to management.
Another great lesson was in presentation styles. I (hopefully) find myself in the Sam Newman style of presenting more that the Robert Johnson style.