Remember when you last felt queasy before deploying a feature to production? And when you were anxiously thinking about all the unintended consequences and fixes you would need to implement afterwards? We sure do. In fact, sometimes it still makes us restless. Yet several months ago, we set out on a journey to automate deployment and make things much easier and reliable for everyone.
Our starting situation was not optimal, despite having a fairly advanced deployment set-up with continuous integration and an intricate deployment pipeline in place. The system consisted of manual deployments with manual checks carried out by a small number of people, who were actually able to deploy code. Most developers didn't deploy themselves, so they didn't care about how the features they developed were going to be deployed. To avoid incidences close to weekends, deployments were allowed only until Thursday mornings at the very latest. Of course, we always made sure to document deployed tasks in order to make what had been deployed as transparent as possible for our team's product owner and stakeholders.
The end-result, though, was always the same: our team accumulated large batches of code to deploy at once. This meant insecurity was always highest during and after deployment itself – the more you deploy at once, the more can go wrong. Needless to say, something had to change.
As a result, deploying became less dependent upon specific people. More team members were able to deploy by themselves, because the steps to follow were readily available. This meant we were able to streamline deployment and share the load. More standardization increased our peace of mind. Plus: it decreased our “lottery factor” significantly – if a key person were to win the lottery and stop working, we would still be able to manage deployments in our project.
Still, people struggled with deployment. A lot. This is mainly because they didn’t have to do it very often: imagine doing one deployment per day with a team of eight to ten developers, then each developer only has to deploy about once per week at maximum. Not a great way to make everyone comfortable with the process.
We also implemented monitoring via NewRelic and Grafana. Instead of sending status messages via Slack, filling the release sheet and moving the deployed Jira tickets by hand, all of this was done by scripts. We set up specific thresholds, at which errors triggered our alert system, for example if an error persisted over a set percentage of page visits or timespan. All it took from here on out is "make deploy".
This resulted in a highly dynamic deployment situation. The number of deployments per week (even per day) increased dramatically with overall much smaller batches to deploy at a time. Standardization increased even further, as room for individual interpretation got smaller and smaller the more we integrated into the script. Finally, we needed much less time and effort for individual deployments. We felt so sure about the process that we now also started deploying on Fridays.
Automate, automate, automate
If you've made it this far, you'll know what’s coming: automate your deployment process as much as you possibly can – and spare yourself the trouble by doing so right from the beginning. Make deployment as easy as possible, so easy in fact, that even a product owner without a developer background can handle deployments to production. For us, automation increased the transparency in our deployment pipeline and saved countless working hours – time, which we can now put to better use building great things.
Beyond the effects on our deployment, automation had a number of effects on how we develop as a team. It brought us a long way toward implementing trunk-based development (you can read up on how that went here) with shorter production cycles and real continuous delivery. Plus, we had less work in progress on the whole, the psychological effect of which cannot be overstated. Never forget: people who finish things are more happy people.
If our story has inspired you to think more about your deployment set-up and increasing automation, you can get started by doing the DORA's DevOps quick check to see how you measure up. The DevOps research program also has some useful resources on everything from version control to continuous testing and delivery.
Finally, and this cannot be stressed enough, automating your deployment processes will lead not just to more stability and reliability in your deployment, but also to more stable and reliable software.