User Tools

Site Tools


Sidebar

Dave Orme muses about data-first development.

My current work emphasizes data engineering and analysis using Kubernetes, Clojure, Scala, Eclipse, and Google Cloud Platform or AWS.


Blog

The Cloud

Scala, Clojure, and FP

Data-First Development

Agile

Older work

Coconut Palm Software home


Donate Bitcoin:

1Ecnr9vtkC8b9FvmQjQaJ9ZsHB127UzVD6

Keywords:

Kubernetes, Docker, Streaming Data, Spark, Scala, Clojure, OSGi, Karaf, GCP, AWS, SQL

Disclaimer:

Everything I say here is my own opinion and not necessarily that of my employer.

blog:agile-_stress_testing_a_release_process

Agile- stress testing a release process

Sometimes you get on an airplane–or into a release process, and partway through the flight, something exciting happens.

You never want this to happen, but sometimes it happens anyway. And it becomes an opportunity to find out just how resilient your processes are.

One time I was on the way between Chicago O'Hare and Los Angeles International, and suddenly the airplane hit a really big bump, the captain instantly turned on the “Fasten Seatbelts” sign, and announced that we'd run into a bit of “unexpected weather”. As it was night, and I had a window seat, I then had a fantastic view of the lightning flashing around us when I looked out the window and saw thunder clouds towering from far below us to high above us.

This week, our agile team had the equivalent of the “fasten seatbelts” sign come on when our team's build server went down 1 week before our ship date and just as we were entering the testing phase of our release process.

No worries, we thought. We had prepared for this eventuality and simply moved the build to a backup machine while we waited for the primary machine to get the attention it needed to come back online.

Then yesterday–one day before our scheduled ship date and the day we were scheduled to build our gold master release–the backup build server suffered a catastrophic hardware failure. New machine needed. We had a day to rebuild our machine from original software images and configuration files, build the release, perform our due and administrative diligence, and ship.

However, being part of an extremely risk adverse organization, we had been forced by the organization several months ago to document all of our processes. We had all of the build server configuration files backed up in SVN. And we had the documentation we needed to quickly rebuild the server from scratch if we needed to.

In an 8 hour day's work, two of our team members rebuilt the server. Today, we built the gold release, did our due diligence, and shipped. On time and on budget.

Kudos to my manager and to all the team members who worked hard to implement the agile processes and documentation that made this possible. This is truly one of the greatest teams I've had the privilege to serve on–but we couldn't have done it without the agile practices that we have implemented from day one: agile practices based on Scrum, but with lots of input from XP and Eclipse Way.

blog/agile-_stress_testing_a_release_process.txt · Last modified: 2018/02/01 11:43 by djo