Development, Sievo Culture

#SIEVODEV: Turning the Big Ship Around

Sievo has a history of shedding its skin. Years ago, the product was a desktop application behind our customer’s firewall. It was originally written in Visual Basic by our founders. Now it’s a SaaS web application. The current technology stack is mostly C# with a touch of F# on the backend and React in the front.

I cannot tell you that story. I was not there to see it. I joined only eleven months ago.

Here is my war story

In May 2016, I joined a company with an ache. Each customer was a separate entity in our systems. With a bit of simplification, this was the structure: there was a database, the service itself and QlikView reporting. All of the mentioned must be in the same version or else.

Before, when a new version of the software was created, we would contact our customers to schedule an upgrade. First, we ran a test upgrade to test the process and validate the new version. After a few days of waiting for our busy people to give us a thumbs up, we took the service down for maintenance and performed the real upgrade. In May, the average version running in a customer environment was almost half a year old. That was a great job from a few very diligent people to keep everything up to date. Even this was an achievement: six months prior, the delay had been close to a year.

In practice, our customers were getting neither the latest features nor the fixes to defects in a timely way. In addition, we were supporting more than twenty versions of our product.

In June, we started a BuSu project. These are key initiatives to “Build Success” that have a sponsor from our top management. This time the sponsor was our CEO, Matti. The goal was to make version upgrades automatic at the press of a button for ten customers during 2016. The team consisted of a customer relations manager, software engineers, a delivery engineer, myself and the project was led by our chief architect.

The complicated button

To start, the process was automated in a rudimentary fashion. It all started by editing an Excel description of the environment, running a script on a specific host, specifying the intent with a string of parameters and waiting. Naturally, this was after checking that all the parameters are correct.

Our customer relations manager started looking for customers who would be happy to get the latest versions as soon as possible. We, the techies, started simplifying the tools in small increments. I started on a spy mission to see what the people performing the upgrade validations were doing.

Instead of letting our diligent delivery engineer execute the upgrades, we told our chief architect to pretend to be the button that runs the upgrades. One could call it a concierge prototype.

Every time a little better

With every run of the mill, we made small adjustments. In mid-June I knew that validations were not exposing any issues at all. There were 26 steps in the process to upgrade a single environment. In August, we had dropped the execution time by a third by skipping unnecessary calculations. Also, an ExterminateExcels script was created to migrate the old description files to a new format. I smuggled in some ASCII art of a Dalek into the script.

Script

In September, all the usual causes for failures had been removed. There were 20 steps in the process to upgrade the ten customers who opted in. The execution would not halt to prompt the user. The test upgrade and actual upgrade were run back to back.

In December, there were just 5 steps in the process. We had 22 customers opted in to the new process.

Today

The entire process to schedule upgrades for 75% of our internal and customer environments looks like this:

1. Log in

2. Set the dates in `batches.json`

3. Execute `JsonScheduler batches.json`

These three simple steps will schedule most of our systems at preset times. The status will be reported by a bot on our Slack chat.

Leaner Meaner

This is not yet perfect. In a few weeks we will remove step 2 by letting the script infer the next available times for upgrades.

The benefit

What’s the moral of the story? Through decisive action and working together, we have freed the time of our delivery team almost entirely. Instead of starting our mornings with an upgrade, we have increased our coffee intake, mood and quality of humor.

For our customers, there is no stress in finding days when the service can be down. The new versions get installed within a week. The largest get the most tangible benefits in service availability: The old method took the service out for 17 hours. The last upgrade took their service offline for two hours.

We are very proud of this, but we’re not nearly done.

I am pressing delete on the ExterminateExcels script, stage complete.

We even have an IoT button waiting…

IoT Button

 

All articles

Related articles

Tommi Salmi

Sievo Culture

Summer at Sievo - what is it made of?

As the fall starts to be lurking around the corner, it's time to take a look at past few months at Sievo as Summer Seal – and how the COVID-19 flavoured summer turned out for me.OnboardingOnboarding of the summer seals happened during the worst corona...

Elena Pitkänen

Sievo Culture

Welcome Summer Seals!

This year 13 summer seals joined Sievo for the summertime. Despite the unusual Covid-19 situation the seals are now part of the Sievo family and they have thrived in their duties! Here are some greetings from a few of us:

Never miss news from Sievo

Get updates

Never miss news from Sievo