Holiday Tech Freezes – Stay C.A.L.M ( Culture, Automation, Lean, Measurement )

“Spokojnych swiat” – is one of most popular Christmas wishes in Poland. In English it means “We wish you a calm and peaceful holiday season”. However, when we try to imagine the atmosphere in an IT department of a medium or large company during December and early January, it rarely is quiet and peaceful. Specialists are either planning and implementing changes into IT environment or getting ready for their freeze period.

The choice of whether to freeze or not to freeze makes Heads of IT Departments contemplate almost every year. Moreover, how to ensure that in any case you and your team is going to have a calm Christmas time? We have prepared a short overview of the reasons to limit the changes and the reasons to go full power to support the business based on our experience and attitude.

So what is a freeze in IT? We propose to define it as a period of time when the IT department has restricted or limited changes to IT infrastructure or related resources.

In order to give well-grounded advice to our clients about freezing, we start with the analysis of the following factors:

• complexity and structure of their business processes,
• the criticality of IT for business,
• the influence of certain people on the mission-critical IT systems
• dependency from load on IT systems on certain time periods.

The main reasons to go on a freeze:

1. Lack of resources internally and among the vendors. Even if your company decides to continue working at a normal pace, the other parties of the business process can be simply not available. In some cases, even the customers may be quiet, especially in b2b companies.
2. Taking some time to re-evaluate – some organizations take this time to prepare for the next financial year and measure the results for the current one. Some think that deploying any changes during this period can come with a lot of risk.
3. Using this time to pay off the technical debt. During the freeze, some companies choose to work on low-priority incidents and requests. Sometimes the dept stocks up due to the lack of resources or because of change resistance (in the organizations that have been undergoing major changes).
4. Deciding to go without freeze too late this year (in case all the previous years you usually went on freeze). Getting rid of the freezes should be planned in advance.

We support several big accounts in different industries (telecom, banking, insurance, retail) and we see that not all attempts to do changes in this period work out well. Even worse, if you combine the factors of the timing and improper change management, you get to see awful mission-critical consequences.

From our experience, there are three types of will-hardly-work-out-well changes during this period:

1.   Initiated by the IT department

Often IT wants to enter a new year with a new equipment, hire new staff, purchase training or try to spend somehow the rest of dedicated annual budget in December. If the changes are not natural, artificial (like spending budget for useless staff in hurry) and are carried out without proper QA, they usually result in many problems and incidents. One of the examples: your vendors had no idea about the upcoming changes, as a result – you cannot carry them out without your vendors that are “enjoying” their freeze.

If you run your own business and mission critical systems, don’t forget to organize 1st, 2nd levels of support (and stricter peak-time SLA with 3d level) and adopt your usual notification and escalations scheme for the whole holidays period in advance.

We had a case when an IT department of a big retailer company initiated a project of migration from an internally developed e-commerce platform into Magento. They wanted to complete the project before December 10th, but were running behind as often happens. As a result, they forgot to test Visa cards processing in a country, which supposed to be their key market during Christmas peak sales period. This oversight resulted in huge business damage, as valuable time was lost when every second was precious.

2.  Initiated by business

Most B2C businesses want to launch a Christmas marketing campaign, or introduce new functionality at this time of the year, and as IT we do our best to help them benefit from these profitable times. Smooth transfer, full QA and organizational change management are critical here. We need to communicate, explain to business why it’s better to launch 80% of tested marketing campaigns, then 100% of untested. It’s also better to sign up stricter SLAs with vendors for peak periods to make sure you are safe. Try to work closely with your change advisory board (CAB). If you are sure that people or infrastructure will not cope with the change, the board may veto a proposed change.

When I was responsible for change management in a Telecom corporation, we planned, implemented and tested two large campaigns for subscribers. One of them demanded many calculations on the database side.  Everything was supposed to be fine. However, on the 20^th of December our Commercial department introduced two more campaigns, albeit simple ones. We checked them on test users and put them in production. This lead to high overload in our Oracle database and delays in generic customers services. We managed to tackle the problem by temporarily turning-off the collection of data for management reports; however, that Christmas was not the quietest one.

3. Initiated by the environment, government, local or national authorities

Well, you can hardly do anything apart from just going ahead with the required changes.

We had a large international bank as our client. On the 4^th of January, tax authorities of one of their operating countries introduced new taxes rates from 1^st of January! Even more, they decided to segment tax rates by registration addresses which had never happened before. We managed to implement this change and recalculate clients’ charges for the past period within one week with the help of the IT department only, and without vendor engagement. The key success factors were collaboration within Change Advisory Board, CSI approach, communication with the end clients and smart leadership.

So is there a way to control the freeze or even do it in a smart and safe way when your business cannot avoid it? We say there is – just do it in a DevOps way. Even exploring the basics of DevOps such as Three Ways and C.A.L.M.S. can shed light on they way to enjoy calm and quiet all year long.

Think about a river.  It flows well in the spring and summer but as autumn approaches and leaves or broken branches fill the river, the river begins to dam or flow more slowly.  The first way of  DevOps encourages flow – so remove the constraints.  But people know that freeze time is coming, so they throw things into the mix to try and get them in before the freeze, without proper testing or instructions. Disaster?  More incidents?  Poor planning and approval?

You need to encourage the second ‘Way of Feedback’ so that the river keeps flowing.  Why do you stop work?  Why do you freeze? Is it a trust issue? If you have a good flow and great feedback then freezing need not happen.  Work is done in small amounts so it is easy to remove or recover.  The third ‘Way of Continuous Experimentation’ and learning lets us see where we went wrong and quickly make things better.  This has a CALMing effect on our Culture, the approach we use allows automation to help our business and people, the Lean impact of supporting customers and improving work, the Metrics we use to guide us and the sharing leaders should support all year around.

DevOps never lets the river freeze

With DevOps system thinking you will never pass a known defect, and this ultimately leads to being debt free as you’ve been continuously paying it off in small amounts. With constant feedback between Dev and Ops and great communication, your team is prepared to handle complex situations. You also equip your team with automation tools and transparent measuring systems for greater flexibility and the ability to handle all possible hardships. Ultimately you end up with resilient systems that can cope with overload and threats, and a team that has the time and resources to support business in challenging times. As a bonus, with DevOps you will hardly ever end up over-flooded with requests after the freeze period. Thus, you will not put the system at risk, and your team under extra pressure because of the overload.

DevOps encourages looking at the whole river, not just a part.  Remember that when the river thaws (we let change occur) then things can happen quite quickly that are bad.  Just like rivers, Spring comes and can cause havoc, so can the releasing of changes.  Looking at the whole river and seeing the impact can prevent issues.

Whatever decision you reach, we urge you to develop a comprehensive strategy for this part of the year. Find out what will work best for your IT department and the business you support. We wish a Merry Christmas for everyone and a very happy and calm Holidays Season!

I would  like to thank Daniel Breston, Business Technology Innovation Transition – Lead Consultant, Coach & Trainer DevOps @ Ranger4 for his input to this article.