You’ve integrated DevOps to help drive your company’s digital transformation. But how do you track the success of your DevOps initiatives and communicate these objectives in a meaningful way to your organization? This can a tricky process because, by design, DevOps isn’t a formal framework and provides limited guidance. The key to success is to create metrics that tie DevOps processes to real business outcomes.
Outcomes that deliver more value for the customer and aligning with the business.
Understanding the Business Challenge
Expectations are high for DevOps to get it right every time in order to meet the demands of today’s always-on, highly competitive digital economy. However, this is often easier said than done. The interaction between DevOps initiatives and continuous delivery is essential for business agility and revenue growth but there are many challenges associated with streamlining collaboration across the teams and different stages of the lifecycle.
Focusing on the speed of delivering new products is critical but cannot be the only focus if you want to tackle the challenges of digital transformation. DevOps teams need the right tools to manage these environments including comprehensive release lifecycle management capabilities to keep pace with the cadence of agile and continuous delivery work streams.
But they also need metrics or Key Performance Indicators (KPIs) to track their success as well as uncover areas for improvement or modification.
Best Practices for Choosing the Right KPIs for Your Organization
When crafting your DevOps KPIs, consider the following best practices:
Tie DevOps Metrics to Business Outcomes
It’s important to think carefully about which metrics you need to track and remember that there’s no one-size-fits-all approach. The key is to choose the metrics that will answer the questions that are most important to your organization — those that focus on specific business outcomes. In solving this puzzle, think about the drivers of organizational success that accelerate return on investment and business value. For instance, if customer experience is a key focus, then you would choose metrics such as the Net Promoter Score. On the other hand, if efficiency is a priority then focusing on more cost-centric measurements would be more appropriate.
Getting Buy-In from All Stakeholders
Answering these questions will require the strategic input of all business stakeholders who must come up with a common set of goals that are clear and measurable. Overall, providing this framework also allows Dev and Ops to be more closely aligned in the determination of the organization’s ever-changing digital capability. It ensures that the focus is placed on improved customer satisfaction and saving time and money.
It might also be helpful to note that more and more organizations are now utilizing a data-driven approach to make informed, reliable decisions and support challenges with service delivery. In fact, according to a recent Gartner survey, by 2020, “50% of IT organizations will apply advanced analytics in application development to improve application quality and speed of delivery.” The report goes on to say that organizations that fail to embrace this data-driven methodology will be, “forced through cost-optimization measures to make ad hoc cuts that may ultimately lead to catastrophic consequences.”
Four main types of metrics
KPI’s fit into one of the following categories:
- Efficiency & Effectiveness. Traditionally these metrics referred to the conventional measurement of development capacity and operational capabilities; for instance, looking at server to admin ratios. However, more recently, there’s been a shift in focus to incorporate more customer-centric ratios as well, such as FTEs to customers.
- Velocity. Accelerating application delivery is one of the key drivers of successful DevOps. Tracking velocity shows whether you are improving application delivery by underscoring IT’s responsiveness to increasing business demand. These metrics will indicate how well the IT organization has been able to embrace automation and eliminate manual or isolated tasks in order to increase their time to market significantly.
- Quality. According to Puppet’s 2017 State of DevOps report, the pressure to deploy faster and more frequently causes lower IT performers to pay “insufficient attention to building in quality.” On the other hand, high performers know that they don’t have to sacrifice speed for stability. They understand that quality has become a shared responsibility of all players in the software delivery lifecycle and that quality should be built into the software earlier in the development process.
- Culture, Collaboration, and Sharing. The importance of communication, collaboration and integration between development and operations can’t be stressed enough when it comes to DevOps. Just having the right processes and technology alone isn’t good enough. Incorporating metrics focused on the people aspect of the organization provides a good barometer of acceptance or resistance to DevOps. This shift in approach is critical to adopting an agile, continuous delivery and deployment process because it requires Operations to get involved earlier in the lifecycle and Dev to focus more on application support.
Five KPIs to Measure DevOps Success
- Frequency of deployment
Boosting deployment frequency has been a powerful motivator for shifts in development practices. The ability to make code changes quickly and easily is provides a key competitive advantage for any company that needs to deliver new features quickly to customers and respond to their changing needs.
Did you know that the highest-performing IT organizations deploy more applications more frequently, such as multiple deployments per day compared to between once a month and once every six months for lower performing IT organizations? More frequent deployments reduce the probability of disruption resulting from more complex and infrequent deployments. - Deployment Speed
This measures how long it takes for a single deployment to go from commit to code that’s successfully running in production. While it takes high performing organizations less than one hour, it can take between one and six months for low IT performing organizations. Low performers are likely not automating nor standardizing their job definitions leading to wasted time in production, hard to fix errors and unplanned work, all because of poor communication and manual intervention.
High performers spend 50% less time on unnecessary rework than low and medium performers. This allows them to dramatically increase revenue by using that extra time to develop more value-added services. - Failure Rates
It’s great to deploy more frequently and quickly, but if changes fail just as frequently, you’ve gained nothing. The average cost of critical application failure is between $500K to $1M per hour and failed deployments can take services down, resulting in lost revenue and frustrated customers. - Mean Time to Recovery (MTTR)
Mean Time to Recovery is an important performance metric that refers to the average time that a device takes to recover from any failure. MTTR underscores key business outcomes that directly relate to customer experience, customer acquisition, and retention. Here are some important questions to answer when it comes to measuring MTTR:
- Were the users able to access your IT services?
- Did the customers experience application errors?
- Did users abandon the application before completion?
- What is your customer satisfaction rating?
Once DevOps has a better grip on where its capabilities stand today it can focus on shortening recovery time. The goal is to improve MTTR dramatically in order to reduce costs and accelerate application delivery.
- Measuring Culture
As mentioned above, focusing on culture must be an intrinsic part of any DevOps process. What are the key components in fostering a collaborative environment between Dev and Ops? To thrive within a DevOps ecosystem, the team must be supported by:
- A culture that encourages innovation that focuses on integrating lean principles and shorter implementation cycles.
- A structure that removes organizational silos thus empowering strategic alignment and reducing conflict.
- A culture of high mutual trust. This is a key requirement to ensure that Dev and Ops can accomplish incredible things – together as one cohesive DevOps effort. One way to do this is to designate a “Transformational Leader” who has the characteristics to contribute to such progressive momentum.
When it comes to measuring these outcomes, you can look at the following metrics: employee morale, retention rates, and responsiveness to change within the organization.
You can read about BMC’s new DevOps Capabilities here
Summary:
How to Measure DevOps for Success
You’ve integrated DevOps to help drive your company’s digital transformation. But how do you track the success of your DevOps initiatives and communicate these objectives in a meaningful way to your organization? This can a tricky process because, by design, DevOps isn’t a formal framework and provides limited guidance. The key to success is to create metrics that tie DevOps processes to real business outcomes. Best Practices for Choosing the Right KPIs for Your Organization. Tie DevOps Metrics to Business Outcomes. Getting Buy-In from All Stakeholders. Five KPIs to Measure DevOps Success. Frequency of deployment. Deployment Speed. Failure Rates. Measuring Culture.
