Blog

Agile Metrics

September 20, 2020

Per the Agile Manifesto “Working software is the primary measure of progress” and in order to measure progress we need metrics.

As an agile coach and practitioner, I respect the relevance of metrics at all levels from teams to programs to portfolios. However please be do not fall victim to the “street light effect” which means we tend to look for data where it can be easily found but does not have much value to the outcome or impact like number of lines of code, team velocity etc. that are most of the time mere numbers that does not help with real outcome .

It is important to note that metrics when evaluated in isolation can cause false positives or negatives and result in missing the signals due to the noise being tracked. Consider metrics as indicators of problems that need root cause analysis to find the root causes.

With that preface, I am going to share few key metrics that really matter and also the concept of correlating metrics where you always correlate one metric with another one to make sure they tell the same story ,so that we see holistic progress at all levels from a Epic to Feature to Story. This correlation also helps to detect any gaming of the individual metrics that may be attempted.

We will cover basic metrics and not so common metrics and share tips on correlation of metrics.

Lead time:

This is the time taken for any request fulfillment start to finish and typically you look at how long it takes any product vision to get to the customer via value streams. For software project management you can say this is the time taken for any project proposal to go thru the review process and get funded and executed and working software released to the end-user.

Cycle Time:

Cycle time refers to the time it takes to start and end a specific step in a series of steps in the value stream activities of request fulfillment. In software projects this can be a requirement phase, development phase or testing phase.

Lead time includes the waiting time the request is in the queue waiting for available capacity to deliver the request.

The goal is to minimize the lead time and cycle time to accelerate time to market.

Correlation metrics Tip:

Reducing cycle time should result in reduced Lead time as well and if not the cycle time data may be gamed.

Burndown Chart: These are one of the easiest to use and very helpful tools to track progress over time on work items. This is a 2-dimensional chart with work items like stories, features, hours on the Y axis and timeline such as sprints, releases on the X-axis. A good burndown chart tends to be nice and linear trending down from left to right over a period of time. This chart helps to track progress at any level of granularity as desired from days to months

Cumulative Flow Diagram: this is a visual radiator so sharing a picture

The CFD is a great way to visualize work and allows quick check on where the bottlenecks are by showing the lead time and Work in Progress (WIP).

This is great for showing status reports to audience to quickly bring visibility to progress of work in various stages like backlog, in progress, done etc. due to the visual nature

All the metrics mentioned so far as available as built-in gadgets in most ALM tools like Rally and Jira.

Now let’s look at some derived metrics that helps an agile team

Correlation Metrics Tip:

Teams with good CFD and Burndown charts should also have higher levels of predictability measures

Number of Deployments

This is an important metric but don’t get influenced by popular facts like Amazon deploys 5000 times a day! Identify what is the realistic number of deployments relevant to your organization and nature of work.

Overall Quality

This is measured by many things like production defects after a release, performance of the system under load, availability of the system to users 24 by 7 etc.

Predictability – Can be applied at team and program level

Predictability is an indicator of the sustainable quality of the team to deliver to expectations and this is a simple measure of “Say “vs “Done “!

Examples:

Features planned vs Features delivered in a release, Stories committed vs Stories accepted during a sprint.

This can be created in plain excel sheets or some times ALM tools comes with built-in gadgets.

Correlation Metrics Tip:

Teams with good predictability should also have good clean burn down charts and optimized cycle and lead times

Employee Engagement

The ways to measure this is by looking at the employee churn, employee satisfaction surveys, daily attendance, 360 feedback reports on managers, self-organizing teams who are cross-functional in skill set.

Correlation Metrics Tip:

If the employee engagement indicators are positive and high, the cycle time and lead time and quality of work should be high as well for the measuring period.

Customer Satisfaction

NPS (Net Promotor score) is an easy way to gauge the customer feedback as these are open ended and ask the customer “How is your customer experience with us?”

Metrics for start-ups

Most of the metrics we shared so far are relevant to Start-ups and established organizations alike but some metrics are more relevant and important for start-ups. There are no baseline data for start-ups so typical metrics wont help in assessing success or failure of the products.

In his book, “The Lean Startup”, Eric Ries defines innovation accounting as,

“…a way of evaluating progress when all the metrics typically used in an established company (revenue, customers, ROI, market share) are effectively zero.”

One way to measure innovation accounting will be to utilize the Pirate metric or AARRR metric popularized by Dave McClure to help with validated learning based on market response to a new product.

Pirate metrics or AARRR metric:
- When your product is new to market measure these key aspects AARRR
- Acquisition: Customers coming to finding and coming to the product.
- Activation: Customers experiencing the core value of the product.
- Retention: Customers coming back to use the product again.
- Referral: Customers sharing the product with others.
- Revenue: Monetizing customers

I can go on with more metric as there are many more but at this time what has been shared is quite comprehensive for most teams.

Please let us know if we can help you with your challenges with measuring progress or business agility.

About the Author: Ram Kollengode is an agile coach and transformational consultant with more than 30 years of experience across a variety of sectors like Banking, Insurance, Telcom etc. Ram is passionate about agile mindset and enjoys time mentoring and training leaders on embracing agility

Reference

https://www.product-frameworks.com/AARRR-Metrics.html