The metrics enigma - 2

Second part of guest post from Mauro Bagnato about organizational metrics. Enjoy!

In my last post The metrics enigma I tried to look at the metrics from a different perspective, starting from the method proposed by D. Hubbard in his fantastic book How to measure anything. In order to turn these insights into practice, I arranged a four-hours workshop with the whole leadership team. Main goal was to share knowledge and to come up with a bunch of brand new organization metrics.

This “clarification workshop” was arranged in the following way:

Step 1. Learning&sharing. 1 hour.
Goal of the first timeslot was to introduce a new way of looking at metrics and share insights and reflections. Definition, purpose, risks and incentives were on the agenda.

Step 2. Understanding why. 30 minutes
The second timeslot was dedicated to the WHY. We tried to answer the question: “Why do we need metrics?” After a very interesting and intense brainstorming session, the leadership team came up with this answer: “We need metrics because we want to understand if we’re improving”. Good start, clarifying the purpose helps tracing the direction to get the metrics we really need.

Step 3. Undestanding what. 30 minutes
The third timeslot was focused on the OBJECT OF MEASUREMENT. Now the question was:” What are we going to measure?” Well, starting from the why found in the previous step, we just needed to measure the improvement. Unfortunately the word improvement was definitely something too vague and hard to measure, unless we had been able to turn it into something tangible. “What does improvement mean for us?” or better “Let’s suppose to be able to clone our organization and instill in the cloned one a massive dose of improvement, while holding the amount constant in the original one. What change do you imagine you would actually observe in the cloned organization?”. Those questions triggered an interesting discussion ended up with the statement: if we want to understand whether our organization is improving or not, we need to observe certain dimensions: delivered value, external perception, learning, innovation and climate. Now we had five objects of measurement even if they still needed further clarification to be measured.

Step 4. Metrics definition. Two one-hour iterations
The fourth timeslot was focused on turning each dimension into metric. In order to facilitate the discussion and to work on more items in parallel, I split the leadership team into two groups and asked them to go through a four phases discussion. The following picture describes the proposed logical path.

If we are able to turn each dimension into something really tangible, it should be easy to directly derive the metric…basically this was the main idea. For example, if innovation meant generating ideas only, we could assume that measuring the number of ideas would tell us how innovative we are. Thus we must understand the meaning of innovation first. In other words the further clarification phase (see the picture) means keep repeating the actions in the step 3 till the vague concept of innovation becomes tangible.

A deep clarification of the object of measurement should lead us to the metrics but… how can we check if those metrics are really good? Did we consider all the aspects? Did we miss something? Next “control” phases in the path help answering those questions. Before going deeper into those phases, we need to make some reflections first. Let’s start from the assumption that measuring means getting information and that getting information requires a certain investment. The amount of this investment should be tied to the importance of the information. Since the reason why we collect information is that we want to reduce the uncertainty related to a certain decision, then the importance of the information is directly related to the relevance of the decision we need to take. (in the financial context reducing the uncertainty related to an investment decision could save a lot of money). Starting from these considerations, the phase 3 requires to answer the question: “which decision does this measure inform?” or in a different way “do we really need this measure?”.

The last “control” phase comes at the end of the chain. Setting a metric in itself inevitably influences people behavior in ways that may or may not be the intended outcome. Here is an example of possible side effects or incentives a metric may produce. Let’s assume that the metric used to monitor a help-desk performance is the number of handled calls. This metric may generate the side effect that help-desk workers are encouraged to conclude the call as soon as possible without solving the problem. In this case the real outcome (side effect) of the metric is far from the expected benefit. On the other hand if the metric was the customer feedback collected at the end of the call, then help-desk workers would be encouraged to give the best service possible to their customers.

It was interesting to see that only few of the metrics found at the beginning of the path, passed the final control phases!

The outcome of this intense, tiring, but interesting workshop was a bunch of metrics related to two dimensions only. Yes…we didn’t manage to complete all the work, but we learnt together how to tackle the metric problem in a different and structured way. We found out how to solve the metrics enigma!
It was a collaborative work of the whole leadership team that gained the fundamental result of building a shared vision around the metrics.

The metrics enigma

This is the third post I host from Mauro Bagnato: my friend, if you keep producing at this rate in 2014, I will be gladly forced to open a special column for you!

A hot issue is touched this time: organizational metrics. The ones among you who are constantly reading (R)Evolutionary Agility know that I wrote already an article about this subject and I have plans to write more about it. 

Whether you celebrate Christmas or not, I wish you all restful and peaceful season’s holidays!

I’ve been struggling against organizational metrics for almost two years.

I must say that it’s a bit frustrating to look for a reasonable answer to the question: “which are the right metrics that an Agile organization should have?” and get the feeling to be very close to the goal, but without being able to hit the target!

These days I recalled a famous quote from Charles Kettering saying: “A problem well stated is a problem half solved”. It made me reflecting. 

Couldn’t it be that it’s so hard to find the solution of the metric enigma just because of the wrong question? Got convinced about it, I decided to start from scratch and to ask myself: Why do we need metrics?

Here you are the answers which came up in my mind.

We might need metrics because we want to:

•  find the evidence that the Agile transformation is worth the investment
• understand if we’re improving
• understand if we’re getting closer to our vision
• show our stakeholders we’re a fantastic organization
•  etc. (I could go on…)

This approach looked promising. Clarifying the purpose traces the direction and helps identifying the metric we really need, the ones linked to our goal. So far so good I’d say!

And now? How to move further? How to turn this goal into metrics? How to measure something so vague and undefined like stakeholders’ perception, improvement, ROI of the agile transformation etc.? In other words, how to measure intangibles?

The answer, or better, the method to measure this kind of things is described in the book “How to measure anything” from Douglas Hubbard. This very interesting book proposes a three steps way to measure intangibles:

1. Concept of measurement. What does measurement means?

Usually measurement is thought as the process aiming at quantifying something or giving something an exact value. This common conception implies that measurement means reaching a sort of certainty (numbers or values are expression of certainty).

This idea fits very well with the measurement of lengths, widths, etc. but when it comes to the intangibles (like happiness, empowerment or motivation) it doesn’t help a lot.

Scientists use a different definition: A measurement is a quantitatively expressed reduction of uncertainty based on one or more observations.

This definition introduces two interesting points:

·     Measurement is no more connected to the idea of certainty but is defined as reduction of uncertainty.

·     It’s possible to measure anything (intangible or not) observing the impacts, the effects, the results.     

2. Object of measurement. What do we have to measure?

Even using the new definition, something could still seem immeasurable simply because the object of measurement is not clear or is not clear enough. Very often clarifying the meaning of what we want to measure makes the problem much easier. What does it mean? That’s the fundamental question to answer.

How to measure happiness? One million dollar if someone is able to find the answer!
Let’s try to change perspective and decompose the problem by answering the question: What does happiness mean? For me happiness means having enough time to dedicate to my family and to my hobbies. Measuring the time spent that way gives me an indication of how happy I am!

I found something measurable to observe reducing the uncertainty related to the measurement of happiness!

It might happen that the answer to the fundamental question is still something intangible. In this case the trick is to repeat the question till the answer is become something measurable.

D. Hubbard suggests also a different way to clarify the object looking at the effect or the consequences: Let’s suppose to be able to clone someone and to be able to instill a massive dose of happiness in the cloned guy, holding the amount constant in the original one. What do you imagine you would actually observe that would change for the cloned guy?

Whatever the method, once the object of measurement is clear, it’s possible to find the way to measure it.    

3. Methods of measurement. How do we measure?

I’ll not dig too much into this point but D. Hubbard suggests to make use of whatever methods of measurement based on:

·     Sampling. It simplifies a lot the process of measurement. We need much less data than we think to get a valid measurement.

·       Experiment. Anyone can develop an intuitive method for measurement and the best approach is to try it and learn from it.

Let’s come back to the starting problem and see if the method works. Assuming that we need metrics because we want to know if we are improving, next step is understanding what improvement means. Assuming that improvement means delivering more value, now we need to understand what delivering more value means. Assuming that delivering more value means delivering what customers really need, what happens when customer have what they need? Assuming that when customers have what they need, they simply use it, we could start observing how many customers really use what we deliver.
We got the object of measurement! Now the point is how to measure it? How many data are we going to collect? Making a sampling of the customers could simplify a lot the measurement itself without compromising the results (see “The rule of five” proposed by D. Hubbard).

It seems we got the point but…could we get there without wasting our time with all this stuff?

Before answering the question, we should consider that the outcome of this method is not only metric itself, but mainly the path to get there. Questioning the need (what we want to achieve), decomposing it reflecting on its meaning/effect and then finding what to measure, builds a sort of picture communicating:

• what we want to achieve
• how we want to achieve it
• What we consider important, what we care of.

Then metric becomes the logic consequence of this journey.

In the next post I’ll tell you how I applied this method in a 4-hours workshop for defining the metric of the organization I work with.