Reasoning before measuring
I often got this question - what do you measure while adopting Agile?
The conversation usually went like this:
[me] What is about adopting Agile?
[him] ... doing Scrum.
[me] What effect do you expect after adopting Agile?
[him] higher efficiency.
[me] How would you define higher efficiency?
[him] ... finishing more tasks.
[me] Why would adopting Agile lead to higher efficiency?
[him] Agile means higher efficiency, doesn't it?!
The thinking is fuzzy, to say the least. I suggest to focus on reasoning before measuring.
Let's take Specification by Example (SbE) as one example of agile practices. What do you measure while adopting SbE?
Practice and Purpose
Firstly, we learn about SbE by asking 1) what is SbE; and 2) why SbE.
A good source of learning is of course the book "Specification by Example".
1) What is SbE?
"Specification by Example is a set of process patterns that facilitate change in software products to ensure the right product is delivered effectively."
Suppose that among those process patterns, the below two are our main focus of initial adoption. We dive deep on them.
- Specifying collaboratively
"Instead of relying on a single person to get the specifications right in isolation, successful delivery teams collaborate with the business users to specify the solution."
- Illustrating using examples
"The team works with the business users to identify key examples that describe the expected functionality. During this process, developers and testers often suggest additional examples that illustrate edge cases or address areas of the system that are particularly problematic."
2) Why SbE?
There are various benefits from the whole set of SbE process patterns. Let's focus on the benefits from "Specifying collaboratively" and "Illustrating using examples". In other words, let's understand the purposes of these two specific practices.
"Specification by Example helps to improve the quality of software products, significantly reduces rework, and enables teams to better align analysis, development, and testing activities."
There are three areas of benefits.
- Higher product quality
"Specification by Example improves collaboration between delivery team members, facilitates better engagement with business users, and provides clear objective targets for delivery - leading to big improvement in product quality."
- Less rework
"Specification by Example helps teams establish a collaborative specification process that lowers problems in the middle of an iteration. Most teams have significantly reduced or completely eliminated rework that occurred as a result of misunderstood requirements or neglected customer expectations."
- Better work alignment
"Specification by Example enables tams to clearly define a target that's universally understood and objectively measured. As a result, many teams find that their analysis, development, and testing activities became better aligned."
The above reasoning could be summarized as below.
Only then do we go to the measuring part.
"Practice has purpose, and we measure both the practice and the purpose." I learned this in one private workshop by David Hussman for Odd-e in 2013 - one of the most memorable experiences with "the dude".
We may come up with the following measures.
- For the practice
- %stories done with SbE (measuring the extent of doing SbE)
- %stories with cross-functional conversations (measuring the extent of specifying collaboratively)
- %stories specified with examples (measuring the extent of illustrating using examples)
- For the purpose
- #defects due to misunderstood/neglected requirements after done (measuring product quality)
- #defects due to misunderstood/neglected requirements during the sprint (measuring rework)
- %stories reaching completion at the end of the sprint (measuring work alignment)
Causal Diagram
In the "The Book of Why", it introduces causal diagram to move from the correlation to the causation. Causal diagram is similar to causal loop diagram in systems thinking, as they both are based on the causation.
After understanding the practice and the purpose, we reason further for finer-grained causation by expanding with the intermediates. Why doing SbE leading to higher product quality, less rework and better work alignment?
The below causal diagram illustrates one reasoning.
Then, we measure the intermediates, so as to learn the whole chain of causation via validation or invalidation. Note that some variables are harder to measure than others. For example, "amount of rework" is harder to measure than "internal defects". We don't have to measure all the variables. With a few intermediate ones, we build our confidence in this causation.
We may end up with measuring:
- %stories of doing SbE (i.e. specifying collaboratively and illustrating using examples)
- #internal defects during the sprint
- #external defects after done
- %story completion at the end of the sprint
Conclusion
We reason about the cause and effect, before getting to the measure. On the other hand, measuring also improves our reasoning in two senses - 1) thinking about how to measure increases the rigor in the reasoning; 2) the measuring result validates or invalidates the reasoning.
Notice that measuring here is for learning and improving. See my previous blog "Two Guidelines for Metrics" for further reference.
