Two Guidelines for Metrics
During the recent Advanced ScrumMaster course, two guidelines for performance metrics emerged.
Continuous Improvement over Performance Evaluation
"Measuring and Managing Performance in Organizations" by Robert D. Austin is a great source for the topic of metrics. He made the distinction in terms of the purpose of using the metrics, either as motivational or as informational.
Motivational purpose ties to Performance Evaluation, while informational purpose ties to Continuous Improvement. Metrics could be the same, but used with different purposes. Take the example of unit test coverage. Management uses this metric as one KPI for team, the higher, the better. This is motivational. Team itself may find that coverage provides meaningful insights to guide their improvement in unit testing, thus, they decide to measure it. This is informational.
Austin claims that all metrics with motivational purpose inevitably lead to measurement dysfunction to some extent. This is what we see in reality that some teams don't add any check/assert in their unit test purely for the sake of achieving measured coverage target.
This leads to the first guidance: Continuous Improvement over Performance Evaluation
Collaboration over Accountability
When you cannot yet avoid motivational use of metrics, the technique of measuring up helps. Traditionally, we measure based on the span of control. By measuring up, we measure based on the span of influence.
Take a couple of examples. Traditionally, we measure developers based on the development output, e.g. lines of code; we measure testers based on the testing output, e.g. number of bugs found. By measuring up, we measure the collective output from the cross-functional team. This applies at broader scale too. Suppose that multiple teams work on different features and those features together form a complete customer scenario. Traditionally, we measure single team's output, e.g. delivered feature. By measuring up, we measure the collective output by multiple teams, e.g. delivered customer scenario.
Through measuring up, we promote the bigger common goal, thus more collaboration. Would this sacrifice clear accountability? Probably yes, but does it matter?
Yves Morieux's Ted Talk "How too many rules at work keep you from getting things done" makes a great point on this question.
In relay race, who's to blame (sorry, who's accountable:) when the baton drops? It's not clear, so, in order to have more clear accountability, you may introduce a third person, whose sole responsibility is to take the baton from one and pass that to the other, thus, he would be accountable when it fails. This is a system designed for failure, failure with clear accountability. However, would that help win the race?
Moreover, you may try measuring one-level up, thus, still take accountability into consideration to some extent.
This leads to the second guidance: Collaboration over Accountability
So often, people ask what metrics we shall use in Agile context. In my humble opinion, these two guidances are more critical for your success in effectively using metrics.
Note: special thanks to my co-trainer Sun Ni for great insights.
When work and skill mismatch
You have more work in one area than what the people in that area can do. You also have available people in the organization, but their skill does not match the work.
Will this happen? Believe me, it will always happen at some point, either at team level or at individual level, or both. This is simply due to dynamic work vs. "static" skill (really? maybe as snapshot, and hopefully not in the life span). When that happens, there are a few strategies to deal with it.
If you are working in matrix and responsible for one project, the obvious strategy is to grab more people (or resource, are people resources?) with the right skill and allocate them to the area having more work. What if other projects suffer even more? It's none of your business.
If you fail to get more people, another strategy is that you anyway push this through the developers working in your project. This inevitably leads to the application of developer's secret toolbox, which sacrifices quality and gradually leads to more and more legacy code, and eventually slows down the development, a lot.
If you take a bigger and longer-term view, you see that "push it" strategy creates more problems than it solves. Another strategy is simply accept the fact, pull the work based on the capacity, with the acknowledgement that the output is constrained by the bottleneck. You either extend the schedule or reduce the scope, or a bit of both.
This is wise in the sense that it respects the system, thus the system would not backfire. However, the speed or the value of delivery suffers and you may not survive.
This strategy is quite logical, as bottleneck area constrains the output, you grow the capacity in that area. It does solve the problem of delivery. However, as bottleneck area moves, it leads to staffing based on the worst scenario.
This is not lean. As it is practically hard to shrink in size, some teams and individuals will inevitably work on low-value stuff and create wastes. And, the more important waste is the underutilization of the people potential. People differ greatly from resource because people can learn. By limiting people only in their familiar areas does not show the respect for people, which is one of the two pillars in lean thinking.
Then it comes to the most sensible strategy. You treat mismatch as an opportunity to learn and expand skills. This does not mean that specialization is bad. You still utilize the specialization for efficiency, meanwhile take advantage of mismatch for people to learn new areas. This is lean. Long-lived feature team adopts this strategy, and LeSS explicitly promotes feature team structure.
- feature team primer
- more on feature team chapter in "Scaling Lean & Agile Development: Thinking and Organizational Tools for Large-Scale Scrum" book
What's in Product backlog?
What's appropriate to put into Product backlog? In order to answer this question, we first look at different views on Scrum.
Two views of Scrum
One is viewing Scrum as planning framework, then, PBI (Product Backlog Item) is the unit for work planning. The other is viewing Scrum as product development framework, then, PBI is the unit for product inspection and adaptation.
The different views lead to different thoughts about what are appropriate items in Product backlog.
What's in Product backlog?
With the view of Scrum as planning framework, everything that needs to be done can be put into Product backlog and planned in Sprints. This still makes sense until things not directly related to product come in, and some ought to be tasks in Sprint backlog. Some Product Owners mis-use Product backlog as a tool for execution. When ex-project manager takes the role of Product Owner, as they used to be execution oriented, this actually becomes a common pitfall. Over time, there is tendency to incorporate "how" items into Product backlog.
With the view of Scrum as product development framework, only things that help product inspection and adaptation are put into Product backlog. How valuable is it to inspect so as to adapt towards product vision and goals? Learning about the real customer needs is a valid PBI, as it helps move closer to successful product, assuming that it is validated learning. Prototypes and spikes are valid PBIs, as we build the product iteratively. Tasks are not, as they do not necessarily build working software and/or validated learning. Product Owners and great product managers use Product backlog as a tool for empirically developing a successful product. My colleague used Sprint goals to drive a new product development. A series of Sprint goals are actually items in Product backlog. Over time, there is tendency to incorporate "why" items into Product backlog.
Product backlog at scale
When we look at two most popular scaling framework, SAFe and LeSS. They take different views for Scrum, thus, their guides on Product backlog are also different.
In SAFe, there's no Product backlog, but team backlog, program backlog and portfolio backlog. At team level, it's defined as pretty much standard Scrum, while the use of team backlog indicates that it views Scrum as planning framework. As the team may or may not be feature team, items in team backlog may or may not be product features. At team level, the focus is mainly on execution of planned work.
In LeSS, it keeps product focus and scales only teams. LeSS views Scrum as product development framework, thus Product backlog is still product backlog. Any PBI by any team can be inspected then adapted at product level. As LeSS requires that the majority of teams are feature teams, the focus is mainly on product inspection and adaptation on Sprint basis.
To me, Product backlog is the backlog for product. It's critical to understand what our product is, then we can use Product backlog as a tool to iteratively and incrementally build it.
Split work and people
How to split work (from big to small) and how to split people (into multiple teams) are two essential and very related topics in the context of scaling Agile.
In these two dimensions (work and people), there are two common splitting strategies, through component or through feature. This leads to the below diagram.
Let's dive deep into each quadrant.
- Component work by Component team
This is the traditional way of working, and strongly related to waterfall development. Feature is split into components via design by architecture group, then developed and (component) tested by component teams, eventually integrated and (system) tested by system testing group. The problems with this strategy are handoff waste, prolonged cycle time, delayed feedback, lack of flexibility, etc.
- Feature work by Component team(s)
First split from big feature to small features, then split into component tasks is the key idea behind Agile and helps tremendously on the speed and flexibility in value delivery. When related component teams are able to collaboratively deliver the same small feature at the same time, it is quite ok. However, in practice, those component work around same feature is often out of sync, which delays feedback and value delivery. Moreover, this is a systematic problem, and can not be solved by stronger coordination. The deep assumption behind component team structure is that people are most efficient in working on familiar areas, and learning other areas is costly and slow.
I put SAFe into this quadrant, not because SAFe demands component team structure, but because SAFe implementation tends to retain current organizational structure, while component team structure is prevalent in organizations.
- Feature work by Feature team
Feature team takes feature, and splits into smaller features to deliver. This is great in achieving agility. This systematically addresses the problems with component team. Meanwhile, it has its own challenges such as coordinating in code, maintaining the component integrity, efficient learning, etc. Furthermore, it demands organization (re)design, which usually means more radical change thus is perceived as more risky.
I put LeSS into this quadrant, because LeSS explicitly requires that the majority of teams are feature teams. Organization design is vital in LeSS adoption.
- Component work by Feature team
This is rarely seen in practice. If you already have feature team, it makes no sense to feed them with component work. Rather feature team will take feature as input and split them into component tasks as part of their development.
Split work via feature dimension is critical element in Agile adoption, while split teams via feature dimension is critical element in LeSS adoption. This is the place we shall strive for.
Contract game is the traditional way that Business and R&D work together. At the beginning of the project, Business asks R&D to commit a big batch of work and holds R&D accountable for execution. The mindset behind is "plan then execute".
Clearly this is not Agile way. One of Agile values is "Customer collaboration over Contract negotiation", which exactly addresses the dysfunction of contract game. Agile is about adaptive planning, and Scrum is about empirical process. The mindset behind is "guided by goal, inspect & adapt".
- Is Release planning predicative or adaptive?
Release planning can be a beneficial practice, as it creates the initial understanding about the work. However, in practice, it is often mistaken as upfront and predictive planning. It is even used to provide the input for contract game. Once you see that work is split into every sprint in the release with great details, the focus is primarily on contract negotiation. With the mindset of contract game, you may see that business people only join Release planning, but not the inspection and adaptation on sprint basis.
So, in Scrum, Release planning creates the initial product backlog, and you may regard that as the first backlog refinement, which is from none to the initial. Then, every backlog refinement updates the release plan and new requirements emerge, thus, Release planning is continuous.
Backlog Refinement = Continuous Release Planning
The key to break contract game in this level is that Business people and PO are working with R&D on sprint basis, and making decisions regarding ROI every sprint.
- How about Sprint planning?
If more people have realized that the release should be managed empirically, the sprint is still largely managed predictively - we plan in Sprint planning, then, we execute the sprint. This is a misconception, because if it is a predictive process, then, what is the point of having Daily Scrum, whose purpose is to manage it empirically with daily inspection and adaption? With the mindset of contract game, you may see that the sprint commitment is regarded as certainty, and PO disappears after Sprint planning and only sees team again in the Sprint review.
So, in Scrum, Sprint planning creates the initial sprint plan (aka. sprint backlog), and we don't sign up for all tasks in Sprint planning, only those you start doing next day. Then, every Daily Scrum updates the sprint plan and new tasks emerge, thus, Sprint planning is continuous too.
Daily Scrum = Continuous Sprint Planning
The key to break contract game in this level is that PO collaborates with team in the sprint through just-in-time review, defining sprint goal, re-negotiating the scope when necessary, occasionally joining Daily Scrum, and just talking to each other.
While PO manages release and team manages sprint to large extent, contract game is harmful at both levels. Are you in the mode of contract game?
Specialization vs. Responsibility
Specialization is about being good at something, while responsibility is about having a duty to deal with something. While they are related, it may not be good idea to couple them together.
Component specialization and Feature responsibility
Conway's law states, "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations"
Once you setup team with certain component responsibility, they specialize on the component naturally. Then, this becomes the source of inflexibility, when specialization is used to justify tying those teams with those components. Evolving architecture becomes harder when it is coupled with organizational structure.
The solution is feature teams with collective code ownership. The responsibility is on feature, while component responsibility is shared, do we still have component specialization? Most likely yes, at least at individual level. Some people are more knowledgable thus better at doing certain component work. Generalizing specialists starts with some speciality, then expands to other specialities. For example, you are the main developer on component A, and you expand to work on component B and develop specialty there over time. This may apply to team level too, since some features may touch certain part of system and the related components more often than other features. Specialization just happens. Since the responsibility is on feature, thus, the component specialization does not limit us from picking the most valuable feature items.
In short, have feature responsibility, and let feature guide component specialization.
Feature specialization and Product responsibility
The same dynamic happens with feature specialization. When the team has feature responsibility, e.g. it is a "Payment" team, the team specializes on the relevant domain. Then, this becomes the source of inflexibility, when specialization is used to justify tying those teams with those features. Evolving product becomes harder when it is coupled with organizational structure.
The solution is feature teams with common product backlog. Those teams have product responsibility, and they don't have special feature area responsibility such as "Payment". Do they still have feature specialization? Most likely yes. Given the team developed features in this area, when another high-value feature comes to same area, the team likely picks this one. Over time, they specialize on this feature area. This is even a good thing since it helps develop the deep knowledge and skills, but we don't want to label them as "payment" team and define their responsibility around this feature area. When that happens, we start to select features because we have suitable team available, rather than customer value driven. Therefore, every team is product team without having name tied to feature area. Instead of having "Payment" team, we have "Gryffindor" team. For specialization, we may even want to track and take advantage of it when possible.
In short, have product responsibility, and let product guide feature specialization.
If you work on large-scale development and adopt LeSS, you should have heard of Requirement area. Do we want to introduce areas with clear responsibility on product domains, or areas simply as groups of teams and let specialization on product domains happen and evolve? There is no simple answer. The bottom line is, Requirement area is dynamic. When you give Requirement area clear responsibility and associate it with meaningful area name (e.g. Security), it may help build the identity thus accelerate the specialization. On the other hand, this may also lead to Requirement areas standing still forever.
Specialization and Responsibility are two different things, and we shall not confuse them. Specialization happens, and you may want to track and take advantage of it, while narrow responsibility, defined around specialization and often labelled in name, decreases flexibility and leads to local optimization.
Back to fixed scope
I'd like to revisit the rationale behind moving fixed scope to fixed time in Agile development. By understanding what is essential, we may get back to the thinking of fixed scope.
Fixed scope in traditional development
In traditional development, we often start by fixing the scope (of the release), then work on how much time and how many people we need. The number of people is the main cost driver in software product development.
Fixed time in Agile development
In Agile development, we often start with fixed time and fixed cost, then work on how much scope we can deliver within those constraints. Fixed time is implemented as iteration and is also called timebox. When the team is stable, we have pretty much fixed cost. Release consists of multiple iterations, and the number of iterations may or may not be fixed.
The rationale behind moving from fixed scope to fixed time:
- Scope often has the most flexibility, particularly when you look into details. For complex product development, we learn the right scope over time, while fixed scope reduces flexibility and makes it difficult to respond to change.
- Increasing the number of people, although increasing cost, may not increase the speed. This is best illustrated by Brooks's law - adding manpower to a late software project makes it later.
- Time has less flexibility due to the growing need for short time to market and even occasions when the time delay is impossible (e.g. Christmas). Timebox helps prioritize and focus, as well as build development discipline.
Back to fixed scope
If you look at the rationale, it assumes that the fixed scope is big. When it is small and minimum, the problems with fixed scope disappear. Therefore, the key problem is big fixed scope. Timebox is one approach to reduce fixed scope. Another approach is to limit WIP directly as done in Kanban. Limiting WIP helps prioritize and focus too. The remaining advantage from timebox may be the support for building development discipline.
With further scope optimization, our focus moves towards identifying the meaningful minimum. It is MMF (Minimum Marketable Feature). In terms of story mapping, it is the minimum slice rather than a single story. The time to deliver MMF is not fixed, but usually short due to the minimum scope. Once we identify MMF, we develop it and release it, with the discipline of continuous delivery. We are back to the thinking of fixed scope, but small fixed scope.
The thinking of fixed scope is not the problem. The problem is that fixed scope is too big. We solve it by reducing it. We may apply timebox, which is the way behind moving fixed scope to fixed time. We may also limit WIP directly, which is the Kanban way. Eventually, if we identify one MMF each time and make continuous delivery, we are back to fixed scope, but very small.
Tighten or loosen roles?
In my recent CSPO course, I got a question about when we should focus on the whole team (team, PO and SM), and when we should highlight different roles. Some people are somewhat confused because on one hand we define different roles and responsibilities, on the other hand we talk about the whole team. How strictly should those roles be defined and responsibilities be respected? Should we tighten or loosen roles?
MOI and Agile value
That question made me think and reminded me of MOI (Motivation, Organization, Information) mode from Jerry Weinberg. In MOI, neither too little nor too much leads to effectiveness. The most effectiveness is achieved when you strike a balance. This applies onto Motivation, Organization and Information, and organization is the dimension related to the question.
It implies that it depends on your context. You observe what makes your situation less effective, is it due to too loose or too tight organization? Roles belong to organization. Your effectiveness is low. When it is caused by lack of organization, you increase it by for example highlighting roles. When it is caused by too much organization, you decrease it by for example focusing on the whole team.
The first Agile value is "Individuals and interactions over processes and tools", roles are part of processes. If what you do depends on the situation, does it conflict with that we value more on Individuals and interactions, which implies loosening roles? No, it does not. In general, we lean towards loosening roles and favor individuals and interactions, while in your specific situation, it is still possible that you lean too far away from organization, thus you actually benefit from having a bit more processes by for example tightening roles a bit.
How do we tighten or loosen roles? CDE (Container, Difference, Exchange) from Glenda Eoyang provides useful insights, in particular, we influence the organization by changing container. Expanding container leads to more loosen roles and more room for self-organization. This includes practices such as collective code ownership, PO and team collaborating on requirement clarification, creating the whole product team, etc.
LeSS is natural
I realize that LeSS is a natural way to scale. And let me illustrate.
LeSS is still Scrum
LeSS provides two different large-scale frameworks: LeSS for 2-8 teams and LeSS huge for 8+ teams. We focus on LeSS here. LeSS is trying to reach the same purpose as one-team Scrum while staying within the constraints of the standard Scrum rules. So, let's examine what's essential for one-team Scrum.
One summary could be "deliver value on sprint basis". This is achieved by one Product Backlog (PB) by one Product Owner (PO), representing value; and one Team self-organizing to deliver at the end of each Sprint, getting Done and leading to one Potentially Shippable Product Increment (PSPI).
That basically has:
- One PO and one PB
- One Done and one PSPI
- One Sprint
- One Team
What's the challenge when scaling? Team size is too big. We need to split them into multiple teams. Think about 20 people, we may split them into 3 teams. But meanwhile keep the rest as intact as possible.
That leads to:
- One PO and one PB
- One Done and one PSPI
- One Sprint
- Multiple Teams
The design goal for team structure is to enable any Team (after splitting) take items from one PB, get to common Done and become part of common PSPI (integrated with other items done by other Teams). Feature team (the majority of the teams) is thus a natural choice.
This is LeSS way, and it is natural to scale this way. More reference here.
Scale Scrum when we grow
In a recent scaling workshop, an interesting comment was raised. "We pretty much did LeSS even without knowing it." I dug deeper on how it evolved. It turned out that they started with one-team Scrum, but grew to the size that is too big for one-team any more. They tried to introduce minimal change by splitting the big team into several small parallel teams. By "parallel", i mean that they could work on any item in the common backlog and didn't create specialization area to constraint themselves. This worked well for them.
This reminded me of my previous experience. Back to 2005-2006, in my first Scrum project, we started from one team Scrum. After 6 months, it grew into 3 teams. We kept one PO and one Product Backlog. We shared the Sprint rhythm, defined common Done, had joint sprint planning and sprint review to keep the whole product focus. This is in line with LeSS. Again in 2008-2009, we had one team in my department, growing into 3 teams. It used similar approach leading to similar LeSS structure.
If you start with one team Scrum, it is rather natural to scale to LeSS. Moreover, regardless of how big the product eventually becomes, it is almost always wise to start from one team. Therefore, scale to LeSS, not start from LeSS.
How Scrum team benefits from Kanban practices
While discussing some struggles of Scrum team with my friend He Mian during his Kanban course recently, i realize that Scrum team can benefit from many Kanban practices.
There are 6 core practices defined in Kanban method. They are:
- Limit Work-in-progress
- Manage Flow
- Make Policies Explicit
- Implement Feedback Loops
- Improve Collaboratively, Evolve Experimentally
Let's look through them to see how Scrum team or any team doing iterations can benefit from those Kanban practices.
Scrum team usually adopts some form of task boards to help coordinate their work in sprint. The key is to effectively inspect where we are so as to adapt accordingly. Visualization in Kanban goes deeper than usual task board. When the cycle time of your stories is still long (e.g. a week or more), the additional details expose problems earlier and help us adapt faster. Kanban uses more elements when visualizing, such as area, color, shape, number, etc. For example, the impediment could be visualized as attached note with different color on the story. This brings everybody's attention immediately.
Scrum limits WIP indirectly by iteration. It's recommended to work on one story at a time, however, it may not be viable with small size story (e.g. 2-3 days, 2-3 people) and big size team (e.g. 7-9 people). It is good practice to limit WIP further inside sprint, and visualize that by limiting the number of lanes for example. One argument against doing that may be the risks involved in not starting the work, considering that the goal is to complete them all by the end of the sprint. We shall take risk factors into account when prioritizing stories. By limiting WIP, we actually improve the chance of completing them all by the end of sprint.
When managing the progress, reminder of sprint goal is a good step forward, while focusing on flow provides more clear guidance. Anything that prevents flow becomes impediment.
Bottleneck is one common reason that prevents flow. In Kanban, a few ways are suggested to address those both in the short term and in the long term. For example, when testing becomes bottleneck, you may first consider removing any non-bottleneck work from those people who are testing; then, you may consider improving quality of the work flowing there by doing more developer tests; then, you may consider having developers help testing. Scrum team benefits from ideas of flow management.
By measuring cycle time - the time you start working on a story till the time you get the story done, you get insights from control chart and distribution chart on how to improve the flow in the long term.
The focus on managing flow is also reflected in the way that daily standup is done in Kanban. We walk through the board from right to left, story by story. This helps making more effective inspection and adaptation.
The Definition of Done (DoD) is one of the most important policies in Scrum. The Definition of Ready (DoR) gets popular in Scrum community, which is another policy. BTW, in practice those may lead to the wrong focus on handover from one group to another, but those could be and are achieved collaboratively in many contexts.
Many Scrum teams create and evolve their working agreement, which forms a dynamic set of processes and policies. Kanban points out more opportunities to make policies explicit, and they become areas for improvement. This fits well with the inspection and adaptation on process. Only when you understand how you do things now, can you improve further. The improvement involves the update of policies, which becomes the baseline of next improvement. Scrum teams benefit from treating working agreement as the carrier of continuous improvement. And the working agreement is visualized in the board.
There are 3 practices defined as feedback loops - daily standup, improvement kata and operational review. We have talked about daily standup in managing flow, and let's look at the other two.
Improvement kata is the daily improvement activity. I have been promoting the paring of manager and ScrumMaster on process improvement. Usually, they work together on removing impediments. The impediments may come from daily scrum or retrospective. Kanban measures flow, and provides the feedback with data. The data is systematic, and supplements well for the impediments. Operational review acts as feedback loop in large scale. The key again lies at the analysis of data.
In Scrum, two levels of feedback are built-in, daily and sprint. Daily feedback is provided through status sharing in daily scrum and sprint burndown. Sprint feedback is provided through conversation in sprint review and release burndown, as well as sprint retrospective. Comparing to metrics defined in Kanban, such as cycle time control chart and distribution chart, CFD, etc., those feedback in Scrum is more subjective. Scrum team benefits from integrating more data in the feedback loop. In particular, we shall consider gathering objective data in sprint retrospective to achieve more balanced view and better insights.
- Improve Collaboratively, Evolve Experimentally
The improvement practice in Kanban emphasizes using models and the scientific method. The requires more rigidity in our improvement activities. While it is one direction to improve retrospective through more effective facilitation so as to increase team engagement and promote ownership, it is another direction to improve retrospective through more rigid analysis and followup. I have seen teams applying PDCA in their retrospective and reaping solid benefits.
Even though flow and iteration are different concepts and practices, when we dive deep, we find that Scrum teams benefit from doing some Kanban practices.
- use visualization to help inspection
- limit WIP for flow
- use flow to guide adaptation
- improve on DoD, DoR and working agreement
- create feedback from data
- retrospective with data