Improving a development team's velocity is no easy task for a leader.
In 2015, I led a team that was taking a long time to build new features, but it was tough to pinpoint a single cause for the slowness. Every week a new problem seemed to arise. A simple change took time because of tightly coupled code. The following week we had six pull requests waiting for code review from one of the engineers. The following change did not have enough test automation, so it took more time to validate.
And it's not that that the team was not doing anything about it. We were saving 30% of the time to tackle technical debt. We were doing refactors, improving test coverage, and the tooling for automating part of the code reviews.
Nothing seemed to work, and we started to run out of ideas.
What industrial factories taught us about improving velocity
There are many learnings the software industry grabbed from industrial factories that helped us improve our velocity.
Finish stuff before starting new stuff
This is the art of managing the team's work-in-progress.
Juggling too many balls at once will lead to a lot of context switching. The team won't move fast if work is often interrupted or priorities switch too often.
The work-in-progress (WIP) is all the work you have started but not yet finished. In 1954, John Little published a paper in Queue Theory, which proved the following: Avg Cycle Time = WIP / Throughput. The takeaway is that the lower the WIP, the shorter it takes to get things done. Kanban is a management method that exploits this concept by setting WIP limits to a system. By limiting the amount of WIP, teams can finish stuff before starting new stuff and increase their velocity.
We adopted a Kanban board and set WIP limits for our work. We used the board to check it from right to left in our team check-ins, focusing on finishing the work closer to completion. This step was the first thing that started to give us a consistent velocity increase.
Optimize the goals of the system rather than the parts
Goldratt, a famous Engineer who studied industrial processes, built his fame on top of a crucial insight:
Every system has a bottleneck. Improving the system at any other place than the bottleneck yields no improvement in the overall outcome of the system.
For example, imagine a team composed of a Product Analyst, three Developers, two Testers, and one Operations person following a development process with the following steps: design, development, testing, and release. Each role is working on a different stage.
Let's assume the testing team is overwhelmed:
Improving development capacity will only make the problem worse. Developers will produce more features, giving more work to the testing team, which is already overloaded.
Adding more people to the release team will also not improve the group's overall output, as the release team will be idle, waiting for more changes to be released after the testing team does their work.
The only way to improve the system's output is to improve capacity at the bottleneck, which is, to improve testing capacity.
The main problem with improving the velocity of software development teams
So, great. We are managing WIP. Now let's focus on improving the bottleneck.
Applying Goldratt's ideas to modern software engineering teams is no easy task!
Industrial Factory processes, the ones that Goldratt studied, have two essential characteristics:
- You know at which rate work items arrive at each part of the process
- You know how much time does it take to process each work item
In these processes, the bottlenecks are pretty much static. But software development is not an industrial process; it's a creative process. Software development teams may know or not the arriving rate of work, and for sure, they can't know how much it will take to complete each work item.
What does this mean? Goldratt's ideas do not apply to Software Development?
Goldratt's ideas still apply, but the variability of arriving rate and time it takes to process each work item causes the bottleneck to moving around often. One week testers will be idle waiting for developers to complete their changes; another week, they will be overwhelmed.
That's the reason why most attempts to increase velocity in development teams fail. They fail to address the variability of the software development process.
Characteristics of a team that optimizes for velocity
Ok, now let's go back to the team story. So I knew that if we wanted to build a team that optimizes for velocity, we would have to either find the way to avoid getting these bottlenecks or dynamically adapt the team's capacity to respond to them.
Staffs generalists to deal with the variability of the development process
Generalists are more flexible than specialists.
We need the capacity at each stage of the development process to be flexible. To achieve that flexibility, we need generalists who can perform diverse tasks to deal with temporary bottlenecks at different stages of the development process.
For instance, our team had an Operations Engineer who mainly focused on infrastructure changes and maintenance activities. As he couldn't do development work, he usually did not contribute to helping in the development or testing stages. Specialists typically don't have flexible skills to help tackle bottlenecks outside their primary area of experience. When a team member can't help other team members, they work on their private backlogs, which only have tasks they can perform, regardless of those not being the most valuable to the team.
Generalists can better address the variability of software development by being able to help on the different stages of the development process. Another thing we did with the team was to balance tasks between two or three team members. Over time, we gained the ability to have more than one person tackling the most common activities of the team.
Avoids bottlenecks with radical collaboration practices
Why optimize for dealing with a bottleneck if you can avoid it?
We learned from Xtreme programming and DevOps that we can move beyond the factory-like process model by embracing other collaboration practices.
For instance, code reviews are a common bottleneck in the industry. We can eliminate the code review step by doing pair programming. Pair programming allows for a continuous review of the code from a peer, eliminating the need to have a review step. This practice is radical because it challenges the idea that two programmers parallelizing work will achieve faster results than two programmers working on the same item. The former assumes the constraint is the ability to type code in the computer, while the latter believes it is in problem-solving and learning. We can eliminate review-based bottlenecks and accelerate cross-training by adopting pair and mobbing practices. Consequently, we can improve the team's velocity and the ability of team members to help address bottlenecks in areas that are not their primary expertise.
Although we never eliminated our process's code review or testing steps, we adopted pair-programming and mob-programming techniques. This experience was enabled us to respond better to what got stuck at a given week. For example, after six months of adopting these practices, most team members were able to do minor fixes, code review, or validate changes to all the projects owned by the team.
In a sense, learning IS the bottleneck, and we can minimize its impact by improving the way we collaborate.
What consistently improving velocity is all about:
Many teams adopt management systems that optimize flow (like Kanban). Still, they fail to consistently improve it because they do not embrace the creative development process's ever-changing variability and accept that bottlenecks will often "jump" from one stage to another. Adopting the mindsets that allow the team to reorganize efforts around the bottlenecks and using advanced collaboration techniques to avoid them can help them achieve that consistency.
Special thanks to David Golden for reviewing multiple drafts of this article.
Follow me on Twitter for more articles on troubleshooting software engineering teams' problems.
