Blog

Size does matter! Be careful to use velocity as measure for improvement

24 Nov, 2011

Imagine you are playing a game of rugby against some blacksuited guys who are doing some odd dancing and screaming exercise before you finally get to start playing. You win the game 27 – 3. You can imagine it wasn’t just one beer at the big party after the match and you did not see home before early morning. A year later your team finds itself in the same stadium against the same guys, doing the same little piece of folk dancing, just a little louder than last year. This time you win 27 – 6, only. The coach and the crowd are going mad: your team lost half of its performance in just a year time! You take a shower, no beers, go home and go to bed early. Measuring the improvement in performance is easy!
Acceleration is a must
Scrum advocates the use of pokercards quoting the estimation of work items in story points. Velocity is the total number of story points a team can take on in a sprint.
Next to using velocity for planning, velocity is often promoted as a measure for improvement. Learning, solving impediments, implementing improvements, more fun and better team play all should lead to an increased velocity over sprints. Steady or even declining velocity is a signal of a mediocre, non-improving team.
The theory and actual practice of estimations in story points diverge, which makes drawing performance conclusions out of velocity records a tricky business. The question is whether velocity should be used as a measure of improvement at all. This blog explains why you should not, or at least that doing so should be reserved ‘for trained professionals use only’.
How do you like your Story Points?
In a last-summer blog (June 21 2010, its-effort-not-complexity) Mike Cohn qualifies the relabeling of story point to complexity points as ‘wrong’. He states: “Story points are not about the complexity of developing a feature; they are about the effort required to develop a feature.”
If we follow Mike, story points are a function of complexity, amount of work (‘sheer volume of work’) and expertise (of the team on the domain). Maybe uncertainty should be part of the formula as well. The consequence of this definition is that the amount of work reduces when the team gains more expertise, uses smarter solutions or can profit from structural improvements.
So following Freyr in his first comment on Mike’s blog “a story last year which was a 5, now has some process improvements and this year is a 2. The velocity of the team stays steady, story points per story decrease.” Let’s call this method A.
This way to determine story points differs from the way I was raised with: user stories are estimated relative to a fixed (set of) reference user stories. A story of 5 last year is still a 5. Implemented improvements (whether technical or on teamwork, skills, knowledge or competences) should lead to higher performance so that we can do more such stories in one sprint. Velocity increases, story points per story stay steady. Let’s call this method B.
Count on Story Points while planning
In Scrum estimations and velocity recordings are primarily used for planning purposes. User stories are estimated in story points, and the number of stories planned in sprint depends on the sum of story points of these stories and the velocity of the last sprint or two. Furthermore, story points and velocity records can likewise be used for release planning.
Whether using method A or B a team can properly predict the result of a sprint. And in both cases one can calculate and forecast the number of sprints needed to finish a selected set of user stories in a release.
However, using method A the team will need to deal with new competences leading to lower estimates, even for user stories estimated in ‘the past’. Using method B the team needs to deal with improving velocities when predicting the number of sprints needed for a set Release.
Measuring improvement is a different story
Scrum promises an increase in productivity / performance. A growth factor of 4 even up to 10 is often quoted. Soon after the introduction of Scrum management requests proof of this growth, especially after they have been baffled with all formerly hidden impediments in their organisation: Scrum does not solve your problems, but reveals them! Putting a heavy burden on the belief in reaching the benefits promised.
Very often velocity is used to measure this improvement. This is only valid if we keep the story points for the same kind of stories steady, as shown in method B. In fact this should only be done if we restrict the formula of estimating story points to size, the size of a piece of work. This is shown by an example by Jose in another comment on Mike’s blog:
Imagine that we have a team of painters and their job is to paint walls. The team sees pictures of the walls and they estimate the wall area to be painted. There are small walls, medium walls and some really big walls. The team estimates each area using relative points. They start painting the walls and after some time, they are able to calculate how many points they can do on each sprint, the velocity. Now the customer can calculate how long the project will take, using the velocity and the product backlog.
The problem is that determining size is one of the most difficult challenges in the field of software development. There certainly are environments where this works fine. Recently we coached a team in a Business Intelligence environment, who were able to define their workitems in predefined steps, and who had derived formula’s from experience to calculate the size of work for each step. Their estimations were remarkably accurate.As soon as the painters get better tools and paint, or gain in experience and concentration they will be able to color bigger walls in the same amount of time.
For most other teams the thing coming closest is Function Point Analysis, which is far from easy and a not a common competence in Development Teammembers.
About every other team we have seen estimates effort. Better tooling, a higher grade of automation, more competences, all lead to less effort and are discounted in the final story points. Fine for planning, but using velocity trends based on these story points is a very tricky business and will not reflect the growth teams realized to the amount they deserve.
The point of this story: size matters
Story points are calculated using different methods and formulas. If your formula does not equal just sheer size, you should not use your velocity measures as an indicator for performance growth. Like you cannot learn your performance development by simply comparing the scores of two games of rugby. If there are many other factors influencing the scores you will do wrong to the team and you should stick to use these records ‘to start a conversation’ only. Measuring the improvement in performance ain’t easy.

Questions?