Accuracy and Precision

Today I got asked what I thought was the difference between accuracy and precision as it pertains to relative estimation, I wasn’t ready for the question though and it’s been sitting on my mind bugging me so here are my thoughts:

I would think accuracy would be how close the estimation was to the truth and precision is how often you predict the same estimation given the same or similar complexity, in other words how repeatable you are. Accuracy is how right or wrong you are and precision is about how consistent you are (irrespective of right or wrong).

So, if you correctly estimate over and over, you’d be accurate and precise but if you estimate incorrectly, but are consistent about the “incorrectness” then you would be precise but not accurate. If you estimate correctly only every now and then for the same given complexity then you would be accurate but lack precision.

Interesting to hear what others think (like if I’ve lost my mind completely 🙂 )…

 

Velocity for the layman

WARNING:This is a brain dump of an idea I’ve been playing around with, some confusion may occur!

As a ScrumMaster I often struggle to explain the concept of velocity to people, a critical concept to grasp to understand how to report on progress and capability. I often get these questions thrown at me:

  • How many features can we complete in X weeks? or…
  • How many bugs can one developer fix per day ?
The danger of these questions, for me, is not so much in the question (which doesn’t make sense) but in what happens afterwards:
  • 1 developer = 1 story per sprint so 2 developers = 2 stories per sprint, right?
  • 1 developer can fix one bug in 1 day, we could double our rate by doubling our people!! or…
 So, to the challenge: how to explain the danger is making all pieces of work equal and how to explain the benefits and logical sense of measuring in velocity.
Let’s start with the analogy: You’re going on holiday, you need to travel from city A to city B (your goal) with 5 stops between.
Doing it wrong would be to assume that you multiply the time to the first stop by the number of remaining stops and that would be your final travelling time.
Doing it right would be to look at the distance between each stop, calculate the average speed you can maintain, and then add up the travel time between each stop. Even better would be to:
  • Adapt your average speed based on environmental conditions. Bad weather, day/night travelling, fatigue, additional rest stops, travelling through mountains? These all affect your average speed.
  • Keep track of your average speed as you travel, this should give you an indicator of exactly what average you can maintain.
  • Check what affects the distance between towns like road closures, alternate routes.
  • Be ready to adapt to changing conditions, like accidents and sudden road closures.

Given the above you should be able to accurately predict when you will arrive at your goal. As a bonus you can also work backwards. Given your need to arrive at a particular time, you can sum up the total distance and divide it by the time you have and this should give you the speed you should try to average.

So, to bring it back to velocity, you average travel time can be equated to the teams average velocity (or capability). Stops are user stories/bugs and distance between cities is the effort of doing the story. Conditions of road, road closures, mountain travelling, predicted bad weather etc is your complexity.

Here’s hoping this analogy makes sense, if it doesn’t please let me know. I still think it’s missing something though so I may add to it at some point…

When will we be finished?

I’m a firm believer that all work should have an agreed end date, but end dates are always a bone of contention and cause endless confusion and wasted arguments and disagreements. These usually hover around the “when will we be finished?” question and, more to the point, no properly defined (empirical) way of working it out.

That in mind I thought I’d lay down the process we use to arrive at our end date. I will also attempt to do this without using the word scrum or agile, not because I don’t like or agree with them, but because those words tend to create predefined thoughts and have a life of their own (unfortunately).

 

So first the basics: In a nutshell we work in iterations, releasing something (hopefully of value) every 2 weeks based on a prioritized backlog. The backlog is constantly updated as stories are added / removed / completed.

Now for the fun. In order to calculate an end date with reasonable empirical accuracy you need the following: remaining effort and an average effort per sprint / day the team is capable of.

Effort Scale

Effort is difficult to define but key to the entire process. Everyone should agree and understand what is meant when we say “X amount of effort”. What you must realize is that it does not matter what unit of measure you use to define your “effort” as long as:

  • it is consistently used across all stories
  • it is understood by everyone
  • something of effort X closely matches another thing of effort X,  for small values of X. 
  • the units should increase roughly exponentially. This is to allow for indication of greater uncertainty with significantly higher numbers. Some people use the Fibonacci sequence (1, 2, 3, 5, 8, 13 …). Also remember to allow for zero effort.

We struggled to settle on our effort scale but eventually had to settle on a time relative scale (for various reasons):

  • Very Easy (1 point) – roughly a day to complete
  • Easy (2 points) – between a day and 2
  • Medium (4 points)  – roughly half a sprint
  • Hard (7 points) – roughly a full sprint
  • Epic (20 points) – unknown or way more than a sprint

and of course 0 points for quickies.

Effort per story

Assigning effort to stories is an exercise the entire team should do but failing that at least the majority of the team. Stories should also be continuously rechecked to see if the effort is still correct, especially stories with high effort assigned to them.

The process we use is as follows:

  1. Discuss the story
  2. Vote on an effort, if there’s dispute, use the highest of the choices, if there’s huge uncertainty, assign the highest effort.
  3. Rinse and repeat

Remember also to compare stories of similar effort. Another way or form of checksumming is to, during planning, assign tasks to the story (as in what you’re going to do to achieve the story) and then assign a rough hour estimate to do the tasks. Sum up the hours for the story and an equivalent effort story should probably have a similar amount of hours, although this is only really useful for well understood stories.

Average Effort Per Iteration

Calculating the effort per iteration (sometimes called the velocity) is a matter of adding all effort units (points in our case) for all completed stories for a sprint. It’s important to only tally up completed stories or the whole process is pointless. Average effort is just the sum of the effort for X number of iterations divided by X. Obviously the more data you have the better. What we also do is remove the highest and lowest values (as anomalies) from the calculation, to account for those strange iterations (over Christmas, Easter etc.)

Remaining Effort

Remaining effort is simply the sum of all effort that has not been completed.

The Formula

Calculating the end date then becomes the following:

By Day: date today + (remaining effort / average velocity / number of work days in iteration) divided by 5 and then multiplied 7 to account for weekends = your end date

By Iteration: current iteration number + (remaining effort / average velocity)  rounded up to whole number = final iteration of work, end of iteration is end date

and that’s it, you’ve predicted a future date based on historical data using a reasonably empirical method. You can improve the accuracy of this estimation by doing the following on a regular basis:

  • Re-evaluate the backlog from an effort point of view, focusing particularly on high effort stories.
  • Recalculate your average velocity at the end of each sprint and factor it in.
  • Factor in holidays and leave into your calculation.

Comments or suggestions welcome

Change is good

I’m often amazed at the number of dogmatic “prophets” out there when it comes to the latest fad’s of agile development. I myself am about getting productive and like to take the best of ALL worlds; change is not only inevitable but necessary, even changing of a process BUT, only if the change leads to improvement! I believe a process of continual introspection and change by a team, combined with a good way of measuring the effect of that change,  is essential for good team performance.

That in mind we made the following changes to the way we practice scrum:

  1. We added a default story to cover recovering technical debt (or refactoring if you will) in that story. Essentially we assign a “story point cap” to this story (say 5 points) that’s based on our current velocity and duration of sprint. We then assign “technical debt recovery” tasks to this story until the team feels there are enough tasks to fill the story points. I know, sounds like the wagon pushing the horse but it’s something we’re trying that other teams have used effectively.
  2. We’ve started estimating (wait for it…) HOURS on tasks. Stories are still broken into story points but the tasks are estimated in “hour blocks” (2, 4, 8, 16) as they are put up. The purpose of this is to act as a type of “sanity check” for story point estimation. An added benefit I found was that the team members seemed to think a lot more about a task when they had to assign time to it.
  3. We’ve started using a round table instead of the tradition square meeting tables. I found this very effective in encouraging participation and collaboration.
  4. Story point estimation was done using a technique I learned on training that speeds up the estimation incredibly (works well with lots of stories). This is how it works (well, how we did it anyway):
    1. All the stories are explained to the team and placed in a pile on the table.
    2. A wall/board is divided into sections, each representing a story amount (we used 1, 3, 5, 8, 13, 20, 40, 100).
    3. Each team member takes a pile of cards and sticks them in the column(s) they believe it belongs, no-one talks or interacts other than with the product owner to understand the story further.
    4. If a team member disagrees with where a card is, he/she simply moves it to where he believes it SHOULD be.
    5. This carries on until all the tasks are done. Remember, the key is NO COMMUNICATION between team members during this process, this has the effect of forcing people to think about why someone made a decision to place something in a particular column, or why they’re moving it.
  5. We’ve started experimenting with TDD and Paired Programming, which seems to be going well.

One additional thing I’m toying with is moving to a more Kanban based board, the team has problems with taking on too much work and I think creating a way of limiting the pulling of work will work.

Finally on how to measure if these changes are effective, well, to be honest I’m not entirely sure. I think the best way is simply to see if the team becomes better at actually doing what they predict they can do while maintaining a sustainable pace and having fun; other than that I guess we wait and see…

I’d be interested to know what kind of changes you brought about in your team and the effect of them (both good and bad), let me know…

Affinity Estimation

“Affinity Estimating is a technique many teams use to quickly and easily estimate (in Story Points) a large number of user stories. This is a great technique if you’re just starting a project and have a backlog that hasn’t been estimated yet.”

I remember this topic came up during the ScrumMaster course and I found it quite interesting, I’ve finally found an article related to it on Kane Mar’s blog, have a look at the entry here.  I’ve yet to try it but will when I have a project with a huge amount of stories, it sounds like it can work quite well.

Epiphany…

Epiphany: a sudden, intuitive perception of or insight into the reality or essential meaning of something, usually initiated by some simple, homely, or commonplace occurrence or experience.

I’ve always struggled with accepting the concept of estimating using story points and NOT effort hours. I could never get my mind around the concept and yes, I’ve read a lot about how estimating with hours is bad (people pad, management argues that real time <> estimated time, team members feel pressured etc etc…), and I believe this, however I did not understand how bad it was until today.

During our estimation (still doing hours effort estimation) we were trying to define the tasks necessary for a story to be completed, however the story had a “task A” that would lead to several “task B’s” that were known, however no-one could (and neither should they be expected to) estimate the hours effort on the “task B’s” simply because the input from “task A” was critical to the timing! Now you’re stuck with tasks that are KNOWN, but cannot be TIMED. And then the light went on and I realized why estimation with time was so bad…You may KNOW everything you need to do (the complexity), however you may not necessarily know how LONG everything will take (the effort), in other words, you know COMPLEXITY but you cannot measure the EFFORT. It now makes sense…

Scrum in a Nutshell

I found a nice, concise, description of SCRUM here:

Scrum – a knol by Dan R Greening

I found particularly interesting the first comment made that clarified my questions regarding the fundamental difference of the feature/story point estimation vs the task/hours estimation techniques. It appears that the reason is pretty much around the comfort of the team and their willingness to participate effectively in the estimation process. I strongly suggest reading it…

First Sprint Planning…

Today was interesting… we did our first sprint planning meeting with a team of 5 people. After I’d sat with the designated Product Owner (a BA with a passion for the project) and created what I think is a sufficient product backlog,  I explained the concepts of Scrum to the team (especially around estimation and effort) as I was taught and we actually managed to determine our first sprint backlog. I did have one question regarding the burndown chart though, and having researched it in “Agile Software Development with Scum” and I realized what I was taught is different (in implementation) to what they say in the book. Boris estimates using what I think are “weightings” for tasks (relative to one another), the book demonstrates the use of “hours” to measure effort, although the “hours” are not commitments but only estimations made “as a team”. This seems a little more logical to me as well. I’ll take it to the team and see what they think, afterall, in the end its only important that you have a decent burndown chart with decent tracking isn’t it?