Usefully estimating software projects is difficult, but not impossible.
A lack of understanding about why we estimate, what to estimate, and how to estimate leads to a breakdown of trust and communication between developers and managers.
Developers end up feeling guilty that they're not meeting their estimates, and at the same time defensive: they were just estimates after all, right? Managers feel exasperated that everything is taking three times as long as it should. What are they doing all day?
This article is about fixing your estimates, and your estimation process. It's split in to two parts - the part you're reading, titled "How to Estimate like an Adolescent", and the part you're not yet reading, titled "How to Estimate like an Adult - What to Steal from Agile".
As an aside, if you're in a position where someone else is estimating work you're doing, get out. The work will be late, you will be blamed, and you will be miserable. Programming is meant to be fun, and setting yourself up for accusations of professional incompetence and the niggling feeling that maybe you are incompetent is the antithesis of fun. Seriously, get out.
Estimates are not the same as deadlines (which someone else will be setting), largely because deadlines are the business's problem, not yours.
What does a business do? It turns money in to more money (usually).
It spends money on your time, a computer for you to use, and a (hopefully) endless supply of second-rate coffee. In exchange, it wants you to write software, which it will use to obtain more money.
Fine, and so far, so obvious. So why does a business need estimates from developers?
Firstly, so they can plan more effectively - so that those funny Gantt chart things line up in the right places. This means the marketing department know exactly when to make sure their diaries are free for wining and dining journalists, and this means the CEO's secretary can make sure he's not off hiking the Appalachian Trail around product launch.
Secondly, so they can estimate resource allocation properly. Remember when I said estimates aren't the same as deadlines? If the estimates bubbling up at the start of the project show there's no way two developers can deliver the electric eggbeater calibration software by the deadline, then at this point, the business can do anything about it; it's still the business's problem to solve.
And of course, if it only becomes apparent towards the end, that's when people start monitoring your Facebook usage. There is a way to solve this, and you should keep reading.
That's Economics 101 out the way.
Project Managers spend all their days getting Agile Certification Scrum Master Training, but in the brief moments when they're not doing that, their job is to monitor progress on the project, and communicate that upward, updating their Gantt charts so the CEO can rebook his holiday to Argentina.
At the same time, their responsibility is to make sure resources aren't being wasted. That can involve anything from procuring software, arranging meetings with suppliers, and shouting at you for spending all day updating your LinkedIn profile.
The earlier and more accurately the Project Manager can communicate changes in the project status to the people they report to, the easier their life is, and the nicer they are to developers.
The more meetings they go to where the project has slipped, and they can't produce a good reason for that, the harder their life becomes, and the more chance there is of a poisoned donut among the decreasingly common Krispy Kremes.
Any software you write, ever, will have some degree of hidden complexity. Whether this is the piece of open-source software you were hoping to rely on being riddled with bugs, or whether this is working with your coworker's poor grasp of service architecture, or whether this is that line in the specification that no-one thought about too hard that triples the workload, there are nasty surprises lurking everywhere.
When you're in your first job out of school, you're allowed to be bewildered and surprised by this. You're allowed to go home and cry in to your KFC Famous Bowl while Great Gig in the Sky plays, angry that you spent the 2 days you were meant to be setting up the AMQ Server trying instead to configure Entourage 2008 to talk to Exchange 2011.
But once you're an adult, this should no longer surprise you.
Not only should this no longer surprise you, but you should start getting an eye for which sorts of stories these situations are most likely to occur in. This is complexity. And this is what you, a software engineering adult, should be estimating (and what we'll get on to in Part 2).
Before we go any further down the Agile Velocity rabbit hole (and we're going all the way down in Part 2), we're going to look at the easy way out here. That's because the easy way out usually works well enough, and it's important to look at why it works.
Here is the easy way out for estimating a development task/story:
Break your task in to smaller components
The more you break your task in to components, the more you'll think about how you need to approach it, the better understanding you'll gain of the issue.
Estimate those components optimistically, and then triple those estimates
As above, every software project has hidden complexity. Over the aggregate of software tasks, the degree of extra complexity (ie: the multiplier effect) tends to be constant. This is hugely important, and this aggregation effect is the key to estimating like an adult.
Why triple? That number was pulled from the air, by which I mean over a decade of software development experience.
Any time someone gives you anything extra to do (extra functionality, something that takes you away from the project), tell your Project Manager to make a note of it, and increase the estimate
Your Project Manager is responsible for your time being used effectively. If they're unaware, or forget (or just as often, you forget), where your time is being diverted, they can't do anything about it, and you end up looking like a slacker for not delivering on time.
This is the single biggest area of communication breakdown. Keep a note of when the scope is being added to, or your time is being diverted, and make sure your Project Manager is informed early, and often.
Give your Project Manager daily updates of the percentage progress you've made through the task
If you've tripled your optimistic estimates, some days you will be zooming through, and some days you will be moving slowly, but the overall completion of the task will probably come in on time. Cut to a shot of Gantt charts and a smiling CEO.
If you stick to this religiously, you will have a mostly easy life. You may initially get pushback from other developers. Any developer dumb enough to push back on such an estimate should have attention drawn to their previous estimation efforts vs elapsed calendar time.
And you know what? You don't have to wait until Part 2. We've covered the four most important points:
In Part 2, we'll be looking at how Agile solves all your problems ever. Or, more accurately, how this whole Agile thing has a couple of really good ideas on estimation, how to use them, and how to avoid some gigantic pitfalls associated with them.
Until next week...
Wow. You nailed this, concisely.
I've been a developer, team lead, project manager, consultant and worked in an CMMI-5 environment (that's "optimizing" processes).
Use project managers to help you. They aren't out to get you unless you don't provide updates when something unplanned happens. As soon as you realize an issue, get the PM involved. She/he might be able to get you some help.
Using 3x the estimate works well too. It allows the necessary time for testing, documentation, error recovery, input validations, helping team members, progress meetings, or just recovering from a dead HDD. Stuff happens.
When I'm estimating, I try to get down to 4 hour tasks. This may seem excessive, but it really forces me to think at a detailed implementation level and uncover enough of the problem areas during the estimation process to set realistic expectations. During the development process, it also lets me know when I'm falling behind and a few hours this evening will help to get back on schedule.
Whatever you do, don't be the guy who gets to 95% complete in 3 weeks and needs another 6 weeks to get 2% more done. You know who you are and your boss will soon, if he doesn't already.
Update: This is quite a long article. If you're looking for a quick read, it breaks down a bit like this:
Update 2: There are active and interesting discussions of this article on HackerNews and Reddit
I have a lot of love for Scrum, the software development process. I have my own little box of Index Cards and Sharpies, and I have sized Backlogs for many of my side projects. Scrum has potential to empower developers more than almost any other set of techniques.
But in almost every implementation of Scrum I've seen in The Real World™, managers are incentivized to help their team deliver broken software to a deadline, and usually end up succeeding in the former and failing in the latter. And when implemented like that, a system that should be an absolutely overwhelming win for developers becomes a tool to beat them around the head with...
So here's Scrum, simplified, and as it's meant to work: You have a Backlog of work to complete, broken down in to Stories, which are distinct pieces of work that should take a few hours to a few days to complete. These are frequently displayed on a Story Board, real or virtual.
We've been using scrum all over the company for the past year, and the more I use it, the more i love it. I previously wrote a blog about using the scrum agile development methodologies in a non-development capacity -- i.e., with a sales or marketing team. I got a lot of questions from that blog about exactly why Scrum was so great, so in this post my goal is to tell you why I love Scrum so much, and why I recommend it to be adopted within any company (and if the US Government ever implemented Scrum -- correctly -- we'd seriously save trillions of dollars. That's probably an impossible wish, but after all, thinking big is free.)
Here's a good 7 minute video describing Scrum. If you're not at all familiar with the methodology, you might want to read my case study below before you watch the video -- it'll make more sense that way.
Video: Scrum in 7 Minutes:
Scrum Case Study: