Update: This is quite a long article. If you're looking for a quick read, it breaks down a bit like this:
I have a lot of love for Scrum, the software development process. I have my own little box of Index Cards and Sharpies, and I have sized Backlogs for many of my side projects. Scrum has potential to empower developers more than almost any other set of techniques.
But in almost every implementation of Scrum I've seen in The Real World™, managers are incentivized to help their team deliver broken software to a deadline, and usually end up succeeding in the former and failing in the latter. And when implemented like that, a system that should be an absolutely overwhelming win for developers becomes a tool to beat them around the head with...
So here's Scrum, simplified, and as it's meant to work: You have a Backlog of work to complete, broken down in to Stories, which are distinct pieces of work that should take a few hours to a few days to complete. These are frequently displayed on a Story Board, real or virtual.
You have some actors:
|Role||Also Known As||Responsibilities|
|Product Owner|| || |
|Scrum Master|| || |
|Team Members|| || |
(Any time someone tells me that a Cross-Functional Team means the QAs and Designers should be programming, I try to explain to them that they shouldn’t be anywhere near the development process – it’s better than jabbing them with my Biro. Generally – and a little sadly - you’ll only hear this rubbish from Agile Consultants.)
You have a Sprint which is a short-ish time period, typically two-three weeks, and you have Story Points assigned to each Story based on complexity, which are specifically not time-based estimates. You have a Capacity, based on your Velocity, which says how many Story Points you have historically completed per Sprint, and thus how many you expect to complete in the next one.
Now here's the awesome part, in theory, for the Developers. The Product Owner prioritizes every story sequentially. And in this way he/she gets to 'spend' your Capacity – and only up to your Capacity - each Sprint on the Stories they want to get completed. They've got a budget, and they can only spend that budget, and no more.
And any time a piece of work that's super-duper-mooper important comes in, someone writes a Story for it, and the Product Owner places it where they want in the Backlog. If they place it above any of the work you're intending to get done this Sprint, something has to drop off the bottom. As a Developer, you can illustrate this beautifully, in front of them, by removing Story Cards from the list of items you are intending to complete this Sprint.
(Some people say that this should never happen – once the Sprint is planned, it’s sacred, and that any changes to it need full team buy-in – any significant number of changes need a complete replan of the Sprint. I’ve not seen this so far in practice, but it sounds intriguing.)
This empowers Developers tremendously. Stories cannot be ‘snuck in’ or their lack of completion put down to Developer laziness or disorganization. This gives Developers all the right kind of accountability, and removes from them the responsibility for owning a time-machine when Feature Creep starts to bloat a project.
When it comes to crunch-time on a project, the Developers can show that they’ve been working efficiently, effectively, and to deadline, and push back against the idea that they should be morally obligated to start putting in long hours because they “didn’t get the work done”.
If the Organizers (the ScrumMaster, and in most teams I’ve worked on, the Product Owner) of the team is unhappy with how new Stories will affect a deadline, they can - according to the Agile canon - change three things:
Which bring us to the central problem, and what this article is really about: The Organizers are rarely empowered to change any of those thing, so they change the one thing they can, but shouldn’t: the build quality.
The Organizers’ incentives are broken in most organizations. The Organizers are incentivized to get projects out of the door on the original deadline, at the original price (ie: staff levels) – if they deliver this, they’ve done their job well. When bugs, corners cut, and ungodly amounts of Technical Debt start to surface, all too often this is attributed to Developer incompetence…
(In the small number of software projects with sufficient number and quality of dedicated test engineers / QAs, observers will notice how what constitutes a bug of release-blocking severity starts to change as the deadline approaches. But in the unlikely event you have enough testing resource, you've probably also got a fairly enlightened organization supporting your work...)
Taking Feature Creep and Unexpected Events as a given, there will come a point in the life of almost every project where it becomes clear it can't be done to the original estimate, with the original resources. This appears to be an absolute constant in software development.
And this is the point where the Organizers need to find a solution. The most appealing solution is to find a magical wand that will somehow increase perceived Developer productivity or Velocity, and this is where Scrum usually suddenly morphs from an incredibly useful tool for Developers and Organizers alike in to a world of pain.
How do you magically increase Velocity? Here’s how: any tickets the Developers have created that represent code quality are history. Code quality is a “nice-to-have’ if this project is going to be delivered on time, sorry. If dropping the automated testing tickets means the project can be 'released', and the alternative is a delay or the Organizers having to ask for more resources, all too often those tickets are toast. Finished a feature, but now it needs proper documentation? Suddenly it can wait until after the release.
Before you know it, the project is leveraged to all hell on Technical Debt, and the result is a shitty product.
This is Scrum's biggest failing - the one knob that many Organizers are empowered and incentivized to twizzle is code quality, and usually only in one direction.
If your Organizers are experienced, talented, and confident, and have buy-in from the rest of the business, the effect of this will be minimized. Developers will be shielded from the heat, and allowed to get on with building an excellent project. Teams like this exist - a client I'm currently doing some work for are really good at this - in no small part due to absolute buy-in in to Doing Scrum Right that goes from the Board level downward. This is rare, though, and even they don't always get it right (but they do offer free pizza, time off in lieu, and heart-felt thanks when Developers work weekends – time off in lieu dwarfing the others in importance).
The right solution to this problem is unambiguous business and customer buy-in and visibility - a situation where the business allows and trusts the Developers to be the guardians of code-quality and estimates, and it deals with "business realities" regarding deadlines - by managing resources and customer expectations properly.
The right solution to this problem also involves incentivizing the Organizers properly by making them accountable for code quality – both via honest feedback from Developers on the level of Technical Debt added, and from a perspective of how much continuing support the project requires after it’s “released”.
In lieu of being able to do the most, there are some activist options for Developers wishing to take matters in to their own hands:
Firstly, stop highlighting time spent maintaining code quality. If your test and documentation tickets get consistently de-scoped, stop displaying them differently, and start padding out your estimates to opaquely contain them. Task estimates belong to Developers, and to Developers alone. If Organizers challenge them, offer to put their estimate on the story too, and "we'll see how long it takes".
This is - obviously - antithetical to the Agile ideal of transparency and clear communication. But so is squeezing code quality to make releases. Be like the Internet, and simply route around the damage - you have a responsibility to your co-Developers and to the customer here as a software professional.
Secondly, if your Backlog doesn’t highlight that adding items to it knocks items out of the Sprint, do something about it. Make sure that views of the Backlog contain a marker that shows where the team capacity for the Sprint is. Move that marker to reflect the work that can actually be achieved every time new Stories are added to the Sprint.
Essentially: start encouraging your Organizers to confront as early on as possible that the work that needs to get done may not fit in the time available. The sooner that that crops up as an unavoidable issue, the more time there is to find a real solution. The longer everyone ignores the problem, the more likely it is that the closing Sprints will become Developer Death Marches™.
Thirdly, start keeping a note of areas where you've had to sacrifice on code quality, and where you’ve created Technical Debt. Create Technical Debt stories to represent this, and insist they go on the Backlog, even if “they’ll never get done”. Find the person who is pushing Agile in your organization, and if they're even slightly empowered, enlist their help on this. Find the person who has ultimate ownership of the wider codebase, and make sure they have visibility of the Technical Debt you’re creating, and why.
Some level of Technical Debt is acceptable when you have a tight release schedule. But hiding the creation of Technical Debt is criminal. Make sure it’s out in the open for everyone to see.
Fourthly, start and relentlessly pursue a discussion on what Done means to your team. When is a piece of work truly Done? Is an untested feature really Done? Is it really releasable? What are the impacts for the company and customer on releasing code in which you have little confidence? Hammer out a team statement, with as much Organizer and business buy-in as possible (if you don’t have one), and print it out and stick it on your Story Board.
Leave. The world is crying out for talented developers, and if you care enough about the software development process, you probably fit in that category. There are many places making bona-fide efforts to do this the right way, and a few well-placed questions at interviews ("How does the business deal with it if not all stories have been completed in a Sprint?" - I had a potential client tell me that all work HAD to be done or developers didn't go home... every Sprint) should help you work out which is which.
The biggest problems I encountered were not so much the situation as described; we were working together with 'mature' SM's and PO's. More so, that the enveloping Resource Management (RM) of the whole resource pool over all teams, couldn't get used to the fact that individuals of a well organized team, could no longer be moved around from one team to another, because that actions interfere with each the team's own characteristics. But still, RM had the empowerment to do so, at that point in time. Maybe, in near futere RM have spare time to look out of the window.
Good article though.
A good SM will run interference if anyone tries to interrupt a Sprint. Scrum allows for it to happen but the effects are so deleterious that a good SM will have no trouble explaining why it's a really bad idea.
If, in the face of simple facts, someone with authority still decides to interrupt a Sprint without a massively compelling reason (and they do exist) then you have an individual, or a company, that does not buy in to Scrum. The life of a SM just got MUCH harder :(
I completely agree. The hardest battle any SM will ever have is in ensuring buy in from the senior executive management.
I'm surprised that the PO and SM are insisting on the high level of done. A good Scrum Team will define the requirement of "done" themselves.
Scrum teams must be self managing for best effect. That's how you empower and enable Scrum.
Sorry but I have to disagree with you here. Scrum does contain a mechanism whereby the PO (or anyone senior to them) can insert an item of higher importance into the present Sprint.
A good PO and/or Scrum Master, explaining the deleterious effects of doing so, will often dissuade the person from doing this in favour of placing the item on the Product Backlog with a very high priority.
The key thing about Scrum is that it's essentially a culture of quality.
And if your organization doesn't truly value quality then your poor old organizers will forever be pushed back forth between what the business demands VS what is good for the business (quality).
The best way of solving this problem is probably removing the organizers and allowing business management to experience 'software' related pain directly without any filters. That's feasible in a small company environment and that's what you usually see in a start-up but in a larger company it's likely to be pretty painful.. but highly amusing ;)
Another boat-sinker for the Agile process is assuming that you own all the resources assigned to your project. You don't, and unfortunately this leads to college-professor syndrome: Where each college professor (manager) thinks that you can focus 100% of your time on their class (project).
Great post. Another point worth mentioning, is that one of the intermediate steps in the "Do a lot of coding -> Acquire Tech Debt -> Dont get to address it -> Deliver a product" is that coding gets much slower when you're constantly fighting bugs from poor design leading to the code debt, which can lead to delays in delivering an mVp (with stress on the V). I've found that a fairly reliable warning of code debt is that time estimates on stories start getting missed, some by a couple of days. The reverse can also be true as code that is well factored can make future stories that depend on said code, easier to develop. Conclusion: taking the extra time to remove code debt with an eye to future stories actually lets you go faster.
Very well written. Looks like you have a lot of experience working with the Agile Scrum methodology. As a business analyst, the first agile project that i worked on, failed miserably because the exact same reason mentioned above "empowerment". We were all asked to just do our jobs, and any highlighting of issues, leaks was not encouraged for some very weird reason. The result was a big failure.
I am guessing ideal conditions exist no where, but then recognizing and accepting that there is a problem is half the battle won. Thanks for your inputs.
I am a bit surprised that the focus here is on Scrum or Agile. Why? What was so different before?
Say "Software Business - Delivering broken software since the very beginning"
In my opinion (backed up by 22 years experience within software development and management of it) it is about the culture of your organisation, not what development toolkit you are using. If your team has code quality as an intrinsic value that you as PM share with them, then the conflict interface (quality vs. delivery scope or/and date) will be between the PM and the Owners. ... and this is how it's supposed to be.
Did you expect Agile approach to be a silver bullet changing everything just because you use the word scrum, have a backlog and daily stand-ups?
Don't blame Agile for lack of the culture or spineless managers. What Agile is really good at is to make these problems visible.
This is a series of articles discussing the best pieces of Scrum. The pieces you - as a developer or Project Manager - can steal and start using independently of a wider Scrum implementation. We'll discuss which bits of Scrum you can rip-off without drinking any of the Certified Agile Consultant Kool-Aid - in short, we'll be looking at how to do Scrum for small teams and startups.
Scrum is a business process, and like any business process, it sits between you and the work you need to get done. If it doesn't help you get that work done more effectively it's a big waste of time. Good developers, like everyone else, hate having their time wasted.
Scrum is often implemented by someone who went on a Scrum Master Certification (NOW 90% MORE AGILE™) once, and is attempting to implement it as a set of cargo-cult rituals in the wild hope it'll make everything better, and more Agile or something. Sometimes this'll be a Project Manager, and sometimes this'll be someone billing their consulting time as a Scrum Expert. Planning sessions become elaborate Arts and Crafts sessions with Post-Its and Sharpies, the Product Backlog becomes time sheets by any other name, and Velocity gets used as a personal productivity measure.
In short, in the wrong hands, Scrum becomes a tool for wasting developer time, and neatly tracking, graphing, and reporting the resulting drop-off in productivity.
This is a huge shame, because done right, Scrum protects and empowers developers. It pushes commercial and business considerations back in to the hands of the business and the customers, while placing implementation decisions back with the developers where it belongs.
We've been using the Scrum agile development framework (or just "scrum") at Socialize since early this year. Hats off to Jason, our VP of Engineering, for really championing it and to Sean and Isaac for implementing it with the developers. Our workflow is to use Basecamp to discuss ideas (Basecamp is like Democracy: it's not great, but it's the best thing out there -- you can read my rants & raves about Basecamp here), then Github for issues & code repository, and Pivotal Tracker to manage our dev process. We do fast iteration cycles with one-week sprints, planning meetings every Monday and daily stand-ups to reconnect.
I'll explain what all that means below, if you don't already know, but here's the point of this blog post: Scrum has worked so well on the dev side -- what if it was possible to implement scrum across an entire company, including its non-development components like user acquisition, sales, even accounting, HR & finance?
We decided to find out. I'm going to write a series of blog posts with our experience implementing Scrum throughout our company, with a focus on the non-development parts of the organization, as I haven't really seen other companies do this yet. I figure this is a good opportunity to share our knowledge and learn from yours, so please post your experiences with Scrum, agile and general, and other workflow approaches in the comments.
In this first blog, I'll talk about what scrum is (at least to me) and show you a video of Jeremia, our developer evangelist, and Christine, our wordsmithstress, discussing the pro's & con's of implementing scrum in the user acquisition department (the first non-dev team we decided to test it with).
There are individuals within our organization that know scrum way better than I do because they're using it every day -- Jason, Sean, Isaac and our incredible developers, for example, and I'll invite their comments. Our goal is to institutionalize this agile mentality throughout the company.
We've been using the Scrum agile development framework (or just "scrum") at Socialize since early this year. Hats off to Jason, our VP of Engineering, for really championing it and to Sean and Isaac for implementing it with the developers. Our workflow is to use Basecamp to discuss ideas (Basecamp is like Democracy: it's not great, but it's the best thing out there -- you can read my rants & raves about Basecamp here), then Github for issues & code repository, and Pivotal Tracker to manage our dev process. We do fast iteration cycles with one-week sprints, planning meetings every Monday and daily stand-ups to reconnect. I'll explain what all that means below, if you don't already know, but here's the point of this blog post: Scrum has worked so well on the dev side -- what if it was possible to implement scrum across an entire company, including its non-development components like user acquisition, sales, even accounting, HR & finance? We decided to find out. I'm going to write a series of blog posts with our experience implementing Scrum throughout our company, with a focus on the non-development parts of the organization, as I haven't really seen other companies do this yet. I figure this is a good opportunity to share our knowledge and learn from yours, so please post your experiences with Scrum, agile and general, and other workflow approaches in the comments. In this first blog, I'll talk about what scrum is (at least to me) and show you a video of Jeremia, our developer evangelist, and Christine, our wordsmithstress, discussing the pro's & con's of implementing scrum in the user acquisition department (the first non-dev team we decided to test it with). There are individuals within our organization that know scrum way better than I do because they're using it every day -- Jason, Sean, Isaac and our incredible developers, for example, and I'll invite their comments. Our goal is to institutionalize this agile mentality throughout the company. So what is scrum, really? To me it's simply a way of thinking about getting things done. There are traditional development models like waterfall development, which (like a waterfall) takes a staged, or phased approach to development. I'm going to abstract "development" into "workflow" for the purposes of this blog, since we're talking about using these models in a larger sense throughout the company's divisions. The magic of scrum is that it takes time out of the equation. Or to put it another way, assuming you have employees that are dedicated and are working hard, things take as long as they take. You're not doing anyone any favors by setting arbitrary deadlines. "This project needs to be finished by Friday" or "This deliverable is due in 60 days" cause way more problems than they solve. When the team inevitably misses the goals, everyone is disappointed -- the goal setters blame those responsible for stepping up to an arbitrary date, and the implementers resent having a date put on them that didn't reflect reality. The scrum attitude is that things take as long as they take, so the best thing to do is embrace that reality and instead of setting deliverable-based goals, instead focus on prioritizing everyone's time effectively so the most important and value-added items are being worked on first. It's a very simple but monumental shift in thought. Things take as long as they take. Focus on what matters. Here's a video of Jeremia & Christine talking about implementing scrum in the UA department. In the next blog, I'll discuss some of the difficulties we've had implementing scrum in a non-development environment. >