“What is the difference between an up-front requirements document and a fish?” said Gollum, hoping to catch the Hobbit out with his riddle. But Bilbo Baggins, though a Halfling, was an experienced Agilist. He knew the answer. “A fish rots from the head down, “ he said, “but a requirements documents rots from the bottom up”. “Curses!”, screamed Gollum, “You cheats us”. But Bilbo wasn’t cheating. He was just drawing on experience.
Detailing all of the requirements out up front is a colossal waste of effort in the development of software products. The average churn after the document has been signed off is said to be about 35%, although most software professionals when I ask them to give an estimate, tend to give me a much higher figure. In any case, the point is that by the time the requirement is reached, there’s a good chance it will have either significantly changed or disappeared completely from the To-Do list. The effort spent in eliciting needs, analyzing them, and documenting the system requirements that meets those needs has been wasted. That, in turn, entails a cost: the salaries of the professionals involved for a start. And now at least part of that effort and cost has to be spent again in understanding the new situation.
There are broadly three responses to this problem:
The Scrum Guide says that “Higher ordered Product Backlog Items are usually clearer and more detailed than lower ordered ones”. This implies just-in-time requirements, but the Scrum framework doesn’t describe any process or mechanism for achieving this. This is entirely in line with the philosophy of Scrum. It is for self-organizing teams to figure out what is best for their own context.
Undeniably the popular choice is User Stories. These originated from XP and, in particular, from Ward Cunningham and Kent Beck. Maybe nine in every ten Agile teams use stories as Product Backlog Items (PBIs). Used properly they are very effective in driving just-in-time requirements. But in my experience, they are rarely used properly.
One of the biggest mistakes teams make is to confuse User Stories with System Requirements documentation. This either leads to stories that themselves are long documents, or to there being no documentation at all other than the first versions of the stories – often just three-line statements of what a user wants.
PBIs are not requirements themselves, they represent requirements. They are items in a list that is ordered by the Product Owner. So, when stories are used as PBIs they are not requirements either. Actually, it is more accurate to say that they are not system requirements. They are short statements of user needs (the IIBA calls these stakeholder requirements) for which the development organization is charged with providing a solution. This requires intense collaboration and multiple conversations between customers and developers to decide which solutions fit those needs best. Stories are designed to trigger those conversations, while system requirements documentation should be their outcome. How much documentation and what form it should take is again a decision for the development team itself.
Once it is properly understood that stories are there to trigger conversations, the team can decide how and when is the best time to have them. Of course, when a story is about to be pulled into a Sprint for development it must be “ready”. In other words, the development team must understand it well enough to start work on it straight away. Most high-performing Scrum teams will have enough “ready” stories at the top of the Backlog to occupy them for between 1 and 3 Sprints. This basically means all of the necessary conversations will already have taken place.
The remaining stories in the Backlog are often called epics. This just means they are too big and vague to be considered ready for development. There are more conversations to be had yet. In the earliest days of the development cycle, the Product Owner will be discussing these stories with what I call commissioning customers. These are the people funding the project or other important stakeholders who are not themselves necessarily users of the product. They see it as a commodity. They need the product for some workflow they are responsible for, but they are more directly interested in the cost of the development and the date of delivery than they are in details of its usage, for example. Later on, end users will be engaged to ensure the product is really helping them do their job. The process of Backlog refinement is exactly that of getting these high-level epics to the point where they can be pulled into the Sprint.
The Product Owner decides which epics to decompose first for more detailed conversations. Ultimately, it is her responsibility to get stories or PBIs to “ready”. Her judgments about the relative importance of the items are reflected in her ordering of the Product Backlog. These are themselves made in the context of her understanding of the Product Vision. One of the most difficult aspects of her role is to keep the big picture of the vision in the minds of the development team even while they are focused on the short-term Sprint Goal. That’s a skill that has to be learned – but it’s a lot easier than creating those up-front requirements documents. Plans based on those contain as much fiction and fantasy The Hobbit and the Lord of the Rings put together.
Looking to learn more about just-in-time and other strategies? Learning Tree offers a wide variety of Agile and Scrum training.