Scrum and XP: reconciliating Agile with its roots

I really encourage you to have a look at this awesome session from Robert C. Martin, called The land that Scrum forgot. 

Besides the usual pleasure to listen to Uncle Bob, it is an impressive overview of the history of Agile, admirably told by an eyewitness and one of the main actors: you can really see in action the giants (on whose shoulders we're all standing) and feel them like closer friends, more than myths.
Among the different enlightning reflections, Bob Martin draws a clear and interesting perpective of the last decade (and more) about why Agile is somehow  not completely fulfilling the original expectations: Agile was originated by developers (all Agile signatories were basically developers), then it got conquered by project managers and lost part of its original intent.
Scrum unintentionally contributed to this, being seemingly so simple to be understood from managers, but so generic about how to build a potentially shippable product increment every sprint to be incredibly hard to adopt. Thus Scrum was actually the way people with project management background used to enter the Agile world either with good intent or opportunism.
On the other side XP was THE Agile method of developers.

I will try to analyze and compare Scrum and XP here to show how they're aiming at the same purpose and how both effective Scrum and effective XP embody the same genuine Agile principles and values.

Scrum and XP have actually a lot in common, more than one might think. 

They are both based on iterations and have some practices in common:

• the Planning Game in XP overlaps with the Sprint Planning in Scrum
• both produce a working SW increment of the most valuable functionalities at the end of each iteration
• the team is self-organized and decide how to build a certain function
• business and developers work together on daily basis
• the system metaphor in XP overlaps with the concept of User Story which is basically used in every Scrum Team I know, even though it is not explicitly requested by Scrum which basically only mention generically a backlog item as input to the Sprint
• both encourage similar values, and actually share two values: Courage and Respect

One might also claim they have some slight differences:

• Scrum prescribes Sprints which can be 1 to 4 weeks long, while the length of an iteration in XP is 1-3 weeks long
• the customer works together with developers in XP, while Scrum proposes the role of Product Owner as customer representative to suit the cases where the customer cannot physically work with the team (btw that’s one of the reasons why Scrum scales and XP doesn’t)
• strictly speaking, Scrum teams do not allow changes to their Sprint Backlog. XP teams instead can allow changes within their iterations: as long as the team hasn’t started working on a particular feature, a new feature of equivalent size can enter the iteration in exchange for the not yet started feature (however many times I teach the same approach to the Scrum teams I coach, especially when the environment allows and it does not become the norm).

However the biggest difference between Scrum and XP is that Scrum is basically focusing on how to organize the work, but it does not say anything about how to do the work, XP instead is more focusing on practices to develop high quality software effectively.

At the same time I learned (and thus agree with Dave Rooney) that Scrum is not enough to sustain an effective Agile SW development. Scrum is a great starting point if you want to learn how to be Agile, but I saw many Scrum teams, getting more mature with Scrum and not improving their performance overtime without using proper technical practices, because they accumulated a too big technical debt. Instead they have to learn to build the product in a different way.

Scrum explicitly avoids prescribing technical practices, but a Scrum team will hardly succeed without using certain XP engineering practices, like Test Driven Development, Continuous Integration, Pair Programming, Collective Ownership, Coding standards and Refactoring.

Effective Scrum and XP together can really unleash the full potential of a team, allow delivering effectively high quality products that customers love and stay true to the original spirit of the Agile manifesto.

Scrum is Lean

I believe that Scrum is the most Lean among all Agile methods. 

scrum by David / CC BY 2.0

And not only because the word "scrum" was first used in 1986 by two professors at Hitotsubashi Univeristy in Japan, Hirotaka Takehuchi and Ikujiro Nonaka, in their famous HBR-published article             "The new new product development game".

In 2003 Mary and Tom Poppendieck coined the term Lean Software Development by publishing a book with the same name and proposing a translation of Lean thinking and practices to the software development domain. Lean Sofware development is based on 7 principles, which may be implemented by means of a number of tools:

1. Eliminate waste
2. Amplify learning
3. Decide as late as possible
4. Deliver as fast as possible
5. Empower the team
6. Build integrity in
7. See the whole

If you want to know more, Mary and Tom published two more books:

• Implementing Lean Software Development: From Concept to Cash
• Leading Lean Software Development: Results Are not the Point

Now, even though Scrum and Lean Sofware development might seem to have formally nothing in common (different principles and different practices),  I will try to explain why and how I think the seven Lean SW Development principles are somehow mapped (or at least found) in Scrum.

1. Eliminate waste

The book reports seven wastes in SW development: partially done work, extra features, extra processes, task switching, waiting, motion, defects.

I think that the Scrum framework as such contributes to eliminate those sources of waste:

- the iteration-based approach, through the use of the Sprint Backlog, and the focus on doing whatever it takes to get things done and transform the items in the Sprint Backlog into a Potentially Shippable Product Increment, contributes the eliminate partially done work

- the Product Backlog handling, by building first the features which are more valuable for the customer and not spend too much time on the ones that are coming later, contributes to minimize partially done work and not needed features

- self-organized teams, who have the best knowledge, who are empowered to decide how to do the work and who continuously adjust their process through Sprint retrospectives, contributes to remove extra processes

- teams focusing on the Sprint goal minimize task switching

- cross-functional development teams working together side by side and daily with the Product Owner can remove waiting and handover of work items

- high skilled development teams, feeling responsibility for product quality, minimize defects

2. Amplify learning

Scrum is all about seeing SW development as a learning process. The Sprint will help create knowledge both on the product and on the process: the Sprint Review and the Sprint Retrospective are the main points in time where the knowledge on the product and the process are built respectively via feedback loops. The Daily Scrum also creates knowledge about how far the team is from the Sprint goal

Cross-functional teams learn continuously from each other skills and share knowledge inside the team and between different teams.

3. Decide as late as possible

A Product Backlog which is more fine grained on the top and more coarse on the bottom and “just enough just in time” planning via a Sprint based approach, allow a Scrum Team to reduce complexity and defer decisions which are not necessary at a certain point in time.

They can keep instead options open up to the last responsible moment: they can use time to collect more information and learn more about the product and the context in order to take better decision later, based on facts. They will of course learn fast by means of quick feedback loops provided by the Sprints.

This allows a Scrum team to be more flexible to react to the changes that will surely happen in the market and in the technology: they can truly embrace change inside their process of developing SW.

4. Deliver as fast as possible

Short delivery cycles represented by Sprints allow deliver quickly to maximize return on investment, get feedback from real customers and users and reduce risk that you won’t deliver a product that meets customers’ needs, due to misunderstanding or any changes in markets and technology.

A team focusing only on delivering the Sprint backlog will produce potentially shippable SW increments faster.

5. Empower the team

Scrum is all about empowering self-organized teams to find the best way to accomplish work. Self-organized teams can leverage all members’ talents and intelligence in the best possible way to achieve their goal as fast as possible.

Real empowerment is achieved by having the Product Owner to decide the “What” and the “Why” and explicit it by means of Product Backlog items in the form of User Stories: the development team will then decide the “How” and find the fastest possible way to get a User Story done.

6. Build integrity in

In a waterfall approach, piling up code without testing often result in a very expensive re-factoring effort, due to bugs, but especially to requirement change or misinterpretation.

Quality in a complex environment like SW development can be built only via iterative and fast feedback loops. A Sprint provides the main feedback loop mechanism in Scrum. Inside the Sprint there might be other even quicker feedback loops:

- the Product Owner (customer representative) works together with development: they are part of the same Scrum teams and work daily together: they collaborate on the Product Backlog and the real Product and this ensures a continuous feedback loop on requirements and system behavior

- Scrum teams adopting XP practices perform continuous refactoring, continuous testing, and continuous integration, which are Agile ways to build quality in.

7. See the whole

End-to-end Product Backlog items (slicing the product end-to-end vertically and explaining why) and cross-functional teams allow looking at the whole system: they can adapt the work items and the process to maximize the value delivered and achieve the Product Vision in the fastest possible way.

An empowered and self-organized team, working closely and daily with a customer representative (the Product Owner), have the possibility to look at the whole development process, find the weakest link or the bottleneck at hand and do whatever they see as necessary to fix it.

They have enough visibility of the whole to avoid sub-optimizations which typically, in traditional environments, try to optimize some small part of the system, which may not help the whole system.

If you want to implement a Lean thinking in your organization, try to practice effective Scrum first and you will get an extraordinary teacher. Scrum is Agile and Lean.

What do you think?