How to Scale Agile?

Scaling agile is not simple. The complexity increases not in proportion to the size of the team but at the square of the size of the team. That means a 20 member team project will be 4 times as complex as a 10 member team.

To ensure that complexity does not turn into chaos it is inevitable that the pure agile way of working may have to be sacrificed – is it possible without violating agile manifesto? It is also inevitable that the agile practices will need to integrate into other practices of the enterprise.

To be able to scale agile we need to understand:

  • What the dimensions to this complexity?
  • How to split the problem and the team?
  • How to introduce processes and tools?
  • How to manage collaboration among distributed team members?
  • How to plan and manage workflow?

Dimensions of scaling and the need to integrate with other enterprise processes

Scaling Agile: An Executive Guide – Scott Ambler

To scale agile to an enterprise level, eight dimensions of complexity needs to be handled.

  1. Team size – from less than ten to hundreds
  2. Geographical distribution – from collocated to globally distributed
  3. Regulatory compliance – need for mandatory compliance
  4. Domain complexity – straightforward to complex and emerging
  5. Organizational distribution – from all in-house to multiple external parties
  6. Technical complexity – from homogeneous to heterogeneous including legacy
  7. Organizational complexity – from flexible to rigid
  8. Enterprise discipline – from no external constrain to need to adhere to enterprise standards

Different options of splitting the teams

Scaling agile teams – by features or by component? – David Draper

There are four options to split a large team into smaller sub-teams

  1. Component orientation: (Pro) Typically this is how developers align themselves. (Con) Need to split user-centric feature – focus shifts away from business value.
  2. Discipline: (Pro) This is how enterprises are already organized. (Con) Risk longer lead times with a “throw over the wall” attitude.
  3. Location: (Pro) Collocated teams tend to be more effective. (Con) Need to match feature set with specific team.
  4. Feature: (Pro) Aligns well with goal of delivering value. (Con) Integration issues may arise.

In practice a mix-and-match approach may be best where the split is done primarily based of feature and location.

Harmonizing the tool and process dimension with agile practices

Scaling Agile for Project Teams – Alan Bustamante & Rahul Sawhney

There is a need to harmonize three dimensions of people, process and tool. Here are some of the recommendations on how to achieve it.

  1. People issues are best tackled through structure, empathy, communication, and people-oriented policies.
  2. Addressing process challenges for scaling requires organizations to understand their process focus and validate that it is aligned with the organization’s vision, mission, and values.
  3. Focus on tools that support and facilitate the team and sub-team’s processes by eliminating or minimizing non-value added work.
  4. Configuring tools around a flawed process will still yield a flawed process. Fix the people and process issues first.

Making distributed teams effective through communities of practice

Scaling Agile to work with Distributed Teams – Mike Cohn

Total team size increases complexity. Distribution increases complexity. It becomes impossible for all team members to know each other let alone trust each other. One way to build the bridge is to set up communities of practice.

  • A group of likeminded or like skilled individuals which is self-organizing, organic and can span projects / location
  • All communities need not be officially sanctioned – they can completely informal to highly institutionalized
  • Creating proper environment for the communities is important

Managing the product backlog at multiple levels

Scaling Agile Processes: Five Levels of Planning – Hubert Smits

Agile planning activities for large-scale development efforts should rely on five levels:

  1. Product Visioning
  2. Product Roadmap
  3. Release Planning
  4. Iteration Planning
  5. Daily Planning

Product Vision is more like a desire of what the product will deliver once it is finished and would not talk about specific features. On the other hand Product Roadmap would talk about high level product backlog from which backlog for lower levels will be derived. It will communicate when releases are needed, what would be sufficient functionality and what business value would be delivered.

Like the two previous levels Release Planning is also for all the teams. Releases are defined by fixed date, theme, and planned feature set to which all teams must commit to. Typically, the product release would be once a quarter and the iteration cycle would be 2 to 4 weeks.

Managing workflow between multiple teams using Kanban

Large-Scale Agile – James Shore

Use Kanban for workflow not within a team but between teams.

  1. Treat each agile team as a work cell
  2. Use Kanban to manage cross-team workflow
  3. Create a Product Portfolio team which is collocated, cross-functional, intact team consisting of product managers, business experts, UX designers, architects etc.
  4. Partition the teams in such a way as to minimize the dependencies
  5. Monitor the system using lean techniques which is Visual control, Throughput and Value-stream mapping
  6. Sacrifice reuse in favor of throughput
  7. Keep communication flowing with Scrums of Scrums

Finally…

Since, to scale agile, we may have to deviate from the spirit of the original manifesto, the question is how much can we deviate and still call the resultant methodology agile? And, when does it become an iterative development methodology?

Incidentally, iterative development seems to be as successful as agile development. Have a look at this survey outcome.

Advertisements

Agile Projects are more Successful – Survey Result

Survey conducted by Scott Ambler has consistently (2008, 2010 & 2011) shown that Agile and Iterative Projects have been more successful.

Apart from the fact that Agile has been consistently been more successful compared to traditional approach, these survey result show two more interesting result – one of them is very surprising.

1) The gap between “Iterative + Agile” and “Traditional + Ad-hoc” has been increasing

It is not clear why there is a dip in the success rate of all types of project in 2010. But, leaving that aside, the gap has been consistently been increasing (2008=7%, 2010=13% and 2011=18%).

So, does it mean that people have really mastered the art of how to manage Agile and Iterative projects?

Or, does it been, people who are good at project management have abandoned Traditional approach and migrated to Agile or Iterative approach?

2) Iterative approach has consistently performed better than Agile

Though the difference is not very significant, all the 3 survey has indicated that iterative is slightly more successful compared to agile.

Does it imply that all the stuff in Agile Manifesto about “People over Process”, “Co-located Team”, “Face to Face communication”, “Cross functional team” etc. has very little to contribute to project success? When you scale Agile, you may need to violate some of them anyway.

Agile@Scale ≠ Agile@Manifesto

Word of Caution from Scott Ambler

He is quick to point out the known challenges with any survey.

  • You will only get responses from people willing to be surveyed
  • You risk getting responses from people with strong feelings about the topic
  • Very often questions capture opinions, not facts
  • The biases of the communities will be reflected in the results

This means (1) your survey sample may not represent the real world, and (2) your result may be based on opinion and not fact.

BUT, the question that we need to ask ourselves is…

…Has the time come for us to examine, which of the agile practices have a +ve correlation with project success and which are only rituals with no impact on the project success?

Scaling Agile – Is it possible without violating Agile Manifesto?

In short the answer is a big NO.

Agile@Scale Agile@Manifesto

Why do I say that Agile@Manifesto cannot be scaled?

Apart from what is explicitly stated in the manifesto, there is an unstated belief that “any” software can be developed by a small group of highly talented & motivated individual as long as they are sitting within shouting distance, have continuous access to people who can decide what the software should do and they can work the way they deem fit. This belief is elaborated through 4 value statements and 12 principles behind it. These 16 statements can broadly be classified into 5 categories of “how to” or “preferred method of” …

  1. …Interaction and Collaboration (4)
  2. …Measure of Progress (4)
  3. …Responding to Change (2)
  4. …Forming Self-Organizing Team (3)
  5. …General Gyan (I mean advice) (3)

However, when complexity increases beyond a certain point, it becomes infeasible for a small collocated team to handle it. The reason can be many. For example, you may not find enough people like Dennis Nedry (Jurassic Park) “…who can network 8 connection machines and debug 2 million lines of code…”

Or, you software size may become much more than 2 million lines of code.

Or, you are not able to, for whatever reason, to assemble the whole team in one location.

Or, you are unable to find “business people” who know everything about what is needed and can take an instant decision.

I am sure you can think of many more reason of why enterprises have to go beyond one co-located cross-functional team.

When you scale, you will have multiple teams and they may not be located in the same place. So, let us examine the statements in each category and see if they can be applied in the context of a large and distributed enterprise engaged in software development and what their limitations are.

Interaction and Collaboration

  • Individuals and interactions over processes and tools
  • Customer collaboration over contract negotiation
  • The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  • Business people and developers must work together daily throughout the project.

When you have teams distributed across multiple locations is it possible to function effectively without a clearly defined process (which everybody understands) and without having a tooling & technology infrastructure (which facilitates interaction among individuals across teams)?

When you have the involvement of multiple organizations, is it possible to start work without a clearly defined contract?

When you are targeting applications which potentially have thousands of users, will it be possible for few “business people” to define what the software should be? What about those pieces of software which is only interacts with another piece of software – what role will “business people” play in specifying them?

Is it always feasible to get the right “business people” to interact with the developer on a daily basis?

In short:

1) Tools, processes and contracts are the backbone of any distributed large scale software development project. Without them interaction and collaboration will collapse.

2) To say that you need only business people and developers to complete a software project is very simplistic. You need specific expertise from different fields like usability, architecting, performance testing etc.  

Measure of Progress

  • Working software over comprehensive documentation
  • Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
  • Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
  • Working software is the primary measure of progress.

If “working” and “valuable” software means “potentially shippable” software, then can customer always take continuous delivery of such software?

In a complex project which has many components and tiers, is it always technically feasible to partition the application in such a way as to deliver “working” or “potentially shippable” software?

Can you take up large scale software development without comprehensive documentation of all the interfaces – interfaces between different software components – interfaces between the human and the system?

In short:

3) Slicing a complex software project into working, valuable software may not always be feasible. Attempting to split it into testable chunks is more practical.

4) Measuring progress only through working software is too limiting. There are so many other things that we need to keep track – Performance tuning, UI design, Prototyping, Code refactoring …

Responding to Change

  • Responding to change over following a plan
  • Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.

In the real world where the delivery of the software is tied to many downstream activities – is it always feasible to change your plan?

Can you always makes changes to the software late in the development based on changed requirement and still guarantee the reliability of the software?

In short:

5) Any change that is required to be done on already developed software has potential side effect. Any decision to incorporate such change needs to carefully weigh the potential cost and benefit.

Forming Self-Organizing Team

  • Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  • The best architectures, requirements, and designs emerge from self-organizing teams.
  • Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

Is team of motivated individual always sufficient to get the job done – doesn’t skill, experience and knowledge play a role?

Is there any guarantee that a group of motivated individual will always self-organize into an efficient and cohesive team?

Is there any hard evidence to support the statement that “the best architectures, requirements, and designs emerge from self-organizing teams” – especially when we are talking about many teams?

Isn’t “maintaining a constant pace indefinitely” a utopia – haven’t we heard of things like loss of interest, fatigue, boredom, attrition etc.?

In short:

6) It is naive to assume that any team of motivated individual will always self-organize. The team may require mentoring, periodic review of effectiveness as so on.

General Advice

  • Continuous attention to technical excellence and good design enhances agility.
  • Simplicity–the art of maximizing the amount of work not done–is essential.
  • At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

These are good advice and can be applied to anything.

Finally

So, if Agile@Scale ≠ Agile@Manifesto, what is it?

Since we don’t have any well accepted definition of agile to fall back on – we need to go by what the popular opinion is.

Everybody agrees that agile means “iterative development”, “progressive refinement of the requirement” and “emergence of the design”.

Most people want the teams to be “self-organizing” but some people are totally silent about it.

Many people feel that there is more to software development than just “working code” and we need to look at how the software development gels into the overall enterprise context. There are also suggestions that we should borrow idea from traditional software engineering techniques where appropriate.

In short:

As long as you…

  1. …Develop software iteratively,
  2. …Have multiple teams of small size and allow them to self-organize, and
  3. …Do not try to document the complete requirement and design upfront

…you can assume that you are following Agile Methodology.

Agile practices now have research support

Adam Smith was wrong. Well … he was not wrong in his conclusion but he was partially wrong in his basic assumption that human always pursue their self-interest.

Through the work of many scientists, we have begun to see evidence across several disciplines that people are in fact more cooperative and selfless—or behave far less selfishly—than we have assumed. In fact, recent research shows that in any society majority of us behave cooperatively rather than selfishly (though some people do behave selfishly).

The essence of agile is iterative development and self-organizing team (What makes Agile agile?). Latest research suggests that iterative approach with trial and error is the best way to navigate through our environment which has become exceedingly complex. Such research is inspired by biology and evolution.

Now you have research evidence that we are indeed tuned to work like that. Here are the references to 2 books and couple of article which highlight the latest development in this field.

However, what surprises me most is that there not a single reference to agile manifesto or agile development methodology in any of them. All of them invariably refer to Wikipedia and Open Source movement but I have not seen any similar publication referring to Agile.

The Unselfish Gene

Ted Cadsby, in an article in HBR, neatly summarizes the status of research in this field. He points out the following:

  • In experiment about cooperative behavior – in no society examined under controlled conditions have the majority of people consistently behaved selfishly. Most organizations would be better off helping us to engage and embrace our collaborative, generous sentiments than assuming that we are driven purely by self-interest.
  • Wikipedia and open source movement – people contribute because they want to cooperate. The interpretation that the voluntary contributions of participants are an attempt to improve their reputations and long-term employment prospects has been refuted by the decade of empirical research.
  • Cognitively and emotionally, we may be able to “feel” what others are feeling – Neuroscience also shows that a reward circuit is triggered in our brains when we cooperate with one another, and that provides a scientific basis for saying that at least some people want to cooperate, given a choice, because it feels good.

Agile proponents have always maintained that “Self-organization” through “Collaboration” and “Trust” is the key to successful software development. If you look at these 3 of the 12 principles behind agile manifesto, you will notice that it revolves around trust, face-to-face conversation, motivated individuals and self-organizing teams.

  1. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
  2. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
  3. The best architectures, requirements, and designs emerge from self-organizing teams.

The article goes on to point out 7 building blocks of cooperative systems. The first building block is Communication – in agile manifesto the first item is just that “Individuals and interactions”.

There is another article by the same author which emphasizes that many of our current problems are too complex to be reduced to a single-cause explanation. Complexity arises from the interconnections between things — how parts within a system interact via intricate feedback mechanisms. The information signals we need to make sense of complex things are buried in a lot of noise, and we, unfortunately, are not adept at digging for cues.

Adapt: Why Success Always Starts With Failure

This book is by Tim Hartford, the author of “The Undercover Economist”. If you have not read the book the here is a nice summarization of the key ideas. There are references to numerous recent researches in this field. The main ideas brought forward in the book are:

  • Astounding complexity emerges in response to a simple process: try out a few variants on what you already have, weed out the failures, copy the success – and repeat forever. Trial and error is a tremendously powerful process for solving problems in a complex world.
  • Given that life is so unpredictable, what seemed initially like an inferior option may turn out to be exactly the solution we need. It’s sensible in many areas in life to leave room for exploring parallel possibilities.

In addition to these the author makes a case for making systems loosely coupled so that failures can be tolerated. This will ensure that the cost of one failure is not too much and it does not bring down everything with it.

The key to Agile is “iteration”. The third principle states “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.” The emphasis is on doing – learning – adapting.

The Origin of Wealth: Evolution, Complexity, and the Radical Remaking Of Economics

This book, written by Eric Beinhocker, is rich in survey of new ideas in economics, incorporating discoveries from science and psychology. This presentation highlights the key ideas in the book. The key premise in the book is that our economy is a complex, adaptive system which implies that:

  1. Equilibrium, and with it much of the economic thinking that has dominated the past century, is out.
  2. A new tool set with its own techniques and theories are available to explain economic phenomena.
  3. Wealth is a product of evolutionary processes.

This book also emphasizes on the need to iteration and self-organization.

Here is a diagram from the presentation showing how cycles have evolved.

Is it Blasphemous to Criticize Agile?

This is how I was planning to start my post …

“…believe me … I do think agile works … in most … well if not in most cases then in many cases. That by implication means it is not a silver bullet for all situation. In fact, no technique, no methodology, no solution can be applied all situations and agile is no exception.

However, I have noticed a tendency among agilist to proclaim that if agile has not worked in a specific situation then the fault lies squarely on improper usage rather than any limitation of agile. All you have to do is apply agile properly and it would work. Who is to say what proper agile is? Agile manifesto does not help in determining what makes agile agile. Conversely, how do we judge what agile is not…”

…but then I saw this – Agile’s Teenage Crisis?. It provides 20 point criticism of Agile and how it is practiced. It is nice to know that the thought leaders are thinking about the same issues.

I don’t want to list the points – you better read the post.

However, there is one point that has been bothering me for some time and it is not fully addressed in the 20 points mentioned.

The point is about “Continuous delivery of valuable software”. Can we compare this to “CEO working to create value for shareholder”?

Do you see any similarity?

Both focus on the short term. CEO focuses on the next quarter result and its impact on stock price. They assume that anyway on the long term everybody is dead. This post nicely summarizes the dilemma of the CEO.

However, visionary CEO’s will always balance the short term and the long term.

Agile developers concentrate only on the current sprint and assume that if there is any architectural problem later code refactoring can take care of it. Big upfront design is frowned upon. Similarly if the agile team has experienced developers who can visualize the impact of every design decision things generally work out.

However, are all developers all so experienced that they can intuitively figure out what the right design is? How about the need to debate, introspect and visualize the impact of any design decision? Can it be done it your focus is always on delivering the next user story and how much business value it delivers?

Quoting Philippe Kruchten from Agile’s Teenage Crisis?

“…While agile practices could help control technical debt, they are also often at the root, the cause of massive technical debt…”

Agile development and Enterprise Architecture practice – Can they coexist

Your reaction to the title is probably one of the following:

  1. Why should it be a challenge?
  2. Co-existence would be impossible.
  3. There will be difficulties but it can be done.

Why should this be a challenge?

After all, EA is agnostic to software development methodologies. Irrespective of which EA definition is accepted, no direct link with any software development methodology exists. Therefore, you would argue, that there would be no conflict between any Enterprise Architecture practice and adoption of agile development methodology. (If you want a quick premier on what EA is then have a look at: A Comparison of the Top Four Enterprise-Architecture Methodologies).

If you are thinking along these lines then I think most Enterprise Architects will agree with you.

Co-existence would be impossible

EA and Agile comes from different mindset and different world view. The challenge of co-existence is nicely captured by Dave Nicolette. This is what he has to say about it (see this):

“…the mission, scope, tools, mentality, culture, and personality types of enterprise architects and agilists are so radically different …. not only is the nature of their work different, but the management styles, employment incentives, and career path options that work with one group don’t work with the other …”

In EA you start with a business goal and from the goal work out what needs to change (people, process, technology, application, data …). As the next step you spell out in complete detail, how the changed scenario will look like. This is to be followed by an implementation plan, execution of the plan and proper governance. The approach is top-down, structured and planned.

In Agile, though you have a business goal in sight, the approach is always evolutionary. You take one step at a time, delivering working solution and reorienting you plan as learning from that step. You create the environment for the team and empower them so that they can self-organize and be more productive.

If this was your answer then most agilists will agree with you.

There will be difficulties but it can be done

Since there happens to be a common ground – both EA and Agile believes that delivering business value is the key – it should be possible for them to co-exist. Not only can they co-exist, they can also complement each other. It is explained succinctly by Ronald van Luttikhuizen (see this):

“…enterprise architecture (top-down), works toward a long-range vision … (agile – bottom-up) addresses questions coming out of projects right now – questions that should be answered quickly … but if either the top-down or bottom-up architecture is missing, you’re not going to end up in the situation you want…”

As stated earlier, in EA you always start with a future business scenario work out what you need to do to achieve that. Similarly, the first two of the 12 principles of Agile Manifesto talks about “…delivery of valuable software…” and “…the customer’s competitive advantage…”.

If you are thinking like this then you are in the august company of Scot Ambler who is the Chief Methodologist for Agile and Lean within IBM Rational. Here is what he has to say (see this for detail)

“…when project teams work under the assumption that they can do anything that they want … chaos typically results … although each individual project may be very successful, as a portfolio they may have serious challenges…”

Now let me add a variant to this thought. I am sure I will invite the wrath of both enterprise architect community as well as the agile community for saying this but I think both EA and Agile have a UTOPIAN world view. In the real world you have to find a middle ground.

Neither emergence nor up-front design can be the right answer to all problems. A middle ground, a HYBRID approach is what is needed.

Imposed structure may be right in some situation; self-organizing team may perform better in other. It is important to be ADAPTABLE so that you can judge which way is better for a given team.

I am sure if you are a realist who understands the dynamics of a large enterprise, you will agree with me.

Agile Trends – Minus the Hype

Surprise, surprises … Agile has never appeared in Gartner Hype Cycle for Emerging Technologies. So, the task of separating the hype from reality becomes simpler. The reality, Scott Ambler says, is that “…you’d have a hard time these days trying to find people who don’t want to be agile…”

Agile is like starfish – you can cut one arm of an (starfish) Agile methodology and let it grow to (a full starfish) a tailored agile methodology to suited for your needs.

Now coming back to question 3 & 4 – [You need to read this post in conjunction with my earlier post where I had raised 4 questions and answered 2 of them].

3. If the current trend continues then where will it be in one year time?

For the majority, there will be 2 distinct style of agile adoption where the focus will be on …

  1. checklist based adoption: as long as you follow a series of steps recommended by the chosen methodology …
  2. iterative development: as long as you develop in iteration where complete specs have not been written down up front …

…it would be deemed that you are following agile methodology.

However, few questions will need to get answered:

4. What happens if you take no action on the specific technology for next one year?

This one is easy – if you take no action then you would have postponed the inevitable by one year.

The good news is that you can do it in your own way – the way that suits you the most.

What is Agile Methodology?

Agile has taken many forms and beyond the Agile Manifesto there is no commonly accepted definition of agile (What is the Definition of “Agile Methodology”?). There are so many different methodologies which are classified under agile – these methodologies have very little in common among themselves except that all of them recommend iterative development lifecycle.

Does any methodology which follows iterative development with evolving requirement become an agile methodology? Is self organizing team a necessity? Forrester, in a survey, has grouped “Lean” and “Six-Sigma” under agile but have a separate category for “Iterative Development”.

Has Agile gone main stream?

Forrester says “Agile Development is rapidly becoming the Norm”. As per their survey report on Agile Development Management Tools, Q2 2010, 35% of the organizations surveyed described Agile as their primary development tool. Another 16% uses iterative development. Waterfall represents only 13%. Here is the chart:

Infoworld.com had asked the question Agile programming 10 years on: Did it deliver? In this interesting blog post by Paul Krill, the answer is mostly on the affirmative. However, in another post
Bob Lewis raises a concern “…I hear about how it’s being taught as a series of steps you have to follow, not as a style of relationship management…”

About Starfish

Starfish (sea stars) are beautiful animals that resemble a star. Like agile methodologies they come in a variety of colors, shapes and sizes. They have the curious characteristics that not only can they regenerate lost arms but also some can even regenerate an entirely new sea star from just one arm and a portion of the star’s central disc.

Here is couple of references: