Evolving & Emerging Architecture: Agile-EA

Going into companies is always an interesting experience. You get to view the way they work and in agile organisations using physical boards, you can walk the floor and view how the work is progressing. That is well known in the agile project management arena, product owner roles etc. but it's also an extremely useful technique in the Agile Enterprise Architecture world (Agile-EA). Before going into why, we need to recap a couple of definitions.

1. Conway's Law

In 1968, Melvin Conway provided this now tried and tested gem.

"organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations"

This is a huge revelation for some companies, but it got hidden by process oriented techniques, which masked the problem for decades.  

In the agile space, it's a very different story. The reliance on teams to develop their own processes, and having the greatest expressive power within them as opposed to across them has resulted in this being more applicable than ever. You can turn this to your advantage and having elements of a workflow which make up the greatest proportion of it, contained within teams really improves the rate of delivery of work, as I have covered in previous posts and is a much better facilitator of outcome than having a distributed development team. This also means that cross-team communication encourages the development of a system through that communication channel.

2. Consumer Driven Contracts

A consumer driven contract can be considered a Kanban pull signal. This pulls a feature from another team and in software development spaces, this also includes a sub-suite of tests from the consuming team. When a feature provided by another team is needed, a cross-team card, often of a different colour is placed on the team board who are to provide that functionality (at Laterooms,com for example, they used rainbow coloured cards at one point). That team then pick the card up and code against the tests to make them pass. 

Consumer driven contracts are pretty much the most central concept in true just in time, cross team software delivery. They often have a subtask link to the source board and effectively provide an architectural link between application components or business features.

This provides opportunities for architects in both the enterprise and/or solution sphere to see how their estate is evolving by simply going to all stand-up meetings, and building a picture of the emerging architecture or estate by following the chain of tickets. Once such a ticket appears, this introduces an architectural dependency and over time, more and more tickets appears which fill the space between things that architecture lives in.

Walking the Floor

Architects live between things. As architects we should aim to get to every stand-up within our sphere. When we do, we should look out for these cross team tickets, or features being played which we are aware touch something else or perhaps could leverage a feature elsewhere in our estate. 

Imagine we go to the stand-ups and see the following boards after, iteration/sprint 10. 

Sprint 10 board states

Sprint 11 sees a purpose ticket appear on team board 1, required by team board 2 (who are pulling it). The purple ticket represents a cross-team 'pull' (indicated in team 2's 'X' ticket for convenience) since it requires a feature from team 2. The dotted line now represents the relationship between the two teams, which remember by Conway's law represents a link between the system or business features:

Sprint 11 emerges an architectural link

This continues to be played and supposing by the end of Sprint 11 it and its compatriot pull are both done. This ticket is now complete and the architectural link has been delivered. So the architecture of the estate now looks like the following (Arrowed lines represent a dependency):

Sprint 11 Architecture - Note, no Team 4 features live, so architect doesn't care

Sprint 12 and two purple tickets appear. One from team 3 to 1 and one 4 to 2. The solid line represents the delivered link from Sprint 11:

Sprint 12 Architecture

When this is delivered, the architecture then looks like:

Architecture after Sprint 12

And so the cycle continues. 

Conclusion

Each one of the dotted lines represents a communication flow between two teams and brings them closer for that phase of the development. When it's done, the link still exists in the system (solid line), but they can separate and form relationships with other teams, as team 1 did with team 3, after team 1 finished with team 2. Of course, there are times when the platform will need to delete those links, removing them from the estate, but the communication still happens regardless. 

As an architect, attending the stand-ups is a great way to see how the system evolves. You can walk the boards looking for these special cards and draw out the architecture as you proceed. Indeed, if you're an active contributor (and you should be) you can and should facilitate the teams to make decisions which are systemically optimal and forming links when the team could otherwise expend a lot of effort would save them and the company time and money. As the organisation matures and the role of software architect embeds in the teams (as roles over job descriptions), it is these individuals who will go to other team stand-ups and hence, facilitate the creation, updating or destruction of these architectural links.

Q: What Do Agility & Astrophysics Have In Common?

There is an agile coaching game called the static points game. It's an extremely useful illustration of how complexity evolves from really simple rules which in the enterprise world, shows how businesses always change under multiple forces, which should be familiar to those in the change management space. I've played it twice, the first was an introduction by Ash Moran whilst I was at Laterooms.com and more recently with Ian Carroll. It's pretty simply, to play:

1. Get everyone to stand up and move to the edge of the room (or form a circle if the room is too big) 
2. Tell each person to pick two other folk from the group 
3. The simple rule is to stay equidistant (the same distance away) from both of them. 
4. Then let them go.

What you'll see is the group organise and shift about, jostling as the distances come to an equilibrium, then eventually settle. You can play it again, telling people to keep the original two people,  and see where they settle this time. Chances are they settle differently from where they did previously (a digital camera might and high angle come in handy for this variation of the game :).

Reset the game, and with everyone keeping their two folk, fix any one person from the group where they are, perhaps using a chair and send everyone else back to the edge of the room and play it again. They settle quicker. Do it again with that one person fixed, and photograph. You can keep fixing more and more folk and the organisation comes to settle much quicker, with much less movement.

What does this Illustrate?

As an abstract systems game, it naturally covers a multitude of arena!

• How departmental level business changes relative to other departments as politics plays a part in how departmental heads compete for work or pass blame. Imagine the people in that game are working in an organisation and trying to balance the needs of two sets of stakeholders.
• How uncoordinated systems work with one another as they evolve (which I believe is what you think this refers to here). Imagine the people in that game are subsystems taking with interfaces to two other systems.
•  How uncoordinated work-streams work with one another as they evolve (in agile environments, this is what I think happens with shared systems. They pull against the shared systems). Imagine the people in that game are subsystems communicating with interfaces to two other systems.
• It shows how complexity can manifest from a really simple rule-set. This one is self-explanatory. Intelligent agents (i.e. people, bees etc.) using a really simple rule can still produce a significant amount of very complex behaviour. This, like all the rest, is called [mathematical] chaos.
• It shows that relationships are easily equally as important as the entities themselves
• How organisms relate to one another
• It is a manifestation of planets being influenced by each other’s gravity
• …

Mathematically speaking, they are ALL a manifestation of what is called the n-body problem. The planetary example that ends that list above is where this originated.

A Lesson in Planetary motion

We orbit the sun because our mass is significantly less than it. The effect we have on the sun is near negligible. This is like a big CEO of a company, keeping things in order by imposing forces upon the lower weighted levels, who can’t respond in any meaningful or significant way. However, the behaviour of the system is predictable and has been like that for billions of years. With the big, overarching, autocratic CEO (or C-Suite) in it, and with the absence of any other influential factors, the environment rarely changes, so there is no need for it to change. That is one sun and several planets and moons and their orbits (statics and dynamics) that systemically stay the same for millions if not billions of years, even if what goes on on the surface changes. As far as systems are concerned, architecturally, these are all static points. That’s a high level block diagram!

However, in agile environments, you are empowering folk, quite rightly. Hence, the gravity they have and are allowed to have, relative to the system, is much higher. However, returning to the n-body system, If you have two equally weighted planets orbiting around each other, they will pull each other’s orbits. If you have three orbiting each other, it has been proven that the behaviour of an unconstrained system, is near unpredictable aside from very restricted contexts (i.e. akin to how often waterfall delivered on time and on budget, which some would argue has the same probability that our solar system came into existence the way it has :) Plus, because the class of problem is the same for all of the above, this chaos or ‘randomness’ applies to all of the above examples too.

OK, how can you test if the result is random?

Firstly, this is a complexity problem. You can't necessarily test the system internals [aka business] as a whole if the result isn't predictable or consistent. However, what you can test, is that the unit which is the individual person, does manifest the rules correctly! i.e. they keep the same distance from their two folk at all points of change, including all points of jostling! After all, most software systems test that rules manifest correctly. For example, you can't open a bank account if you don't have ID or you can't board a plane for an international flight without a passport. Each and every one of the individual entities, as well as the whole in deterministic systems, is defined by: 

• A pre-condition, which includes the initial state of the system - The position they are in in the room 
• An action - Someone moves
• A post-condition - They have to remain equidistant

With an invariant that they have picked two constant people to apply the rule with.

Which in BDD/Gherkin syntax is akin to:

Background each person has two different folk to focus on
##...

Given the person is in the room
When someone moves
Then the person has to be the same distance from their left-person and right-person

Remember, a static point is not just the structure or entity, it is also the behaviour it exhibits. Hence, the best way to make a change and make it testable, with the minimum of risk. is to nail all but one of the folk (read systems, which include the entities and how they behave – that is the aggregate of business, application, data and technology for each feature), make that small change, test they manifest the rules, then release everything, make sure the system didn't disintegrate (i.e. all the other elements correctly adhere to their own rules, which may be the old ones), then for the next change, nail all but a different one, make a change, release etc. The key is to test that the one body itself adheres to the simple rule you want of it! This provides parallels in systems?

For example, if in the static points game above, if we took just one person (which is a business unit or capability), let's call them 'delta' and made the rule that they stay an arms length away from everyone else and can pull the folk together if they are not, then played the game again. They may get jostled about a bit by the rest of the business going about its [old] business, which is still testable, whilst simultaneously pulling the two folk to their arms length distance. Note, through pulling folk, the rest of the system, that means folk who have picked one or other or even both of the people pulled by delta, and their 2nd, 3rd and nth degree of separation, will also change. So this new rule influences the system as a whole, but crucially, the rest of the system adhered to the old 'equidistance' rule, which like the new one, you can still measure individually (as mentioned at the top of this section). You measure that Delta is doing their job by ensuring they keep their two within 'punch distance' at all points of change, i.e. jostling.

Conclusion

Trust me, this is a very difficult concept for some folk to get and it requires some 'micro-thinking'. Indeed, it always provides a learning point in communication to me. Whilst there is almost nothing anybody in the IT world can tell me about non-linear system dynamics, there is a lot I have to learn about communicating the concept across in language people understand. That is why I attend community events, such as TechNights, Lean Agile Manchester etc. it’s to learn to effectively communicate the ideas to people who do not have that bridge or background. I've been through this sort of discussion a couple of dozen (or more) times and I still communicate this wrong now, because there may be levels of knowledge or skill between the person I am trying to communicate this to and the understanding of non-linear dynamics that would help illustrate the benefits of the knowledge. I'd be interested to see how others communicate this to analytical and non-analytical audiences alike.

So the best I can do for now is probably illustrate it with video. In the external links below, take a look and see if the systems look the same as each other after they've been running for a while. Indeed, you can run the YouTube vids side by side if your broadband is up to it.

External Links
n-Body Gravity Simulation like folk at the edge of a room - https://www.youtube.com/watch?v=XAlzniN6L94

n-Body simulation with 50 million entities - https://www.youtube.com/watch?v=OJaE9J39A8s

VIew these two 3 body problems side by side:
https://www.youtube.com/watch?v=VX9IdCnNWJI
http://vimeo.com/11993047 (from 24 seconds in - This also has multiple runs with different starting points)