Tackling Complexity in Projects & Enterprises

There is an analogy doing the rounds at the moment which I think is pretty good as a necessary intro, but not at all sufficient to understand some of the complexity enterprise programmes and transformations face. It's the comparison between simple pendulums and double or compound pendulums. For those of you who don't know what the difference is, the first part of this video shows the simple pendulum, the second part, after the unbolting (turning the single rod into two parts  at 0:10) shows the double pendulum.

YouTube vid stevenbtroy

Now, simple enough, predict the motion of the pendulum in each case.

Dependency

The key point to note here is that when the fixed point is freed to become a compound double pendulum, there is a single variable dependency between the two parts of the compound pendulum. That's where they couple.

Coupling and dependency certainly isn't new. It exists in code, in relationships between component, people, tasks, cars, concepts etc. The term 'conessience' is making the rounds, which for maths folk is simply a dependency.

These dependencies exist all over every type of system you can think of. Indeed, it's impossible for there not to be in a sustainable way (even mythical perpetual motion machines are dependent on something such as gravity). An enterprise and indeed, a project isn't any different.

Consider the following example Gantt chart:

The key thing to note are the arrows between the tasks as these define the dependency. The main variable of note is time (hinted at by the calendar along the top). The on-time deliverability [I don't think that's a word] of tasks on the right is wholly dependent on the delivery of the tasks to the left of it in the dependency chain. Each link is like the pivot on the video at the start of this blog. So in essence, even the red stream alone of this simple Gantt chart has 4 dependencies (tasks 2, 4, 6, 8) and each of these dependencies is a pivot in a pendulum. How successful were you in predicting the motion of the compound pendulum ahead of time? Now quadruple the number of pivots and imagine predicting that. Note, in enterprise projects, 7 tasks across two teams isn't exactly considered a large project. So why do we subject ourselves to such unpredictability and risk?

Fixed Points

In august of last year, I wrote a post about the similarities of enterprise agility and astrophysics. I explain the n-body problem and how it relates to enterprises. I'm only going to expand a little more on this by asking you to imagine that as a CxO, you pay for everything in the path the base of the pendulum takes and get a return when a pendulum hits its maximum position on the other extreme (in the video, it is swung from the right, so you get a return every time it reaches the maximum horizontal distance on the left hand side).

To make this next bit easy, I've run an experiment for 10 for you using the pendulum simulator at http://labs.minutelabs.io/Chaotic-Pendulum/

Simple Pendulum

In 10 seconds, the simple pendulum reaches it's maximum horizontal latitude twice. The amount of 'money' (which is simply the length of the arc) is about the same each time and is represented by the path traced out by the furthest point away from the pivot.

Compound Pendulum
Contrast this with the path covered by the double pendulum in the same 10 second period. Remember, the path covered is the direction all your projects are going in with the money you have invested in their success. I have set the pendulum to be about the same length, which I show you how to do, and I run it for 10 seconds. Whilst watching this, answer the following:

1. How many times did the goal of hitting the maximum left hand side happen? 
2. Where did the money go? 
3. What value or return did the investment yield?

Additionally, remembering that entropy always increases, this sort of disorder just gets worse the more energy that's put into it.

Side-by-Side

Consider how long the arcs are and where they are relative to the direction of travel. Which looks more chaotic and unpredictable? 

Simple versus Compound paths

Conclusion

This analogous post highlights the importance of understanding complexity in enterprises. Whilst a basic understanding of trigonometry will see you through the simple harmonic motion of the simple pendulum, it isn't sufficient to understand the full chaotic motion of the compound system, which is both non-linear and sensitively dependent anyway (so calculus of variations as a base skill comes in handy. Though you're not taught that at A-level or in pretty much any undergraduate or postgraduate computing course - You have to go to subjects like physics or mathematics for that. If you're from business, sales or marketing backgrounds, you'll really struggle). Just like simple IT or business only thinking will see you through localised problems in individual architecture domains, but isn't sufficient for the enterprise as a whole. The trouble comes in the relationship (aka dependency) within the organisational system. The relationship between functions, disciplines, people, teams etc. etc. this is where architecture lives. Between the things.

Enterprise projects are still regarded as somewhat simple ideas when they are anything but. There are so many factors to consider. As I tried to illustrate at the top, the likelihood of delivering bigger projects, with higher numbers of dependencies get slimmer by the link. It's the whole chain that matters. This equally well applies to activities. Enterprise functions are also always changing. Whether they should be if it's not aligned to the overall enterprise goal is another matter. So why do we still persist in managing projects like this? As you can see, you're just wasting money!


E

P.S. The other edge to this double edged sword is how complexity can arise out of simple behaviour. Shhh... I'll tell you a secret. I will come to this another day ;)

Evolving & Emerging Architecture: Agile-EA

Going into companies is always an interesting experience. You get to view the way they work and in agile organisations using physical boards, you can walk the floor and view how the work is progressing. That is well known in the agile project management arena, product owner roles etc. but it's also an extremely useful technique in the Agile Enterprise Architecture world (Agile-EA). Before going into why, we need to recap a couple of definitions.

1. Conway's Law

In 1968, Melvin Conway provided this now tried and tested gem.

"organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations"

This is a huge revelation for some companies, but it got hidden by process oriented techniques, which masked the problem for decades.  

In the agile space, it's a very different story. The reliance on teams to develop their own processes, and having the greatest expressive power within them as opposed to across them has resulted in this being more applicable than ever. You can turn this to your advantage and having elements of a workflow which make up the greatest proportion of it, contained within teams really improves the rate of delivery of work, as I have covered in previous posts and is a much better facilitator of outcome than having a distributed development team. This also means that cross-team communication encourages the development of a system through that communication channel.

2. Consumer Driven Contracts

A consumer driven contract can be considered a Kanban pull signal. This pulls a feature from another team and in software development spaces, this also includes a sub-suite of tests from the consuming team. When a feature provided by another team is needed, a cross-team card, often of a different colour is placed on the team board who are to provide that functionality (at Laterooms,com for example, they used rainbow coloured cards at one point). That team then pick the card up and code against the tests to make them pass. 

Consumer driven contracts are pretty much the most central concept in true just in time, cross team software delivery. They often have a subtask link to the source board and effectively provide an architectural link between application components or business features.

This provides opportunities for architects in both the enterprise and/or solution sphere to see how their estate is evolving by simply going to all stand-up meetings, and building a picture of the emerging architecture or estate by following the chain of tickets. Once such a ticket appears, this introduces an architectural dependency and over time, more and more tickets appears which fill the space between things that architecture lives in.

Walking the Floor

Architects live between things. As architects we should aim to get to every stand-up within our sphere. When we do, we should look out for these cross team tickets, or features being played which we are aware touch something else or perhaps could leverage a feature elsewhere in our estate. 

Imagine we go to the stand-ups and see the following boards after, iteration/sprint 10. 

Sprint 10 board states

Sprint 11 sees a purpose ticket appear on team board 1, required by team board 2 (who are pulling it). The purple ticket represents a cross-team 'pull' (indicated in team 2's 'X' ticket for convenience) since it requires a feature from team 2. The dotted line now represents the relationship between the two teams, which remember by Conway's law represents a link between the system or business features:

Sprint 11 emerges an architectural link

This continues to be played and supposing by the end of Sprint 11 it and its compatriot pull are both done. This ticket is now complete and the architectural link has been delivered. So the architecture of the estate now looks like the following (Arrowed lines represent a dependency):

Sprint 11 Architecture - Note, no Team 4 features live, so architect doesn't care

Sprint 12 and two purple tickets appear. One from team 3 to 1 and one 4 to 2. The solid line represents the delivered link from Sprint 11:

Sprint 12 Architecture

When this is delivered, the architecture then looks like:

Architecture after Sprint 12

And so the cycle continues. 

Conclusion

Each one of the dotted lines represents a communication flow between two teams and brings them closer for that phase of the development. When it's done, the link still exists in the system (solid line), but they can separate and form relationships with other teams, as team 1 did with team 3, after team 1 finished with team 2. Of course, there are times when the platform will need to delete those links, removing them from the estate, but the communication still happens regardless. 

As an architect, attending the stand-ups is a great way to see how the system evolves. You can walk the boards looking for these special cards and draw out the architecture as you proceed. Indeed, if you're an active contributor (and you should be) you can and should facilitate the teams to make decisions which are systemically optimal and forming links when the team could otherwise expend a lot of effort would save them and the company time and money. As the organisation matures and the role of software architect embeds in the teams (as roles over job descriptions), it is these individuals who will go to other team stand-ups and hence, facilitate the creation, updating or destruction of these architectural links.

Q: What Do Agility & Astrophysics Have In Common?

There is an agile coaching game called the static points game. It's an extremely useful illustration of how complexity evolves from really simple rules which in the enterprise world, shows how businesses always change under multiple forces, which should be familiar to those in the change management space. I've played it twice, the first was an introduction by Ash Moran whilst I was at Laterooms.com and more recently with Ian Carroll. It's pretty simply, to play:

1. Get everyone to stand up and move to the edge of the room (or form a circle if the room is too big) 
2. Tell each person to pick two other folk from the group 
3. The simple rule is to stay equidistant (the same distance away) from both of them. 
4. Then let them go.

What you'll see is the group organise and shift about, jostling as the distances come to an equilibrium, then eventually settle. You can play it again, telling people to keep the original two people,  and see where they settle this time. Chances are they settle differently from where they did previously (a digital camera might and high angle come in handy for this variation of the game :).

Reset the game, and with everyone keeping their two folk, fix any one person from the group where they are, perhaps using a chair and send everyone else back to the edge of the room and play it again. They settle quicker. Do it again with that one person fixed, and photograph. You can keep fixing more and more folk and the organisation comes to settle much quicker, with much less movement.

What does this Illustrate?

As an abstract systems game, it naturally covers a multitude of arena!

• How departmental level business changes relative to other departments as politics plays a part in how departmental heads compete for work or pass blame. Imagine the people in that game are working in an organisation and trying to balance the needs of two sets of stakeholders.
• How uncoordinated systems work with one another as they evolve (which I believe is what you think this refers to here). Imagine the people in that game are subsystems taking with interfaces to two other systems.
•  How uncoordinated work-streams work with one another as they evolve (in agile environments, this is what I think happens with shared systems. They pull against the shared systems). Imagine the people in that game are subsystems communicating with interfaces to two other systems.
• It shows how complexity can manifest from a really simple rule-set. This one is self-explanatory. Intelligent agents (i.e. people, bees etc.) using a really simple rule can still produce a significant amount of very complex behaviour. This, like all the rest, is called [mathematical] chaos.
• It shows that relationships are easily equally as important as the entities themselves
• How organisms relate to one another
• It is a manifestation of planets being influenced by each other’s gravity
• …

Mathematically speaking, they are ALL a manifestation of what is called the n-body problem. The planetary example that ends that list above is where this originated.

A Lesson in Planetary motion

We orbit the sun because our mass is significantly less than it. The effect we have on the sun is near negligible. This is like a big CEO of a company, keeping things in order by imposing forces upon the lower weighted levels, who can’t respond in any meaningful or significant way. However, the behaviour of the system is predictable and has been like that for billions of years. With the big, overarching, autocratic CEO (or C-Suite) in it, and with the absence of any other influential factors, the environment rarely changes, so there is no need for it to change. That is one sun and several planets and moons and their orbits (statics and dynamics) that systemically stay the same for millions if not billions of years, even if what goes on on the surface changes. As far as systems are concerned, architecturally, these are all static points. That’s a high level block diagram!

However, in agile environments, you are empowering folk, quite rightly. Hence, the gravity they have and are allowed to have, relative to the system, is much higher. However, returning to the n-body system, If you have two equally weighted planets orbiting around each other, they will pull each other’s orbits. If you have three orbiting each other, it has been proven that the behaviour of an unconstrained system, is near unpredictable aside from very restricted contexts (i.e. akin to how often waterfall delivered on time and on budget, which some would argue has the same probability that our solar system came into existence the way it has :) Plus, because the class of problem is the same for all of the above, this chaos or ‘randomness’ applies to all of the above examples too.

OK, how can you test if the result is random?

Firstly, this is a complexity problem. You can't necessarily test the system internals [aka business] as a whole if the result isn't predictable or consistent. However, what you can test, is that the unit which is the individual person, does manifest the rules correctly! i.e. they keep the same distance from their two folk at all points of change, including all points of jostling! After all, most software systems test that rules manifest correctly. For example, you can't open a bank account if you don't have ID or you can't board a plane for an international flight without a passport. Each and every one of the individual entities, as well as the whole in deterministic systems, is defined by: 

• A pre-condition, which includes the initial state of the system - The position they are in in the room 
• An action - Someone moves
• A post-condition - They have to remain equidistant

With an invariant that they have picked two constant people to apply the rule with.

Which in BDD/Gherkin syntax is akin to:

Background each person has two different folk to focus on
##...

Given the person is in the room
When someone moves
Then the person has to be the same distance from their left-person and right-person

Remember, a static point is not just the structure or entity, it is also the behaviour it exhibits. Hence, the best way to make a change and make it testable, with the minimum of risk. is to nail all but one of the folk (read systems, which include the entities and how they behave – that is the aggregate of business, application, data and technology for each feature), make that small change, test they manifest the rules, then release everything, make sure the system didn't disintegrate (i.e. all the other elements correctly adhere to their own rules, which may be the old ones), then for the next change, nail all but a different one, make a change, release etc. The key is to test that the one body itself adheres to the simple rule you want of it! This provides parallels in systems?

For example, if in the static points game above, if we took just one person (which is a business unit or capability), let's call them 'delta' and made the rule that they stay an arms length away from everyone else and can pull the folk together if they are not, then played the game again. They may get jostled about a bit by the rest of the business going about its [old] business, which is still testable, whilst simultaneously pulling the two folk to their arms length distance. Note, through pulling folk, the rest of the system, that means folk who have picked one or other or even both of the people pulled by delta, and their 2nd, 3rd and nth degree of separation, will also change. So this new rule influences the system as a whole, but crucially, the rest of the system adhered to the old 'equidistance' rule, which like the new one, you can still measure individually (as mentioned at the top of this section). You measure that Delta is doing their job by ensuring they keep their two within 'punch distance' at all points of change, i.e. jostling.

Conclusion

Trust me, this is a very difficult concept for some folk to get and it requires some 'micro-thinking'. Indeed, it always provides a learning point in communication to me. Whilst there is almost nothing anybody in the IT world can tell me about non-linear system dynamics, there is a lot I have to learn about communicating the concept across in language people understand. That is why I attend community events, such as TechNights, Lean Agile Manchester etc. it’s to learn to effectively communicate the ideas to people who do not have that bridge or background. I've been through this sort of discussion a couple of dozen (or more) times and I still communicate this wrong now, because there may be levels of knowledge or skill between the person I am trying to communicate this to and the understanding of non-linear dynamics that would help illustrate the benefits of the knowledge. I'd be interested to see how others communicate this to analytical and non-analytical audiences alike.

So the best I can do for now is probably illustrate it with video. In the external links below, take a look and see if the systems look the same as each other after they've been running for a while. Indeed, you can run the YouTube vids side by side if your broadband is up to it.

External Links
n-Body Gravity Simulation like folk at the edge of a room - https://www.youtube.com/watch?v=XAlzniN6L94

n-Body simulation with 50 million entities - https://www.youtube.com/watch?v=OJaE9J39A8s

VIew these two 3 body problems side by side:
https://www.youtube.com/watch?v=VX9IdCnNWJI
http://vimeo.com/11993047 (from 24 seconds in - This also has multiple runs with different starting points)

Agile software development under TOGAF

Phases G & H: the Architecture Governance Iteration

In a blog post last year, I covered how Agile and TOGAF are perfectly compatible and outlined the similarities that they have.

The title of the post won't resonate much with agilists out there, as the term 'governance' is something that most think of as 'management bureaucracy'. However, the truth of the matter is that if you look at Kent Beck's seminal work on XP and consider the principles and practises section, you'll see that these are in effect, principles and governance guidelines.What I mean by this is probably best illustrated via an example.

Imagine you are an agile QA. In order to adjudge whether or not a piece of software has been developed solidly, agile QA's often carry out the role of checking code coverage and complexity measures and monitor that coverage against metrics defined in their testing strategy. This can be automated and they instantly see a governance measure that they can check against what are effectively their organisational unit's (software development department, say) principles of TDD/BDD etc.

For software developers and architects out there, remember, TOGAF is a much broader and more abstract framework to develop enterprises under (I steer clear of just applying it to software, since it isn't technically a software development methodology. It is an EA framework, or systems development methodology if you look at it through those particular eyes). It covers business architecture, data architecture and technical architecture as well as application architecture and at the highest level, it does this without recourse to any technological concepts at all in those stages (so there are no explicit, detailed references to Visual Studio 2012, SQL Server 2012, MySql, PHP etc.). The aim is to consider the organisation as a whole as a system.

Statics and Dynamics

Any system, in whatever form, only require two high-level concepts to fully define it's operation. These are:

• Statics - which in the case of business are things like the structures, entities, artefacts, roles, people, systems etc. etc. 
• Dynamics - How these static elements interact and indeed how the organisation as a whole behaves.

In the world of software, we have seem this many times. For example, the GoF books were split into patterns of structure and those of behaviour and we also have languages to express those interactions, such as UML class and collaboration diagrams. These days, systems thinking is finally making inroads into the software world, despite those of us already familiar with the concepts having used it for nearly 15 years (some of you out there will have used it longer than I have).

Just like businesses as a whole, you have different levels of abstraction which define the organisation. You can define the system dynamics at varying levels of details and just like you have, say, different levels of UML diagram (use-cases to compartmentalise the operation of a software unit, with activity diagrams which define the functions within it. In themselves, these functions can be another use case diagram's use case, in the case of say, a component or package), the same is true of business. BPMN's ability to run sub tasks is another example of this. However, I don't want to get hung up on the language of expressing a business, as this is different from the business itself.

fig 1 - TOGAF ADM with different sub iterations (from Mike Walker's blog on MSDN, copyright The Open Group)

And just like a business, TOGAF allows different levels of EA for each level of the architecture partition or indeed organisational unit.

fig 2 - TOGAF ADM applied at different levels of partitioned architecture (copyright The Open Group)

How does this fit?

Well, a business is a system. Software is a system. Software development is a system. Everything in this context is just a system. Apologies to people who really hold on to the psychological element of agile methods, but you are being kept a happy cog in a big wheel :-) This is not to say it shouldn't happen, but you are there to develop software at the end of the day. Indeed, remember, you and your team use elements like Kanban to 'manufacture' software just like a they do in a factory.

AGILIST: "OK, stop flaming us! Really, how does it fit?"

Joking aside, during a software development project, you will often use BDD/TDD to develop acceptance criteria and the amount of coverage in these scenarios is your governance metric. Additionally, some agile teams are aware of risks and issues that they have to monitor. They also use metrics to define 'productivity' such as throughput and cycle time They also run retrospectives to continuously improve. Software dev peeps should pay particular attention to phase G in TOGAF gives you all that when defining a system representing your organisation. Software, Business, Data and Tech Architects may be involved in earlier phases, maybe even A to E, depending on the model of TOGAF used at each level of architecture partition. But for devs, phase G of  the ADM gives you:

• Project name, description and objectives [Project name/Description/Epics]
• Scope, deliverables and constraints
• Efficacy metrics [Throughput/Cycle time/Business Value]
• Acceptance criteria [BDD]
• Risks and issues [Risk Log]

I have linked the Agile concepts in square brackets. As you can see, we have everything required for the initiation of an agile development project right there. 

Remember, when going through an iterative enterprise architecture process such as TOGAF, there is no BDUF exercise. Each transition goes through an iteration of the ADM, which creates an incremental change to the organisational system as a whole. Within each iteration, at phase B, businesses are architected or rearchitected and IT systems are developed to facilitate that business operation. Same as always. 

Some developers see themselves as coming into play quite late in the process (Phases G and H - last n the cycle). It seems to them that there is a BDUF exercise going on, but really the business is defining itself at the beginning of the cycle. After all, you can't have acceptance criteria defined by the business if there is no business :-)

By contrast, when you get a BDD spec for an existing, well understood business, the business system (with it's statics and dynamics) is already defined for the business owners. They have lived it and worked it for years. The business owner knows what they want and work with the 'three headed monster' (BA, QA, dev) to spec that out in Gherkin for example. In this case, there is no need to reengineer the business to then submit acceptance criteria to the devs. In TOGAF, it is exactly like going through the Architecture Definition Iteration (cycles of ADM phases B to F) with the existing formal baselines already in place, but having no EA change requirement. So you can get through those in very little time and go straight to phase G, which is where these BDD criteria form the acceptance criteria for the IT systems (I appreciate using TOGAF here is overkill, but the point is about illustrating the pattern).

Remember this...

Whilst TOGAF is an EA framework and, say, SCRUM is an Agile software development practise, they both work to define systems. Both are iterative and incremental in nature. They just work at different levels and to some degree, can have very different cycle times. After all, TOGAF would also concern itself with the definition of an organisation's services, people required to carry out enterprise functions within that service, the information (not 'data') they require to do their job, and provide services which may or may not use IT at all (think shop assistants, call centre operatives, cleaners, warehouse staff etc. etc.) and their roles have to be at least understood if not engineered into the business system as a whole. This is something IT personnel take for granted when defining software. Even in RUP they assume actors already exist, but I would say how do they think those actors came to exist? TOGAf is a way of making that happen.