Tackling Complexity in Projects & Enterprises

There is an analogy doing the rounds at the moment which I think is pretty good as a necessary intro, but not at all sufficient to understand some of the complexity enterprise programmes and transformations face. It's the comparison between simple pendulums and double or compound pendulums. For those of you who don't know what the difference is, the first part of this video shows the simple pendulum, the second part, after the unbolting (turning the single rod into two parts  at 0:10) shows the double pendulum.

YouTube vid stevenbtroy

Now, simple enough, predict the motion of the pendulum in each case.

Dependency

The key point to note here is that when the fixed point is freed to become a compound double pendulum, there is a single variable dependency between the two parts of the compound pendulum. That's where they couple.

Coupling and dependency certainly isn't new. It exists in code, in relationships between component, people, tasks, cars, concepts etc. The term 'conessience' is making the rounds, which for maths folk is simply a dependency.

These dependencies exist all over every type of system you can think of. Indeed, it's impossible for there not to be in a sustainable way (even mythical perpetual motion machines are dependent on something such as gravity). An enterprise and indeed, a project isn't any different.

Consider the following example Gantt chart:

The key thing to note are the arrows between the tasks as these define the dependency. The main variable of note is time (hinted at by the calendar along the top). The on-time deliverability [I don't think that's a word] of tasks on the right is wholly dependent on the delivery of the tasks to the left of it in the dependency chain. Each link is like the pivot on the video at the start of this blog. So in essence, even the red stream alone of this simple Gantt chart has 4 dependencies (tasks 2, 4, 6, 8) and each of these dependencies is a pivot in a pendulum. How successful were you in predicting the motion of the compound pendulum ahead of time? Now quadruple the number of pivots and imagine predicting that. Note, in enterprise projects, 7 tasks across two teams isn't exactly considered a large project. So why do we subject ourselves to such unpredictability and risk?

Fixed Points

In august of last year, I wrote a post about the similarities of enterprise agility and astrophysics. I explain the n-body problem and how it relates to enterprises. I'm only going to expand a little more on this by asking you to imagine that as a CxO, you pay for everything in the path the base of the pendulum takes and get a return when a pendulum hits its maximum position on the other extreme (in the video, it is swung from the right, so you get a return every time it reaches the maximum horizontal distance on the left hand side).

To make this next bit easy, I've run an experiment for 10 for you using the pendulum simulator at http://labs.minutelabs.io/Chaotic-Pendulum/

Simple Pendulum

In 10 seconds, the simple pendulum reaches it's maximum horizontal latitude twice. The amount of 'money' (which is simply the length of the arc) is about the same each time and is represented by the path traced out by the furthest point away from the pivot.

Compound Pendulum
Contrast this with the path covered by the double pendulum in the same 10 second period. Remember, the path covered is the direction all your projects are going in with the money you have invested in their success. I have set the pendulum to be about the same length, which I show you how to do, and I run it for 10 seconds. Whilst watching this, answer the following:

1. How many times did the goal of hitting the maximum left hand side happen? 
2. Where did the money go? 
3. What value or return did the investment yield?

Additionally, remembering that entropy always increases, this sort of disorder just gets worse the more energy that's put into it.

Side-by-Side

Consider how long the arcs are and where they are relative to the direction of travel. Which looks more chaotic and unpredictable? 

Simple versus Compound paths

Conclusion

This analogous post highlights the importance of understanding complexity in enterprises. Whilst a basic understanding of trigonometry will see you through the simple harmonic motion of the simple pendulum, it isn't sufficient to understand the full chaotic motion of the compound system, which is both non-linear and sensitively dependent anyway (so calculus of variations as a base skill comes in handy. Though you're not taught that at A-level or in pretty much any undergraduate or postgraduate computing course - You have to go to subjects like physics or mathematics for that. If you're from business, sales or marketing backgrounds, you'll really struggle). Just like simple IT or business only thinking will see you through localised problems in individual architecture domains, but isn't sufficient for the enterprise as a whole. The trouble comes in the relationship (aka dependency) within the organisational system. The relationship between functions, disciplines, people, teams etc. etc. this is where architecture lives. Between the things.

Enterprise projects are still regarded as somewhat simple ideas when they are anything but. There are so many factors to consider. As I tried to illustrate at the top, the likelihood of delivering bigger projects, with higher numbers of dependencies get slimmer by the link. It's the whole chain that matters. This equally well applies to activities. Enterprise functions are also always changing. Whether they should be if it's not aligned to the overall enterprise goal is another matter. So why do we still persist in managing projects like this? As you can see, you're just wasting money!


E

P.S. The other edge to this double edged sword is how complexity can arise out of simple behaviour. Shhh... I'll tell you a secret. I will come to this another day ;)

Q: What Do Agility & Astrophysics Have In Common?

There is an agile coaching game called the static points game. It's an extremely useful illustration of how complexity evolves from really simple rules which in the enterprise world, shows how businesses always change under multiple forces, which should be familiar to those in the change management space. I've played it twice, the first was an introduction by Ash Moran whilst I was at Laterooms.com and more recently with Ian Carroll. It's pretty simply, to play:

1. Get everyone to stand up and move to the edge of the room (or form a circle if the room is too big) 
2. Tell each person to pick two other folk from the group 
3. The simple rule is to stay equidistant (the same distance away) from both of them. 
4. Then let them go.

What you'll see is the group organise and shift about, jostling as the distances come to an equilibrium, then eventually settle. You can play it again, telling people to keep the original two people,  and see where they settle this time. Chances are they settle differently from where they did previously (a digital camera might and high angle come in handy for this variation of the game :).

Reset the game, and with everyone keeping their two folk, fix any one person from the group where they are, perhaps using a chair and send everyone else back to the edge of the room and play it again. They settle quicker. Do it again with that one person fixed, and photograph. You can keep fixing more and more folk and the organisation comes to settle much quicker, with much less movement.

What does this Illustrate?

As an abstract systems game, it naturally covers a multitude of arena!

• How departmental level business changes relative to other departments as politics plays a part in how departmental heads compete for work or pass blame. Imagine the people in that game are working in an organisation and trying to balance the needs of two sets of stakeholders.
• How uncoordinated systems work with one another as they evolve (which I believe is what you think this refers to here). Imagine the people in that game are subsystems taking with interfaces to two other systems.
•  How uncoordinated work-streams work with one another as they evolve (in agile environments, this is what I think happens with shared systems. They pull against the shared systems). Imagine the people in that game are subsystems communicating with interfaces to two other systems.
• It shows how complexity can manifest from a really simple rule-set. This one is self-explanatory. Intelligent agents (i.e. people, bees etc.) using a really simple rule can still produce a significant amount of very complex behaviour. This, like all the rest, is called [mathematical] chaos.
• It shows that relationships are easily equally as important as the entities themselves
• How organisms relate to one another
• It is a manifestation of planets being influenced by each other’s gravity
• …

Mathematically speaking, they are ALL a manifestation of what is called the n-body problem. The planetary example that ends that list above is where this originated.

A Lesson in Planetary motion

We orbit the sun because our mass is significantly less than it. The effect we have on the sun is near negligible. This is like a big CEO of a company, keeping things in order by imposing forces upon the lower weighted levels, who can’t respond in any meaningful or significant way. However, the behaviour of the system is predictable and has been like that for billions of years. With the big, overarching, autocratic CEO (or C-Suite) in it, and with the absence of any other influential factors, the environment rarely changes, so there is no need for it to change. That is one sun and several planets and moons and their orbits (statics and dynamics) that systemically stay the same for millions if not billions of years, even if what goes on on the surface changes. As far as systems are concerned, architecturally, these are all static points. That’s a high level block diagram!

However, in agile environments, you are empowering folk, quite rightly. Hence, the gravity they have and are allowed to have, relative to the system, is much higher. However, returning to the n-body system, If you have two equally weighted planets orbiting around each other, they will pull each other’s orbits. If you have three orbiting each other, it has been proven that the behaviour of an unconstrained system, is near unpredictable aside from very restricted contexts (i.e. akin to how often waterfall delivered on time and on budget, which some would argue has the same probability that our solar system came into existence the way it has :) Plus, because the class of problem is the same for all of the above, this chaos or ‘randomness’ applies to all of the above examples too.

OK, how can you test if the result is random?

Firstly, this is a complexity problem. You can't necessarily test the system internals [aka business] as a whole if the result isn't predictable or consistent. However, what you can test, is that the unit which is the individual person, does manifest the rules correctly! i.e. they keep the same distance from their two folk at all points of change, including all points of jostling! After all, most software systems test that rules manifest correctly. For example, you can't open a bank account if you don't have ID or you can't board a plane for an international flight without a passport. Each and every one of the individual entities, as well as the whole in deterministic systems, is defined by: 

• A pre-condition, which includes the initial state of the system - The position they are in in the room 
• An action - Someone moves
• A post-condition - They have to remain equidistant

With an invariant that they have picked two constant people to apply the rule with.

Which in BDD/Gherkin syntax is akin to:

Background each person has two different folk to focus on
##...

Given the person is in the room
When someone moves
Then the person has to be the same distance from their left-person and right-person

Remember, a static point is not just the structure or entity, it is also the behaviour it exhibits. Hence, the best way to make a change and make it testable, with the minimum of risk. is to nail all but one of the folk (read systems, which include the entities and how they behave – that is the aggregate of business, application, data and technology for each feature), make that small change, test they manifest the rules, then release everything, make sure the system didn't disintegrate (i.e. all the other elements correctly adhere to their own rules, which may be the old ones), then for the next change, nail all but a different one, make a change, release etc. The key is to test that the one body itself adheres to the simple rule you want of it! This provides parallels in systems?

For example, if in the static points game above, if we took just one person (which is a business unit or capability), let's call them 'delta' and made the rule that they stay an arms length away from everyone else and can pull the folk together if they are not, then played the game again. They may get jostled about a bit by the rest of the business going about its [old] business, which is still testable, whilst simultaneously pulling the two folk to their arms length distance. Note, through pulling folk, the rest of the system, that means folk who have picked one or other or even both of the people pulled by delta, and their 2nd, 3rd and nth degree of separation, will also change. So this new rule influences the system as a whole, but crucially, the rest of the system adhered to the old 'equidistance' rule, which like the new one, you can still measure individually (as mentioned at the top of this section). You measure that Delta is doing their job by ensuring they keep their two within 'punch distance' at all points of change, i.e. jostling.

Conclusion

Trust me, this is a very difficult concept for some folk to get and it requires some 'micro-thinking'. Indeed, it always provides a learning point in communication to me. Whilst there is almost nothing anybody in the IT world can tell me about non-linear system dynamics, there is a lot I have to learn about communicating the concept across in language people understand. That is why I attend community events, such as TechNights, Lean Agile Manchester etc. it’s to learn to effectively communicate the ideas to people who do not have that bridge or background. I've been through this sort of discussion a couple of dozen (or more) times and I still communicate this wrong now, because there may be levels of knowledge or skill between the person I am trying to communicate this to and the understanding of non-linear dynamics that would help illustrate the benefits of the knowledge. I'd be interested to see how others communicate this to analytical and non-analytical audiences alike.

So the best I can do for now is probably illustrate it with video. In the external links below, take a look and see if the systems look the same as each other after they've been running for a while. Indeed, you can run the YouTube vids side by side if your broadband is up to it.

External Links
n-Body Gravity Simulation like folk at the edge of a room - https://www.youtube.com/watch?v=XAlzniN6L94

n-Body simulation with 50 million entities - https://www.youtube.com/watch?v=OJaE9J39A8s

VIew these two 3 body problems side by side:
https://www.youtube.com/watch?v=VX9IdCnNWJI
http://vimeo.com/11993047 (from 24 seconds in - This also has multiple runs with different starting points)