TOGAF luuuurvs Agile!

There is a common misconception that I come across day-to-day in some environments which seems to indicate a disconnect between the opinion of TOGAF in the development community and what it truly entails.

This is particularly evident in the world of agile methods, where such things as ‘up front design’, ‘documentation’ and ‘long term strategy’ are often cited as reasons to compete in the ‘urinational’ championships.

The truth of the matter is Agile and Enterprise architecture frameworks cover different domains, though agile Methodists can’t seem to generalise their thinking into the enterprise architecture world whilst TOGAF practitioners definitely can scale down (even if they can’t write code). This post will attempt to address the 12 principles of the agile manifesto and relate them to TOGAF 9. However, in order to do that, an incredibly quick, summary crash course in TOGAF will be ‘just enough’ information to get the job done :-)

TOGAF 9 Overview

The Open Group released the version 9 of the TOGAF standard in 2009. It is a set of structured tools, including an iterative Architecture Development Method (ADM for short), reference models, content framework,
enterprise continuum and repositories for the production of a set of interconnected functional and reusable enterprise capabilities for the satisfaction of varying levels of strategy, when constraining a business (which is just a system like any other) by factors such as legislation, finance, time, organisational capabilities, business goals and other constraints.

What this allows is the ability to bridge the gap between the CEO/CIO vision and strategy and the operation of a company in terms of its capabilities to deliver that vision, including the IT functions.

TOGAF concentrates its architectural effort on four domains of architecture. Namely:

• Business Architecture
• Information Systems Architecture, which is subdivided into:
• Data Architecture
• Application Architecture
• Technology Architecture

...and it can be applied at many different levels by going through an ‘architecture partitioning’ exercise which create strategic architecture (which may be a 5 year plan for the whole enterprise), segmented architectures and capability architectures, in order of increasing levels of detail and decreasing levels of scope.

An IT department is but one capability in a myriad of others, such as HR, Customer Services and Finance. It just so happens that it is significant enough a capability to have dedicated responsibilities in phases of the ADM at the capability architecture level.

Additionally, TOGAF specifies the use of Architectural Principles, which are reasoned, qualitative, yet normative statements which constrain the architectural solution space in some way. They can be based on regulatory, best-practice, performance, security, auditing dimensions, amongst a host of others, but they generally pertain to one of the four groupings of:

• Assurance
• Adaptability
• Availability
• Usability

However, in all cases, they must be SMART goals.

• Specific
• Measurable
• Achievable
• Realistic
• Time-boxed

When the ADM is started, there can be many cycles at many levels and they may all use the ADM to deliver the capability. So a strategic enterprise architecture exercise may initiate a 5 year plan together with segmented architectures to look at which are segmented deliverables, which will in turn be broken down through their own ADM instance to capabilities for each segment, which can be delivered using the ADM. Alternatively, an iteration of the ADM may deliver a higher level architecture, which initiates the development of lower level capabilities after it has been completed.

An IT capability can easily be delivered using agile principles, but a few things have to come to the fore and the TOGAF terminology, which is foundational in nature, needs to be augmented with and a mapped to common systems terminology (such as security, auditing, monitoring functions), industry architecture concepts (such as those general to all travel agents, or HR systems) and specific organisational terms.

How does Agile fit within this?

Enterprise architecture, whether agile or not, is the bridge between the business strategy delivered by a board and all the different capabilities that exits in an organisation. The corporation’s vision, portfolio, strategy and governance are input to it and the capabilities (of which IT is but one part) are output after each iteration of the ADM, either in the form of transition architectures or target architecture.

Agile methods deliver the functional technology that makes the application and data architectures happen, which are in turn delivered by the technology architecture phase. However, half the problem is that an exercise connecting the terminology used by enterprise architects and solution level developers is normally never carried out.

TOGAF mandates this exercise as the framework is very abstract and The Open Group were so aware of the effects of this abstraction on the use and uptake of TOGAF, that they strongly encourage the use of a reference library which maps TOGAF terminology to the terms used in specific capabilities in the enterprise and/or to the enterprise as a whole. Without this, a large amount of confusion can occur, for example what does a ‘service’ mean to different people?

The Agile Manifesto’s 12 principles can be mapped something like the following (amongst a myriad of other ways):

1. Customer satisfaction by rapid delivery of useful software

The difference with TOGAF only comes to the fore when the definition of ‘customer’ is looked at.

In an enterprise architecture exercise, the customer is internal. Indeed, there may be solution or enterprise architects in the interested stakeholder groups. The value realisation may take months to come to fruition, but the agile development of a piece of functionality by the IT capability may be a small section which only delivers its value when the whole jigsaw is in place. This is in itself, valuable! As the business value has already been assessed by the business participants and this is the reason for the development exercise and it may indeed contribute to the exercise as a whole with a monetary value.

To link TOGAF to Agile, we have to be aware that the product owner is usually the solution architect responsible for the delivery of the solution building block in the solution continuum. It is categorically NOT the development manager and may not even be the business owner. The business owner will have delivered their capability requirements to the enterprise architects who will have delivered their prioritised strategy to the solution architects who in turn will have segmented the solution architecture into software components (which deliver capabilities) which they will pull from each team.

2. Welcome changing requirements, even late in development

This is good from TOGAF’s perspective. The reason is because TOGAF encourages ’Just Enough’ design, as it is under no illusions that the requirements (fed by the organisation’s environment) will change during the lifetime of the larger partitions’ realisation. This ties in very well with the agile proponents’ viewpoints on the matter, though the ‘no turning back’ point in TOGAF is obviously of a different scale to the Agile processes.

In order to facilitate ‘welcoming changing operations,even late in the strategy’ certain infrastructural matters need to be carried out and they may include using an emergent/middle-out architecture. However, this is just the same as a Kanban board, metrics, elicitation meetings, spring planning or anything else. It certainly is not an alien
concept to agile developers.

Enterprise architects, together with the Solution architects may use Service Oriented Architecture to deliver the building block service capabilities, including, say the use of an enterprise service bus (ESB) to facilitate the communication. Once done, orchestration processes, which are basically workflows, can be designed by relatively small teams to change the operation of the system within very short timescales if new capabilities are not required. Indeed, it is possible to re-orchestrate well-designed, existing services in an operation significantly faster than an agile development team can create a new software component.

3. Working software is delivered frequently (weeks rather than months)

This is the same as stating that an organisation needs to be able to change direction quickly and change its internal operation to match. SOA platforms deliver this using orchestration processes where flow of process or information is sent around the company to perform a particular task. Consider it like be able to restructure [aka refactor to agile dev readers] the whole organisation in a very short space of time, to take the company where it needs to go next.

In TOGAF, each iteration of the ADM generates a new transition architecture, which becomes the baseline architecture for the next iteration starting at the architectural vision phase. A GAP analysis is carried to identify the difference in capability between the baseline and next transition/target and those that are selected for the ADM are realised through its phases.

Agile methods work with smaller ‘granules’ of functions, until the whole functioning system is delivered. This is the system capability of the whole enterprise capability and is the responsibility of solution architects to deliver. Again, if this is delivered through incremental delivery, TOGAF doesn’t really care. At the solution architecture level, as long as ‘symmetry of conformance’ principle are adhered to, the development teams can work in tandem. For software, this means systems on either side of an interface boundary must conform to the same interface specification at the same time and may necessitate a contract definition, which the QA roles on the team would ensure conformance to.

Each iteration of the ADM will deliver business value in a much shorter period than the whole business strategy. Additionally, the same benefits of pipelining tasks (like an assembly line) that agile developers see, can also be realised at the enterprise level by performing the ADM’s architecture phases in the same ‘assembly line’ way (Business, Data, Application and Technical Architectures – Phases B - D). Each finished iteration will deliver business value and do so in a much shorter time than trying to carry them out in sequence. The key is to partition the individual architectures to be independent of one another.

However, without taking the time to identify the segments, this won’t be possible. This is the same as discussions surrounding success criteria and requirements elicitation in agile processes. So again, it is not new.

4. Working software is the principal measure of progress

I personally have a problem with this definition anyway. ‘Working software’ as the agile developers of today know it, is necessary but not sufficient. If the definition of working software is to include ‘fit for purpose’ criteria, which TOGAF encourages, then this can encompass qualitative non-functional elements in the final delivered system. Additionally, a software component is not ‘Done’ until it has been fully tested, documented and delivered together with any necessary supporting artefacts and processes. All these together with similar criteria for the other architecture domains in the enterprise, deliver the whole capability. After all, what use is a piece of software that it is claimed ‘delivers value’ when nobody can use it because documents and training don’t exist?

Checking ‘fit for purpose’ criteria, together with architectural principles make up the bulk of the QA role in enterprise scale systems.

Often an architectural principle can be decomposed into a QA usable element. Consider the system principle:

Name:Logarithmic Performance Degradation

Statement:The system latency increases logarithmically when there is a linear growth of call centre users.

Rationale:The cost to handle linear increases in capacity is £1 million for 100 new call centre CPU licenses purchasable in blocks of 10. Logarithmic reductions in performance will allow a cumulative reduction of £100 per bulk license cost for each 10 new CPU licenses.

Impact: …

Here we can see that the bounds of ‘working software’ that is ‘fit for purpose’ includes a qualitative statement of the costs of performance in the principle. This may further be broken down into success criteria that after involvement from BA, Development and QA personnel, might derive the following non-functional requirement:

“Under heavy load, the system must maintain a throughput proportional to a function of O(n log₁₀ n).”

Now, I appreciate the vast majority of you won’t have come across a requirement worded like this. Indeed, it is entirely possible you consider agile to not worry about the performance of software until it becomes an issue (whether a company may lose millions in the process in either opportunity costs or real world losses seems immaterial to those who think like this, despite it delivering a ‘negative’ business value). But this gives architects the means by which to reduce the solution space by filtering out solutions which are inadequate and unsatisfactory. This in turn can give the technical staff the means to be able to deliver an environment to be able to test and stage pre-production and production deployments for each typical agile iteration that you will do. The idea of ‘fail fast and fail early’ which a lot of agile adopters have taken on-board applies equally well to the environments that the software is hosted in. The last thing you want is to have a piece of software which you have carefully sculpted using agile principles failing to deliver the performance gains in live or worse still, losing the company business value because considerations outside the agile domain have not been accounted for.

TOGAF includes the use of architectural principles in part to address non-functional concerns from the outset. Together with business architecture requirements, the statement of architecture work, the existing baseline architecture, specific industry trends, computing technology trends and the like, TOGAF moulds the enterprise architecture (including its technical capability and resources) to fit in with the vision of the very highest stakeholders on the company. When addressed from this perspective, the solution architects (whether of software or technical systems) deliver specific capability architectures for their architectural domain to the enterprise architects, who are often the principle customers of the solution architects. In turn, the solution architects then become the principle customers of the development teams along application horizontals whilst the business customers are along the verticals.

Unfortunately, the architects are normally vastly outnumbered by the development personnel, so can’t be in several agile teams at the same time. It may be that an organisation works to include junior solution architects in their staff-base, which would allow the segmentation of components of the overall architecture to spread across several development teams or may choose another way of segmenting that work, such as using development contracts with which the QAs would apply best practise principles such as ‘symmetry of conformance’.

5. Sustainable development, able to maintain a constant pace

TOGAF maintains the status quo and doesn’t explicitly address tempo in the standard. Again, it is more interested in the delivery of capabilities to the business and it is the programme management office’s responsibility to schedule the delivery of the enterprise architecture by priority of the requests for architecture work. You will note though, that this is akin to the prioritising of a product backlog by a stakeholder.

6. Close, daily co-operation between business people and developers

Remember, in TOGAF everything and everyone is a capability at a high level. Looking down from a strategic architecture, through segmented architecture at the capabilities, we see that IT is a capability, customer services is a capability, accounting is a capability etc. so everyone is a capability within the business and as a result, can be a business customer to any agile development team. This specifically means that developers and solution architects can be customers of development teams. It also allows the dissemination of the bigger picture as part of facilitating knowledge transfer from team to team via the architectural members participating in it.

An example of enterprise capabilities

The difficulty is that architects are not all available to be involved in direct code writing for the whole team all the time. As a result, a clear statement of intent for the day or indeed sprint backlog review should be used to prioritise the work and should come about, in part by the use of stand-up meetings, which architects should always attend.

The role of governance in the team should be delegated to the QAs who should ensure all architectural principles are adhered to, the interface definitions are conformed to and that the required performance and other related principles are encompassed in the system.

For some organisations, it may be decided that a ‘scrum-of-scrums’ take place to allow the solution architects to ensure that the required criteria are being satisfied at a group level.

7. Face-to-face conversation is the best form of communication (co-location)

In a previous blog, I indicated why this principle is highly deficient when applied to anyone outside the immediate development team. Additionally, there is a strict reluctance from both development and architecture to include enterprise architects in the development process (quite correctly, as it is a level of detail they should be unconcerned with outside the segmentation of the capability and the governance of its production).

Once the interfaces, acceptance criteria, principles and constraints (of which requirements, functional or otherwise, are part) and documented and understood, the team can choose to work whatever way they wish. If there is no clarity in the understanding of what deliverables the team should produce, the exercise will result in wasted development effort or high risk deployments which may fail catastrophically to deliver the value of that capability.

Also, it is the QA and PM’s responsibility to make sure that all communication into and out of the team is strictly controlled and is in accordance with the TOGAF communication strategy documentation. Use of importance-interest stakeholder quadrants (e.g. Eisenhower matrices) and stakeholder analyses/management techniques is the domain of the enterprise and business architects. Once agreed, it is not the responsibility of the development team to deviate from that strategy.

In larger organisations, developers are not the best members of the team to communicate information to non- technical audiences who are unconcerned with the detail of the realisation exercise, especially if they are unaware or unconcerned with the bigger picture (which is an area a lot of practical agile developers normally shun). As a result, they may make development decision on the fly or make the client aware of system level decisions which are incongruent with the enterprise strategy at a higher level, meaning the customer will go away thinking one thing, when the EA has stated something altogether different. These ‘double binds’ lead to confusion, which potentially can feed back into the team when the ultimate stakeholder makes what they think is a ‘corrective’ change to requirements, which may encompass the work of other teams which fall outside the remit of the source developers.

This results in a violation of architectural principles or governance procedures, duplicated and incompatible development deliverables (as the symmetry of conformance principles may be unable to be applied to both sides of interface contracts), overlaps in responsibility which fall outside what is known about in development capability (at the enterprise architecture level) and many other problems.

A slight modification can see RACI matrices help delineate responsibilities for capability deliverable by team. Once delineated, it should be strictly enforced and governance procedures/logs used to determine the decisions that were made and why they were made.

8. Projects are built around motivated individuals, who should be trusted

A project may contain sub-projects. Each sub-project is in itself a project to deliver a capability (aka a piece of functionality) in accordance with success criteria which includes all constraining factors. Once inside the bounds of a project, the motivated individuals are free and trusted to develop whichever way they see fit. TOGAF is unconcerned with how the deliverable is created. All it is concerned with is that the deliverable is ‘fit for purpose’ (Agile’s statement on ‘working software as the final arbiter’) and it does this by using governance processes to test the software deliverable as well as the development capability as a whole.

9. Continuous attention to technical excellence and good design, and… 10. Simplicity

In truth, this is at a level which is way above the understanding and capability of almost all agile developers on the market. If they understood it, they would usually be architects. The concern with very low level detail has blinded the majority of developers into believing that an understanding of a good design pattern is enough, without the need to understand the trade-offs that each has which make it particularly suited to one situation and not another.

Also, what low level development teams don’t have sight of, are points of optimisation in an overall design to allow and facilitate reuse of code or indeed components across teams. This is where solution architects can bridge the gap in understanding, by having a higher level view which allows them to see the deliverables as they are deployed and can govern the reuse of these software assets, which become the solution building blocks that get added to the enterprise repository as part of the solutions continuum. Indeed, it is very very easy for duplication of work across teams to occur, where this step is missed. After all, if one development team is trusting another development team to deliver against a ‘pull’ signal from the first, whilst the first is delivering their own related functionality and is being trusted to deliver this by the client or worse the second team, there is nothing to stop a duplication of work which is one of the cardinal sins of lean practises as it is a direct form of waste.

The enterprise architects don’t have responsibilities under principle 9, aside from producing the architectural capabilities that must be delivered to allow an enterprise engagement. Their view of simplicity (principle 10) is that of combinatorial optimisation, which reduces combinatorial complexity, which in turn reduces the amount of effort it would take for each set of iterations from involved development teams to deliver software, as well as the greater prize of the amount of effort required for a business to change direction.

Consider the two scenarios below.

A development team produces a piece of software (say, a service) which implements a business process which in itself, consumes functionality in 4 other services dotted around the company. The 4 other services are all linked, bi-directionally, to one another, but not the first (that is, for the 4 internal services, 12 communication links, through 6 channels). There are 5 services here and supposing one of the 4 internal services changes its service contract. The development team responsible for this service contract will develop the new interface and publish it, which will break every other connection to a service (which is 3 others). The other 3 teams then have to change their consumption of that service contract and code accordingly. The amount of effort involved is large. This has already broken the symmetry of conformance governance principle.

An enterprise architect will look at the capabilities involved, decide if they are necessary in any transition or target architecture and take steps to change this. This may include using SOA to deliver an ESB architecture pattern, with a data architecture which is independent of vendor, but adequately addresses the specific enterprise requirements (aka the Canonical Data Model or CDM for short) which is the movement from a foundation data architecture, including a common systems architecture, through industry architecture and finally organisation architecture (working from most general to most specific along the enterprise continuum). This can then be gradually rolled out to specific data and/or integration solution architects to realise that vision.

The solution architects may choose an ESB platform, create the contracts for a normaliser for each service to be attached to the ESB, deliver the CDM and also specify the normalising transformation between the specific messaging format of the service to be integrated and the CDM. The development teams all deliver their own normalising transformations to their systems and attach them to the ESB. This means there is only a requirement to support the 4 links on to the ESB not 6, making 8 bidirectional links not 12!

However, the real saving comes in the form of the non-functional requirements associated with adaptability. Should the scenario arise as above, where a contract changes, the development team involved need only change the transformation associated with the normaliser for that one service, massively reducing the workload on every other development team, which will be as good as zero. This is one single team, who have to deliver one single conformant interface (the tests for which will have been created already around the conformance and governance process for the CDM) and this reduces the overall development effort for the scenario above from 3 channels to 1, saving the equivalent to 2 whole teams’ development times over the first method.

Additionally, should the system have 20 internal services on it, each dependent on every other, the number of communication links could reach 380, whilst the number of communication to and from the ESB is only 40, massively reducing the workload.

The trade-off for the developers is the introduction of the normalization process. This is NOT waste! The ‘investment’, which is guaranteed to save money going forward, thereby delivering direct business value, including a monetary increase in profits, needs to be completed for the bigger picture to realise the benefit. To not provide this investment, and govern it, is akin to not writing unit tests up front (as a unit test up-front is an investment that pays dividends later).

So be aware that good design includes architecture! Technical excellence includes architecture! Unfortunately, there are many more programmers in the world than there are architects. So views on agile without architecture (which is in complete contrast to these principles) seem to have taken a hold in the UK simply because of the numbers. Agile methods will not save you alone! Pay attention to the ‘internal stakeholders’ :-)

11. Self-organizing teams

TOGAF, as mentioned already, facilitates the identification of capabilities at different levels. This can include the functions of QA, Business Analysis, Development, Architecture, Test, Technical Infrastructure and the like within the IT capability. As long as the vision of the capability is understood at each stage from baseline to target, then the capability to deliver story level functionality can be made up of self-organising teams within the IT capability architecture. Indeed, capability assessments are often the first elements to be decided during the ADM iterations, initially as part of the preliminary phase and refined in each pass of the architecture vision. It answers questions such as ‘Do we have the skills to deliver this capability and do these necessary skills exist in the required numbers to facilitate each role required by the capability?’

When forming teams to carry out particular pieces of IT development work, managers should be aware of the overall strategy of why they are doing what they are doing. Once they have a full understanding of what is required from them, they can hand it over to the team, who will be aware of how long tasks actually take to perform.

Whichever methodology school you come from, you know you have ask people who will actually carry out the work how much effort is required. Whether that translates to a time based system, story point or the like, it is those people who are responsible for the delivery that need to make the specific, concrete decisions. The boundary of ‘management’, just like all other agile project teams, is the prioritised product backlog driven by a product owner. The product owner is a role and not a job description and can be anyone who is pulling tasks from the team. This can and should include solution architects, if architecture members are not part of the team itself and/or enterprise architects, who will pull deliverables from the agile teams and deliver those solution building blocks into the architecture repository.

12. Regular adaptation to changing circumstances

With self-organising teams, this is a reality at the development level. However, what about the enterprise?

Agility is a big buzzword at the moment and in a lot of cases, quite rightly too. However, an enterprise as an entity has to be just as agile in the face of changing market dynamics, competitive practises, regulatory landscape, financial market position, technology innovation and other influential factors.

At the developer level, self-organising teams (principle 11) facilitate this by allowing a development team to change tack in mid-flight (on the product/sprint backlog). Whatever direction you pull it in, the team’s ‘organism’ will find a new equilibrium position.

An enterprise is often just like any other organism. In order to survive, it needs to adapt to its environment. Scaling up a development team (where each role can be mapped to a capability) then it should be possible to scale up such practises to the entire enterprise, which should adapt to its enterprise landscape. This is the Holy Grail of Agile-EA and indeed there has been some work done by the people following in the footsteps of Ken Schwaber to map Scrum methods up to the enterprise level.

As part of the architecture vision phase of each iteration, TOGAF specifies that an assessment of the enterprise, in its context, is carried out. What is in the landscape? Where are we now? Where do we want to be? Additionally, TOGAF’s ADM can basically be redefined as…

…a series of iterations each of which delivers business objectives through the incremental delivery of enterprise capability.

Let’s look at what agile software methods often suggest is the process of developing software. Agile development is

…a series of iterations each of which delivers business value through the incremental delivery of software.

I have highlighted the different word just for comparison, but they are ultimately, the same. Business value, to most enterprises, is not the software, it is what capability the software and the associated business function can allow the enterprise to have.

In short, just like any other system that can be developed, all the benefits that befit agile development can also be delivered at the enterprise level and just like at the software development level, it has a preference for tools and techniques which facilitate that context specific agile adaptation.

The Final Words…

What I would hope is that the fight between the resistive members of both sides of the TOGAF and Agile divide will come to realise that they have far more in common than they would readily admit. Agile proponents have to remember that it was not so long ago that they were the ones fighting to change the minds of the ‘anti-agile’ community, so they are sitting on the other side of the fence and some are not liking it.

Also, Enterprise Architects have to be aware that the tasks which involve communication management have to be carried out. This includes the mapping of the language of enterprise architecture to the language of IT solutions (or any other architecture domain), so reference library tasks must be completed and made public to facilitate that.

Both sides have to know their stakeholders and how to communicate with them. EAs and BAs should be doing this as part of their stakeholder management processes, but they must remember that solution architects, devs and QAs are stakeholders too and have to use language which facilitates communication with these stakeholder groups too. An example that has been put forward by an agile coach in my current client’s site may be the use of CRC cards to help developers understand high level services and their communication.

I would hope to run through an example of how TOGAF derived architectures can be developed using agile principles in the not too distant future. But that is obviously time dependent.

[UPDATE: 01/04/2012]
I just watched a presentation from Dennis Stevens, an enterprise Agile coach on how to scale agile thinking to the enterprise. It is a very good presentation and I agree almost entirely with his professional viewpoint when scaling agile to the enterprise level.

He doesn't explicitly mention TOGAF at all in the presentation, but it is clear to those who have been through the TOGAF certification process that he is talking about concepts which are also covered in the framework.

You can find the presentation on the InfoQ website at:

www.infoq.com/presentations/Scaling-Agile-to-the-Enterprise