XP Updated

Taking a look at a selection of practises from the 2nd edition of Beck's "eXtreme Programming Explained" I and others have noted a few clarifications and substantial changes in the practises, values and principles sections of the book. To me, these added a lot of credibility to the XP agile method and I want to focus my attention on some of the elements presented in the book for the next few posts.

Documentation

Many agile developers claim that documentation is waste, because it is never read or kept up to date. I have covered this elsewhere in this blog, but Kent's take on this in the second edition is that he doesn't like this claim. He stated that those who do not value documentation value communication in less in general. I totally agree with this statement and most companies that I have been to where this is espoused are incredible poor communicators, having silo developers or even pairs who are not communicating when sat right next to one another (you just get a developer in 'the zone' who doesn't interact with the pair. This just generates waste for the paired programmer and increases communication waste going forward).

Jim Coplien in his book on Lean Architecture quite rightly introduces the reader to 'as-built' documents in the construction industry. Additionally, Beck says documents can be generated from the code. You can to that through reverse engineering tools, documentation comments (such as JavaDoc/Sandcastle) and all these can be automated. However, the focus in most companies is to get something out of the door, so this sort of aspect suffers badly, as developers claim it doesn't add value. Additionally, if self-documenting code is going to tell you what it does, as a new developer, where can I find the code? This was a criticism of XP after I read the 1st edition, but Beck gets around this by stating that teams or members leaving projects have a "Rosetta stone" document to decipher where to find this self-documenting code. You could argue that another developer will just tell you, but if that portion of the project has not been touched in a long while, then will they know? If not, the tribal memory has failed or evolved and that information is simply lost. So start digging!

"Does my bum look big in this? OK, I'll get down to the best value gym"

Reflection, Improvement and Economics are the part of agile methods that I consistently keep seeing performed ineffectively or not at all. How does a team know it is getting better? How does a team show it is providing value for the client's money? How does a team continuously improve if it is not measuring anything?

Velocity is the aggregate of a significant amount of information. A story point (which btw, Beck now ditches as a quantifying unit in the second edition, preferring to go with fixed time periods) is bad enough for most people. I personally don't mind them, as long as we know what influences them. Sure, they get better as they go along, but is that because of more of an understanding of a domain? Is it just the natural consequence of moving along the cone of uncertainty? Are the developers getting better at segregating their work and not blocking each other? Was it because someone on the team had a holiday or the team shrink? Was it because of a bank holiday? Without metrics, you have no idea!

CFD's are another pretty useless mechanism for quantifying these aspects. However, you see the majority of teams attempt to use tools which give that information but nothing else. Remember individuals and interactions over processes and tools? Yes? So reflect, as a team, in your retro's and do so compared to last week and indeed further back! If you don't make a note of the amount of work you delivered, what influenced it, how much waste you generated, how many blockers you had (and freed), your defect rate or anything else, you have nothing to use to measure any improvements at all. So anything you say at that point is speculative, which is not in accordance with XP's values.

You can tell I am very passionate about this part, since every team I have seen has almost always got this wrong if they did it at all. Having a postgrad in applied maths and also pertinent commitments in other facets of my life probably makes me much more likely to see waste and other problems before others do. However, this is a practise that those who espouse Beck's values themselves make a mess of time and time again and stagnate as a team, sometimes losing the benefits they gained earlier in projects, as the knowledge of that success fades or is lost (maybe due to no docs eh? :o)

Really, a lot of reflection ties in with the value of economics and improvement as described in Beck's book. Indeed, over the years, measures such as cycle time and throughput have come to be seen as the best indicators of the performance of the team. As a reductionist, I think simplifying to the pertinent set of variables (let's use a mathematical term and call these bases which must be independent), where required, is a really good idea and can almost always be done in some form (I have yet to find a situation which can't be 'metricised'). So looking at how this links on to more objective economic and accounting criteria such as cash flow and ROI.

Technically, if you reduce to the throughput alone and add the costs of employment of the individual staff members, you can combine these to determine the cash flow for each project, where the business value is measured directly on the card (which is something Beck now strongly encourages). I appreciate that a lot of situations will see the card not deliver value immediately, such as companies in seasonal markets, where the feature may be deployed in spring and the value doesn't get realised until summer, such as some retail, leisure and tourism and travel industries (This often adds a temporal element where the phasing of a release is very important).

For example, a service for beach donkey bookings need a website to organise the bookings for several agents. 99% of the value is delivered in two weeks in the summertime (last week in July and first week in August) and the whole industry is worth £1,000,000 a year to Beach Donkey Booking Co.


The company Beach Donkey Bookings Co employ two people full-time at £25,000 a year each, starting in the spring-time, 8 weeks before peak, to deliver the software system to manage this. Their project consists of 50 separate stories, of the same value (to make the maths easy, I know this is not the case in real-life), which the team of two deliver in pairs at a throughput of 5 x 8 hour cards a week (effectively, the 2 team members are delivering 40 hours worth of work a week between them), taking them 10 weeks to deliver the software at the very end of the peak period.

As a result, assuming the dregs of the value for the investment are spread normally outside these values, they gain £10,000 (i.e. 1%) from the investment in the development of the software for this year, whilst paying the costs of the developers at £25,000 for that year. So for this year, on the balance sheet, it produces a net loss of £15,000!!!

'Hero' Programmers

Personally, I agree with Beck's assessment that you get more out of a set of average developers than you do out of one hero developer in a team of less than averages. Indeed, in the second edition of XP explained, Beck states that XP is about getting the best out of ordinary developers at any time. So where do the 1% to 2% of top level developers go?

Those who have an interest in anthropology/sociology/marketing/non-linear maths, will know that information takes hold when the population is predisposed to that meme taking hold. For example, if your local area has a bunch of gossips who are interested in everyone else's business and they hear that so and so from number 52 has had several women arrive and leave that house, that meme will spread virally. Put that same meme in the hands of one gossip in a population who keep themselves to themselves and that information goes nowhere.

Often, in IT, that one person is the smartest person in a company. Given the distribution of these people in the wider populous, where are these people going to go? What do they do? Should they be sacrificed for the good of the wider groups, given they are the people who brought science, computing, technology, engineering, mathematics and the like to the very society that now shuns them (even if they are just as extrovert and inclusive as everyone else)?

XP introduced practises that appealed to programmers, who are the vast majority of developers on the market. That is really why it took off the first time. In its aim to bring consistency to mediocrity, it certainly lived up to its expectations and I agree that everyone should strive for better. Hopefully that means some of the programmers look wider afield than their preferred area of work. Including into metrics and continuous improvement, as well as truly understanding the element of constraints referenced in the second edition of the book.

Planning Waste

Adequate planning is regarded by a lot of developers to be a waste of time. The problem planning is not the planning itself, after all, failure to plan is planning to fail. It is that the best laid plans of mice and men get laid to waste. So where is that balance?

This book doesn't really address this delicate balancing act in any great detail. Indeed, you could argue that this will very much differ from team to team and company to company, so how can it? However, it does give a strong hint as to when the planning activities should take place.

The two cycles of weekly and quarterly planning are effectively attempting to bring together the cycle and architectural planning activities, with the developers being the 'glue' that drives the smaller cycled from the bigger ones, architectural ones.

Roles

Beck delves much deeper and broader into the roles that team members have compared to the first edition of his book. He shows how project and program managers roles work under XP and also explicitly defines and delineates the role of Architect in a project.

The lack of architectural responsibility was a huge criticism that I had of the first edition of the book. Apparently, I was not the only one during the intervening years. He was always rather vague about this responsibility after the first publication, but after this publication, he is much clearer about what the role of the architect should be and I, for one, am delighted with this clarification.


Many developers have traditionally fought against architectural guidance, which they saw as being at odds with agile methods as they think it introducing BDUF. They really don't understand the role of the architect, how important they are in seaming together the systems, translating business domain knowledge into systems, dealing with risk, stakeholders and NFRs. I sincerely hope people who read the second edition of this book will get a glimpse of what the role of architect is in this framework.

XP Revisited: Part 2 - Beck's Coming of Age

As mentioned in my last post, this weekend I got hold of, and finished, the second edition of "eXtreme Programming Explained", written by Kent Beck and Cynthia Andres.

For at least the last decade, this has probably been one of the most influential books to be placed in the hands of software developers and engineers. With it and its subject matter having been mainstream for a good while now and readers on Amazon stating how this edition of the book is very different to the original (and not always in a good way), I decided to read through the second edition to compare it to both what I remember about the first edition and how well or badly the software development field has applied these concepts in practise.

I read the first edition in 2002 but was totally unimpressed. The book was far too software developer focussed and called for practises which claimed to deliver software faster and at less cost which I felt would not automatically be the case, as it would introduce a significant amount of rework (aka Refactoring) at each stage. I thought this would do nothing to shorten a 'complete' release of software. If you consider one release of software in a 'big bang' and all the smaller releases of software in XP or agile methods in general, the amount of development is, of course, about the same to get a finished product. The benefits lay elsewhere, in areas such as risk and incremental value creation.

I am a big believer in software engineering and not 'craftsmanship' and at the time was a heavy user of formal methods and languages, such as UML, OCL, Z/VDM and RUP (Indeed, UML with RUP is still my preferred method or development, but with short iterative cycles, placing collections of use cases, analogous to stories and features in the hands of the business users at each release). Seeing as RUP is in itself an iterative method, I didn't think there was anything strange or unusual about XP doing this too.

Additionally, I had been using self-testing classes for a good few years by the point at which I was introduced to the DUnit testing framework for Delphi in late 2001. Self-testing classes, with methods segregated from the main bulk of the code in the same class by the use of compiler directives, allowed the software to test itself and obviously the class access to its own private and protected methods within 'Foo.Test()'.

There are a few drawbacks with this, don't get me wrong (such as needing to know to switch the configuration over or disable to compiler directive or more seriously, the need to create new tests each time you refactor if you wish to keep the advantage of private method testing) but the ability to test private/protected methods allowed much finer grained debugging that can be done when only testing the public methods if a class.

I spent a significant amount of time playing e-mail tennis with Ron Jeffries batting the XP ball around at the time. What surprised me was the effort he put in to critique the fairly small-fry elements of other methods. Indeed, sometimes, some of his comments seemed a little like critique for critique's sake. I still remember the conversation about sequence diagrams, where we discussed how to check things work according to the activity diagrams before building any code - Note, I have come to argue this can be considered a 'test first' activity. He used the statement "How do you tell the difference between a sequence diagram showing a whole system and a black cat at midnight?", which is a fantastic analogy, even though that is not what I was saying at all :-D I break sequences down into component and package level entities too, as the software is assumed to be loosely coupled, most scenarios which result in a high linkage between two packages can be seen to not be cohesive and you can tell the segmentation into those specific package is wrong. So there are as many ways to figure things out from models and diagrams as programmers can see in code.


I attempted discussing the pros and cons with many different people over the years but found no reason to say that XP or Agile methods in general was any better than the RUP process I was using. The lack of strong, cohesive, non-contradictory reasoning from anyone I discussed XP with over the years (that didn't appeal to the programmer) didn't help and indeed, for the first few years of the 'agile revolution' I could easily argue that a company's investment in, say, RUP or later Scrum would be a much better fit with existing corporate structure, at least initially. After all, the vast majority of companies are not developer led. The aim fo the majority of companies is not to develop software. So unless a company is willing to structure itself to segregate the development of software to a different organisational unit, including the use of transfer pricing for charging models (i.e. inter-departmental), then unmodified XP was a loser methodology in terms of adoption. This was not helped by Beck's insistence on the source code being the final arbiter in the first edition, which I felt was both narrow and small minded, as was a lot of the vehement statements in the first edition.


In the intervening years, the references to other fields that agile developers often quoted, without any analytical or empirical evidence to support their claims was startling (indeed, even the conjectures were very badly thought out and this is still the case today). They claimed to be lean, but don't understand it. They claimed to revel in the importance of Kanban, but again, don't understand it (ask pretty much any developer what the equation is for, say, a container/column size and the response you often get is "There's an equation?" *facepalm*). They quote religiously continuous improvement but don't measure anything (sometimes not even the points flow!?!), so have no idea if what they are doing is working and what caused it. Woe betide anyone who mentions alternative confounding paths or variables (most developers won't have a clue about factor analysis).


So all in, given the years of garbage I was often quoted by some development teams, I was fully up for a fight when reading the second edition. I got my pen and a notebook and noted down the reasoning for each of the salient points, including the values and principles (which I didn't have too many significant issues with the first time) and the new split of 13 primary and 11 corollary practises plus any comments he made that I didn't immediately agree with, so see how he addressed the reason for them as the book progressed.

Surprise!

What I ended up reading was a complete shock! The only way I can describe Beck's take on software development in the second edition is mature! Very mature! Having read the first edition, I started wanting to tear this work to pieces, but actually, with about a third of the book being rewritten, his slight U-turns on some of the things he presented in the first edition and his admission that he wrote the first edition of the book with the narrow focus of a software developer, increased his stature substantially in my eyes. 

So that leaves me to point the finger of software engineering mediocrity in this day and age firmly at the software developers themselves (indeed, Beck himself has criticised some of the claims by modern agile developers). If you read the second edition you will see what I mean. I shall cover some of the more salient points here over the next few blog posts, but I just wanted to say that if you have adopted XP from the first edition, then read the second edition! There is a whole world view you are missing out on.

XP Revisited: Part 1

I got hold of the second edition of Kent Beck's seminal work "Extreme Programming Explained" and am now making my way through it (again).

Although the 2nd edition was published in 2005, I figured given changes over time that I would look at how it has changed and also how people have implemented XP in retrospect (effectively feedback).

I read the first edition of this book back in 2002 and thought that it was the biggest load of codswallop I had ever read and still do. The practises it advocated were certainly no better than I was already using (in some cases they were significantly worse) and there was such a lot of ambiguity, contradiction and conjecture that I felt like I had really wasted my time reading it.

So I didn't hold out much hope for the second edition. Reviewers had criticised the book for deviating too much from the original, not being specific and indeed recommended the original copy be found.

However, having got so far into it, it has become clear through some of the rewrite that our industry, which has worn the 'agile' badge for so long and the critics who so vehemently defend XP against other methods and claim to work in that way, are fundamentally wrong in their implementation of the principles and practises.

I am surprised to now be sat defending Kent's principles presented in the book. It has validated my stance that the problem with 'agility' is the manifestation of it in industry and not necessarily the method.

People claiming to be agile are simply not employing the principles as stated in the book. However, I shall defer judgement long enough to finished the book, just in case I have to report on 'embracing change' in the tone or inference of the author.

Watch this space!

'Dr' Richard Stallman...

...has wasted me an hour of my life that I will never get back and I am miffed!

I went along to a lecture given by the eminent founder of GNU and the school of Free software which was timetables for an hour and a half... or an hour depending on which site you read. If I am being kind, I would have to agree with the filibusters calling for the free software movement to find a new voice.

Introduction to Merchandising

'Dr' Richard Stallman, with honorary doctorates from several universities, began the talk by selling free software badges and memorabilia, such as "don't SaaS me" badges, for between £2 and £8 pounds. OK, he is taking advantage of the capitalist movement to further his cause for free software. Fine, I have no problem with that, since that is somewhat the way I would go on a crusade. He went on to espouse his already familiar belief that software should be free to anyone and that software should be inclusive.

As part of his request to put videos or audio of his talk on sites running only free software and in a free software format (such as .ogg files instead of .mp3), he mentioned that Facebook is a surveillance engine.

Interesting point of view I thought. To me, Facebook is a site offering a service, which gathers data and one of the ways that this data 'could' be used would be 'surveillance' but you could equally argue that is also building useful marketing trends, usage stats to improve Facebook and optimise specific areas of the site amongst a host of other things. I happen to agree with the marketing edict that "If you are getting a service for free, then you are the product not the service", so I can see where this could be pertinent, but in reality, in the basic form it is just 'data' (and remember kids, data is just data until meaning is attached to it and in that that case and that case only, it becomes 'information').

He also went on to criticise Windows for destroying the resource that is you. It destroys the people and the freedom of the people. The reasons he stated this are not 100% clear, but he attempted to state that it blocks access for that resources to software and mass computing and so are spoiling the resource that is you.

To me, this was the point at which I figured this was going to be an abysmal talk. But I initially did the respectful thing and stayed for an hour and 10 minutes before having to walk out in disgust.

Thought: Cultural Tech Knowledge As A Sliding Window

I have had many a discussion on various types of technology over the years and have come to the conclusion that technology shifts cultural knowledge along as a kind of 'sliding window' over time. For example, the introduction of the calculator, made the populus as a whole less able to do arithmetic, but we gained how to use a calculator or anything with a numeric keypad... including the numeric keypad. The introduction of computers with WIMP/WYSIWYG editors, means people lost the understanding of formatting syntax, an understanding of command consoles and to some degree typing skills. Those buying computers with GUIs made people program less. In all case, feeding the apparent human need to find the path of least resistance meant we got to learn these 'labour saving' devices and forget the harder ways of doing the same job, sometimes to our detriment in the 'intermediate years' of each.

Stallman started to talk through 9 threats to freedom and free software. He went on to mention surveillance as a recurring theme throughout the talk, citing Facebook's compliance in handing over your data to the authorities on request, the anti-copyright lobby initiating a propaganda campaign against free software, human rights abuses in the iPhone/iPad factories etc.

He mentioned that proprietary software logs usage data on people, so companies can keep tabs on what you do (there was certainly machine code disassembled in the Windows 3.x environment that indicated that if a network was present, certain information could be passed back to a host).

4 Levels of Freedom

He stated that Free software allows you four levels of freedom (0 to 3) which included freedom 0, running the software as you wish, which would apply like freedom 1 to individuals, but also freedoms 2 and 3 exist would would allow you to build a community with people "if they cooperate" (which I thought was a very authoritarian stance to take for someone with his standpoint).

He claimed 'the community' would tell you if there was something wrong, 'the community' would give you support and help. He identified that not everyone is a programmer or has the skills to program, so the community could do that for them.

Please Sir Linus, can I have my ball back?

Stallman started to point out that whilst working on the Kernel for his Free software OS, he discovered that him and his team were in for a long-haul and thought that they would be around for years trying to get the kernel done. Then along came Linus Torvalds and slotted in his Linux kernel into the middle of all the other things that Stallman and his team had created and so the platform had become 'Linux'. So he would like us to call Linux GNU/Linux and give him and his team equal credit.

This happens to be a story I have heard from other sources, so I am not actually miffed about it.

The Digital Divide

Stallman went on to state that proprietary software creates a society who are divided and helpless. They either can or can't program and can't modify the software. Aside from that being complete rubbish (you can modify almost any at the machine code level/IL if you work hard enough at it. Let's not forget this is how crackers make it happen), using free software doesn't solve this problem at all. In fact, it makes the divide much worse, as less people from this time or at any time in computing history would be able to program their own software, so most would be both divided from those that can program and are experts and be helpless to deal with a problem without the support of those people. If he is arguing that writing software should be part of the fabric of society, then making free software available in the sense he means it would be wholly counter-productive.


He criticised Steve Jobs on his passing last year and drew a lot of criticism for it. Indeed, as other bloggers have already pointed out, Steve Jobs brought computing to the masses and changed the game fundamentally. Something Stallman has failed to do since 1983. I agree that very little of Apple/Jobs' work was new, however, what he did was identify the profile in society which needed a particular device, created a market for it, and then sold to that market. Stallman has failed to do this at any point, preferring his stance to come from a purely technocratic crusade when all people want is the labour saving device to save them time, keep in touch, same them space, get online, share things etc. Stallman fundamentally failed to show the world there was a problem and offer a solution like Apple (and indeed a lot of proprietary software did). Apple happened to identify the problem at the right time and marketed in the right way. Even though I  personally don't like their products much at all, I have to commend the marketing skills that Apple had under Steve Jobs. The had their finger on the pulse at all points, their market and brand awareness were exemplary and very few companies have matched them since. Maybe Samsung since, but they obviously were no the first. 


The supply of proprietary desktops to the classroom was another issue that he went on to target. My counter argument is that schools are woefully under prepared should anything goes wrong. Generally the UK public sector ICT jobs are incredibly low paid relative to the private sector. As such, won't appeal to the very highly skilled who can earn 6 times as much. So the support isn't there. People often purchase support for piece of mind and IT retail businesses know that. That is why the "extended 3 year warrantee" is often purchased by those not in the know. They want that piece of mind. 


Similarly, schools need the support contract and they need it with people who know the infrastructure in detail (usually having fitted it), understand the platform and are reliable. Free software doesn't have that one person/organisation they can turn to. So understandably, they are worried. After all, if 999 didn't exist (it celebrates it's 75th birthday this month), who would you call in a major emergency? Your mum? Your mate the badge selling Dick-Stall man?... sorry, typo :-S

Hypocrisy 

He said words to the effect of "Proprietary software is supplied to schools in the same way drugs are supplied to children!" and then in his pitch about "the war on sharing" (copyright and legislative frameworks designed to stop it) several minutes later, made the comment about anti-copyright propaganda. 


"WTF!?!?!" I hear you ask "Did he honestly make the connection between school kids and drugs, then complain about a propaganda campaign against him and his organisation?" Yes, I can confirm, he definitely did! That was the point at which the man lost all credibility with me and reduced him to a a giant, hairy blob of hypocrisy.

Don't SaaS Me!!

His 9 threats to freedom then included a criticism of SaaS services such as file storage apps (implying the likes of dropbox and G-Drive) and referred to how the "Pat Riot Act" (Patriot act) gives the US authorities access to your data from a provider without needing a court order. He also criticised PaaS/SaaS environments because the user effectively has to upload their data onto the service to run... which in my mind is the same as punch cards/mainframe system of days of yore. Mainframes can still store data or pipe it to a PC to be stored on disk and yet he kept exclaiming that any systems which the user has no control over is a threat to [the] free software [movement]. In any case, there is no difference in security risk as both mainframes and SaaS introduce a system that the 'resource' has no control over.


In reality, people would struggle. For example, how many people in your friends list are not programmers? Given you are reading this blog, there is a good chance the figure you came to is overstating it, as being nerds/geeks we tend to stick arond our ilk. We are the people too many others turn to when they have computer problems. Indeed, some geeks have developed coming strategies, so that when asked what they do for a living the reply is "erm... I am a refuse collector. I collect refuse!"

Don't Vote On Computer!

Cool! Thanks! I won't vote for you Stallman, whomever the Free Software Foundation decide should be their next voice, I will attempt to vote for them. I don't care who it is! Torvalds, Gates, Balmer, Cook whoever! Just get the chair from under that guy! 


I used to have a certain respect for the free software foundation several years ago. The FSF/OSS movement brought the battle to the Windows/UNIX platforms in the enterprise, at one point making up 60% of company servers and caused Microsoft to really look again at its server platforms. Indeed, I was in a focus group in London in 2001 where Linux was brought up in a question by the facilitator.

Summary

The Open Source community were right to splinter off from him and his ethos. Myself, having held the free software movement in fairly high regard for its achievements in pushing well in to proprietary software territory in the server space, I was sorely disappointed with Stallman's contradictory, hypocritical and nonsensical rantings which seemed somewhat detached from the way the market dynamics have worked. It is not that a lot of his sourced statements were wrong, but the meaning he attached to that 'data' was so far off it bordered on, dare I say, lunacy!


I finished my working day 30 minutes early to drag myself and a poor unfortunate to see this guy. I lost income due to this and I am very definitely not going to recommend seeing Stallman talk. I have to say, I should have heeded the advice of others who had experienced his one man 'cult' at work (I am afraid that is how I see it) and I can't say I can recommend this at all. The FSF need to find a better voice to take them to the next level. This has to happen to keep their crusade alive and give consumers options, as the more Stallman talks, the worse it will get. 


I can imagine some free software veterans saying "God MAN! Shut-up SHUT-UP SHUT-UUUUPPP!" and frankly, 70 minutes into the lecture, I really wished he would too!

Windows Azure, my first look

I started writing this post from the first class lounge at Euston station, having finished my busman's holiday in London. I was at the London Windows Azure User Group's showcase of the new features of the Windows Azure platform.

To demonstrate the platform, a number of the Windows Azure Evangelist team arrived at Fulham Broadway's Vue cinema (all screens booked for the event) to demonstrate the platform across a number of track with supporting acts from the likes of SolidSoft, ElastaCloud, Derivitec, some of which were headline sponsors of the event.

The one and only Scott Guthrie, Corporate VP of the Windows Azure Application Platform, started the presentations the by outlining the new functionality available on the cloud platform released in the last couple of weeks. This coincides with the release of the new SDK. Unfortunately, despite his stated 99.95% Azure SLA availability, he had no control over the availability of the internet connection within the cinema itself. So predictably perhaps, the network connection went down and despite the efforts of some brave souls in handing over 3G dongles and using their phone data plans, it took a long while to get back online, with the help of a huge 100-BaseTX cable, which was so long the limits of that standard were breached, which meant  that an impromptu break was organised and a laptop which could work with such a poor wired signal was found to run the rest of the presentation on.

Scott Guthrie, before the network connection went down.

I went in there with my architecture hat on, to see what the platform had to offer and to see how the platform can be used to lower costs and deliver better value and to take away how to approach the decisions on whether or not to support or use Azure in the enterprise.

An Introduction to/Recap of Cloud Computing

To recap on the phenomenon that is cloud computing, the idea behind hosting on cloud computing infrastructure is to provide potentially infinite scaling of computing power to meet the demands of the services you wish to provide to your customers, whilst lowering the total cost of operating the platforms you would traditionally run in house.

This includes the ability to deliver extra computing cores, extra memory and the like for a linearly scaling cost through  a 'pay-as-you-go' business model which allows you to pay only for what you use.

The general provision of cloud serves take three main forms:

• Infrastructure as a Service (IaaS) - This provides the 'bare-bones' platform for the end-user. The services are operated and maintained by the Azure platform. This often takes the form of the provision of a virtual machine with guest operating system under Azure.
• Platform as a Service (PaaS) - On Azure, in addition to the services provided for IaaS, platform serves such as database server, e-mail services, and web services are provided, operated and maintained for the end-user by the data-center. In addition, Azure offers the ability to provision services such as the Azure Service Bus and WebSites through this service model type.
• Software as a Service (SaaS) - In addition to the provision of the PaaS services, SaaS provides software services for the end user. On the Microsoft platform, the MS Office 365 environment is an example of a Software as a Service provision model. 

Below is a diagram depicting the relationship between the three types of service provision above.

cloud service provision types.

IaaS was available in the previous release of Azure. However, what interested me with my enterprise solution architect hat on was the provision of PaaS infrastructure. The Service Bus element especially ties in very nicely with ESB platforms that are currently making the rounds as the latest fad. So over the next few weeks, I am going to spend some of my included MSDN Azure minutes on finding out about that part of the platform.

Other benefits also include the implicit 'outsourcing' of the management of the platform to the in house Azure data-center staff, of which there are not many at all. The data-centers are designed to be operated and managed remotely, with racks of 25,000 servers set into a modified shipping container which is just hooked up to power, network and cooling before being let loose on the world. 

Scott showed a short montage of clips showing how the containers are put in place in the data-centers. When servers fail, they are simply left in place until a large enough number of servers in that container have failed, before the entire container is disconnected and shipped out to be refurbished/repaired.

Azure Cloud Availability 

Windows Azure claims a 99.95% availability per month for their cloud infrastructure. This is their SLA commitment. 

Now, as was made clear in other presentations on the day that there are no guarantees. The 99.95% SLA commitment is just a reflection of their confidence in the Azure platform. For those of us that have any experience with infrastructure, or have an understanding of terms such as 'three-9s', 'four-9s', 'five-9s' etc. then you will appreciate the sentiment and also the costs involved in claiming any more. Their SLA put them at the same level of availability as Amazon EC2, but higher than Google's Cloud service offering. 

The service provision of worker processes or VM instances is kept at that level by distributing 3 instances of your image offering (whether that be PaaS, SaaS, IaaS website offering or whatever) across three servers which have no shared single points of failure, thereby reducing the probability that your entire platform would be affected by an outage in any one of them.

This makes perfect sense, as distributing the load across diverse servers distributes the risk across a wider set of failure points (thereby reducing the risk that any single failure takes more than one server out). In addition, the Azure data centers replicate their server data across at least 500 miles of geographical space into another Azure data center. There are allegedly secure links to do this, so we were assured that the channels used to replicate the data are uncompromisable.

Cloud Services Available

Azure services are divided into 4 main streams:

• Websites - An PaaS option which allow you to host up to 10 websites for free. This applies to anybody using the Azure platform, but bandwidth is chargeable out of the data-center. 2GB of data is provided at 24 cents per month. Again, you can increase this limit if you wish, but be aware it is an opt out service and not an opt-in. So you will be charged should you not change the default. 
• Virtual Machines - An IaaS provision which allows for the creation of a number of virtual environments in Windows, different flavours of Linux or both. Again, georedundant storage is available.
• Cloud Services - Additional computing functions, such as Service bus, worker role assignments, storage and the like.
• Data Management - Different types of computing storage, such as BLOB storage, DB storage and management on platforms such as MS SQL Server and now MySql.

A number of additional cloud services across the three layers are available. However, more are being added each month. Unfortunately, I didn't see the Service Bus elements in any detail, but cloud services can be added to standard packages. These can be any or all of:

• Web and Worker role instances - Sold in the computing unit sizes of XS, S, M, L, XL. Having had a more detailed look at the website, Apart from the extra small computing unit (Single 1 GHz CPU, 768MB Memory and 20GB storage), the rest of the options are based around the 'single computing unit' being (1.6GHz, 1.75GB memory and 225GB storage space). These scale linearly in the two dimensions of computing unit and number of units.
• Storage - Extra Georedundant storage elements (where the data is stored in a different regional data-center) can be purchased up to 100 Terabytes for each processing unit. We were told that it could result in a Petabytes of data for some services.
• Bandwidth - Same as usual
• SQL Database - Unlike the others cloud services, this is the only service that doesn't scale linearly for the whole pricing model. The first GB is $9.99 per month, but after that, and especially when you get towards the 150GB mark, it is dirt cheap.

A lot of these options are replicated in the Data Storage part of the cloud service delivery model. You can choose not to have your data stored georedundantly, as Azure effectively creates a mirroring of your data across three Azure storage units. The presentations around the data storage elements indicated that there were both SSD and mechanical drives present in data-centers, but that the SSDs were being used to cache data. I asked Scott what the ratios were compared to the mechanical sizes and whether they were shared across all computing units for all users, but he couldn't give me the ratios and he was half way through trying to fix the internet connection at the time to answer the question fully. 

Multi-platform Support

Various demonstrations were set up to show the use of multi-platform deployment. These included the use of open source as well as Microsoft platforms with the aim of showing how these run out of the box without any extra config. 

Scott showed examples of nodejs and PHP code running straight out, whilst other tracks saw the use of Java on the Azure platform and Brady Gaster showed the open source track the use of a multilanguage site using classic ASP and PHP as well as the standard .NET toolkit. 

Whilst I can't imagine any of this being incredibly difficult in Windows, given that it only requires an ISAPI library to be able to run any of these as it stands, it is useful not to have to do the config yourself.

There was also a demonstration centered around the HPC capabilities in Azure, using the Azure HPC SDK elements. However, again, due to the internet connection having problems, the demonstrations were left a little lacking in response times.

Yosi Dahan explained the use of Hadoop to the uninitiated (somewhat including myself), though the information presented didn't include much that I didn't know already. There was no demo for this one, despite the billing, but given the problems with the internet that day, it wasn't likely to have been very good.

Microsoft are aiming to embrace the Hadoop platform for use in the cloud. Yosi stated the standard OSS version of the code was not enterprise ready, given there is hardly any security surrounding it. Microsoft are working to improve this and other aspects of the platform, before giving the changes back the Hadoop community. This was the second of two open source presentations which showcased the Azure platform as a place to host OSS sites. It is an interesting tack and one which Yosi himself stated that Microsoft has not always been good at (...at all I would say ;-)

Clouded Architecture Considerations

The 'Thinking Architecturally' presentation by Charles Young from SolidSoft highlighted that the cloud offers a unique way in which to provision infrastructure and platform service to end-users. Charles asked the audience if anyone could bill themselves as working as an Enterprise architect or work in or with enterprise architecture. Given I have a TOGAF certification and am a member of the Association of Enterprise Architects, I figured I could just about raise my hand... and was the only person to do so.... cringe city! A similar question to the floor for solution architects, for which there was a much better response, including my second vote ;-)

He presented the two sides of the architecture domains to the audience. Initially starting with enterprise architecture, he used the cloud costing models to illustrate typical investment forces which could lead down one path or another. However, Charles didn't sing the praises of every bit of the cloud infrastructure, in either the enterprise architecture or solutions architecture domains. I happened to like that as it showed a balanced viewpoint, which is what I was there to see. Note, architecture is often about trade-offs, and in order to do that, you need to know what those trade-offs points actually are. 

Charles referred to cloud computing as a 'game changer' which I certainly agree with, as the costing structure will certainly influence the financial forces at work in migration planning stages of any enterprise architecture strategy. I would suffix the words 'once it reaches critical mass'. This will most certainly applied across the board and industry. The usual question with such innovation is when will it reach the critical mass necessary to make this spread like wildfire? This would take it into all facets of the industry and become the de facto standard for deployment. 

Given the extreme examples of costings that Charles used as examples from his client list (the latter of which appeared to show that the operational costs for 10 year deployment would be 0.33% of what they would be in a traditional in-house hosted solution). However, Charles did indicate that those were extreme examples of money saving effects and that most will be much closer. However, even then, the savings would be big enough to be a 'no-brainer' for most accounting functions or investment committees. So there would not be any concern from this set of functions within an enterprise.

Security

Despite the insistence of SolidSoft and others that the network infrastructure is secure (and I have no doubt it is) the traditional in house functions responsible for the day-to-day operations of a company's infrastructure seem to win out from the 'safety' aspect. Security managers/architects still tend to have problems with the idea of cloud infrastructures and the security mitigation that the Azure data centers have put in place do not cover all of them at all. For one, development and infrastructure teams will have to become more adept at dealing with security issues outside their control, and make more use of secure channels to and from these data centers.

The worry, which I certainly think is a legitimate one is how to ensure compliance according to the legislative frameworks we currently have in place in some architecture landscapes. Some of the organisations that stand to benefit the most from cloud computing are the very ones who can both invest in it, pushing the market simply by the numbers and also the very ones who stand to be hit the most by any legislative data security issues. 

Unfortunately, my question to Charles surrounding the PCI-DSS standard were not answered, though this is due to the SolidSoft representative not having had experience of implementing it in the cloud. Also, given I was told that a delegate before me in the queue had already asked a similar question, it is certainly something that will have to be addressed before companies falling into the higher levels of the standards, who stand to lose the most should any of them be found in violation could sensibly take this on. For all the ease of scaling that cloud services provide, the trade off is that there will have to be greater emphasis by companies on the securing of the channels that would be needed to make it work realistically, against the backdrop of said legislative frameworks.

Sky High Costs?

For those who pay (or will pay) for cloud services, what is interesting about the costs of cloud models is the way it scales.

A traditional data center setup would involve an enterprise setting up their own hardware resources and running their own operations. Imagining a badly paid IT manager, with 25 servers running 24/365, but requests to them only run during a working day, plus the electricity for the cooling and the servers. The servers and cooling infrastructure alone are an upfront payment towards resources which may or may not be fully utilized. Similarly, so will the fixed cost of salaries for the poor badly paid IT manager and the cooling and electricity for servers which are on all night when very little is being processed. 

Contrast this with the per hour model of Azure's cloud service. 

In their Paas/IaaS model, depending upon the processing resource you require, it is a linear cost. If you need a processor with a single 1 GHz processor core, this is their extra small processing unit and costs very little. So much so, this model you (or anyone else, regardless of whether they have purchased Azure time or not) get for free in their Websites environment. Getting a single dedicated 1.6GHz processor requires their Small computing unit setup. This can either be a Windows and now a Linux virtual machine, which affords the purchaser one of two ways to target and distribute their services.

Additionally, the WebSites cloud service offering can provide 10 'small' scale websites for your business, included pre-created templates (such as WordPress), potentially with a 100MB SQL Server database for free (though very definitely big enough for a lot of small business needs). Both ANAME and CNAME records can be used on Azure, as there were previous concerns that the Azure platform had trouble with linking to domain name registrations which would override the 'mysitename.azurewebsites.net' style ofnaming. This will go some way to appeasing these concerns. 

Summary

On the whole, the day provided a useful insight into cloud computing on the Azure platform. There were  a number of presentations and it was not possible to catch all talks from all tracks. So there will no doubt be others who will enlighten use with their different viewpoints. 

The latest version of Azure certainly offers a richer environment from which to work and the rolling, potentially monthly, deployment of other cloud services, templates, platforms etc. I am looking forward to jumping in to the service bus elements of the platform, to see how it stacks up and what functionality it has (or has not) got in comparison to an in house ESB offering.

Watch this space...

I got the gift of TOGAF

I passed my TOGAF certification at the beginning of last week.Whilst studying for it in my spare time over the past few months and trying to apply it to my everyday role, I came across a large number of statement which validate my stance that enterprise architecture methods such as TOGAF and agile methods can live very happily together. TOGAF is regarded (by misguided developers in the agile sphere) as 'a bureaucratic nightmare' because it is "up front design" using "documentation and models" which delivers everything in a "big bang" - All of which is entirely false.

Some snippets from the TOGAF 9 specification and related literature (there are many many more):

"A capability will take an extended time to deliver (specifics will be a function of the organization and industry vertical) and will normally involve many projects delivering numerous increments. In addition, the capability needs to provide real business value to stakeholders as soon as possible and maintain momentum to achieve the Target Architecture as well as the associated executive support and corporate funding. Therefore, it is useful to break the capability into capability increments that deliver discrete, visible, and quantifiable outcomes as well as providing the focus for Transition Architectures and the deliverables from numerous inter-dependent projects. These outcomes are the Critical Success Factors (CSFs) for continued capability support."

Now map:

Critical Success Factors -> Acceptance Criteria

Capability increments -> Story cards in a feature or features in an epic

Indeed, you can do the same with:

"Just Enough EA" -> "Just Enough Up-front Design"

Where 'Just enough EA' is usually characterised by just enough detail (and volume) to allow directed initiative (the Gartner research group goes one step further and states "Just enough Enterprise Architecture, Just In Time", the latter half of that statement should resonate with Agilists.

NB, TOGAF is a very abstract EA framework. It can be applied in a variety of ways tailored to the organisation and can leverage other frameworks and methodologies to actually deliver the capability of which IT would be one part. It works across the four domains of architecture of Business, Data, Application and Technology. So not only does IT have to bring itself to the party, but the business has to do their bit first, with information systems and then technology following on from there. So it is much more than delivering IT, but it currently has that focus, given a lot of solution architects find this a requirement for their jobs.

What I would recommend for those agilists who want to see more light and scale agile methods to the enterprise, is to view Dennis Stevens' presentation on "Scaling Agile to the Enterprise" where he outlines methods to deliver an agile enterprise (not just agile development). For those with good TOGAF knowledge, the mapping to TOGAF ADM, artefacts and deliverables is too striking to be able to ignore, especially when addressing the whole system in transformation, not just the individual elements.

TOGAF luuuurvs Agile!

There is a common misconception that I come across day-to-day in some environments which seems to indicate a disconnect between the opinion of TOGAF in the development community and what it truly entails.

This is particularly evident in the world of agile methods, where such things as ‘up front design’, ‘documentation’ and ‘long term strategy’ are often cited as reasons to compete in the ‘urinational’ championships.

The truth of the matter is Agile and Enterprise architecture frameworks cover different domains, though agile Methodists can’t seem to generalise their thinking into the enterprise architecture world whilst TOGAF practitioners definitely can scale down (even if they can’t write code). This post will attempt to address the 12 principles of the agile manifesto and relate them to TOGAF 9. However, in order to do that, an incredibly quick, summary crash course in TOGAF will be ‘just enough’ information to get the job done :-)

TOGAF 9 Overview

The Open Group released the version 9 of the TOGAF standard in 2009. It is a set of structured tools, including an iterative Architecture Development Method (ADM for short), reference models, content framework,
enterprise continuum and repositories for the production of a set of interconnected functional and reusable enterprise capabilities for the satisfaction of varying levels of strategy, when constraining a business (which is just a system like any other) by factors such as legislation, finance, time, organisational capabilities, business goals and other constraints.

What this allows is the ability to bridge the gap between the CEO/CIO vision and strategy and the operation of a company in terms of its capabilities to deliver that vision, including the IT functions.

TOGAF concentrates its architectural effort on four domains of architecture. Namely:

• Business Architecture
• Information Systems Architecture, which is subdivided into:
• Data Architecture
• Application Architecture
• Technology Architecture

...and it can be applied at many different levels by going through an ‘architecture partitioning’ exercise which create strategic architecture (which may be a 5 year plan for the whole enterprise), segmented architectures and capability architectures, in order of increasing levels of detail and decreasing levels of scope.

An IT department is but one capability in a myriad of others, such as HR, Customer Services and Finance. It just so happens that it is significant enough a capability to have dedicated responsibilities in phases of the ADM at the capability architecture level.

Additionally, TOGAF specifies the use of Architectural Principles, which are reasoned, qualitative, yet normative statements which constrain the architectural solution space in some way. They can be based on regulatory, best-practice, performance, security, auditing dimensions, amongst a host of others, but they generally pertain to one of the four groupings of:

• Assurance
• Adaptability
• Availability
• Usability

However, in all cases, they must be SMART goals.

• Specific
• Measurable
• Achievable
• Realistic
• Time-boxed

When the ADM is started, there can be many cycles at many levels and they may all use the ADM to deliver the capability. So a strategic enterprise architecture exercise may initiate a 5 year plan together with segmented architectures to look at which are segmented deliverables, which will in turn be broken down through their own ADM instance to capabilities for each segment, which can be delivered using the ADM. Alternatively, an iteration of the ADM may deliver a higher level architecture, which initiates the development of lower level capabilities after it has been completed.

An IT capability can easily be delivered using agile principles, but a few things have to come to the fore and the TOGAF terminology, which is foundational in nature, needs to be augmented with and a mapped to common systems terminology (such as security, auditing, monitoring functions), industry architecture concepts (such as those general to all travel agents, or HR systems) and specific organisational terms.

How does Agile fit within this?

Enterprise architecture, whether agile or not, is the bridge between the business strategy delivered by a board and all the different capabilities that exits in an organisation. The corporation’s vision, portfolio, strategy and governance are input to it and the capabilities (of which IT is but one part) are output after each iteration of the ADM, either in the form of transition architectures or target architecture.

Agile methods deliver the functional technology that makes the application and data architectures happen, which are in turn delivered by the technology architecture phase. However, half the problem is that an exercise connecting the terminology used by enterprise architects and solution level developers is normally never carried out.

TOGAF mandates this exercise as the framework is very abstract and The Open Group were so aware of the effects of this abstraction on the use and uptake of TOGAF, that they strongly encourage the use of a reference library which maps TOGAF terminology to the terms used in specific capabilities in the enterprise and/or to the enterprise as a whole. Without this, a large amount of confusion can occur, for example what does a ‘service’ mean to different people?

The Agile Manifesto’s 12 principles can be mapped something like the following (amongst a myriad of other ways):

1. Customer satisfaction by rapid delivery of useful software

The difference with TOGAF only comes to the fore when the definition of ‘customer’ is looked at.

In an enterprise architecture exercise, the customer is internal. Indeed, there may be solution or enterprise architects in the interested stakeholder groups. The value realisation may take months to come to fruition, but the agile development of a piece of functionality by the IT capability may be a small section which only delivers its value when the whole jigsaw is in place. This is in itself, valuable! As the business value has already been assessed by the business participants and this is the reason for the development exercise and it may indeed contribute to the exercise as a whole with a monetary value.

To link TOGAF to Agile, we have to be aware that the product owner is usually the solution architect responsible for the delivery of the solution building block in the solution continuum. It is categorically NOT the development manager and may not even be the business owner. The business owner will have delivered their capability requirements to the enterprise architects who will have delivered their prioritised strategy to the solution architects who in turn will have segmented the solution architecture into software components (which deliver capabilities) which they will pull from each team.

2. Welcome changing requirements, even late in development

This is good from TOGAF’s perspective. The reason is because TOGAF encourages ’Just Enough’ design, as it is under no illusions that the requirements (fed by the organisation’s environment) will change during the lifetime of the larger partitions’ realisation. This ties in very well with the agile proponents’ viewpoints on the matter, though the ‘no turning back’ point in TOGAF is obviously of a different scale to the Agile processes.

In order to facilitate ‘welcoming changing operations,even late in the strategy’ certain infrastructural matters need to be carried out and they may include using an emergent/middle-out architecture. However, this is just the same as a Kanban board, metrics, elicitation meetings, spring planning or anything else. It certainly is not an alien
concept to agile developers.

Enterprise architects, together with the Solution architects may use Service Oriented Architecture to deliver the building block service capabilities, including, say the use of an enterprise service bus (ESB) to facilitate the communication. Once done, orchestration processes, which are basically workflows, can be designed by relatively small teams to change the operation of the system within very short timescales if new capabilities are not required. Indeed, it is possible to re-orchestrate well-designed, existing services in an operation significantly faster than an agile development team can create a new software component.

3. Working software is delivered frequently (weeks rather than months)

This is the same as stating that an organisation needs to be able to change direction quickly and change its internal operation to match. SOA platforms deliver this using orchestration processes where flow of process or information is sent around the company to perform a particular task. Consider it like be able to restructure [aka refactor to agile dev readers] the whole organisation in a very short space of time, to take the company where it needs to go next.

In TOGAF, each iteration of the ADM generates a new transition architecture, which becomes the baseline architecture for the next iteration starting at the architectural vision phase. A GAP analysis is carried to identify the difference in capability between the baseline and next transition/target and those that are selected for the ADM are realised through its phases.

Agile methods work with smaller ‘granules’ of functions, until the whole functioning system is delivered. This is the system capability of the whole enterprise capability and is the responsibility of solution architects to deliver. Again, if this is delivered through incremental delivery, TOGAF doesn’t really care. At the solution architecture level, as long as ‘symmetry of conformance’ principle are adhered to, the development teams can work in tandem. For software, this means systems on either side of an interface boundary must conform to the same interface specification at the same time and may necessitate a contract definition, which the QA roles on the team would ensure conformance to.

Each iteration of the ADM will deliver business value in a much shorter period than the whole business strategy. Additionally, the same benefits of pipelining tasks (like an assembly line) that agile developers see, can also be realised at the enterprise level by performing the ADM’s architecture phases in the same ‘assembly line’ way (Business, Data, Application and Technical Architectures – Phases B - D). Each finished iteration will deliver business value and do so in a much shorter time than trying to carry them out in sequence. The key is to partition the individual architectures to be independent of one another.

However, without taking the time to identify the segments, this won’t be possible. This is the same as discussions surrounding success criteria and requirements elicitation in agile processes. So again, it is not new.

4. Working software is the principal measure of progress

I personally have a problem with this definition anyway. ‘Working software’ as the agile developers of today know it, is necessary but not sufficient. If the definition of working software is to include ‘fit for purpose’ criteria, which TOGAF encourages, then this can encompass qualitative non-functional elements in the final delivered system. Additionally, a software component is not ‘Done’ until it has been fully tested, documented and delivered together with any necessary supporting artefacts and processes. All these together with similar criteria for the other architecture domains in the enterprise, deliver the whole capability. After all, what use is a piece of software that it is claimed ‘delivers value’ when nobody can use it because documents and training don’t exist?

Checking ‘fit for purpose’ criteria, together with architectural principles make up the bulk of the QA role in enterprise scale systems.

Often an architectural principle can be decomposed into a QA usable element. Consider the system principle:

Name:Logarithmic Performance Degradation

Statement:The system latency increases logarithmically when there is a linear growth of call centre users.

Rationale:The cost to handle linear increases in capacity is £1 million for 100 new call centre CPU licenses purchasable in blocks of 10. Logarithmic reductions in performance will allow a cumulative reduction of £100 per bulk license cost for each 10 new CPU licenses.

Impact: …

Here we can see that the bounds of ‘working software’ that is ‘fit for purpose’ includes a qualitative statement of the costs of performance in the principle. This may further be broken down into success criteria that after involvement from BA, Development and QA personnel, might derive the following non-functional requirement:

“Under heavy load, the system must maintain a throughput proportional to a function of O(n log₁₀ n).”

Now, I appreciate the vast majority of you won’t have come across a requirement worded like this. Indeed, it is entirely possible you consider agile to not worry about the performance of software until it becomes an issue (whether a company may lose millions in the process in either opportunity costs or real world losses seems immaterial to those who think like this, despite it delivering a ‘negative’ business value). But this gives architects the means by which to reduce the solution space by filtering out solutions which are inadequate and unsatisfactory. This in turn can give the technical staff the means to be able to deliver an environment to be able to test and stage pre-production and production deployments for each typical agile iteration that you will do. The idea of ‘fail fast and fail early’ which a lot of agile adopters have taken on-board applies equally well to the environments that the software is hosted in. The last thing you want is to have a piece of software which you have carefully sculpted using agile principles failing to deliver the performance gains in live or worse still, losing the company business value because considerations outside the agile domain have not been accounted for.

TOGAF includes the use of architectural principles in part to address non-functional concerns from the outset. Together with business architecture requirements, the statement of architecture work, the existing baseline architecture, specific industry trends, computing technology trends and the like, TOGAF moulds the enterprise architecture (including its technical capability and resources) to fit in with the vision of the very highest stakeholders on the company. When addressed from this perspective, the solution architects (whether of software or technical systems) deliver specific capability architectures for their architectural domain to the enterprise architects, who are often the principle customers of the solution architects. In turn, the solution architects then become the principle customers of the development teams along application horizontals whilst the business customers are along the verticals.

Unfortunately, the architects are normally vastly outnumbered by the development personnel, so can’t be in several agile teams at the same time. It may be that an organisation works to include junior solution architects in their staff-base, which would allow the segmentation of components of the overall architecture to spread across several development teams or may choose another way of segmenting that work, such as using development contracts with which the QAs would apply best practise principles such as ‘symmetry of conformance’.

5. Sustainable development, able to maintain a constant pace

TOGAF maintains the status quo and doesn’t explicitly address tempo in the standard. Again, it is more interested in the delivery of capabilities to the business and it is the programme management office’s responsibility to schedule the delivery of the enterprise architecture by priority of the requests for architecture work. You will note though, that this is akin to the prioritising of a product backlog by a stakeholder.

6. Close, daily co-operation between business people and developers

Remember, in TOGAF everything and everyone is a capability at a high level. Looking down from a strategic architecture, through segmented architecture at the capabilities, we see that IT is a capability, customer services is a capability, accounting is a capability etc. so everyone is a capability within the business and as a result, can be a business customer to any agile development team. This specifically means that developers and solution architects can be customers of development teams. It also allows the dissemination of the bigger picture as part of facilitating knowledge transfer from team to team via the architectural members participating in it.

An example of enterprise capabilities

The difficulty is that architects are not all available to be involved in direct code writing for the whole team all the time. As a result, a clear statement of intent for the day or indeed sprint backlog review should be used to prioritise the work and should come about, in part by the use of stand-up meetings, which architects should always attend.

The role of governance in the team should be delegated to the QAs who should ensure all architectural principles are adhered to, the interface definitions are conformed to and that the required performance and other related principles are encompassed in the system.

For some organisations, it may be decided that a ‘scrum-of-scrums’ take place to allow the solution architects to ensure that the required criteria are being satisfied at a group level.

7. Face-to-face conversation is the best form of communication (co-location)

In a previous blog, I indicated why this principle is highly deficient when applied to anyone outside the immediate development team. Additionally, there is a strict reluctance from both development and architecture to include enterprise architects in the development process (quite correctly, as it is a level of detail they should be unconcerned with outside the segmentation of the capability and the governance of its production).

Once the interfaces, acceptance criteria, principles and constraints (of which requirements, functional or otherwise, are part) and documented and understood, the team can choose to work whatever way they wish. If there is no clarity in the understanding of what deliverables the team should produce, the exercise will result in wasted development effort or high risk deployments which may fail catastrophically to deliver the value of that capability.

Also, it is the QA and PM’s responsibility to make sure that all communication into and out of the team is strictly controlled and is in accordance with the TOGAF communication strategy documentation. Use of importance-interest stakeholder quadrants (e.g. Eisenhower matrices) and stakeholder analyses/management techniques is the domain of the enterprise and business architects. Once agreed, it is not the responsibility of the development team to deviate from that strategy.

In larger organisations, developers are not the best members of the team to communicate information to non- technical audiences who are unconcerned with the detail of the realisation exercise, especially if they are unaware or unconcerned with the bigger picture (which is an area a lot of practical agile developers normally shun). As a result, they may make development decision on the fly or make the client aware of system level decisions which are incongruent with the enterprise strategy at a higher level, meaning the customer will go away thinking one thing, when the EA has stated something altogether different. These ‘double binds’ lead to confusion, which potentially can feed back into the team when the ultimate stakeholder makes what they think is a ‘corrective’ change to requirements, which may encompass the work of other teams which fall outside the remit of the source developers.

This results in a violation of architectural principles or governance procedures, duplicated and incompatible development deliverables (as the symmetry of conformance principles may be unable to be applied to both sides of interface contracts), overlaps in responsibility which fall outside what is known about in development capability (at the enterprise architecture level) and many other problems.

A slight modification can see RACI matrices help delineate responsibilities for capability deliverable by team. Once delineated, it should be strictly enforced and governance procedures/logs used to determine the decisions that were made and why they were made.

8. Projects are built around motivated individuals, who should be trusted

A project may contain sub-projects. Each sub-project is in itself a project to deliver a capability (aka a piece of functionality) in accordance with success criteria which includes all constraining factors. Once inside the bounds of a project, the motivated individuals are free and trusted to develop whichever way they see fit. TOGAF is unconcerned with how the deliverable is created. All it is concerned with is that the deliverable is ‘fit for purpose’ (Agile’s statement on ‘working software as the final arbiter’) and it does this by using governance processes to test the software deliverable as well as the development capability as a whole.

9. Continuous attention to technical excellence and good design, and… 10. Simplicity

In truth, this is at a level which is way above the understanding and capability of almost all agile developers on the market. If they understood it, they would usually be architects. The concern with very low level detail has blinded the majority of developers into believing that an understanding of a good design pattern is enough, without the need to understand the trade-offs that each has which make it particularly suited to one situation and not another.

Also, what low level development teams don’t have sight of, are points of optimisation in an overall design to allow and facilitate reuse of code or indeed components across teams. This is where solution architects can bridge the gap in understanding, by having a higher level view which allows them to see the deliverables as they are deployed and can govern the reuse of these software assets, which become the solution building blocks that get added to the enterprise repository as part of the solutions continuum. Indeed, it is very very easy for duplication of work across teams to occur, where this step is missed. After all, if one development team is trusting another development team to deliver against a ‘pull’ signal from the first, whilst the first is delivering their own related functionality and is being trusted to deliver this by the client or worse the second team, there is nothing to stop a duplication of work which is one of the cardinal sins of lean practises as it is a direct form of waste.

The enterprise architects don’t have responsibilities under principle 9, aside from producing the architectural capabilities that must be delivered to allow an enterprise engagement. Their view of simplicity (principle 10) is that of combinatorial optimisation, which reduces combinatorial complexity, which in turn reduces the amount of effort it would take for each set of iterations from involved development teams to deliver software, as well as the greater prize of the amount of effort required for a business to change direction.

Consider the two scenarios below.

A development team produces a piece of software (say, a service) which implements a business process which in itself, consumes functionality in 4 other services dotted around the company. The 4 other services are all linked, bi-directionally, to one another, but not the first (that is, for the 4 internal services, 12 communication links, through 6 channels). There are 5 services here and supposing one of the 4 internal services changes its service contract. The development team responsible for this service contract will develop the new interface and publish it, which will break every other connection to a service (which is 3 others). The other 3 teams then have to change their consumption of that service contract and code accordingly. The amount of effort involved is large. This has already broken the symmetry of conformance governance principle.

An enterprise architect will look at the capabilities involved, decide if they are necessary in any transition or target architecture and take steps to change this. This may include using SOA to deliver an ESB architecture pattern, with a data architecture which is independent of vendor, but adequately addresses the specific enterprise requirements (aka the Canonical Data Model or CDM for short) which is the movement from a foundation data architecture, including a common systems architecture, through industry architecture and finally organisation architecture (working from most general to most specific along the enterprise continuum). This can then be gradually rolled out to specific data and/or integration solution architects to realise that vision.

The solution architects may choose an ESB platform, create the contracts for a normaliser for each service to be attached to the ESB, deliver the CDM and also specify the normalising transformation between the specific messaging format of the service to be integrated and the CDM. The development teams all deliver their own normalising transformations to their systems and attach them to the ESB. This means there is only a requirement to support the 4 links on to the ESB not 6, making 8 bidirectional links not 12!

However, the real saving comes in the form of the non-functional requirements associated with adaptability. Should the scenario arise as above, where a contract changes, the development team involved need only change the transformation associated with the normaliser for that one service, massively reducing the workload on every other development team, which will be as good as zero. This is one single team, who have to deliver one single conformant interface (the tests for which will have been created already around the conformance and governance process for the CDM) and this reduces the overall development effort for the scenario above from 3 channels to 1, saving the equivalent to 2 whole teams’ development times over the first method.

Additionally, should the system have 20 internal services on it, each dependent on every other, the number of communication links could reach 380, whilst the number of communication to and from the ESB is only 40, massively reducing the workload.

The trade-off for the developers is the introduction of the normalization process. This is NOT waste! The ‘investment’, which is guaranteed to save money going forward, thereby delivering direct business value, including a monetary increase in profits, needs to be completed for the bigger picture to realise the benefit. To not provide this investment, and govern it, is akin to not writing unit tests up front (as a unit test up-front is an investment that pays dividends later).

So be aware that good design includes architecture! Technical excellence includes architecture! Unfortunately, there are many more programmers in the world than there are architects. So views on agile without architecture (which is in complete contrast to these principles) seem to have taken a hold in the UK simply because of the numbers. Agile methods will not save you alone! Pay attention to the ‘internal stakeholders’ :-)

11. Self-organizing teams

TOGAF, as mentioned already, facilitates the identification of capabilities at different levels. This can include the functions of QA, Business Analysis, Development, Architecture, Test, Technical Infrastructure and the like within the IT capability. As long as the vision of the capability is understood at each stage from baseline to target, then the capability to deliver story level functionality can be made up of self-organising teams within the IT capability architecture. Indeed, capability assessments are often the first elements to be decided during the ADM iterations, initially as part of the preliminary phase and refined in each pass of the architecture vision. It answers questions such as ‘Do we have the skills to deliver this capability and do these necessary skills exist in the required numbers to facilitate each role required by the capability?’

When forming teams to carry out particular pieces of IT development work, managers should be aware of the overall strategy of why they are doing what they are doing. Once they have a full understanding of what is required from them, they can hand it over to the team, who will be aware of how long tasks actually take to perform.

Whichever methodology school you come from, you know you have ask people who will actually carry out the work how much effort is required. Whether that translates to a time based system, story point or the like, it is those people who are responsible for the delivery that need to make the specific, concrete decisions. The boundary of ‘management’, just like all other agile project teams, is the prioritised product backlog driven by a product owner. The product owner is a role and not a job description and can be anyone who is pulling tasks from the team. This can and should include solution architects, if architecture members are not part of the team itself and/or enterprise architects, who will pull deliverables from the agile teams and deliver those solution building blocks into the architecture repository.

12. Regular adaptation to changing circumstances

With self-organising teams, this is a reality at the development level. However, what about the enterprise?

Agility is a big buzzword at the moment and in a lot of cases, quite rightly too. However, an enterprise as an entity has to be just as agile in the face of changing market dynamics, competitive practises, regulatory landscape, financial market position, technology innovation and other influential factors.

At the developer level, self-organising teams (principle 11) facilitate this by allowing a development team to change tack in mid-flight (on the product/sprint backlog). Whatever direction you pull it in, the team’s ‘organism’ will find a new equilibrium position.

An enterprise is often just like any other organism. In order to survive, it needs to adapt to its environment. Scaling up a development team (where each role can be mapped to a capability) then it should be possible to scale up such practises to the entire enterprise, which should adapt to its enterprise landscape. This is the Holy Grail of Agile-EA and indeed there has been some work done by the people following in the footsteps of Ken Schwaber to map Scrum methods up to the enterprise level.

As part of the architecture vision phase of each iteration, TOGAF specifies that an assessment of the enterprise, in its context, is carried out. What is in the landscape? Where are we now? Where do we want to be? Additionally, TOGAF’s ADM can basically be redefined as…

…a series of iterations each of which delivers business objectives through the incremental delivery of enterprise capability.

Let’s look at what agile software methods often suggest is the process of developing software. Agile development is

…a series of iterations each of which delivers business value through the incremental delivery of software.

I have highlighted the different word just for comparison, but they are ultimately, the same. Business value, to most enterprises, is not the software, it is what capability the software and the associated business function can allow the enterprise to have.

In short, just like any other system that can be developed, all the benefits that befit agile development can also be delivered at the enterprise level and just like at the software development level, it has a preference for tools and techniques which facilitate that context specific agile adaptation.

The Final Words…

What I would hope is that the fight between the resistive members of both sides of the TOGAF and Agile divide will come to realise that they have far more in common than they would readily admit. Agile proponents have to remember that it was not so long ago that they were the ones fighting to change the minds of the ‘anti-agile’ community, so they are sitting on the other side of the fence and some are not liking it.

Also, Enterprise Architects have to be aware that the tasks which involve communication management have to be carried out. This includes the mapping of the language of enterprise architecture to the language of IT solutions (or any other architecture domain), so reference library tasks must be completed and made public to facilitate that.

Both sides have to know their stakeholders and how to communicate with them. EAs and BAs should be doing this as part of their stakeholder management processes, but they must remember that solution architects, devs and QAs are stakeholders too and have to use language which facilitates communication with these stakeholder groups too. An example that has been put forward by an agile coach in my current client’s site may be the use of CRC cards to help developers understand high level services and their communication.

I would hope to run through an example of how TOGAF derived architectures can be developed using agile principles in the not too distant future. But that is obviously time dependent.

[UPDATE: 01/04/2012]
I just watched a presentation from Dennis Stevens, an enterprise Agile coach on how to scale agile thinking to the enterprise. It is a very good presentation and I agree almost entirely with his professional viewpoint when scaling agile to the enterprise level.

He doesn't explicitly mention TOGAF at all in the presentation, but it is clear to those who have been through the TOGAF certification process that he is talking about concepts which are also covered in the framework.

You can find the presentation on the InfoQ website at:

www.infoq.com/presentations/Scaling-Agile-to-the-Enterprise