Saturday 25 July 2015

FAIL is not a dirty word!

Short one this one. I'm spending a bit of time on an OSS project, so don't have the time to go into this in more detail.

TL;DR; Failing tests identify where your system is incomplete or inconsistent

I was at the North West Tester Gathering a couple of weeks ago. The theme of the night was Failure. There were a diverse variety of speakers, some from the BBC, Sage and one from SkyBet, Leigh Rathbone (@villabone) presented a talk entitled "FAIL is not a dirty word". It reminded me of yet another blog post I've been meaning to write for a few months on it, so I figured I'd get this down on a screen somewhere before I forgot or my time got chewed up yet again.

The thing with failure is it is a central part of doing anything in uncertain environments. Whatever the environment, whether it is Marketing, Lean-Startup, TDD'ing software or anything else with a high degree of uncertainty or variance, it is important to fail for many reasons.

Failure in TDD

"Write a failing test" - This is one of the most crucial mantras that is often espoused by us in the lean/agile world. This actually deals with a number of different problems all at once.

Aside from the orthodox answers, it also addresses two fundamental concepts to all systems thinking. If you're into theoretical computer science or mathematical logic (predicate logic or propositional calculus), these two concepts will be very familiar. I'll introduce the concepts first then name the theorem for those not familiar with them.

  • Developers who start to code a new story, start with a test. A failing test shows the boundary of the system relative to it's context. When you modify or expand the code to make the test pass, then you have made the software more complete. If the test correctly codifies the story and it happens to pass, then the system and your knowledge was more complete than you thought it was. Our knowledge is now more complete, which the code and tests happen to also represent ("code communicates intent" - @datoon83).
  • Bug tests - Those issues resulting in live (or UAT if your team works like that) show you that the software, and our associated knowledge, which satisfied the acceptance tests, isn't consistent. You write a test which exposes the bug, then you fix the system to satisfy the test. Thereby making your system more consistent.

Those familiar with these two ideas will immediately notice Godel's [1st] incompleteness theorem, which for us IT software/systems folk basically translates to:

"A system cannot be both complete and consistent at exactly the same time"

So, we have a choice. We can try to code the world, which would make it complete, putting in all the possible use cases that anyone would ever want (and perhaps many they won't) never delivering anything and we'd lose the consistency of the system anyway. Alternatively, we can constrain ourselves and accept a level of incompleteness and go for consistency (low bug count). Software development/engineering naturally lends itself to the latter. This is natural, since the system is complete to the stories that are done, not in the backlog or in-progress. Lean-Startup also introduces the concept of an MVP (Minimum Viable Product) with the aim of solidifying that MVP over time.


Fixing a bug by starting with a red test which surfaces the bug, identifies where your software is inconsistent.

Starting a new scenario with a red test, helps you identify the bounds of your system, gain more knowledge about what it should and can do, and naturally makes you extend that system, increasing the sphere of completeness.

It is important to recognise the contribution failure makes to software development. I am often frustrated when I look at code which hasn't been developed that way. It often has far too much coverage in one area, not enough coverage in others and I see the odd Assert.True() thrown in. Crucially, you can go on proving something is true the same way forever.

Code can also 'suffer' from confirmation bias as much as we can as humans. After all, the code is a manifestation of our knowledge of the domain. If we don't have that failing test, that appreciation that we have stretched the code, the system and ourselves past our limit of knowledge, we don't have that ability to fill in any gaps in that context.

So I'd certainly go further than Leigh on this one. Not only is failure not a dirty word, it's absolutely and unequivocally MANDATORY!