In my trawls across the web, I come across a lot of OSS which always has the promise to deliver lots, but certain things let it down really really badly. This time round, it is the turn of some of the Javascript testing frameworks.
Unfortunately, there are far far too many and they are split across TDD and BDD and a large proportion of them have appalling documentation that is either inaccurate, non-existent or incomplete. So the provisions are awful, no two ways about it.
...Be warned, I may rant a lot in this post.
Javascript unit-testing?
Yes, Javascript testing. Why not? You unit test everything else (or should be) and even if you test the output generated in the HTML in some form of test, you don't automatically test the Javascript and it certainly won't be at the unit level if you're testing it through Selenium. This leaves both a massive hole in your test coverage and leaves the testing of Javascript far too late into the development cycle. So take note QAs, does your team cover their Javascript code?
OK, So What Do We Do?
The lesson I learned the hard way is it depends what you want to do with it and how you run it. If you want the tests to form part of a CI testing process, which integrates with Jenkins, TFS or
"Insert your favourite task runner here" then you will likely end up working with
mocha, which is a NodeJS application. I am going to be covering this here. If you want to build it into a web-page, to tack on to somehting like Fitnesse (remember that?) then you'll need something else, as whilst mocha contains that ability, it is rubbish out of the box, despite protests. Also, despite the tagline of "simple, flexible, fun" it is anything but if you are not experienced in Node, have an understanding of RequireJS and JavaScript AMDs and are used to a packages solution, because Mocha is anything but that. Read:
- Simple - Can't be bothered writing decent documentation, you work it out
- Flexible - Needs to be amalgamated with other tools just to do the simplest of things
- Fun - ...If you like hunting for holy-grails
The docs is the bit that gets my goat a bit. With my architecture hat on and a lean one at that, I have to be very aware how fast I can scale teams up or down. The requirement to have new team members get productive quickly is a massive factor as this can make a huge mess of the velocity/throughput of a team if they have to go digging in a framework or worse, pull time from others on the team who working at close to capacity.
On top of that, you have to be aware of how the addition of frameworks affects the CI build process. Adding tech not already in the stack has to be justifiable via a kind of 'cost-benefit'. For example, adding a DI/IoC container is very useful for the development of modular, decoupled software, but the cost or trade-off is it couples your application to the framework and hence your ALM/CI/CD process has to account for it (even if it is only one line) and account for upgrades/modifications to it.
So all-in-all, the introduction of a new framework into a system is not something that I take lightly unless there is an obvious benefit and it significantly outweighs the cost of its introduction. For example, if a dev decides that they wanted to put a DI framework into a project which didn't need the more advanced features of it yet (such as profiles, configurations or something else), I tell them to shove it, since this simple dev level introduction can cause headaches for DevOps and tech services guys, since they may have to open connections in restricted environments, maintain a package environment of their own, of something else, just to save the one dev from writing what can be tantamount to a key-value dictionary. Plus, it may delay the project because of it by introducing a blocker into the flow if the environment takes time to process change requests for example.
In short, mochas distinct lack of coherent documentation (note, there are docs, but none of it flows well) means that it would not be something I would normally introduce, since it would hit flow really really badly and require new team developers to spend an age getting to grips with. However, given it's free and downloadable and it based on Node, I figured I'd give it a go.
What does JS Unit-testing on Mocha Look Like?
Woah horsey! First you have to install node, then install it, then set it up with an assertion library and then write the test, as the assertion library will dictate somewhat the structure of your tests and of course, if you pick a BDD like language, that will look different again.
A Shot of mocha
Mocha is based on NodeJS. indeed, it is deployed as a Node Packaged Module (npm) which is the Node equivalent of nuGet. However, first you have it install NodeJS. You can get Node from
here.
Unlike mocha, Node's site is actually very good and the documentation is quite extensive and they make good use of examples.
Once you have installed Node, you can install mocha by opening a command prompt As Administrator and typing:
npm install -g mocha
This globally installs mocha so you can run it from anywhere in your directory tree. Hence once the installation competes, you can literally run mocha by typing
mocha from the command line (which doesn't have to be opened in Administrator mode).
 |
| first run of an empty mocha |
Nothing special. What this has done is run mocha on a directory, it has found no unit tests (as there is no directory with unit tests in it) and has existed with a green "0 passing" message. It is very important to note that this produces this message.
Do I need an assertion library... What is an assertion library?
An assertion library is a file containing assertion modules to perform TDD or BDD style test validation. For the .NET chaps it is the .NET library in Assert.AreEqual in MSTest and similarly for NUnit. However, if you've ever downloaded fluent assertions separately, this is also a form of assertion library and is what we ahve to do with mocha via the 'require' RequireJS function.
Mocha Directories
One thing that the documentation doesn't make it easy to find is that out of the box, mocha requires that a 'test\' directory exist relative to where you run it from. You can recurse into other directories from the command line by including
--recursive in the command-line switches.
So let's try to set up to run some unit tests. The first stage is to create the directory structure. I created the following:
c:\spikes\mocha\ <-- Main subdirectory holding the classes/functions under test.
c:\spikes\mocha\test <-- Which holds the unit-tests
To test this out, I intend to TDD a created a calculator class, which will just perform the four main arithmetic operations and
Creating Tests
Assertion library: assert
Within the test sub-directory, create a file called CalculatorTest.js and at the top of that file, put:
assert = require("assert");
Calculator = require("../Calculator.js").Calculator
The first line defines the requirement for assert, which is the vanilla assertion library available from node. The second line specifies a requirement for the calculator in the main subdirectory (i.e. the test directory parent) and after the dot (.) also specifies the actual class we want to use. For those with a .NET background, this is akin to aliasing in C# with:
using Calculator = Calculator.Calculator
When you run mocha, Node will follow the path to the assert library and download it if necessary. If it is not local it goes to the default npm location and aliases it to there. Calculator is then located in the subdirectory.
Run mocha and you will see your blank test file has produced the same 0 passing message as before. So be aware that mocha will run whatever it finds in the test directory, but also will return green tests if there are no tests. Not too problematic so far.
So let's build our first Calculator test. By default, using the
assert library with mocha means you are going to be using the
Jasmine style of assertion. For the calculator, this basically takes the form of:
describe("A calculator", function () {
describe("adding 3 and 4 together", function () {
it("should return 7", function () {
var result = new Calculator().AddNumbers(3, 4);
assert.equal(7, result);
});
});
});
In general, you will put a describe option to describe the test harness and methods/scenarios, which is just a string that appears in the summary view, so can be anything you like. I prefer the usual BDD 'given... when... then...' syntax, but when reading the test in code, it was surprisingly strange to split it across this syntax so that the description on screen and the partial-fluent-like test read the same.
Additionally, the mocha documentation seemed to confuse me a little by prefixing hashes to method names, which initially made me think there was some form of JQuery, but then I came to my senses and figured I'd test it with just a string and a failing test and it worked fine.
The describe syntax just groups the tests into convenient units. In the above, I am effectively blocking up the test into a class level calculator tests, with a method test for adding 3 and 4. You will note that the supplied anonymous function is a callback, which then runs the expectation. Indeed, you can have many descriptions and many 'it' statements within them.
 |
| without a calculator class defined, you get this |
So running this test, without a calculator class shows a thrown ReferenceError, with calculator not defined. Fine, that's OK, we can learn from that. Despite the dynamic nature of JavaScript, it conceptually makes sense anyway, as you'd expect the C# compiler to throw an error if there was no such class defined.
So defining a Calculator class in a file in the main subdirectory, which we can do using our usual standard OO JavaScript knowledge, we get:
function Calculator() {
}
Calculator.prototype.AddNumbers = function (p1, p2) {
return 0;
}
This basically creates our class, with a public function AddNumbers, defined to return a zero instead of the sum of two numbers, as failing test. Remember, Red-Green-Refactor.
Our directory structure should now look something like:
c:\spikes\mocha\Calculator.js
c:\spikes\mocha\test\CalculatorTest.js
Cool. So running mocha shows us:
 |
| mocha run with calculator (without module defined) |
At this point I was totally scratching my head. Under normal circumstances, this should have worked without issue. What else did I need to do?
After a bit of trawling, I managed to figure that under the RequireJS usage, which I highlighted previously is used in Node, I needed to export the module. That initially made me a little uncomfortable, but actually, given the AMD nature of the code, and thinking about how you'd want to use this, I figured that I wouldn't really have much of an issue with the idea of exporting a module, since this is a better loading system than you'd otherwise have (for example, loading on the command line or worse, static loading in a web page).
So I added the following line to the bottom of the Calculator.js file:
module.exports.Calculator = Calculator;
Run mocha again and you get the first legitimate failure.
 |
| Woohoo! First test failure!! :) |
The error message is somewhat rubbish, so I figured I'd change it in the CalculatorTest.js file to include a message at the end of the the assertion:
describe("A calculator", function () {
describe("adding 3 and 4 together", function () {
it("should return 7", function () {
var result = new Calculator().AddNumbers(3, 4);
assert.equal(7, result, "But the number " + result + " was returned instead");
});
});
Which then output the following message (highlighted region shows the new message):
 |
| Meaningful error message |
So let's make the test pass. Change the AddNumbers function in Calculator.js to return the sum of both parameters and run the test. You should get:
 |
| First green test pass :) |
Note the number of full stops output. This shows the number of tests run and of course, the green text shows the number of tests passed.
The same process is used for the rest of the TDD. You can find references to the assertion options for your chosen library at their website.. but it might be rubbish so be warned.
Fast forwarding through those to an exception scenario. The divide by zero error. I have added this at the bottom of the CalculatorTest.js file:
//-- CalculatorTest.js
assert = require("assert");
Calculator = require("../Calculator.js").Calculator
describe("A calculator", function () {
describe("adding 3 and 4 together", function () {
it("should return 7", function () {
var result = new Calculator().AddNumbers(3, 4);
assert.equal(7, result, "But the number " + result + " was returned instead");
});
});
describe("aubtracting 3 from 5", function () {
it("should return 2", function () {
var result = new Calculator().SubtractNumbers(5, 3);
assert.equal(2, result, "But the number " + result + " was returned instead");
});
});
describe("multiplying 7 and 8", function () {
it("should return 56", function () {
var result = new Calculator().MultiplyNumbers(7, 8);
assert.equal(56, result, "But the number " + result + " was returned instead");
});
});
describe("dividing 8 by 4", function () {
it("should return 2", function () {
var result = new Calculator().DivideNumbers(8, 4);
assert.equal(2, result, "But the number " + result + " was returned instead");
});
});
// Catching exceptions
describe("dividing 5 by 0", function () {
it("should throw an exception", function () {
// The throw in the next line is caught by the assert and hence registers as a pass.
assert.throws(function(){ new Calculator().DivideNumbers(5, 0) }, Error, "This did not throw the expected error");
});
});
});
Javascript throws exceptions just like every other language. The selection of assertion library will determine what syntax is used to capture those exceptions. In the case of assert, it is simply uses assert's "throws" method, which basically runs the supplied function inside a try...catch. However, note one very important feature, which left me scratching my head for 5 minutes until this StackOverflow post:
http://stackoverflow.com/questions/6645559/is-nodes-assert-throws-completely-broken
It doesn't take a parameter for the FUT. So you have to wrap it in a function to test it.
Within the 'A calculator' description, I deliberately throw an exception inside DivideNumbers() when the second parameter is zero.
function Calculator() {
}
Calculator.prototype.AddNumbers = function (p1, p2) {
return p1 + p2;
}
Calculator.prototype.SubtractNumbers = function (p1, p2) {
return p1 - p2;
}
Calculator.prototype.MultiplyNumbers = function (p1, p2) {
return p1 * p2;
};
Calculator.prototype.DivideNumbers = function (p1, p2) {
if (p2 === 0)
throw "Attempt to divide by zero!";
return p1 / p2;
};
module.exports.Calculator = Calculator;
Additionally, when I looked at the above code and back at Jasmine, there is the possibility of using multiple 'it' statements, which makes sense when describing muliple tests (such as positive and negative tests). The DivideNumbers() method call is ideal for this as it has a number of scenarios which you need to test. So I refactored the test code into:
assert = require("assert");
Calculator = require("../Calculator.js").Calculator
describe("A calculator", function () {
describe("adding 3 and 4 together", function () {
it("should return 7", function () {
var result = new Calculator().AddNumbers(3, 4);
assert.equal(7, result, "But the number " + result + " was returned instead");
});
});
describe("aubtracting 3 from 5", function () {
it("should return 2", function () {
var result = new Calculator().SubtractNumbers(5, 3);
assert.equal(2, result, "But the number " + result + " was returned instead");
});
});
describe("multiplying 7 and 8", function () {
it("should return 56", function () {
var result = new Calculator().MultiplyNumbers(7, 8);
assert.equal(56, result, "But the number " + result + " was returned instead");
});
});
describe("dividing", function () {
it("8 by 4 should return 2", function () {
var result = new Calculator().DivideNumbers(8, 4);
assert.equal(2, result, "But the number " + result + " was returned instead");
});
it("7 by 2 should return 3.5", function () {
var result = new Calculator().DivideNumbers(7, 2);
assert.equal(3.5, result, "But the number " + result + " was returned instead");
});
it("5 by 0 should throw an exception", function () {
assert.throws(function () { new Calculator().DivideNumbers(5, 0); }, Error, "This did not throw the expected error");
});
});
});
...plus added a real number test as well, testing the tests against the unmodified code all the time (remember, when everything is green and you refactor your tests only, you are effectively using your code as the mould your tests fit into. If you have to, red out the tests one at a time by changing only the test code, refactor the test code and make it green again by fixing the test code ONLY in an exploratory testing kind of way). All six tests now pass:
 |
| 6 passing, refactored tests |
Conclusion
JavaScript unit testing has the potential to be great and it is much needed. However, the state of the documentation varies greatly and you really need to pick the one where you can get up to speed quickly. In this case, I had to bear in mind that this is a combination of assert, require (and AMDs), mocha, and node.
This needs further coverage, so I will look at the combination of mocha with expectJS, chai (in TDD and BDD mode - note the documentation looks nicer for chai, but it still pretty thin on the ground when it comes to examples). However, just to get this far was a day's work. This is a substantial learning curve which means that turning around new staff, however smart they are, will take longer than necessary.
The main thing for me from this experience, is that if the frameworks the OSS community wish to supply to the commercial world have poor documentation or learning resources, the introduction of such frameworks into agile teams will greatly slow down the team. That means teams can't scale upwards as fast as they need to, to keep up the velocity/throughput, especially if there is no slack in the development process. The lack of slack means there is no time to learn and that increases risks around adoption of new frameworks and new team members. This applies across developers, analysts, QAs, DevOps and tech services. The introduction of frameworks is something that needs thinking about as a fairly typical trade-off analysis.
