This is part 2 of the JavaScript unit test framework series. If you haven't read the first part, here it is: JavaScript unit test framework, part 1 of 4

While adding new features to a project, you're supposed to add only one or a few unit tests at a time, see them fail, and then implement as little code as possible to make it/them pass. The current version of the unit test framework only shows the number of passing/failing tests, so you cannot be sure about which test is actually failing – especially after making some code changes. Also, there is no way of seeing the actual error message thrown when running a failing unit test. I really need to improve on this, so I introduce the failures property, which will contain names and error messages of all failing tests.

There is also the case of testing asynchronous code, such as timers, event handlers, AJAX requests, CSS transitions and so on. The way the run method is currently implemented, there is no way of properly waiting for it to complete if some unit test contains event handlers or timers.

New requirements

• The engine should keep track of failing tests and their error messages
  • after eng.run() is done, the eng.failures should contain a list of objects containing the name and error message of each failing test
• Each test function should be called with a ut.TestContext object as its first argument
  • testContext.actAndWait(timeout, actFunc) should run the actFunc function, and wait timeout milliseconds before continuing
  • if the actFunc function crashes, the test is marked as a failed test
  • the actFunc function should receive a ut.TestContext object as its first argument
  • calling testContext.actDone() from within an asynchronous test function stops the waiting immediately
  • the testContext.actAndWait method should return the test context object itself, for call chaining purposes
  • calling testContext.thenAssert(assertFunc) should make the assertFunc function get called after the actFunc function is either timed out, or marked as done using the actDone() method
  • the assertFunc function should receive a ut.TestContext object as its first argument
• The framework should provide a function hook to be called after all unit tests have been run
  • the run() method of ut.Engine should return the engine itself
  • the then(func) method should register a function to be run after all unit tests are done running, passing the engine as the first parameter to the func function
  • If none of the registered unit tests contain any asynchronous code, calling run() should run all tests before returning, and the caller of run() shouldn't need to use the then() method

Fifth requirement

// Tests

engine.add("The engine should keep track of which tests fail/succeed",
    function() {
        var eng = new ut.Engine();
        var failFunc = function() {
            throw "I did crash!";
        };
        eng.add("Failing", failFunc);

        // Act
        eng.run();

        // Assert
        if (eng.failures.length !== 1) {
            throw "failures.length should be 1, but is " + eng.failures.length;
        }
        if (eng.failures[0].name != "Failing") {
            throw "failures[0].name should be 'Failing', but is " + eng.failures[0].name;
        }
        if (eng.failures[0].message != "I did crash!") {
            throw "failures[0].message should be 'I did crash!', but is " + eng.failures[0].message;
        }
    });

The implementation is pretty simple, and after this refactoring, I can use the failures property to list failing tests and their error messages.

// Implementation

run : function() {
    this.failures = [];
    for (var name in this.tests) {
        var testfunc = this.tests[name];
        try {
            testfunc.call();
            ++this.successCount;
        } catch(e) {
            this.failures.push({
                name : name,
                message : e.toString()
            });
            ++this.failureCount;
        }
    }
}

// Test run

engine.run();

console.log(engine.failureCount + " failures and " +
            engine.successCount + " successes, out of " +
            engine.testCount + " tests");

for (var i = 0; i < engine.failures.length; ++i) {
    var fail = engine.failures[i];
    console.log(fail.name + ": " + fail.message);
}

Sixth and seventh requirement

Requirements for an API or a framework are often expressed better in code. This is how I want to be able to use the features described in requirements 6/7:

// Requirement code vision

var eng = new ut.Engine();

eng.add("Asynch test",
    function(testContext) {
        // Arrange
        var someObj = new SomeClass();
        var result = null;

        // Act
        testContext.actAndWait(1000, function(tc) {
            someObj.someAsyncMethod({
                success : function(r) {
                    result = r;
                    tc.actDone();
                }
            });
        }). // < -- notice the dot here... call chaining!

        // Assert
        thenAssert(function(tc) {
            if (result == null) {
                throw "Timeout!";
            }
            if (result.foo !== "bar") {
                throw "Expected 'bar', but found " + result.foo;
            }
        });
    });

eng.run().then(function(engine) {
    // Display test results
});

Requirements 6/7 – first batch

The first batch of eight unit tests addresses every piece of the sixth requirement except for the actDone() function. `

// Tests

engine.add("Test functions should be called with a ut.TestContext as its first argument",
    function() {
        // Arrange
        var innerTc;
        var eng = new ut.Engine();
        eng.add("set inner test context", function(tc) {
            innerTc = tc;
        });

        // Act
        eng.run();

        // Assert
        if (!(innerTc instanceof ut.TestContext)) {
            throw "innerTc is not a ut.TestContext object";
        }
    });

engine.add("testContext.actAndWait should return the testContext itself",
    function() {
        // Arrange
        var innerTc;
        var returnedTc;
        var eng = new ut.Engine();
        eng.add("set inner test context", function(tc) {
            innerTc = tc;
            returnedTc = tc.actAndWait(1, function() {});
        });

        // Act
        eng.run();

        // Assert
        if (innerTc !== returnedTc) {
            throw "actAndWait did not return the testContext itself";
        }
    });

engine.add("actAndWait(timeout, actFunc) should run the actFunc, and wait (at least) timeout milliseconds",
    function(testContext) {
        // Arrange
        var timeout = 100;
        var calledAt = 0, endedAt = 0;
        var eng = new ut.Engine();
        var actFunc = function() { calledAt = new Date().getTime(); }
        var testFunc = function(tc) { tc.actAndWait(timeout, actFunc); };
        eng.add("actAndWait should wait correct amount of ms", testFunc);

        // Act
        testContext.actAndWait(timeout + 100, function() {
            eng.run().then(function() { endedAt = new Date().getTime(); });
        }).

        // Assert
        thenAssert(function() {
            if (calledAt == 0) {
                throw "Did not call the actFunc function";
            }
            if (endedAt == 0) {
                throw "Did not finish running the tests properly";
            }
            // Minor timing issue: one or two milliseconds off is not a big deal
            if (endedAt < calledAt + timeout) {
                throw "Did not wait enough ms (waited " + (endedAt - calledAt) + " ms";
            }
        });
    });

engine.add("thenAssert(func) should called the assert function after (at least) the registered number of milliseconds",
    function(testContext) {
        // Arrange
        var timeout = 100;
        var calledAt = 0, assertedAt = 0;
        var eng = new ut.Engine();
        var actFunc = function() { calledAt = new Date().getTime(); };
        var assertFunc = function() { assertedAt = new Date().getTime(); };
        var testFunc = function(tc) {
            tc.actAndWait(timeout, actFunc).thenAssert(assertFunc);
        }
        eng.add("thenAssert should wait correct amount of ms", testFunc);

        // Act
        testContext.actAndWait(timeout + 100, function() {
            eng.run();
        }).

        // Assert
        thenAssert(function() {
            if (calledAt == 0) {
                throw "Did not call the actFunc function";
            }
            if (assertedAt == 0) {
                throw "Did not call the assertFunc function";
            }
            if (assertedAt < calledAt + timeout) {
                throw "Did not wait enough ms (waited " + (assertedAt - calledAt) + " ms";
            }
        });
    });

engine.add("if the actFunc for actAndWait crashes, the test should be failed",
    function(testContext) {
        // Arrange
        var eng = new ut.Engine();
        eng.add("This should crash", function(tc) {
            tc.actAndWait(100, function() { throw "Crashing!"; });
        });

        // Run
        testContext.actAndWait(200, function() {
            eng.run();
        }).

        // Assert
        thenAssert(function() {
            if (eng.failures.length !== 1) {
                throw "Did not register exactly one failure";
            }
        });
    });

engine.add("then(func) should run func immediately if there are no asynchronous unit tests",
    function() {
        // Arrange
        var thenCalled = false;
        var eng = new ut.Engine();
        eng.add("no-op test", function() { });

        // Run
        eng.run().then(function() { thenCalled = true; });

        // Assert
        if (!thenCalled) {
            throw "the thenFunc was not called";
        }
    });

engine.add("then(func) should NOT run func immediately if there are some asynchronous unit test",
    function() {
        // Arrange
        var thenCalled = false;
        var eng = new ut.Engine();
        eng.add("async test", function(tc) {
            tc.actAndWait(100, function() { });
        });

        // Run
        eng.run().then(function() { thenCalled = true; });

        // Assert
        if (thenCalled) {
            throw "the thenFunc was called, but shouldn't!";
        }
    });

engine.add("then(func) should run func after all asynchronous tests are done",
    function(testContext) {
        // Arrange
        var thenCalled = false;
        var eng = new ut.Engine();
        eng.add("async test", function(tc) {
            tc.actAndWait(100, function() { });
        });

        // Run
        testContext.actAndWait(200, function() {
            eng.run().then(function() { thenCalled = true; });
        }).

        // Assert
        thenAssert(function() {
            if (!thenCalled) {
                throw "the thenFunc wasn't called";
            }
        });
    });

// new way of printing successes/failures
engine.run().then(function() {
    console.log(engine.failureCount + " failures and " +
                engine.successCount + " successes, out of " +
                engine.testCount + " tests");

    for (var i = 0; i < engine.failures.length; ++i) {
        var fail = engine.failures[i];
        console.log(fail.name + ": " + fail.message);
    }
});

This takes a pretty big piece of refactoring. I'm essentially transforming a sequential traversal of the tests property into a "wait-and-continue" loop using window.setTimeout to save engine state, halt the unit test engine, let a test run its course, then continue with the assert function or the next test.

First, the new ut.TestContext class:

// Implementation – TestContext

var utTestContext = function(engine) {
    this.engine = engine;
};

utTestContext.prototype = {
    actAndWait : function(timeout, actFunc) {
        this.engine.actAndWaitFunc = actFunc;
        this.engine.actAndWaitContext = this;
        this.engine.actAndWaitTimeout = timeout;
        return this;
    },

    thenAssert : function(assertFunc) {
        this.engine.thenAssertFunc = assertFunc;
        this.engine.thenAssertContext = this;
    }
};

window.ut = {
    "Engine" : utEngine,
    "TestContext" : utTestContext
};

Then the new ut.Engine implementation:

// Implementation – Engine

var utEngine = function() {
    this.tests = {};
    this.testCount =
        this.successCount =
        this.failureCount = 0;
};

// private function, not exposed
var runOneTestOrAssertFunc = function(engine, func, context) {
    try {
        func.call(null, context);
        if (!engine.actAndWaitFunc) {
            ++engine.successCount;
        }
    } catch(e) {
        engine.failures.push({
            name : engine.currentTestName,
            message : e.toString()
        });
        ++engine.failureCount;
    }
};

utEngine.prototype = {
    add : function(name, testfunc) {
        this.tests[name] = testfunc;
        ++this.testCount;
    },

    run : function() {
        if (this.initialized !== true) {
            this.initialized = true;

            this.failures = [];

            this.testNameIndex = 0;
            this.testNames = [];
            for (var name in this.tests) this.testNames.push(name);
        }

        this.running = true;

        if (this.actAndWaitFunc) {
            runOneTestOrAssertFunc(this, this.actAndWaitFunc, this.actAndWaitContext);

            delete this.actAndWaitFunc;
            var self = this;

            // pause the engine for a number of milliseconds
            this.actAndWaitTimeoutId = window.setTimeout(function() {
                self.run();
            }, this.actAndWaitTimeout);

            return this;
        }

        if (this.thenAssertFunc) {
            runOneTestOrAssertFunc(this, this.thenAssertFunc, this.thenAssertContext);

            delete this.thenAssertFunc;
            delete this.thenAssertContext;
        }

        while (this.testNameIndex < this.testNames.length) {
            var name = this.testNames[this.testNameIndex++];
            var testFunc = this.tests[name];
            var context = new ut.TestContext(this);
            this.currentTestName = name;

            runOneTestOrAssertFunc(this, testFunc, context);

            if (this.actAndWaitFunc) {
                var self = this;
                window.setTimeout(function() {
                    self.run();
                }, 0);
                return this;
            }
        }

        this.running = false;

        if (this.thenFunc) {
            this.thenFunc.call(null, this);
        }

        return this;
    },

    then : function(thenFunc) {
        if (this.running) {
            this.thenFunc = thenFunc;
        } else {
            thenFunc.call(null, this);
        }
    }
};

The unit test framework is starting to be really useful now, but is still only just over a hundred lines of production code, and about 350 lines of test code.

Requirements 6/7 – second batch

If you have lots of asynchronous unit tests where you need a large timeout value, but the tests still might finish quickly, it doesn't really feel good to always wait for the maximum expected timeout before moving on to the next test. You should be able to move on instantly if a test finishes early. That's why I add the actDone() method to tell the engine to move on to assertion and/or the next test instantly.

// Tests

engine.add("actDone() should move immediately to the thenAssert assert function",
    function(testContext) {
        // Arrange
        var calledAt = 0, assertAt = 0;
        var eng = new ut.Engine();
        eng.add("10ms func with 10000ms timeout", function(tc) {
            tc.actAndWait(10000, function() {
                calledAt = new Date().getTime();
                window.setTimeout(function() {
                    tc.actDone();
                }, 10);
            }).thenAssert(function() {
                assertAt = new Date().getTime();
            });
        });

        // Act
        testContext.actAndWait(500, function() {
           eng.run();
        }).

        // Assert
        thenAssert(function() {
            if (assertAt === 0) {
                throw "Assert wasn't called!";
            }
            if (assertAt > (calledAt + 100)) {
                throw "Assert took way too long to get called!";
            }
        });
    });

The solution is to add this to the ut.TestContext prototype:

// Implementation

actDone : function() {
    var engine = this.engine;
    if (engine.actAndWaitTimeoutId) {
        window.clearTimeout(engine.actAndWaitTimeoutId);
        delete engine.actAndWaitTimeoutId;

        window.setTimeout(function() {
            engine.run();
        }, 0);
    }
}

There it is. Sixteen unit tests have rendered 137 lines of production code, which actually makes a pretty decent unit test framework. What is missing is a group of convenient helper functions and hooks for asserting and pretty output. If you want to use this in a automated testing environment it is already pretty much good to go. In the then() function, you could add a jQuery.ajax call to automatically post the test results to some server. Then just add a post-build script to your CI environment of choice to run your unit tests in a number of different browsers.

Next part will focus on assert helper functions and output hooks. Then I will look into mocking and some inversion of control magic.

The latest version of the code is always available in the GitHub repository.

Articles in this series:

• JavaScript unit test framework, part 1 of 4
• JavaScript unit test framework, part 2 of 4 (this part)
• JavaScript unit test framework, part 3 of 4
• JavaScript unit test framework, part 4 of 4