The Rontologist

Code coverage reports are neat. At a glimpse you can determine what pieces of code have been executed in a batch of unit tests, and get a good guesstimate on how well your unit testing efforts are going. Unfortunately there is a tendency to jump to the conclusion that covered code is tested code. If only it was that simple!

Assume we have a simple method:

function calculateTotal(amount, includeTaxes) {
    if (amount < 0) {
        raiseError();
    }
    var total = amount;

    if (includeTaxes) {
        total *= 1.05;
    }

    return total;
}

This can be tested fully with a few simple asserts

assertEquals(100, calculateTotal(100, false));
assertEquals(105, calculateTotal(100, true));
assertEquals(0, calculateTotal(0, false));
assertEquals(0, calculateTotal(0, true));
assertError(calculateTotal(-5. true));
assertError(calculateTotal(-5. false));

Running these tests will provide a coverage report that indicates we have 100% code coverage. That is good, since any fully tested function will have 100% code coverage, and our tool would be broken if it didn't say that. However, what happens if we need to add a small bit of functionality to this function? Maybe we want to include the option to include a default tip. Our function could be changed like this:

function calculateTotal(amount, includeTaxes, includeTip) {
    if (amount < 0) {
        raiseError();
    }
    var total = amount;

    if (includeTip) {
        total *= 1.15;
    }

    if (includeTaxes) {
        total *= 1.05;
    }

    return total;
}

Since we have added to the function we need to modify our tests or the compiler will ball at us. There also needs to be additional testing included to cover the new functionality. Ideally we would have done this TDD, but that's a topic for another day.

//existing tests modified for third param
assertEquals(100, calculateTotal(100, false, false));
assertEquals(105, calculateTotal(100, true, false));
assertEquals(0, calculateTotal(0, false, false));
assertEquals(0, calculateTotal(0, true, false));
assertError(calculateTotal(-5. true, false));
assertError(calculateTotal(-5. false, false));

//new test
assertError(115, calculateTotal(100, false, true));

If we run our tests again we will find that we have 100% code coverage again. It is tempting to assume that it is 100% tested, but jumping to the conclusion is wrong since we did not test how the function behaves if we are calculating both the tip and the taxes. Never assume that you have 100% test coverage if you have 100% code coverage.

Does this mean that the code coverage report is useless? I don't think so. While you cannot determine if a covered line is tested, you can determine that an uncovered line is not. This may give you a clue on the best place to start your unit testing effort.