Why Code Coverage Alone Doesn’t Mean Squat

Agile software development is all the rage these days. One of Agile’s cornerstones is the concept of test driven development (TDD). With TDD, you write the test first, and write only enough code to make the test pass. You then repeat this process until all functionality has been implemented, and all tests pass. TDD leads to more modular, more flexible, and better designed code. It also gives you, by the mere nature of the process, a unit test suite that executes 100% of the code. This can be a very nice thing to have.

However, like most things in life, people often focus on the destination, and pay little attention to the journey required to get there. We as human beings are always looking for short cuts. Some software managers see 100% code coverage as a must have, not really caring how that goal is achieved. But it is the journey to 100% code coverage that provides the benefits that most associate with simply having 100% code coverage. Without taking the correct roads, you can easily create a unit test suite that exercises 100% of your code base, and still end up with a buggy, brittle, and poorly designed code base.

100% code coverage does not mean that your code is bug free. It doesn’t even mean that your code is being properly tested. Let me walk through a very simple example.

I’ve created a class, MathHelper that I want to test. MathHelper has one method, average, that takes a List of Integers.

/**
 * Helper for some simple math operations.
 */
public class MathHelper {

    /**
     * Average a list of integers.
     * 
     * @param integerList The list of integers to average.
     * @return The average of the integers.
     */
    public float average(List<Integer> integerList) {
        ...
    }
}

Caving into managerial pressure to get 100% code coverage, we quickly whip up a test for this class. Abracadabra, and poof! 100% code coverage!

coverage-green-bar

So, we’re done. 100% code coverage means our class is adequately tested and bug free. Right? Wrong!

Let’s take a look the the test suite we put together to reach that goal of 100% code coverage.

public class MathHelperTest {

    private MathHelper _testMe;

    @Before
    public void setup() {
        _testMe = new MathHelper();
    }

    @Test
    public void poor_example_of_a_test() {
        List<Integer> nums = Arrays.asList(2, 4, 6, 8);
        _testMe.average(nums);
    }
}

Ugh! What are we really testing here? Not much at all. poor_example_of_a_test is simply verifying that the call to average doesn’t throw an exception. That’s not much of a test at all. Now, this may seem like a contrived example, but I assure you it is not. I have seen several tests like this testing production code, and I assume that you probably have too.

So, let’s fix this test by actually adding a test!

    @Test
    public void a_better_example_of_a_test() {
        List<Integer> nums = Arrays.asList(2, 4, 6, 8);
        _testMe.average(nums);
        assertEquals(5.0, result);
    }

Let’s run it, and see what we get.

java.lang.AssertionError: expected:<5.0> but was:<2.0>

Well, that’s certainly not good. How could the average of 2, 4, 6, and 8 be 2? Let’s take a look at the method under test.

    public float average(List<Integer> integerList) {
        long sum = 0;
        for (int i = 0; i < integerList.size() - 1; i++) {
            sum += integerList.get(i);
        }
        return sum / integerList.size() - 1;
    }

Ok, there’s the bug. We’re not iterating over the full list of integers that we have been passed. Let’s fix it.

    public float average(List<Integer> integerList) {
        long sum = 0;
        for (Integer i : integerList) {
            sum += i;
        }
        return sum / integerList.size();
    }

We run the test once again, and very that our test now passes. That’s better. But, let’s take a step back for a second. We had a method with unit tests exercising 100% of the code that still contained this very critical, very basic error.

With this bug now fixed, we commit the code to source control, and push a patch to production. All is fine and dandy until we start getting hammered with bug reports describing NullPointerExceptions and ArithmeticExceptions being thrown from our method. Taking another look at the code above, we realize that we have not done any validation of the input parameter to our method. If the integerList is null, the for loop will throw a NullPointerException when it tries to iterate over the list. If the integerList is an empty list, we will end up trying to divide by 0, giving us an ArithmeticException.

First, let’s write some tests that expose these problems. The average method should probably throw an IllegalArgumentException if the argument is invalid, so let’s write our tests to expect that.

    @Test(expected=IllegalArgumentException.class)
    public void test_average_with_an_empty_list() {
        _testMe.average(new ArrayList<Integer>());
    }
    
    @Test(expected=IllegalArgumentException.class)
    public void test_average_with_a_null_list() {
        _testMe.average(null);
    }

We first verify that the new tests fail with the expected NullPointerException and ArithmeticException. Now, let’s fix the method.

    public float average(List<Integer> integerList) 
            throws IllegalArgumentException {
        
        if (integerList == null || integerList.isEmpty()) {
            throw new IllegalArgumentException(
                "integerList must contain at least one integer");
        }
        
        long sum = 0;
        for (Integer i : integerList) {
            sum += i;
        }
        return sum / integerList.size();
    }

We run the tests again, and verify everything now passes. So, there wasn’t just one bug that slipped in, but three! And, all in code that had 100% code coverage!

As I said in the beginning of the post, having a test suite that exercises 100% of your code can be a very valuable thing. If achieved using TDD, you will see many or all of the benefits I list at the top of the post. Having a solid test suite also shields you from introducing regressions into your code base, letting you find and fix bugs earlier in the development cycle. However, it is very important to remember that the goal is not to have 100% coverage, but to have complete and comprehensive unit tests. If your tests aren’t really testing anything, or the right thing, then they become virtually useless.

Code coverage tools are great for highlighting areas of your code that you are neglecting in your unit testing. However, they should not be used to determine when you are done writing unit tests for your code.

Moved Rails Apps to GitHub

I’ve spent some time over the past couple of months cleaning up the Rails apps I’ve written with the goal of making them available for download on this site. I wrapped that up last week with the release of Addressbook. However, I soon realized that making the code available on this site was not ideal, and it would be much better if I made the code available via some code hosting site, like Sourceforge or GitHub. This gives the projects a little more visibility, allows for easier contributions to the projects, and simply provides a better home for the code. So, I spent the past few days making Diners Club, Addressbook, and Karate Journal available on GitHub. Please check the project pages on this site for links to the GitHub repositories.

Choosing GitHub was fairly easy. I’ve been wanting to learn Git for a while now, GitHub has all of the features I needed (plus some nice-to-haves), and it seemed pretty straight forward to use. So far so good!

I did however have to do some additional cleanup before the move to GitHub. Previously, I had a bash script that I ran that would export the code from my local Subversion repository, and wipe out any user names, passwords, or other configuration that I didn’t want to publish with the code. The script would then tar and zip up the code, and push it to the site. This wouldn’t fly with GitHub, since everybody would be able to see the code in the repository. So, some work was needed to provide template config files (database.yml.template for example) for people to specify their own configuration, and I removed the original files from source control.

All things considered, I’m very happy with the move, and wish I’d done it sooner. I think from now on, I’ll be using GitHub from the start with new projects.

Merry Christmas everybody!

Addressbook Webapp Has Been Released

Yay for me! I set a personal goal to have the code for all of my Rails apps on this site by the end of the year. Tonight, I can check that one off the list. The code for the Addressbook webapp is now available for download. Addressbook was not only my first Rails app, but also my first experience with Ajax. And trust me, it shows. I learned a lot from this project, especially what not to do. However, I must also say that I use Addressbook more than any other personal project I have ever completed. So, it can’t be that bad :) I love the fact that I can access my contact information from anywhere, and that I can manage groups of addresses and print mailing labels with the click of a button. Sure, the UI can be a bit unintuitive, but it’s not that big of an issue for me. There are a few more things I’d like to do with this project. We’ll see where it goes from here.

Really Cool Site Containing Java Conference Videos

Yesterday I stumbled upon Parleys.com. It appears that Parleys has been around for a while, providing podcasts of various tech presentations (audio only). However, the new version of the site, still in Beta, takes their offerings to an entirely new level. It now contains several videos of presentations from various Java conferences over the past few years. Not only is the content fantastic, but the user interface is very slick as well. The UI displays not only the audio in sync with the slides, but also a video of the presenter, giving you the full experience. The navigation options are also great, providing a table of contents and a time line for the presentation…letting your jump to a specific point in the talk. In addition, Parley’s provides information about the speaker and the talk, a list of related talks, and the ability to post tags and comments at specific points in the talk.

In my opinion, the best feature provided by Parleys.com is the ability to watch the presentations, with all of the UI features, offline. This is HUGE for me, as I spend almost two hours every day sitting on a train. Parley’s provides an Adobe AIR application that makes all of this possible. Adobe AIR is a runtime environment that “lets developers use proven web technologies to build rich Internet applications that run outside the browser on multiple operating systems”. The Linux version of AIR, still in Beta, works wonderfully. The AIR application operates just like the site, and provides some additional functionality to download the presentations for later viewing.

I’ve been spending quite a bit of time lately looking for a site like this. With the current economic conditions, I’d imagine that most companies are cutting back on the number of conferences they send their developers to. According to the site, they plan on continuing to add content. If they do, I see this as an invaluable resource for keeping up to date on what is going on in the Java community.