Testing is Hard

Tests don’t verify that our code works; they verify that a particular edge in
our code works and only if we wrote the test case correctly.

Thus, tests are useful when:

1. We’re testing something that we might’ve gotten wrong,
2. The test case correctly catches that mistake, AND
3. The test case can be implemented efficiently

The reason that I don’t just assume #2 upfront that #2 is because the
whole reason that we’d want to write test cases in the first place is that the logic is complicated enough that the implementation might be wrong. So, it matters
whether or not the test case is more likely to be correct than the
implementation.

The reason that I mention #3 is because as our test cases become more fine
grained, testing more edges and verifying the original functionality more
thoroughly, they’ll also tend to take the shape of the original implementation.
So, very very fine grained testing is going to take a lot of time and it’s also
going to be error prone…just like the original.

Therefore, what we really want are simple tests that elicit complicated and
error prone behaviors across many edges and verify them. Because we’re eliciting complicated behaviors we know that #1 is likely true. Because our test cases are simple we know that #2 and #3 are likely true. That way we can get the most value out of our test cases.

But, obviously, simple test cases are going to have their limits — there will
always be big areas of the underlying code which aren’t thoroughly tested
because we have to keep our tests simple. So, we need to leverage a combination of techniques to really get good coverage and fix all or nearly all of our mistakes. This is the topic discussed in Error Bars.

PS: I’ve stated a lot of assumptions here about what tests do and don’t do well. I’ll discuss that in more detail in a future post. This post is just intended as a practical set of suggested guidelines for people who are already like-minded to me.