Skip to main content

The perils of over testing

Test Driven Development (TDD) is a great thing. It allows us to write code "safe" in the knowledge that if we accidentally break something in another part of the system, we will be alerted to it and can re-write or fix accordingly.

It's such a good thing that often in large teams people will insist on this idea of 100% code coverage (i.e. every piece of working code has a corresponding test). Also, some people are compelled to test every single permutation and variation of how someone will access their program/website.

TDD does have some costs however. Some are evident, but others not so.

Cost 1 - Tests take time to run
True, automated testing is faster than manual testing, but they do take time to run. This time means every time you make a change to the code, you need to wait for the suite to finish running in order to feel that your code is "safe" for deployment.

Most of this time is a necessary evil.

A lot of it is unnecessary...

Should you write a feature that checks to see that a form is disabled if a text box is not checked? Well, it depends...

If this is essential to the end user getting to a process, then possibly yes.

If this is just to check a possible error condition that has little or no bearing on the final result, then possibly no.

You could maybe write it as you are developing a feature and then remove it when you are done.

Or else you can write an Object Oriented JS component (FormDisabler) and then test its logic with Jasmine.

Bear in mind that testing JavaScript in a feature will bring up the whole stack which takes time. Time is valuable.

One analogy is that testing is like packing for a trip. You might think that you will need to bring everything you have to cover every possible situation, but really you probably only need an overnight bag with a couple of changes of clothes. The cost is that the more you bring, the more you have to carry.

Cost 2 - Brittleness
Over testing can lead to a very brittle code base where one small change can lead to many hours of fixing broken tests. In Agile development you will constantly make changes to the code base and often what is wanted in the end is not what is expressed in the beginning.

If you test every single possible scenario, then a small business change can lead to a domino effect of broken tests.

The key is to identify what aspects/methods of a feature are most important and to test those.

Classes should be made bulletproof, but not features.

Martin Fowler's Test Pyramid is a good guide.

At the end of the day, you have to use your brain and judgement to determine what tests are important and what tests are not. It is a subjective thing and can vary from team to team (and application to application). Back to the packing analogy, you should only pack what you need for the trip. If you are writing something mission critical that can never go down and the slightest mistake can lead to potential loss of human life, then by all means go for 100% test coverage. On the other hand, if a mistake in part of your app means a comment or review might not get posted, then test accordingly.


Popular posts from this blog

Master of my domain

Hi All, I just got myself a new domain ( ). The reason is that is offering cheap domain via and I thought after having this nickname for nigh on 10 years it was time to buy the domain before someone else did (also I read somewhere that using in your domain is the equivalent of an or email address...shudder...). Of course I forgot that I would have to re-register my blog everywhere (which is taking ages) not to mention set up all my stats stuff again. *sigh*. It's a blogger's life... In any case, don't forget to bookmark the new address and to vote me up on Technorati !

Speeding up RSpec

So today I have been looking into getting our enormous battery of tests to run faster. I have yet to find anything that works for Cucumber, but I did find an interesting way to speed up RSpec which is detailed here. Basically, it seems that by not collecting garbage too frequently, you can make your tests run much faster (at the expense of memory management of course). We observed a 30% reduction in the time it takes to run an RSpec test suite. I did try to implement this on Cucumber, however because we need to store much more in memory to set up and tear down our objects, it meant that I kept running out of memory when I wasn't using the default Garbage Collection and the tests took even longer (so, buyer beware). I suppose if you had a small set of features though you might see some benefit.

Elixir - destructuring, function overloading and pattern matching

Why am I covering 3 Elixir topics at once? Well, perhaps it is to show you how the three are used together. Individually, any of these 3 are interesting, but combined, they provide you with a means of essentially getting rid of conditionals and spaghetti logic. Consider the following function. def greet_beatle(person) do case person.first_name do "John" -> "Hello John." "Paul" -> "Good day Paul." "George" -> "Georgie boy, how you doing?" "Ringo" -> "What a drummer!" _-> "You are not a Beatle, #{person.first_name}" end end Yeah, it basically works, but there is a big old case statement in there. If you wanted to do something more as well depending on the person, you could easily end up with some spaghetti logic. Let's see how we can simplify this a little. def greet_beatle(%{first_name: first_name}) do case first_name d