Skip to main content

The perils of over testing

Test Driven Development (TDD) is a great thing. It allows us to write code "safe" in the knowledge that if we accidentally break something in another part of the system, we will be alerted to it and can re-write or fix accordingly.

It's such a good thing that often in large teams people will insist on this idea of 100% code coverage (i.e. every piece of working code has a corresponding test). Also, some people are compelled to test every single permutation and variation of how someone will access their program/website.

TDD does have some costs however. Some are evident, but others not so.

Cost 1 - Tests take time to run
True, automated testing is faster than manual testing, but they do take time to run. This time means every time you make a change to the code, you need to wait for the suite to finish running in order to feel that your code is "safe" for deployment.

Most of this time is a necessary evil.

A lot of it is unnecessary...

Should you write a feature that checks to see that a form is disabled if a text box is not checked? Well, it depends...

If this is essential to the end user getting to a process, then possibly yes.

If this is just to check a possible error condition that has little or no bearing on the final result, then possibly no.

You could maybe write it as you are developing a feature and then remove it when you are done.

Or else you can write an Object Oriented JS component (FormDisabler) and then test its logic with Jasmine.

Bear in mind that testing JavaScript in a feature will bring up the whole stack which takes time. Time is valuable.

One analogy is that testing is like packing for a trip. You might think that you will need to bring everything you have to cover every possible situation, but really you probably only need an overnight bag with a couple of changes of clothes. The cost is that the more you bring, the more you have to carry.

Cost 2 - Brittleness
Over testing can lead to a very brittle code base where one small change can lead to many hours of fixing broken tests. In Agile development you will constantly make changes to the code base and often what is wanted in the end is not what is expressed in the beginning.

If you test every single possible scenario, then a small business change can lead to a domino effect of broken tests.

The key is to identify what aspects/methods of a feature are most important and to test those.

Classes should be made bulletproof, but not features.

Martin Fowler's Test Pyramid is a good guide.

http://martinfowler.com/bliki/TestPyramid.html

 Summary
At the end of the day, you have to use your brain and judgement to determine what tests are important and what tests are not. It is a subjective thing and can vary from team to team (and application to application). Back to the packing analogy, you should only pack what you need for the trip. If you are writing something mission critical that can never go down and the slightest mistake can lead to potential loss of human life, then by all means go for 100% test coverage. On the other hand, if a mistake in part of your app means a comment or review might not get posted, then test accordingly.

Comments

Popular posts from this blog

Freezing Gems

What is a gem and why would you want to freeze it? In Ruby, there are times when you want to access pieces of functionality that other people of written (3rd party libraries) and you normally have 2 options. You can install a plug in or install a gem. Normally the method you use is determined by which ever is made available by the author. Gems are installed on the host machine and are pretty handy when you want to run things in the command line or else across lots of projects, but their downside is that if you use a gem in a Rails project there is no automatic publishing mechanism when you deploy your site. You will need to log onto the remote host machine and install the gem manually. Plugins are specific to Rails and are similar to gems in that they are also 3rd party libraries. However they are associated with your Rails project as opposed to your machine so they will get posted to the server on a regular deploy. Freezing a gem is the process of transforming a gem into a plug in

Unit/Functional Testing RubyAMF

One of my current projects is using RubyAMF to communicate with Flash (http://rubyforge.org/projects/rubyamf/). On the whole this is really nice because it allows you to transfer Ruby objects directly to ActionScript ones (as opposed to translating the object into XML, sending the XML and then recreating the object in ActionScript). However, Rails does not provide a built in transport mechanism for AMF, so we cannot run functional testing directly on the data call (as we could for an XML or HTML transport layer). This is a show stopper for a lot of people (Rails w/o Unit testing = a big mess of trouble when something goes wrong). We can though serve both the HTML and the AMF formats depending on the request format. This means that we can test the object instantiation logic and make sure there are no errors in the controllers (though we cannot check the actual format of the data being served). In the controller, instead of rendering AMF alone, do the following respond_to do |format|

Responsive Web Design

I wanted to go over Responsive Web Design using CSS. In the old days of web development, we had to code to common screen sizes (i.e. 800 X 600, 1024 X 768) and we patiently waited for people to upgrade their computers to have a decent amount of screen real estate so we could design things the way we really wanted. We also took on semi stretchy web layouts etc to expand and contract appropriately. Then about 2 or 3 years ago, Apple released this little device called an iPhone with a 320 X 480 resolution which took the world by storm and suddenly a lot of people were viewing your website on a tiny screen again... Anyways, as it can be difficult to design a site which looks good on 320 X 480 AND 1680 X 1050, we need to come up with some kind of solution. One way is to sniff the client and then use an appropriate stylesheet, but then you are mixing CSS with either JavaScript or server side programming and also potentially maintaining a list of appropriate clients and stylesheets. Also,