I was asked 'how much does it cost to get the last 15% of code coverage?', which also led to the question of 'how much does a unit test cost?'   So I went on a mission to determine this.

For this post, substitute $$ for whatever your hourly rate is, or your companies 'billable' rate, or any 'constant' number that represents a cost per hour rate.   For this post, to keep things simple on math I'll use $60 for my $$ constant for our fictious company we'll use as an example.

So let's use a project I'm working on which is an average project of average size and complexity for my skill set to get the time per test which we can use with our constant to determine our cost per test.  Based upon my development time that has been logged on that project and the number of unit tests in that project, assuming all I’ve done is wrote unit tests, no new screens, no stored procedures, no tables, no integration of external libraries or services, no reusable components created, etc; the cost of EACH unit test would be 12 minutes or using our variable for $$ this equates to $12.00/test.

Ok, now let’s be slightly more realistic and say I spent 50% of my time doing tests.  Probably still high, but some books/white papers I've read might suggest that is how much of your coding time might be writing unit tests.  This change lowers the cost to $6.00/test.

Next, let's look at the cost of the bug.   There's the time it takes for the user to report the bug (assuming theu doesn't think it's their mistake and they try several times before reporting), the time it takes for the whomever handles the call from the user to enter the bug into your tracking system, the time it takes to reproduce and verify the bug, the time it takes to fix, deploy to your QA department or equivelant, QA to test the fix, and deploy to product the fix for the bug.  And let's assume you got it fixed right the first time.   So a totally unrealistic best cost of a bug would be 2 hours (and I feel I'm being EXTREMELY generous here), using our constant would be $240.  If you don't feel '2 hours' is accurate, substitute a time you think is appropriate and adjust the numbers below.   But I urge you to look at all of your bugs for the past year and see what the TOTAL average is, not just the 'developer coding' average for time submitted against it.

In the cost of $240 using the rate we mentioned above, using the average cost of a test we determined above ($6.00), we could have wrote 40 unit tests.

All it takes is 1 bug to cost our fictitious company at least $240 using these numbers, which is nothing to a company.   Lets say there where 25 bugs for the year (extremely good year for most companies I've worked at), now we're talking $6000, and that's just ONE application.  In reality we all know that often a bug takes a lot longer than 2 cumulative hours between all parties involved to fix a bug from start to stop and most companies that don't have good testing departments/practices surpass 25 bugs per year easily.  

All it costs our fictitious company to write a single unit test that would’ve potentially caught that single bug is $6, or $150 for all 25 for the year (assuming one unit test would've caught each bug).  So we can say that a bug cost 40X that of a unit test that would've caught the bug to begin with.  Or to put it another way, writing the unit test instead of allowing the single bug it covered to go to production could have been a cost savings of 3900% to our fictitious company.

So now, the question of code coverage – how much does it cost to get that last 15% of code coverage.   Take the number of tests it would take to cover that 15% and multiply it by $6 (using our constant) and there is your cost.   To know how much it potentially cost our fictitous company not to have that coverage and have it show up as bug, take the number of tests it would’ve took to get 100% coverage and multiply that by $240 because each of those paths is a potential bug.

Let’s say it would take 20 tests to cover that last 15%.   This would cost the company $120.   On the flip side, not to have it could cost our fictitious company as much as $4800 worst case where a unique SINGLE bug would've been caught by each of the 20 tests and fixed the first time.

Now these are the extremes and averages and rely on using my average test writing time for this particular project (which happens to be close to the average with 'suggested' average unit test writing time I've seen in books).   Certainly there are cases where 15% coverage won’t cost the company a dime and tests that cost more than our fictitous $6/test, just as there are ones that will barely cost a dime.   But at the same time, we also have keep in mind those bugs that have 20, 30, 50 hours on them which cost our theoritical company thousands; we all have had them and don't pretend you haven't!   Really, it’s all a gamble, that at $6/test on average in comparission to $120 for a bug I’m not willing to take if at all possible unless I can say definitively some other way that the 15% isn’t a risk.   This also doesn’t mean bugs won’t be found as there are no silver bullets, but I rather have the unit tests there and have the confidence that I covered everything rather as much as possible at the time than putting it out there and seeing what happens and hoping for the best.

Unit testing is all about safety and confidence.   If your building where you work was missing 15% of it's fire saftey features, would you feel comfortable there?   What was in that 15% that's not covered?   The sprinkler automatic valve that release the water?   The alarm siren that goes off?   The automatic movement to the first floor and shut down of your elevators?  How confident would you be in your safety there?   Would you just sit there while you smelt smoke coming from the office kitchen and say 'well we'll see what happens, may not be a fire, maybe burnt popcorn in the microwave'?   I guess that's okay to do if you're on the first floor near an exit, or your going to stop what your doing and go manually check each time you 'think' something might be a problem.

Just some things to think about.   I emplore anyone who would disagree with the statement of 'not writing a test costs more than writing a test in a enterprise application' to post their comments as I'd be very interested to here about that and meet what is apparently the perfect programmer or near perfect programmer who can find, fix, and deploy bugs nearly the instant they are found.

For those who will say 'but what about the maintenance costs of unit tests?'.   I'll say the same thing I read in a very good book by Roy Osherove - the only time a test should fail or require changing is because of requirements changes.  It's that simple.   So your question now is essentially 'but what about the maintenance costs of changing requirements?' - and that has been answered in twice as many books as there are on the cost saving of unit testing.   In short, please show me list that contains 1/20th the amount of printed books on why you shouldn't have unit tests as there are on why you should and I might entertain a response to your question.