One problem I’ve encountered is that some people in organisations (companies, standards bodies) got into their positions before the renaissance in testing began -one which Kent Beck and colleagues deserve the credit- and as a result, they often view the time spent testing as a waste. You have to lie and say you are debugging, which is what they can relate to.

I think you and Kent simply differ in your determination of ‘high cost’ and so navigate the tradeoffs differently; perhaps as you both differ (greatly) in focus.

We can’t avoid bugs. But we can do a lot to ensure that the bugs we do write are ‘higher quality bugs’. IBM’s Orthogonal Defect Classification (ODC) scheme/methodology is one good example.

http://www.research.ibm.com/softeng/ODC/ODC.HTM

The trick it seems, is ensuring project teams all agree on how to navigate the cost benefit tradeoffs and are informed enough (continuously) so that these tradeoffs can be tweaked and tuned over time.

Defining an explicit test strategy on day one certainly helps focus efforts in the right direction. The maturity to follow that strategy empowers teams to meet quality demands. The wisdom to adapt/tailor it to changing needs/focus is needed to scale it over time.

Cheers,

Darach.

And the other way round: we receive the most bug reports on features not tested – the GUI and client side. Thats another story on how those kinds of tests could be automated.

I develop and coach almost entirely at startups, and I teach people to test from day 1. Testing supports refactoring, and it enables rapid, radical changes in product direction, both vital in startups. If the question is “test or no test”, my answer is always: test, test, test!

Still, Kent’s absolutely right: there are cost-benefit tradeoffs involved. If somebody has a couple years experience with TDD and they have the discipline to actually backfill tests later on, I have no problem with them leaving small things untested if they’re currently hard to test. What’s hard to test today may be easy to test in a couple of weeks.

More accurately, I have no problem just as long as they’re working in the release-early-and-often context that Kent is. If you make it very easy to report bugs (and hopefully automatically report exceptions), then your customers will quickly tell you when you’ve cut too many corners on quality. Being too cavalier for a few hours or a few days is survivable, especially, as Steve Vinoski points out, on the left edge of the adoption curve. Being sloppy for weeks or months, though, is one of the many ways to kill your startup dead.

Even though I am at an early stage of the product, I still write quite a few tests. The difference from my usual practice is that if the cost is high and the risk is low I don’t insist on testing everything. The question I ask, because this is such an early stage product and I need customer feedback to survive, is, “Will this test help me get more customer feedback sooner?” In a more mature product I ask, “Will this test reduce cost over time?” That’s the distinction I was trying to make.

I was surprised because I had thought that the “correct” rule wasn’t flexible. Turns out everything we do as programmers should serve business purposes. That shouldn’t have been a surprise but it was.