Regression Testing |
 |
 |
 |
| Written by Graham Stoney | |||
|
Regression testing is vital to ensuring steadily increasing software quality in any non-trivial application. Once something is working, you want to be sure that you don't inadvertently break it as development progresses. This is what regression testing is all about. The regression test suite should ideally be sufficiently fast to run that developers will routinely run it before checking code in. If you also take my advice about committing one change at a time, that means you'll be doing this a lot. So the regression test needs to be designed to run fast, possibly with a slower variant that does more thorough checking overnight. For this to work, you also need to do automated nightly rebuilds. The regression test should run all the module tests, preferably from the bottom up so that lower level failures are caught first, as they tend to cascade into higher level failures where they are harder to locate. It should also be run against both the optimized production version of the software (with assertions disabled) and the debug version (with assertions enabled). System level tests should also be included in the regression test, but if you're following my advice on module testing, the system level test will already exist, as they are simply the module tests for the top-level module, or whatever you're calling “the system”. Don't ignore seemingly trivial regression test failures. If a single bit is different in the output today from yesterday, work out why. Regression testing often relies on a “golden set” of known-good output; but often the only practical way of generating this output is by using the application itself, and verifying manually that the golden set really is correct. It is conceivable that changes in the golden set occur over time as new bugs are found; things like rounding errors can go unnoticed and cause very minor changes in output. But take great care about updating files in the “golden set”; don't convince yourself that the change is expected, especially if you weren't expecting it ahead of time and only find out during a regression test run. Social Bookmark this page:         
|