An Experiment in Specification-based Testing


Ian MacColl
Department of Computer Science, The University of Queensland, Queensland 4072, Australia.
ianm@cs.uq.edu.au

David Carrington
Department of Computer Science, The University of Queensland, Queensland 4072, Australia.
davec@cs.uq.edu.au

Phil Stocks
Department of Computer Science, Hill Center, Busch Campus, Rutgers University, Piscataway, NJ 08855, USA.
pstocks@cs.rutgers.edu


Abstract

The Test Template Framework (TTF) is a formal, abstract model of tests used to derive a hierarchy of test information, including test cases and oracles. The hierarchy is derived from a formal model-based specification, for example, one written in the Z notation. A case study presents a Test Template Hierarchy (TTH) derived from a specification for topological sorting. Test suites were also prepared from the same specification by two classes of testers: experts and non-experts. Qualitative analysis of the test suites indicated a lack of structured process in their development. The suites were analysed quantitatively by executing all valid test inputs on five implementations seeded with faults and the TTF-derived suite was the only one to identify all faults. The most structured of the non-TTF suites identified more faults than any other non-TTF suite, and it was found to be covered by the TTF-derived suite. This informal comparison demonstrates that a systematic, formal approach can raise the quality of suites produced by non-expert testers.
Conference Home Page