15-11-2012, 04:39 PM
The Model-Driven Test Design Process
1The Model-Driven Test.ppt (Size: 4.16 MB / Downloads: 49)
The First Bugs
“an analyzing process must equally have been performed in order to furnish the Analytical Engine with the necessary operative data; and that herein may also lie a possible source of error. Granted that the actual mechanism is unerring in its processes, the cards may give it wrong orders. ” – Ada, Countess Lovelace (notes on Babbage’s Analytical Engine)
Costly Software Failures
NIST report, “The Economic Impacts of Inadequate Infrastructure for Software Testing” (2002)
Inadequate software testing costs the US alone between $22 and $59 billion annually
Better approaches could cut this amount in half
Huge losses due to web application failures
Financial services : $6.5 million per hour (just in USA!)
Credit card sales applications : $2.4 million per hour (in USA)
In Dec 2006, amazon.com’s BOGO offer turned into a double discount
2007 : Symantec says that most security vulnerabilities are due to faulty software
Testing in the 21st Century
More safety critical, real-time software
Embedded software is ubiquitous … check your pockets
Enterprise applications means bigger programs, more users
Paradoxically, free software increases our expectations !
Security is now all about software faults
Secure software is reliable software
The web offers a new deployment platform
Very competitive and very available to more users
Web apps are distributed
Web apps must be highly reliable
Test Design in Context
Test Design is the process of designing input values that will effectively test software
Test design is one of several activities for testing software
Most mathematical
Most technically challenging
Test Design – (b) Human-Based
This is much harder than it may seem to developers
Criteria-based approaches can be blind to special situations
Requires knowledge of :
Domain, testing, and user interfaces
Requires almost no traditional CS
A background in the domain of the software is essential
An empirical background is very helpful (biology, psychology, …)
A logic background is very helpful (law, philosophy, math, …)
This is intellectually stimulating, rewarding, and challenging
But not to typical CS majors – they want to solve problems and build things
Test Automation
This is slightly less technical
Requires knowledge of programming
Fairly straightforward programming – small pieces and simple algorithms
Requires very little theory
Very boring for test designers
Programming is out of reach for many domain experts
Who is responsible for determining and embedding the expected outputs ?
Test designers may not always know the expected outputs
Test evaluators need to get involved early to help with this
Test Execution
This is easy – and trivial if the tests are well automated
Requires basic computer skills
Interns
Employees with no technical background
Asking qualified test designers to execute tests is a sure way to convince them to look for a development job
If, for example, GUI tests are not well automated, this requires a lot of manual labor
Test executors have to be very careful and meticulous with bookkeeping
Other Activities
Test management : Sets policy, organizes team, interfaces with development, chooses criteria, decides how much automation is needed,
Test maintenance : Save tests for reuse as software evolves
Requires cooperation of test designers and automators
Deciding when to trim the test suite is partly policy and partly technical – and in general, very hard !
Tests should be put in configuration control
Test documentation : All parties participate
Each test must document “why” – criterion and test requirement satisfied or a rationale for human-designed tests
Ensure traceability throughout the process
Keep documentation in the automated tests
Using MDTD in Practice
This approach lets one test designer do the math
Then traditional testers and programmers can do their parts
Find values
Automate the tests
Run the tests
Evaluate the tests
Just like in traditional engineering … an engineer constructs models with calculus, then gives direction to carpenters, electricians, technicians, …
Source of Structures
These structures can be extracted from lots of software artifacts
Graphs can be extracted from UML use cases, finite state machines, source code, …
Logical expressions can be extracted from decisions in program source, guards on transitions, conditionals in use cases, …
This is not the same as “model-based testing,” which derives tests from a model that describes some aspects of the system under test
The model usually describes part of the behavior
The source is usually not considered a model