26-06-2014, 11:22 AM
Using Property-Based Oracles when Testing Embedded System Applications
Using Property.pdf (Size: 152.03 KB / Downloads: 21)
Abstract
—Embedded systems are becoming increasingly
ubiquitous, controlling a wide variety of popular and safetycritical
devices. Effective testing techniques could improve the
dependability of these systems. In prior work we presented an
approach for testing embedded systems, focusing on embedded
system applications and the tasks that comprise them. In this
work we focus on a second but equally important aspect
of testing embedded systems; namely, the need to provide
observability of system behavior sufficient to allow engineers
to detect failures. We present several property-based oracles
that can be instantiated in embedded systems through program
analysis and instrumentation, and can detect failures for which
simple output-based oracles are inadequate. An empirical study
of our approach shows that it can be effective.
INTRODUCTION
Embedded systems are used to control a wide variety of
applications, ranging from non-safety-critical systems such
as cellular phones and televisions to safety-critical systems
such as automobiles, airplanes, and medical devices. Clearly,
systems such as these must be sufficiently dependable. There
is ample evidence, however, that faults in embedded systems
have led to numerous failures [16].
To address dependability problems in embedded systems,
researchers have proposed various approaches ranging from
formal verification to testing techniques (e.g., [6], [14], [21],
[32]). On our analysis of prior work, however (Section II-C),
one need that has not been sufficiently addressed is to
provide methodologies by which the developers of embedded
system applications— which run in environments supported
by a wide range of underlying software components such as
middle-ware, operating system kernels, device drivers, and
hardware-related utilities — can cost-effectively test those
applications in the context of those environments.
In previous work [31], we presented an approach to help
developers of embedded system applications detect faults
that occur as their applications interact with underlying
system components. Our approach involves two dataflowbased
test adequacy criteria. First, we use dataflow analysis
to identify inter-layer interactions between application code
and lower-level (kernel and hardware-related) components
in embedded systems. Second, we use a further dataflow
analysis to identify inter-task interactions between tasks that
are initiated by the application. Application developers then
create and execute test cases targeting these interactions.
Helping engineers create test cases is important, but
test cases are useful only if they can reveal faults. To
reveal faults, test cases must produce observable failures.
Observability requires appropriate test oracles. The “oracle
problem” is a challenging problem in many testing domains,
but with embedded systems it can be particularly difficult.
Embedded systems employing multiple tasks can have nondeterministic
output, which complicates the determination
of expected outputs for given inputs, and oracle automation.
Faults in embedded systems can produce effects on program
behavior or state which, in the context of particular test
executions, do not propagate to output, but do surface later in
the field. Thus, oracles that are strictly “output-based”, i.e.,
focusing on externally visible aspects of program behavior
such as writes to stdout and stderr, file outputs, and
observable signals, may fail to detect faults.
PROPERTY-BASED ORACLES
In this section, we describe our approach for creating
test oracles based on properties of various synchronization
primitives and common interprocess communications. There
are three general steps involved in applying the approach:
1) Create test oracles based on properties – this can be
done by analyzing the source programs to extract their
interfaces and the program semantics of particular
software components.
2) Execute test cases on the system and generate runtime
trace information – this requires instrumentation of the
source programs, OS, libraries, and runtime systems.
3) Analyze trace information to detect violations – this
involves checking generated traces against oracles to
verify conformance with properties of interest.
Our approach requires test engineers to create properties
relative to particular embedded systems platforms, and thus,
manual effort and expertise are required and some properties
will be platform dependent. Once created, however, these
properties are relevant to all applications built on those
platforms so the cost of defining properties for them is
amortized. This differs from output-based oracles, which are
also typically defined manually, but must be defined for each
test case and program individually.
CONCLUSIONS AND FUTURE WORK
We have presented an approach for using property-based
oracles when testing embedded systems. We have conducted
an empirical study applying our techniques to two commercial
embedded system applications, and demonstrated that
they can be effective at revealing faults.
In addition to the need to perform additional empirical
work, there are several other avenues for future work. We
have already discussed prospects for using simulation platforms
to create non-intrusive instrumentation environments.
We also see the potential for creating other types of propertybased
oracles, including those for high-level concurrent
programming constructs for multi-task embedded software
(e.g., monitors) and real-time properties. Finally, the insights
obtained from this work can lead to the development of
tools that can automate the instrumentation process, monitor
runtime events, and detect faults.
ACKNOWLEDGEMENTS
This work was supported in part by the AFOSR through
award FA9550-09-1-0129 to the University of Nebraska -
Lincoln, and through the Korea Research Foundation under
award KRF-2007-357-D00215.Wayne Motycka helped with
the setup for the empirical study. Moonzoo Kim provided
helpful advice on content and presentation.