03-10-2012, 11:17 AM
Invariant-Based Automatic Testing of Modern Web Applications
Invariant-Based Automatic.pdf (Size: 2.65 MB / Downloads: 32)
Abstract
AJAX-based Web 2.0 applications rely on stateful asynchronous client/server communication, and client-side runtime
manipulation of the DOM tree. This not only makes them fundamentally different from traditional web applications, but also more errorprone
and harder to test. We propose a method for testing AJAX applications automatically, based on a crawler to infer a state-flow
graph for all (client-side) user interface states. We identify AJAX-specific faults that can occur in such states (related to, e.g., DOM
validity, error messages, discoverability, back-button compatibility) as well as DOM-tree invariants that can serve as oracles to detect
such faults. Our approach, called ATUSA, is implemented in a tool offering generic invariant checking components, a pluginmechanism
to add application-specific state validators, and generation of a test suite covering the paths obtained during crawling. We
describe three case studies, consisting of six subjects.
INTRODUCTION
THERE is a growing trend to move applications toward the
web. Well-known examples include Google’s mail and
office software comprising spreadsheet, word processing,
and calendar applications. The reasons for this move to the
web are manifold:
. no installation effort for end users;
. automatic use of the most recent software version
by all users, thus reducing maintenance and
support costs;
. universal access from any browser on any machine
with Internet access, not only to the application but
also to user data;
. new collaboration and community building opportunities
as supported by Web 2.0 applications.
For today’s web applications, one of the key technologies
facilitating this move is AJAX, an acronym for “Asynchronous
JAVASCRIPT and XML” [13]. With AJAX, web
browsers not only offer the user navigation through a
sequence of HTML pages, but also dynamic rich interaction
via graphical user interface (UI) components.
RELATED WORK
Modern web interfaces incorporate client-side scripting and
user interface manipulation which is increasingly separated
from server-side application logic [36]. Although the field of
rich web interface testing is mainly unexplored, much
knowledge may be derived from two closely related fields:
traditional web testing and GUI application testing. We
survey these in Sections 2.1 and 2.2. We describe current
AJAX testing approaches in Section 2.3, after which we
provide a short overview of the use of invariants for web
testing in Section 2.4.
Traditional Web Testing
Benedikt et al. [4] present VeriWeb, a tool for automatically
exploring paths of multipage websites through a crawler
and detector for abnormalities such as navigation and page
errors (which are configurable through plugins). VeriWeb
uses SmartProfiles to extract candidate input values for
form-based pages. Although VeriWeb’s crawling algorithm
has some support for client-side scripting execution, the
paper provides insufficient detail to determine whether it
would be able to cope with modern AJAX web applications.
VeriWeb offers no support for generating test suites as we
do in Section 6.
Tools such as WAVES [18] and SecuBat [19] have been
proposed for automatically assessing web application
security. The general approach is based on a crawler
capable of detecting data entry points which can be seen
as possible points of security attack. Malicious patterns, e.g.,
SQL and XSS vulnerabilities, are then injected into these
entry points and the response from the server is analyzed to
determine vulnerable parts of the web application.
GUI Application Testing
Reverse engineering a model of the desktop (GUI) in order
to generate test cases has been proposed by Memon [23].
AJAX applications can be seen as a hybrid of desktop and
web applications since the user interface is composed of
components and the interaction is event-based [27]. However,
AJAX applications have specific features, such as the
asynchronous client/server communication and dynamic
DOM-based user interface, which make them different from
traditional GUI applications [22], and therefore require
other testing tools and techniques.
Invariants
The concept of using invariants to assert program behavior at
runtime is as old as programming itself [8]. For the domain of
web applications, any approach that performs validation of
the HTML output (e.g., [4], [2]) could be considered to be
using invariants on the DOM. This paper makes the use of
invariants for testing web applications explicit by defining
different types of (client side) invariants, providing a
mechanism for expressing those invariants, and automatically
checking them through dynamic analysis.
Automatic detection of invariants is another direction that
has gained momentum. The best-known work is that of Ernst
et al. on Daikon [12], a tool capable of inferring likely invariants
from program execution traces.Amore recent tool is DoDom
[30], capable of inferring DOM invariants. We have also
started exploring ways of automatically detecting DOM and
JAVASCRIPT invariants in web applications [14]
Creating and Comparing States
After firing an event on a candidate clickable, the algorithm
inspects the resulting DOM tree to see if the event results in
a modified state (line 5 Algorithm 2). If a similar state is part
of the state flow graph already, merely an edge is created,
identifying the type of click and the location clicked. If the
next state is not part of the graph already, a new state is
created and added first (line 7 Algorithm 2).
The level of abstraction achieved in the resulting stateflow
graph is largely determined by the algorithm used to
compare DOM trees (which reflect the states in the flow
graph). A generic and effective way is to use a simple string
edit distance algorithm such as Levenshtein [20]. This has
the advantage that it does not require application-specific
knowledge and that the algorithm can be fine-tuned by
means of a similarity threshold (between 0 and 1).
TESTING AJAX STATES THROUGH INVARIANTS
With access to different dynamic web states we can check
the user interface against different constraints. We propose
to express those as invariants, which we can use as an oracle
to automatically conduct sanity checks in any state.
Although the notion of invariants has predominantly been
applied to programming languages for software evolution
[12] and verification [3], we believe that invariants can also
be adopted for testing modern web applications to specify
and constrain DOM elements’ properties, their relations,
and occurrences.
In this work, we distinguish between generic and
application-specific invariants on the DOM-tree, between
DOM-tree states, and on the runtime JAVASCRIPT variables.
Each invariant is based on a fault model [7], representing
AJAX-specific faults that are likely to occur and which can
be captured through the given invariant.
Performance and Scalability
The state space of any realistic web application is huge
and can cause the well-known state explosion problem. To
constrain the state space, we provide the tester with a set
of configurable options. These constraints include the
maximum search depth level, similarity threshold for
comparing states, maximum number of states per domain,
maximum crawling time, and the option of ignoring
external links and links that match some predefined set of
regular expressions. The main component that can
influence the performance and scalability is the crawling
part. The performance of crawling an AJAX site depends
on many factors, such as the speed at which the server
can handle requests, how fast the browser and client-side
JAVASCRIPT can update the interface, and the size of the
DOM tree.
Application Size
The six experimental subjects involve around 20,000 lines
of JAVASCRIPT library code, several hundred lines of
custom application code, and several thousand dynamic
DOM states. One might wonder whether the size of the
subjects counts against the external validity of our study.
Our results, however, are based on dynamic analysis rather
than static code analysis; hence, the amount of JAVASCRIPT
code is not the determining factor in our view. The number
of dynamic states is, in this case, a more realistic measure.
The limiting factor for the number of states to be examined
is the amount of memory available and the size of the
DOM tree. Based on our experiments, the maximum
number of states can be calculated by sizeOfðmemoryÞ
3sizeOfðDOMÞ .
CONCLUDING REMARKS
In this paper, we have proposed a method for testing AJAX
applications automatically. Our starting point for supporting
AJAX-testing is CRAWLJAX, a crawler for AJAX
applications that we proposed in our earlier work [24],
which can dynamically make a full pass over an AJAX
application. Our current work consists of extending the
crawler substantially for supporting automated testing of
modern web applications. We developed a series of
plugins, collectively called ATUSA, for invariant-based
testing and test suite generation.