29-11-2012, 05:31 PM
Monitoring Online Tests through Data Visualization
Monitoring Online.pdf (Size: 2.35 MB / Downloads: 35)
Abstract
We present an approach and a system to let tutors monitor several important aspects related to online tests, such as
learner behavior and test quality. The approach includes the logging of important data related to learner interaction with the system
during the execution of online tests and exploits data visualization to highlight information useful to let tutors review and improve the
whole assessment process. We have focused on the discovery of behavioral patterns of learners and conceptual relationships among
test items. Furthermore, we have led several experiments in our faculty in order to assess the whole approach. In particular, by
analyzing the data visualization charts, we have detected several previously unknown test strategies used by the learners. Last, we
have detected several correlations among questions, which gave us useful feedbacks on the test quality.
INTRODUCTION
E-TESTING systems are being widely adopted in academic
environments, as well as in combination with other
assessment means, providing tutors with powerful tools to
submit different types of tests in order to assess learners’
knowledge. Among these, multiple-choice tests are extremely
popular, since they can be automatically corrected. However,
many learners do not welcome this type of test,
because often, it does not let them properly express their
capacity, due to the characteristics of multiple-choice questions
of being “closed-ended.” Even many examiners doubt
about the real effectiveness of structured tests in assessing
learners’ knowledge, and they wonder whether learners are
more conditioned by the question type than by its actual
difficulty.
In order to teach learners how to improve their
performances on structured tests, in the past, several
experiments have been carried out to track learners’
behavior during tests by using the think-out-loud method:
learners were informed of the experiment and had to speak
during the test to explain what they were thinking, while an
operator was storing their words using a tape recorder. This
technique might be quite invasive, since it requires learners
to modify their behavior in order to record the information
to analyze [1], [2], [3], [4], [5], which might vanish the
experiment goals, since it adds considerable noise in the
tracked data.
INFORMATION VISUALIZATION FOR KNOWLEDGE
DISCOVERY
In the last decade, the database community has devoted
particular efforts to the extraction of knowledge from data.
One of the main approaches for knowledge extraction is data
mining, which applies automatic algorithms to recognize
patterns in huge data collections [6], [7], [13]. Alternatively,
visual data mining presents data in a visual form to
stimulate user interaction in the pattern detection process.
A combination of visual and automatic data mining draws
together human cognitive skills and computer efficiency,
which permits faster and more efficient KDD.
Since in this paper, we are interested in a knowledge
extraction process in which the tutor plays a central role, in
what follows, we will briefly review the basic concepts
underlying data visualization and visual data mining.
Data Visualization
As opposed to textual or verbal communication of information,
data visualization provides a graphical representation
of data, documents, and structures, which turns out to be
useful for various purposes. Data visualization provides an
overview of complex and large data sets, shows a summary
of the data, and helps humans in the identification of
possible patterns and structures in the data. Thus, the goal
of data visualization is to simplify the representation of a
given data set, minimizing the loss of information [13], [14].
Visualization methods can be either geometric or symbolic.
In a geometric visualization, data are represented by
using lines, surfaces, or volumes and are usually obtained
from a physical model or as a result of a simulation or a
generic computation. Symbolic visualization represents
non-numeric data using pixels, icons, arrays, or graphs.
A general classification of visualization methods is
presented in [15] and [16]. It constructs a 3D visualization
space by classifying the visualization methods according to
three orthogonal criteria, the data type, the type of the
visualization technique, and the interaction methods, as
shown in Fig. 1.
Data Collection
The proposed approach aims to gather data concerning the
learner’s activities during online tests. As said above, a wellknown
strategy to collect data concerning learner’s activities
is the think-out-loud method [1], [2], [3], [4], [5]. However, we
have already mentioned the numerous drawbacks of this
method. In fact, it is quite invasive, which can significantly
influence user behavior. Further, it is difficult to use in
practice, since it requires considerable effort in terms of staff
personnel to analyze tape-recorded data.
One way of collecting data with a less invasive method
is to track the eyes’ movements. However, this would
require the use of expensive tools, which becomes even
worse if large-scale simultaneous experiments need to be
conducted, like it happens in the e-learning field. Nevertheless,
it has been shown that similar results can also be
inferred by tracking mouse movements. In particular, an
important experiment has revealed that there is a significant
correlation between the eyes’ and the mouse’s
movements [34]. This means that tracking the trajectory
drawn by the mouse pointer could be useful for deriving
the probable trajectory of the user’s eyes, which, in our
case, might reveal the strategy the learner is using to
answer multiple-choice questions.
CONCLUSION
We have presented an approach and a system to let tutors
monitor learners’ strategies during online tests. The approach
exploits data visualization to draw the data characterizing the
learner’s test strategy, in order to trigger the tutor’s attention
and to let him/her discover previously unknown behavioral
patterns of the learners and conceptual relationships among
test items. In this way, the tutor is provided with a powerful
tool that lets him/her review the whole assessment process
and evaluate possible improvements.
We have extensively used the implemented system
experimentally to evaluate online test strategies in the
courses of our faculty, in order to assess the whole approach.
This lets us discover several relevant patterns regarding the
test quality, the characteristics of used strategies, and the
impact on the final score.