25-08-2014, 01:10 PM
Skinput
[attachment=66987]
ABSTRACT
We present Skinput, a technology that appropriates the human body for acoustic transmission, allowing the skin to be
used as an input surface. In particular, we resolve the location of finger taps on the arm and hand by analyzing mechanical vibrations that propagate through the body. We
collect these signals using a novel array of sensors worn as
an armband. This approach provides an always available,
naturally portable, and on-body finger input system. We
assess the capabilities, accuracy and limitations of our technique through a two-part, twenty-participant user study. To
further illustrate the utility of our approach, we conclude
with several proof-of-concept applications we developed.
INTRODUCTION
Devices with significant computational power and capabili
ties can now be easily carried on our bodies. However, their
small size typically leads to limited interaction space (e.g.,
diminutive screens, buttons, and jog wheels) and consequently diminishes their usability and functionality. Since
we cannot simply make buttons and screens larger without
losing the primary benefit of small size, we consider alternative approaches that enhance interactions with small mobile system.
RELATED WORK
Always-Available Input
The primary goal of Skinput is to provide an alwaysavailable mobile input system – that is, an input system that
does not require a user to carry or pick up a device. A number of alternative approaches have been proposed that operate in this space. Techniques based on computer vision are
popular (e.g. [3,26,27], see [7] for a recent survey). These,
however, are computationally expensive and error prone in
mobile scenarios (where, e.g., non-input optical flow is
prevalent). Speech input (e.g. [13,15]) is a logical choice
for always-available input, but is limited in its precision in
unpredictable acoustic environments, and suffers from privacy and scalability issues in shared environments.
Other approaches have taken the form of wearable computing. This typically involves a physical input device built in
a form considered to be part of one’s clothing. For example,
glove-based input systems (see [25] for a review) allow
users to retain most of their natural hand movements, but
are cumbersome, uncomfortable, and disruptive to tactile
sensation. Post and Orth [22] present a “smart fabric” systaking this approach to always-available input necessitates
embedding technology in all clothing, which would be prohibitively complex and expensive
Bio-Sensing
Skinput leverages the natural acoustic conduction properties
of the human body to provide an input system, and is thus
related to previous work in the use of biological signals for
computer input. Signals traditionally used for diagnostic
medicine, such as heart rate and skin resistance, have been
appropriated for assessing a user’s emotional state (e.g.
[16,17,20]).
Acoustic Input
Our approach is also inspired by systems that leverage
acoustic transmission through (non-body) input surfaces.
Paradiso et al. [21] measured the arrival time of a sound at
multiple sensors to locate hand taps on a glass window.
Ishii et al. [12] use a similar approach to localize a ball hiting a table, for computer augmentation of a real-world
game. Both of these systems use acoustic time-of-flight for
localization, which we explored, but found to be insuft ciently robust on the human body, leading to the finger printing approach described in this paper.
SKINPUT
To expand the range of sensing modalities for always available input systems, we introduce Skinput, a novel input
technique that allows the skin to be used as a finger input
surface. In our prototype system, we choose to focus on the
arm (although the technique could be applied elsewhere).
This is an attractive area to appropriate as it provides considerable surface area for interaction, including a contiguous
and flat area for projection (discussed subsequently).
Bio-Acoustics
When a finger taps the skin, several distinct forms of acoustic energy are produced. Some energy is radiated into the
air as sound waves; this energy is not captured by the Skinput system. Among the acoustic energy transmitted through
the arm, the most readily visible are transverse waves,
created by the displacement of the skin from a finger impact
(Figure 2). When shot with a high-speed camera, these appear as ripples, which propagate outward from the point of
contact (see video). The amplitude of these ripples is correlated to both the tapping force and to the volume and compliance of soft tissues under the impact area. In general,
tapping on soft regions of the arm creates higher amplitude
transverse waves than tapping on boney areas (e.g., wrist,
palm, fingers), which have negligible compliance.
Armband Prototype
Our final prototype, shown in Figures 1 and 5, features two
arrays of five sensing elements, incorporated into an arm band form factor. The decision to have two sensor packages
was motivated by our focus on the arm for input. In particuhoped to collect acoustic information from the fleshy bicep
area in addition to the firmer area on the underside of the
arm, with better acoustic coupling to the Humerus, the main
bone that runs from shoulder to elbow. When the sensor
was placed below the elbow, on the forearm, one package
was located near the Radius, the bone that runs from the
lateral side of the elbow to the thumb side of the wrist, and
the other near the Ulna, which runs parallel to this on the
medial side of the arm closest to the body. Each location
thus provided slightly different acoustic coverage and in formation, helpful in disambiguating input location.
Based on pilot data collection, we selected a different set of
resonant frequencies for each sensor package (Table 1).
Processing
In our prototype system, we employ a Mackie Onyx 1200F
audio interface to digitally capture data from the ten sensors
(http://mackie.com). This was connected via Firewire to a
conventional desktop computer, where a thin client written
in C interfaced with the device using the Audio Stream In put/Output (ASIO) protocol.
Each channel was sampled at 5.5kHz, a sampling rate that
would be considered too low for speech or environmental
audio, but was able to represent the relevant spectrum of
frequencies transmitted through the arm.
EXPERIMENT
Participants
To evaluate the performance of our system, we recruited 13
participants (7 female) from the Greater Seattle area. These
participants represented a diverse cross-section of potential
ages and body types. Ages ranged from 20 to 56 (mean
38.3), and computed body mass indexes (BMIs) ranged
from 20.5 (normal) to 31.9 (obese).
Experimental Conditions
We selected three input groupings from the multitude of
possible location combinations to test. We believe that
these groupings, illustrated in Figure 7, are of particular
interest with respect to interface design, and at the same
time, push the limits of our sensing capability. From these
three groupings, we derived five different experimental
conditions, described below.
Design and Setup
We employed a within-subjects design, with each participant performing tasks in each of the five conditions in randomized order: five fingers with sensors below elbow; five
points on the whole arm with the sensors above the elbow;
the same points with sensors below the elbow, both sighted
and blind; and ten marked points on the forearm with the
sensors above the elbow.
Procedure
For each condition, the experimenter walked through the
input locations to be tested and demonstrated finger taps on
each. Participants practiced duplicating these motions for
approximately one minute with each gesture set. This constituted one training round. In total, three
rounds of training data were collected per input location set
(30 examples per location, 150 data points total)
RESULTS
In this section, we report on the classification accuracies for
the test phases in the five different conditions. Overall,
classification rates were high, with an average accuracy
Forearm
Classification accuracy for the ten-location forearm condi
across conditions of 87.6%. Additionally, we present pretion stood at 81.5% (SD=10.5%, chance=10%), a surprsingly strong result for an input set we devised to push our
system’s sensing limit (K=0.72, considered very strong).
Following the experiment, we considered different ways to
improve accuracy by collapsing the ten locations into larger
input groupings. The goal of this exercise was to explore
the tradeoff between classification accuracy and number of
input locations on the forearm, which represents a particularly valuable input surface for application designers. We
grouped targets into sets based on what we believed to be
logical spatial groupings (Figure 9, A-E and G). In addition
to exploring classification accuracies for layouts that we
considered to be intuitive, we also performed an exhaustive
search (programmatically) over all possible groupings. For
most location counts, this search confirmed that our intutive groupings were optimal; however, this search revealed
one plausible, although irregular, layout with high accuracy
at six input locations (Figure 9, F).
SUPPLEMENTAL EXPERIMENTS
We conducted a series of smaller, targeted experiments to
explore the feasibility of our approach for other applicaformance of the system while users walked and jogged, we
recruited one male (age 23) and one female (age 26) for a
single-purpose experiment. For the rest of the experiments,
we recruited seven new participants (3 female, mean age
26.9) from within our institution. In all cases, the sensor
armband was placed just below the elbow. Similar to the
previous experiment, each additional experiment consisted
of a training phase, where participants provided between 10
and 20 examples for each input type, and a testing phase, in
which participants were prompted to provide a particular
input (ten times per input type).
Walking and Jogging
As discussed previously, acoustically-driven input techniques are often sensitive to environmental noise. In regard
to bio-acoustic sensing, with sensors coupled to the body,
noise created during other motions is particularly troublesome, and walking and jogging represent perhaps the most
common types of whole-body motion. This experiment
explored the accuracy of our system in these scenarios
Single-Handed Gestures
In the experiments discussed thus far, we considered only
bimanual gestures, where the sensor-free arm, and in paticular the fingers, are used to provide input. However,
there are a range of gestures that can be performed with just
the fingers of one hand. This was the focus of [2], although
this work did not evaluate classification accuracy.
Surface and Object Recognition
During piloting, it became apparent that our system had
some ability to identify the type of material on which the
user was operating. Using a similar setup to the main expriment, we asked participants to tap their index finger
against 1) a finger on their other hand, 2) a paper pad approximately 80 pages thick, and 3) an LCD screen. Results
show that we can identify the contacted object with about
87.1% (SD=8.3%, chance=33%) accuracy. This capability
was never considered when designing the system, so superior acoustic features may exist. Even as accuracy stands
now, there are several interesting applications that could
take advantage of this functionality, including workstations
or devices composed of different interactive surfaces, or
recognition of different objects grasped in the environment.
Identification of Finger Tap Type
Users can “tap” surfaces with their fingers in several distinct ways. For example, one can use the tip of their finger
(potentially even their finger nail) or the pad (flat, bottom)
of their finger. The former tends to be quite boney, while
the latter more fleshy. It is also possible to use the knuckles
(both major and minor metacarpophalangeal joints).
To evaluate our approach’s ability to distinguish these input
types, we had participants tap on a table situated in front of
them in three ways (ten times each): finger tip, finger pad,
and major knuckle. A classifier trained on this data yielded
an average accuracy of 89.5% (SD=4.7%, chance=33%)
during the testing period.
EXAMPLE INTERFACES AND INTERACTIONS
We conceived and built several prototype interfaces that
demonstrate our ability to appropriate the human body, in
this case the arm, and use it as an interactive surface. These
interfaces can be seen in Figure 11, as well as in the accompanying video.
While the bio-acoustic input modality is not strictly tethered
to a particular output modality, we believe the sensor form
factors we explored could be readily coupled with visual
output provided by an integrated pico-projector. There are
two nice properties of wearing such a projection device on
the arm that permit us to sidestep many calibration issues.
First, the arm is a relatively rigid structure - the projector,
when attached appropriately, will naturally track with the
arm (see video). Second, since we have fine-grained control
of the arm, making minute adjustments to align the projected image with the arm is trivial (e.g., projected horizontal stripes for alignment with the wrist and elbow).
To illustrate the utility of coupling projection and finger
input on the body (as researchers have proposed to do with
projection and computer vision-based techniques [19]), we
developed three proof-of-concept projected interfaces built
on top of our system’s live input classification. In the first
interface, we project a series of buttons onto the forearm, on
which a user can finger tap to navigate a hierarchical menu
(Figure 11, left)
[attachment=66987]
ABSTRACT
We present Skinput, a technology that appropriates the human body for acoustic transmission, allowing the skin to be
used as an input surface. In particular, we resolve the location of finger taps on the arm and hand by analyzing mechanical vibrations that propagate through the body. We
collect these signals using a novel array of sensors worn as
an armband. This approach provides an always available,
naturally portable, and on-body finger input system. We
assess the capabilities, accuracy and limitations of our technique through a two-part, twenty-participant user study. To
further illustrate the utility of our approach, we conclude
with several proof-of-concept applications we developed.
INTRODUCTION
Devices with significant computational power and capabili
ties can now be easily carried on our bodies. However, their
small size typically leads to limited interaction space (e.g.,
diminutive screens, buttons, and jog wheels) and consequently diminishes their usability and functionality. Since
we cannot simply make buttons and screens larger without
losing the primary benefit of small size, we consider alternative approaches that enhance interactions with small mobile system.
RELATED WORK
Always-Available Input
The primary goal of Skinput is to provide an alwaysavailable mobile input system – that is, an input system that
does not require a user to carry or pick up a device. A number of alternative approaches have been proposed that operate in this space. Techniques based on computer vision are
popular (e.g. [3,26,27], see [7] for a recent survey). These,
however, are computationally expensive and error prone in
mobile scenarios (where, e.g., non-input optical flow is
prevalent). Speech input (e.g. [13,15]) is a logical choice
for always-available input, but is limited in its precision in
unpredictable acoustic environments, and suffers from privacy and scalability issues in shared environments.
Other approaches have taken the form of wearable computing. This typically involves a physical input device built in
a form considered to be part of one’s clothing. For example,
glove-based input systems (see [25] for a review) allow
users to retain most of their natural hand movements, but
are cumbersome, uncomfortable, and disruptive to tactile
sensation. Post and Orth [22] present a “smart fabric” systaking this approach to always-available input necessitates
embedding technology in all clothing, which would be prohibitively complex and expensive
Bio-Sensing
Skinput leverages the natural acoustic conduction properties
of the human body to provide an input system, and is thus
related to previous work in the use of biological signals for
computer input. Signals traditionally used for diagnostic
medicine, such as heart rate and skin resistance, have been
appropriated for assessing a user’s emotional state (e.g.
[16,17,20]).
Acoustic Input
Our approach is also inspired by systems that leverage
acoustic transmission through (non-body) input surfaces.
Paradiso et al. [21] measured the arrival time of a sound at
multiple sensors to locate hand taps on a glass window.
Ishii et al. [12] use a similar approach to localize a ball hiting a table, for computer augmentation of a real-world
game. Both of these systems use acoustic time-of-flight for
localization, which we explored, but found to be insuft ciently robust on the human body, leading to the finger printing approach described in this paper.
SKINPUT
To expand the range of sensing modalities for always available input systems, we introduce Skinput, a novel input
technique that allows the skin to be used as a finger input
surface. In our prototype system, we choose to focus on the
arm (although the technique could be applied elsewhere).
This is an attractive area to appropriate as it provides considerable surface area for interaction, including a contiguous
and flat area for projection (discussed subsequently).
Bio-Acoustics
When a finger taps the skin, several distinct forms of acoustic energy are produced. Some energy is radiated into the
air as sound waves; this energy is not captured by the Skinput system. Among the acoustic energy transmitted through
the arm, the most readily visible are transverse waves,
created by the displacement of the skin from a finger impact
(Figure 2). When shot with a high-speed camera, these appear as ripples, which propagate outward from the point of
contact (see video). The amplitude of these ripples is correlated to both the tapping force and to the volume and compliance of soft tissues under the impact area. In general,
tapping on soft regions of the arm creates higher amplitude
transverse waves than tapping on boney areas (e.g., wrist,
palm, fingers), which have negligible compliance.
Armband Prototype
Our final prototype, shown in Figures 1 and 5, features two
arrays of five sensing elements, incorporated into an arm band form factor. The decision to have two sensor packages
was motivated by our focus on the arm for input. In particuhoped to collect acoustic information from the fleshy bicep
area in addition to the firmer area on the underside of the
arm, with better acoustic coupling to the Humerus, the main
bone that runs from shoulder to elbow. When the sensor
was placed below the elbow, on the forearm, one package
was located near the Radius, the bone that runs from the
lateral side of the elbow to the thumb side of the wrist, and
the other near the Ulna, which runs parallel to this on the
medial side of the arm closest to the body. Each location
thus provided slightly different acoustic coverage and in formation, helpful in disambiguating input location.
Based on pilot data collection, we selected a different set of
resonant frequencies for each sensor package (Table 1).
Processing
In our prototype system, we employ a Mackie Onyx 1200F
audio interface to digitally capture data from the ten sensors
(http://mackie.com). This was connected via Firewire to a
conventional desktop computer, where a thin client written
in C interfaced with the device using the Audio Stream In put/Output (ASIO) protocol.
Each channel was sampled at 5.5kHz, a sampling rate that
would be considered too low for speech or environmental
audio, but was able to represent the relevant spectrum of
frequencies transmitted through the arm.
EXPERIMENT
Participants
To evaluate the performance of our system, we recruited 13
participants (7 female) from the Greater Seattle area. These
participants represented a diverse cross-section of potential
ages and body types. Ages ranged from 20 to 56 (mean
38.3), and computed body mass indexes (BMIs) ranged
from 20.5 (normal) to 31.9 (obese).
Experimental Conditions
We selected three input groupings from the multitude of
possible location combinations to test. We believe that
these groupings, illustrated in Figure 7, are of particular
interest with respect to interface design, and at the same
time, push the limits of our sensing capability. From these
three groupings, we derived five different experimental
conditions, described below.
Design and Setup
We employed a within-subjects design, with each participant performing tasks in each of the five conditions in randomized order: five fingers with sensors below elbow; five
points on the whole arm with the sensors above the elbow;
the same points with sensors below the elbow, both sighted
and blind; and ten marked points on the forearm with the
sensors above the elbow.
Procedure
For each condition, the experimenter walked through the
input locations to be tested and demonstrated finger taps on
each. Participants practiced duplicating these motions for
approximately one minute with each gesture set. This constituted one training round. In total, three
rounds of training data were collected per input location set
(30 examples per location, 150 data points total)
RESULTS
In this section, we report on the classification accuracies for
the test phases in the five different conditions. Overall,
classification rates were high, with an average accuracy
Forearm
Classification accuracy for the ten-location forearm condi
across conditions of 87.6%. Additionally, we present pretion stood at 81.5% (SD=10.5%, chance=10%), a surprsingly strong result for an input set we devised to push our
system’s sensing limit (K=0.72, considered very strong).
Following the experiment, we considered different ways to
improve accuracy by collapsing the ten locations into larger
input groupings. The goal of this exercise was to explore
the tradeoff between classification accuracy and number of
input locations on the forearm, which represents a particularly valuable input surface for application designers. We
grouped targets into sets based on what we believed to be
logical spatial groupings (Figure 9, A-E and G). In addition
to exploring classification accuracies for layouts that we
considered to be intuitive, we also performed an exhaustive
search (programmatically) over all possible groupings. For
most location counts, this search confirmed that our intutive groupings were optimal; however, this search revealed
one plausible, although irregular, layout with high accuracy
at six input locations (Figure 9, F).
SUPPLEMENTAL EXPERIMENTS
We conducted a series of smaller, targeted experiments to
explore the feasibility of our approach for other applicaformance of the system while users walked and jogged, we
recruited one male (age 23) and one female (age 26) for a
single-purpose experiment. For the rest of the experiments,
we recruited seven new participants (3 female, mean age
26.9) from within our institution. In all cases, the sensor
armband was placed just below the elbow. Similar to the
previous experiment, each additional experiment consisted
of a training phase, where participants provided between 10
and 20 examples for each input type, and a testing phase, in
which participants were prompted to provide a particular
input (ten times per input type).
Walking and Jogging
As discussed previously, acoustically-driven input techniques are often sensitive to environmental noise. In regard
to bio-acoustic sensing, with sensors coupled to the body,
noise created during other motions is particularly troublesome, and walking and jogging represent perhaps the most
common types of whole-body motion. This experiment
explored the accuracy of our system in these scenarios
Single-Handed Gestures
In the experiments discussed thus far, we considered only
bimanual gestures, where the sensor-free arm, and in paticular the fingers, are used to provide input. However,
there are a range of gestures that can be performed with just
the fingers of one hand. This was the focus of [2], although
this work did not evaluate classification accuracy.
Surface and Object Recognition
During piloting, it became apparent that our system had
some ability to identify the type of material on which the
user was operating. Using a similar setup to the main expriment, we asked participants to tap their index finger
against 1) a finger on their other hand, 2) a paper pad approximately 80 pages thick, and 3) an LCD screen. Results
show that we can identify the contacted object with about
87.1% (SD=8.3%, chance=33%) accuracy. This capability
was never considered when designing the system, so superior acoustic features may exist. Even as accuracy stands
now, there are several interesting applications that could
take advantage of this functionality, including workstations
or devices composed of different interactive surfaces, or
recognition of different objects grasped in the environment.
Identification of Finger Tap Type
Users can “tap” surfaces with their fingers in several distinct ways. For example, one can use the tip of their finger
(potentially even their finger nail) or the pad (flat, bottom)
of their finger. The former tends to be quite boney, while
the latter more fleshy. It is also possible to use the knuckles
(both major and minor metacarpophalangeal joints).
To evaluate our approach’s ability to distinguish these input
types, we had participants tap on a table situated in front of
them in three ways (ten times each): finger tip, finger pad,
and major knuckle. A classifier trained on this data yielded
an average accuracy of 89.5% (SD=4.7%, chance=33%)
during the testing period.
EXAMPLE INTERFACES AND INTERACTIONS
We conceived and built several prototype interfaces that
demonstrate our ability to appropriate the human body, in
this case the arm, and use it as an interactive surface. These
interfaces can be seen in Figure 11, as well as in the accompanying video.
While the bio-acoustic input modality is not strictly tethered
to a particular output modality, we believe the sensor form
factors we explored could be readily coupled with visual
output provided by an integrated pico-projector. There are
two nice properties of wearing such a projection device on
the arm that permit us to sidestep many calibration issues.
First, the arm is a relatively rigid structure - the projector,
when attached appropriately, will naturally track with the
arm (see video). Second, since we have fine-grained control
of the arm, making minute adjustments to align the projected image with the arm is trivial (e.g., projected horizontal stripes for alignment with the wrist and elbow).
To illustrate the utility of coupling projection and finger
input on the body (as researchers have proposed to do with
projection and computer vision-based techniques [19]), we
developed three proof-of-concept projected interfaces built
on top of our system’s live input classification. In the first
interface, we project a series of buttons onto the forearm, on
which a user can finger tap to navigate a hierarchical menu
(Figure 11, left)