01-09-2016, 12:37 PM
1452140984-PHONEPOINTPEN.docx (Size: 400.42 KB / Downloads: 5)
Imagine the following scenario. While driving to of¬fice, Leslie stops at a traffic light. As she mentally sifts through her tasks for the day, she remembers that she needs to call her friend, Jane, very soon. Since Leslie tends to forget her personal commitments, she decides to make a note of this task. Therefore, while keep¬ing her gaze on the traffic lights, she reaches for her phone from the pocket, and by holding it like a pen, she writes "JANE" in the air. She also gestures a check¬mark to email the written note to herself. She does not look at any of these hand-gestures she makes. Once in her office, she finds an email in her mailbox that reads "Phone Point Pen - JANE". Leslie calls Jane, talks to her, and deletes the email. The figure below shows the out¬put of writing Jane using PhonePoint Pen.
Fig 1.1hows the output of writing Jane using PhonePoint Pen.
The above is a fictional scenario, however, represen¬tative of a niche in pervasive computing applications. In particular, i believe that there is a class of appli¬cations that will benefit from a technology that can quickly and effortlessly "note down" short pieces of in¬formation. Although existing technologies have made important advances to meet the needs, the quality of user-experience can perhaps be improved. I discuss some avenues of improvement, and motivate the po¬tential of Phone Point Pens
Typing an SMS, while popular among the youth, has been unpopular among a moderate section of society. Studies report user dissatisfaction with mobile phone typing [1, 2, 3]. The major sources of discomfort arise from small key sizes, short inter-key spacing’s, and the need for multi-tapping in some phone keyboards. With increasingly smaller phones, keyboard sizes may de¬crease, exacerbating the problem of physical typing.
Even if future keyboards [4] improve the typing expe¬rience, some problems may still persist. While walking, or with one hand occupied, typing in information may be inconvenient. Using the mobile phone accelerometer to capture hand gestures, and carefully laying them out in text or image, can improve the user experience. The ability to write without having to look at the phone keypad may offer an added advantage.
One may argue that voice recorder applications on mo¬bile phones may be an easy way to input short pieces of information. However, searching and editing voice- recorded content is difficult (unless processed through a separate speech-to-text software). Further, browsing through multiple voice messages is time-consuming. Writing in air, and converting them to typed text, may alleviate these problems.
Current approaches are largely ad hoc. People use whatever is quickly reachable, including pen-and- paper, sticky notes, one's own palm, etc. None of these scale because they are not always handy, and more im¬portantly, not always connected to the Internet. Thus, hurriedly noted information gets scattered, making in¬formation organization and retrieval hard.
This paper proposes to use the in-built accelerometer in modern mobile phones as a quick and ubiquitous way of capturing (short) written information. The problem definition bears similarity to known problems in gesture recognition. However, as we will see later, recognizing actual alphabets in air (using the phone processor, a noisy accelerometer, and no software training), raises a number of new challenges. For instance, as a part of writing the alphabet "A" on paper, one must write "/\" first, lift and reposition the pen on the paper, and then write the "—". When writing in air, the phone cannot easily say which part of the
hand-movement is intended to be the "re-positioning" of the pen. The problem is further complicated by the inherent noise in mobile phone accelerometers, the user's involuntary wrist-rotation, and practical difficulties in deriving displacement from noisy acceleration.
The PhonePoint Pen addresses majority of these challenges by treat¬ing the accelerometer readings as a digital signal, and successively refining it through simple numerical and signal processing algorithms. The simplicity is impor¬tant to ensure that the operations can be performed on the phone processor. Once individual geometric move¬ments have been tracked, their sequence of occurrence is matched against a decision tree (a simple grammar). The outcome of the matching operation yields the English character.
The PhonePoint Pen is not yet like a true pen-in-the-air, and requires the user to get used to a few soft con¬straints. Users that do not rotate their wrists while moving, do not write too fast, and write 15-inch sized capital letters, achieve an average accuracy of 83%. WithEnglish alphabets. The geometric representations of the characters are quite legi¬ble, except in 23% cases. The performance degrades as these constraints get violated, such as with new users. However, after writing around 20 characters, most users observed greater than 70% accuracy. Surveys and verbal feedback from random student users, as well as from speech-impaired patients in Duke Hospital, were positive. The absence of visual feedback while writing did not appear to be a concern at all. While more research is certainly necessary, our current find¬ings give us confidence that the Phone Point Pen could become a publicly-usable technology in the near future.
The conception of the ideas and a preliminary design of the Phone Point Pen (P3) was published in MobiHeld 2009 [5], a workshop collocated with ACM Sigcomm.
Besides a mature design, full implementation, and a real-user based evaluation of the system, this paper adds a number of functional capabilities:
1. The workshop version was only capable of geo¬metric representations of characters; this paper al¬lows for actual character recognition leading to (editable/searchable) text.
2. The workshop version focused on identifying a single character, while this paper attempts to rec¬ognize transitions from one character to another, forming words.
3. The workshop version used a back-end server for processing; this paper is capable of on-phone analysis, and can display the results on the phone's screen with 2-3 seconds latency.
4. Finally, this paper adds a few miscellaneous fea¬tures such as character deletion, spaces between characters, digit recognition, and the ability to email with a gesture (a check-mark in the air).
The overall system is implemented on the platform of Nokia N95 phones using Python as the programming platform.