03-09-2014, 11:09 AM
Surface Computing
Surface Computing.doc (Size: 369 KB / Downloads: 12)
INTRODUCTION
Over the past couple of years, a new class of interactive device has begun to emerge, what can best be described as “surface computing”. Two examples are illustrated in this report. They are- Surface Table top and Perceptive Pixel.
The Surface table top typically incorporates a rear-projection display coupled with an optical system to capture touch points by detecting shadows from below. Different approaches to doing the detection have been used, but most employ some form of IR illumination coupled with IR cameras. With today’s camera and signal-processing capability, reliable responsive and accurate multi-touch capabilities can be achieved.
The multitouch pioneer and his company, Perceptive Pixel, have devoted the better part of two years to building an entirely new multitouch framework from the ground up. Instead of simply mapping multitouch technology to familiar interfaces and devices, Han's goal is far more sweeping: To use the technology as a foundation for an entirely new operating system.
Because they are new to most, the tendency in seeing these systems is to assume that they are all more-or-less alike. Well, in a way that is true. But on the other hand, that is perhaps no more so than to say that all ICs are more-or-less alike, since they are black plastic things with feet like centipedes which contain a bunch of transistors and other stuff. In short, the more that you know, the more you can differentiate. But even looking at the two systems in the photo, there is evidence of really significant difference.
The really significant difference is that one is vertical and the other is horizontal. Why is this significant? Well, this is one of those questions perhaps best answered by a child in kindergarten. They will tell you that if you put a glass of water on the vertical one, it will fall to the floor, leading to a bout of sitting in the corner. On the other hand, it is perfectly safe to put things on a table. They will stay there.
WHAT IS SURFACE COMPUTING?
Surface computing is a new way of working with computers that moves beyond the traditional mouse-and-keyboard experience. It is a natural user interface that allows people to interact with digital content the same way they have interacted with everyday items such as photos, paintbrushes and music their entire life: with their hands, with gestures and by putting real-world objects on the surface. Surface computing opens up a whole new category of products for users to interact with.
Surface computing is a completely intuitive and liberating way to interact with digital content. It blurs the lines between the physical and virtual worlds. By using your hands or placing other unique everyday objects on the surface – such as an item you’re going to purchase at a retail store or a paint brush – you can interact with, share and collaborate like you’ve never done before. Imagine you’re out at a restaurant with friends and you each place your beverage on the table – and all kinds of information appears by your glass, such as wine pairings with a restaurant’s menu. Then, with the flick of your finger, you order dessert and split the bill. We really see this as broadening content opportunities and delivery systems.
Surface computing is a powerful movement. In fact, it’s as significant as the move from DOS [Disk Operating System] to GUI [Graphic User Interface]. Our research shows that many people are intimidated and isolated by today’s technology. Many features available in mobile phones, PCs and other electronic devices like digital cameras aren’t even used because the technology is intimidating. Surface computing breaks down those traditional barriers to technology so that people can interact with all kinds of digital content in a more intuitive, engaging and efficient manner. It’s about technology adapting to the user, rather than the user adapting to the technology. Bringing this kind of natural user interface innovation to the computing space is what Surface Computing is all about
HISTORY OF SURFACE COMPUTING
Surface computing is a major advancement that moves beyond the traditional user interface to a more natural way of interacting with digital content. Microsoft Surface™, Microsoft Corp.’s first commercially available surface computer, breaks down the traditional barriers between people and technology to provide effortless interaction with all forms of digital content through natural gestures, touch and physical objects instead of a mouse and keyboard. The people will be able to interact with Surface in select restaurants, hotels, retail establishments and public entertainment.
In 2001, Stevie Bathiche of Microsoft Hardware and Andy Wilson of Microsoft Research began working together on various projects that took advantage of their complementary expertise in the areas of hardware and software. In one of their regular brainstorm sessions, they started talking about an idea for an interactive table that could understand the manipulation of physical pieces. Although there were related efforts happening in academia, Bathiche and Wilson saw the need for a product where the interaction was richer and more intuitive, and at the same time practical for everyone to use. This conversation was the beginning of an idea that would later result in the development of Surface, and over the course of the following year, various people at Microsoft involved in developing new product concepts, including the gaming-specific PlayTable, continued to think through the possibilities and feasibility of the project. Then in October 2001 a virtual team was formed to fully pursue bringing the idea to the next stage of development; Bathiche and Wilson were key members of the team
Hardware Design
By late 2004, the software development platform of Surface was well-established and attention turned to the form factor. A number of different experimental prototypes were built including “the tub” model, which was encased in a rounded plastic shell, a desk-height model with a square top and cloth-covered sides, and even a bar-height model that could be used while standing. After extensive testing and user research, the final hardware design (seen today) was finalized in 2005. Also in 2005, Wilson and Bathiche introduced the concept of surface computing in a paper for Gates’ twice-yearly “Think Week,” a time Gates takes to evaluate new ideas and technologies for the company
From Prototype to Product
The next phase of the development of Surface focused on continuing the journey from concept to product. Although much of what would later ship as Surface was determined, there was significant work to be done to develop a market-ready product that could be scaled to mass production.
In early 2006, Pete Thompson joined the group as general manager, tasked with driving end-to-end business and growing development and marketing. Under his leadership, the group has grown to more than 100 employees. Today Surface has become the market-ready product once only envisioned by the group, a 30-inch display in a table-like form factor that’s easy for individuals or small groups to use collaboratively. The sleek, translucent surface lets people engage with Surface using touch, natural hand gestures and physical objects placed on the surface. Years in the making, Microsoft Surface is now poised to transform the way people shop, dine, entertain and live. This is a radically different user-interface experience than anything and it’s really a testament to the innovation that comes from marrying brilliance and creativity
TECHNOLOGY BEHIND SURFACE COMPUTING
Microsoft Surface uses cameras to sense objects, hand gestures and touch. This user input is then processed and displayed using rear projection. Specifically, Microsoft Surface uses a rear projection system which displays an image onto the underside of a thin diffuser. Objects such as fingers are visible through the diffuser by series of infrared–sensitive cameras, positioned underneath the display. An image processing system processes the camera images to detect fingers, custom tags and other objects such as paint brushes when touching the display. The objects recognized with this system are reported to applications running in the computer so that they can react to object shapes, 2D tags, movement and touch.
One of the key components of surface computing is a "multitouch" screen. It is an idea that has been floating around the research community since the 1980s and is swiftly becoming a hip new product interface — Apple's new iPhone has multitouch scrolling and picture manipulation. Multitouch devices accept input from multiple fingers and multiple users simultaneously, allowing for complex gestures, including grabbing, stretching, swiveling and sliding virtual objects across the table. And the Surface has the added advantage of a horizontal screen, so several people can gather around and use it together. Its interface is the exact opposite of the personal computer: cooperative, hands-on, and designed for public spaces.
System software
Microsoft Surface works much like another Microsoft product, Media Center, in that the main application runs on top of Windows and takes over the whole screen. Like Media Center, it is designed to be difficult to exit the application without using a mouse or keyboard. I asked if the Surface team considered allowing the user to drop into Windows mode while retaining the touch functionality, but they felt that the product worked better if it stayed in this mode.
The various demonstration programs are accessed from a main menu, which scrolls left and right in an endless loop. The user moves the selection by swiping back and forth and selects an application with a single tap. This works reasonably well and feels quite natural. When an application is selected, a swirly purple ring appears in the center of the screen to indicate that the program is loading.
There were eight different programs available: Water, Video Puzzle, Paint, Music, Photos, Casino, a T-Mobile demonstration app, and Dining. Much of the software was written using Microsoft's WPF (Windows Presentation Foundation), though the XNA development toolkit, a framework originally created for writing PC and Xbox 360 games, is also supported. XNA allows programmers to use managed code written in C# to manipulate various DirectX features; managed code frees the programmer from worrying about handling memory, allocating and discarding memory automatically. This approach has allowed Microsoft and its partners to write impressive-looking demonstration programs for Surface more quickly than would otherwise be possible
Video Puzzle
Video Puzzle showcases the power of the little identification tags mentioned above. The tags consist of a pattern of variously-sized dots; Keam mentioned that the dots currently represent an 8-bit code (256 permutations) but that 128-bit tags were in the works. The neat thing about the tags is that they can be very nearly transparent and the system will still pick them up. Not only can the tags transmit numerical information, but the geometrical arrangement of the dots means that Surface can also tell, to a high degree of accuracy, how much the tag (and therefore the object) has rotated.
In Video Puzzle, these virtually invisible tags are placed upon small squares of glass. When the pieces of glass are put on the table, the screen starts playing video clips underneath each one. Because the video moves whenever you move the squares, it creates the illusion that the glass itself is displaying the video, which looks very futuristic. As you move the squares around, you quickly realize that the video clips are all pieces of a larger video. Flipping the glass squares over inverts the video playing underneath, making completing the puzzle even more of a challenge.
When you complete the puzzle correctly, the system senses the achievement, congratulates you, and shows you the time taken to finish. According to Mark Bolger, the current record for finishing when the pieces are fully randomized is 1 minute and 53 seconds. On my first attempt, I finished in just over 2 minutes, but the squares were all right side up to begin with (Microsoft is nice to journalists, it seems).
Music
The Music application works like a virtual jukebox, displaying music arranged by album and allowing the user to flip over albums, select songs, and drag them to the "Now Playing" section. The album browser works a bit like Apple’s Cover Flow, although many albums are visible at once without scrolling.
In addition to playing music that is already stored on the unit's hard drive, Music can also transfer songs from portable music players. Mark Bolger demonstrated this by placing two Zunes on top of the Surface and using the wireless connection to drag and drop songs between the units, the song list, and the Now Playing section. I mentioned to the team that this was the first time I had ever seen even one Zune "in the wild," and they joked that Microsoft headquarters didn’t really count as being in the wild. Bolger noted that sharing songs in this manner would be "subject to DRM restrictions, of course."
Casino
The Casino application was developed in cooperation with Harrah's of Las Vegas and is a good example of how Surface can be used in a hospitality environment. The background image is a giant map of the hotel and casino, with all the attractions marked for further inspection. Hotel customers can place their card anywhere on the screen and reserve tickets to any of these shows. The background map can be easily scrolled with a brush of the hand, and zoomed in and out by performing the two-finger pinch.
SURFACE COMPUTING - IN FUTURE
Although surface computing is a new experience for consumers, over time Microsoft believes there will be a whole range of surface computing devices and the technology will become pervasive in people’s lives in a variety of environments.
As form factors continue to evolve, surface computing will be in any number of environments— schools, businesses, homes — and in any number of form factors — part of the countertop, the wall or the refrigerator.
CONCLUSION
Some people will look at Surface and claim that it does nothing that hasn't been tried before: computers with touch screens have been around for years and have already found niches in ATMs, ticket ordering machines, and restaurant point-of-sale devices.
This view largely misses the point of the product. Like most projects, Surface takes existing technology and presents it in a new way. It isn't simply a touch screen, but more of a touch-grab-move-slide-resize-and-place-objects-on-top-of-screen, and this opens up new possibilities that weren't there before.
Playing with the unit felt a bit like being in the movie Minority Report (in a good way), but it also felt like a more natural and enjoyable method of doing certain computing tasks. Sharing and looking at family photos, for example, is more fun on Surface than on any other device. The retail applications, particularly the dining application, show how businesses could use the technology to really stand out from competitors, though one wonders how diners will react when their table locks up and needs a reboot.
Many people who viewed the early Xerox PARC demonstrations of the GUI came out of that experience knowing that every computer would work that way someday, and they were right. Playing with Surface, one gets the sense that although not every computer will work like this someday, many of them will. More importantly, computers running Surface-like software will end up in places that never had computers before, and the potential applications are exciting. Imagine a multiplayer real-time strategy game where you and another human opponent can move units around as quickly as you can point to them or perhaps an educational environment, where university students could assemble and disassemble anything from molecules to skyscrapers quickly and easily.