Beyond the Surface 1

This project outlines the progress of the development of several multi-modal, multi-user interfaces for use in urban planning and architectural visualizations. It will utilize as a central, interactive display the Microsoft Surface, which serves as a communal area where data can be shared among several users. We will also utilize mobile devices, such as smartphones and netbooks, as well as head-mounted displays (HMDs), to augment the user’s experience by providing information desired by the user and ensuring it is only visible by that user.

Computing displays and devices present an increasing diversity of form and deployment. Some are hand-held, such as smartphones, personal media players, UMPCs, and Tablet PCs. Others reside on the desk, and include PCs with single and multiple monitor configurations. Some cover tabletops, walls, or floors. Others are head-worn. How can these heterogeneous displays and devices complement each other to create an effective environment in which to work, play, and learn? We propose research into the design and development of hybrid user interfaces [Feiner and Shamash, 1991] that put the multiuser tabletop user interface afforded by Microsoft Surface into a rich context of additional displays and devices, ranging from hand-held, to head-worn, to stationary. The idea is to supplement the display and interaction volume established by and anchored on Surface with the capability to interact both above and around it.

Our goal in this proposal is to build on previous work to create novel software infrastructure, user interface techniques, and example applications that emphasize their usability through real-world scenarios. Together, these will support collaborative interaction inside a computational volume, within many users and their devices are located—displaying, using, and exchanging data. We propose to do this by using a combination of software infrastructure including Microsoft Surface, Windows 7, and Goblin XNA (our system for research in 3D user interfaces and games, built on top of XNA Game Studio 3.0 and distributed at http://goblinxna.codeplex.com).

For example, hand-held devices such as smartphones and UMPCs can be tracked on Surface to present additional information. These devices can also be tracked in 3D position and orientation above Surface using a range of possible approaches (e.g., a sub-Surface camera, as employed in Second Light; optical tracking by a camera in the hand-held device itself, viewing known material displayed on Surface; or the use of a variety of 3D tracking systems currently available in our lab). See-through head-worn displays can also be supported and tracked similarly, augmenting the 3D volume above Surface in stereo, customized to the information needs of the individual user. Additional stationary displays can increase the amount of shared space available. One domain that we propose to use as a test case is that of urban visualization, creating a vivid, manipulable representation of an urban environment that can be simultaneously explored and modified by multiple users. Here, we will leverage our ongoing work on situated visualization of the site of Columbia’s planned northern campus expansion in Manhattanville [White and Feiner, 2009].

Videos shot so far:

Beyond the Surface: Sneak preview from Nicolas Dedual on Vimeo.

UPDATE

These videos have been a long time coming. They are the latest videos shown in Microsoft Research Faculty Summit by my advisor, Prof. Steven Feiner

Beyond the Surface – Flatiron Building – External Camera Version from Nicolas Dedual on Vimeo.

Beyond the Surface – Flatiron building – CamStudio version from Nicolas Dedual on Vimeo.