Multitouch Interface Summer 2007 Project

During the summer of 2007, I worked with a very energetic undergraduate student, Casey Johnson, who was very interested in doing some sort of research work with me. At the same time, the Apple iPhone was coming out, which piqued my interest, as it implemented what’s known as a multi-touch interface. After some preliminary research into trying to understand how the iPhone worked, I came across the research of Jefferson Han and the multi-touch interface he designed using Frustrated Total Internal Reflectance (FTIR). So, eager to understand first-hand how a multi-touch interface worked, I created this project for Casey to work on.

We met weekly throughout the entire 2007 summer semester, where we discussed ideas and his progress so far. I reviewed with Casey some concepts in linear algebra and optics, as well as assisted him whenever he encountered problems with hardware and software, guiding him to online sources that addressed our problems whenever it was necessary. At the end of the summer semester, he was able to successfully build a proof-of-concept multi-touch interface. Casey created the frame embedded with IR LEDs, and hacked a webcam in order to see in the infrared range. He also was able to run the touchlib programming library, and run the default programs.

I’ve decided to take some of the lessons learned over the summer to create my own fully-functional multi-touch interface as my final project for my HCI course.

Below are some of the images of the interface in action:

This is Casey, holding with his hand the multi-touch screen.

This is the same board seen in the infrared range. Note how it appears illuminated on the edges, due to the presence of the LEDs.

This is an image of Casey’s hand as it made contact with the multi-touch screen. Note how his fingertips are brightly illuminated, as predicted due to FTIR.

These are some of the touchlib filtering programs in action. Note how the two topmost images Casey’s hand appears illuminated and how, by applying a simple threshold filter (along with some rectification), we are able to simply get the contact points of the fingertips. The image on the bottom right is the output of the threshold filter, while the image on the bottom right is the output of the rectifier.