Microsoft Interns Create Ultimate Photo-Tagging Spy App

TagSense, a prototype app designed by two Microsoft interns, can automatically tag a picture with a person's name, physical activities, facial expression, and exact physical location--all without human input.

A new, creepily awesome Android application developed by two Microsoft interns turns Android smartphone cameras into full-on spy machines. The app, called TagSense, relies on smartphone sensors to automatically tag photographs with the identities and activities of whoever's in them. No human input is required.
Although it also makes use of facial recognition technology, TagSense mainly relies on
the myriad peripherals integrated into the average smartphone. Android phones' built-in accelerometers are used to determine the exact physical activity a picture subject is involved in, while light sensors in the camera are used to determine weather conditions outside in conjunction with location data obtained via GPS. The phones' microphones can then determine whether the picture subject is talking, laughing, crying, or silent. All this information is then tagged onto the photos.
But the kicker for TagSense is that it is designed to tap into information stored on other, nearby smartphones. The application will be able to interact with data stored on adjacent phones that are also using the app, enabling collaborative tagging and verification of photo subjects' identities. The application's creators claim that there will be a strict opt-in requirement for use of that particular feature.
According to co-creator Chaun Qin, “phones have many different kinds of sensors that you can take advantage of. [...] They collect diverse information like motion, orientation, location, sound and light. By putting all that information together, you can sense the setting of a photograph and describe its attributes.”
Both Qin and co-creator Xuan Bao are currently summer interns for Microsoft Research. Bao and Qin are working on computer science PhDs at the University of South Carolina and Duke University respectively. They both collaborated on the project with Romit Roy Choudhury, a professor at Duke University.
Qin and Bao's application also pioneers technology that could be of interest to everyone from tech firms to the media to the military. An excerpt from their technical paper gives an idea of the possibilities:

One may imagine improved image search in the Internet, or even within one’s own computer--Bob may query his personal photo collection for all pictures of Alice and Eve playing together in the snow. Another application may tag videos with important event/activity markers; a user of this application may be able to move the video slider to the exact time-point where President Obama actually walks up to the podium, or starts speaking. Today, such functionalities may be available in select images and videos, where some humans have painstakingly tagged them. TagSense aims to automate this process via sensor-assisted tagging.

 More on Fastcompany.com