Scientists at the MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) developed a “Computational Mirror” – a system that can reconstruct hidden videos solely from shadows and reflections outside the camera’s field of view.
The team built off of previous MIT research that used lasers to to understand information about hidden scenes. The lasers allowed the system to produce 3D images from reflections on floors, doors, and walls, which acted like “mirrors”. But this time, no lasers or special equipment were necessary: only a video camera and an algorithm.
PHOTO: http://compmirrors.csail.mit.edu/
How do Computational Mirrors work?
This computer vision system can reconstruct a motion taking place outside it’s field of view by observing changes in indirect illumination in it’s visible region. It relies on non-negative matrix factorization and randomly initialized convolutional neural networks to unscramble and make sense of these lighting cues.
“Mathematically, it’s like if I told you that I’m thinking of two secret numbers, and their product is 80. Can you guess what they are? Maybe 40 and 2? Or perhaps 371.8 and 0.2152? In our problem, we face a similar situation at every pixel”
Miika Aittala, lead researcher on “Computational Mirrors”
Potential benefits of the system
This new imaging system “Computational Mirrors” has several potential benefits, such as improving the safety of self-driving cars, by providing insights of what’s going on behind corners, or improving the ability of search-and-rescue teams to navigate obstructed areas.
Authors of the paper on “computational mirrors”
- Miika Aittala, former CSAIL postdoc, current research scientist at NVIDIA, lead researcher
- Prafull Sharma, CSAIL PhD student
- Lukas Murmann, CSAIL PhD student
- Adam Yedidia, CSAIL PhD student
- Fredo Durand, MIT professor
- Bill Freeman, MIT professor
- Gregory Wornell, MIT professor
References:
1. Computational Mirrors: Blind Inverse Light Transport by Deep Matrix Factorization (MIT CSAIL)
2. Using computers to view the unseen (MIT News)
