A new science of looking around the corner
The researchers of computer vision have discovered the hidden world of visual signals that we have at our disposal, where there are inconspicuous movements that give out what was said and vague images of what is around the corner of
Specialist in computer vision Antonio Torralba , resting on the coast of Spain in 201? noticed on the wall of his room in the hotel random shadows, which seemed to be nothing to discard. As a result, Torralba realized that the color-changing spots on the wall were not shadows, but dull, inverted images of the patio that was outside. The window worked like pinhole - the simplest kind of camera, in which the light rays pass through a small hole and form on the other side an inverted image. On the sun-drenched wall this image could hardly be discerned. But Torralba realized that our world is filled with visual information that our eyes do not perceive.
"These images are hidden from us," he said, "but they constantly surround us."
Bill Freeman , also a professor at the Massachusetts Institute of Technology, to realize that the world is filled with "random chambers", as they are called: windows, corners, houseplants and other ordinary objects creating hidden images of their environment. These images are 1000 times less bright than anything else, and usually they are not visible to the unaided eye. "We came up with ways to isolate these images and make them visible," Freeman explained.
They learned how much visual information is hidden right in front of everyone. In the first work of they showed that when shooting with the help of an ordinary iPhone changes the light on the wall of the room, you can recreate the scene outside the window from the video you received. Last fall they and their colleagues reported , that you can detect a person moving around the corner, shooting the camera on the ground near the corner. This summer they are have demonstrated that they can shoot a home plant on the video, and then recreate a three-dimensional image of the entire room based on the shadows cast by the leaves of the plant. Or they can turn the leaves into a " visual microphone ", Increasing their vibrations and recognizing speech.
1) Patio outside the hotel room where Antonio Torralba noticed that the window works like a pinhole. 2) Blurred image of the patio on the wall; 3) it can be sharpened by covering most of the window with cardboard to reduce the size of the hole. 4) If you turn it upside down, you can see the scene from the outside.
"Our Mary had a sheep," says the man on the audio record, recreated from the movements of an empty bag of chips that scientists shot through the soundproof window in 2014 (these are the first words recorded by Thomas Edison in 1877 on a phonograph).
Investigations about looking at corners and constructing assumptions about objects that are not directly visible, or "constructing images not in line of sight", began in 2012 with the work of Torralba and Freeman on a random camera, and with another breaking work , conducted by a separate group of scientists from MIT under the leadership of Ramesh's Ramesh . In 201? in particular, and thanks to their results, the Office of Advanced Research Projects of the US Department of Defense (DARPA) launched a $ 27 million REVEAL program (Revolutionary Enhancement of Visibility by Exploiting Active Light Fields). The program finances emerging laboratories throughout the country. Since then, the flow of new ideas and mathematical tricks makes the construction of images not in line of sight ever more powerful and practical.
In addition to the obvious use in military and intelligence purposes, researchers are studying the use of technology in robotic vehicles, robotic vision, medical photography, astronomy, space exploration and rescue missions.
Torralba said that he and Freeman did not have any ideas on the practical application of technology at the very beginning of the work. They simply understood the basics of image formation and what a camera is, from which a more complete study of the behavior of light and its interaction with objects and surfaces naturally developed. They began to see things that no one could think of looking. Psychological research, according to Torralba, shows that "people are terribly bad at interpreting shadows. Perhaps one of the reasons for this is that many of the things we see are not shadows. And in the end, my eyes gave up trying to comprehend them. "
The rays of light, carrying the image of the world outside our field of vision, constantly fall on the walls and other surfaces, after which they are reflected and come into our eyes. But why are these visual remnants so weak? Just too many rays go on too many directions, and the images are blurred.
To form an image, it is necessary to seriously limit the rays incident on the surface and to see only a certain set of them. This is what pinhole camera does. The original idea of Torralba and Freeman in 2012 was that in our environment there are quite a lot of objects and various properties that naturally limit the rays of light and form weak images that the computer can recognize.
The smaller the pinhole aperture, the sharper the image, since each point of the object being studied will emit only one light beam at the right angle, which it will be possible to pass through the hole. The window in the Torralba hotel was too big for the image to be sharp, and he and Freeman understood that in general useful occasional pinhole cameras were rare. However, they realized that anti-pinholes ("point" cameras), consisting of any small object that blocks light, form images in abundance.
Imagine that you are removing the inner wall of the room through a slot in the blinds. You will not see much. Suddenly, a hand appears in your field of vision. Comparing the intensity of light on a wall with and without a hand gives useful information about the scene. A set of rays of light falling on the wall in the first frame is momentarily blocked by the hand in the next. Subtracting the data from the second frame from the data of the first, as Freeman says, "you can calculate what the hand blocked" - a set of light rays representing the image of a part of the room. "If you study what blocks light, and what passes light," he said, "you can expand the set of places where you can find pinhole cameras."
Together with the work on the study of random cameras that perceive small changes in intensity, Freeman and his colleagues developed algorithms that determine and enhance small color changes-such as changing the color of a person's face at high tide or low tide, and tiny movements-this is what it was possible to record the conversation by removing a bag of chips. Now they can easily notice movement in one-hundredth pixel, which under ordinary conditions would simply sink in the noise. Their method mathematically converts images into a sinusoidal configuration. In the resulting space, noise does not dominate the signal, since the sinusoids represent the mean values taken over many pixels, so the noise is distributed over them. Thanks to this, researchers can determine the shifts of the sinusoid from one frame of video to another, amplify these shifts, and then convert the data back.
Now they began to combine all these tricks to extract hidden visual information. In a study described last October, which conducted Cathy Bowman (then a student under the direction of Freeman, and now a scientist from the Harvard-Smithsonian Astrophysical Center), it was shown that the corners of buildings function as cameras, creating a rough image of what is around the corner.
By removing the penumbra on the ground next to the angle (1), you can get information about objects that are around the corner (2). When invisible objects begin to move, light and shadows from them move at different angles relative to the wall. Small changes in intensity and color can not usually be discerned with the unaided eye (3), but can be enhanced by algorithms. Primitive videos with light coming from different angles to the penumbra give out the presence around the corner of one moving person (4) and two (5).
The edges and corners, like pinholes with point cameras, prevent the passage of sunlight. Using ordinary cameras, the same iPhone, in daylight, Bowman and his colleagues removed the penumbra at the corner of the building - an area with shadows, highlighted by a subset of light rays coming from a hidden area around the corner. If, for example, there is a man in a red shirt, this shirt will send a small amount of red light to the penumbra, and this light will move around the penumbra, while the person walks, invisible by the ordinary eye, but detected after post-processing.
In a revolutionary work published in June, Freeman and colleagues recreated the "light field" of the room - a picture of the intensity and direction of the rays of light in the room - from the shadows cast by the deciduous plant standing next to the wall. The leaves worked as point cells, each of which blocked its set of light rays. Matching the shadow of each sheet with the rest of the shadows gave out this missing set of rays, and allowed to get an image of a part of the hidden scene. Given parallax, the researchers then were able to bring all these images together.
This approach yields much sharper images than the early work with random cameras, since the algorithm incorporates pre-acquired knowledge of the world. Knowing the shape of the plant, believing that natural images should be smooth, and taking into account several other assumptions, the researchers were able to draw certain conclusions about signals containing noise, which helped to make the resulting image sharper. The technology of working with the light field "requires knowledge of the surrounding world to create reconstruction, but also gives you a lot of information," Torralba said.
And while Freeman, Torralba and their proteges uncover images that were hidden, elsewhere on the MIT campus, Ramesh Raskar, a computer vision specialist who spoke at the TED, intends to "change the world" and chooses an approach called "active imaging". He uses specialized expensive laser camera systems to create high-resolution images that represent what's around the corner.
In 201? within the framework of the idea that visited him five years ago, Raskar and his team created for the first time a technology in which it is necessary to produce laser pulses into a wall. A small part of the scattered light will manage to bypass the obstacle. And after a short time after each pulse, they use a "flash camera" recording individual photons at a rate of billions of frames per second to detect photons bouncing off the wall. Measuring the time spent by photons for the return, the researchers can find out how far they have flown away, and in detail to recreate the three-dimensional geometry of objects hidden behind the obstacle, on which the photons are scattered. One of the difficulties is that for the formation of a three-dimensional image, it is necessary to perform raster scanning of the wall with a laser. Let's say that a man is hiding around the corner. "Then the light reflected from a certain point on the head, from a certain point on the shoulder, and from a certain point on the knee, can come to the camera at the same time," said Raskar. But if you shine a laser a little elsewhere, then the light from these three points will not arrive at the camera at the same time. " It is necessary to combine all the signals and solve the "inverse problem" for recreating the hidden three-dimensional geometry.
The original Raskar algorithm for solving the inverse problem required too much computational resources, and the device itself cost half a million dollars. But serious work has been done to simplify mathematics and reduce costs. In March, in the journal Nature, was published. work , set a new standard for efficient and economical construction of three-dimensional images of the object - in the work recreated the figure of a rabbit - located around the corner. The authors, Matthew O'Toole , David Lindel and Gordon Vectaine from Stanford University developed a powerful new algorithm for solving the inverse problem and used relatively inexpensive camera SPAD - semiconductor devices, the frame rate of which is lower than that of flash cameras. Raskar, who previously worked as curator of the two authors of the work, called her "very cunning" and "one of my favorites."
Previous algorithms were drowned in detail: the researchers usually tried to find the returning photons that reflected not from the point of the wall into which the laser was shining so that the camera could avoid collecting scattered laser light. But by directing the laser and camera almost to one point, the researchers were able to map outgoing and incoming photons from one " light cone ". Dissipating from the surface, light forms an expanding sphere of photons, which draws a cone, spreading in space-time. O'Toole (who has since changed jobs with Stanford to Carnegie Mellon University) has translated the physics of light cones - developed by the teacher Albert Einstein, Hermann Minkowski at the beginning of the 20th century - into a laconic expression connecting the time of the flight of a photon with the location of scattering surfaces. He called his translation "light cone transformation".
Robotic vehicles already use LIDAR systems to build direct images, and one can imagine that someday they will acquire SPAD to look around the corner. "In the near future, such sensors will be available in a portable format," predicts Andreas Welten , the first author of the initial work of Raskar from 201? who is now in charge of the active image imaging team at Wisconsin University. Now the task is to "handle more complex scenes" and realistic scenarios, said Welten, "and not just a careful creation of a scene with a white object and a black backdrop. We need a technology that allows us to direct the device and press the button. "
Where are the things
Researchers from the Freeman group began to combine passive and active approaches. In a work conducted under the guidance of the researcher Christos Trumpulidis, it is shown that with the active construction of images by means of a laser, a point cloud of a known shape located around the corner can be used to reconstruct a hidden scene without using the information about the flight time of photons. "And this should happen with us using the usual CCD matrix ", Said Trumpulidis.
The construction of images not in direct line of sight will someday be able to help rescue teams and autonomous robots. Velten cooperates with the NASA Jet Propulsion Laboratory, working on a project aimed at building distant images of objects within the caves of the Moon. And Raskar and the company use their approach to read the first few pages of a closed book, and in order to see in the fog.
In addition to audio reconstruction, the Freeman motion amplification algorithm can help create medical devices and security systems, as well as detectors for small astronomical movements. This algorithm is "a very good idea," said David Hogg, an astronomer and data specialist from New York University and the Flatiron Institute. "I thought - we just have to use it in astronomy."
As for the privacy issues raised by recent discoveries, Freeman turns to his experience. "I've been thinking about this problem very, very much my whole career," he says. Ochkarik, an amateur tinker with cameras, all his life engaged in photography, Freeman said that at the beginning of his career he did not want to work on anything, which would have some kind of military or spy potential. But over time, he began to think that "technology is a tool that can be used in many ways. If you try to avoid everything that can have at least some military application, then you can not think of anything useful. " He says that even in the case of the military, "there is a very wide range of possibilities for using things. You can help someone survive. And, in principle, to know where the things are is useful. "
But its most pleasing is not the technological possibilities, but simply the discovery of the phenomenon, hiding from everyone in sight. "It seems to me that the world is full of everything that has yet to be discovered," he said.
It may be interesting
Thttps://clubessay.com/here is definately a great deal to know about this subject. I like all of the points you've made.
VK Mobile ChallengE is best challange I have ever seen. Thanks for sharing such awesome news.
I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more.
Situs QQ Online
Situs QQ Online
After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.
Situs QQ Online
Situs QQ Online