Laboratory for Image and Media Understanding,Department of Advanced Information Technology, Kyushu University

Light Field Vision

Outline

We propose light field vision, a new approach that uses light field cameras in research of visual recognition. We also established a technique for recognizing transparent and specular objects as one of the applications for supporting the advantage of the proposed approach. Light field represents all of the appearance in a scene and is often modeled as the four-dimensional function I(x, y, u, v), where (u, v) represents the two-dimensional position of the camera viewpoint, and (x, y) represents the two-dimensional image at that point. The conventional study of image understanding, computer vision, has been developed by using a normal, single viewpoint image. Hence, computer vision has been restricted to considering the subspaces of the light field represented as the image I(x, y, 0, 0) for solving a problem. We propose light field vision, a new visual recognition approach that uses light field images as inputs and establish it as a new field of research area. By using rich four-dimensional appearance information, including camera viewpoint differences provided by the light field, we try to solve the problems that have been difficult to solve in the area of computer vision based on normal images.

Detail

This study aims to propose light field vision as a breakthrough that essentially overcomes the limitations of traditional image recognition, as well as experimentally demonstrate the effectiveness of it. In general, an image is two-dimensional appearance information (shape, brightness, and color) in the actual three-dimensional world observed from a specific viewpoint. It is an extremely limited subset space of the original three-dimensional world. Consequently, it is essentially difficult to recognize objects and scenes in the real three-dimensional world from the image. If, however, we can collect images from all viewpoints, then it is easier, in principle, to understand the three-dimensional world from the collected images since they contain richer appearance information from the three-dimensional world and less ambiguity for understanding the object and scene.

In this study, information representation equivalent to the collection of images at all viewpoints is called the light field (LF) and the framework for recognizing real-world from LF is called light field vision. In particular, the study tries to demonstrate that light field vision allows you to recognize transparent and specular objects, which are almost unrecognizable with traditional image recognition, and that it can essentially overcome the limitations of traditional image recognition.

Specifically, LF is represented as the four-dimensional function I(x, y, u, v) (here, for intuitive understanding, x and y should be considered the image coordinates and u and v the camera viewpoint) and can model all the appearance in the three-dimensional space from every viewpoint. It can be considered that traditional camera images are LF subspaces I(x, y, 0, 0) with camera viewpoints u and v fixed. Stereo vision and motion stereo (structure from motion) are no exception and are still a subset space of LF, since each image from these techniques is merely provided with two viewpoints in the (u, v) space, instead of a single viewpoint, or a single viewpoint extended along a line. On the other hand, light field vision allows us to determine all vision issues (e.g., image recognition, three-dimensional reconstruction, and moving object tracking) more stably by using all viewpoints represented as (u, v) surfaces. It is expected to improve practicality in the application areas covered by computer vision.

Three-dimensional reconstruction based on a stereo camera, for example, usually provides only lateral parallaxes, allowing no depth against the lateral edges to be determined. On the other hand, the multidirectional parallaxes provided by LF are considered to allow for a depth estimation independent from the texture orientation. In moving object tracking as well, the depth information contained in LF can be used to track not only the conventional (x, y) direction movements parallel to images but also three-dimensional (x, y, z) movements, including the depth-direction movements. As these examples indicate, light field vision proposed by the study has the possibility of creating a variety of new approaches to image understanding.

Research Promotion

This work was supported by
P&P in Kyushu University
KAKENHI Grant-in-Aid for Scientific Research (A) (25240027)