No icon

Light Field Inpainting Propagation via Low Rank Matrix Completion

Light Field Inpainting Propagation via Low Rank Matrix Completion


Building up on the advances in low rank matrix completion, this paper presents a novel method for propagating the inpainting of the central view of a light field to all the other views. After generating a set of warped versions of the inpainted central view with random homographies, both the original light field views and the warped ones are vectorized and concatenated into a matrix. Because of the redundancy between the views, the matrix satisfies a low rank assumption enabling us to fill the region to inpaint with low rank matrix completion. To this end, a new matrix completion algorithm, better suited to the inpainting application than existing methods, is also developed in this paper. In its simple form, our method does not require any depth prior, unlike most existing light field inpainting algorithms. The method has then been extended to better handle the case where the area to inpaint contains depth discontinuities. In this case, a segmentation map of the different depth layers of the inpainted central view is required. This information is used to  warp the depth layers with different homographies. Our experiments with natural light fields captured with plenoptic cameras demonstrate the robustness of the low rank approach to noisy data as well as large color and illumination variations between the views of the light field.

Existing System:

As the capture of 4D light fields from real scenes grows in popularity, the need for a generalization of traditional image editing tools is expected to rise as well. In particular, for inpainting, which is the recovery of missing parts of an image, plenty of algorithms already exist in the 2D case.

For instance, diffusion based methods can efficiently recover thin holes. Exemplar based methods are better suited for applications such as object removal thanks to their ability to fill larger holes using patches of texture taken from the known parts of the image. However, applying independently these methods to each sub-aperture image of a light field is very unlikely to result in a consistent inpainting in the angular direction.

Proposed System:

The definition of a new low rank matrix completion algorithm suitable for completing large parts of a matrix without requiring the exact matrix rank.

The use of this algorithm for propagating an inpainted area from the central view to all the other views of a light field. In case of depth discontinuities within the inpainted area, an extension of the method using a depth layer segmentation map is also proposed.

As an intermediate step of the inpainting propagation, we define a method for randomly generating homography warpings that are consistent with the displacement between views in actual light field data.

Comment As:

Comment (0)