Selected Publications
Immersive Video Technologies
Get a broad overview of the different modalities of immersive video technologies—from omnidirectional video to light fields and volumetric video—from a multimedia processing perspective. From capture to representation, coding, and display, video technologies have been evolving significantly and in many different directions over the last few decades, with the ultimate goal of providing a truly immersive experience to users. After setting up a common background for these technologies, based on the plenoptic function theoretical concept, Immersive Video Technologies offers a comprehensive overview of the leading technologies enabling visual immersion, including omnidirectional (360 degrees) video, light fields, and volumetric video. Following the critical components of the typical content production and delivery pipeline, the book presents acquisition, representation, coding, rendering, and quality assessment approaches for each immersive video modality. The text also reviews current standardization efforts and explores new research directions. With this book the reader will a) gain a broad understanding of immersive video technologies that use three different modalities: omnidirectional video, light fields, and volumetric video; b) learn about the most recent scientific results in the field, including the recent learning-based methodologies; and c) understand the challenges and perspectives for immersive video technologies.
Elsevier
Spectral Analysis of Re-parameterized Light Fields
In this paper, we study the spectral properties of re-parameterized light field. Following previous studies of the light field spectrum, which notably provided sampling guidelines, we focus on the two plane parameterization of the light field. However, we introduce additional flexibility by allowing the image plane to be tilted and not only parallel. A formal theoretical analysis is first presented, which shows that more flexible sampling guidelines (i.e. wider camera baselines) can be used to sample the light field when adapting the image plane orientation to the scene geometry. We then present our simulations and results to support these theoretical findings. While the work introduced in this paper is mostly theoretical, we believe these new findings open exciting avenues for more practical application of light fields, such as view synthesis or compact representation.
SPIC 2022
A spatio-angular filter for high quality sparse light field refocusing
The ability to render synthetic depth-of-field effects post capture is a flagship application of light field imaging. However, it is known that many existing light field refocusing methods suffer from severe artefacts when applied to sparse light fields, known as angular aliasing. We propose in this paper a method for high quality sparse light field refocusing based on insights from depth-based bokeh rendering techniques. We first provide an in-depth analysis of the geometry of the defocus blur in light field refocusing by analogy with the defocus geometry in a traditional camera using the thin lens model. Based on this analysis, we propose a filter for removing angular aliasing artefacts in light field refocusing, which consists in modifying the well known shift-and-sum algorithm to apply a depth-dependent blur to the light field in between the shift and the sum operations. We show that our method can achieve significant quality improvements compared to existing approaches for a reasonable computational cost.
ICMEW 2021
Light Field Style Transfer With Local Angular Consistency
Style transfer involves combining the style of one image with the content of another to form a new image. Unlike traditional two-dimensional images which only capture the spatial intensity of light rays, four-dimensional light fields also capture the angular direction of the light rays. Thus, applying style transfer to a light field requires to not only render convincing style transfer for each view, but also to preserve its angular structure. We present a novel optimization-based method for light field style transfer which iteratively propagates the style transfer from the centre view towards the outer views while enforcing local angular consistency. For this purpose, a new initialisation method and angular loss function is proposed for the optimization process. In addition, since style transfer for light field is an emerging topic, no clear evaluation procedure is available. Thus, we investigate the use of a recently proposed metric designed to evaluate light field angular consistency, as well as a proposed variant.
ICASSP 2021
Light Field Super-Resolution via LFBM5D Sparse Coding
In this paper, we propose a spatial super-resolution method for light fields, which combines the SR-BM3D single image super-resolution filter and the recently introduced LFBM5D light field denoising filter. The proposed algorithm iteratively alternates between an LFBM5D filtering step and a back-projection step. The LFBM5D filter creates disparity compensated 4D patches which are then stacked together with similar 4D patches along a 5th dimension. The 5D patches are then filtered in the 5D transform domain to enforce a sparse coding of the high-resolution light field, which is a powerful prior to solve the ill-posed super-resolution problem. The back-projection step then impose the consistency between the known low-resolution light field and the-high resolution estimate. We further improve this step by using image guided filtering to remove ringing artifacts. Results show that significant improvement can be achieved compared to state-of-the-art methods, for both light fields captured with a lenslet camera or a gantry.
ICIP 2018
Scalable image coding based on epitomes
In this paper, we propose a novel scheme for scalable image coding based on the concept of epitome. An epitome can be seen as a factorized representation of an image. Focusing on spatial scalability, the enhancement layer of the proposed scheme contains only the epitome of the input image. The pixels of the enhancement layer not contained in the epitome are then restored using two approaches inspired from local learning-based super-resolution methods. In the first method, a locally linear embedding model is learned on base layer patches and then applied to the corresponding epitome patches to reconstruct the enhancement layer. The second approach learns linear mappings between pairs of co-located base layer and epitome patches. Experiments have shown that the significant improvement of the rate-distortion performances can be achieved compared with the Scalable extension of HEVC (SHVC).
IEEE TIP 2017
