Epitome inpainting with in-loop residue coding for image compression

Abstract

This paper describes a novel image coding scheme based on epitome inpainting. An epitome containing a factorized texture representation of the image is first coded and transmitted. The decoded epitome is then inpainted by propagating its structure and texture with extended H.264 Intra directional prediction modes and advanced neighbor embedding methods. The blocks to be inpainted are processed according to an adaptive scanning order which adapts to the image content and gives a higher priority to the blocks having strong structures. The inpainting step has been incorporated in the coding loop of an H.264 encoder, leading to a bitrate saving of up to 48.12% with respect to H.264 intra.

Type
Conference proceedings
Publication
IEEE International Conference on Image Processing
Date
Links
PDF IEEE Xplore INRIA

Context and Goal

Image/video compression standards such as JPEG, H.264, and more recently HEVC, achieve state-of-the-art compression performances. However, modern image formats call for improved coding performances. Independently from the standard image/video compression techniques emerged the concept of epitome, first presented by Jojic et al., which is defined as a condensed representation of an image or video containing the essence of its textural properties.
In this work, we propose a novel image coding scheme exploiting the epitome concept.

Approach

The main steps of the proposed coding scheme are depicted in Fig. 1.

First, an epitome consisting of texture pieces from the input image is generated and encoded. More explanations on the epitome generation are available here or in the papers listed at the end of this page.

Second, the epitome is decoded and used to predict the missing non-epitome blocks in an inpainting fashion, which implies changing the scanning order of the blocks in the encoder. An adaptive scanning order is thus used, based on the priority order proposed by criminisi et al. for object removal, adapted here to fit with the block structure of the encoder. Blocks surrounded by strong linear structures and a larger number of reliable pixels are assigned the highest priority and processed first. The inpainting of the epitome is performed by propagating texture and structure using extended multi-directional H.264 Intra prediction modes in competition with extra modes based on block matching and weighted template matching techniques.

Finally, the best mode is selected using an RDO criterion and once the prediction obtained, the residue of the block is computed, transformed, quantized and entropy coded with the H.264 encoding tools. At the decoder side the residue is obtained by inverse transform and dequantization. The block is immediately reconstructed by adding the decoded residue to the prediction. The reconstructed block is then available in the causal part of the image and the new front line is taken into account when computing the new filling order for the inpainting of the next block.

Proposed compression scheme based on epitome inpainting and in-loop residue coding.

Proposed compression scheme based on epitome inpainting and in-loop residue coding.

Experimental Results

To validate the approach, the inpainting step has been integrated in an H.264 codec with the block size set to 8x8. Objective assessments of the method were obtained using the Bjontegaard metric at low (Quantization parameter (QP) = 26, 31, 36, 41) and high bitrates (QP = 16, 21, 26, 31). Note that in the following results, the measured bitstream does take into account both the epitome texture and the non-epitome block residue bitstreams. The method was tested on a set of four images: Barbara, Building, Spincalendar and Wool.

In the videos below, we show the comparison between the usual raster scanning order and the adaptive scanning order of the proposed method for the Barbara and Spincalendar images.
In each video, the raster scan can be seen on the left and the adaptive scan on the right. The top row represents the encoding at high bitrate (QP=16) while the bottom row represents the encoding at low bitrate (QP=41). One can thus notice the changes in the adaptive scan depending on the image quality.
Note that we show here the prediction images.
We recommend Firefox or Chrome browsers to play the videos.

We show in Table 1 the RD performances against H.264, compared to a modified H.264 codec integrating the additional prediction modes used in the proposed method.

Table 1 - Comparison between the inpainting scheme integrated with neighbor embedding and H.264 intra modes with additional neighbor embedding
Low bitrates High bitrates
Raster scan Adapt. scan Raster scan Adapt. scan
dB % rate dB % rate dB % rate dB % rate
Barbara 1.19 -17.07 1.13 -16.16 0.92 -10.53 0.92 -10.58
Building 4.23 -46.78 4.40 -48.12 4.32 -32.80 4.48 -33.89
Spincalendar 2.87 -37.59 3.15 -39.99 2.64 -26.83 2.97 -29.92
Wool 2.28 -32.73 2.33 -33.39 1.76 -18.93 1.81 -19.39
Average 2.64 -33.54 2.75 -34.42 2.41 -22.27 2.54 -23.44

References