Light Field Super-Resolution via LFBM5D Sparse Coding

Abstract

In this paper, we propose a spatial super-resolution method for light fields, which combines the SR-BM3D single image super-resolution filter and the recently introduced LFBM5D light field denoising filter. The proposed algorithm iteratively alternates between an LFBM5D filtering step and a back-projection step. The LFBM5D filter creates disparity compensated 4D patches which are then stacked together with similar 4D patches along a 5th dimension. The 5D patches are then filtered in the 5D transform domain to enforce a sparse coding of the high-resolution light field, which is a powerful prior to solve the ill-posed super-resolution problem. The back-projection step then impose the consistency between the known low-resolution light field and the-high resolution estimate. We further improve this step by using image guided filtering to remove ringing artifacts. Results show that significant improvement can be achieved compared to state-of-the-art methods, for both light fields captured with a lenslet camera or a gantry.

Type
Conference proceedings
Publication
IEEE International Conference on Image Processing
Date
Links
PDF IEEE Xplore V-SENSE

Implementation

The MATLAB/C/C++ source code is available on github.

Additional results

Visual results complementing the paper are shown below. We then show in Tables 1 and 2 the average results presented in the paper. The corresponding detailed results are shown in Table 3 and 4.

Lytro Illum dataset

We use in our experiments a Lytro Illum dataset consisting of light fields taken from the EPFL and the INRIA datasets. These light fields have been further processed with the V-SENSE enhancement pipeline described here.

Visual results

We show in videos below (click on an image to start a video) side by side comparisons of low-resolution (uspampled with a bicubic filter) and super-resolved light fields for different scaling factors α. On the top left corner of each video is highlighted the sub-aperture image being displayed. Note that some videos may exhibit encoding artifacts.

α=2 α=3 α=4
Chess α=2 α=3 α=4
Lego Bulldozer α=2 α=3 α=4
Lego Knights α=2 α=3 α=4
Tarot Cards and Crystal Ball (Large Angle) α=2 α=3 α=4
Bee 2 α=2 α=3 α=4
Bikes α=2 α=3 α=4
Danger de mort α=2 α=3 α=4
Vespa α=2 α=3 α=4
Below, we show results for α=4 without (left) and with (right) the light field guided filtering described in section 2.2 of the paper to improve back-projection (click on an image to start a video).
Chess Lego Knights Bikes Vespa
Chess Lego Knights Bikes Vespa

Average PSNR results

Values highlighted in bold and italic correspond to the best and second best performing methods respectively for a given magnification factor.
Table 1 - Average SR performances in PSNR for the Lytro Illum dataset.
α=2 α=3 α=4
Bicubic 27.78 26.08 24.62
SR-BM3D 30.21 28.45 26.58
BM+PCA+RR 29.95 28.55 27.08
GB 29.80 28.65 27.45
SR-LFBM5D 1st step 30.17 28.62 26.87
SR-LFBM5D 2nd step 30.25 28.60 26.82
Table 2 - Average SR performances in PSNR for the Stanford dataset.
α=2 α=3 α=4
Bicubic 29.00 26.81 25.06
SR-BM3D 34.10 30.90 28.00
BM+PCA+RR 32.81 30.85 28.73
GB 33.01 31.42 29.44
SR-LFBM5D 1st step 34.15 31.81 29.10
SR-LFBM5D 2nd step 34.27 31.77 29.02

Detailed PSNR results

Table 3 - SR performances in PSNR for each light field in the Lytro Illum dataset.
Ankylosorus & Dipplodocus 1 Bee 1 Bee 2 Bikes Chez Edgar Danger de Mort Friends 1 Fruits Magnets 1 Posts Rose Vespa
Bicubic
α=2 33.5414 27.0979 27.5808 26.2794 22.3877 24.3675 28.713 26.0159 31.0621 30.4165 28.5095 27.3734
α=3 31.5025 25.4013 25.9392 24.5069 21.1371 22.8849 26.589 24.3747 29.0306 29.2154 26.8529 25.5526
α=4 29.7729 23.8938 24.4516 22.9869 20.0919 21.7029 24.769 22.9985 27.2962 28.0506 25.4262 24.0135
SR-BM3D
α=2 36.075 29.023 29.346 30.058 24.220 27.101 32.185 28.791 33.858 30.958 30.597 30.354
α=3 34.207 27.892 28.265 27.978 22.433 24.979 30.423 26.827 31.500 29.667 28.733 28.534
α=4 32.477 26.421 26.959 25.369 20.908 22.756 28.256 24.748 29.300 28.299 26.801 26.698
BM+PCA+RR
α=2 35.6127 28.7435 29.0858 29.5911 24.2046 26.8869 31.8674 28.6242 33.4409 30.8375 30.4213 30.1102
α=3 34.3607 27.6675 28.225 27.8739 22.7327 25.2586 30.4348 27.1949 31.5474 29.704 28.9552 28.6078
α=4 32.9217 26.4726 27.1327 26.0074 21.4623 23.7469 28.7222 25.6303 29.7057 28.5436 27.481 27.1216
GB
α=2 35.7291 28.7144 29.159 29.329 23.753 26.618 31.629 28.249 33.353 30.9148 30.292 29.918
α=3 34.606 27.774 28.395 28.073 22.644 25.356 30.531 27.159 31.672 29.781 28.976 28.781
α=4 33.441 26.757 27.588 26.643 21.536 24.126 29.114 26.108 29.991 28.627 27.793 27.619
SR-LFBM5D 1st step
α=2 36.044 28.992 29.303 29.977 24.222 27.059 32.136 28.714 33.840 30.917 30.495 30.326
α=3 34.559 27.921 28.366 28.127 22.459 25.171 30.529 27.139 31.586 29.767 29.039 28.756
α=4 33.068 26.587 27.203 25.678 20.977 22.794 28.649 25.246 29.415 28.456 27.262 27.133
SR-LFBM5D 2nd step
α=2 36.123 29.037 29.350 30.098 24.282 27.144 32.242 28.816 33.899 30.966 30.602 30.425
α=3 34.532 27.908 28.348 28.115 22.439 25.133 30.513 27.106 31.573 29.768 29.007 28.731
α=4 33.032 26.561 27.170 25.614 20.909 22.650 28.595 25.161 29.379 28.447 27.201 27.074
Table 4 - SR performances in PSNR for each light field in the Stanford dataset.
Amethyst Bracelet Chess Eucalyptus Flowers Jelly beans Lego Bulldozer Lego Knights Lego Truck Tarot Cards and Crystal Ball (Large Angle) Tarot Cards and Crystal Ball (Small Angle) The Stanford Bunny Treasure Chest
Bicubic
α=2 27.9143 26.3944 31.0063 23.3153 41.5464 26.9215 28.5825 27.7637 27.0215 27.103 36.0681 24.3827
α=3 25.846 24.1524 28.707 22.1331 38.0155 24.7319 26.3982 25.8623 25.2307 24.8391 32.8963 22.9615
α=4 24.1607 22.3826 26.8413 21.2074 35.4326 22.9613 24.5788 24.2874 23.5296 23.1016 30.4554 21.745
SR-BM3D
α=2 32.591 34.325 37.250 26.424 45.191 32.029 34.014 31.820 32.241 33.764 41.051 28.487
α=3 29.935 29.172 33.646 23.929 42.740 29.198 30.944 28.929 29.048 28.878 38.889 25.505
α=4 27.491 25.111 30.247 22.269 39.682 26.305 28.053 26.103 26.087 25.339 36.061 23.264
BM+PCA+RR
α=2 32.1632 32.8753 36.1938 26.2293 43.3373 30.5675 32.8232 31.4071 28.4677 31.6728 39.7559 28.1899
α=3 30.281 30.260 33.903 24.775 41.575 28.482 30.898 29.497 26.363 29.435 38.600 26.169
α=4 28.428 26.799 31.713 23.344 39.239 26.367 28.695 27.502 24.485 27.127 36.452 24.578
GB
α=2 32.164 32.164 35.715 25.529 45.301 30.928 32.768 30.968 31.457 30.939 41.017 27.192
α=3 30.245 30.34 34.136 24.595 43.182 29.293 31.475 29.549 29.298 29.620 39.330 25.976
α=4 29.127 28.16 32.636 24.012 40.835 27.828 29.995 28.007 21.534 28.148 37.816 25.203
SR-LFBM5D 1st step
α=2 32.620 34.214 37.455 26.450 45.396 32.007 34.210 31.896 32.278 33.673 41.109 28.514
α=3 30.440 30.535 35.242 24.653 43.099 29.804 32.467 29.949 30.077 29.863 39.412 26.141
α=4 28.125 26.351 31.670 22.838 40.689 27.241 29.767 27.510 26.837 26.724 37.157 24.324
SR-LFBM5D 2nd step
α=2 32.775 34.508 37.567 26.584 45.261 32.125 34.312 31.982 32.445 33.902 41.172 28.659
α=3 30.400 30.477 35.206 24.629 43.032 29.769 32.427 29.923 30.046 29.811 39.385 26.112
α=4 28.046 26.209 31.618 22.754 40.626 27.152 29.707 27.463 26.738 26.614 37.073 24.236