A Spatio-Angular Binary Descriptor for Fast Light Field Inter View Matching

Abstract

Light fields are able to capture light rays from a scene arriving at different angles, effectively creating multiple perspective views of the same scene. Thus, one of the flagship applications of light fields is to estimate the captured scene geometry, which can notably be achieved by establishing correspondences between the perspective views, usually in the form of a disparity map. Such correspondence estimation has been a long standing research topic in computer vision, with application to stereo vision or optical flow. Research in this area has shown the importance of well designed descriptors to enable fast and accurate matching. We propose in this paper a binary descriptor exploiting the light field gradient over both the spatial and the angular dimensions in order to improve inter view matching. We demonstrate in a disparity estimation application that it can achieve comparable accuracy compared to existing descriptors while being faster to compute.

Publication
IEEE International Conference on Image Processing
Date

Implementation

The C++ code will soon be available on github.

Presentation

Additional results

The proposed Spatio-Angular Binarised Orientation Maps (SABOM) descriptor is compared to SIFT, DAISY, and the original BOOM descriptor.

ROC performance

HCI datasetINRIA Dense datasetINRIA Sparse dataset

Disparity estimation

CPM - HCICPM - INRIA
CPM+PF - HCICPM+PF - INRIA

Visual results

We show here comparisons between the disparity maps obtained with the SIFT descriptor and the proposed SABOM descriptor for the HCI benchmark, the INRIA synthetic dataset, and the Stanford Lego Gantry dataset. As mentioned in the paper, the disparity estimation is used to compare existing descriptors but would require further improvement to compete with the best state-of-the-art method for disparity estimation. The disparity maps are color-coded with the Turbo colormap.
Overall, similar quality is obtained for both descriptors for while SABOM is much faster to compute.

HCI dataset

Antinous Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Boardgames Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Dishes Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Greek Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Kitchen Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Medieval2 Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Museum Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Pens Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Pillows Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Platonic Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Rosemary Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Table Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Tomb Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Tower Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Town Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Vinyl Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

INRIA Dense dataset

Antiques Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Big_clock Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Blue_room Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Bottles Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Camera_brush Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Dinosaur Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Dishes Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Electro_devices Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Flowers_clock Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Flying_dice Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Flying_furniture Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Flying_toys Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Furniture Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Green_balloon Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Kiwi_bike Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Lonely_man Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Microphone Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Microphone_rooster Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Origami Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Pinenuts_blue Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Robots Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Rooster_clock Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Roses_bed Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Roses_table Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Smiling_crowd Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Smiling_crowd_roses Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Three_pillows Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Toy_bricks Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Toy_friends Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Two_vases Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
White_lamp Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
White_roses Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Stanford dataset

Amethyst Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
Chess Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
EucalyptusFlowers Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
JellyBeans Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
LegoBulldozer Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
LegoKnights Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
TarotCardsandCrystalBall_LargeAngle Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
TarotCardsandCrystalBall_SmallAngle Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
TheStanfordBunny Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
TreasureChest Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF
https://youtu.be/SyhGy75BMtc

Implementation

The C++ code will soon be available on github.

Additional results

The proposed Spatio-Angular Binarised Orientation Maps (SABOM) descriptor is compared to SIFT, DAISY, and the original BOOM descriptor.

ROC performance

HCI datasetINRIA Dense datasetINRIA Sparse dataset

Disparity estimation

CPM - HCICPM - INRIA
CPM+PF - HCICPM+PF - INRIA

Visual results

We show here comparisons between the disparity maps obtained with the SIFT descriptor and the proposed SABOM descriptor for the HCI benchmark, the INRIA synthetic dataset, and the Stanford Lego Gantry dataset. As mentioned in the paper, the disparity estimation is used to compare existing descriptors but would require further improvement to compete with the best state-of-the-art method for disparity estimation. The disparity maps are color-coded with the Turbo colormap.
Overall, similar quality is obtained for both descriptors for while SABOM is much faster to compute.

HCI dataset

Antinous Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Boardgames Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Dishes Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Greek Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Kitchen Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Medieval2 Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Museum Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Pens Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Pillows Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Platonic Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Rosemary Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Table Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Tomb Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Tower Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Town Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Vinyl Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

INRIA Dense dataset

Antiques Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Big_clock Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Blue_room Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Bottles Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Camera_brush Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Dinosaur Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Dishes Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Electro_devices Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Flowers_clock Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Flying_dice Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Flying_furniture Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Flying_toys Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Furniture Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Green_balloon Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Kiwi_bike Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Lonely_man Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Microphone Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Microphone_rooster Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Origami Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Pinenuts_blue Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Robots Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Rooster_clock Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Roses_bed Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Roses_table Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Smiling_crowd Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Smiling_crowd_roses Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Three_pillows Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Toy_bricks Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Toy_friends Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Two_vases Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

White_lamp Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

White_roses Center ViewGround truth disparity map
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Stanford dataset

Amethyst Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

Chess Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

EucalyptusFlowers Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

JellyBeans Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

LegoBulldozer Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

LegoKnights Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

TarotCardsandCrystalBall_LargeAngle Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

TarotCardsandCrystalBall_SmallAngle Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

TheStanfordBunny Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF

TreasureChest Center ViewGround truth disparity map not available
 
SIFT CPMSIFT CPM+PF
SABOM CPMSABOM CPM+PF