Light Fields — from Display to 4D Algorithms

The light field is a 4D record of a scene's rays — how do we present it to humans, interact with it, and process it computationally?

A light field captures the radiance at every point in space, in every direction — a 4D function that fully describes how light fills a scene. Captured light fields enable refocusing, depth recovery, and parallax view synthesis; displayed light fields offer glasses-free 3D. The challenge is data density: 4D content stresses capture devices, display hardware, and processing pipelines.

Two sub-arcs sit in this thread. The first (2012–2015) targets light field displays — an Emerging Technologies demo of painting directly into a glasses-free 3D display, content-adaptive lenticular prints that reshape the lenslet array to the captured light field, and a UIST paper that turns that lenslet array into a joint display-and-pen-input surface. The second (2019–2021), led by Numair Khan with Min H. Kim at KAIST, develops dense algorithms over captured 4D content: view-consistent superpixels via epipolar-plane image segmentation, edge-aware bidirectional diffusion for depth, and a differentiable diffusion routine for sparse-to-dense depth from multi-view images.

Authors

Marc Alexa · Simon Heinzle · Stanislav Jakuschevskij · Lucas Kasser · Jan Kautz · Numair Khan · Min H. Kim · Wojciech Matusik · James McCann · Jim McCann · Samuel Muff · Hanspeter Pfister · Henry Stone · Qian Zhang

Papers in this thread

Interactive Light Field Painting

SIGGRAPH Emerging Technologies, 2012

An early SIGGRAPH Emerging Technologies demo of a dual-purpose lenslet array that both displays a light field and senses a 3D light-pen position — the live precursor to the UIST 2015 write-up.

Content-adaptive Lenticular Prints

ACM Transactions on Graphics (SIGGRAPH), 2013

Treats the lenslet array as something to optimise rather than fix in advance — given an input light field, solve for lenslet size, shape, and arrangement that trade spatial against angular resolution where it matters. Validated by 3D printing the resulting arrays.

Joint 5D Pen Input for Light Field Displays

User Interface Software and Technology (UIST), 2015

One lenslet array does double duty — light field output and 5D pen input (3D position plus 2D orientation) at 150 Hz, with millimetre-scale accuracy. The display surface and the input surface are the same surface.

View-consistent 4D Light Field Superpixel Segmentation

International Conference on Computer Vision (ICCV), 2019

Segments horizontal and vertical EPIs first, then clusters and propagates across all sub-aperture views — so superpixels stay consistent and respect occlusion as the viewpoint shifts, rather than being propagated outward from a single central view.

Differentiable Diffusion for Dense Depth Estimation from Multi-view Images

Computer Vision and Pattern Recognition (CVPR), 2021

Dense depth is obtained by diffusing a sparse set of points whose positions, depths, and weights are differentiably optimised through Gaussian splatting against a multi-view RGB reprojection loss. Scales to the 50k+ points needed for non-trivial scenes.

4D Light Field Depth Estimation

BMVC, 2021

A pair of BMVC papers that estimate 4D depth from sparse EPI-derived edges and diffuse them outward — the 2021 paper separates depth from texture edges via bidirectional diffusion, and the 2020 paper propagates the central-view depth to every other sub-aperture view in an occlusion-aware way.