Mansi Sharma

Recently, there has been extensive progress in developing autostereoscopic platforms for display purposes to present real-world 3D scenes. Light fields are the best emerging choice for computational multi-view autostereoscopic displays since they provide an optimized solution to support direction-dependent outputs simultaneously without sacrificing the resolution. We present a novel light field representation, coding and streaming scheme that efficiently handles large tensor data. Intrinsic redundancies in light field subsets are eliminated through low-rank representation using Tucker decomposition with tensor sketching for various ranks and sketch dimension parameters, making it ideal for streaming and transmission. Apart from removing spatial redundancies, the approximated light field is used to construct a Fourier disparity layers representation to further exploit other non-linear, temporal, intra and inter-view correlations present among the approximated sub-aperture images. Four scanning or view prediction patterns are utilized and the subsets in each pattern hierarchically construct the FDL representation and synthesize subsequent views. Iterative refinement and encoding with HEVC are followed by the final light field reconstruction. The complete end-to-end processing pipeline can flexibly work for multiple bitrates and is adaptable for a variety of multi-view autostereoscopic platforms. The compression performance of the proposed scheme is analyzed on real light fields. We achieved substantial bitrate savings compared to state-of-the-art codecs, while maintaining good reconstruction quality.

Computational multi-view displays involving light fields are a fast emerging choice for 3D presentation of real-world scenes. Tensor autostereoscopic glasses-free displays use just few light attenuating layers in front of a backlight to output high quality light field. We propose three novel schemes, Focal Stack - Hybrid Tucker-TensorSketch Vector Quantization (FS-HTTSVQ), Focal Stack - Tucker-TensorSketch (FS-TTS), and Focal Stack - Tucker Alternating Least-Squares (FS-TALS), for efficient representation, streaming and coding of light fields using a stack of differently focused images. Working with a focal stack instead of the entire light field majorly reduces the data acquisition cost as well as the computation and processing cost. Extensive experiments with real world light field focal stacks demonstrate that proposed novel one-pass Tucker decomposition using TensorSketch with hybrid vector quantization in FS-HTTSVQ, compactly represents the approximated focal stack in codebook form for better transmission and streaming. Encoding with High Efficiency Video Coding (HEVC) eliminates all intrinsic redundancies present in the approximated focal stack. Resultant low-rank approximated and coded focal stack is then employed to analytically optimize layer patterns for the tensor display. The complete end-to-end light field processing pipelines flexibly work for multiple bitrates and are adaptable for a variety of multi-view autostereoscopic platforms. Our schemes exhibit note-worthy performances on focal stacks compared to direct encoding of an entire light field using a standard codec like HEVC.

We present a novel hierarchical coding scheme for light fields based on transmittance patterns of low-rank multiplicative layers and Fourier disparity layers. The proposed scheme identifies multiplicative layers of light field view subsets optimized using convolutional neural networks for different scanning orders. Our approach exploits the hidden low-rank structure in the multiplicative layers obtained from the subsets of different scanning patterns. The spatial redundancies in the multiplicative layers can be efficiently removed by performing low-rank approximation at different ranks on the Krylov subspace. The intra-view and inter-view redundancies between approximated layers are further removed by HEVC encoding. Next, a Fourier disparity layer representation is constructed from the first subset of the approximated light field based on the chosen hierarchical order. Subsequent view subsets are synthesized by modeling the Fourier disparity layers that iteratively refine the representation with improved accuracy. The critical advantage of the proposed hybrid layered representation and coding scheme is that it utilizes not just spatial and temporal redundancies in light fields, but also efficiently exploits intrinsic similarities among neighboring sub-aperture images in both horizontal and vertical directions as specified by different predication orders. In addition, the scheme is flexible to realize a range of multiple bitrates at the decoder within a single integrated system. Comparison with state-of-the-art light field coders exhibits superior compression performance of the proposed scheme for real-world light fields. We achieve substantial bitrate savings and also maintain good light field reconstruction quality.

This paper presents a novel hierarchical coding scheme for light fields based on transmittance patterns of low-rank multiplicative layers and Fourier disparity layers. The proposed scheme learns stacked multiplicative layers from subsets of light field views determined from different scanning orders. The multiplicative layers are optimized using a fast data-driven convolutional neural network (CNN). The essential factor for multiplicative layers representation, which has not been considered in previous compression approaches, is the origin of redundancy, i.e., the low-rank structure of light field data. The spatial correlation in layer patterns is exploited with varying low ranks through factorization derived from singular value decomposition on a Krylov subspace. Further, encoding with HEVC efficiently removes intra-view and inter-view correlation in low-rank approximated layers. The initial subset of approximated decoded views from multiplicative representation is used to construct Fourier disparity layer (FDL) representation. The FDL model synthesizes the second subset of views identified by a pre-defined hierarchical prediction order. The correlations between the prediction residue of synthesized views are further eliminated by encoding the residual signal. The set of views obtained from decoding the residual is employed to refine the FDL model and predict the next view subsets with improved accuracy. This hierarchical procedure is repeated until all light field views are encoded. The critical advantage of the proposed hybrid layered representation and coding scheme is that it utilizes not just spatial and temporal redundancies but efficiently exploits the strong intrinsic similarities among neighboring sub-aperture images in both horizontal and vertical directions as specified by different predication orders. Besides, the scheme is flexible to realize a range of multiple bitrates at the decoder within a single integrated system. The compression performance analyzed with real light field shows substantial bitrate savings, maintaining good reconstruction quality.

To create a realistic 3D perception on glasses-free displays, it is critical to support continuous motion parallax, greater depths of field, and wider fields of view. A new type of Layered or Tensor light field 3D display has attracted greater attention these days. Using only a few light-attenuating pixelized layers (e.g., LCD panels), it supports many views from different viewing directions that can be displayed simultaneously with a high resolution. This paper presents a novel flexible scheme for efficient layer-based representation and lossy compression of light fields on layered displays. The proposed scheme learns stacked multiplicative layers optimized using a convolutional neural network (CNN). The intrinsic redundancy in light field data is efficiently removed by analyzing the hidden low-rank structure of multiplicative layers on a Krylov subspace. Factorization derived from Block Krylov singular value decomposition (BK-SVD) exploits the spatial correlation in layer patterns for multiplicative layers with varying low ranks. Further, encoding with HEVC eliminates inter-frame and intra-frame redundancies in the low-rank approximated representation of layers and improves the compression efficiency. The scheme is flexible to realize multiple bitrates at the decoder by adjusting the ranks of BK-SVD representation and HEVC quantization. Thus, it would complement the generality and flexibility of a data-driven CNN-based method for coding with multiple bitrates within a single training framework for practical display applications. Extensive experiments demonstrate that the proposed coding scheme achieves substantial bitrate savings compared with pseudo-sequence-based light field compression approaches and state-of-the-art JPEG and HEVC coders.

This study presents a scheme for efficient representation, coding and streaming of static or dynamic light fields using the authors’ novel hybrid Tucker-TensorSketch Karhunen–Loève transform-singular value decomposition via double sketching (HTTS-KLTSVD-DS) algorithm. A deep learning model is employed to obtain acquired images from the light fields by simulating coded aperture patterns. These acquired images can represent the entire light field and are low-rank approximated using HTTS-KLTSVD-DS. Incorporation of double sketching using TensorSketch allows the authors’ algorithm to work faster in a single pass itself and there is no need to store large Kronecker products of Tucker decomposition in the memory. This provides an efficient transmission and streaming adaptability of the light field, making it suitable for 3D display applications. Besides, compact representation of factor matrices by KLT-SVD in the authors’ proposed model acts as an optimal transform with good energy compaction property. Encoding of low-rank approximated acquired images using HEVC eliminates intra-frame, inter-frame and other intrinsic redundancies in the light field. The authors’ complete light field processing pipeline flexibly works for multiple bitrates and is adaptable for a variety of multi-view autostereoscopic platforms. Comparison with state-of-the-art codecs shows reasonable savings and PSNR gains for low and high bitrates, while maintaining good reconstruction quality.

Emerging computational light field displays are a suitable choice for realistic presentation of 3D scenes on autostereoscopic glasses-free platforms. However, the enormous size of light field limits their utilization for streaming and 3D display applications. In this paper, we propose a novel representation, coding and streaming scheme for dynamic light fields based on a novel Hybrid Tucker TensorSketch Vector Quantization (HTTSVQ) algorithm. A dynamic light field can be generated from a static light field to capture a moving 3D scene. We acquire images through different coded aperture patterns for a dynamic light field and perform their low-rank approximation using our HTTSVQ scheme, followed by encoding with High Efficiency Video Coding (HEVC). The proposed single pass coding scheme can incrementally handle tensor elements and thus enables to stream and compress light field data without the need to store it in full. Additional encoding of low-rank approximated acquired images by HEVC eliminates intra-frame, inter-frame and intrinsic redundancies in light field data. Comparison with state-of-the-art coders HEVC and its multi-view extension (MV-HEVC) exhibits superior compression performance of the proposed scheme for real-world light fields.