Kaushik Mitra

Lensless cameras are ultra-thin imaging systems that replace the lens with a thin passive optical mask and computation. Passive mask-based lensless cameras encode depth information in their measurements for a certain depth range. Early works have shown that this encoded depth can be used to perform 3D reconstruction of close-range scenes. However, these approaches for 3D reconstructions are typically optimization based and require strong hand-crafted priors and hundreds of iterations to reconstruct. Moreover, the reconstructions suffer from low resolution, noise, and artifacts. In this work, we propose FlatNet3D-a feed-forward deep network that can estimate both depth and intensity from a single lensless capture. FlatNet3D is an end-to-end trainable deep network that directly reconstructs depth and intensity froma lensless measurement using an efficient physics-based3Dmapping stage and a fully convolutional network.Our algorithm is fast and produces high-quality results, whichwe validate using both simulated and real scenes captured usingPhlatCam.

The standard pipeline for many vision tasks uses a conventional camera to capture an image that is then passed to a digital processor for information extraction. In some deployments, such as private locations, the captured digital imagery contains sensitive information exposed to digital vulnerabilities such as spyware, Trojans, etc. However, in many applications, the full imagery is unnecessary for the vision task at hand. In this paper we propose an optical and analog system that preprocesses the light from the scene before it reaches the digital imager to destroy sensitive information. We explore analog and optical encodings consisting of easily implementable operations such as convolution, pooling, and quantization. We perform a case study to evaluate how such encodings can destroy face identity information while preserving enough information for face detection. The encoding parameters are learned via an alternating optimization scheme based on adversarial learning with deep neural networks. We name our system CAnOPIC (Camera with Analog and Optical Privacy-Integrating Computations) and show that it has better performance in terms of both privacy and utility than conventional optical privacy-enhancing methods such as blurring and pixelation.

We propose an improved saliency guided wavelet compression scheme for low-bitrate image/video coding applications. Important regions (faces in security camera feeds, vehicles in traffic surveillance) get degraded significantly at low bitrates by existing compression standards, such as JPEG/JPEG-2000/MPEG-4, since these do not explicitly utilize any knowledge of which regions are salient. We design a compression algorithm which, given an image/video and a saliency value for each pixel, computes a corresponding saliency value in the wavelet transform domain. Our algorithm ensures wavelet coefficients representing salient regions have a high saliency value. The coefficients are transmitted in decreasing order of their saliency. This allows important regions in the image/video to have high fidelity even at very low bitrates. Further, our compression scheme can handle several salient regions with different relative importance. We compare the performance of our method with the JPEG/JPEG-2000 image standards and the MPEG-4 video standard through two experiments: face detection and vehicle tracking. We show improved detection rates and quality of reconstructed images/videos using our Saliency Based Compression (SBC) algorithm.

Recent advancements in fields like Internet of Things (IoT), augmented reality, etc. have led to an unprecedented demand for miniature cameras with low cost that can be integrated anywhere and can be used for distributed monitoring. Mask-based lensless imaging systems make such inexpensive and compact models realizable. However, reduction in the size and cost of these imagers comes at the expense of their image quality due to the high degree of multiplexing inherent in their design. In this paper, we present a method to obtain image reconstructions from mask-based lensless measurements that are more photorealistic than those currently available in the literature. We particularly focus on FlatCam, a lensless imager consisting of a coded mask placed over a bare CMOS sensor. Existing techniques for reconstructing FlatCam measurements suffer from several drawbacks including lower resolution and dynamic range than lens-based cameras. Our approach overcomes these drawbacks using a fully trainable non-iterative deep learning based model. Our approach is based on two stages: An inversion stage that maps the measurement into the space of intermediate reconstruction and a perceptual enhancement stage that improves this intermediate reconstruction based on perceptual and signal distortion metrics. Our proposed method is fast and produces photo-realistic reconstruction as demonstrated on many real and challenging scenes.

Time-of-flight (ToF) imaging is an active method that utilizes a temporally modulated light source and a correlation-based (or lock-in) imager that computes the round-trip travel time from source to scene and back. Much like conventional imaging ToF cameras suffer from the trade-off between depth of field (DOF) and light throughput-larger apertures allow for more light collection but results in lower DoF. This trade-off is especially crucial in ToF systems since they require active illumination and have limited power, which limits performance in long-range imaging or imaging in strong ambient illumination (such as outdoors). Motivated by recent work in extended depth of field imaging for photography, we propose a focal sweep-based image acquisition methodology to increase depth-of-field and eliminate defocus blur. Our approach allows for a simple inversion algorithm to recover all-in-focus images. We validate our technique through simulation and experimental results. We demonstrate a proof-of-concept focal sweep time-of-flight acquisition system and show results for a real scene.

Transient imaging techniques capture the propagation of an ultra-short pulse of light through a scene, which in effect captures the optical impulse response of the scene. Recently, it has been shown that we can capture transient images using commercial, correlation imager based Time-of-Flight (ToF) systems. But the temporal resolution of these transient images are currently limited by high-speed electronics. In this paper, we propose 'Spatial Phase-Sweep' (SPS), a technique that exploits the speed of light to increase the temporal resolution of transient imaging beyond the limit imposed by electronic circuits in these commercial ToF sensors. SPS uses a linear array of light sources with a controlled spatial separation between these sources. The differential positioning of these sources introduce sub nano-second time shifts in the light wavefront, improving the time resolution of captured transients. As a proof of concept, we demonstrate a prototype which improves the temporal resolution of transient imaging by a factor of 10x, without any modification to the underlying electronics.