A new approach is proposed, capable of restoring a single high-quality image from a given image sequence distorted by atmospheric turbulence. This approach reduces the space and time-varying deblurring problem to a shift invariant one. It first registers each frame to suppress geometric deformation through non-rigid registration. Next, a temporal regression (fusion) process is carried out to produce an image from the registered frames, which can be viewed as being convolved with a space invariant near-diffractionlimited blur. Finally, a blind deconvolution algorithm is implemented to deblur the fused image, generating a high quality output. Experiments using real data illustrate that this approach can effectively alleviate blur and distortions, recover details of the scene, and significantly improve visual quality.
We formulate the fundamental limits of image denoising using a statistical framework. Next, we propose a practical algorithm with the explicit aim of achieving the bound. The proposed method is a patch-based Wiener filter that takes advantage of both geometrically and photometrically similar patches. The resultant approach has a solid statistical foundation while producing denoising results that are comparable to or exceeding the current state-of-the-art, both visually and quantitatively.
Filtering 2- and 3-D data (i.e. images and video) is fundamental and common to many fields. From graphics, machine vision and imaging, to applied mathematics and statistics, important innovations have been introduced in the past decade which have advanced the state of the art in applications such as denoising and deblurring. While the impact of these contributions has been significant, little has been said to illuminate the relationship between the theoretical foundations, and the practical implementations. Furthermore, new algorithms and results continue to be produced, but generally without reference to fundamental statistical performance bounds. In this talk, I will present a wide-angle view of filtering, from the practical to the theoretical, and discuss performance analysis and bounds, indicating perhaps where the biggest (if any) improvements can be expected. Furthermore, I will describe various broadly applicable techniques for improving the performance of any given denoising algorithm in the mean-squared error sense, without assuming prior information, and based on the given noisy data alone.
Recent developments in computational imaging and restoration have heralded the arrival and convergence of several powerful methods for adaptive processing of multidimensional data. Examples include Moving Least Square (from Graphics), the Bilateral Filter and Anisotropic Diffusion (from Vision), Boosting and Spectral Methods (from Machine Learning), Non-local Means (from Signal Processing), Bregman Iterations (from Applied Math), Kernel Regression and Iterative Scaling (from Statistics). While these approaches found their inspirations in diverse fields of nascence, they are deeply connected. In this talk, I present a practical and unified framework for understanding some common underpinnings of these methods. This leads to new insights and a broad understanding of how these diverse methods interrelate. I also discuss several applications, and the statistical performance of the resulting algorithms. Finally I briefly illustrate connections between these techniques and classical Bayesian approaches. An updated version of the slides for this talk is available here.
More or less the same abstract as the above! Slides available also here.
Locally Adaptive Regression Kernels (LARKs) are visual descriptors that adapt to local characteristics of the given data, capturing both the spatial density of the data samples ("the geometry"), and the actual values of those samples ("the photometry"). These descriptors are exceedingly robust in expressing the underlying structure of multidimensional signals even in the presence of significant noise, missing data, and other disturbances. As the framework does not rely upon strong assumptions about noise or signal models, it is applicable to a wide variety of problems. Of particular interest in two and three dimensions are state of the art denoising and upscaling of images and video, and novel applications in computer vision such as "visual search".
We present a generic detection/localization algorithm capable of searching for a 2- or 3-D visual object of interest without training. The proposed method operates using a single example (query) of an object of interest to find similar matches; does not require prior knowledge (learning) about objects being sought; and does not require any pre-processing step or segmentation of a target image/video. Our method is based on the computation of local regression kernels as descriptors from a query, which measure the likeness of a pixel to its surroundings. State of the art performance is demonstrated on several challenging datasets, indicating successful detection of objects in diverse contexts and under different imaging conditions.
We propose K-LLD: a patch-based, locally adaptive denoising method based on clustering the given noisy image into regions of similar geometric structure. In order to effectively perform such clustering, we employ as features the local weight functions derived from our earlier work on steering kernel regression. These weights are exceedingly informative and robust in conveying reliable local structural information about the image even in the presence of significant amounts of noise. Next, we model each region (or cluster) – which may not be spatially contiguous – by “learning” a best basis describing the patches within that cluster using principal components analysis. This learned basis (or “dictionary”) is then employed to optimally estimate the underlying pixel values using a kernel regression framework. An iterated version of the proposed algorithm is also presented which leads to further performance enhancements. We also introduce a novel mechanism for optimally choosing the local patch size for each cluster using Stein’s Unbiased Risk Estimator (SURE). We illustrate the overall algorithm’s capabilities with several examples. These indicate that the proposed method appears to be competitive with some of the most recently published state of the art denoising methods.
We have developed a class of robust nonparametric estimation methods which are ideally suited for the reconstruction of signals and images from noise-corrupted and sparse or irregularly sampled data. The framework results in locally adapted kernels which take into account both the spatial density of the available samples, and the actual values of those samples. As such, they are automatically steered and adapted to both the given sampling "geometry", and the samples' "radiometry". As the framework we propose does not rely upon strong assumptions about noise or sampling distributions, it is applicable to a wide variety of problems, including image upscaling, high quality interpolation from irregular, sparse and noisy samples, state of the art denoising, and deblurring.
We introduce a novel framework for adaptive enhancement and upscaling of videos containing complex activities based on multidimensional kernel regression. In this framework, each pixel in the video sequence is approximated with a 3-D local (Taylor) series, capturing the essential local behavior of its spatio-temporal neighborhood. The coefficients of this series are estimated by solving a local weighted least-squares problem, where the weights are a function of the 3-D space-time orientation in the neighborhood. As this framework is fundamentally based upon the comparison of neighboring pixels in both space and time, it implicitly contains information about the local motion of the pixels across time. The proposed approach not only significantly widens the applicability of super-resolution methods to a broad variety of video sequences containing complex motions, but also yields improved overall performance.
In addressing the problem of estimating the relative motion between the frames of a video sequence. In comparison with the commonly applied pairwise image registration methods, the proposed method considers global consistency conditions for the overall multi frame motion estimation problem, and is more accurate. We review the recent work on this subject and propose an optimal framework, which directly applies the consistency conditions as both hard constraints in the estimation problem, or as soft constraints in the form of stochastic (Bayesian) priors. The framework is applicable to virtually any motion model and enables us to develop a robust approach, which is resilient against the effects of outliers and noise, and therefore useful for demanding applications such as superresolution.
In this talk, I present an overview of our work in the statistical analysis of resolution, its computational enhancement in imaging, and its inherent fundamental limits. On the computational image reconstruction front, we have developed efficient, and statistically optimal algorithms for superresolution from video, spanning a wide range of problems in this field, including robust multi-frame image fusion, simultaneous demosaicing and super-resolution, and color video-to-video super-resolution.On the more fundamental analysis, having developed a proper statistical definition of resolution, we have been able to analyze the resolution/SNR tradeoff for general optical imaging systems, deriving a practically useful scaling law relating them. Furthermore, we have performed related analysis demonstrating and quantifying the deep interdependence between the motion estimation and image reconstruction problems, thereby yielding insight into fundamental limits of each, and how the knowledge of one set of parameters affects the other. I touch on most of the above inter-related topics, and describe some of them at higher resolution.
Optical microlithography, a technique similar to photographic printing, is used for transferring circuit patterns onto silicon wafers. The above process introduces distortions arising from optical limits and non-linear resist effects, leading to poor pattern fidelity and yield loss. The input to the above system is a photo-mask (or reticle), which can be controlled (or engineered) such that it cancels out (or compensates for) the process losses to come. This forms the basis of optical proximity correction (OPC) and phase shift masks (PSM), two commonly employed resolution enhancement techniques for patterning very small features (close to the optical limit). In this talk, we discuss a novel inverse lithography technology (ILT) framework to synthesize OPC and PSM for high-fidelity patterning of random logic 2-D features. ILT attempts to synthesize the input mask which leads to the desired output wafer pattern by inverting the mathematical forward model from mask to wafer. Our framework employs a pixel-based mask parameterization, continuous function formulation, and analytic gradient-based optimization techniques to synthesize the masks. We also introduce a regularization framework to control the tone and complexity of the synthesized masks, and inculcate other user-defined properties. The results indicate automatic generation and placement of assist bars, which are very popular in the semiconductor industry. We conclude by briefly discussing ILT-based mask design for double exposure lithography systems, which are deemed as key technology enablers for the future.
In this talk we address the problem of reconstructing the shape of a convex object from measurements of the areas of its shadows in several directions. This type of very weak measurement is sometime referred to as the brightness function of the object, and may be observed in an imaging scenario by recording the total number of pixels where the object's image appears. Related measurements, collected as a function of viewing angle, are also referred to as "lightcurves" in the astrophysics community, and are employed in estimating the shape of atmosphereless rotating bodies (e.g. asteroids). We address the problem of shape reconstruction from brightness functions by constructing a least-squares optimization framework for approximating the underlying shapes with polygons in two dimensions, or polyhedra in three dimensions, from noisy, and possibly sparse measurements of the brightness values.