Department of Image Processing

Title: Inverse problems in image restoration: from basic principles to current deep-learning trends

Authors: Filip Šroubek, Jan Flusser and Barbara Zitová

Various types of degradation such as noise, blur, aliasing and geometric deformation are present in digital images and hamper further processing of images. The degradation results in smoothing high-frequency details, which makes the image analysis problematic. Heavy degradation may corrupt images to such an extent, that neither automatic analysis nor visual interpretation of the content are possible. Degradation often conveys valuable information both about the measuring device and the measured object. For example, motion blur gives us hints about the camera and/or object motion.

Two major approaches to handling degradation exist: restoration and invariants. They are more complementary rather than concurrent; each of them is appropriate for different tasks and employs different mathematical methods and algorithms.

Image restoration is one of the oldest areas of image processing. It appeared as early as in 1960's and 1970's in the work of the pioneers A. Rosenfeld, H. Andrews, B. Hunt, and others. In the last ten years, this area has received new impulses and has undergone a quick development. We have witnessed the appearance of multichannel techniques, blind techniques, and superresolution enhancement resolved by means of variational calculus in very high-dimensional spaces and recently also by deep learning (DL). A common point of all these methods is that they suppress or even remove the degradation from the input image and produce an image of a high visual quality. However, image restoration methods are often ill-posed and computationally expensive.

On the contrary, the invariant approach, proposed originally in 1995, works directly with the degraded data without any preprocessing. Degraded image is described by features, which are invariant with respect to noise, convolution and/or geometric deformation. Image analysis is then performed in the feature space. This approach is suitable for object recognition, template matching, and other tasks where we want to recognize/localize objects rather than to completely restore the image. The mathematics behind it is based on projection operators and moment invariants.

In this tutorial, we focus on both approaches. We start with modelling all types of degradation, analyzing their sources and providing recommendation to avoid them in practical applications. In the image restoration part of the tutorial, we review traditional as well as modern techniques for tackling problems of denosing, superresolution, image rectification and blind deconvolution in various forms. In the invariant part, we show invariants to noise, blur and geometric transformations. The tutorial is enriched with numerous demonstrations and practical examples. The content of the tutorial originates from speakers’ 20-year experience in image restoration, deconvolution, invariants, and related fields.

Outline:

There is no specific required knowledge of the tutorial participants except standard undergraduate courses of image processing and pattern recognition. The tutorial is self-contained and consists of four parts: modeling degradation, image restoration, invariants to degradation, and applications. 

1) Modeling degradation

• Relation between the true latent image u(x,y) and the degraded observed image g(x,y), g(x,y) = Hu(T(x,y)) +n(x,y), where H is the degradation operator, T is geometric transformation and n is additive noise.
• Types and sources of degradation: noise, aliasing, blur, geometric warping
• Common examples of blur: out-of-focus, motion, camera shake, or turbulence
• Space-invariant versus space-variant blur model
• Blind versus non-blind methods

2) Image restoration

• Denoising methods: bilateral filters, non-local means, DL approach
• Traditional non-blind deconvolution methods: Wiener filter, constrained optimization, role of image priors
• Single-channel blind deconvolution methods: maximum a posteriori method, marginalization and the Variational Bayesian strategy
• Multichannel deconvolution: a better-posed problem of multiple blurred observations
• Space-variant case: parametric models, patch-based approaches, DL approach
• Removing aliasing and increasing resolution by superresolution: single-channel and multichannel methods, DL approach
• Blur as a cue to understand complex object motion
• Image registration
• Numerical methods for solving non-linear systems of equations that appear in restoration problems

3) Invariants

• The notion of blur invariance
• Projection operators on kernel subspaces
• Blur invariants in frequency domain. The notion of the primordial image. Particular cases for centrosymmetric, radial, N-fold symmetric, dihedral, and Gaussian blur
• Invariants to Gaussian blur reformulated as invariants to additive Gaussian noise
• Invariants to geometric transformation: rotation and affine

4) Applications

• Numerous practical applications of signal restoration as well as of blur invariants will be presented during the tutorial. We show the use in monitoring atmospheric pollution, remote sensing, astronomy, security, forensic imaging, and biomedical imaging. We will also demonstrate an application in consumer photography implemented in smartphones.
• The invariants to blur have found successful applications in recognition of noisy signals, solution of heat equation, in face recognition on out-of-focused photographs, in normalizing blurred images into the canonical forms, in template-to-scene matching of satellite images, in blurred digit and character recognition, in registration of images obtained by digital subtraction angiography, and in focus/defocus quantitative measurement. Many of these applications will be presented in the tutorial.

Tutorial slides are available here.