Chapter 3 - Local Models for Optic Flow

Optic flow is one of the most fundamental and intensely researched problems in computer vision, with a multitude of applications. We investigate why it is so important and what the key problems are in determining motion. A fundamental technical challenge is motion ambiguity, formalized in the aperture problem. We investigate several approaches to resolve the ambiguity, which collect additional information either in the spatial or spatio-temporal domain.

Table of contents

Part 1 - What is Optic Flow?

Slides 1-16

We give our working definition of optic flow as the motion of brightness patterns within the image plane, in contrast to the physical motion field. A difficulty in recovering optic flow is ambiguity of motion due to temporal sampling, which we need to counter by certain assumptions. We investigate the many example applications in computer vision and image anlysis, and introduce and discuss our key assumptions to arrive at practical algorithms to recover optic flow. Finally, we talk about some practical issues caused by the fact that we deal with digital images, and hint at a more fundamental issue to be investigated in later talks.


Index

00:00 Introduction, motion field (physical motion) vs. optic flow (apparant motion)
06:00 Key difficulty: ambiguity of motion and resulting ill-posedness
08:10 Example applications of optic flow
15:43 Key assumptions we make when designing algorithms for optic flow
22:55 Example scene to discuss realism of the assumptions
26:00 Practical issues: limitations based on how scenes are recorded
32:05 Outlook: a more fundamental issue is the aperture problem


Part 2 - The Aperture Problem and Normal Flow

Slides 17-26

A fundamental problem in determining optic flow is the aperture problem. One way to understand it on a conceptual level is that the motion of lines can not be determined by just looking locally, i.e. through a small aperture. To understand the problem on a more technical level, we move into the mathematics of optic flow. The most important equation is the optic flow constraint, which is derived from the gray value constancy assumption by linearizing the image. We show that optic flow in a single pixel can not be determined just from the optic flow constraint, the flow tangent to image edges is impossible to determine. If we resolve the ambiguity by just computing the component orthogonal to local edges, we end up with the normal flow, and terrible results. Clearly, some better idea is needed.


Index

00:00 Introduction: the aperture problem
05:30 Mathematics of optic flow: the optic flow constraint
20:00 Ambiguity of the OFC solution at a single point
26:00 Trying to resolve the ambiguity: the normal flow
29:40 Remark: computing discrete partial derivatives in time
30:22 Example results of the normal flow, which are terrible


Part 3 - The method by Lucas-Kanade

Slides 27-37

As an alternative to just computing the normal flow, Lucas-Kanade resolve the ambiguity by collecting also equations in a neighbourhood of a point to determine the flow vector at that location. The assumption is that within this small neighbourhood, the flow is constant. We arrive at an overdetermined system, since we have now more equations than unknowns, and solve it in a least squares sense, i.e. looking for the vector which minimizes the residual. However, we follow the original continuous approach by Lucas-Kanade and derive an analytic solution, which leads to two linear equations for the two unknowns. It turns out that if we want to determine the coefficients of the system in every point, we arrive at the structure tensor again. This is not surprising, as local solvability of flow is strongly linked to the local image structure: we can determine flow precisely if we have a corner-like structure, and fail if we are on an edge or in a flat region. Results are now much better than before, and we also can determine where the method failed.


Index

00:00 Introduction, the idea and assumptions of Lucas-Kanade's approach
04:30 Discrete interpretation of the continuous cost function to be minimized
12:20 Analytic approach to minimize the energy in the continuous setting
20:55 Final system of two equations for our two unknown components of the flow vector
25:54 Setting up the system in every pixel: the structure tensor appears again
34:05 Results of the method of Lucas-Kanade
36:10 Identifying regions where a unique solution exists, and where we run into the aperture problem
41:25 Example for pixel classification with regards to solvability of local flow
43:45 Discussion of advantages and disadvantages, outlook


Part 4 - Spatio-temporal methods, the method by Bigün et al.

Slides 38-49

We look at spatio-temporal approaches, which consider a three-dimensional neighborhood in space-time to arrive at solutions. We first understand spatio-temporal volumes by looking at different 2D slices through the volume. A first idea is a straight-forward generalization of Lucas-Kanade, where the equations are accumulated over the whole 3D volume. However, more interesting is the approach by Biguen et al., who find a direction orthogonal to the gradient as the direction of least change, which means that in this direction, grayscale values should be approximately constant. The resulting flow classification is more accurate, in particular, we can identify the case of conflicting flow directions, which typically appear at object boundaries.


Index

00:00 Introduction, the spatio-temporal volume
04:55 Idea of spatio-temporal approaches, spatio-temporal neighbourhoods
05:55 Spatio-temporal Lucas-Kanade
09:30 Spatio-temporal approach of Bigün et al.: idea and assumptions
11:50 Mathematical background: directional derivative
19:00 Formalizing the idea into a mathematical model
26:00 Solving the constrained minimization problem
30:25 Interpreting the result, checking whether we can find a solution
34:30 Example results, discussion
40:20 Chapter summary, outlook