Chapter 4 - Tracking
The KLT tracker is a standard solution if you just need something robust and solid which tracks points of interest (here meaning corners) throughout a video sequence. While it seems ancient technology given the developments in computer vision, and there are certainly more fancy algorithms around nowadays, it is hard to beat in terms of speed vs. accuracy, simple to implement, and easily real-time on modern hardware. Moreover, for many applications, in particular structure-from-motion, it is just good enough, and there is no need for fifty-layer convolutional networks.
Table of contents
Part 1 - The KLT Tracker
Slides 1-24
We introduce the KLT tracker, a very robust, real-time method to track points of interest, i.e. corners, throughout a video sequence. The basic procedure consists of feature detection (structure tensor), inter-frame displacements (Lucas-Kanade optic flow), feature monitoring (patch comparison old vs. estimated new location). We briefly review those techniques, which we all know already, but introduce iterative refinement of Lucas-Kanade optical flow as a new idea. We note that over time, even for good features, the translational motion model becomes invalid, so we need something better here.
Index
00:00 Introduction, what is our view on tracking in this lecture?
05:50 Origins of the KL(S)T tracker, basic tracking procedure
10:00 Detour (just for fun): on papers and citation counts
17:50 Inter-frame displacement, iterative refinement in Lucas-Kanade
30:40 Remarks on image warping and interpolation
33:22 Feature selection based on which points probably can be tracked well
39:30 Feature monitoring: comparison of neighbourhoods in first and current frame
42:50 Summary, outlook: better motion model required
Part 2 - Affine Motion Model for Feature Monitoring
Slides 25-35
Over time, the constant motion model becomes invalid, although it might be a good approximation for frame-to-frame motion. Thus, we require a more sophisticated model for feature monitoring. Here, we investigate the affine model, where the flow varies across the patch according to an affine function of patch coordinates. We learn how to solve for the unknown coefficents of the model, again by minimizing a quadratic form, and verify that the performance when monitoring patches is indeed improved.
Index
00:00 Introduction, locally affine flow assumption
04:15 Mathematical formulation of the affine motion model
07:40 Examples for motion which are approximately affine
10:55 Solving for the coefficients of the model
16:20 Improvements when using the model in the KLT tracker
22:00 Summary and outlook