I’ve been mucking around with video stabilization for the past two weeks after a masters student got me interested in the topic. The algorithm is pretty simple yet produces surprisingly good stabilization for panning videos and forwarding moving (eg. on a motorbike looking ahead). The algorithm works as follows:
- Find the transformation from previous to current frame using optical flow for all frames. The transformation only consists of three parameters: dx, dy, da (angle). Basically, a rigid Euclidean transform, no scaling, no sharing.
- Accumulate the transformations to get the “trajectory” for x, y, angle, at each frame.
- Smooth out the trajectory using a sliding average window. The user defines the window radius, where the radius is the number of frames used for smoothing.
- Create a new transformation such that new_transformation = transformation + (smoothed_trajectory – trajectory).
- Apply the new transformation to the video.
Here’s an example video of the algorithm in action using a smoothing radius of +- 30 frames.
We can see what’s happening under the hood by plotting some graphs for each of the steps mentioned above on the example video.
This graph shows the dx, dy transformation for previous to current frame, at each frame. I’ve omitted da (angle) because it’s not particularly interesting for this video since there is very little rotation. You can see it’s quite a bumpy graph, which correlates with our observation of the video being shaky, though still orders of magnitude better than Hollywood’s shaky cam effect. I’m looking at you Bourne Supremacy.
Step 2 and 3
I’ve shown both the accumulated x and y, and their smoothed version so you get a better idea of what the smoothing is doing. The red is the original trajectory and the green is the smoothed trajectory.
It is worth noting that the trajectory is a rather abstract quantity that doesn’t necessarily have a direct relationship to the motion induced by the camera. For a simple panning scene with static objects it probably has a direct relationship with the absolute position of the image but for scenes with a forward moving camera, eg. on a car, then it’s hard to see any.
The important thing is that the trajectory can be smoothed, even if it doesn’t have any physical interpretation.
This is the final transformation applied to the video.
videostabKalman.cpp (live version by Chen Jia using a Kalman Filter)
You just need OpenCV 2.x or above.
Once compile run it from the command line via
Footages I took during my travels.