1. Overview
In this tutorial, we’ll learn how to add frame interpolation to videos using FFmpeg.
First, we’ll learn how the frame interpolation works. Then, we’ll have a hands-on approach to adding frame interpolation to videos. For that, we’ll make use of the tblend and minterpolate video filters in FFmpeg.
2. How Does Frame Interpolation Work?
Frame interpolation increases the frame rate of a video, which can help improve the smoothness of motion.
It works by generating additional frames between existing frames in a video sequence. It’s useful in scenarios where the original footage has a lower frame rate.
Generally, it involves the following steps:
- Frame interpolation starts with the input video, which contains a sequence of discrete frames captured at a specific frame rate
- The video goes through an algorithm that analyzes the movements of objects between consecutive frames by detecting the motion vectors
- The algorithm generates intermediate frames by interpolating pixel values between the original frames
- The sequence usually flows through temporal filtering to maintain video quality by smoothing the interpolated frames
- An output video is produced with a higher frame rate, which appears smoother than the original
However, the result isn’t always solid. Sometimes, the process produces frames that appear distorted.
Besides that, frame interpolation can be computationally intensive when using advanced algorithms and deep learning models.
In the next sections, we’ll apply frame interpolation to a short video.
3. Frame Interpolation in Action
On Linux, we can use FFmpeg to carry out frame interpolation in a video. FFmpeg is a powerful command-line utility that lets us manipulate media files.
By default, it’s not installed on most Linux distributions. Nevertheless, it’s available on most official package repositories under its canonical name ffmpeg:
$ sudo apt install -y ffmpeg
Once installed, let’s verify it:
$ ffmpeg --version
ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers
For our example, we’ll use a demo video with the following metadata:
$ mediainfo buck-bunny.mp4
Video
ID : 1
Format : MPEG-4 Visual
Format profile : Simple@L1
Format settings, BVOP : No
Format settings, QPel : No
Format settings, GMC : No warppoints
Format settings, Matrix : Default (H.263)
Codec ID : mp4v-20
Duration : 6 s 400 ms
Bit rate mode : Variable
Bit rate : 933 kb/s
Maximum bit rate : 1 350 kb/s
Width : 640 pixels
Height : 368 pixels
Display aspect ratio : 1.739
Frame rate mode : Constant
Frame rate : 25.000 FPS
...
Additionally, let’s also note down the total number of frames in the video:
$ mediainfo --Inform="Video;%FrameCount%" buck-bunny.mp4
160
For frame interpolation, we’ll make use of two video filters: tblend and minterpolate.
3.1. tblend
We can perform temporal frame blending on a video:
$ ffmpeg -i buck-bunny.mp4 -filter:v tblend -r 120 buck-bunny_tblended.mp4
Let’s break this down:
- -i sets the input file to be buck-bunny.mp4
- -filter:v applies the tblend filter to the video
- -r sets the frame rate to 120
- buck-bunny_tblender.mp4 is the output video file
tblend performs temporal frame blending. It generates intermediate frames between the original frames to achieve the specified frame rate. In addition, it calculates the average of pixel values between two consecutive frames, which creates smoother transitions between frames.
For that reason, the total number of frames increases effectively:
$ mediainfo --Inform="Video;%FrameCount%" buck-bunny_tblended.mp4
767
3.2. minterpolate
minterpolate is another filter for the same purpose. However, it works differently from tblend. It works through motion estimation and optical flow techniques.
Let’s apply minterpolate to the same video:
$ ffmpeg -i buck-bunny.mp4 -filter:v minterpolate -r 120 buck-bunny_minterpolated.mp4
Next, let’s check the total frames:
$ mediainfo --Inform="Video;%FrameCount%" buck-bunny_minterpolated.mp4
759
3.3. tblend vs. minterpolate
Now, the question is, which one should we use? Well, both of these filters have tradeoffs. The minterpolate filter is more performance intensive, but the output is a more smooth and more high-quality video. So, it should be the go-to choice for slow-motion videos.
On the other hand, tblend reduces the flicker and visual artifacts in interlaced video. However, it’s not recommended for general frame rate conversion or slow-motion effects.
For demonstration, we’ll stitch both videos side by side into a single video. The one on the left uses tblend, while the other uses minterpolate:
$ ffmpeg -i buck-bunny_tblended.mp4 -i buck-bunny_minterpolated.mp4 \
-filter_complex "[0:v][1:v]hstack=inputs=2[v]" -map "[v]" \
-an interpolate-filter-comparison.mp4
Here’s the result:
Notably, there is a delay in the one on the left, while the other one is much more smoother. On the other hand, the one on the left has much better colors.
4. Conclusion
In this article, we learned about frame interpolation. We learned how frame interpolation works as well as how to apply frame interpolation to videos.
For that purpose, we discussed two frame interpolation filters offered by FFmpeg: tblend and minterpolate. Finally, we saw how these two filters are suitable for different scenarios.