Applying Filters To Camera Feed Using OpenCV And GStreamer

Introduction

Hello! 😃

In this tutorial, I will show you how to apply a few filters to the camera video feed using C++, OpenCV, and GStreamer.

Prerequisites

  • A Working installation of OpenCV and GStreamer.
  • A webcam or video input source.
  • A configured CMake environment.
  • Basic knowledge of C++ (and prefereably GStreamer).

Writing The Code

First we need to write the source code for the project. Create a new file called "main.cpp".

At the top of the file, include the headers that will be used by this project. Add the following header imports to the top of the file:

#include <iostream>
#include <opencv2/opencv.hpp>
#include <gst/gst.h>
#include <gst/app/gstappsink.h>

Next we'll define a method for each of the filters, in this tutorial I will show you how to implement the sepia, invert and grayscale filters.

void applySepia(cv::Mat &frame)
{
  cv::Mat temp = frame.clone();
  cv::transform(temp, frame, cv::Matx33f(
    0.272, 0.534, 0.131,
    0.349, 0.686, 0.168,
    0.393, 0.769, 0.189)
  );
}

void invertColors(cv::Mat &frame)
{
  cv::bitwise_not(frame, frame);
}

void applyGrayscale(cv::Mat &frame)
{
  cv::cvtColor(frame, frame, cv::COLOR_BGR2GRAY);
}

To process the frame with OpenCV we need to extract the video frame from the GStreamer pipeline and convert it so that it's processable with OpenCV. To do that we use the following function:

cv::Mat get_frame_from_sink(GstElement *sink, int &width, int &height)
{
  GstSample *sample = gst_app_sink_pull_sample(GST_APP_SINK(sink));

  if (!sample)
  {
    return cv::Mat();
  }

  if (width == 0 || height == 0)
  {
    GstCaps *caps = gst_sample_get_caps(sample);
    GstStructure *structure = gst_caps_get_structure(caps, 0);
    gst_structure_get_int(structure, "width", &width);
    gst_structure_get_int(structure, "height", &height);
  }

  GstBuffer *buffer = gst_sample_get_buffer(sample);
  GstMapInfo map_info;
  gst_buffer_map(buffer, &map_info, GST_MAP_READ);

  cv::Mat frame(height, width, CV_8UC4, map_info.data); // Dynamic resolution

  gst_buffer_unmap(buffer, &map_info);
  gst_sample_unref(sample);

  cv::cvtColor(frame, frame, cv::COLOR_BGRA2BGR);

  return frame;
}

The above creates a new OpenCV Mat object that matches the resolution of the frame in the GStreamer pipeline.

Finally we need to create the main method, which gets executed when the program is executed, the main function looks like this:

int main(int argc, char *argv[])
{
  gst_init(&argc, &argv);

  GstElement *pipeline = gst_pipeline_new("video-pipeline");
  GstElement *source = gst_element_factory_make("v4l2src", "video-source");
  GstElement *convert = gst_element_factory_make("videoconvert", "converter");
  GstElement *sink = gst_element_factory_make("appsink", "video-sink");

  if (!pipeline || !source || !convert || !sink)
  {
    std::cerr << "Failed to create elements." << std::endl;
    return -1;
  }

  g_object_set(G_OBJECT(sink), "emit-signals", TRUE, "caps", gst_caps_from_string("video/x-raw, format=(string)BGRx"), NULL);
  gst_bin_add_many(GST_BIN(pipeline), source, convert, sink, NULL);
  gst_element_link_many(source, convert, sink, NULL);

  gst_element_set_state(pipeline, GST_STATE_PLAYING);

  int width = 0, height = 0;
  cv::Mat frame, gray_frame;

  while (true)
  {
    frame = get_frame_from_sink(sink, width, height); // Pass width and height

    if (frame.empty())
    {
      std::cerr << "Empty frame!" << std::endl;
      break;
    }

    applyGrayscale(frame);
    cv::imshow("Filtered Frame", frame);

    if (cv::waitKey(1) == 27)
    {
      break;
    }
  }

  gst_element_set_state(pipeline, GST_STATE_NULL);
  gst_object_unref(GST_OBJECT(pipeline));

  return 0;
}

The above function creates a new GStreamer pipeline and generates three GStreamer elements, one is the video source which in this case is the user's web camera, two a converter in order to convert the video so it's usable by the third element the appsink.

Next we add the elements to the pipeline and link them all together. Once linked we set the status of the pipeline to playing.

Once everything is set we can start capturing frames from the camera, process them and apply the filter. Once the filter is set the frame is displayed to the user.

This continues until the "Esc" key is pressed on the keyboard, once pressed the program cleans up the elements and the program exits.

Phew! Now that we have got the code for the program next we need to create a CMakeLists.txt file in order to build and run the program.

Creating The CMakeLists.txt File

Next create the CMakeLists.txt file and fill it with the following:

cmake_minimum_required(VERSION 3.10)

# Project name
project(VideoFilter)

# Find required packages
find_package(OpenCV REQUIRED)
find_package(PkgConfig REQUIRED)
pkg_check_modules(GSTREAMER REQUIRED gstreamer-1.0)
pkg_check_modules(GST_APP REQUIRED gstreamer-app-1.0)

# Include directories
include_directories(${OpenCV_INCLUDE_DIRS})
include_directories(${GSTREAMER_INCLUDE_DIRS})
include_directories(${GST_APP_INCLUDE_DIRS})

# Link directories
link_directories(${OpenCV_LIBRARY_DIRS})
link_directories(${GSTREAMER_LIBRARY_DIRS})
link_directories(${GST_APP_LIBRARY_DIRS})

# Add executable
add_executable(VideoFilter main.cpp)

# Link libraries
target_link_libraries(VideoFilter ${OpenCV_LIBRARIES})
target_link_libraries(VideoFilter ${GSTREAMER_LIBRARIES})
target_link_libraries(VideoFilter ${GST_APP_LIBRARIES})

The comments basically describe what the above lines do, this file enables us to build the project and create an executable which we can run.

Save the above file and create a build directory:

mkdir build && cd build

Build the source code:

cmake ..
make

The program should build with no errors, and you should be able to run the project via the following command:

./VideoFilter

When done currently you should see your camera with a grayscale filter.

Done! 😄

Conclusion

In this tutorial I showed you how to apply three filters to a webcam video stream using C++, OpenCV and GStreamer. Feel free to play with the other two filters by replacing the call to the grayscale function.

Feel free to try add over filters and maybe even try streaming the filtered video etc.

I hope this tutorial as helped you as much as I had fun making it. 😆

As always you can find the source of this tutorial on my Github. https://github.com/ethand91/gstreamer-filter

Happy Coding! 😎

If you appreciate my work? I cover a variety of topics. For more, please like and follow me. Also, I love coffee.

“Buy Me A Coffee”

If you are looking to learn Algorithm Patterns to ace the coding interview I recommend the following course