Skip to content
This repository was archived by the owner on Jan 3, 2023. It is now read-only.

Getting Started Guide [v0.4 2020]

Jump to bottom
Lin Xie edited this page Jan 8, 2020 · 3 revisions

Table of contents

1.  Introduction
2.  Acquire data and sources
      2.1.  Clone FFmpeg* Video Analytics patches repository
      2.2.  Download models in IR format
      2.3.  Prepare video files
3.  Install FFmpeg* Video Analytics plugin
      3.1.  Install in Docker* image
      3.2.  Install on host machine
4.  Run samples
5.  Advanced usage
      5.1.  Use Caffe* models
      5.2.  Use TensorFlow* models

1. Introduction

This Getting Started Guide provides detailed instructions on how to start working with FFmpeg* Video Analytics plugin (FFVA plugin). The FFVA plugin contains multiple FFmpeg elements(filter/muxer) to construct various pipelines. Some elements can run inference operations (detection, classification, identification), other elements can output inference results to screen (for instance, as overlay on top of the video) and dump results to file or publish results to popular message brokers (Kafka* ).
Consider a simple FFmpeg pipeline:

ffmpeg -i test.mp4 -vf detect=model=face-detection-adas-0001.xml:model_proc=face-detection-adas-0001.json,classify=model=emotions-recognition-retail-0003.xml:model_proc=emotions-recognition-retail-0003.json -an -f null -

This pipeline contains typical usage of ffmpeg command line: -i test.mp4 specifies the input source location and a software decoder will be created implicitly for decoding. -vf indicates the following section contains one or more filters to do raw data post processing.

  • FFVA filter detect uses provided model to perform detection on each frame of the video file test.mp4. detect generates regions of interest (ROIs) - a list of regions where detection happened. These ROIs are used by the the next filter in the pipeline - classify.
  • FFVA filter classify uses provided model and model-proc to classify on each ROIs on video frame. model-proc is a json file which configures output layer name and labels list. classify refers to this file to interpret the results correctly.
    Inorder to show the results on top of a playing video, we need to use another tool ffplay and the command looks like:
ffplay -i test.mp4 -vf detect=model=face-detection-adas-0001.xml:model_proc=face-detection-adas-0001.json,classify=model=emotions-recognition-retail-0003.xml:model_proc=emotions-recognition-retail-0003.json,ocv_overlay
  • You may have noticed that filter ocv_overlay is added into the filter chain, now it can support drawing detected bounding box and labels on top of the video. This article is divided into the following sections:

Paragraph 2 Acquire data and sources describes what data to prepare on host machine. FFVA plugin uses OpenVINO™ Toolkit's Inference Engine which uses CNN (Convolutional Neural Network) models in IR (Intermediate Representation) format. IR format includes a pair of files: .xml for describing topology and .bin containing weights. The input video files will be also required.

Paragraph 3 Install FFmpeg Video Analytics plugin describes two options to install FFVA plugin: on a Docker image or on host machine. If you choose Docker image, you can build it through OpenVisualCloud/Dockerfiles. If you choose host machine, you will need to build FFVA plugin from source.

Paragraph 4 Run samples describes how to run samples. The data preparation for this section is described in section 2. For Docker image, the folders including video files and model files are mounted from host and environment variables are set; for host machine, only environment variables are set.

Paragraph 5 Advanced usage describes more FFVA plugin samples. It will show examples of how to convert the original CNN models in Caffe* and TensorFlow* into IR format by OpenVINO™ Toolkit's Model Optimizer. The converted models will be used by the FFVA inference filters.

2. Acquire data and sources

First, define you workspace path and create directories to store the data:

mkdir ~/ffva
mkdir ~/ffva/data
mkdir ~/ffva/data/models
mkdir ~/ffva/data/models/intel
mkdir ~/ffva/data/models/common
mkdir ~/ffva/data/video

2.1. Clone FFmpeg Video Analytics patches repository 

# Download FFVA patch repository
git clone -b ffmpeg4.2_va https://github.com/VCDP/FFmpeg-patch.git ~/ffva/FFmpeg-patch

2.2. Download models in IR format

Models in IR format can be downloaded by Model Downloader tool of Open Model Zoo (OMZ). IR model is the format which can be used directly by samples, please use following commands to download a set of IR models used by section 4.

# Download OMZ to have Model Downloader
git clone https://github.com/opencv/open_model_zoo.git ~/ffva/open_model_zoo

# Download models listed in intel_models_for_samples.LST into ~/ffva/data/models/intel folder
cd ~/ffva/open_model_zoo/tools/downloader
python3 downloader.py --list ~/ffva/FFmpeg-patch/samples/model_downloader_configs/intel_models_for_samples.LST -o ~/ffva/data/models

The above commands will download several models in IR format which are ready to be used with FFVA filters and which are required for samples to run.

2.3. Prepare video files

FFVA plugin is not distributed with any video files. You can supply your own or use freely licensed content from sites like Pexels. Any video with cars, pedestrians, humans, human faces can be used. Refer to following commands to copy downloaded video files to ~/ffva/data/video directory:

# uncomment below and use with appropriate videos path
# cp <path/to/downloaded/videos/folder>/* ~/ffva/data/video

3. Install FFmpeg Video Analytics plugin

There are two options to install FFVA plugin:

  • Install in Docker image (recommended). Installation of FFVA plugin and its dependencies will be performed in a docker image on the host. Host environment will not be mofidied; FFVA plugin will be used from inside a running docker container.
  • Install on host machine FFVA plugin and its dependencies will be installed and built directly on host. This will modify the host system. Installation process will take more time.

3.1. Install in Docker image

3.1.1. Support Matrix:

The project supports the following platforms and OS'es:

Supported Platforms Supported OS'es
Xeon Ubuntu 16.04 LTS, Ubuntu 18.04 LTS, CentOS 7.4, 7.5, and 7.6
Xeon E3 Ubuntu 16.04 LTS, Ubuntu 18.04 LTS, CentOS 7.4, 7.5, and 7.6
VCA2 Ubuntu 16.04 LTS, Ubuntu 18.04 LTS, CentOS 7.4, 7.5, and 7.6
VCAC-A Ubuntu 16.04 LTS, Ubuntu 18.04 LTS

3.1.2. Host Platform Setup:

  • Follow the instructions to setup host date and time.
  • Follow the instructions to install docker.ce.
  • If you are behind a firewall, setup proxy as follows:
(1) sudo mkdir -p /etc/systemd/system/docker.service.d    
(2) printf "[Service]\nEnvironment=\"HTTPS_PROXY=$https_proxy\" \"NO_PROXY=$no_proxy\"\n" | sudo tee /etc/systemd/system/docker.service.d/proxy.conf    
(3) sudo systemctl daemon-reload     
(4) sudo systemctl restart docker

3.1.3. Build docker image:

Option 1:

You can choose to use Dockerfiles in OpenVisualCloud/Dockerfiles

git clone https://github.com/OpenVisualCloud/Dockerfiles.git Dockerfiles
cd Dockerfiles
mkdir build && cd build     
cmake ..     
cd Xeon/ubuntu-18.04/analytics/ffmpeg # please build your specific <_platform_>/<_OS_>/<_usage_>/<_image_> only as a full build takes a long time.     
make # build on the target processor for best performance.    
ctest # optional

Option 2 (recommend):

Use the Dockerfile under ~/ffva/FFmpeg-patch/docker

Build Docker image from sources

Run the following commands to start build process:

  1. If you want to use open source version of OpenVINO™ Toolkit, please use this command:
cd ~/ffva/FFmpeg-patch/docker  # was cloned on step 2.1 of current instruction
./build_docker_image.sh opensource
  1. If you want to use binary version, please use this command:
cd ~/ffva/FFmpeg-patch/docker  # was cloned on step 2.1 of current instruction
./build_docker_image.sh package

Note: if you want to use HDDL/HKH device, you should use binary version (option #2)

The build process runs automatically. It will build Docker image ffmpeg-video-analytics:latest.
When build process is completed, go to Run Docker image

3.1.4. Run Docker image

Validate Docker image openvisualcloud/xeon-ubuntu1804-analytics-ffmpeg or ffmpeg-video-analytics:

docker images | grep analytics | grep latest

This command will return a line with image xeon-ubuntu1804-analytics-ffmpeg:latest or ffmpeg-video-analytics description. If the image is absent in your output, please take a step back to Install in Docker image.
If validation is successfull, the docker image is ready to run. There are two options to run the Docker image:

For VCAC-A, please refer to the following link for the extra steps to setup VCAC-A to run docker containers:
https://github.com/OpenVisualCloud/Dockerfiles/blob/master/VCAC-A/README.md

Run Docker image with command

Some FFVA plugin samples use display, so the following command should be used to connect the host's system X server from the Docker image:

xhost local:root

Run container:

docker run -it --privileged --net=host \
\
-v ~/.Xauthority:/root/.Xauthority \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-e DISPLAY=$DISPLAY \
-e HTTP_PROXY=$HTTP_PROXY \
-e HTTPS_PROXY=$HTTPS_PROXY \
-e http_proxy=$http_proxy \
-e https_proxy=$https_proxy \
\
-v ~/ffva/data/models/intel:/root/intel_models:ro \
-v ~/ffva/data/models/common:/root/common_models:ro \
-e MODELS_PATH=/root/intel_models:/root/common_models \
\
-v ~/ffva/data/video:/root/video-examples:ro \
-e VIDEO_EXAMPLES_DIR=/root/video-examples \
\
ffmpeg-video-analytics-<type>

Here is the meaning of some options in the command:

  • Option --privileged is required for Docker container to access the host system's GPU
  • Option --net=host provides host network access to container. It is needed for correct interaction with X server
  • Files ~/.Xauthority and /tmp/.X11-unix mapped to the container are needed to ensure smooth authentication with X server
  • -v instances are needed to map host system directories inside Docker container
  • -e instances set Docker container environment variables. Samples need them some of them set correctly to operate. Proxy variables are needed if host is behind firewall.

After command docker run, you should be inside the Docker image with FFVA plugin installed; now go to Run samples.

Note: if you want to use video from web camera as an input in sample face_detection_and_classification.sh you should mount the device with this command (add this command when running the container):

-v /dev/video0:/dev/video0

Now you can run the sample with video from web camera:

./face_detection_and_classification.sh /dev/video0

Run Docker image with script

Run help message and print the script usage:

cd ~/ffva/FFmpeg-patch/docker  
./run_docker_container.sh -h  # call help

Example output:

usage: sudo ./run_docker_container.sh [--video-examples-path=<path>]
[--intel-models-path=<path>] [--models-path=<path>] [--image-name=<name>]

Run script:

cd ~/ffva/FFmpeg-patch/docker  # essential
export DATA_PATH=~/ffva/data  # essential
sudo ./run_docker_container.sh --video-examples-path=$DATA_PATH/video --intel-models-path=$DATA_PATH/models/intel --models-path=$DATA_PATH/models/common --image-name=ffmpeg-video-analytics-<type>

After running the script, you should be inside the Docker image with FFVA plugin installed. Now go to Run samples.

3.2. Install on host machine

The instruction of this section is validated on Ubuntu 16.04 & 18.04

Build dependencies

Please install the required system packages:

#build tools
sudo apt update && sudo apt install -y --no-install-recommends \
build-essential autoconf automake cmake make git wget pciutils cpio libtool lsb-release \
ca-certificates pkg-config bison flex libcurl4-gnutls-dev zlib1g-dev  

#ffmpeg dependencies
sudo apt-get install -y -q --no-install-recommends nasm yasm libass-dev libfreetype6-dev \
libvdpau-dev libsdl2-dev libxcb1-dev libxcb-shm0-dev libxcb-xfixes0-dev zlib1g-dev

Install third party codec [OPTIONAL]

x264

sudo apt-get install libx264-dev  #if you want to use the latest version, please compile with source code

x265

sudo apt-get install libx265-dev libnuma-dev  #if you want to use the latest version, please compile with source code

svt hevc/vp9/av1

please refer the instructions from section build-and-install in
SVT-HEVC
SVT-VP9
SVT-AV1

install libvpx/libfdk-aac/libmp3lame/libopus/libaom

Please refer the steps in ffmpeg wiki Compilation & Installation to install other codes you need.
If you meet git clone error: gnutls_handshake() failed: An unexpected TLS packet was received, please do

git config --global http.proxy http://<my_proxy>:<my_port>
git config --global https.proxy https://<my_proxy>:<my_port>

Download and install OpenVINO™ Toolkit 2019 R3

Please register and download 2019 R3 OpenVINO™ Toolkit for Linux*. At the installation wizard, you will be offered 2 installation options. Please choose customizable package. When choosing components to install, exclude OpenVX* and Intel® Media SDK. After this, keep the default installation settings and start the installation process. When installation finishes, you will be redirected to the OpenVINO™ Toolkit Installation Guide. You may return to this Getting Started Guide to continue FFVA plugin installation process.

Now please run the installation script:

sudo ./install_GUI.sh

Install OpenVINO™ Toolkit dependencies

After successful installation you should get root folder /opt/intel/openvino (if you did not change it during OpenVINO™ Toolkit installation). Navigate to this folder and install dependencies:

cd /opt/intel/openvino/install_dependencies/
sudo -E ./install_openvino_dependencies.sh

Install OpenVINO™ C API library

Before you install C API library, please make sure your CMake version is higher than 3.13. You can update cmake by script ~/ffva/FFmpeg-patch/script/install_cmake.sh.
So far there's no C interface support in OpenVINO™ inference engine, a temporary bridge is implemented and the source is placed at thirdpart/dldt-c-api/source, please follow the script to install it:

cd ~/ffva/FFmpeg-patch/scripts && ./install_ie_c_api.sh

If you are going to use Gen based graphics to accerlate the media processing(encode/decode/vpp/inference), following packages must be installed

Install libdrm, libva, libva-utils, media driver

please refer the script below to install it:

cd ~/ffva/FFmpeg-patch/scripts && ./build_install_libva_media_driver.sh

Install OpenCL™

Check if OpenCL™ drivers are installed:

sudo apt-get install -y clinfo && clinfo

If output message includes "Number of platforms: 0", the driver is not installed yet, so go here and install the latest available OpenCL™ drivers version, or use the script provided by OpenVINO™ Toolkit

cd /opt/intel/openvino/install_dependencies/
sudo -E ./install_NEO_OCL_driver.sh

Setup OpenVINO™ Toolkit environment variables into current shell:

source /opt/intel/openvino/bin/setupvars.sh

Check components installation

Before proceeding further, please reboot your machine. After the reboot please check components installation as shown below.

  • Check OpenCL™: 
    clinfo

The output should include "Number of platforms: x", where "x" is a non-zero integer.

  • Check VA-API:
    Run vainfo and check that iHD driver is found and initialized successfully 
    vainfo

The output shouldn't contain any error messages, iHD driver must be found. If no errors occur, please proceed further to the next step.

If you receive error message like the one below, please reboot your machine and try again.

libva info: va_openDriver() returns -1
vaInitialize failed with error code -1 (unknown libva error), exit

If you receive any other errors, please retry installation process. If it's not even after re-installation, please submit an issue for support on GitHub here

Install message brokers [OPTIONAL]

Installation of message brokers will enable capability of metapublish muxer to publish inference results to Kafka. Run following script to install it:

cd ~/ffva/FFmpeg-patch/scripts/  # was cloned on step 2.1 of current instruction
./install_metapublish_dependencies.sh

Build ffmpeg with FFVA plugin

With all dependencies installed, proceed to build FFVA plugin:

source /opt/intel/openvino/bin/setupvars.sh

# get ffmpeg code base
FFMPEG_VER=n4.2
FFMPEG_REPO=https://github.com/FFmpeg/FFmpeg/archive/${FFMPEG_VER}.tar.gz
cd ~/ffva && wget -O - ${FFMPEG_REPO} | tar xz && mv FFmpeg-${FFMPEG_VER} ffmpeg 

# apply analytics patches
cd ~/ffva/ffmpeg
find ~/ffva/FFmpeg-patch/patches -type f -name '*.patch' -print0 | sort -z | xargs -t -0 -n 1 patch -p1 -i

# Build ffmpeg
# copy one of the scripts to ffmpeg source root directory and run it
# For static libraries refer:
~/ffva/FFmpeg-patch/scripts/build_ffmpeg-static.sh
# (Recommend) For dynamic libraries refer:
~/ffva/FFmpeg-patch/scripts/build_ffmpeg-dynamic.sh

One more thing, if you want to draw the results of analytics, such as bounding box, and overlay on top of the original video, please install OpenCV* 4.1 and add append option --enable-libopencv when run configure.

cd ~/ffva/FFmpeg-patch/scripts && ./install_opencv.sh
export CPATH=$CPATH:/usr/local/include/opencv4/
sudo cp ~/ffva/FFmpeg-patch/docker/patch/opencv.pc /usr/local/lib/pkgconfig/
sudo cp ~/ffva/FFmpeg-patch/docker/patch/cvdef.h /usr/local/include/opencv4/opencv2/core/cvdef.h

Note: the command source /opt/intel/openvino/bin/setupvars.sh is required each time to use FFVA plugin. To set these environment variables automatically, you can add these commands to file ~/.bashrc.

The manual installation on the host machine may have environment issues; if this happens, please consider other ways we recommend - Install in Docker image.

Also, you can create an issue on GitHub here.

4. Run samples

Before running samples, let’s confirm that your environment is properly configured

FFVA plugin installation check

Check the environment:

env | grep MODELS_PATH  # should output one ore more directories. Check that these directories contain models as expected
env | grep VIDEO_EXAMPLES_DIR  # should output directory. Check that this directory contains videos as expected

Check FFVA plugin installation:

ffmpeg -filters | grep Inference # should output FFVA filters like classify, detect, etc.

The last command should output FFVA elements like detect, classify, etc. If not, the environment is not ready to run samples. Please check if you missed any steps above.

Run samples

If you used host machine to build FFVA plugin, please use the following command to run the basic samples:

cd ~/ffva/FFmpeg-patch  # was cloned on step 2.1 of current instruction
source /opt/intel/openvino/bin/setupvars.sh

# Here and further, running samples, you need to enter the name of your video instead of "your-video".
./samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/your-video  # shell samples.

When running sample, a video with object detection and attributes recognition (bounding boxes around faces with recognized attributes) should be played.

Sample face_detection_and_classification.sh supports input from web camera. To try this feature, please run:

./face_detection_and_classification.sh /dev/video0

Folder ~/ffva/FFmpeg-patch (was cloned during step 2.1) should include a list of shell samples in sub-directory ~/ffva/FFmpeg-patch/samples/shell which is described in README.md file. You can also check this document online.

Note:

  • Always make sure to configure OpenVINO™ Toolkit environment variables with file "setupvars.sh" before running samples:
source /opt/intel/openvino/bin/setupvars.sh

5. Advanced usage

The examples in this section include extra steps to download the original model from public site and convert it to IR format for usage by the FFVA plugin. This shows how to use FFVA plugin in real scenario.

git clone https://github.com/opencv/dldt.git ~/ffva/dldt  # contains folder model-optimizer
pip install tensorflow==1.12.0 --upgrade

5.1. Use Caffe models

Example 1. Run object detection with ssd300 model

mkdir -p ~/ffva/data/models/non_ir/ssd300  # to hold downloads
cd ~/ffva/open_model_zoo/tools/downloader  # was cloned on step 2.2 of current instruction
python3 downloader.py --name ssd300 -o ~/ffva/data/models/non_ir/ssd300  # download ssd300

# convert model
cd ~/ffva/dldt/model-optimizer  # was cloned on step 5.1 of current instruction
python3 mo.py --framework caffe --input_model ~/ffva/data/models/non_ir/ssd300/public/ssd300/models/VGGNet/VOC0712Plus/SSD_300x300_ft/VGG_VOC0712Plus_SSD_300x300_ft_iter_160000.caffemodel --input_proto ~/ffva/data/models/non_ir/ssd300/public/ssd300/models/VGGNet/VOC0712Plus/SSD_300x300_ft/deploy.prototxt --data_type FP32 --mean_values data[104.0,117.0,123.0] --output_dir ~/ffva/data/models/common/ssd300-fp32/

# if you meet ImportError: No module named 'generator', please install following packages via
pip3 install coverage m2r pyenchant pylint Sphinx safety test-generator

# give handy names
mv ~/ffva/data/models/common/ssd300-fp32/VGG_VOC0712Plus_SSD_300x300_ft_iter_160000.xml ~/ffva/data/models/common/ssd300-fp32/ssd300.xml
mv ~/ffva/data/models/common/ssd300-fp32/VGG_VOC0712Plus_SSD_300x300_ft_iter_160000.bin ~/ffva/data/models/common/ssd300-fp32/ssd300.bin

rm -rf ~/ffva/data/models/non_ir/ssd300  # delete ssd300 in Caffe format

# run object detection sample
cd ~/ffva/FFmpeg-patch  # was cloned on step 2.1 of current instruction
source /opt/intel/openvino/bin/setupvars.sh
./samples/shell/object_detection_ssd.sh $VIDEO_EXAMPLES_DIR/your-video

Example 2. Run object detection with mobilenet-ssd model

mkdir -p ~/ffva/data/models/non_ir/mobilenet-ssd  # to hold downloads
cd ~/ffva/open_model_zoo/tools/downloader  # was cloned on step 2.2 of current instruction
python3 downloader.py --name mobilenet-ssd -o ~/ffva/data/models/non_ir/mobilenet-ssd  # download mobilenet-ssd

# convert model
cd ~/ffva/dldt/model-optimizer  # was cloned on step 5.1 of current instruction
python3 mo.py --framework caffe --input_model ~/ffva/data/models/non_ir/mobilenet-ssd/public/mobilenet-ssd/mobilenet-ssd.caffemodel --input_proto ~/ffva/data/models/non_ir/mobilenet-ssd/public/mobilenet-ssd/mobilenet-ssd.prototxt --data_type FP32 --mean_values data[127.5,127.5,127.5] --scale_values data[127.50223128904757] --output_dir ~/ffva/data/models/common/mobilenet-ssd-fp32/

rm -rf ~/ffva/data/models/non_ir/mobilenet-ssd  # delete mobilenet-ssd in Caffe format

# run object detection sample
cd ~/ffva/FFmpeg-patch  # was cloned on step 2.1 of current instruction
source /opt/intel/openvino/bin/setupvars.sh
./samples/shell/object_detection_and_transcode.sh $VIDEO_EXAMPLES_DIR/your-video

5.2. Use TensorFlow models

Example 1. Run object detection with yolov3 model

You can find more detailed instructions here, but here is the quick way:

git clone https://github.com/mystic123/tensorflow-yolo-v3.git ~/ffva/data/models/non_ir/yolov3  # download yolov3
cd ~/ffva/data/models/non_ir/yolov3
git checkout ed60b90  # choose particular commit
wget https://raw.githubusercontent.com/pjreddie/darknet/master/data/coco.names  # download coco labels 
wget https://pjreddie.com/media/files/yolov3.weights  # download weights

# convert weights
python3 convert_weights_pb.py --class_names coco.names --data_format NHWC --weights_file yolov3.weights --size 320 

# convert model
cd ~/ffva/dldt/model-optimizer  # was cloned on step 5 of current instruction
python3 mo.py --framework tf --input_model ~/ffva/data/models/non_ir/yolov3/frozen_darknet_yolov3_model.pb --tensorflow_use_custom_operations_config ~/ffva/FFmpeg-patch/samples/model_optimizer_configs/yolo_v3.json --reverse_input_channels --input_shape [1,320,320,3] --data_type=FP32 -o ~/ffva/data/models/common/yolov3-fp32

rm -rf ~/ffva/data/models/non_ir/yolov3  # delete yolov3 in TensorFlow format

# Run object detection sample
cd ~/ffva/FFmpeg-patch  # was cloned on step 2.1 of current instruction
source /opt/intel/openvino/bin/setupvars.sh
./samples/shell/object_detection_yolov3.sh $VIDEO_EXAMPLES_DIR/your-video
* Other names and brands may be claimed as the property of others.

Getting started Guide [v0.5]

  1. Introduction
  2. Acquire data and sources
  3. Install FFmpeg Video Analytics plugin
  4. Run samples
  5. Advanced usage

FFmpeg Video Analytics Plugin

Getting Started Guides archive:

Contribute

Pipelines integration:

Clone this wiki locally