diff --git a/.gitignore b/.gitignore index 4dda9b7..64f5b52 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,4 @@ out-*/ *.tar a.out +*.so diff --git a/README.md b/README.md index c8ad45b..ccca8a4 100644 --- a/README.md +++ b/README.md @@ -7,27 +7,34 @@ Motion Vector Extractor -This tool extracts frames, motion vectors and frame types from H.264 and MPEG-4 Part 2 encoded videos. +This tool extracts motion vectors, frames, and frame types from H.264 and MPEG-4 Part 2 encoded videos. -This class is a replacement for OpenCV's [VideoCapture](https://docs.opencv.org/4.1.0/d8/dfe/classcv_1_1VideoCapture.html) and can be used to read and decode video frames from a H.264 or MPEG-4 Part 2 encoded video stream/file. It returns the following values for each frame: -- decoded frame as BGR image +A replacement for OpenCV's [VideoCapture](https://docs.opencv.org/4.1.0/d8/dfe/classcv_1_1VideoCapture.html) that returns for each frame: +- Frame type (I, P, or B) - motion vectors -- Frame type (keyframe, P- or B-frame) +- Optional decoded frame as BGR image -You can use these for applications, such as fast visual object tracking. Both a C++ and a Python API is provided. Under the hood [FFMPEG](https://github.com/FFmpeg/FFmpeg) is used. +Frame decoding can be skipped for very fast motion vector extraction, ideal for, e.g., fast visual object tracking. Both a C++ and a Python API is provided. The image below shows a video frame with extracted motion vectors overlaid. ![motion_vector_demo_image](https://raw.githubusercontent.com/LukasBommes/mv-extractor/cb8e08f4c1e161d103d5382ded93134f26e96f05/mvs.png) -A usage example can be found [here](https://github.com/LukasBommes/mv-extractor/blob/master/src/mvextractor/__main__.py). +
+ Note on Deprecation of Timestamp Extraction -*Note*: Versions 1.x of the mv-extractor additionally returned the timestamps of video frames. For RTSP streams the UTC wall time of the moment the sender sent out a frame was returned (as opposed to an easily retrievable timestamp for the frame reception). Since this feature required patching FFMPEG-internals it proofed difficult to maintain. Hence, I decided to remove this feature in the 2.0 release. If you rely on this feature, please use version 1.1.0. + Versions 1.x of the motion vector extractor additionally returned the timestamps of video frames. For RTSP streams, the UTC wall time of when the sender transmitted a frame was returned (rather than the more easily retrievable reception timestamp). + + Since this feature required patching FFmpeg internals, it became difficult to maintain and prevented compatibility with newer versions of FFmpeg. + + As a result, timestamp extraction was removed in the 2.0.0 release. If you rely on this feature, please use version **1.1.0**. +
## News -### Changes in Upcoming Release 2.0.0 +### Recent Changes in Release 2.0.0 +- New motion-vectors-only mode, in which frame decoding is skipped for better performance (thanks to [@microa](https://github.com/LukasBommes/mv-extractor/pull/78)) - Dropped extraction of timestamps as this feature was complex and difficult to maintain. Note the breaking API change to the `read` and `retrieve` methods of the `VideoCapture` class ```diff @@ -37,88 +44,61 @@ A usage example can be found [here](https://github.com/LukasBommes/mv-extractor/ - Added support for Python 3.13 and 3.14 - Moved installation of FFMPEG and OpenCV from script files directly into Dockerfile - -### Recent Changes in Release 1.1.0 - -- Included community contributions (many thanks to @luowyan and @xyperias) -- Added support for Python 3.11 and 3.12 and dropped support for Python 3.8 -- Upgraded Docker image from deprecated manylinux_2_24_x86_64 to manylinux_2_28_x86_64 -- Improved CI pipeline to run unit tests on every push to a feature branch -- Improved the test suite -- Upgraded build dependencies (OpenCV 4.5.5 -> 4.10.0, numpy 1.x -> 2.0.0) -- Support numpy 2.x as runtime dependency (see this [issue](https://github.com/LukasBommes/mv-extractor/issues/57)) +- Improved quickstart section of the readme ## Quickstart ### Step 1: Install -You can install the motion vector extractor via pip -``` -pip install --upgrade pip +```bash pip install motion-vector-extractor ``` -Note, that we currently provide the package only for x86-64 linux, such as Ubuntu or Debian, and Python 3.9, 3.10, 3.11, 3.12, 3.13, and 3.14. If you are on a different platform, please use the Docker image as described [below](#installation-via-docker). +Note, that we currently provide the package only for x86-64 linux, such as Ubuntu or Debian, and Python 3.9 to 3.14. If you are on a different platform, please use the Docker image as described [below](#installation-via-docker). ### Step 2: Extract Motion Vectors -Download the example video [`vid_h264.mp4`](https://github.com/LukasBommes/mv-extractor/blob/master/vid_h264.mp4) from the repo and place it somewhere. To extract the motion vectors, open a terminal at the same location and run -``` -extract_mvs vid_h264.mp4 --preview --verbose -``` - -The extraction script provides command line options to store extracted motion vectors to disk, and to enable/disable graphical output. For all options type -``` -extract_mvs -h -``` -For example, to store extracted frames and motion vectors to disk without showing graphical output run -``` -extract_mvs vid_h264.mp4 --dump -``` -The `--dump` parameter also takes an optional destination directory. +You can follow along the examples below using the example video [`vid_h264.mp4`](https://github.com/LukasBommes/mv-extractor/blob/master/vid_h264.mp4) from the repo. +#### Command Line -## Advanced Usage +```bash +# Extract motion vectors and show live preview +extract_mvs vid_h264.mp4 --preview --verbose -### Run Tests +# Extract motion vectors and skip frame decoding (faster) +extract_mvs vid_h264.mp4 --verbose --skip-decoding-frames -You can run the test suite either directly on your machine or (easier) within the provided Docker container. Both methods require you to first clone the repository. To this end, change into the desired installation directory on your machine and run -``` -git clone https://github.com/LukasBommes/mv-extractor.git mv_extractor -``` - -#### In Docker Container +# Extract and store motion vectors and frames to disk without showing live preview +extract_mvs vid_h264.mp4 --dump -To run the tests in the Docker container, change into the `mv_extractor` directory, and run -``` -./run.sh /bin/bash -c 'yum install -y compat-openssl10 && python3.12 -m unittest discover -s tests -p "*tests.py"' +# See all available options +extract_mvs -h ``` -#### On Host - -To run the tests directly on your machine, you need to install the motion vector extractor as explained [above](#step-1-install). +#### Python API +```python +from mvextractor.videocap import VideoCap -Now, change into the `mv_extractor` directory and run the tests with -``` -python3.12 -m unittest discover -s tests -p "*tests.py" -``` -Confirm that all tests pass. +cap = VideoCap() +cap.open("vid_h264.mp4") -Some tests run the [LIVE555 Media Server](http://www.live555.com/mediaServer/), which has dependencies on its own, such as OpenSSL. Make sure these dependencies are installed correctly on your machine, or otherwise you will get test failures with messages, such as "error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory". E.g. in Alma Linux you could fix this issue by installing OpenSSL with -``` -yum install -y compat-openssl10 -``` -For other operating systems you may be lacking additional dependencies, and the package names and installation command may differ. +# (optional) skip decoding frames +cap.set_decode_frames(False) -### Importing mvextractor into Your Own Scripts +while True: + ret, frame, motion_vectors, frame_type = cap.read() + if not ret: + break + print(f"Num. motion vectors: {len(motion_vectors)}") + print(f"Frame type: {frame_type}") + if frame is not None: + print(f"Frame size: {frame.shape}") -If you want to use the motion vector extractor in your own Python script import it via +cap.release() ``` -from mvextractor.videocap import VideoCap -``` -You can then use it according to the example in `extract_mvs.py`. -Generally, a video file is opened by `VideoCap.open()` and frames, motion vectors and frame types are read by calling `VideoCap.read()` repeatedly. Before exiting the program, the video file has to be closed by `VideoCap.release()`. For a more detailed explanation see the API documentation below. +## Advanced Usage ### Installation via Docker @@ -127,14 +107,14 @@ Instead of installing the motion vector extractor via PyPI you can also use the #### Prerequisites To use the Docker image you need to install [Docker](https://docs.docker.com/). Furthermore, you need to clone the source code with -``` +```bash git clone https://github.com/LukasBommes/mv-extractor.git mv_extractor ``` #### Run Motion Vector Extraction in Docker Afterwards, you can run the extraction script in the `mv_extractor` directory as follows -``` +```bash ./run.sh python3.12 extract_mvs.py vid_h264.mp4 --preview --verbose ``` This pulls the prebuild Docker image from DockerHub and runs the extraction script inside the Docker container. @@ -143,13 +123,13 @@ This pulls the prebuild Docker image from DockerHub and runs the extraction scri This step is not required and for faster installation, we recommend using the prebuilt image. If you still want to build the Docker image locally, you can do so by running the following command in the `mv_extractor` directory -``` +```bash docker build . --tag=mv-extractor ``` Note that building can take more than one hour. Now, run the docker container with -``` +```bash docker run -it --ipc=host --env="DISPLAY" -v $(pwd):/home/video_cap -v /tmp/.X11-unix:/tmp/.X11-unix:rw mv-extractor /bin/bash ``` @@ -166,12 +146,17 @@ This module provides a Python API which is very similar to that of OpenCV [Video | open() | Open a video file or url | | grab() | Reads the next video frame and motion vectors from the stream | | retrieve() | Decodes and returns the grabbed frame and motion vectors | -| read() | Convenience function which combines a call of grab() and retrieve(). | +| read() | Convenience function which combines a call of grab() and retrieve() | | release() | Close a video file or url and release all ressources | +| set_decode_frames() | Enable/disable decoding of video frames | + +| Attributes | Description | +| --- | --- | +| decode_frames | Getter to check if frame decoding is enabled (True) or skipped (False) | ##### Method :: VideoCap() -Constructor. Takes no input arguments. +Constructor. Takes no input arguments and returns nothing. ##### Method :: open() @@ -204,7 +189,7 @@ Takes no input arguments and returns a tuple with the elements described in the | Index | Name | Type | Description | | --- | --- | --- | --- | | 0 | success | bool | True in case the frame and motion vectors could be retrieved sucessfully, false otherwise or in case the end of stream is reached. When false, the other tuple elements are set to empty numpy arrays or 0. | -| 1 | frame | numpy array | Array of dtype uint8 shape (h, w, 3) containing the decoded video frame. w and h are the width and height of this frame in pixels. Channels are in BGR order. If no frame could be decoded an empty numpy ndarray of shape (0, 0, 3) and dtype uint8 is returned. | +| 1 | frame | numpy array | Array of dtype uint8 shape (h, w, 3) containing the decoded video frame. w and h are the width and height of this frame in pixels. Channels are in BGR order. If no frame could be decoded an empty numpy ndarray of shape (0, 0, 3) and dtype uint8 is returned. If frame decoding is disabled with set_decode_frames(False) None is returned instead. | | 2 | motion vectors | numpy array | Array of dtype int32 and shape (N, 10) containing the N motion vectors of the frame. Each row of the array corresponds to one motion vector. If no motion vectors are present in a frame, e.g. if the frame is an `I` frame an empty numpy array of shape (0, 10) and dtype int32 is returned. The columns of each vector have the following meaning (also refer to [AVMotionVector](https://ffmpeg.org/doxygen/4.1/structAVMotionVector.html) in FFMPEG documentation):
- 0: `source`: offset of the reference frame from the current frame. The reference frame is the frame where the motion vector points to and where the corresponding macroblock comes from. If `source < 0`, the reference frame is in the past. For `source > 0` the it is in the future (in display order).
- 1: `w`: width of the vector's macroblock.
- 2: `h`: height of the vector's macroblock.
- 3: `src_x`: x-location (in pixels) where the motion vector points to in the reference frame.
- 4: `src_y`: y-location (in pixels) where the motion vector points to in the reference frame.
- 5: `dst_x`: x-location of the vector's origin in the current frame (in pixels). Corresponds to the x-center coordinate of the corresponding macroblock.
- 6: `dst_y`: y-location of the vector's origin in the current frame (in pixels). Corresponds to the y-center coordinate of the corresponding macroblock.
- 7: `motion_x`: Macroblock displacement in x-direction, multiplied by `motion_scale` to become integer. Used to compute fractional value for `src_x` as `src_x = dst_x + motion_x / motion_scale`.
- 8: `motion_y`: Macroblock displacement in y-direction, multiplied by `motion_scale` to become integer. Used to compute fractional value for `src_y` as `src_y = dst_y + motion_y / motion_scale`.
- 9: `motion_scale`: see definiton of columns 7 and 8. Used to scale up the motion components to integer values. E.g. if `motion_scale = 4`, motion components can be integer values but encode a float with 1/4 pixel precision.

Note: `src_x` and `src_y` are only in integer resolution. They are contained in the [AVMotionVector](https://ffmpeg.org/doxygen/4.1/structAVMotionVector.html) struct and exported only for the sake of completeness. Use equations in field 7 and 8 to get more accurate fractional values for `src_x` and `src_y`. | | 3 | frame_type | string | Unicode string representing the type of frame. Can be `"I"` for a keyframe, `"P"` for a frame with references to only past frames and `"B"` for a frame with references to both past and future frames. A `"?"` string indicates an unknown frame type. | @@ -216,6 +201,14 @@ Convenience function which internally calls first grab() and then retrieve(). It Close a video file or url and release all ressources. Takes no input arguments and returns nothing. +##### Method :: set_decode_frames() + +Enable/disable decoding of video frames. May be called anytime, even mid-stream. Returns nothing. + +| Parameter | Type | Description | +| --- | --- | --- | +| enable | bool | If True (default) RGB frames are decoded and returned in addition to extracted motion vectors. If False, frame decoding is skipped, yielding much higher extraction througput. | + ## C++ API @@ -243,7 +236,7 @@ The frame type is either "P", "B" or "I" and refers to the H.264 encoding mode o ## About -This software is written by [**Lukas Bommes**](https://lukasbommes.de/). +This software is maintained by [**Lukas Bommes**](https://lukasbommes.de/). It is based on [MV-Tractus](https://github.com/jishnujayakumar/MV-Tractus/tree/master/include) and OpenCV's [videoio module](https://github.com/opencv/opencv/tree/master/modules/videoio). @@ -267,3 +260,5 @@ If you use our work for academic research please cite pages={1419-1424}, doi={10.1109/ICIEA48937.2020.9248145}} ``` + + diff --git a/src/mvextractor/__main__.py b/src/mvextractor/__main__.py index 7b99b53..a1fad22 100644 --- a/src/mvextractor/__main__.py +++ b/src/mvextractor/__main__.py @@ -30,6 +30,7 @@ def main(args=None): parser.add_argument('video_url', type=str, nargs='?', help='file path or url of the video stream') parser.add_argument('-p', '--preview', action='store_true', help='show a preview video with overlaid motion vectors') parser.add_argument('-v', '--verbose', action='store_true', help='show detailled text output') + parser.add_argument('-s', '--skip-decoding-frames', action='store_true', help='skip decoding RGB frames and return only motion vectors (faster)') parser.add_argument('-d', '--dump', nargs='?', const=True, help='dump frames, motion vectors and frame types to optionally specified output directory') args = parser.parse_args() @@ -53,6 +54,9 @@ def main(args=None): if args.verbose: print("Sucessfully opened video file") + if args.skip_decoding_frames: + cap.set_decode_frames(False) + step = 0 times = [] @@ -79,22 +83,28 @@ def main(args=None): # print results if args.verbose: print("frame type: {} | ".format(frame_type), end=" ") - print("frame size: {} | ".format(np.shape(frame)), end=" ") + if frame is not None: + print("frame size: {} | ".format(np.shape(frame)), end=" ") + else: + print("frame size: () | ", end=" ") print("motion vectors: {} | ".format(np.shape(motion_vectors)), end=" ") print("elapsed time: {} s".format(telapsed)) - frame = draw_motion_vectors(frame, motion_vectors) + # draw vectors on frames + if not args.skip_decoding_frames and frame is not None: + frame = draw_motion_vectors(frame, motion_vectors) - # store motion vectors, frames, etc. in output directory + # store motion vectors, frames, and fraem types in output directory if args.dump: - cv2.imwrite(os.path.join(dumpdir, "frames", f"frame-{step}.jpg"), frame) np.save(os.path.join(dumpdir, "motion_vectors", f"mvs-{step}.npy"), motion_vectors) with open(os.path.join(dumpdir, "frame_types.txt"), "a") as f: f.write(frame_type+"\n") + if not args.skip_decoding_frames and frame is not None: + cv2.imwrite(os.path.join(dumpdir, "frames", f"frame-{step}.jpg"), frame) step += 1 - if args.preview: + if args.preview and not args.skip_decoding_frames: cv2.imshow("Frame", frame) # if user presses "q" key stop program diff --git a/src/mvextractor/py_video_cap.cpp b/src/mvextractor/py_video_cap.cpp index dc2cadb..4a35efd 100755 --- a/src/mvextractor/py_video_cap.cpp +++ b/src/mvextractor/py_video_cap.cpp @@ -9,10 +9,17 @@ typedef struct { PyObject_HEAD VideoCap vcap; - //NDArrayConverter mat_to_ndarray_cvt; } VideoCapObject; +static int +VideoCap_init(VideoCapObject *self, PyObject *args, PyObject *kwds) +{ + new(&self->vcap) VideoCap(); + return 0; +} + + static void VideoCap_dealloc(VideoCapObject *self) { @@ -71,11 +78,16 @@ VideoCap_retrieve(VideoCapObject *self, PyObject *Py_UNUSED(ignored)) } // copy frame buffer into new cv::Mat - cv::Mat(height, width, CV_MAKETYPE(CV_8U, cn), frame, step).copyTo(frame_cv); - - // convert frame cv::Mat to numpy.ndarray - NDArrayConverter cvt; - PyObject* frame_nd = cvt.toNDArray(frame_cv); + PyObject* frame_nd = Py_None; + if (self->vcap.getDecodeFrames()) { + cv::Mat(height, width, CV_MAKETYPE(CV_8U, cn), frame, step).copyTo(frame_cv); + + // convert frame cv::Mat to numpy.ndarray + NDArrayConverter cvt; + frame_nd = cvt.toNDArray(frame_cv); + } else { + Py_INCREF(Py_None); + } // convert motion vector buffer into numpy array npy_intp dims_mvs[2] = {(npy_intp)num_mvs, 10}; @@ -111,12 +123,16 @@ VideoCap_read(VideoCapObject *self, PyObject *Py_UNUSED(ignored)) ret = Py_False; } - // copy frame buffer into new cv::Mat - cv::Mat(height, width, CV_MAKETYPE(CV_8U, cn), frame, step).copyTo(frame_cv); + PyObject* frame_nd = Py_None; + if (self->vcap.getDecodeFrames()) { + cv::Mat(height, width, CV_MAKETYPE(CV_8U, cn), frame, step).copyTo(frame_cv); - // convert frame cv::Mat to numpy.ndarray - NDArrayConverter cvt; - PyObject* frame_nd = cvt.toNDArray(frame_cv); + // convert frame cv::Mat to numpy.ndarray + NDArrayConverter cvt; + frame_nd = cvt.toNDArray(frame_cv); + } else { + Py_INCREF(Py_None); + } // convert motion vector buffer into numpy array npy_intp dims_mvs[2] = {(npy_intp)num_mvs, 10}; @@ -135,16 +151,46 @@ VideoCap_release(VideoCapObject *self, PyObject *Py_UNUSED(ignored)) } +static PyObject * +VideoCap_set_decode_frames(VideoCapObject *self, PyObject *args) +{ + int enable = 0; + if (!PyArg_ParseTuple(args, "p", &enable)) + Py_RETURN_NONE; + + self->vcap.setDecodeFrames(enable != 0); + Py_RETURN_NONE; +} + + +static PyObject * +VideoCap_get_decode_frames(VideoCapObject *self, PyObject *Py_UNUSED(ignored)) +{ + if (self->vcap.getDecodeFrames()) + Py_RETURN_TRUE; + else + Py_RETURN_FALSE; +} + + static PyMethodDef VideoCap_methods[] = { {"open", (PyCFunction) VideoCap_open, METH_VARARGS, "Open a video file or device with given filename/url"}, {"read", (PyCFunction) VideoCap_read, METH_NOARGS, "Grab and decode the next frame and motion vectors"}, {"grab", (PyCFunction) VideoCap_grab, METH_NOARGS, "Grab the next frame and motion vectors from the stream"}, {"retrieve", (PyCFunction) VideoCap_retrieve, METH_NOARGS, "Decode the grabbed frame and motion vectors"}, {"release", (PyCFunction) VideoCap_release, METH_NOARGS, "Release the video device and free ressources"}, + {"set_decode_frames", (PyCFunction) VideoCap_set_decode_frames, METH_VARARGS, "Enable/disable decoding of RGB frames"}, {NULL} /* Sentinel */ }; +static PyGetSetDef VideoCap_getset[] = { + {"decode_frames", (getter)VideoCap_get_decode_frames, NULL, + "Whether RGB frames are decoded (True) or only motion vectors (False)", NULL}, + {NULL} // Sentinel +}; + + static PyTypeObject VideoCapType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "videocap.VideoCap", @@ -175,13 +221,13 @@ static PyTypeObject VideoCapType = { .tp_iternext = NULL, .tp_methods = VideoCap_methods, .tp_members = NULL, - .tp_getset = NULL, + .tp_getset = VideoCap_getset, .tp_base = NULL, .tp_dict = NULL, .tp_descr_get = NULL, .tp_descr_set = NULL, .tp_dictoffset = 0, - .tp_init = NULL, + .tp_init = (initproc) VideoCap_init, .tp_alloc = NULL, .tp_new = PyType_GenericNew, .tp_free = NULL, diff --git a/src/mvextractor/video_cap.cpp b/src/mvextractor/video_cap.cpp index 298255a..fa58a39 100644 --- a/src/mvextractor/video_cap.cpp +++ b/src/mvextractor/video_cap.cpp @@ -11,6 +11,7 @@ VideoCap::VideoCap() { this->frame = NULL; this->img_convert_ctx = NULL; this->frame_number = 0; + this->decode_frames = true; memset(&(this->rgb_frame), 0, sizeof(this->rgb_frame)); memset(&(this->picture), 0, sizeof(this->picture)); @@ -60,6 +61,7 @@ void VideoCap::release(void) { this->video_stream = NULL; this->video_stream_idx = -1; this->frame_number = 0; + this->decode_frames = true; } @@ -141,6 +143,9 @@ bool VideoCap::open(const char *url) { if (!this->frame) goto error; + // default: decode frames + this->decode_frames = true; + if (this->video_stream_idx >= 0) valid = true; @@ -152,6 +157,14 @@ bool VideoCap::open(const char *url) { return valid; } +void VideoCap::setDecodeFrames(bool enable) { + this->decode_frames = enable; +} + +bool VideoCap::getDecodeFrames() { + return this->decode_frames; +} + bool VideoCap::grab(void) { @@ -195,7 +208,6 @@ bool VideoCap::grab(void) { if(got_frame) { this->frame_number++; valid = true; - } else { count_errs++; @@ -214,58 +226,70 @@ bool VideoCap::retrieve(uint8_t **frame, int *step, int *width, int *height, int if (!this->video_stream || !(this->frame->data[0])) return false; - if (this->img_convert_ctx == NULL || - this->picture.width != this->video_dec_ctx->width || - this->picture.height != this->video_dec_ctx->height || - this->picture.data == NULL) { - - // Some sws_scale optimizations have some assumptions about alignment of data/step/width/height - // Also we use coded_width/height to workaround problem with legacy ffmpeg versions (like n0.8) - int buffer_width = this->video_dec_ctx->coded_width; - int buffer_height = this->video_dec_ctx->coded_height; - - this->img_convert_ctx = sws_getCachedContext( - this->img_convert_ctx, - buffer_width, buffer_height, - this->video_dec_ctx->pix_fmt, - buffer_width, buffer_height, - AV_PIX_FMT_BGR24, - SWS_BICUBIC, - NULL, NULL, NULL - ); - - if (this->img_convert_ctx == NULL) - return false; - - av_frame_unref(&(this->rgb_frame)); - this->rgb_frame.format = AV_PIX_FMT_BGR24; - this->rgb_frame.width = buffer_width; - this->rgb_frame.height = buffer_height; - if (0 != av_frame_get_buffer(&(this->rgb_frame), 32)) - return false; - - this->picture.width = this->video_dec_ctx->width; - this->picture.height = this->video_dec_ctx->height; - this->picture.data = this->rgb_frame.data[0]; - this->picture.step = this->rgb_frame.linesize[0]; - this->picture.cn = 3; - } + // perform color conversion and return frame buffer + if (this->decode_frames) { + + if (this->img_convert_ctx == NULL || + this->picture.width != this->video_dec_ctx->width || + this->picture.height != this->video_dec_ctx->height || + this->picture.data == NULL) { + + // Some sws_scale optimizations have some assumptions about alignment of data/step/width/height + // Also we use coded_width/height to workaround problem with legacy ffmpeg versions (like n0.8) + int buffer_width = this->video_dec_ctx->coded_width; + int buffer_height = this->video_dec_ctx->coded_height; + + this->img_convert_ctx = sws_getCachedContext( + this->img_convert_ctx, + buffer_width, buffer_height, + this->video_dec_ctx->pix_fmt, + buffer_width, buffer_height, + AV_PIX_FMT_BGR24, + SWS_BICUBIC, + NULL, NULL, NULL + ); + + if (this->img_convert_ctx == NULL) + return false; + + av_frame_unref(&(this->rgb_frame)); + this->rgb_frame.format = AV_PIX_FMT_BGR24; + this->rgb_frame.width = buffer_width; + this->rgb_frame.height = buffer_height; + if (0 != av_frame_get_buffer(&(this->rgb_frame), 32)) + return false; - // change color space of frame - sws_scale( - this->img_convert_ctx, - this->frame->data, - this->frame->linesize, - 0, this->video_dec_ctx->coded_height, - this->rgb_frame.data, - this->rgb_frame.linesize - ); - - *frame = this->picture.data; - *width = this->picture.width; - *height = this->picture.height; - *step = this->picture.step; - *cn = this->picture.cn; + this->picture.width = this->video_dec_ctx->width; + this->picture.height = this->video_dec_ctx->height; + this->picture.data = this->rgb_frame.data[0]; + this->picture.step = this->rgb_frame.linesize[0]; + this->picture.cn = 3; + } + + // change color space of frame + sws_scale( + this->img_convert_ctx, + this->frame->data, + this->frame->linesize, + 0, this->video_dec_ctx->coded_height, + this->rgb_frame.data, + this->rgb_frame.linesize + ); + + *frame = this->picture.data; + *width = this->picture.width; + *height = this->picture.height; + *step = this->picture.step; + *cn = this->picture.cn; + + } else { + // when not decoding frames, don't allocate or return frame buffer + *frame = NULL; + *width = 0; + *height = 0; + *step = 0; + *cn = 0; + } // get motion vectors AVFrameSideData *sd = av_frame_get_side_data(this->frame, AV_FRAME_DATA_MOTION_VECTORS); diff --git a/src/mvextractor/video_cap.hpp b/src/mvextractor/video_cap.hpp index 912fc9a..1a87095 100644 --- a/src/mvextractor/video_cap.hpp +++ b/src/mvextractor/video_cap.hpp @@ -65,6 +65,8 @@ class VideoCap { Image_FFMPEG picture; struct SwsContext *img_convert_ctx; int64_t frame_number; + // When true, retrieve only motion vectors and skip RGB/color conversion + bool decode_frames; #if USE_AV_INTERRUPT_CALLBACK AVInterruptCallbackMetadata interrupt_metadata; #endif @@ -163,4 +165,11 @@ class VideoCap { * The parameters and return value correspond to the `retrieve` method. */ bool read(uint8_t **frame, int *step, int *width, int *height, int *cn, char *frame_type, MVS_DTYPE **motion_vectors, MVS_DTYPE *num_mvs); + + /** Enable/disable decoding frames in addition to extracting motion vectors. + * If decoding is disabled (false), retrieve() will skip color space conversion + * and not fill the frame buffer to avoid costly RGB decoding/copying. + */ + void setDecodeFrames(bool enable); + bool getDecodeFrames(); }; diff --git a/tests/README.md b/tests/README.md index 9ac6e10..42d8c2a 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,16 +1,63 @@ -# Reference Data +# Tests -This README explains how the reference datasets were obtained from the provided video files `vid_h264.mp4` and `vid_mpeg4_part2.mp4`. +## Run Tests -## reference/h264 +You can run the test suite either directly on your machine or (easier) within the provided Docker container. Both methods require you to first clone the repository. To this end, change into the desired installation directory on your machine and run +```bash +git clone https://github.com/LukasBommes/mv-extractor.git mv_extractor +``` + +### In Docker Container + +To run the tests in the Docker container, change into the `mv_extractor` directory, and run +```bash +./run.sh /bin/bash -c 'yum install -y compat-openssl10 && python3.12 -m unittest discover -s tests -p "*tests.py"' +``` + +### On Host + +To run the tests directly on your machine, you need to install the motion vector extractor as explained [above](#step-1-install). + +Now, change into the `mv_extractor` directory and run the tests with +```bash +python3.12 -m unittest discover -s tests -p "*tests.py" +``` +Confirm that all tests pass. + +Some tests run the [LIVE555 Media Server](http://www.live555.com/mediaServer/), which has dependencies on its own, such as OpenSSL. Make sure these dependencies are installed correctly on your machine, or otherwise you will get test failures with messages, such as "error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory". E.g. in Alma Linux you could fix this issue by installing OpenSSL with +```bash +yum install -y compat-openssl10 +``` +For other operating systems you may be lacking additional dependencies, and the package names and installation command may differ. + + +## Reference Test Data + +This directory contains reference test data for validating mv-extractor output. The test suite compares current output against this reference data to ensure no regressions. More specifically the test suite verifies that: +1. Motion vector extraction produces consistent results +2. Frame decoding works correctly +3. Frame types are correctly identified + +### Structure + +- `h264/` - H.264 test video reference data +- `mpeg4_part2/` - MPEG-4 Part 2 test video reference data +- `rtsp/` - RTSP stream reference data + +### Data Format + +Each subdirectory contains: +- `motion_vectors/` - Motion vector .npy files +- `frames/` - Frame image .jpg files +- `frame_types.txt` - Frame type information -## reference/mpeg4_part2 +### Reference Data Creation -## reference/rtsp +Reference datasets for H.264 and MPEG-4 PART 2 were obtained by running the `extract_mvs` command of a manually verified version of the mvextractor on the provided video files `vid_h264.mp4` and `vid_mpeg4_part2.mp4` -This reference data was obtained by streaming one of the video files with the [LIVE555 Media Server](http://www.live555.com/mediaServer/) and then reading the RTSP stream with the motion vector extractor. To reproduce the reference data, follow the steps below. +RTSP reference data was obtained by streaming one of the video files with the [LIVE555 Media Server](http://www.live555.com/mediaServer/) and then reading the RTSP stream with the motion vector extractor. To reproduce the reference data, follow the steps below. -### Convert input file into H.264 video elementary stream +#### Convert input file into H.264 video elementary stream First, convert the `vid_h264.mp4` file into a H.264 video elementary stream file. To this end, run ``` @@ -26,7 +73,7 @@ MultiFramedRTPSink::afterGettingFrame1(): The input frame data was too large for ``` and the resulting video frame is truncated at the bottom. -### Serve the video with LIVE555 Media Server +#### Serve the video with LIVE555 Media Server Now, we serve the file `vid_h264.264` with LIVE555 Media Server. Place the file in a folder named `data` ``` @@ -47,7 +94,7 @@ live555MediaServer & ``` You may have to hit `CTRL+C` now to dismiss the log of the server. The server will continue running in the background. -### Consume the RTSP stream with the motion vector extractor +#### Consume the RTSP stream with the motion vector extractor Still in the Docker container, install the motion vector extractor ``` @@ -58,7 +105,7 @@ and run it to read and dump the RTSP stream to a folder named `out-reference` /opt/python/cp312-cp312/bin/extract_mvs 'rtsp://localhost:554/vid_h264.264' --verbose --dump out-reference ``` -### Preserve reference data and cleanup +#### Preserve reference data and cleanup Finally, exist the container with ``` diff --git a/tests/end_to_end_tests.py b/tests/end_to_end_tests.py index 0407de0..e7e8c5e 100644 --- a/tests/end_to_end_tests.py +++ b/tests/end_to_end_tests.py @@ -11,58 +11,82 @@ PROJECT_ROOT = os.getenv("PROJECT_ROOT", "") -class TestEndToEnd(unittest.TestCase): - - def motions_vectors_valid(self, outdir, refdir): - equal = [] - num_mvs = len(os.listdir(os.path.join(refdir, "motion_vectors"))) - for i in range(num_mvs): - mvs = np.load(os.path.join(outdir, "motion_vectors", f"mvs-{i}.npy")) - mvs_ref = np.load(os.path.join(refdir, "motion_vectors", f"mvs-{i}.npy")) - equal.append(np.all(mvs == mvs_ref)) - return all(equal) - - - def frame_types_valid(self, outdir, refdir): - with open(os.path.join(outdir, "frame_types.txt"), "r") as file: - frame_types = [line.strip() for line in file] - with open(os.path.join(refdir, "frame_types.txt"), "r") as file: - frame_types_ref = [line.strip() for line in file] - return frame_types == frame_types_ref +def motions_vectors_valid(outdir, refdir): + equal = [] + num_mvs = len(os.listdir(os.path.join(refdir, "motion_vectors"))) + for i in range(num_mvs): + mvs = np.load(os.path.join(outdir, "motion_vectors", f"mvs-{i}.npy")) + mvs_ref = np.load(os.path.join(refdir, "motion_vectors", f"mvs-{i}.npy")) + equal.append(np.all(mvs == mvs_ref)) + return all(equal) + + +def frame_types_valid(outdir, refdir): + with open(os.path.join(outdir, "frame_types.txt"), "r") as file: + frame_types = [line.strip() for line in file] + with open(os.path.join(refdir, "frame_types.txt"), "r") as file: + frame_types_ref = [line.strip() for line in file] + return frame_types == frame_types_ref + + +def frames_valid(outdir, refdir): + equal = [] + num_frames = len(os.listdir(os.path.join(refdir, "frames"))) + for i in range(num_frames): + frame = cv2.imread(os.path.join(outdir, "frames", f"frame-{i}.jpg")) + frame_ref = cv2.imread(os.path.join(refdir, "frames", f"frame-{i}.jpg")) + equal.append(np.all(frame == frame_ref)) + return all(equal) - def frames_valid(self, outdir, refdir): - equal = [] - num_frames = len(os.listdir(os.path.join(refdir, "frames"))) - for i in range(num_frames): - frame = cv2.imread(os.path.join(outdir, "frames", f"frame-{i}.jpg")) - frame_ref = cv2.imread(os.path.join(refdir, "frames", f"frame-{i}.jpg")) - equal.append(np.all(frame == frame_ref)) - return all(equal) - +class TestEndToEnd(unittest.TestCase): def test_end_to_end_h264(self): with tempfile.TemporaryDirectory() as outdir: print("Running extraction for H.264") - subprocess.run(f"extract_mvs {os.path.join(PROJECT_ROOT, 'vid_h264.mp4')} --dump {outdir}", shell=True, check=True) + video_path = os.path.join(PROJECT_ROOT, 'vid_h264.mp4') + subprocess.run(f"extract_mvs {video_path} --dump {outdir}", shell=True, check=True) refdir = os.path.join(PROJECT_ROOT, "tests/reference/h264") - self.assertTrue(self.motions_vectors_valid(outdir, refdir), msg="motion vectors are invalid") - self.assertTrue(self.frame_types_valid(outdir, refdir), msg="frame types are invalid") - self.assertTrue(self.frames_valid(outdir, refdir), msg="frames are invalid") + self.assertTrue(motions_vectors_valid(outdir, refdir), msg="motion vectors are invalid") + self.assertTrue(frame_types_valid(outdir, refdir), msg="frame types are invalid") + self.assertTrue(frames_valid(outdir, refdir), msg="frames are invalid") + + + def test_end_to_end_motion_vectors_only_h264(self): + with tempfile.TemporaryDirectory() as outdir: + print("Running motion-vectors-only extraction for H.264") + video_path = os.path.join(PROJECT_ROOT, 'vid_h264.mp4') + subprocess.run(f"extract_mvs {video_path} --skip-decoding-frames --dump {outdir}", shell=True, check=True) + refdir = os.path.join(PROJECT_ROOT, "tests/reference/h264") + + self.assertTrue(motions_vectors_valid(outdir, refdir), msg="motion vectors are invalid") + self.assertTrue(frame_types_valid(outdir, refdir), msg="frame types are invalid") def test_end_to_end_mpeg4_part2(self): with tempfile.TemporaryDirectory() as outdir: print("Running extraction for MPEG-4 Part 2") - subprocess.run(f"extract_mvs {os.path.join(PROJECT_ROOT, 'vid_mpeg4_part2.mp4')} --dump {outdir}", shell=True, check=True) + video_path = os.path.join(PROJECT_ROOT, 'vid_mpeg4_part2.mp4') + subprocess.run(f"extract_mvs {video_path} --dump {outdir}", shell=True, check=True) refdir = os.path.join(PROJECT_ROOT, "tests/reference/mpeg4_part2") - self.assertTrue(self.motions_vectors_valid(outdir, refdir), msg="motion vectors are invalid") - self.assertTrue(self.frame_types_valid(outdir, refdir), msg="frame types are invalid") - self.assertTrue(self.frames_valid(outdir, refdir), msg="frames are invalid") + self.assertTrue(motions_vectors_valid(outdir, refdir), msg="motion vectors are invalid") + self.assertTrue(frame_types_valid(outdir, refdir), msg="frame types are invalid") + self.assertTrue(frames_valid(outdir, refdir), msg="frames are invalid") + + + def test_end_to_end_motion_vectors_only_mpeg4_part2(self): + with tempfile.TemporaryDirectory() as outdir: + print("Running motion-vectors-only extraction for MPEG-4 Part 2") + video_path = os.path.join(PROJECT_ROOT, 'vid_mpeg4_part2.mp4') + subprocess.run(f"extract_mvs {video_path} --skip-decoding-frames --dump {outdir}", shell=True, check=True) + refdir = os.path.join(PROJECT_ROOT, "tests/reference/mpeg4_part2") + self.assertTrue(motions_vectors_valid(outdir, refdir), msg="motion vectors are invalid") + self.assertTrue(frame_types_valid(outdir, refdir), msg="frame types are invalid") + def test_end_to_end_rtsp(self): with tempfile.TemporaryDirectory() as outdir: print("Setting up end to end test for RTSP") @@ -75,9 +99,9 @@ def test_end_to_end_rtsp(self): subprocess.run(f"extract_mvs {rtsp_url} --dump {outdir}", shell=True, check=True) refdir = os.path.join(PROJECT_ROOT, "tests/reference/rtsp") - self.assertTrue(self.motions_vectors_valid(outdir, refdir), msg="motion vectors are invalid") - self.assertTrue(self.frame_types_valid(outdir, refdir), msg="frame types are invalid") - self.assertTrue(self.frames_valid(outdir, refdir), msg="frames are invalid") + self.assertTrue(motions_vectors_valid(outdir, refdir), msg="motion vectors are invalid") + self.assertTrue(frame_types_valid(outdir, refdir), msg="frame types are invalid") + self.assertTrue(frames_valid(outdir, refdir), msg="frames are invalid") finally: rtsp_server.terminate() diff --git a/tests/unit_tests.py b/tests/unit_tests.py index 7582db5..6827eb1 100644 --- a/tests/unit_tests.py +++ b/tests/unit_tests.py @@ -13,15 +13,16 @@ class TestMotionVectorExtraction(unittest.TestCase): def validate_frame(self, frame): - self.assertEqual(type(frame), np.ndarray) - self.assertEqual(frame.dtype, np.uint8) - self.assertEqual(frame.shape, (720, 1280, 3)) + self.assertEqual(type(frame), np.ndarray, "Frame should be numpy array") + self.assertEqual(frame.dtype, np.uint8, "Frame dtype should be uint8") + self.assertEqual(frame.shape, (720, 1280, 3), "Frams hape should be (720, 1280, 3)") def validate_motion_vectors(self, motion_vectors, shape=(0, 10)): - self.assertEqual(type(motion_vectors), np.ndarray) - self.assertEqual(motion_vectors.dtype, np.int32) - self.assertEqual(motion_vectors.shape, shape) + self.assertEqual(type(motion_vectors), np.ndarray, "Motion vectors should be numpy array") + self.assertEqual(motion_vectors.dtype, np.int32, "Motion vectors dtype should be int32") + self.assertEqual(motion_vectors.shape, shape, "Motion vectors shape not matching expected shape") + # run before every test def setUp(self): @@ -44,33 +45,50 @@ def test_init_cap(self): self.assertIn('read', dir(self.cap)) self.assertIn('release', dir(self.cap)) self.assertIn('retrieve', dir(self.cap)) + self.assertIn('set_decode_frames', dir(self.cap)) + self.assertIn('decode_frames', dir(self.cap)) + + + def test_decode_frames_mode(self): + self.cap = VideoCap() + self.assertTrue(self.cap.decode_frames, "Frame decoding is expected to be actived by default") + self.cap.set_decode_frames(True) + self.assertTrue(self.cap.decode_frames, "Frame decoding is expected to be active") + self.cap.set_decode_frames(False) + self.assertFalse(self.cap.decode_frames, "Frame decoding is expected to be inactive") + self.open_video() + self.assertTrue(self.cap.decode_frames, "Frame decoding is expected to be actived after opening a video") + self.cap.set_decode_frames(False) + self.assertFalse(self.cap.decode_frames, "Frame decoding is expected to be inactive") + self.cap.release() + self.assertTrue(self.cap.decode_frames, "Frame decoding is expected to be active") def test_open_video(self): ret = self.open_video() - self.assertTrue(ret) + self.assertTrue(ret, "Should open video file successfully") def test_open_invalid_video(self): ret = self.cap.open("vid_not_existent.mp4") - self.assertFalse(ret) + self.assertFalse(ret, "Should fail to open non-existent video file") def test_read_not_opened_cap(self): ret = self.cap.open("vid_not_existent.mp4") - self.assertFalse(ret) + self.assertFalse(ret, "Should fail to open non-existent video file") ret, frame, motion_vectors, frame_type = self.cap.read() - self.assertEqual(frame_type, "?") - self.assertFalse(ret) - self.assertIsNone(frame) + self.assertEqual(frame_type, "?", "Frame type should be ?") + self.assertFalse(ret, "Should fail to read from non-existent video file") + self.assertIsNone(frame, "Frame read from non-existent video should be None") self.validate_motion_vectors(motion_vectors) def test_read_first_I_frame(self): self.open_video() ret, frame, motion_vectors, frame_type = self.cap.read() - self.assertTrue(ret) - self.assertEqual(frame_type, "I") + self.assertTrue(ret, "Should succeed to read from video file") + self.assertEqual(frame_type, "I", "Frame type of first frame should be I") self.validate_frame(frame) self.validate_motion_vectors(motion_vectors) @@ -79,8 +97,8 @@ def test_read_first_P_frame(self): self.open_video() self.cap.read() # skip first frame (I frame) ret, frame, motion_vectors, frame_type = self.cap.read() - self.assertTrue(ret) - self.assertEqual(frame_type, "P") + self.assertTrue(ret, "Should succeed to read from video file") + self.assertEqual(frame_type, "P", "Frame type of second frame should be P") self.validate_frame(frame) self.validate_motion_vectors(motion_vectors, shape=(3665, 10)) self.assertTrue(np.all(motion_vectors[:10, :] == np.array([ @@ -94,7 +112,7 @@ def test_read_first_P_frame(self): [-1, 16, 16, 120, 8, 120, 8, 0, 0, 4], [-1, 16, 16, 136, 8, 136, 8, 0, 0, 4], [-1, 16, 16, 152, 8, 152, 8, 0, 0, 4], - ]))) + ])), "Motion vectors should match the expected values") def test_read_first_ten_frames(self): @@ -110,7 +128,7 @@ def test_read_first_ten_frames(self): motion_vectors.append(motion_vector) frame_types.append(frame_type) - self.assertTrue(all(rets)) + self.assertTrue(all(rets), "All frames should be read successfully") self.assertEqual(frame_types, ['I', 'P', 'P', 'P', 'P', 'P', 'P', 'P', 'P', 'P']) [self.validate_frame(frame) for frame in frames] shapes = [ @@ -128,7 +146,7 @@ def test_frame_count(self): if not ret: break frame_count += 1 - self.assertEqual(frame_count, 337) + self.assertEqual(frame_count, 337, "Video file is expected to have 337 frames") def test_timings(self): @@ -151,6 +169,113 @@ def test_timings(self): self.assertLess(dt_std, 0.003, msg=f"Standard deviation of frame read duration exceeds maximum ({dt_std} s > {0.003} s)") + def test_skipping_frame_decoding_does_not_raise(self): + self.cap.set_decode_frames(False) + self.cap.set_decode_frames(True) + + + def test_read_first_I_frame_skipping_frame_decoding(self): + self.open_video() + self.cap.set_decode_frames(False) + ret, frame, motion_vectors, frame_type = self.cap.read() + self.assertTrue(ret, "Should succeed to read from video file") + self.assertEqual(frame_type, "I", "Frame type of first frame should be I") + self.assertIsNone(frame, "Frame should be None when skipping frame decoding") + self.validate_motion_vectors(motion_vectors) + + + def test_read_first_P_frame_skipping_frame_decoding(self): + self.open_video() + self.cap.set_decode_frames(False) + self.cap.read() # skip first frame (I frame) + ret, frame, motion_vectors, frame_type = self.cap.read() + self.assertTrue(ret, "Should succeed to read from video file") + self.assertEqual(frame_type, "P", "Frame type of second frame should be P") + self.assertIsNone(frame, "Frame should be None when skipping frame decoding") + self.validate_motion_vectors(motion_vectors, shape=(3665, 10)) + self.assertTrue(np.all(motion_vectors[:10, :] == np.array([ + [-1, 16, 16, 8, 8, 8, 8, 0, 0, 4], + [-1, 16, 16, 24, 8, 24, 8, 0, 0, 4], + [-1, 16, 16, 40, 8, 40, 8, 0, 0, 4], + [-1, 16, 16, 56, 8, 56, 8, 0, 0, 4], + [-1, 16, 16, 72, 8, 72, 8, 0, 0, 4], + [-1, 16, 16, 88, 8, 88, 8, 0, 0, 4], + [-1, 16, 16, 104, 8, 104, 8, 0, 0, 4], + [-1, 16, 16, 120, 8, 120, 8, 0, 0, 4], + [-1, 16, 16, 136, 8, 136, 8, 0, 0, 4], + [-1, 16, 16, 152, 8, 152, 8, 0, 0, 4], + ])), "Motion vectors should match the expected values") + + + def test_read_first_ten_frames_skipping_frame_decoding(self): + rets = [] + frames = [] + motion_vectors = [] + frame_types = [] + self.open_video() + self.cap.set_decode_frames(False) + for _ in range(10): + ret, frame, motion_vector, frame_type = self.cap.read() + rets.append(ret) + frames.append(frame) + motion_vectors.append(motion_vector) + frame_types.append(frame_type) + + self.assertTrue(all(rets), "All frames should be read successfully") + self.assertEqual(frame_types, ['I', 'P', 'P', 'P', 'P', 'P', 'P', 'P', 'P', 'P']) + [self.assertIsNone(frame) for frame in frames] + shapes = [ + (0, 10), (3665, 10), (3696, 10), (3722, 10), (3807, 10), + (3953, 10), (4155, 10), (3617, 10), (4115, 10), (4192, 10) + ] + [self.validate_motion_vectors(motion_vector, shape) for motion_vector, shape in zip(motion_vectors, shapes)] + + + def test_frame_count_skipping_frame_decoding(self): + self.open_video() + self.cap.set_decode_frames(False) + frame_count = 0 + while True: + ret, _, _, _ = self.cap.read() + if not ret: + break + frame_count += 1 + self.assertEqual(frame_count, 337, "Video file is expected to have 337 frames") + + + def test_skipping_frame_decoding_is_faster_than_not_skipping(self): + self.open_video() + # skip frame decoding + self.cap.set_decode_frames(False) + start_time = time.perf_counter() + frame_count = 0 + for _ in range(50): # read 50 frames + ret, _, _, _ = self.cap.read() + if not ret: + break + frame_count += 1 + mvo_time = time.perf_counter() - start_time + + # do not skip frame decoding + self.cap.set_decode_frames(True) + start_time = time.perf_counter() + frame_count_full = 0 + for i in range(50): # Read 50 frames + ret, _, _, _ = self.cap.read() + if not ret: + break + frame_count_full += 1 + full_time = time.perf_counter() - start_time + + self.assertEqual(frame_count, 50, "Should read 50 frames") + self.assertEqual(frame_count_full, 50, "Should read 50 frames") + + # Performance comparison (skipping decoding should be at least as fast as not skipping decoding mode) + if mvo_time > 0 and full_time > 0: + speedup = full_time / mvo_time + print(f"Speedup by skipping frame decoding: {speedup:.2f}x") + self.assertGreaterEqual(speedup, 1.0, "Skipping frame decoding should be reasonably fast") + if __name__ == '__main__': unittest.main()