Video over the network

[josuah]

This is a thread to gather information and keep track of the progress of video over the network for anyone interested in implementing it.

Topics include:

• Control protocol implementation (i.e. ONVIF, WebRTC, SIP)
• Streaming protocol implementation (i.e. SRT, RTP)
• Video stream container formats (i.e. MPEG-TS)
• Video pipeline control framework (i.e. Gstreamer, FFmpeg)
• Video compression support (i.e. H.264)
• Other initiatives (OpenHD, OpenIPC)
• Collaboration with other RTOS and ecosystems.
• Integration with other areas (i.e. NPU for detecting events on the feed, displays)

[josuah]

General discussion

When a topic does not really fit other categories, or simply to say hi.

[kartben]

what's the status of https://docs.zephyrproject.org/latest/samples/drivers/video/tcpserversink/README.html -- has someone tested it recently?

[josuah]

I tried it with native_sim about a month ago, and am testing now again:

Full details here.

P.S. I should remember to subscribe to my own discussion threads so that I get notifications.

[josuah]

Maybe some day we can have a Zephyr <-> Zephyr video streaming served on a static webpage: the USB Video Webcam sample for the local camera, SRTP (WebRTC) for the remote camera.

[josuah]

Streaming/control protocol (SRT, RTP, WebRTC, SIP, ONVIF...)

• https://en.wikipedia.org/wiki/Real-time_Transport_Protocol

[sayednasermoravej]

These are three types of protocols.

1. Control protocols.
   Streaming control protocols are communication protocols that manage how media (audio, video, etc.) is delivered and controlled during streaming over a network (especially IP networks). These protocols do not necessarily carry the media itself, but rather handle setup, control, synchronization, and teardown of media streams.

Common Streaming Control Protocols: SIP, SDP, ONVIF, WebRTC signaling.

SIP (Session Initiation Protocol):
Initiates, modifies, and terminates multimedia sessions (calls). Used with SDP for session description.
Common Use: VoIP, video calls

SDP (Session Description Protocol):
Describes multimedia sessions (used within RTSP, SIP). Includes codec info, ports, etc.
Common Use: VoIP, conferencing, RTSP

ONVIF(Open Network Video Interface Forum):
A control standard over SOAP/HTTP used in IP-based security products. Includes stream control and PTZ camera control.
Common Use: CCTV, security systems

WebRTC signaling (custom via WebSocket, SIP, etc.):
Media exchange in WebRTC is P2P, but signaling (control) uses external protocols like SIP or custom WebSocket.
Common Use: Web video calls, real-time communications

2. Streaming protocols.
   Streaming protocols define how media data is transmitted over a network so that it can be played back in real time or near-real time. These protocols manage, Media packetization, Buffering, Adaptation to bandwidth, and Live or on-demand playback.

Common Streaming Protocols: SRT, RTMP, RTP, WebRTC.

SRT (Secure Reliable Transport):
Transport Layer: UDP
Use Case: Remote video contribution, broadcasting

RTMP (Real-Time Messaging Protocol):
Transport Layer: TCP
Use Case: OBS → YouTube/Facebook

RTP (Real-time Transport Protocol): Low-latency, used with SIP/RTSP
Transport Layer: UDP
Use Case: VoIP, WebRTC, IP cameras

WebRTC
Transport Layer: UDP +SRTP
Use Case: Video chat, conferencing

3. Hybrid or Control-Enabled Protocols (Control protocols + Streaming protocols)

RTSP (Real Time Streaming Protocol):
Client-server protocol to control media sessions (e.g., play, pause, teardown). Often used with RTP for transport.
Common Use: IP cameras, surveillance, VoD

HLS (HTTP Live Streaming) – control built into HTTP:
Apple's streaming protocol using chunked files and playlists. Control via HTTP playlist updates.
Common Use: Web, mobile, live events

DASH (MPEG-DASH):
Adaptive bitrate streaming over HTTP. Similar to HLS. Control via manifest files (MPD).
Common Use: YouTube, Netflix

[josuah]

Thanks for the investigation @sayednasermoravej! That's great to have a high-level overview.

Through this it seems like RTP/SRTP are building blocks for about everything. Sometimes with long chains like SIP -> SDP -> RTSP -> RTP. Sometimes as only one supported protocol like WebRTC (SRTP = RTP + AES).

That could be some interesting upgrade path to the tcpserversink sample so that the user does not have to specify the format/resolution etc.

So if I understand, no camera control (PTZ, exposure) and no format negotiation (which resolution/FPS/bandwidth is supported on each end) yet with only RTP (a streaming protocol). This would need an extra control protocol to be supported, which would become a lot easier to implement once RTP is there... and be helped by some framework to deal with media (audio/video/metadata) like Gstreamer.

[sayednasermoravej]

You're welcome!

You're right — RTP/SRTP are indeed the main building blocks.

RTP typically runs on top of UDP.

And yes, to support features like PTZ control, an additional control protocol is needed.

[beriberikix]

Summoning @Sean-Der, who is the WebRTC expert and has been doing more with embedded lately.

[josuah]

Video stream container formats (MPEG-TS...)

• https://en.wikipedia.org/wiki/Comparison_of_video_container_formats

[josuah]

Video pipeline framework (like Gstreamer)

On Linux there is Gstreamer, FFmpeg, PipeWire, libcamera...

Together they provide a fairly complete abstraction of the video <-> network connection and are bedrock for supporting video streaming protocols.

[josuah]

Hardware compression support (MJPEG, H.264, H.265...)

[josuah]

H.264 decoding, test data

I ordered this camera and captured a raw stream with this command:

v4l2-ctl --stream-mmap=100 --device /dev/video6 --stream-to=h264_raw_data_from_arducam_b0261.bin

h264_raw_data_from_arducam_b0261.bin.zip

ffprobe /tmp/h264_raw_data_from_arducam_b0261.bin:

Input #0, h264, from '/tmp/h264_raw_data_from_arducam_b0261.bin':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1280x720, 25 fps, 1200k tbr, 1200k tbn

file /tmp/h264_raw_data_from_arducam_b0261.bin:

/tmp/h264_raw_data_from_arducam_b0261.bin: JVT NAL sequence, H.264 video, main @ L 41

hexdump -C /tmp/h264_raw_data_from_arducam_b0261.bin | head

00000000  00 00 00 01 67 4d 40 29  96 54 02 80 2d c8 00 00  |....gM@).T..-...|
00000010  00 01 68 ee 38 80 00 00  00 01 65 88 80 40 00 49  |[email protected]|
00000020  f5 00 f1 01 28 e0 06 e9  8a 07 7d f3 00 bc d9 f5  |....(.....}.....|
00000030  b7 19 ab e3 7b 16 16 ae  f3 95 7e db 27 d4 5b fd  |....{.....~.'.[.|
00000040  2f 0a 6d 09 5a e5 de 78  cb 6e ce b7 4a 50 98 65  |/.m.Z..x.n..JP.e|
00000050  a5 c4 74 0e 73 56 71 3f  d2 5a f4 19 61 a6 cf 94  |..t.sVq?.Z..a...|
00000060  43 6f 83 fb ba 13 68 3f  bb ca 15 f7 6a f8 35 a0  |Co....h?....j.5.|
00000070  42 fa 3f 8f e0 b2 f5 10  b2 40 b9 bc 47 e1 6e 66  |[email protected]|
00000080  39 34 d2 dd 0e f1 9f 89  2d 5d 6a 45 a0 0b bc 60  |94......-]jE...`|
00000090  e5 71 97 b3 61 f9 ae 11  6f be 7f 76 b1 ff b3 62  |.q..a...o..v...b|

h264_analyze -o /dev/stdout /tmp/h264_raw_data_from_arducam_b0261.bin | head

!! Found NAL at offset 4 (0x0004), size 10 (0x000A)
0.8: forbidden_zero_bit: 0
0.7: nal->nal_ref_idc: 3
0.5: nal->nal_unit_type: 7
1.8: sps->profile_idc: 77
2.8: sps->constraint_set0_flag: 0
2.7: sps->constraint_set1_flag: 1
2.6: sps->constraint_set2_flag: 0
2.5: sps->constraint_set3_flag: 0
2.4: sps->constraint_set4_flag: 0

• h264_analyze can be obtained by compiling the https://github.com/aizvorski/h264bitstream library (no dependency, compiles easily).

• Online software: https://mradionov.github.io/h264-bitstream-viewer/

• Guide for decoding H.264 using ffmpeg and other tools: https://developer.ridgerun.com/wiki/index.php/H264_Analysis_Tools

• The raw "H.264 bitstream" (#1) is documented in the ITU document annex B: https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-H.264-200305-S!!PDF-E&type=items (PDF of annex B only)

[thecapn32]

Here is a brief H.264 introduction for anyone who might be interested to start working on this topic.

H.264

H.264 is a compression standard that converts raw video frames into compressed
binary bitstream for storage and transmission.

NAL(Network Abstraction Layer)

NAL(Network Abstraction Layer) Units are the fundamental building blocks of
every H.264 bitstream. Think of them as "packets" or "containers" that hold
different types of video data.

NAL Unit Structure

  ┌──────────────┬────────────────────────┐
  │  NAL Header  │     NAL Payload        │
  │   (1 byte)   │   (variable length)    │
  └──────────────┴────────────────────────┘

NAL Unit Header

Bit:  7   6   5   4   3   2   1   0
    ┌───┬───┬───┬───┬───┬───┬───┬───┐
    │ F │NRI│NRI│ T │ T │ T │ T │ T │
    └───┴───┴───┴───┴───┴───┴───┴───┘
     │   └─────┘   └─────────────┘
     │      │             │
     │      │             └── NAL Unit Type (5 bits)
     │      └─────────────────── NRI - Reference Indicator (2 bits)
     └────────────────────────── F - Forbidden Zero Bit (1 bit)

NAL Unit Types

Type	Value	Meaning
1	0x01	Non-IDR Slice (P or B frame)
5	0x05	IDR Slice (I-frame)
6	0x06	SEI (Supplemental Info)
7	0x07	SPS (Sequence Param Set)
8	0x08	PPS (Picture Param Set)
9	0x09	Access Unit Delimiter
0	0x00	Unspecified
10+	10+	Reserved or for extensions

Bitstream Organization Patterns

SPS → PPS → IDR → B → B → P → B → B → P → ...
 │     │     │    │           │
 │     │     │    └─ Depends on IDR and P
 │     │     └ Keyframe       └─ Depends on previous P and next P
 │     └ Picture params
 └─ Sequence params

NAL Unit Framing

Annex B Format

In annex b format NAL units are seperated by start codes. Some example usecase
would be in broadcasting, realtime encoding(V4L2) and raw h264 files, etc.

Hex: 00 00 00 01 [NAL Unit 1] 00 00 00 01 [NAL Unit 2] ...
    └Start Code┘             └Start Code┘

AVCC Format

In AVCC format there is no start code and each NAL unit is prefixed by its
length (usually 4 bytes). Some example usecases would be in ISO BMFF(MP4)
container, WebRTC and FLV, etc.

Hex: 00 00 01 23 [291 bytes of NAL data] 00 00 00 45 [69 bytes of NAL data]
     └Length=291┘                       └Length=69─┘

[josuah]

Very neat! It's immediately possible to understand the raw data captured from the webcam (like you said in another chat):

00000000  00 00 00 01 67 4d 40 29  96 54 02 80 2d c8 00 00  |....gM@).T..-...|
00000010  00 01 68 ee 38 80 00 00  00 01 65 88 80 40 00 49  |[email protected]|

We recognize the 00 00 00 01 start code there.

This means some conversion to do while getting the data from the hardware (Annex B framing) to the network (AVCC framing)

[josuah]

Other initiatives (OpenHD, OpenIPC, other RTOS)

• https://github.com/OpenIPC
• https://github.com/OpenHD

[josuah]

Good reminder about Espressif who will probably want something robust for ESP32-P4 (H.264).
Perhaps worth sharing some common dependencies to allow both to patch the same libraries.

https://zoneminder.com/ is auto-supporting ONVIF, which uses RTMP, which uses (S)RTP/RTCP, so this is all points at the same low-level level block (S)RTP for streaming.

Audio-side, using all the infrastructure from SOF sounds like a good opportunity!

Thank you!

[beriberikix]

Several other vendors have h264 acceleration, like the STM32N6, so worth getting their input as well!

[josuah]

Good point, things seems to be moving vendor-side.

• STM32N6 driver is underway! video: introduction of STM32 VENC driver for H264 hardware video compression #92884
• ESP32P4 is not yet ported to Zephyr (but apparently planned to happen)
• i.MXRT1170 is capable of H.264 via the GPL x264 library
• USB Video Host would allow to plug H.264-enabled webcams with bandwidth fitting USB FullSpeed 12 MHz

[josuah]

Looking at libpeer dependencies, almost all parts have Zephyr equivalents:

It seems to be feasible to use an unmodified version of Cisco's libsrtp, which is mostly srtp.c + crypto integration layer, with mbedtls as one of the option.

This can be leveraged for each use-cases of CCTV (ONVIF), real-time chat (WebRTC, VoIP/SIP), and media broadcasting.
Maybe libsrtp folks would be interested back in Zephyr support.

[josuah]

Gstreamer also uses libsrtp:
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-bad/ext/srtp/gstsrtpenc.c#L61