Video Stream Analysis & WebRTC Playback #82
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lysis Add comprehensive video and audio stream analysis capabilities: Video Detection (video_detect.c/h): - RTP payload type identification for H.264, H.265, MPEG-4, MJPEG - NAL unit parsing with emulation prevention byte removal - SPS/PPS extraction for codec parameter detection - Resolution and framerate extraction from video headers - Support for both single NAL and aggregation packets (STAP-A, FU-A) Video Metrics (video_metrics.c/h): - RTP sequence tracking with loss detection - Inter-packet gap (IPG) measurement and histograms - Video jitter calculation per RFC 3550 - Frame size histograms (20 buckets covering VoIP to MTU sizes) - Packets-per-second (PPS) distribution histograms - Bitrate calculation from RTP timestamps RTSP Signaling Tap (rtsp_tap.c/h): - Passive observation of RTSP DESCRIBE/SETUP/PLAY - SDP parsing for media stream parameters - Track-to-port mapping for RTP stream correlation - Session lifecycle tracking TCP Analysis Enhancements: - RTT histogram with 14 log-scale buckets (10µs to 10s+) - Health status classification (good/warning/problem) - Health flags for specific issues (tail latency, loss, stalls) - IPG histogram for all flows (12 log-scale buckets) Extended Flow Record (flow.h): - New fields for video codec, resolution, framerate - Audio type and codec detection - Histogram arrays for RTT, IPG, jitter, frame size, PPS - Health status and flags
WebRTC Video Bridge (webrtc_bridge.c/h): - GStreamer-based RTP to WebRTC transcoding pipeline - Dynamic H.264 profile selection from browser SDP capabilities - Supports Baseline, Main, and High profiles with level negotiation - Session management with automatic cleanup on disconnect - Thread-safe design with mutex-protected state Protocol Extensions (messages/): - New message types for WebRTC signaling (180-184): - JT_MSG_WEBRTC_OFFER/ANSWER for SDP exchange - JT_MSG_WEBRTC_ICE for ICE candidate exchange - JT_MSG_WEBRTC_START/STOP for session control - Extended toptalk message format with: - Video telemetry: codec, resolution, framerate, bitrate, jitter - Audio telemetry: type, codec, jitter - TCP health: status, flags, RTT histogram - Flow histograms: IPG, frame size, PPS Server Integration: - WebRTC message handlers in jt_server_message_handler.c - Toptalk thread integration for video/audio stream detection - Updated compression dictionary for new telemetry fields - Increased MAX_JSON_MSG_LEN to 64KB for histogram data Build System: - Optional ENABLE_WEBRTC_PLAYBACK flag for GStreamer dependency - Debug/release build modes with sanitizer support - New test targets for video functionality
…ements WebRTC Video Playback (jittertrap-webrtc.js): - In-browser video playback via WebRTC - SDP offer/answer exchange with server - ICE candidate handling for NAT traversal - Play button in Top Talkers legend for video flows - Draggable, resizable video overlay with close button Top Talkers Enhancements (jittertrap-chart-toptalk.js): - Expandable video/audio telemetry rows showing codec, resolution, bitrate - TCP health indicators (✓/⚠/✗) based on RTT distribution - Expandable health details with RTT histogram visualization - IPG, frame size, and PPS histogram displays - Click-to-select on byte distribution bar segments - Bitrate column in flow legend - Removed redundant hover tooltips (~200 lines) Chart Improvements: - Flow selection highlighting synced across all charts - Logarithmic Y-axis scale for RTT chart (better for wide value ranges) - SVG clip paths to prevent overdraw during scrolling - Improved axis label readability Performance Optimizations (jittertrap-core.js): - Object pooling for data points to reduce GC pressure - Custom stack implementation reusing arrays - Cached tick formatters to avoid closure allocation - Throttled legend updates (10Hz vs 60Hz) - Smooth chart scrolling with interval refinement - Shared X-axis domain calculation across charts Documentation (modal-help.part.html): - New TCP Analysis section with troubleshooting guidance - Enhanced Video & Audio Stream Analysis section - NAL unit explanation with diagrams and encoding table Styling (jittertrap.css): - Video overlay and controls styling - Health indicator and histogram bar styles - Expandable row animations
GitHub Actions: - New test_asan job running AddressSanitizer tests - Installs build dependencies and runs make test-asan Build System: - Debug/release build modes (BUILD_TYPE variable) - AddressSanitizer targets: test-asan, test-video-asan - Fuzzing targets: fuzz-build, fuzz, fuzz-quick - Fix scan-build to use $(MAKE) for proper recursion Test Infrastructure: - Comprehensive video detection test suite (test_video_detect.c) - Tests for H.264/H.265 NAL parsing, SPS extraction - RTP packet decoding and aggregation packet handling - Emulation prevention byte removal verification
Fix two memory leaks found when running the test suite with ASan:
1. messages/src/jt_msg_toptalk.c (jt_toptalk_packer):
- Use json_object_set_new() instead of json_object_set() for
timestamp fields to properly transfer ownership
- Remove redundant json_decref(timestamp) since ownership is
now transferred to params via json_object_set_new()
2. server/test_slist.c:
- Add cleanup code at end of test to free allocated nodes
- Free the slist meta structure and head node
- Add return statement
These leaks were pre-existing but only caught now that ASan
testing was added to the CI workflow.
Add documentation for the optional WebRTC video playback feature: Makefile: - Add ENABLE_WEBRTC_PLAYBACK to `make config` output under Feature Flags README.md: - Add "Video & Audio Stream Analysis" to Key Features section - Add "Optional Features" section explaining how to enable WebRTC playback and the libdatachannel dependency The in-app Help modal already documents this feature comprehensively, and a runtime alert guides users if the server lacks WebRTC support.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive video/audio stream detection, quality metrics tracking, and in-browser WebRTC video playback to JitterTrap.
It enables real-time visibility into RTP and MPEG-TS streams with deep packet inspection—no configuration required.
Features
🎬 Video Stream Detection
📊 Video Quality Metrics
📺 WebRTC Video Playback
🩺 TCP Health Indicators
⚡ Client Performance
Build Options
Standard build (video detection enabled, WebRTC disabled)
make
With WebRTC playback support (requires libdatachannel)
ENABLE_WEBRTC_PLAYBACK=1 make
CI Improvements
Added AddressSanitizer workflow to catch memory leaks and buffer overflows. This already identified and fixed two pre-existing memory
leaks:
Screenshots
Add screenshots of the video stream detection UI and WebRTC player overlay here
Testing
Documentation