English Version ---- 中文介绍
AI-Powered Hard Subtitle Removal, Translation and Dubbing Suite for Global Short Drama Distribution
Simvoo AI is a digital service platform focused on overseas short drama and film & TV content distribution, intelligent video processing and global communication. We deliver efficient, intelligent and one-stop technical solutions for the short drama localization industry.
Powered by cutting-edge large AI models, audio-video codec and computer vision technologies, we have independently developed a full-stack video processing toolchain. It covers hard subtitle erasure, text detection, translation, dubbing and subtitle regeneration, greatly improving the efficiency of short drama content localization and global distribution.
This repository provides an open-source, AI-driven subtitle erasure and processing pipeline, purpose-built for short drama re-creation and localization across YouTube, TikTok, Facebook and other global platforms.
- Hard subtitle erasure & inpainting: Automatically detects and removes embedded hard subtitles, watermarks and logos from video frames via AI inpainting, restoring clean background footage for secondary localization.
- One-click subtitle extraction & dubbing: End-to-end workflow for dubbed drama production, which extracts subtitle content, completes translation and voiceover generation in a single pipeline, optimized for overseas short drama release.
- Multi-language subtitle translation & regeneration: One-click full-language subtitle translation for Chinese short dramas, supporting both subtitle-only localization and regenerated standard subtitles after erasure.
- Precise audio-video separation & speaker diarization: Accurately separates audio tracks and identifies all speakers in the plot, accurately mapping subtitles to corresponding characters to boost production efficiency.
- Full-cycle subtitle management: Smart subtitle generation, proofreading, localization adaptation and multi-platform compatibility, supporting global short drama dissemination across multiple language versions.
SVC (Simvoo Video Completion) is a video processing system focused on burned-in subtitle removal. It has been integrated into the Simvoo AI (Simvoo.com) content processing platform.
The system is primarily designed for handling subtitle overlays in short-form videos, film and television content, and social media videos. While preserving the structural integrity of the original footage, SVC enables high-quality reconstruction of subtitle-covered regions.
SVC is not a single model. Instead, it is a video processing system built through the engineering integration and optimization of existing open-source video restoration research, including image inpainting and temporal consistency modeling approaches.
Burned-in subtitle removal is essentially a form of:
Structured Occlusion Video Inpainting
Unlike general image restoration tasks, subtitles exhibit several unique characteristics:
-
High-contrast edges (outlines and shadows)
-
Fixed or semi-fixed placement (typically near the bottom of the frame)
-
Potential overlap with foreground subjects
-
Strong consistency of occlusion across consecutive frames
Therefore, this task is not simply image completion, but also involves video-level consistency reconstruction.
SVC adopts a multi-stage processing pipeline. The core modules are reconstructed and combined based on existing open-source methods.
The system first identifies subtitle regions, primarily based on:
-
Text detection models (OCR + text-region detectors)
-
Edge enhancement analysis
-
Visual saliency region filtering
This stage outputs pixel-level masks rather than simple bounding boxes.
To address subtitle characteristics such as outlines, shadows, and transparency, masks are refined through:
-
Edge expansion
-
Semi-transparent region compensation
-
Multi-frame consistency alignment
This step is designed to reduce the residual artifacts commonly caused by traditional OCR-based bounding boxes.
The core restoration stage is reconstructed based on existing open-source video restoration frameworks, including:
-
LaMa (image-level inpainting)
-
ProPainter (video-level temporal restoration)
-
STTN-style temporal consistency methods
SVC does not rely on a single model. Instead, restoration strategies are dynamically selected according to scene complexity:
-
Static backgrounds → Image-level restoration
-
Moderate motion → Hybrid restoration
-
High motion / complex textures → Video-level temporal restoration
To address potential flickering between video frames, the system introduces:
-
Inter-frame consistency constraints
-
Local optical-flow alignment strategies
-
Stability reprojection for restored regions
These techniques help reduce visual jitter across consecutive frames.
The benchmark dataset consists of a mixture of real-world short-form videos and film/television content.
-
Video clips: 18,200+
-
Total duration: Approximately 150 hours
-
White subtitles with black outlines
-
Colored danmu-style subtitles
-
Semi-transparent subtitle bars
-
Dynamically positioned subtitles
Content cannot currently be disclosed outside internal Feishu documentation.
The system is evaluated using the following metrics:
-
Visual Artifact Rate
-
Temporal Stability Score
-
Structural Consistency
-
Human Preference Score (blind human evaluation)
Content cannot currently be disclosed outside internal Feishu documentation.
-
In natural scenes (such as indoor interviews and short dramas), restored regions are generally imperceptible during normal playback.
-
In complex texture regions (such as striped clothing and checkerboard patterns), minor structural errors may still occur.
-
Compared with single-frame restoration approaches, video flickering is significantly reduced.
Under a standard 1080P testing environment:
- 1.6× ~ 3.5× real-time speed (depending on scene complexity)
- GPU utilization is optimized through staged scheduling.
Compared with diffusion-based frame-by-frame restoration approaches:
- Computational cost is reduced by approximately 45%–70%.
SVC has been integrated as a core video pre-processing module within the Simvoo AI content production platform.
In practical production workflows, SVC is used for:
-
Burned-in subtitle removal
-
Source material preparation for content re-editing
-
Pre-processing for multilingual content adaptation
Users can access and experience the system through Simvoo:
Current optimization efforts are mainly focused on engineering efficiency and performance under extreme scenarios.
Further improvements are still needed for scenarios such as:
-
Striped clothing
-
Checkerboard patterns
-
High-frequency repetitive texture backgrounds
Key optimization targets include:
-
GPU scheduling efficiency
-
Batch processing throughput
-
Long-video segmentation strategies
Ongoing work focuses on reducing accumulated errors during long-duration video processing and improving overall consistency.
SVC (Simvoo Video Completion) is a video subtitle removal system designed for real-world production environments.
By combining existing open-source video restoration methods with engineering-level reconstruction and scheduling optimization, SVC achieves high-quality subtitle removal performance across most real-world video scenarios.
The system is currently running stably within the Simvoo AI platform and continues to improve in areas such as complex texture restoration and computational efficiency.