Yt dlp proxy

ROADMAP.md

@@ -235,11 +235,11 @@
 - [x] Fix edit not shown in production
 - [x] is it fine that supervisord from root starts nginx not appuser
 - [ ] imba feature: on startup we check music folder and for files there and setting the same user rights and group - backend + production dockerfiles
-- [ ] can we show download progress somehow?
-- [ ] after yt-dlp we have to fire modal with saving - there is "awaiting_review" status but does not work now
+- [ ] critical mobile bug: on the end of track it switches to next but doesnt play automatically until open webpage. If i try to play from modal - does nothing first and when disappears
 - [ ] log what takes long on e2e testing
 - [ ] redis pub-sub for events
 - [ ] On Edit - if im not mistaken we do not check if file with same name exists, we should show UI warning and prevent save on such cases! that is server check
+- [ ] use custom play icon on timeline when shuffle because its ugly on w11
 - [ ] on close tab did not restore track - bug. Maybe we have to reimplement. Maybe we have to save that in local storage and send once in a while. UDP - do not need to wait 200.
 - [ ] Player footer desktop - on change windows calculate div for player controls - this will allow to have full size for artist-title
 - [ ] nginx not from root but from appuser - this will require some fixes because container won't run in that moment
@@ -259,13 +259,13 @@
   - [ ] has to be smaller by default and on hover it has to be bigger in size like now
   - [ ] styling for playing music - make it less colored but on hover make blue colored styling for slider
 - [ ] ~~Revert functionality UI~~
-- [ ] Remove non-docker development - not sure if thats needed - actually needed because AI doesnt understand what env im working in currently. Less commands is better
+- [x] Remove non-docker development - not sure if thats needed - actually needed because AI doesnt understand what env im working in currently. Less commands is better
 - [ ] Get rid of SQLModel, only sqlalchemy. Remove all warnings disabling. remove all # noqa: F401 - actually think to move everything in redis so there will be no
   sql db. But - think of relationships such model. Redis might have relationships or something.. I mean we can give up some consistency..
 - [ ] Sort tracks by different fields and ways
 - [x] Continue refactoring effects
 - [ ] fast search - has to be server side to look vk/yt - and download in future!
-- [ ] Download track functionality?
+- [ ] Download track functionality? only in edit window I think?
 - [ ] docker-compose - i think we dont need separated volumes for cover/db, might be single
 - [ ] production image nginx better logging. Logging across app (!!!) - Think about this more
 - [ ] Hotkeys for player controls
@@ -283,6 +283,8 @@
 - [ ] yt-dlp from yt (list domains)
 - [ ] yt-dlp from other sites?
 - [ ] functionality to automatically update yt-dlp? on startup?
+- [ ] can we show download progress somehow?
+- [ ] after yt-dlp we have to fire modal with saving - there is "awaiting_review" status but does not work now
 
 ## Phase vk / other services ?
 
@@ -305,3 +307,4 @@
 
 - [ ] Every minute check if folder still read-only or writeable - this might be one of the little things nobody notices
 - [ ] The part of adding a group/user/etc has to go on executing script - we read music folder and get user rights
+- [ ] Autoplay slider on right sidebar

backend/pyproject.toml

@@ -26,7 +26,6 @@ dependencies = [
   "vulture>=2.14",
   "bandit>=1.8.5",
   "python-multipart>=0.0.20",
-  "yt-dlp>=2025.08.22",
 ]
 
 [build-system]

backend/scripts/update_ytdlp.py

@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""
+Script to update/install yt-dlp binary and yt-dlp-proxy.
+This script handles the installation and updating of both yt-dlp and yt-dlp-proxy.
+"""
+
+import asyncio
+import logging
+import subprocess
+import sys
+from pathlib import Path
+
+logger = logging.getLogger(__name__)
+
+
+class YtDlpUpdater:
+    def __init__(self):
+        pass
+
+    def _run_command(self, cmd: list[str], check: bool = True, timeout: int = 300, cwd: str = None) -> subprocess.CompletedProcess:
+        """Run a command with proper error handling."""
+        logger.info(f"Running command: {' '.join(cmd)}")
+        try:
+            result = subprocess.run(
+                cmd,
+                capture_output=True,
+                text=True,
+                check=check,
+                timeout=timeout,
+                cwd=cwd
+            )
+            if result.stdout:
+                logger.debug(f"Command stdout: {result.stdout}")
+            if result.stderr:
+                logger.debug(f"Command stderr: {result.stderr}")
+            return result
+        except subprocess.TimeoutExpired as e:
+            logger.error(f"Command timed out after {timeout}s: {' '.join(cmd)}")
+            raise
+        except subprocess.CalledProcessError as e:
+            logger.error(f"Command failed with exit code {e.returncode}: {' '.join(cmd)}")
+            logger.error(f"stderr: {e.stderr}")
+            raise
+
+    def check_yt_dlp_available(self) -> bool:
+        """Check if yt-dlp is available."""
+        logger.info("Checking yt-dlp availability...")
+
+        try:
+            result = self._run_command([sys.executable, "-m", "yt_dlp", "--version"])
+            logger.info(f"yt-dlp version: {result.stdout.strip()}")
+            return True
+        except Exception as e:
+            logger.error(f"yt-dlp not available: {e}")
+            return False
+
+    def install_yt_dlp_proxy(self) -> bool:
+        """Install or update yt-dlp-proxy from GitHub."""
+        logger.info("Installing/updating yt-dlp-proxy...")
+
+        import tempfile
+        import shutil
+
+        try:
+            with tempfile.TemporaryDirectory() as temp_dir:
+                temp_path = Path(temp_dir)
+                repo_path = temp_path / "yt-dlp-proxy"
+
+                # Clone the repository
+                self._run_command([
+                    "git", "clone", "https://github.com/Petrprogs/yt-dlp-proxy.git",
+                    str(repo_path)
+                ])
+
+                # Install dependencies
+                self._run_command([
+                    "uv", "pip", "install", "-r", "requirements.txt"
+                ], cwd=str(repo_path))
+
+                # Create permanent installation directory
+                install_path = Path.home() / ".local" / "share" / "yt-dlp-proxy"
+                if install_path.exists():
+                    shutil.rmtree(install_path)
+                shutil.copytree(repo_path, install_path)
+
+                # Create wrapper script
+                wrapper_path = Path.home() / ".local" / "bin" / "yt-dlp-proxy"
+                wrapper_path.parent.mkdir(parents=True, exist_ok=True)
+
+                wrapper_content = f'''#!/usr/bin/env python3
+import sys
+sys.path.insert(0, "{install_path}")
+from main import main
+if __name__ == "__main__":
+    main()
+'''
+                wrapper_path.write_text(wrapper_content)
+                wrapper_path.chmod(0o755)
+
+                # Verify installation
+                self._run_command([str(wrapper_path), "--help"])
+                logger.info("yt-dlp-proxy installed successfully")
+
+                return True
+        except Exception as e:
+            logger.error(f"An error occurred during yt-dlp-proxy installation: {e}", exc_info=True)
+            return False
+
+    def update_yt_dlp_proxy(self, max_workers: int = 4) -> bool:
+        """Update yt-dlp-proxy with specified max workers."""
+        logger.info(f"Updating yt-dlp-proxy with max-workers={max_workers}...")
+        proxy_path = str(Path.home() / ".local" / "bin" / "yt-dlp-proxy")
+
+        try:
+            self._run_command([
+                proxy_path, "update",
+                "--max-workers", str(max_workers)
+            ])
+            logger.info("yt-dlp-proxy updated successfully")
+            return True
+        except Exception as e:
+            logger.error(f"An error occurred while updating yt-dlp-proxy proxy list: {e}", exc_info=True)
+            return False
+
+    def update_all(self, max_workers: int = 4) -> bool:
+        """Update yt-dlp-proxy (yt-dlp is installed during container build)."""
+        logger.info("Starting update process...")
+
+        if not self.check_yt_dlp_available():
+            logger.error("yt-dlp not available - should be installed during container build")
+            return False
+
+        if not self.install_yt_dlp_proxy():
+            logger.error("Failed to install yt-dlp-proxy.")
+            return False
+
+        if not self.update_yt_dlp_proxy(max_workers):
+            logger.warning("Failed to update proxy list for yt-dlp-proxy. The tool is installed but may use stale proxies.")
+
+        logger.info("Update process completed.")
+        return True
+
+
+
+
+async def main():
+    """Main entry point for the script."""
+    logging.basicConfig(
+        level=logging.INFO,
+        format="%(levelname)s: %(name)s - %(message)s"
+    )
+
+    updater = YtDlpUpdater()
+
+    max_workers = 4
+    if len(sys.argv) > 1:
+        try:
+            max_workers = int(sys.argv[1])
+        except ValueError:
+            logger.warning(f"Invalid max_workers value: {sys.argv[1]}, using default: 4")
+
+    success = updater.update_all(max_workers)
+    sys.exit(0 if success else 1)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

backend/src/mus/infrastructure/jobs/download_jobs.py

@@ -1,7 +1,5 @@
-import asyncio
 import logging
 import shutil
-import subprocess  # nosec B404
 import tempfile
 from pathlib import Path
 from urllib.parse import urlparse
@@ -11,6 +9,7 @@
 from src.mus.core.streaq_broker import worker
 from src.mus.infrastructure.api.sse_handler import notify_sse_from_worker
 from src.mus.infrastructure.jobs.file_system_jobs import handle_file_created
+from src.mus.infrastructure.services.ytdlp_service import ytdlp_service
 
 
 def _validate_url(url: str) -> bool:
@@ -37,7 +36,7 @@ async def download_track_from_url(url: str):
         if not _validate_url(url):
             raise ValueError("Invalid URL format")
 
-        output_path = await asyncio.to_thread(_download_audio, url, logger)
+        output_path = await _download_audio(url, logger)
 
         await set_app_write_lock(output_path)
 
@@ -76,7 +75,7 @@ async def download_track_from_url(url: str):
     logger.info(f"WORKER: Completed download for URL: {url}")
 
 
-def _download_audio(url: str, logger: logging.Logger) -> str:
+async def _download_audio(url: str, logger: logging.Logger) -> str:
     with tempfile.TemporaryDirectory(prefix="mus-download-") as temp_dir_str:
         temp_dir = Path(temp_dir_str)
         logger.info(f"WORKER: Using temporary directory for download: {temp_dir}")
@@ -85,42 +84,12 @@ def _download_audio(url: str, logger: logging.Logger) -> str:
             temp_dir / "%(artist,uploader|Unknown Artist)s - %(title)s.%(ext)s"
         )
 
-        cmd = [
-            "yt-dlp",
-            "--format",
-            "bestaudio/best",
-            "--extract-audio",
-            "--audio-format",
-            "mp3",
-            "--audio-quality",
-            "0",
-            "-o",
-            output_template,
-            "--embed-thumbnail",
-            "--convert-thumbnails",
-            "jpg",
-            "--embed-metadata",
-            "--parse-metadata",
-            "title:%(title)s",
-            "--parse-metadata",
-            "artist:%(artist,uploader|Unknown Artist)s",
-            "--sponsorblock-remove",
-            "all",
-            "--embed-chapters",
-            "--concurrent-fragments",
-            "3",
-            "--throttled-rate",
-            "100K",
-            "--retries",
-            "10",
-            "--no-playlist",
-            url,
-        ]
-
         try:
-            result = subprocess.run(
-                cmd, capture_output=True, text=True, check=True, timeout=600
-            )  # nosec B603
+            result = await ytdlp_service.download_with_fallback_update(
+                url=url,
+                output_template=output_template,
+                max_workers=4
+            )
 
             all_output = result.stdout + result.stderr
             downloaded_file_path = None
@@ -137,7 +106,7 @@ def _download_audio(url: str, logger: logging.Logger) -> str:
                             break
                         else:
                             logger.warning(
-                                "WORKER: yt-dlp reported a file outside of temp directory: "
+                                "WORKER: yt-dlp-proxy reported a file outside of temp directory: "
                                 f"{filename_str}"
                             )
 
@@ -154,12 +123,6 @@ def _download_audio(url: str, logger: logging.Logger) -> str:
 
             return str(final_path)
 
-        except subprocess.TimeoutExpired as e:
-            logger.error(f"WORKER: yt-dlp subprocess timed out: {e.stderr}")
-            raise Exception("Download timed out after 10 minutes") from e
-        except subprocess.CalledProcessError as e:
-            logger.error(f"WORKER: yt-dlp subprocess error: {e.stderr}")
-            raise Exception(f"Download failed: {e.stderr}") from e
         except Exception as e:
             logger.error(f"WORKER: Download error: {str(e)}")
             raise