UtsukushiiAI

Generative Manga-to-Shorts Animation Platform

Transform static manga panels into animated short-form videos using AI — panel detection, character animation, depth-based parallax, and beat-synced video composition, all in one pipeline.

---

What It Does

UtsukushiiAI is a full-stack generative AI platform that takes manga pages as input and outputs animated short-form videos (MMV format). The system detects individual panels, segments characters, estimates depth for parallax effects, analyzes audio beats, animates characters using Stable Video Diffusion, and composes the final video with FFmpeg — all orchestrated through an event-driven async architecture.

---

ML Pipeline

Stage	Model	Purpose
1. Panel Detection	YOLOv12 (fine-tuned on Manga109)	Detect panel bounding boxes
2. Instance Segmentation	SAM 2 (Segment Anything)	Isolate characters & foreground
3. Depth Estimation	MiDaS	Generate depth maps for parallax
4. Audio Analysis	Librosa (HPSS)	Extract BPM, onsets, beat markers
5. Character Animation	Stable Video Diffusion	Animate segmented characters
6. Video Composition	FFmpeg	Compose layers, sync to beats → MP4

---

Tech Stack

Frontend (apps/web) Next.js 15 (App Router) React + TypeScript Zustand (state management) WebSocket (real-time progress)

Backend (apps/api) Express.js + TypeScript JWT Authentication MongoDB + Redis WebSocket server

ML Worker (apps/worker) FastAPI (Python) YOLOv12, SAM 2, MiDaS, SVD Librosa, FFmpeg Docker containerized

Monorepo: Turborepo · Shared packages: database, cache, storage, types

---

Architecture

User Upload (Manga Image)
        ↓
Panel Detection (YOLOv12)
        ↓
Instance Segmentation (SAM 2)
        ↓
Depth Estimation (MiDaS)   +   Audio Analysis (Librosa)
        ↓                               ↓
Character Animation (SVD)       Beat Markers
        ↓                               ↓
              Video Composer (FFmpeg)
                        ↓
              Final MP4 (9:16, 30fps)

Design Patterns: Event-Driven · CQRS · Repository · Factory · Observer (WebSocket job updates)

---

Project Structure

utsukushii-ai/
├── apps/
│   ├── web/        # Next.js 15 frontend
│   ├── api/        # Express.js backend
│   └── worker/     # FastAPI ML worker
├── packages/
│   ├── shared/     # TypeScript types & utilities
│   ├── database/   # MongoDB schemas
│   ├── cache/      # Redis client
│   └── storage/    # Storage utilities
├── docs/           # Architecture & API docs
├── docker-compose.yml
└── turbo.json

---

API Overview

Auth

Method	Endpoint	Description
POST	/api/auth/register	Register user
POST	/api/auth/login	Login
GET	/api/auth/me	Get current user

Projects & Panels

Method	Endpoint	Description
GET	/api/projects	List projects
POST	/api/projects	Create project
POST	/api/projects/:id/panels	Add manga panel

Render

Method	Endpoint	Description
POST	/api/render/start	Start render job
GET	/api/render/:jobId	Poll render status
GET	/api/render/:jobId/download	Download output video

WebSocket Events

Event	Direction	Description
render:progress	Server → Client	Live progress updates
render:complete	Server → Client	Job finished
render:error	Server → Client	Job failed

---

Getting Started

Prerequisites

• Node.js 18+
• Python 3.10+
• Docker & Docker Compose
• MongoDB + Redis

Quick Start

# Clone the repo
git clone https://github.com/Mananwebdev160408/UtsukushiiAI.git
cd UtsukushiiAI

# Install dependencies
npm install

# Set up environment variables
cp apps/web/.env.example apps/web/.env.local
cp apps/api/.env.example apps/api/.env
cp apps/worker/.env.example apps/worker/.env

# Start all services
docker-compose up -d

# Run dev servers
npm run dev

---

Contributing

1. Fork the repo
2. Create a feature branch: git checkout -b feature/your-feature
3. Commit: git commit -m 'feat: add your feature'
4. Push & open a Pull Request

Use TypeScript for all new code. Write unit tests for new features.

---

Acknowledgements

• Manga109 Dataset — training data for panel detection
• Ultralytics — YOLOv12
• Meta AI — SAM 2
• Stability AI — Stable Video Diffusion

---

_{Built by Manan Gupta · @Mananwebdev160408}