CLODEX Website Cloner

CLODEX Website Cloner is a Next.js service for creating website capture jobs from the CLODEX dashboard.

Users choose a CLODEX API key, target URL, provider mode, and model. The server queue processes jobs, stores a static archive, and keeps archive downloads private to the job owner or an admin.

Features

• Shared public job queue
• CLODEX session check through new-api
• Per-user archive download authorization
• User key selection for billing attribution
• Static archive worker with persistent job storage
• Docker-ready Next.js standalone build

Local Development

npm install
npm run dev

By default the service expects new-api at http://new-api:3000. For local development you can override it:

NEW_API_INTERNAL_BASE_URL=http://127.0.0.1:3000 npm run dev

Production Build

npm run build
npm run start

Docker

docker compose up app --build

The worker stores queue data and generated archives in .cloner by default inside the container. In production, mount this directory as a persistent volume.

Environment

Variable	Purpose	Default
NEW_API_INTERNAL_BASE_URL	Internal CLODEX/new-api URL for session validation	http://new-api:3000
NEW_API_BASE_URL	Fallback new-api URL	unset
WEBSITE_CLONER_DATA_DIR	Queue and archive storage directory	/app/.cloner

Security

The queue list is visible to everyone, but archive links are only serialized for the job owner or an admin. The download endpoint repeats the same server-side authorization check, so a direct archive URL cannot be used to download another user's result.

License

MIT. See LICENSE and NOTICE.