Skip to content

Custom bot upload holds entire ZIP in memory #191

Description

@alexanderwanyoike

Bug Description

FileInterceptor (Multer) loads the entire uploaded ZIP into file.buffer in memory before passing it to MinIO. For large bots with vendored dependencies (400MB+), this can OOM the API pod.

Steps to Reproduce

  1. Upload a large custom bot ZIP (400MB+) via the API
  2. Observe memory usage of the API pod spike to hold the entire file in memory before MinIO receives it

Expected Behavior

The upload should be streamed directly to MinIO without buffering the full file in memory.

Actual Behavior

The entire ZIP is buffered in memory before being passed to MinIO, risking OOM errors for large uploads.

Component

API

Additional Context

Current flow

Client uploads ZIP
  -> Multer buffers entire file in memory (file.buffer)
  -> storageService.uploadBotFile(file.buffer, ...)
  -> minioClient.putObject(bucket, path, buffer, buffer.length)

Suggested fix

Switch from Multer's memory storage to streaming:

  1. Use Multer disk storage or a custom storage engine that pipes to MinIO
  2. Or use busboy / fastify-multipart to get a readable stream and pipe directly to minioClient.putObject(bucket, path, stream, size)
  3. The MinIO client already supports Readable streams, so uploadBotFile just needs to accept a stream instead of a Buffer

Relevant files

  • api/src/custom-bot/custom-bot.controller.ts (lines 41-44) - FileInterceptor + file.buffer
  • api/src/custom-bot/storage.service.ts (lines 105-132) - uploadBotFile takes Buffer

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions