Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 Store stopped after uploading first chunk #686

Closed
2 tasks done
iamsahebgiri opened this issue Dec 8, 2024 · 9 comments · Fixed by #689
Closed
2 tasks done

S3 Store stopped after uploading first chunk #686

iamsahebgiri opened this issue Dec 8, 2024 · 9 comments · Fixed by #689
Labels
bug

Comments

@iamsahebgiri
Copy link

Initial checklist

  • I understand this is a bug report and questions should be posted in the Community Forum
  • I searched issues and couldn’t find anything (or linked relevant results below)

Steps to reproduce

Size of the video file on disk is 158MB, but minio shows ~70MB (that's my part size).
image

// config/tus.js
const path = require("path");
const { Server, EVENTS } = require("@tus/server");
const { S3Store } = require("@tus/s3-store");

const env = require("./env");
const { s3ClientConfig } = require("./s3");

const s3Store = new S3Store({
  partSize: 80 * 1024 * 1024, // each uploaded part will have ~80MiB,
  s3ClientConfig: {
    ...s3ClientConfig,
    bucket: env.UPLOAD_BUCKET,
  },
});

const fileStore = new FileStore({
  directory: "./uploads",
});

const tusServer = new Server({
  path: "/upload",
  datastore: s3Store,
});

tusServer.on(EVENTS.EVENT_UPLOAD_COMPLETE, (event) => {
  console.log(
    `[${new Date().toLocaleTimeString()}] [EVENT HOOK] Upload complete for file ${
      event.file.id
    }`
  );
});

module.exports = tusServer;
// app.js
const express = require("express");
const cors  = require("cors");
const env = require("./config/env");
const { initializeBuckets } = require("./config/s3");
const tusServer = require("./config/tus");

const app = express();
const uploadApp = express();

const PORT = env.PORT;

// app.use(express.static("public"));

app.use(cors());
uploadApp.all("*", tusServer.handle.bind(tusServer));
app.use("/upload", uploadApp);

async function startServer() {
  await initializeBuckets();

  app.listen(PORT, () => {
    console.log(`Server running on  http://localhost:${PORT}`);
  });
}

startServer();

Frontend code

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Veed - Video transcoding pipeline</title>
    <link
      href="https://releases.transloadit.com/uppy/v4.7.0/uppy.min.css"
      rel="stylesheet"
    />
    <style>
      .uppy-size--md .uppy-Dashboard-AddFiles-info {
        display: none;
      }
    </style>
  </head>
  <body>
    <h1>Veed - Video transcoding pipeline</h1>
    <noscript>You need JavaScript enabled for this example to work.</noscript>
    <button id="uppyModalOpener">Open Modal</button>
    <script type="module">
      import {
        Uppy,
        Dashboard,
        Webcam,
        Tus,
      } from "https://releases.transloadit.com/uppy/v4.8.0/uppy.min.mjs";

      const uppy = new Uppy({
        debug: true,
        autoProceed: false,
        restrictions: {
          maxFileSize: 1024 * 1024 * 1024, // 1GB
          allowedFileTypes: ["video/*"],
        },
      })
        .use(Dashboard, { trigger: "#uppyModalOpener" })
        .use(Webcam, { target: Dashboard })
        // .use(Tus, { endpoint: "https://tusd.tusdemo.net/files/" });
        .use(Tus, { endpoint: "http://localhost:8001/upload" });

      uppy.on("success", (fileCount) => {
        console.log(`${fileCount} files uploaded`);
      });
    </script>
  </body>
</html>

Expected behavior

It should ideally upload the entire file.

Actual behavior

Uploading just the first chunk of the video when s3 store is used. This works fine if s3 store is replaced with file store.
@Murderlon
Copy link
Member

Murderlon commented Dec 17, 2024 •
edited
Loading

Hi, can you remove your partSize and test it the exact same way again? Wondering if it does work for you then.

@iamsahebgiri
Copy link
Author

Tried that as well but that didn't help though, can you point me to an example with minio?

@Murderlon
Copy link
Member

There is no such example. However Supabase is using @tus/server and @tus/s3-store in production at a massive global scale and reported no such issues. So wondering if this is related to minio as your integration code looks fine 🤔

@iamsahebgiri
Copy link
Author

iamsahebgiri commented Dec 19, 2024 •
edited
Loading

This possibly could be minio, but I have tried using tusd binary, which seems to be working fine even with minio.

@fenos
Copy link
Collaborator

fenos commented Dec 19, 2024

@iamsahebgiri can you create a repo that reproduces the issuee, please
That would help a lot

@iamsahebgiri
Copy link
Author

Hi @fenos, here is the repo which reproduce the issue, https://github.com/iamsahebgiri/tus-minio-repro

Hope that helps.

@fenos
Copy link
Collaborator

fenos commented Dec 19, 2024

Hi @iamsahebgiri i can confirm that there is a pretty nasty bug 😄
Will raise a PR shortly

@fenos
Copy link
Collaborator

fenos commented Dec 19, 2024

It was a regression on this commit:
227ae81#diff-fecfad9cf273825148c4be6b30379a6331608acd8b71f5acd9fc60c2286be034L365-R370

Currently the partNumber is never increased, resulting in uploading each chunk always on the part number 2

This was referenced Dec 19, 2024
Merged
Closed
@juliusrennerbosch
Copy link

I was having the same issue and can confirm, that that the latest version (1.6.2) fixes the problem. Thank you! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

@tus/s3-store: fix part number increment tus/tus-node-server
4 participants
@fenos @Murderlon @iamsahebgiri @juliusrennerbosch