You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: docs/_storage_backends/aws-s3.md
+79-25
Original file line number
Diff line number
Diff line change
@@ -4,42 +4,96 @@ layout: default
4
4
nav_order: 3
5
5
---
6
6
7
-
Alternatively, if you want to store the uploads on an AWS S3 bucket, you only have to specify
8
-
the bucket and provide the corresponding access credentials and region information using
9
-
environment variables (if you want to use a S3-compatible store, use can use the `-s3-endpoint`
10
-
option):
7
+
# S3 storage
11
8
12
-
```
9
+
Tusd can store files directly on AWS S3 or other compatible services, e.g. [Minio](https://min.io/). The uploaded file is directly transferred to S3 while the user is performing the upload without storing the entire file on disk first.
10
+
11
+
## Configuration
12
+
13
+
To enable this backend, you must supply the corresponding access credentials and region information using environment variables and specify the bucket name using `-s3-bucket`, for example:
14
+
15
+
```bash
13
16
$ export AWS_ACCESS_KEY_ID=xxxxx
14
17
$ export AWS_SECRET_ACCESS_KEY=xxxxx
15
18
$ export AWS_REGION=eu-west-1
16
19
$ tusd -s3-bucket=my-test-bucket.com
17
20
[tusd] 2019/09/29 21:11:23 Using 's3://my-test-bucket.com' as S3 bucket for storage.
18
-
[tusd] 2019/09/29 21:11:23 Using 0.00MB as maximum size.
19
-
[tusd] 2019/09/29 21:11:23 Using 0.0.0.0:8080 as address to listen.
20
-
[tusd] 2019/09/29 21:11:23 Using /files/ as the base path.
21
-
[tusd] 2019/09/29 21:11:23 Using /metrics as the metrics path.
22
-
[tusd] 2019/09/29 21:11:23 Supported tus extensions: creation,creation-with-upload,termination,concatenation,creation-defer-length
23
-
[tusd] 2019/09/29 21:11:23 You can now upload files to: http://0.0.0.0:8080/files/
21
+
...
24
22
```
25
23
26
-
If your S3 bucket has been configured for AWS S3 Transfer Acceleration and you want to make use of that advanced service,
27
-
you can direct tusd to automatically use the designated AWS acceleration endpoint for your bucket by including the optional
28
-
command line flag `s3-transfer-acceleration` as follows:
24
+
Credentials can also be loaded from a shared credentials file (`~/.aws/credentials`) as described in the [AWS SDK for Go](https://github.com/aws/aws-sdk-go#configuring-credentials). You still need to declare the `AWS_REGION` value which isn't conventionally associated with credentials.
25
+
26
+
### Alternative endpoints
29
27
28
+
If you are using an S3-compatible service other than from AWS, you must configure the correct endpoint using `-s3-endpoint`. Please note that this value must start with `http://` or `https://`, for example:
2024/02/20 15:33:45.474497 Using 'https://mystoreage.example.com/my-test-bucket.com' as S3 endpoint and bucket for storage.
33
+
...
30
34
```
31
-
$ export AWS_ACCESS_KEY_ID=xxxxx
32
-
$ export AWS_SECRET_ACCESS_KEY=xxxxx
33
-
$ export AWS_REGION=eu-west-1
35
+
36
+
### Object prefix
37
+
38
+
If the bucket is also used to store files from other sources than tusd, it makes sense to use a custom prefix for all object relating to tusd. This can be achieved using the `-s3-object-prefix` flag. For example, the following configuration will cause the objects to be put under the `uploads/` prefix in the bucket:
If your S3 bucket has been configured for [AWS S3 Transfer Acceleration](https://aws.amazon.com/s3/transfer-acceleration/) and you want to make use of that advanced service, you can direct tusd to automatically use the designated AWS acceleration endpoint for your bucket by including the optional
47
+
command line flag `-s3-transfer-acceleration` as follows:
[tusd] 2019/09/29 21:11:23 Using 's3://my-test-bucket.com' as S3 bucket for storage with AWS S3 Transfer Acceleration enabled.
36
-
[tusd] 2019/09/29 21:11:23 Using 0.00MB as maximum size.
37
-
[tusd] 2019/09/29 21:11:23 Using 0.0.0.0:8080 as address to listen.
38
-
[tusd] 2019/09/29 21:11:23 Using /files/ as the base path.
39
-
[tusd] 2019/09/29 21:11:23 Using /metrics as the metrics path.
40
-
[tusd] 2019/09/29 21:11:23 Supported tus extensions: creation,creation-with-upload,termination,concatenation,creation-defer-length
41
-
[tusd] 2019/09/29 21:11:23 You can now upload files to: http://0.0.0.0:8080/files/
52
+
...
53
+
```
54
+
55
+
## Permissions
56
+
57
+
For full functionality of the S3 backend, the user accessing the bucket must have at least following AWS IAM policy permissions for the bucket and all of its subresources:
58
+
42
59
```
60
+
s3:AbortMultipartUpload
61
+
s3:DeleteObject
62
+
s3:GetObject
63
+
s3:ListMultipartUploadParts
64
+
s3:PutObject
65
+
```
66
+
67
+
## Storage format
68
+
69
+
Uploads on S3 are stored using multiple objects:
70
+
71
+
- An informational object with the `.info` extension holds meta information about the uploads, as described in [the section for all storage backends](/storage_backends/overview/#storage-format).
72
+
- An [S3 multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) is used to transfer the file piece-by-piece to S3 and reassemble the original file once the upload is finished. It is removed once the upload is finished.
73
+
- A file object will contain the uploaded file. It will only be created once the entire upload is finished.
74
+
- A temporary object with the `.part` extension may be created when the upload has been paused to store some temporary data which cannot be transferred to the S3 multipart upload due to its small size. Once the upload is resumed, the temporary object will be gone.
75
+
76
+
By default, the objects are stored at the root of the bucket. For example the objects for the upload ID `abcdef123` will be:
77
+
78
+
-`abcdef123.info`: Informational object
79
+
-`abcdef123`: File object
80
+
-`abcdef123.part`: Temporary object
81
+
82
+
{: .note }
83
+
84
+
The file object is not visible in the S3 bucket before the upload is finished because the transferred file data is stored in the associated S3 multipart upload. Once the upload is complete, the chunks from the S3 multipart are reassembled into the file, creating the file object and removing the S3 multipart upload. In addition, the S3 multipart upload is not directly visible in the S3 bucket because it does not represent a complete object. Please don't be confused if you don't see the changes in the bucket while the file is being uploaded.
85
+
86
+
### Metadata
87
+
88
+
If [metadata](https://tus.io/protocols/resumable-upload#upload-metadata) is associated with the upload during creation, it will be added to the file object once the upload is finished. Because the metadata on S3 objects must only contain ASCII characters, tusd will replace every non-ASCII character
89
+
with a question mark (for example, "Menü" will be "Men?").
90
+
91
+
In addition, the metadata is also stored in the informational object, which can be used to retrieve the original metadata without any characters being replaced.
92
+
93
+
# Considerations
94
+
95
+
When receiving a `PATCH` request, parts of its body will be temporarily stored on disk before they can be transferred to S3. This is necessary to meet the minimum part size for an S3 multipart upload enforced by S3 and to allow the AWS SDK to calculate a checksum. Once the part has been uploaded to S3, the temporary file will be removed immediately. Therefore, please ensure that the server running this storage backend has enough disk space available to hold these temporary files.
96
+
97
+
# Local setup with Minio
43
98
44
-
tusd is also able to read the credentials automatically from a shared credentials file (~/.aws/credentials) as described in https://github.com/aws/aws-sdk-go#configuring-credentials.
45
-
But be mindful of the need to declare the AWS_REGION value which isn't conventionally associated with credentials.
Tusd can store files directly on Azure Blob Storage or other compatible services, e.g. [Azurite](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=visual-studio%2Cblob-storage). The uploaded file is directly transferred to Azure while the user is performing the upload without storing the entire file on disk first.
10
+
11
+
## Configuration
12
+
13
+
To enable this backend, you must supply the corresponding access credentials using environment variables and specify the container name using `-azure-storage`, for example:
14
+
15
+
```bash
16
+
$ export AZURE_STORAGE_ACCOUNT=xxxxx
17
+
$ export AZURE_STORAGE_KEY=xxxxx
18
+
$ tusd -azure-storage=my-test-container
19
+
[tusd] 2024/02/23 11:34:03.411021 Using Azure endpoint https://xxxxx.blob.core.windows.net.
20
+
...
21
+
```
22
+
23
+
### Alternative endpoints
24
+
25
+
If you want to upload to Azure Blob Storage using a custom endpoint, e.g when using [Azurite](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string#configure-a-connection-string-for-azurite) for local development,
26
+
you can specify the endpoint using the `-azure-endpoint` flag. For example:
[tusd] 2023/02/13 16:15:18.641937 Using Azure endpoint https://my-custom-endpoint.com.
31
+
...
32
+
```
33
+
34
+
### Object prefix
35
+
36
+
If the container is also used to store files from other sources than tusd, it makes sense to use a custom prefix for all object relating to tusd. This can be achieved using the `-azure-object-prefix` flag. For example, the following configuration will cause the objects to be put under the `uploads/` prefix in the bucket:
You can also upload blobs to Azure Blob Storage with a different storage tier than what is set as the default for the storage account. This can be done by using the `-azure-blob-access-tier` flag. For example:
- An informational object with the `.info` extension holds meta information about the uploads, as described in [the section for all storage backends](/storage_backends/overview/#storage-format).
55
+
- A file object will contain the uploaded file. Data is appended to the object while the upload is performed.
56
+
57
+
By default, the objects are stored at the root of the container. For example the objects for the upload ID `abcdef123` will be:
Copy file name to clipboardexpand all lines: docs/_storage_backends/google-cloud-storage.md
+33-6
Original file line number
Diff line number
Diff line change
@@ -4,14 +4,41 @@ layout: default
4
4
nav_order: 5
5
5
---
6
6
7
-
Furthermore, tusd also has support for storing uploads on Google Cloud Storage. In order to enable this feature, supply the path to your account file containing the necessary credentials:
7
+
# Google Cloud Storage
8
8
9
-
```
9
+
Tusd can store files directly on Google Cloud Storage. The uploaded file is directly transferred to S3 while the user is performing the upload without storing the entire file on disk first.
10
+
11
+
## Configuration
12
+
13
+
To enable this backend, you must supply the path to the corresponding account file using environment variables and specify the bucket name using `-gcs-bucket`, for example:
14
+
15
+
```bash
10
16
$ export GCS_SERVICE_ACCOUNT_FILE=./account.json
11
17
$ tusd -gcs-bucket=my-test-bucket.com
12
18
[tusd] Using 'gcs://my-test-bucket.com' as GCS bucket for storage.
13
-
[tusd] Using 0.00MB as maximum size.
14
-
[tusd] Using 0.0.0.0:8080 as address to listen.
15
-
[tusd] Using /files/ as the base path.
16
-
[tusd] Using /metrics as the metrics path.
19
+
...
20
+
```
21
+
22
+
### Object prefix
23
+
24
+
If the container is also used to store files from other sources than tusd, it makes sense to use a custom prefix for all object relating to tusd. This can be achieved using the `-gcs-object-prefix` flag. For example, the following configuration will cause the objects to be put under the `uploads/` prefix in the bucket:
The used service account must have the `https://www.googleapis.com/auth/devstorage.read_write` scope enabled, so tusd can read and write data to the storage buckets associated with the service account file.
33
+
34
+
## Storage format
35
+
36
+
Uploads are stored using multiple objects:
37
+
38
+
- An informational object with the `.info` extension holds meta information about the uploads, as described in [the section for all storage backends](/storage_backends/overview/#storage-format).
39
+
- A file object will contain the uploaded file. Data is appended to the object while the upload is performed.
40
+
41
+
By default, the objects are stored at the root of the bucket. For example the objects for the upload ID `abcdef123` will be:
Copy file name to clipboardexpand all lines: docs/_storage_backends/local-disk.md
+23
Original file line number
Diff line number
Diff line change
@@ -3,3 +3,26 @@ title: Local Disk
3
3
layout: default
4
4
nav_order: 2
5
5
---
6
+
7
+
# Local disk storage
8
+
9
+
Tusd can store uploads on the local disk in a specific directory. This storage backend is the default if no other configuration flags are supplied. The `-upload-dir` flag specifies the directory that will be used. If this directory does not exist, tusd will create it. For example:
10
+
11
+
```sh
12
+
$ tusd -upload-dir=./uploads
13
+
```
14
+
15
+
When a new upload is created, for example with the upload ID `abcdef123`, tusd creates two files:
16
+
17
+
-`./uploads/abcdef123` to hold the file content that is uploaded
18
+
-`./uploads/abcdef123.info` to hold [meta information about the upload](/storage_backends/overview/#storage-format)
19
+
20
+
## Issues with NFS and shared folders
21
+
22
+
Tusd maintains [upload locks](/advanced_topics/locks/) on disk to ensure exclusive access to uploads and prevent data corruption. These disk-based locks utilize hard links, which might not supported by older NFS versions or when a folder is shared in a VM using VirtualBox or Vagrant. In these cases, you might get errors like this:
23
+
24
+
```
25
+
TemporaryErrors (Lockfile created, but doesn't exist)
26
+
```
27
+
28
+
We recommend you to ensure that your file system supports hard links, use a different file system, or use one of tusd's cloud storage abilities. If the problem still persists, please open a bug report.
0 commit comments