Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated documentation. #16

Merged
merged 1 commit into from
Jan 8, 2022
Merged

Updated documentation. #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
312 changes: 266 additions & 46 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,53 +9,75 @@ This implementation has several features to make usage as simple as possible.
* Rustus is robust, since it uses asynchronous Rust;
* It can store information about files in databases;
* You can specify directory structure to organize your uploads;
* It has a lot of hooks options, and hooks can be combined.
* Highly configurable;

### Supported info storages
## Installation

* FileSystem
* PostgresSQL
* Mysql
* SQLite
* Redis
You can download binaries from a [releases page](https://github.com/s3rius/rustus/releases).

### Supported data storages
If you want to use docker, you can use official images from [s3rius/rustus](https://hub.docker.com/r/s3rius/rustus/):
```bash
docker run --rm -it -p 1081:1081 s3rius/rustus:latest
```

* FileSystem
If we don't have a binary file for your operating system you can build it with [cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html).

## Installation
```bash
git clone https://github.com/s3rius/rustus.git
cd rustus
cargo install --path . --features=all
```

### Supported data storages

Since I haven't configured build automation yet, you can build it
from source using `cargo`.
Right now you can only use `file-storage` to store uploads data.
The only two options you can adjust are:
* uploads directory
* directory structure

To upload files in a custom directory other than `./data`
you can provide a `--data-dir` parameter.

```bash
cargo install --path .
rustus --data-dir "./files"
```

Or you can use a docker image.
If you have a lot of uploads, you don't want to store all your files in
a flat structure. So you can set a directory structure for your uploads.

```bash
docker run --rm -it -p 1081:1081 s3rius/rustus:latest
rustus --dir-structure="{env[HOSTNAME]}/{year}/{month}/{day}"
```

Docker image and binaries will be available soon.
```bash
tree data
data
├── 0bd911d4054d41c6a3ad54be67ee3e66.info
├── 5bc9c62384494c439e2a064b82a39cc6.info
└── rtus-68cb5b8746-5mgw9
└── 2022
└── 1
└── 8
├── 0bd911d4054d41c6a3ad54be67ee3e66
└── 5bc9c62384494c439e2a064b82a39cc6

## Architecture
```

Files and info about them are separated from each other.
In order to modify original file rustus searches for information about
the file in information storage.
**Important note:** if you use variable that doesn't exist or incorrect like invalid env variable, it
results in an error and the directory structure will become flat again.

However, automatic migration between different information
storages is not supported yet.
As you can see all info files are stored in a flat structure. It cannot be changed if
you use file info storage. In order to get rid of those `.info` files use different
info storages.

## Info storages

The info storage is a database or directory. The main goal is to keep track
of uploads. Rustus stores information about download in json format inside
database.

File storage is a default one. You can customize the directory of an .info files
File storage is used by default. You can customize the directory of an .info files
by providing `--info-dir` parameter.

```bash
Expand Down Expand Up @@ -90,27 +112,225 @@ you have to use webhooks or File hooks.

Hooks have priorities: file hooks are the most important, then goes webhooks and AMQP hooks have the least priority.
If pre-create hook failed, the upload would not start.
Of course, since AMQP is a protocol that doesn't allow you to track responses.
We can't validate anything to stop uploading.


### Roadmap

* [x] Data storage interface;
* [x] Info storage interface;
* [x] Core TUS protocol;
* [x] Extensions interface;
* [x] Creation extension;
* [x] Creation-defer-length extension;
* [x] Creation-with-upload extension;
* [x] Termination extension;
* [x] Route to get uploaded files;
* [x] Database support for info storage;
* [x] Redis support for info storage;
* [x] Notification interface;
* [x] Notifications via http hooks;
* [x] Notifications via RabbitMQ;
* [X] Executable files notifications;
* [ ] S3 as data storage store support;
* [ ] Rustus helm chart;
* [ ] Cloud native rustus operator.
Of course, since AMQP is a protocol that doesn't allow you to track responses
we can't validate anything to stop uploading.

Hooks can have 2 formats

default:
```json
{
"upload": {
"id": "",
"offset": 0,
"length": 39729945,
"path": null,
"created_at": 1641620821,
"deferred_size": false,
"metadata": {
"filename": "38MB_video.mp4",
"meme": "hehe2"
}
},
"request": {
"URI": "/files",
"method": "POST",
"remote_addr": "127.0.0.1",
"headers": {
"accept-encoding": "gzip, deflate",
"connection": "keep-alive",
"host": "localhost:1081",
"upload-metadata": "meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ=",
"tus-resumable": "1.0.0",
"content-length": "0",
"upload-length": "39729945",
"user-agent": "python-requests/2.26.0",
"accept": "*/*"
}
}
}
```

tusd:
```json
{
"Upload": {
"ID": "",
"Offset": 0,
"Size": 39729945,
"IsFinal": true,
"IsPartial": false,
"PartialUploads": null,
"SizeIsDeferred": false,
"Metadata": {
"filename": "38MB_video.mp4",
"meme": "hehe2"
},
"Storage": {
"Type": "filestore",
"Path": null
}
},
"HTTPRequest": {
"URI": "/files",
"Method": "POST",
"RemoteAddr": "127.0.0.1",
"Header": {
"host": [
"localhost:1081"
],
"user-agent": [
"python-requests/2.26.0"
],
"accept": [
"*/*"
],
"content-length": [
"0"
],
"upload-metadata": [
"meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ="
],
"connection": [
"keep-alive"
],
"tus-resumable": [
"1.0.0"
],
"upload-length": [
"39729945"
],
"accept-encoding": [
"gzip, deflate"
]
}
}
}
```

### File hooks

Rustus can work with two types of file hooks.

1. Single file hook;
2. Hooks directory.

The main difference is that hook name is passed as a command line parameter to a
single file hook, but if you use hooks directory then hook name is used to determine a
file to call. Let's take a look at the examples

Example of a single file hook:

```bash
#!/bin/bash

# Hook name would be "pre-create", "post-create" and so on.
HOOK_NAME="$1"
MEME="$(cat /dev/stdin | jq ".upload .metadata .meme" | xargs)"

# Here we check if name in metadata is equal to pepe.
if [[ $MEME = "pepe" ]]; then
echo "This meme isn't allowed" 1>&2;
exit 1
fi
```

As you can see it uses first CLI parameter as a hook name and all hook data is received from stdin.

Let's make it executable
```bash
chmod +x "hooks/unified_hook"
```

To use it you can add parameter
```bash
rustus --hooks-file "hooks/unified_hook"
```

This hook is going to ignore any file that has "pepe" in metadata.

Let's create a hook directory.

```bash
❯ tree hooks
hooks
├── post-create
├── post-finish
├── post-receive
├── post-terminate
└── pre-create
```

Every file in this directory has an executable flag.
So you can specify a parameter to use hooks directory.

```bash
rustus --hooks-dir "hooks"
```

In this case rustus will append a hook name to the directory you pointed at and call it as
an executable.

Information about hook can be found in stdin.

### Http Hooks

Http hooks use http protocol to notify you about an upload.
You can use HTTP hooks to verify Authorization.


Let's create a FastAPI application that listens to hooks and checks the
authorization header.

```bash
# Installing dependencies
pip install fastapi uvicorn
```

```python
# server.py
from fastapi import FastAPI, Header, HTTPException
from typing import Optional

app = FastAPI()

@app.post("/hooks")
def hook(
authorization: Optional[str] = Header(None),
hook_name: Optional[str] = Header(None),
):
print(f"Received: {hook_name}")
if authorization != "Bearer jwt":
raise HTTPException(401)
return None
```

Now we can start a server.
```bash
uvicorn server:app --port 8080
```

Now you can start rustus, and it will check if Authorization header has a correct value.
```bash
rustus --hooks-http-urls "http://localhost:8000/hooks" --hooks-http-proxy-headers "Authorization"
```


### AMQP hooks

All hooks can be sent with an AMQP protocol.

For example if you have a rabbitMQ you can use it.

```bash
rustus --hooks-amqp-url "amqp://guest:guest@localhost" --hooks-amqp-exchange "my_exchange"
```

This command will create an exchange called "rustus" and queues for every hook.

Every hook is published with routing key "rustus.{hook_name}" like
"rustus.post-create" or "rustus.pre-create" and so on.

The problem with AMQP hooks is that you can't block the upload.
To do this you have to use HTTP or File hooks. But with AMQP your
uploads become non-blocking which is definitely a good thing.
17 changes: 10 additions & 7 deletions src/config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ use std::path::PathBuf;

use chrono::{Datelike, Timelike};
use lazy_static::lazy_static;
use log::error;
use structopt::StructOpt;

use crate::info_storages::AvailableInfoStores;
Expand Down Expand Up @@ -36,11 +37,11 @@ pub struct StorageOptions {
///
/// This directory is used to store files
/// for all *file_storage storages.
#[structopt(long, default_value = "./data")]
#[structopt(long, env = "RUSTUS_DATA_DIR", default_value = "./data")]
pub data_dir: PathBuf,

#[structopt(long, default_value = "")]
pub dis_structure: String,
#[structopt(long, env = "RUSTUS_DIR_STRUCTURE", default_value = "")]
pub dir_structure: String,
}

#[derive(StructOpt, Debug, Clone)]
Expand Down Expand Up @@ -91,8 +92,8 @@ pub struct NotificationsOptions {
///
/// This format will be used in all
/// messages about hooks.
#[structopt(long, default_value = "default", env = "RUSTUS_NOTIFICATION_FORMAT")]
pub notification_format: Format,
#[structopt(long, default_value = "default", env = "RUSTUS_HOOKS_FORMAT")]
pub hooks_format: Format,

/// Enabled hooks for notifications.
#[structopt(
Expand Down Expand Up @@ -236,8 +237,10 @@ impl RustusConf {
vars.insert("year".into(), now.year().to_string());
vars.insert("hour".into(), now.hour().to_string());
vars.insert("minute".into(), now.minute().to_string());
strfmt::strfmt(self.storage_opts.dis_structure.as_str(), &vars)
.unwrap_or_else(|_| "".into())
strfmt::strfmt(self.storage_opts.dir_structure.as_str(), &vars).unwrap_or_else(|err| {
error!("{}", err);
"".into()
})
}

/// List of extensions.
Expand Down
Loading