Skip to content

Commit

Permalink
Merge pull request #16 from s3rius/feature/docs
Browse files Browse the repository at this point in the history 
Updated documentation.
  • Loading branch information
@s3rius
s3rius authored Jan 8, 2022
2 parents b1d6f45 + afe73c8 commit 9a92e0c
Show file tree
Hide file tree
Showing 5 changed files with 280 additions and 57 deletions.
312 changes: 266 additions & 46 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,53 +9,75 @@ This implementation has several features to make usage as simple as possible.
* Rustus is robust, since it uses asynchronous Rust;
* It can store information about files in databases;
* You can specify directory structure to organize your uploads;
* It has a lot of hooks options, and hooks can be combined.
* Highly configurable;

### Supported info storages
## Installation

* FileSystem
* PostgresSQL
* Mysql
* SQLite
* Redis
You can download binaries from a [releases page](https://github.com/s3rius/rustus/releases).

### Supported data storages
If you want to use docker, you can use official images from [s3rius/rustus](https://hub.docker.com/r/s3rius/rustus/):
```bash
docker run --rm -it -p 1081:1081 s3rius/rustus:latest
```

* FileSystem
If we don't have a binary file for your operating system you can build it with [cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html).

## Installation
```bash
git clone https://github.com/s3rius/rustus.git
cd rustus
cargo install --path . --features=all
```

### Supported data storages

Since I haven't configured build automation yet, you can build it
from source using `cargo`.
Right now you can only use `file-storage` to store uploads data.
The only two options you can adjust are:
* uploads directory
* directory structure

To upload files in a custom directory other than `./data`
you can provide a `--data-dir` parameter.

```bash
cargo install --path .
rustus --data-dir "./files"
```

Or you can use a docker image.
If you have a lot of uploads, you don't want to store all your files in
a flat structure. So you can set a directory structure for your uploads.

```bash
docker run --rm -it -p 1081:1081 s3rius/rustus:latest
rustus --dir-structure="{env[HOSTNAME]}/{year}/{month}/{day}"
```

Docker image and binaries will be available soon.
```bash
tree data
data
├── 0bd911d4054d41c6a3ad54be67ee3e66.info
├── 5bc9c62384494c439e2a064b82a39cc6.info
└── rtus-68cb5b8746-5mgw9
└── 2022
└── 1
└── 8
├── 0bd911d4054d41c6a3ad54be67ee3e66
└── 5bc9c62384494c439e2a064b82a39cc6

## Architecture
```

Files and info about them are separated from each other.
In order to modify original file rustus searches for information about
the file in information storage.
**Important note:** if you use variable that doesn't exist or incorrect like invalid env variable, it
results in an error and the directory structure will become flat again.

However, automatic migration between different information
storages is not supported yet.
As you can see all info files are stored in a flat structure. It cannot be changed if
you use file info storage. In order to get rid of those `.info` files use different
info storages.

## Info storages

The info storage is a database or directory. The main goal is to keep track
of uploads. Rustus stores information about download in json format inside
database.

File storage is a default one. You can customize the directory of an .info files
File storage is used by default. You can customize the directory of an .info files
by providing `--info-dir` parameter.

```bash
Expand Down Expand Up @@ -90,27 +112,225 @@ you have to use webhooks or File hooks.

Hooks have priorities: file hooks are the most important, then goes webhooks and AMQP hooks have the least priority.
If pre-create hook failed, the upload would not start.
Of course, since AMQP is a protocol that doesn't allow you to track responses.
We can't validate anything to stop uploading.


### Roadmap

* [x] Data storage interface;
* [x] Info storage interface;
* [x] Core TUS protocol;
* [x] Extensions interface;
* [x] Creation extension;
* [x] Creation-defer-length extension;
* [x] Creation-with-upload extension;
* [x] Termination extension;
* [x] Route to get uploaded files;
* [x] Database support for info storage;
* [x] Redis support for info storage;
* [x] Notification interface;
* [x] Notifications via http hooks;
* [x] Notifications via RabbitMQ;
* [X] Executable files notifications;
* [ ] S3 as data storage store support;
* [ ] Rustus helm chart;
* [ ] Cloud native rustus operator.
Of course, since AMQP is a protocol that doesn't allow you to track responses
we can't validate anything to stop uploading.

Hooks can have 2 formats

default:
```json
{
"upload": {
"id": "",
"offset": 0,
"length": 39729945,
"path": null,
"created_at": 1641620821,
"deferred_size": false,
"metadata": {
"filename": "38MB_video.mp4",
"meme": "hehe2"
}
},
"request": {
"URI": "/files",
"method": "POST",
"remote_addr": "127.0.0.1",
"headers": {
"accept-encoding": "gzip, deflate",
"connection": "keep-alive",
"host": "localhost:1081",
"upload-metadata": "meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ=",
"tus-resumable": "1.0.0",
"content-length": "0",
"upload-length": "39729945",
"user-agent": "python-requests/2.26.0",
"accept": "*/*"
}
}
}
```

tusd:
```json
{
"Upload": {
"ID": "",
"Offset": 0,
"Size": 39729945,
"IsFinal": true,
"IsPartial": false,
"PartialUploads": null,
"SizeIsDeferred": false,
"Metadata": {
"filename": "38MB_video.mp4",
"meme": "hehe2"
},
"Storage": {
"Type": "filestore",
"Path": null
}
},
"HTTPRequest": {
"URI": "/files",
"Method": "POST",
"RemoteAddr": "127.0.0.1",
"Header": {
"host": [
"localhost:1081"
],
"user-agent": [
"python-requests/2.26.0"
],
"accept": [
"*/*"
],
"content-length": [
"0"
],
"upload-metadata": [
"meme aGVoZTI=,filename MzhNQl92aWRlby5tcDQ="
],
"connection": [
"keep-alive"
],
"tus-resumable": [
"1.0.0"
],
"upload-length": [
"39729945"
],
"accept-encoding": [
"gzip, deflate"
]
}
}
}
```

### File hooks

Rustus can work with two types of file hooks.

1. Single file hook;
2. Hooks directory.

The main difference is that hook name is passed as a command line parameter to a
single file hook, but if you use hooks directory then hook name is used to determine a
file to call. Let's take a look at the examples

Example of a single file hook:

```bash
#!/bin/bash

# Hook name would be "pre-create", "post-create" and so on.
HOOK_NAME="$1"
MEME="$(cat /dev/stdin | jq ".upload .metadata .meme" | xargs)"

# Here we check if name in metadata is equal to pepe.
if [[ $MEME = "pepe" ]]; then
echo "This meme isn't allowed" 1>&2;
exit 1
fi
```

As you can see it uses first CLI parameter as a hook name and all hook data is received from stdin.

Let's make it executable
```bash
chmod +x "hooks/unified_hook"
```

To use it you can add parameter
```bash
rustus --hooks-file "hooks/unified_hook"
```

This hook is going to ignore any file that has "pepe" in metadata.

Let's create a hook directory.

```bash
❯ tree hooks
hooks
├── post-create
├── post-finish
├── post-receive
├── post-terminate
└── pre-create
```

Every file in this directory has an executable flag.
So you can specify a parameter to use hooks directory.

```bash
rustus --hooks-dir "hooks"
```

In this case rustus will append a hook name to the directory you pointed at and call it as
an executable.

Information about hook can be found in stdin.

### Http Hooks

Http hooks use http protocol to notify you about an upload.
You can use HTTP hooks to verify Authorization.


Let's create a FastAPI application that listens to hooks and checks the
authorization header.

```bash
# Installing dependencies
pip install fastapi uvicorn
```

```python
# server.py
from fastapi import FastAPI, Header, HTTPException
from typing import Optional

app = FastAPI()

@app.post("/hooks")
def hook(
authorization: Optional[str] = Header(None),
hook_name: Optional[str] = Header(None),
):
print(f"Received: {hook_name}")
if authorization != "Bearer jwt":
raise HTTPException(401)
return None
```

Now we can start a server.
```bash
uvicorn server:app --port 8080
```

Now you can start rustus, and it will check if Authorization header has a correct value.
```bash
rustus --hooks-http-urls "http://localhost:8000/hooks" --hooks-http-proxy-headers "Authorization"
```


### AMQP hooks

All hooks can be sent with an AMQP protocol.

For example if you have a rabbitMQ you can use it.

```bash
rustus --hooks-amqp-url "amqp://guest:guest@localhost" --hooks-amqp-exchange "my_exchange"
```

This command will create an exchange called "rustus" and queues for every hook.

Every hook is published with routing key "rustus.{hook_name}" like
"rustus.post-create" or "rustus.pre-create" and so on.

The problem with AMQP hooks is that you can't block the upload.
To do this you have to use HTTP or File hooks. But with AMQP your
uploads become non-blocking which is definitely a good thing.
17 changes: 10 additions & 7 deletions src/config.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ use std::path::PathBuf;

use chrono::{Datelike, Timelike};
use lazy_static::lazy_static;
use log::error;
use structopt::StructOpt;

use crate::info_storages::AvailableInfoStores;
Expand Down Expand Up @@ -36,11 +37,11 @@ pub struct StorageOptions {
///
/// This directory is used to store files
/// for all *file_storage storages.
#[structopt(long, default_value = "./data")]
#[structopt(long, env = "RUSTUS_DATA_DIR", default_value = "./data")]
pub data_dir: PathBuf,

#[structopt(long, default_value = "")]
pub dis_structure: String,
#[structopt(long, env = "RUSTUS_DIR_STRUCTURE", default_value = "")]
pub dir_structure: String,
}

#[derive(StructOpt, Debug, Clone)]
Expand Down Expand Up @@ -91,8 +92,8 @@ pub struct NotificationsOptions {
///
/// This format will be used in all
/// messages about hooks.
#[structopt(long, default_value = "default", env = "RUSTUS_NOTIFICATION_FORMAT")]
pub notification_format: Format,
#[structopt(long, default_value = "default", env = "RUSTUS_HOOKS_FORMAT")]
pub hooks_format: Format,

/// Enabled hooks for notifications.
#[structopt(
Expand Down Expand Up @@ -236,8 +237,10 @@ impl RustusConf {
vars.insert("year".into(), now.year().to_string());
vars.insert("hour".into(), now.hour().to_string());
vars.insert("minute".into(), now.minute().to_string());
strfmt::strfmt(self.storage_opts.dis_structure.as_str(), &vars)
.unwrap_or_else(|_| "".into())
strfmt::strfmt(self.storage_opts.dir_structure.as_str(), &vars).unwrap_or_else(|err| {
error!("{}", err);
"".into()
})
}

/// List of extensions.
Expand Down
Loading

0 comments on commit 9a92e0c

Please sign in to comment.