s3_store

S3-backed file storage for Frappe with native bench backup / bench restore integration.

Unlike other S3 attachment apps, files uploaded via s3_store are pulled into the standard bench backup --with-files tar (so it's self-contained) and pushed back to S3 after bench restore with a single command — or automatically. No separate scheduled DB pushes, no parallel backup infrastructure.

Compatible with AWS S3, MinIO, and any S3-compatible service.

---

Table of contents

1. Installation
2. Configuration
3. How uploads work
4. How serving works
5. Backup with files
6. Restore
7. Bulk migration of pre-existing local files
8. Settings reference
9. Architecture notes
10. Testing
11. Troubleshooting
12. Limitations

---

Installation

bench get-app https://github.com/Centura-AG/s3_store
bench --site <site> install-app s3_store

This pulls in a single Python dependency: boto3. No node modules, no native extensions.

---

Configuration

Open S3 Store Settings (singleton, System Manager only) in Desk.

Minimum required for AWS S3:

Field	Example
Enabled	✓
Region	eu-central-1
AWS Access Key ID	AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key	wJalrXUt…
Bucket	my-frappe-uploads

For MinIO / S3-compatible providers, also set:

Field	Example
Endpoint URL	https://minio.example.com

Saving the settings runs head_bucket against the configured S3 endpoint — if credentials, region, or bucket name are wrong, the save fails with a clear error.

---

How uploads work

When Enabled = 1, every new File doc routes through the write_file hook:

1. The file's bytes are uploaded to S3 under a generated key: {key_prefix}/{YYYY/MM/DD}/{attached_to_doctype}/{token8}_{sanitized_name}
2. The File doc's file_url is rewritten to the backend serve endpoint: /api/method/s3_store.s3_store.api.serve?key=<encoded>
3. No local copy is kept.

The token8 is a random 4-byte hex prefix — uploads of the same file.pdf from different forms get distinct keys, so they can't overwrite each other in S3 or on disk during backup staging.

Ignoring specific doctypes — list one DocType per line under Advanced → Ignored Doctypes to keep their attachments on the local filesystem (useful for User profile pictures, for example).

---

How serving works

/api/method/s3_store.s3_store.api.serve?key=<encoded> resolves the key to a File doc via an exact-match lookup on file_url, enforces check_permission("read") on that doc, then streams the object's bytes through the backend as a download response (display_content_as: inline).

The S3 object is fetched server-side with get_object and proxied to the client — the bucket stays fully private and credentials never leave the server. Browsers never talk to S3 directly.

Because every read is proxied through a web worker, large files occupy a worker for the duration of the download and are buffered in memory. For very large public assets you may prefer fronting the bucket with a CDN; that is out of scope for this app.

---

Backup with files

bench --site <site> backup --with-files

Behind the scenes, s3_store monkey-patches BackupGenerator.backup_files so that — only when this site has S3 Store Settings.enabled = 1 and include_in_native_backup = 1 — every S3-hosted file is downloaded into public/files/ or private/files/ before tar runs, and removed after. The patch is a no-op on sites without the app installed.

The resulting tar is self-contained: you can move it to a different host, a developer laptop, or a fresh bucket and restore without losing any uploads.

Disk-space pre-flight

Before staging, s3_store totals file_size across all S3-backed File records and requires ~110% free on the disk hosting the site. If insufficient, the backup aborts with:

Insufficient disk space to stage S3 files for backup:
need <N> bytes, have <M> bytes free under <site_path>.
Free up space or disable include_in_native_backup in S3 Store Settings.

Encrypted backups

bench backup --with-files --backup-encryption-key <key> works unchanged. Encryption runs after backup_files() in Frappe core, so the staged S3 contents end up inside the encrypted blob.

Concurrency

A file lock at sites/<site>/.s3_store_backup.lock prevents two backups from staging on top of each other. The second attempt fails fast with: "Another s3_store backup is already running for this site."

Stale-tar shortcut

BackupGenerator.get_recent_backup normally returns recent *-files.tar if one exists within the --older-than window. That would silently skip new S3 uploads — so s3_store neutralises the files-portion only when S3 is enabled, forcing a fresh stage. The DB shortcut is preserved.

---

Restore

bench --site <site> restore <db.sql.gz> \
    --with-public-files <files.tar> \
    --with-private-files <private-files.tar>

After this, the DB has File records whose file_url already points at S3 (?key=…), and the bytes sit on disk under their staging names ({token8}_{sanitized_name}).

To push them back to S3:

bench --site <site> s3-store-push-local

For each staged file, this uploads to the exact existing key referenced by the File doc — no URL rewriting, no fresh keys. Local copies are deleted after a successful push.

Auto-pilot

Set Auto-Push Local Files after Migrate in S3 Store Settings, and the push runs automatically as an after_migrate hook — i.e. at the end of every bench migrate, which conventionally follows bench restore. Cautious users leave this off and run the command explicitly.

---

Bulk migration of pre-existing local files

Use this once, after first installing on a site that already has local attachments:

Via the UI: open S3 Migration Log → New, save. The migration runs as a background job; progress is published to the realtime channel and reflected on the log doc (migrated, failed, errors).

Via API:

bench --site <site> execute s3_store.s3_store.api.start_migration

For every File row with a /files/... or /private/files/... URL, s3_store:

1. Mints a fresh S3 key
2. Uploads the bytes
3. Rewrites file_url to the backend serve endpoint
4. Deletes the local copy (subject to delete_local_after_push)

Errors are collected per-file — one bad file doesn't abort the run. The log doc shows status, counts, and the per-file error list.

---

Settings reference

Field	Default	Description
enabled	0	Master switch. When off, files go to the local filesystem as usual.
endpoint_url	—	S3 endpoint. Blank for AWS; set for MinIO etc.
region	us-east-1	AWS region.
aws_access_key_id	—	Required when enabled.
aws_secret_access_key	—	Required when enabled (stored encrypted via Frappe's password field).
bucket	—	Required when enabled.
key_prefix	—	Optional path prefix (e.g. site name). Objects are stored at {prefix}/{YYYY/MM/DD}/{doctype}/{token}_{name}.
signed_url_expiry	3600	Reserved for future use. Currently unused — all serving goes through the backend proxy endpoint.
delete_from_s3	1	Delete the S3 object when a File doc is trashed.
include_in_native_backup	1	Stage S3 files locally during bench backup --with-files.
auto_push_after_migrate	0	After bench migrate (typically right after bench restore), push local files back to S3.
delete_local_after_push	1	Remove the local copy after uploading to S3.
ignored_doctypes	—	Newline-separated DocType names whose attachments stay on the local filesystem.

---

Testing

bench --site <site> run-tests --app s3_store

The suite covers:

• Settings validation (creds + connection check)
• write_file and delete-on-trash hook
• serve content proxying, missing-key, permission-error paths
• Bulk migration: rewrite URL, delete local, error collection, log doc lifecycle
• Post-restore push: re-upload to existing key, error when staged file is missing
• Backup staging: cleanup on success, cleanup on exception, mixed-mode skip
• Patched BackupGenerator is a no-op when S3 disabled; get_recent_backup neutralises file tars
• URL parsing across virtual-host AWS, path-style MinIO, and the backend serve endpoint

All boto3 calls are mocked — the suite never hits the network.

End-to-end smoke test against MinIO

# Remove any previous attempt before re-creating
docker rm -f s3store-minio 2>/dev/null || true

# Start MinIO (change ports if 9200/9201 are taken)
docker run -d --name s3store-minio \
    -p 9200:9000 -p 9201:9001 \
    -e MINIO_ROOT_USER=admin \
    -e MINIO_ROOT_PASSWORD=adminadmin \
    minio/minio server /data --console-address ":9001"

# Create the bucket
docker run --rm --network container:s3store-minio --entrypoint sh minio/mc \
    -c "mc alias set local http://localhost:9000 admin adminadmin --api s3v4 \
        && mc mb --ignore-existing local/s3store-bucket"

Open S3 Store Settings in Desk:

Field	Value
Enabled	✓
Endpoint URL	http://localhost:9200
Region	us-east-1
AWS Access Key ID	admin
AWS Secret Access Key	adminadmin
Bucket	s3store-bucket

Console is at http://localhost:9201 (admin / adminadmin).

bench --site dev.s3.test install-app s3_store
# upload some files via the UI, then:
bench --site dev.s3.test backup --with-files

# Wipe and restore
bench drop-site dev.s3.test
bench --site dev.s3.test restore <db> \
    --with-public-files <files.tar> --with-private-files <priv-files.tar>
bench --site dev.s3.test s3-store-push-local

After the push, the local public/files/ and private/files/ directories should contain only Frappe's own non-S3 files; the bucket should hold every uploaded attachment under its original key.

---

Troubleshooting

S3 connection check failed: An error occurred (403) when calling the HeadBucket operation Wrong credentials, wrong region, or the IAM user/policy doesn't grant s3:ListBucket on the configured bucket. The same policy needs s3:GetObject, s3:PutObject, s3:DeleteObject on arn:aws:s3:::<bucket>/*.

Another s3_store backup is already running for this site. A previous backup crashed without releasing the lock. Check for hung tar processes; delete sites/<site>/.s3_store_backup.lock if confirmed stale.

Insufficient disk space to stage S3 files for backup Total of all S3 file sizes for the site exceeds 90% of free disk space. Either free up space or set include_in_native_backup = 0 in S3 Store Settings (you'll lose the self-contained-tar property).

Backup tar is missing some files Check the Error Log for "S3 Store backup staging" — individual download failures don't abort the backup, they're logged and skipped. The File doc is preserved so you can investigate.

bench s3-store-push-local reports "staged file missing" The File doc points at an S3 key, but no local copy was found at the expected staging path. Either (a) the file wasn't included in the backup tar (the source backup was made without --with-files), or (b) it was unpacked under an unexpected name (file_name was renamed in the source). The S3 object, if it exists, is unaffected.

---

Limitations

• Disk space is required on the server for both backup and restore. During backup, every S3-hosted file is downloaded to the server's local disk before tar runs; during restore, the tar is extracted to local disk before bench s3-store-push-local (or the auto-push hook) re-uploads everything. In both directions the server must have free disk space roughly equal to the total size of all S3-backed files. If that isn't possible, disable include_in_native_backup and manage S3 backups separately.

• bench backup --exclude-doctypes File is incompatible: the DB dump won't reference the staged files. There's no clean way to detect this from the patch site.

• External file_url (e.g. Google Drive thumbnails) is ignored by both the backup-staging path and the post-restore push — those URLs aren't under our control.

• Mixed-mode: if a file exists both locally and on S3, the local copy wins during backup staging. After restore, the file at the staging path is pushed up to S3 with the existing key. This is intentional — local-wins makes recovery scenarios predictable.

• Patch ordering: if another app monkey-patches BackupGenerator.backup_files after s3_store, our patch is lost. Both patches must chain via the saved original to coexist; s3_store does so.

• File.file_url index: the serve endpoint uses exact-match lookups, so it benefits from an index on file_url. Frappe doesn't ship one by default; on sites with millions of File rows, consider adding idx_file_file_url via a custom DB patch.

---

License

MIT