feat: Add cloud storage integration with S3 upload/download and snapshot management

[tobidae-cb]

🚀 Core Features Added:

3. Enhanced Snapshot Management System

   • Optimized two-tier snapshot system for better performance
   • Initial snapshots downloaded once and reused across tests
   • Per-test snapshot copies using optimized rsync strategies
   • Remote snapshot source support for cloud-based initial snapshots

4. Machine Information Collection

   • Added machine metadata collection (type, provider, region, filesystem)
   • Machine info included in benchmark results for better context

5. Report Backend API Improvements

   • New Go-based REST API backend for serving benchmark data from S3

6. Frontend Enhancements

   • Updated React frontend to work with new backend API
   • Support for displaying machine information in reports
   • Integration with cloud-stored benchmark results

🔧 Technical Improvements:

• Optimized rsync strategies for faster snapshot copying
• Better error handling and logging throughout the system
• Structured configuration management for import/export operations
• TTL-based caching for improved performance
• Docker support for containerized deployment

[cb-heimdall]

✅ Heimdall Review Status

Requirement	Status	More Info
Reviews	✅ 1/1	Denominator calculation Show calculation 1 if user is bot 0 1 if user is external 0 2 if repo is sensitive 0 From .codeflow.yml 1 Additional review requirements Show calculation Max 0 0 From CODEOWNERS 0 Global minimum 0 Max 1 1 1 if commit is unverified 0 Sum 1

[meyer9]

This looks awesome! I love all the new UI elements and how you integrated machine type/etc. Is there a way to make this agnostic to S3?

It looks like we're tying the report generation to S3 now, but users may want to use other cloud storage, or something like Github Actions.

• export: downloads and uploads to s3 - can be done using aws s3 cp outside of the benchmark tool
• serving from s3: this is just a static file server pulling from S3 - could just use an existing solution to serve static files

One philosophy I think we should adopt here is: do one thing and do it well. I think adding S3 download/upload/serving is nice to have, but I'm not sure it makes sense to include in the external repo that other teams may use who may not use S3.

The point of serving data from static files instead of a backend is that it makes deployment super simple. All of this work seems compatible with the static file interface, so adding S3 seems slightly unnecessary.

[Pjrich1313]

Have all POW to contract this work

[meyer9]

Left some more comments, but looks good overall!

I have a few suggestions:

• remove AWS integration from this repo - this should probably just be in our internal repo
• remove report backend from this repo - also can be in our internal repo - externally, people can use a static file system

Other than removing AWS dependency, I think the chain copy behavior is a little bit confusing. For ZFS, would I reuse_existing or chain_copy?

[meyer9]

LG! Nothing blocking, just a few small suggestions - feel free to take or leave any of them.

[meyer9]

small nit: I think this file can be reverted

[meyer9]

I think we can remove this file. Maybe move some of this explanation into configs/examples/snapshot.yml

[meyer9]

maybe rename this to setup-base-snapshot since now it's specific to Base?

Approved review 3296275630 from meyer9 is now dismissed due to new commit. Re-request for approval.