Skip to content

Fastpath writes to disk #971

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 42 commits into
base: master
Choose a base branch
from
Open

Fastpath writes to disk #971

wants to merge 42 commits into from

Conversation

LDiazN
Copy link
Contributor

@LDiazN LDiazN commented Jul 14, 2025

With our current setup the API in the monolith is writing report data to disk with the /report/{id} endpoint while also sending the same data to the fastpath.

This data is meant to be processed by the api uploader.

However since we intend to move this setup to AWS, we need a different approach, see #968

This PR will move the file-writing part to the Fastpath once the measurement is received. This is necessary because the ooniprobe service is unable to share disk with the EC2 machine running fastpath and api uploader

The file-writing feature is under a feature flag and deactivated by default. To activate it we add this in fastpath.conf

[DEFAULT]
...
write_to_disk = true

It will also add the config parameter msmt_spool_dir to the configuration file. It will only be parsed when write_to_disk == true

depends on #969
closes #968

@LDiazN LDiazN requested a review from hellais July 14, 2025 08:51
@LDiazN LDiazN self-assigned this Jul 14, 2025
Copy link
Member

@hellais hellais left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together it looks very good. I have left a few comments for things to change. Once addressed we can move it forward.

@hellais hellais self-requested a review July 16, 2025 13:07
@hellais hellais self-requested a review July 16, 2025 15:18
Copy link
Member

@hellais hellais left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@hellais
Copy link
Member

hellais commented Jul 17, 2025

Next steps to get this deployed in production:

  1. Bump version inside of https://github.com/ooni/backend/blob/master/fastpath/debian/changelog to 0.87
  2. Test deploy 0.87 on backend-hel to make sure the fastpath without write-to-disk enabled still works as intended
  3. Land this into master and deploy the master branch onto dev
  4. Do some testing with ooniprobe and miniooni coordinating with @DecFox
  5. Make a plan to deploy it into production during a specific week (ideally starting deployment on Monday)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for writing files to disk in fastpath
2 participants