Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

engine: submit compressed measurements #2014

Open
bassosimone opened this issue Feb 8, 2022 · 0 comments
Open

engine: submit compressed measurements #2014

bassosimone opened this issue Feb 8, 2022 · 0 comments
Assignees
Labels
enhancement New feature request or improvement to existing functionality GSoC GSoC related issues ooni/probe-cli ooni/probe-engine priority/medium Normal priority issue

Comments

@bassosimone
Copy link
Contributor

bassosimone commented Feb 8, 2022

This issue is about submitting compressed measurements. We're going to start from compressing individual measurements using the stdlib's compress/gzip package. We'll close this issue when we've implemented compression.

This issue is a follow up to #1802.

We discussed this matter in today's backend meeting with @FedericoCeratto and @hellais. Here are notes from the meeting regarding what we think we'll need to do to make this happen:

  1. the probe needs to set a header to indicate the measurement is compressed;
  2. it seems python's gzip implementation could use a context manager to limit the used memory;
  3. the ooni/api does not touch the incoming measurement in any way and just stores it to disk where the upload and the fastpath will pick it up and, respectively, save it, process it and insert it into the fastpath table;
  4. so the ooni/api can just receive the compressed measurement and put it into a can;
  5. we need to use the filename as metadata to tell the rest of the backend the postcan is compressed;
  6. the fastpath needs to know how to handle compressed measurements.

See also ooni/backend#561.

@bassosimone bassosimone added enhancement New feature request or improvement to existing functionality priority/high Important issue that needs attention soon ooni/probe-engine labels Feb 8, 2022
@bassosimone bassosimone self-assigned this Feb 8, 2022
@bassosimone bassosimone changed the title probe: submit compressed measurements engine: submit compressed measurements Feb 8, 2022
@bassosimone bassosimone added GSoC GSoC related issues ooni/probe-cli labels Feb 18, 2022
@hellais hellais added this to Roadmap Jan 7, 2025
@hellais hellais added priority/medium Normal priority issue and removed priority/high Important issue that needs attention soon labels Jan 23, 2025
@hellais hellais assigned DecFox and unassigned bassosimone Jan 23, 2025
@hellais hellais moved this to Backlog in Roadmap Jan 23, 2025
@hellais hellais removed this from Roadmap Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature request or improvement to existing functionality GSoC GSoC related issues ooni/probe-cli ooni/probe-engine priority/medium Normal priority issue
Projects
None yet
Development

No branches or pull requests

3 participants