While reviewing ooni/devops#240 with @LDiazN we realised we are still missing a piece to get this deployed.

Namely, at the moment files are written to disk by the OONI API and not the fastpath (see: https://github.com/ooni/backend/blob/master/api/ooniapi/probe_services.py#L859). This means that we would have to somehow share the disk between fastpath and OONI API, but the API we would like it to live on ECS.

Instead of sharing the disk we concluded that the best way forward is to move the code for writing to disk into the fastpath.

The data payload is already being transfered over the network: https://github.com/ooni/backend/blob/master/api/ooniapi/probe_services.py#L945C22-L945C26.

We should be sure to log several metrics related to this, especially when the upload fails: https://github.com/ooni/backend/blob/master/api/ooniapi/probe_services.py#L949 and upload the unsubmitted file to s3 at a particular prefix.

And then add logic to write the json blob to disk in the msm_processor https://github.com/ooni/backend/blob/master/fastpath/fastpath/core.py#L1579. Be careful to not do it in the process_measurement function, as that's called also by the reprocessor and we don't want to duplicate files when we are reprocessing them.