New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

store intermediate aggregate_spatial output on object storage #274

Open

jdries opened this issue Nov 24, 2022 · 0 comments

Contributor

jdries commented Nov 24, 2022 •

edited

Loading

On Creo, intermediate output of aggregate_spatial is now written to /shared_pod_volume, requiring a shared mount of an S3 bucket.

Spark can also write to S3 directly:
https://spark.apache.org/docs/latest/cloud-integration.html
We can use the batch job output directory if it's a batch job.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment