New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Increase the row limit or add pagination support to the BigQuery executor #5268

Open

arjun-1 opened this issue Feb 11, 2025 · 1 comment

Contributor

arjun-1 commented Feb 11, 2025

Currently, the BigQuery executor is limited to only 10.000 rows:

windmill/backend/windmill-worker/src/bigquery_executor.rs

Line 149 in d0c0eca

    
           "More than 10000 rows were requested, use LIMIT 10000 to limit the number of rows".to_string(),

The limit is rather arbitrary, and not suited for many use cases. The reason seems that no pagination was implemented, and the page size is taken as total limit for the response: https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/query

Contributor

rubenfiszel commented Feb 11, 2025

👍 we should be able to stream and also to store result in s3 when too large @wendrul

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment