Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scalability / Query Size limitations? #39

Open
sephcoster opened this issue Jun 26, 2017 · 4 comments
Open

Scalability / Query Size limitations? #39

sephcoster opened this issue Jun 26, 2017 · 4 comments
Labels

Comments

@sephcoster
Copy link

Hello!

Based on your previous experience with this plugin can you speak to the kind of scale required to support somewhat sizable exports, and the kind of size of data that it supports well?

Does it scale in a similar manner to elasticsearch based on nodes or are there different considerations / limitations I should think about? Is dumping 100,000+ items going to bring my cluster to a halt?

I'm considering dataformat as an option for a data export feature but want to make sure I am able to scale my infrastructure to support the number of requests we're expecting. We'll conduct some performance testing before going to production but appreciate any advice you might have to get us started.

Thank you so much for this awesome library and appreciate your thoughts!

#Question

@marevol
Copy link
Contributor

marevol commented Jun 27, 2017

For the current version, download size depends on heap size of elasticsearch.
If you want to generate a file which is unlimited size, use file request parameter(ex. file=/tmp/filename.csv).

@anujtmr9
Copy link

I m working on same problem. We are exporting at least 100,000 every time. What u need to take care is memory allocation for the plugin so that it can write the result at server and dump to desired location

@sephcoster
Copy link
Author

sephcoster commented Jun 27, 2017

This is great information @marevol and thank you for being so responsive. Also appreciate your insight @anujtmr9 and will check to make sure we have enough memory freed up.

@sephcoster
Copy link
Author

This should give us enough to go on for now - please feel free to close this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants