Skip to content

Serverless ES support (a working draft) #271

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from

Conversation

ccancellieri
Copy link

@ccancellieri ccancellieri commented Jun 17, 2024

Related Issue(s):
Hi all,
I'm trying to use this great app to accomplish my task:

  1. Connect with the ElasticSearch Serverless provided by ElasticSearch: https://elasticsearch-serverless-python.readthedocs.io/en/stable/api.html#module-elasticsearch_serverless

  2. Make use of the right API and async app ( https://elasticsearch-py.readthedocs.io/en/latest/async.html

  3. Ensure post and get capabilities to/from ES Serverless Stac items loaded with the python script.

Description:

This is only a rude but working implementation.
I'm curious to understand:

  • Could it be useful?
  • How can I better integrate the patch with your codebase (a flag during the deploy for the imports)?

I can also contribute with a remote debug setup with debugpy and vscode.

PR Checklist:

  • Code is formatted and linted (run pre-commit run --all-files)
  • Tests pass (run make test)
  • Documentation has been updated to reflect changes, if applicable
  • Changes are added to the changelog

@jonhealy1
Copy link
Collaborator

jonhealy1 commented Jun 18, 2024

Hi @ccancellieri it would be really nice to have this working with elasticsearch serverless. Having a flag could work I think. Maybe someone else will have some thoughts too.

If I am understanding, this is working on elasticsearch serverless with these modifications?

@ccancellieri
Copy link
Author

Hi @ccancellieri it would be really nice to have this working with elasticsearch serverless. Having a flag could work I think. Maybe someone else will have some thoughts too.

Thank you!

If I am understanding, this is working on elasticsearch serverless with these modifications?

Yes, it is :)

I've a working draft in use under our review environment, it's under test at the moment.

I'll be back in september with a refined review of the changes to provide to complete the pull request.

My best

@ccancellieri
Copy link
Author

I'm not sure how to add tests, do you have some feedback on this?

@jamesfisher-geo
Copy link
Collaborator

I'm not sure how to add tests, do you have some feedback on this?

I suppose that you will need to add serverless elasticsearch and a serverless es API to this docker-compose file. https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/blob/main/docker-compose.yml

You can then create a make command to run the tests against that service. https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/blob/main/Makefile

@jonhealy1
Copy link
Collaborator

Hi @ccancellieri what are your thoughts on this pr presently?

@ccancellieri
Copy link
Author

Hi @ccancellieri what are your thoughts on this pr presently?

Dear @jonhealy1 I will close this pull req. we are not maintaining this fork anymore.
Unfortunately several other patches and functionalities have been requested but none of them have been accepted, so now we are on a fork, I'm sorry.

@jonhealy1
Copy link
Collaborator

Hi @ccancellieri I have only started to have some serious time to look at this project again. What fork are you talking about? Can you send me the link - thank you.

@ccancellieri
Copy link
Author

@jonhealy1 the repository is private but I'm discussing some of the problems here

I'm now extending this service overriding only some parts not forking the entire codebase but with the recent updates I'm getting the above problem.

I also have had to switch to the Elastic Cloud Hosted service as the Serverless is still costing too much for us.

To give you more hints the unique difference I've found between the ES Serverless and the Elastic Cloud Hosted services is that the creation of the indices and schemas for the ES Serverless can't be async, for that you could just use the sync client, it will work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants