Skip to content

build: add embedding providers and related configurations #1565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

feliciien
Copy link

This commit introduces the embedding providers feature, including OpenAI, Cohere, HuggingFace, and VoyageAI. It includes the necessary configurations, build scripts, and CI workflows to support embedding generation and reranking functionality. The changes include:

  • Added .cargo/config.toml for WASM target configuration
  • Updated Cargo.toml to include new embedding providers
  • Added .gitignore files for embedding providers
  • Introduced .env and .env.template for API key management
  • Added build and test scripts for embedding providers
  • Included Dockerfile for VoyageAI provider
  • Added WIT bindings and common embedding functionality
  • Implemented logging, rate limiting, and error handling
  • Added benchmarking and testing utilities

The changes enable embedding generation and reranking capabilities across multiple providers, with support for durability, rate limiting, and comprehensive testing. This lays the foundation for integrating embedding functionality into Golem applications. link to the video https://drive.google.com/file/d/1DTFbGmM7z2ZV6T4SuZwLgrNAChZlI-zo/view?usp=sharing
/claim #1510

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant