-
Notifications
You must be signed in to change notification settings - Fork 269
feat: upgraded to Pinecone v7 #633
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
shinobi-josh
wants to merge
66
commits into
main
Choose a base branch
from
semantic-router/pinecone-v7
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
66 commits
Select commit
Hold shift + click to select a range
63f609e
Upgraded to Pinecone v7
shinobi-josh eb74870
secondary commit
shinobi-josh f61b2bd
fixing port to 5080 default or base_url
shinobi-josh 0590e82
updated base url
shinobi-josh b51d5a9
using extra host for container
shinobi-josh 905dd53
tricking dagger CI for url
shinobi-josh 5d31847
removed extra host lines
shinobi-josh c968d3e
experimenting with URL
shinobi-josh 58f88c3
updated pinecone,py
shinobi-josh 389322a
changing back to pinecone
shinobi-josh e12080d
added monkey patch
shinobi-josh d714dbd
initializing indexes if they are not found
shinobi-josh 02fec0d
added exception logic to return empty lists
shinobi-josh 81651d8
lint fix and added NotFoundException to deal with robustnes
shinobi-josh b3fda5d
fixed exception added pinecone.exceptions
shinobi-josh d331559
added logging
shinobi-josh 8ccb23b
added 2 sec delay
shinobi-josh 48345ab
rerunning up to 5 times
shinobi-josh f23e9d5
testing directly on pinecone
shinobi-josh bdf1508
...
shinobi-josh b1fa8e7
....
shinobi-josh 5f3bfbe
...
shinobi-josh d0bb90a
debug printing
shinobi-josh 13f93e9
explicitly stating url
shinobi-josh 779ee1c
...
shinobi-josh ba63a22
...
shinobi-josh ca332c5
added debug log to main to see whats happening
shinobi-josh 2d5583b
updated test.yml to be inline for pinecone v7
shinobi-josh bfc48dc
removed any local emulator stuff and removed prints
shinobi-josh 781e7e7
gone through all of the pinecone.py locals to help cloud
shinobi-josh ca05889
changes to main
shinobi-josh 0fe5986
removed import time
shinobi-josh 79f261b
updated pinecone
shinobi-josh 50b724f
cloud runs never local
shinobi-josh 2e835cb
more bug fixes in ci pipeline
shinobi-josh 0fc0b46
fixes to pineccone.py
shinobi-josh 2014f5d
fixes again for pinecone.py
shinobi-josh c5d9dbf
reworked pinecone.py
shinobi-josh 104d117
...
shinobi-josh 8dad7b1
...
shinobi-josh fa35041
...
shinobi-josh d493f48
updated test
shinobi-josh 56fe8e4
...
shinobi-josh 72e825b
...
shinobi-josh aa0cc37
...
shinobi-josh effe800
Update test_sync.py
shinobi-josh 84ecd57
lint fix
shinobi-josh 7de8422
added pinecone.md (how to use guide)
shinobi-josh da6466b
Merge branch 'main' into pr/616
shinobi-josh 434796a
cleaned main
shinobi-josh 3a4f6bf
updated no more stdout - was using before to debug
shinobi-josh f363936
removed URL - need to test this to make sure
shinobi-josh 22b739e
Merge branch 'main' into semantic-router/pinecone-v7
shinobi-josh bd95c62
quick fix
shinobi-josh 01ecec2
Refactored the repeated env var reassignments into a single chained c…
shinobi-josh ba8c61e
Removed the explicit cloud base URL example from docs/user-guide/guid…
shinobi-josh 85d6223
Simplified the quota-handling logic in semantic_router/index/pinecone…
shinobi-josh d40afc1
Simplified the local emulator host handling in semantic_router/index/…
shinobi-josh 1b8f234
Removed the custom retry logic from _batch_upsert in semantic_router/…
shinobi-josh bc8cf24
Made the requested cleanups in semantic_router/index/pinecone.py
shinobi-josh 39fac0d
Switched async HTTP calls to the Pinecone async SDK
shinobi-josh 40589db
simplification - removing any retries, sleep and none needed complex …
shinobi-josh e23210e
Added Pinecone V7 features
shinobi-josh 9ef3a7f
linter errors fixed
shinobi-josh ffc605d
Revert "linter errors fixed"
shinobi-josh 3966250
Revert "Added Pinecone V7 features"
shinobi-josh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,112 @@ | ||
## Pinecone v7 integration | ||
|
||
This guide shows how to use Semantic Router with the Pinecone Python SDK v7+, including cloud vs local setup, shared-index reuse, and namespaces for isolation. | ||
|
||
### Install | ||
|
||
```bash | ||
pip install "semantic-router[pinecone]" | ||
``` | ||
|
||
### Environment variables | ||
|
||
- `PINECONE_API_KEY` (required): Your Pinecone API key | ||
- `PINECONE_API_BASE_URL` (optional): | ||
- Cloud: `https://api.pinecone.io` (default if a real API key is set) | ||
- Local emulator: `http://localhost:5080` or `http://pinecone:5080` | ||
- `PINECONE_INDEX_NAME` (recommended on cloud): Name of an existing index to reuse | ||
|
||
Why set `PINECONE_INDEX_NAME`? Pinecone serverless has per-project index limits. Reusing a shared index avoids 403 quota errors. Semantic Router will automatically isolate data using namespaces. | ||
|
||
### Basic usage (cloud) | ||
|
||
```python | ||
import os | ||
from semantic_router.encoders import OpenAIEncoder | ||
from semantic_router.index.pinecone import PineconeIndex | ||
from semantic_router.route import Route | ||
from semantic_router.routers import SemanticRouter | ||
|
||
# Required | ||
os.environ["PINECONE_API_KEY"] = "<your-key>" | ||
|
||
# Strongly recommended: reuse an existing index to avoid quota | ||
os.environ["PINECONE_INDEX_NAME"] = "semantic-router-shared" | ||
|
||
encoder = OpenAIEncoder(name="text-embedding-3-small") | ||
|
||
# Use a namespace for isolation (otherwise the router will use the requested | ||
# index name internally as the namespace when reusing a shared index) | ||
index = PineconeIndex(index_name="demo-index", namespace="demo", dimensions=1536) | ||
|
||
routes = [ | ||
Route(name="greeting", utterances=["hello", "hi"]), | ||
Route(name="goodbye", utterances=["bye", "goodbye"]), | ||
] | ||
|
||
router = SemanticRouter(encoder=encoder, routes=routes, index=index, auto_sync="local") | ||
|
||
print(router(text="hi there").name) # -> greeting | ||
``` | ||
|
||
Notes: | ||
- If the shared index exists, Semantic Router reuses it and writes route vectors under your `namespace`. | ||
- If you do not set `PINECONE_INDEX_NAME`, creating a new index requires `dimensions`. If index creation is forbidden (quota), a clear error is raised asking you to set `PINECONE_INDEX_NAME`. | ||
- You do not need to set `PINECONE_API_BASE_URL` for cloud; override it only when using the local emulator for testing. | ||
|
||
### Local emulator | ||
|
||
```bash | ||
export PINECONE_API_KEY=pclocal | ||
export PINECONE_API_BASE_URL=http://localhost:5080 | ||
``` | ||
|
||
In local mode, Semantic Router connects to the emulator at `http://localhost:5080` (or `http://pinecone:5080` in containerized CI) and adds a short delay after create to account for readiness. | ||
|
||
### Async usage | ||
|
||
```python | ||
import asyncio | ||
from semantic_router.routers import SemanticRouter | ||
|
||
async def main(): | ||
result = await router.acall("hello") | ||
print(result.name) | ||
|
||
asyncio.run(main()) | ||
``` | ||
|
||
Internally, the library resolves the Pinecone v7 data-plane host and uses the correct `/vectors/query` endpoint for async queries. | ||
|
||
### Error handling and retries | ||
|
||
- 403 (quota): The library attempts to reuse an existing index. If none is available, it raises an error advising you to set `PINECONE_INDEX_NAME`. | ||
- 404 (eventual consistency): Readiness checks and upserts include brief bounded retries. | ||
|
||
### CI tips (GitHub Actions) | ||
|
||
- Set secrets: | ||
- `PINECONE_API_KEY` | ||
- `PINECONE_INDEX_NAME` (existing shared index) | ||
- Ensure the environment uses cloud: | ||
|
||
```yaml | ||
env: | ||
PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }} | ||
PINECONE_INDEX_NAME: ${{ secrets.PINECONE_INDEX_NAME }} | ||
|
||
steps: | ||
- name: Run tests | ||
run: | | ||
PINECONE_API_BASE_URL="https://api.pinecone.io" \ | ||
pytest -q | ||
``` | ||
|
||
Tests that require Pinecone will automatically skip in cloud mode if `PINECONE_INDEX_NAME` isn’t provided, avoiding quota-based failures. | ||
|
||
### Requirements recap | ||
|
||
- Pinecone Python client v7+ | ||
- Semantic Router ≥ version including Pinecone v7 support (this branch) | ||
- Recommended on cloud: `PINECONE_INDEX_NAME` pointing at an existing index | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these are not needed by postgres in the tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes — not needed. In tests:
The Dagger job binds the Postgres service as postgres:5432.
.dagger/src/semantic_router_ci/main.py sets these envs inside the test container with sensible defaults:
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_DB=postgres
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
Since we set them in the container with defaults, the workflow doesn’t need to pass them.
Kept the defaults in main.py and removed the redundant workflow-level Postgres envs.