Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
63f609e
Upgraded to Pinecone v7
shinobi-josh Aug 4, 2025
eb74870
secondary commit
shinobi-josh Aug 4, 2025
f61b2bd
fixing port to 5080 default or base_url
shinobi-josh Aug 4, 2025
0590e82
updated base url
shinobi-josh Aug 4, 2025
b51d5a9
using extra host for container
shinobi-josh Aug 4, 2025
905dd53
tricking dagger CI for url
shinobi-josh Aug 4, 2025
5d31847
removed extra host lines
shinobi-josh Aug 4, 2025
c968d3e
experimenting with URL
shinobi-josh Aug 4, 2025
58f88c3
updated pinecone,py
shinobi-josh Aug 4, 2025
389322a
changing back to pinecone
shinobi-josh Aug 4, 2025
e12080d
added monkey patch
shinobi-josh Aug 5, 2025
d714dbd
initializing indexes if they are not found
shinobi-josh Aug 5, 2025
02fec0d
added exception logic to return empty lists
shinobi-josh Aug 5, 2025
81651d8
lint fix and added NotFoundException to deal with robustnes
shinobi-josh Aug 5, 2025
b3fda5d
fixed exception added pinecone.exceptions
shinobi-josh Aug 5, 2025
d331559
added logging
shinobi-josh Aug 5, 2025
8ccb23b
added 2 sec delay
shinobi-josh Aug 5, 2025
48345ab
rerunning up to 5 times
shinobi-josh Aug 5, 2025
f23e9d5
testing directly on pinecone
shinobi-josh Aug 5, 2025
bdf1508
...
shinobi-josh Aug 5, 2025
b1fa8e7
....
shinobi-josh Aug 5, 2025
5f3bfbe
...
shinobi-josh Aug 5, 2025
d0bb90a
debug printing
shinobi-josh Aug 5, 2025
13f93e9
explicitly stating url
shinobi-josh Aug 5, 2025
779ee1c
...
shinobi-josh Aug 5, 2025
ba63a22
...
shinobi-josh Aug 5, 2025
ca332c5
added debug log to main to see whats happening
shinobi-josh Aug 11, 2025
2d5583b
updated test.yml to be inline for pinecone v7
shinobi-josh Aug 11, 2025
bfc48dc
removed any local emulator stuff and removed prints
shinobi-josh Aug 11, 2025
781e7e7
gone through all of the pinecone.py locals to help cloud
shinobi-josh Aug 11, 2025
ca05889
changes to main
shinobi-josh Aug 11, 2025
0fe5986
removed import time
shinobi-josh Aug 11, 2025
79f261b
updated pinecone
shinobi-josh Aug 11, 2025
50b724f
cloud runs never local
shinobi-josh Aug 11, 2025
2e835cb
more bug fixes in ci pipeline
shinobi-josh Aug 11, 2025
0fc0b46
fixes to pineccone.py
shinobi-josh Aug 11, 2025
2014f5d
fixes again for pinecone.py
shinobi-josh Aug 11, 2025
c5d9dbf
reworked pinecone.py
shinobi-josh Aug 11, 2025
104d117
...
shinobi-josh Aug 11, 2025
8dad7b1
...
shinobi-josh Aug 11, 2025
fa35041
...
shinobi-josh Aug 11, 2025
d493f48
updated test
shinobi-josh Aug 11, 2025
56fe8e4
...
shinobi-josh Aug 11, 2025
72e825b
...
shinobi-josh Aug 11, 2025
aa0cc37
...
shinobi-josh Aug 11, 2025
effe800
Update test_sync.py
shinobi-josh Aug 11, 2025
84ecd57
lint fix
shinobi-josh Aug 11, 2025
7de8422
added pinecone.md (how to use guide)
shinobi-josh Aug 11, 2025
da6466b
Merge branch 'main' into pr/616
shinobi-josh Aug 11, 2025
434796a
cleaned main
shinobi-josh Aug 11, 2025
3a4f6bf
updated no more stdout - was using before to debug
shinobi-josh Aug 12, 2025
f363936
removed URL - need to test this to make sure
shinobi-josh Aug 12, 2025
22b739e
Merge branch 'main' into semantic-router/pinecone-v7
shinobi-josh Aug 12, 2025
bd95c62
quick fix
shinobi-josh Aug 12, 2025
01ecec2
Refactored the repeated env var reassignments into a single chained c…
shinobi-josh Aug 12, 2025
ba8c61e
Removed the explicit cloud base URL example from docs/user-guide/guid…
shinobi-josh Aug 12, 2025
85d6223
Simplified the quota-handling logic in semantic_router/index/pinecone…
shinobi-josh Aug 12, 2025
d40afc1
Simplified the local emulator host handling in semantic_router/index/…
shinobi-josh Aug 12, 2025
1b8f234
Removed the custom retry logic from _batch_upsert in semantic_router/…
shinobi-josh Aug 12, 2025
bc8cf24
Made the requested cleanups in semantic_router/index/pinecone.py
shinobi-josh Aug 12, 2025
39fac0d
Switched async HTTP calls to the Pinecone async SDK
shinobi-josh Aug 12, 2025
40589db
simplification - removing any retries, sleep and none needed complex …
shinobi-josh Aug 12, 2025
e23210e
Added Pinecone V7 features
shinobi-josh Aug 12, 2025
9ef3a7f
linter errors fixed
shinobi-josh Aug 12, 2025
ffc605d
Revert "linter errors fixed"
shinobi-josh Aug 12, 2025
3966250
Revert "Added Pinecone V7 features"
shinobi-josh Aug 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 28 additions & 9 deletions .dagger/src/semantic_router_ci/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,7 @@ async def test(
pinecone_api_key: str = "",
python_version: str = "3.11",
) -> str:
"""Runs tests for semantic-router, scope can be
set to run for 'unit', 'functional', 'integration',
or 'all'. By default scope is set to 'unit'.
"""
"""Runs tests for semantic-router. Scope: 'unit' (default), 'functional', 'integration', or 'all'."""
# Map scope to pytest arguments
if scope == "all":
pytest_args = [
Expand Down Expand Up @@ -164,10 +161,34 @@ async def test(
container = container.with_env_variable(
"PINECONE_API_KEY", pinecone_api_key
)
# Forward optional shared index name to the test container
pinecone_index_name = os.environ.get("PINECONE_INDEX_NAME")
if pinecone_index_name:
container = container.with_env_variable(
"PINECONE_INDEX_NAME", pinecone_index_name
)
container = container.with_service_binding("postgres", self.postgres_service())
pinecone_api_base_url = os.environ.get("PINECONE_API_BASE_URL")
# Decide cloud vs local
if pinecone_api_base_url is None:
# No explicit base URL provided; infer from API key
if pinecone_api_key and pinecone_api_key != "pclocal":
# Real key provided: prefer cloud
pinecone_api_base_url = "https://api.pinecone.io"
else:
# Local mode
pinecone_api_base_url = "http://pinecone:5080"
# Start local emulator only if pointing to local
if (
pinecone_api_base_url.startswith("http://pinecone:5080")
or "localhost" in pinecone_api_base_url
):
container = container.with_service_binding(
"pinecone", self.pinecone_service()
)
# Set env vars inside test container
container = (
container.with_service_binding("postgres", self.postgres_service())
.with_service_binding("pinecone", self.pinecone_service())
.with_env_variable("PINECONE_API_BASE_URL", "http://pinecone:5080")
container.with_env_variable("PINECONE_API_BASE_URL", pinecone_api_base_url)
.with_env_variable(
"POSTGRES_HOST", os.environ.get("POSTGRES_HOST", "postgres")
)
Expand All @@ -180,7 +201,5 @@ async def test(
"POSTGRES_PASSWORD", os.environ.get("POSTGRES_PASSWORD", "postgres")
)
)
# Debug: print env vars inside the container
container = container.with_exec(["env"])
container = container.with_exec(pytest_args)
return await container.stdout()
12 changes: 7 additions & 5 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,12 @@ on:
jobs:
test:
runs-on: ubuntu-latest
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
# If you have a shared Pinecone index, set this secret to reuse it in CI
PINECONE_INDEX_NAME: ${{ secrets.PINECONE_INDEX_NAME }}
timeout-minutes: 20 # Fail the job if it runs longer than 20 minutes
strategy:
matrix:
Expand Down Expand Up @@ -49,11 +55,7 @@ jobs:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
POSTGRES_HOST: postgres
POSTGRES_PORT: 5432
POSTGRES_DB: postgres
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
Comment on lines -52 to -56
Copy link
Member

@jamescalam jamescalam Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are not needed by postgres in the tests?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes — not needed. In tests:
The Dagger job binds the Postgres service as postgres:5432.
.dagger/src/semantic_router_ci/main.py sets these envs inside the test container with sensible defaults:
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_DB=postgres
POSTGRES_USER=postgres
POSTGRES_PASSWORD=postgres
Since we set them in the container with defaults, the workflow doesn’t need to pass them.
Kept the defaults in main.py and removed the redundant workflow-level Postgres envs.

PINECONE_INDEX_NAME: ${{ secrets.PINECONE_INDEX_NAME }}

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v2
Expand Down
112 changes: 112 additions & 0 deletions docs/user-guide/guides/pinecone-v7.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
## Pinecone v7 integration

This guide shows how to use Semantic Router with the Pinecone Python SDK v7+, including cloud vs local setup, shared-index reuse, and namespaces for isolation.

### Install

```bash
pip install "semantic-router[pinecone]"
```

### Environment variables

- `PINECONE_API_KEY` (required): Your Pinecone API key
- `PINECONE_API_BASE_URL` (optional):
- Cloud: `https://api.pinecone.io` (default if a real API key is set)
- Local emulator: `http://localhost:5080` or `http://pinecone:5080`
- `PINECONE_INDEX_NAME` (recommended on cloud): Name of an existing index to reuse

Why set `PINECONE_INDEX_NAME`? Pinecone serverless has per-project index limits. Reusing a shared index avoids 403 quota errors. Semantic Router will automatically isolate data using namespaces.

### Basic usage (cloud)

```python
import os
from semantic_router.encoders import OpenAIEncoder
from semantic_router.index.pinecone import PineconeIndex
from semantic_router.route import Route
from semantic_router.routers import SemanticRouter

# Required
os.environ["PINECONE_API_KEY"] = "<your-key>"

# Strongly recommended: reuse an existing index to avoid quota
os.environ["PINECONE_INDEX_NAME"] = "semantic-router-shared"

encoder = OpenAIEncoder(name="text-embedding-3-small")

# Use a namespace for isolation (otherwise the router will use the requested
# index name internally as the namespace when reusing a shared index)
index = PineconeIndex(index_name="demo-index", namespace="demo", dimensions=1536)

routes = [
Route(name="greeting", utterances=["hello", "hi"]),
Route(name="goodbye", utterances=["bye", "goodbye"]),
]

router = SemanticRouter(encoder=encoder, routes=routes, index=index, auto_sync="local")

print(router(text="hi there").name) # -> greeting
```

Notes:
- If the shared index exists, Semantic Router reuses it and writes route vectors under your `namespace`.
- If you do not set `PINECONE_INDEX_NAME`, creating a new index requires `dimensions`. If index creation is forbidden (quota), a clear error is raised asking you to set `PINECONE_INDEX_NAME`.
- You do not need to set `PINECONE_API_BASE_URL` for cloud; override it only when using the local emulator for testing.

### Local emulator

```bash
export PINECONE_API_KEY=pclocal
export PINECONE_API_BASE_URL=http://localhost:5080
```

In local mode, Semantic Router connects to the emulator at `http://localhost:5080` (or `http://pinecone:5080` in containerized CI) and adds a short delay after create to account for readiness.

### Async usage

```python
import asyncio
from semantic_router.routers import SemanticRouter

async def main():
result = await router.acall("hello")
print(result.name)

asyncio.run(main())
```

Internally, the library resolves the Pinecone v7 data-plane host and uses the correct `/vectors/query` endpoint for async queries.

### Error handling and retries

- 403 (quota): The library attempts to reuse an existing index. If none is available, it raises an error advising you to set `PINECONE_INDEX_NAME`.
- 404 (eventual consistency): Readiness checks and upserts include brief bounded retries.

### CI tips (GitHub Actions)

- Set secrets:
- `PINECONE_API_KEY`
- `PINECONE_INDEX_NAME` (existing shared index)
- Ensure the environment uses cloud:

```yaml
env:
PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
PINECONE_INDEX_NAME: ${{ secrets.PINECONE_INDEX_NAME }}

steps:
- name: Run tests
run: |
PINECONE_API_BASE_URL="https://api.pinecone.io" \
pytest -q
```

Tests that require Pinecone will automatically skip in cloud mode if `PINECONE_INDEX_NAME` isn’t provided, avoiding quota-based failures.

### Requirements recap

- Pinecone Python client v7+
- Semantic Router ≥ version including Pinecone v7 support (this branch)
- Recommended on cloud: `PINECONE_INDEX_NAME` pointing at an existing index

2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,13 @@ local = [
"sentence-transformers>=5.0.0 ; python_version < '3.13'",
"torch>=2.6.0 ; python_version < '3.13'"
]
pinecone = ["pinecone>=7.0.0,<8.0.0"]
vision = [
"torchvision>=0.17.0 ; python_version < '3.13'",
"transformers>=4.36.2 ; python_version < '3.13'",
"pillow>=10.2.0,<11.0.0 ; python_version < '3.13'",
"torch>=2.6.0 ; python_version < '3.13'"
]
pinecone = ["pinecone>=5.0.0,<6.0.0"]
mistralai = ["mistralai>=0.0.12,<0.1.0"]
qdrant = ["qdrant-client>=1.11.1,<2"]
google = ["google-cloud-aiplatform>=1.45.0,<2"]
Expand Down
1 change: 0 additions & 1 deletion semantic_router/index/hybrid_local.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ class HybridLocalIndex(LocalIndex):
def __init__(self, **data):
super().__init__(**data)
self.metadata = None


def add(
self,
Expand Down
Loading