feat!: Migrate Vector DB IDs to Vector Store IDs (breaking change)

[franciscojavierarceo]

What does this PR do?

This change migrates the VectorDB id generation to Vector Stores.

This is a breaking change for some users that may have application code using the vector_db_id parameter in the request of the VectorDB protocol instead of the VectorDB.identifier in the response.

By default we will now create a Vector Store every time we register a VectorDB. The caveat with this approach is that this maps the vector_db_id → vector_store.name. This is a reasonable tradeoff to transition users towards OpenAI Vector Stores.

As an added benefit, registering VectorDBs will result in them appearing in the VectorStores admin UI.

Why?

This PR makes the POST API call to /v1/vector-dbs swap the vector_db_id parameter in the request body into the VectorStore's name field and sets the vector_db_id to the generated vector store id (e.g., vs_038247dd-4bbb-4dbb-a6be-d5ecfd46cfdb).

That means that users would have to do something like follows in their application code:

res = client.vector_dbs.register(
    vector_db_id='my-vector-db-id', 
    embedding_model='ollama/all-minilm:l6-v2', 
    embedding_dimension=384,
)
vector_db_id = res.identifier

And then the rest of their code would behave, including VectorIO's insert protocol using vector_db_id in the request.

An alternative implementation would be to just delete the vector_db_id parameter in VectorDB but the end result would still require users having to write vector_db_id = res.identifier since VectorStores.create() generates the ID for you.

So this approach felt the easiest way to migrate users towards VectorStores (subsequent PRs will be added to trigger files.create() and vector_stores.files.create()).

Test Plan

Unit tests and integration tests have been added.

[varshaprasad96]

/lgtm
The changes look good. However, in longer term it would be good to rip Vector DB and support only vector stores directly. To be able to use vector dbs now, we are breaking the ability to pass vector_db_id to subsequent calls in vector_io which was possible before and this can be confusing.

[2000krysztof]

/lgtm

[franciscojavierarceo]

/lgtm The changes look good. However, in longer term it would be good to rip Vector DB and support only vector stores directly. To be able to use vector dbs now, we are breaking the ability to pass vector_db_id to subsequent calls in vector_io which was possible before and this can be confusing.

So this PR makes the POST API call to /v1/vector-dbs swap the vector_db_id parameter in the request body into the VectorStore's name field and sets the vector_db_id to the generated vector store id (e.g., vs_038247dd-4bbb-4dbb-a6be-d5ecfd46cfdb).

That means that users would have to do something like follows in their application code:

res = client.vector_dbs.register(
    vector_db_id='my-vector-db-id', 
    embedding_model='ollama/all-minilm:l6-v2', 
    embedding_dimension=384,
)
vector_db_id = res.identifier

And then the rest of their code would behave, including VectorIO's insert protocol using vector_db_id in the request.

An alternative implementation would be to just delete the vector_db_id parameter in VectorDB but the end result would still require users having to write vector_db_id = res.identifier since VectorStores.create() generates the ID for you.

[franciscojavierarceo]

I think this is a reasonable way to highlight this to users.

[franciscojavierarceo]

@leseb mind taking a look at this one?

[leseb]

I don't understand, if it's deprecated (or want to deprecate soon), why convert the call to vector store under the hood? Can't we just fail the call and tell users to use VectorStore?

[franciscojavierarceo]

I wanted to provide a lower friction migration.

The path would be:

1. Create Vector Stores (this PR)
2. Create Files and Vector Store Files for user (would be next PR)

Doing this would make the migration easier for users IMO. Under the hood, Vector Stores uses most of the VectorDB and VectorIO logic.

Given we have so many Red Hat users of RAG, I thought this was an easier path for them.

For some, they may just rip off the bandaid and that'd definitely be preferred but I'm happy to try and reduce the friction as much as possible.

[leseb]

I'm wondering if we should park this for 0.3.0 since it's a breaking change? @mattf @ashwinb thoughts? Thanks!

[franciscojavierarceo]

I'm good with that. I'm happy to also include the other items I mentioned in this PR then too if @mattf and @ashwinb are comfortable with this.

To be clear, I am in favor of deprecating the APIs and I am only doing this because (1) I know many Red Hat+ use cases rely on VectorIO and VectorDB APIs and (2) even though the Stack is still early in its adoption lifecycle, I want to treat users with care as we deprecate APIs and I believe this work will provide that.

[r3v5]

Great work, @franciscojavierarceo ! lgtm

[leseb]

This hasn’t received much attention, and I’d like to move it forward. We’re on the right track, and precautions have been taken to minimize any impact on end users. Thank you!