Docsite rag

[hanna-paasivirta]

Short Description

Replace the Search service with new embed_docsite and search_docsite services.

Fixes #172

Implementation Details

The embed_docsite service downloads, chunks, processes metadata and indexes OpenFn documentation. The service uses Pinecone as a vector database and OpenAI for text embeddings.

The search_docsite searches the documentation through the vector database using and input query.

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[hanna-paasivirta]

Todo: test on full docs; finish DocsiteSearch according to index metadata structure

[josephjclark]

Thank you @hanna-paasivirta! I've gotta tied up with a production issue but we'll get this reviewed and merged tomorrow :)

[josephjclark]

This is really nice and clean, thank you! Even I have a fighting chance of understanding it.

I'd like to test before merging but it looks fantastic. Left a couple of questions in the meantime.

[josephjclark]

Is sleep really the best way to handle this?

What happens if the update isn't complete after 30 seconds?

[hanna-paasivirta]

I added a while loop and a max wait time instead. This works in the current implementation where we’re not expecting several indexing jobs running in parallel.

[josephjclark]

Sorry, just before we merge and forget about this, can we feed docs_to_ignore into the processor from here from user args?

And can we then update the readme with:
a) one example with default values (which will do everything)
b) one example which sets docs_to_upload and _docs_to_ignore

[hanna-paasivirta]

Sorry I forgot to commit this file, I've added it in now along with the readme.