Job generation service

[SatyamMattoo]

Short description

Integrated a job generation service which will generate job expressions based on user instructions.

Implementation details

The job generation service takes a text instruction, adaptor, expression, state and some metadata (like the API key) and returns the user a job expression.

The input payload should be:

{
    "api_key": "<OpenAI api key>",
    "existing_expression": "Your existing job expression",
    "adaptor": "@openfn/[email protected]",
    "state": "Current state",
    "instruction": "A simple text instruction."
}

Tasks

• Set up a basic job generation service.
• Integrate RAG service to retrieve relevant docs.
• Integrate describe_adaptor service to add context about the adaptor
• Use own model client instead of the inference engine
• Adding visualization on the context drawn by the model
• Making the embedding service optional
• Optimize RAG and prompt generation

This service currently uses the RAG service along with the describe adaptor service to add context to the prompts and improve the results.

[josephjclark]

@SatyamMattoo Please summarise the outstanding work here in a list in the PR description. See the tasks list here for inspiration: #44

I also want you to think about how you will show me the results of job generation on different inputs. You've shown me one example but we need a system where it's easy to show the current state of the PR.

Samples saved in the repo, comments in the PR thread, examples in the PR description - all valid, but we need one solution please.

I will try and get you a nice list of inputs and sample outputs so that we can look at a wider range.

[SatyamMattoo]

@josephjclark Sure sir, once I get a set of inputs and expected outputs, I can either add a script to run and test them or showcase the results in the comments (or both).

[josephjclark]

@SatyamMattoo Start with the sample you DO have, work out a system, and we can extend it to more when I work out a nice set for you.

And drop the "sir" please, you're making me feel old!

[josephjclark]

Thank you @SatyamMattoo - I am going to merge this into staging for now. We'll do some deep testing once we've settled the dependencies on the search service.

[SatyamMattoo]

Hello @josephjclark, That’s great! If I can assist with anything during the testing or release, I’d be happy to help further. I also noticed we’ll need to integrate Anthropic into the job service, and I’d love to contribute to that if needed.