Code

Job generation service#92

Code

Merged

josephjclark merged 68 commits intorelease/nextOpenFn/apollo:release/nextfrom

gen_jobOpenFn/apollo:gen_jobCopy head branch name to clipboard

Sep 19, 2024

Commits

Commits on Jun 23, 2024

• initial setup for job expression generation service

  SatyamMattoo

  committed
  9899b4a

  Copy full SHA for 9899b4a

• fixing a typo

  SatyamMattoo

  committed
  b84b04b

  Copy full SHA for b84b04b

Commits on Jun 30, 2024

• Merge pull request #79 from SatyamMattoo/main

  Show description for 31be256
  SatyamMattoo

  authored
  31be256

  Copy full SHA for 31be256

Commits on Jul 4, 2024

• adding milvus as a vector database

  SatyamMattoo

  committed
  6176e48

  Copy full SHA for 6176e48

• adding new env var

  SatyamMattoo

  committed
  942390d

  Copy full SHA for 942390d

• adding new env var

  SatyamMattoo

  committed
  d503de9

  Copy full SHA for d503de9

Commits on Jul 10, 2024

• changed the embedding service to OpenAI

  SatyamMattoo

  committed
  2c25522

  Copy full SHA for 2c25522

Commits on Jul 14, 2024

• Embedding OpenFn docs from the repo

  SatyamMattoo

  committed
  63bd5a2

  Copy full SHA for 63bd5a2

• fixing a typo

  SatyamMattoo

  committed
  2bc8ca0

  Copy full SHA for 2bc8ca0

Commits on Jul 18, 2024

• added a better text splitting strategy to avoid uneven chunks of data

  SatyamMattoo

  committed
  c341639

  Copy full SHA for c341639

Commits on Jul 20, 2024

• embed docs before database connection

  SatyamMattoo

  committed
  55bc7c0

  Copy full SHA for 55bc7c0

Commits on Jul 21, 2024

• changing chunking strategy, size and overlap after testing (this seems to return better search results).

  SatyamMattoo

  committed
  0132654

  Copy full SHA for 0132654

Commits on Jul 24, 2024

• Merge pull request #91 from OpenFn/dmp/620

  Show description for 55ef21f
  SatyamMattoo

  authored
  55ef21f

  Copy full SHA for 55ef21f

• Merge branch 'main' of https://github.com/OpenFn/apollo into milvus-integration

  SatyamMattoo

  committed
  c7d42e4

  Copy full SHA for c7d42e4

Commits on Jul 26, 2024

• integrating gen_job and search service.

  SatyamMattoo

  committed
  49b7df3

  Copy full SHA for 49b7df3

Commits on Jul 27, 2024

• added the adaptor description service to add more context to the prompt and some minor changes to improve the search service.

  SatyamMattoo

  committed
  146f8bd

  Copy full SHA for 146f8bd

Commits on Jul 28, 2024

• adding Readme for job generation service and removing adaptor docs from the embeddings as we can use describe_adaptor service for retreiving adaptor info.

  SatyamMattoo

  committed
  b7b8ec1

  Copy full SHA for b7b8ec1

Commits on Aug 8, 2024

• improving prompts and removing adaptor docs

  SatyamMattoo

  committed
  87ab196

  Copy full SHA for 87ab196

• Remove gen_job folder for search_docs focus

  SatyamMattoo

  committed
  ba63683

  Copy full SHA for ba63683

• Add gen_job folder back to gen_job branch

  SatyamMattoo

  committed
  9aa0a61

  Copy full SHA for 9aa0a61

• improved the visualization of doc spliting by writing the splits to the disk

  SatyamMattoo

  committed
  efd2506

  Copy full SHA for efd2506

Commits on Aug 9, 2024

• collection name can be added to args, logs embeddings generated per file, updated readme

  SatyamMattoo

  committed
  ee221b8

  Copy full SHA for ee221b8

• removing unnecessary commands from dockerfile

  SatyamMattoo

  committed
  1db8e33

  Copy full SHA for 1db8e33

Commits on Aug 11, 2024

• Merge branch 'milvus-integration' of https://github.com/OpenFn/apollo into gen_job

  SatyamMattoo

  committed
  9a05f2a

  Copy full SHA for 9a05f2a

• adding own AI client, improving the prompt for job generation and RAG

  SatyamMattoo

  committed
  9b872c4

  Copy full SHA for 9b872c4

• minor prompt changes

  SatyamMattoo

  committed
  1e596f0

  Copy full SHA for 1e596f0

Commits on Aug 12, 2024

• adding an input parameter for optional embeddings.

  SatyamMattoo

  committed
  887541a

  Copy full SHA for 887541a

• changing search parameters

  SatyamMattoo

  committed
  03aeb0f

  Copy full SHA for 03aeb0f

• Merge branch 'milvus-integration' of https://github.com/OpenFn/apollo into gen_job

  SatyamMattoo

  committed
  3b7d66f

  Copy full SHA for 3b7d66f

• writing the context retrieved and adaptor description to a file.

  SatyamMattoo

  committed
  f20ee94

  Copy full SHA for f20ee94

• deleting temp file.

  SatyamMattoo

  committed
  cdc8dee

  Copy full SHA for cdc8dee

Commits on Aug 13, 2024

• fixing some prompt errors

  SatyamMattoo

  committed
  b9a148a

  Copy full SHA for b9a148a

Commits on Aug 14, 2024

• added a script to test multiple inputs with one command

  SatyamMattoo

  committed
  06509a3

  Copy full SHA for 06509a3

• changing search parameters

  SatyamMattoo

  committed
  e24d148

  Copy full SHA for e24d148

• Merge branch 'milvus-integration' of https://github.com/OpenFn/apollo into gen_job

  SatyamMattoo

  committed
  e0c28fb

  Copy full SHA for e0c28fb

Commits on Aug 31, 2024

• added semantic search

  SatyamMattoo

  committed
  26ed1a7

  Copy full SHA for 26ed1a7

• better handling of paths to be embedded

  SatyamMattoo

  committed
  3d2d1ec

  Copy full SHA for 3d2d1ec

• Merge branch 'milvus-integration' of https://github.com/OpenFn/apollo into gen_job

  SatyamMattoo

  committed
  d94d7df

  Copy full SHA for d94d7df

Commits on Sep 2, 2024

• added some gen_job results along with a input file

  SatyamMattoo

  committed
  2b7840f

  Copy full SHA for 2b7840f

• fixing a typo

  SatyamMattoo

  committed
  b4089d8

  Copy full SHA for b4089d8

Commits on Sep 3, 2024

• updated readme with instructions for RAG and job processor

  SatyamMattoo

  committed
  adaa28d

  Copy full SHA for adaa28d

Commits on Sep 4, 2024

• improving the readme

  SatyamMattoo

  committed
  16ed202

  Copy full SHA for 16ed202

• added more customization to search service and a feature to summarize the docs retirieved

  SatyamMattoo

  committed
  28fe149

  Copy full SHA for 28fe149

• updated the Readme

  SatyamMattoo

  committed
  a4ba665

  Copy full SHA for a4ba665

• using markdown and recursive text spilters

  SatyamMattoo

  committed
  79f471d

  Copy full SHA for 79f471d

• added path.config and updated readme

  SatyamMattoo

  committed
  7d5ac5b

  Copy full SHA for 7d5ac5b

Commits on Sep 5, 2024

• merging changes

  SatyamMattoo

  committed
  b7787e8

  Copy full SHA for b7787e8

• adding range search

  SatyamMattoo

  committed
  cee0ee9

  Copy full SHA for cee0ee9

• Merge branch 'milvus-integration' of https://github.com/OpenFn/apollo into gen_job

  SatyamMattoo

  committed
  7c8dae7

  Copy full SHA for 7c8dae7

• updating results

  SatyamMattoo

  committed
  4573c2e

  Copy full SHA for 4573c2e

Commits on Sep 6, 2024

• adding instructions to outputs

  SatyamMattoo

  committed
  eb183c2

  Copy full SHA for eb183c2

• fixing a limit error

  SatyamMattoo

  committed
  6ef0f07

  Copy full SHA for 6ef0f07

• Merge branch 'milvus-integration' of https://github.com/OpenFn/apollo into gen_job

  SatyamMattoo

  committed
  c54114d

  Copy full SHA for c54114d

• added some search results with the score

  SatyamMattoo

  committed
  a8b4ee0

  Copy full SHA for a8b4ee0

Commits on Sep 7, 2024

• added adaptor doc embeddings and improved the code quality

  SatyamMattoo

  committed
  bb18e40

  Copy full SHA for bb18e40

• fixing typos

  SatyamMattoo

  committed
  6561dac

  Copy full SHA for 6561dac

• adding more validation and improving the readme

  SatyamMattoo

  committed
  4a578ad

  Copy full SHA for 4a578ad

Commits on Sep 8, 2024

• Merge branch 'milvus-integration' of https://github.com/OpenFn/apollo into gen_job

  SatyamMattoo

  committed
  ee880a1

  Copy full SHA for ee880a1

• improving prompt and updating results

  SatyamMattoo

  committed
  0f99532

  Copy full SHA for 0f99532

• resolved minor errors

  SatyamMattoo

  committed
  54a269f

  Copy full SHA for 54a269f

• merging milvus-integration

  SatyamMattoo

  committed
  e651fd4

  Copy full SHA for e651fd4

• fixing typo

  SatyamMattoo

  committed
  ab3c7ba

  Copy full SHA for ab3c7ba

• Merge branch 'milvus-integration' of https://github.com/OpenFn/apollo into gen_job

  SatyamMattoo

  committed
  49275bc

  Copy full SHA for 49275bc

• fixing a typo

  SatyamMattoo

  committed
  682b330

  Copy full SHA for 682b330

Commits on Sep 15, 2024

• adding more parameters to the search

  SatyamMattoo

  committed
  7dddd8e

  Copy full SHA for 7dddd8e

Commits on Sep 19, 2024

• changeset

  josephjclark

  committed
  b83c1e6

  Copy full SHA for b83c1e6

• update readme

  josephjclark

  committed
  d6a9cde

  Copy full SHA for d6a9cde

• Merge branch 'release/next' into gen_job

  josephjclark

  committed
  f493301

  Copy full SHA for f493301