feat: Added llama stack-langchain integration example scripts #3211
Conversation
omaryashraf5
requested review from
ashwinb,
yanxi0830,
hardikjshah,
raghotham,
ehhuang,
terrytangyuan,
leseb,
bbrowning,
reluctantfuturist,
mattf and
slekkala1
as code owners
meta-cla
bot
added
the
CLA Signed
omaryashraf5
changed the title
| @@ -0,0 +1,290 @@ | |||
| import os | |||
There was a problem hiding this comment.
nit: please use langchain small case. why camelCase it?
There was a problem hiding this comment.
also: ghostty-org/ghostty#8289
docs/notebooks/langChain/README.md
Outdated
| This repository contains two different implementations of document processing using LangChain and Llama Stack: | ||
|
|
||
| 1. **`langchain_llamastack.py`** - Interactive CLI version | ||
| 2. **`langchain_llamastack_ray.py`** - Ray Serve API version |
There was a problem hiding this comment.
can you tell the motivation behind providing this? genuinely curious as to what folks are looking for and gain with the Ray example.
There was a problem hiding this comment.
Thanks, @ashwinb for your comment. We looked at serving layer solutions and how langchain applications get served and we wanted to show an example with a serving layer solution like RayServe which is commonly used by langchain developers. I could remove the rayserve script (part 2) because making the source code compatible with ray should be straight forward.
There was a problem hiding this comment.
@ashwinb removed the ray script from this PR.
docs/notebooks/langChain/README.md
Outdated
|
|
||
| --- | ||
|
|
||
| ## 🌐 Option 2: Ray Serve API Version (`langchain_llamastack_ray.py`) |
There was a problem hiding this comment.
Is this needed?
Option 1 seems to be good for Langchain prompt template and Llamastack integration
There was a problem hiding this comment.
@slekkala1 Thank you for your comment. Answered your question above.
Removed ray example
Updated the README and the name of the python script to langchain-llama-stack
Added fixes to the README file.
removed some dead code
Changing the langchain dir name
Changed the langChain dir name to langchain
omaryashraf5
changed the title
| @@ -0,0 +1,288 @@ | |||
| import os | |||
There was a problem hiding this comment.
hm this is not a notebook I guess
There was a problem hiding this comment.
@ashwinb Its a script but it can be converted to a Jupyter notebook.
There was a problem hiding this comment.
this still creates its own LLM and uses a deprecated api. please collaborate with @slekkala1
@mattf now using langchain-openai |
it's not. need to push an update? also, users aren't likely to have |
@mattf the change wasn't pushed for conflicts. Let me change that asap. |
docs/notebooks/langchain/README.md
Outdated
| source llama-env-py312/bin/activate | ||
|
|
||
| # Run the interactive CLI | ||
| cd /home/omara/langchain_llamastack |
There was a problem hiding this comment.
users aren't likely to have /home/omara, though maybe we should 😄
docs/notebooks/langchain/README.md
Outdated
| - Llama Stack server running on `http://localhost:8321/` | ||
| - Ollama or compatible model server | ||
|
|
||
| ### Required Python Packages |
There was a problem hiding this comment.
skip this, it's duplicated in the setup
| Before running either version, ensure your Llama Stack server is running: | ||
| ```bash | ||
| # Start Llama Stack server (example) | ||
| llama stack run your-config --port 8321 |
There was a problem hiding this comment.
what does the config need to include, at least inference=...?
it looks like specific models need to be setup too
There was a problem hiding this comment.
@mattf yes, at least inference but there are instructions on how to start a llama stack server instance. We could add a link to these instructions.
docs/notebooks/langchain/README.md
Outdated
| ### How to Run | ||
| ```bash | ||
| # Activate environment | ||
| source llama-env-py312/bin/activate |
docs/notebooks/langchain/README.md
Outdated
| source llama-env-py312/bin/activate | ||
|
|
||
| # Run the interactive CLI | ||
| cd /langchain_llamastack |
|
|
||
| # Run the interactive CLI | ||
| cd /langchain_llamastack | ||
| python langchain-llama-stack.py |
There was a problem hiding this comment.
it was missing langchain_openai
There was a problem hiding this comment.
@mattf After adding langchain_openai, the script ran successfully.
| @@ -212,7 +212,7 @@ def main(): | |||
| ) | |||
| os.environ["OPENAI_API_KEY"] = "dummy" | |||
| os.environ["OPENAI_BASE_URL"] = "http://0.0.0.0:8321/v1/openai/v1" | |||
| llm = ChatOpenAI(model="ollama/llama3:70b-instruct") | |||
| llm = ChatOpenAI(model="ollama/llama3:70b-instruct", base_url="http://localhost:8321/v1/openai/v1") | |||
There was a problem hiding this comment.
why pass base_url and set the env var?
imho, skip the environ
What does this PR do?
It contains a python script for running llama stack with langchain locally in addition to a README file that explains how to run the script.
The script shows how to build content summarizer with llama stack and langChain.
Test Plan
Tested these scripts on the dev server.
An AI tool was used to optimize code, and create or improve function descriptions and comments in this pull request.