Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add example on LLM offline inference #31

Merged
merged 19 commits into from
Mar 24, 2025
Merged

Add example on LLM offline inference #31

merged 19 commits into from
Mar 24, 2025

Conversation

JasonLo
Copy link
Contributor

@JasonLo JasonLo commented Dec 20, 2024

This example may be valuable for:

  • Generating large volumes of synthetic data with open-source LLMs.
  • Performing large-scale, structured data extraction from text.
  • Embedding extensive text datasets.
  • Cost-effectively executing LLM-driven tasks at scale without relying on expensive commercial solutions.

For more details, please refer to vllm_batch_inference/README.md.

@iross iross self-assigned this Mar 20, 2025
@iross iross self-requested a review March 21, 2025 14:49
Copy link
Contributor

@iross iross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for submitting this! Sorry it took so long to look at it 😂. Let me know what you think of the changes I mentioned. I wish the uv fix was less ugly.. Your suggestion of building it from source sounds painful, but at least then we wouldn't be stomping permissions all over /root/...

@agitter
Copy link
Contributor

agitter commented Mar 21, 2025

Thanks for the contribution @JasonLo!

JasonLo added 4 commits March 21, 2025 15:59
Since v0.8, vLLM has updated its environment management to use `uv`, which can make running the container in user mode slightly more challenging.
@JasonLo
Copy link
Contributor Author

JasonLo commented Mar 21, 2025

Thank you, @iross and @agitter, for your input. I have tested the changes mentioned above, and they seem to be working well now. I tested a job with job_id = 1455169.

@iross iross merged commit efd419a into CHTC:master Mar 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants