Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MultimodalQnA Image and Audio Support Phase 1 #1071

Merged
merged 58 commits into from
Nov 8, 2024
Merged
Changes from 2 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
97f4bc0
Initial implementation of image ingestion
mhbuehler Oct 14, 2024
08956e5
Added ability to change LVM Model (#1)
okhleif-IL Oct 21, 2024
56bd8c3
Use same endpoint for image and video ingestion
mhbuehler Oct 21, 2024
b334dc2
Update tests and docs
mhbuehler Oct 21, 2024
13c752a
Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…
dmsuehir Oct 22, 2024
5f4cf29
Renamed Dataprep Endpoints videos --> files (#3)
okhleif-IL Oct 23, 2024
271117e
added LVM_MODEL_ID var to test file (#4)
okhleif-IL Oct 23, 2024
c4e0259
Updates tests per feedback
mhbuehler Oct 24, 2024
a541140
Merge pull request #2 from mhbuehler/melanie/combined_image_video_ing…
mhbuehler Oct 24, 2024
30c311d
Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…
dmsuehir Oct 24, 2024
e83fc44
Update LVM model for Xeon
dmsuehir Oct 25, 2024
4b8a5ad
Merge pull request #5 from mhbuehler/dina/lvm_model
mhbuehler Oct 25, 2024
a6d826c
Initial setup for ingest_with_text
mhbuehler Oct 24, 2024
547a139
Write and send custom caption file
mhbuehler Oct 25, 2024
7cfc343
Update docs and tests
mhbuehler Oct 28, 2024
69dbdfc
MMQnA doc updates for audio ingestion (#7)
dmsuehir Oct 29, 2024
834c668
Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…
dmsuehir Oct 29, 2024
a8f8dc9
Fix UI request for follow up queries with no image (#8)
dmsuehir Oct 30, 2024
718d02e
Updated for review suggestions
mhbuehler Oct 30, 2024
d05cfb3
Merge branch 'melanie/mm-rag-enhanced' into melanie/images_and_text
mhbuehler Oct 30, 2024
72591c1
Add audio upload functionality to UI
mhbuehler Oct 30, 2024
431e41b
Merge pull request #6 from mhbuehler/melanie/images_and_text
mhbuehler Oct 30, 2024
d535aa7
Merge branch 'melanie/mm-rag-enhanced' into melanie/audio_ingest_ui
mhbuehler Oct 31, 2024
39f43fc
Minor refactor, improve display text, and suppress PDF tab
mhbuehler Oct 31, 2024
afc3c8a
Merge pull request #9 from mhbuehler/melanie/audio_ingest_ui
mhbuehler Oct 31, 2024
ddd5dfb
Small fixes
mhbuehler Nov 1, 2024
426c739
Improve appearance
mhbuehler Nov 1, 2024
cfa1c8c
Improve upload errors and revert multimodal query box
mhbuehler Nov 2, 2024
cdec83f
Small text edit as suggested
mhbuehler Nov 4, 2024
48baceb
Merge pull request #11 from mhbuehler/melanie/mm-fixes
mhbuehler Nov 4, 2024
170b723
Merge branch 'main' of github.com:mhbuehler/GenAIExamples into melani…
dmsuehir Nov 4, 2024
3a23e5b
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 5, 2024
65a8afd
Merge branch 'main' into melanie/mm-rag-enhanced
ashahba Nov 5, 2024
f95b946
updated readmes with MMQA info
okhleif-IL Nov 5, 2024
c4d5138
removed stray char
okhleif-IL Nov 5, 2024
a1350c5
Fixed header
okhleif-IL Nov 5, 2024
c7aadd2
addressed review comments
okhleif-IL Nov 5, 2024
7288faa
removed video
okhleif-IL Nov 5, 2024
e108ee9
Merge branch 'main' into melanie/mm-rag-enhanced
ashahba Nov 6, 2024
9fdd6fe
Reorder new lvm-dependent tests and fix clear textbox
mhbuehler Nov 6, 2024
ee387a2
Merge pull request #13 from mhbuehler/omar/mmqa-docs
mhbuehler Nov 6, 2024
aafcfe1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 6, 2024
359b6f8
Merge pull request #14 from mhbuehler/melanie/fix_tests
mhbuehler Nov 6, 2024
9d3ed45
fixed multimodalqna typos
okhleif-IL Nov 6, 2024
54cff40
Point git clone at specific branch of GenAIComps
mhbuehler Nov 6, 2024
fd9fd84
Merge pull request #15 from mhbuehler/omar/dockimg_doc
mhbuehler Nov 6, 2024
d88513a
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 6, 2024
24438e5
Merge pull request #16 from mhbuehler/melanie/clone_specific_branch
mhbuehler Nov 6, 2024
df2511b
Fix xeon test to use llava-hf/llava-1.5-7b-hf (#17)
dmsuehir Nov 6, 2024
6631601
Update MMQnA xeon test to wait for LVM to be ready (#18)
dmsuehir Nov 7, 2024
59acc77
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 7, 2024
9d4cb5f
Test update: Increase wait time and add more messages (#19)
dmsuehir Nov 7, 2024
c1bfe6a
Test update: Increase sleep
dmsuehir Nov 7, 2024
b88aa3f
Merge pull request #20 from mhbuehler/dina/test-update-wait
mhbuehler Nov 7, 2024
6e5a200
Merge branch 'main' into melanie/mm-rag-enhanced
mhbuehler Nov 7, 2024
e2c585e
Revert change of repo for tests
mhbuehler Nov 8, 2024
45c7cf9
Merge pull request #21 from mhbuehler/melanie/revert_repo_change
mhbuehler Nov 8, 2024
04fede6
Merge branch 'main' into melanie/mm-rag-enhanced
mhbuehler Nov 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docker_images_list.md
Original file line number Diff line number Diff line change
@@ -26,8 +26,8 @@ Take ChatQnA for example. ChatQnA is a chatbot application service based on the
| [opea/faqgen](https://hub.docker.com/r/opea/faqgen) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/FaqGen/Dockerfile) | The docker image served as a faqgen gateway and automatically generating comprehensive, natural sounding Frequently Asked Questions (FAQs) from documents, legal texts, customer inquiries and other sources. |
| [opea/faqgen-ui](https://hub.docker.com/r/opea/faqgen-ui) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/FaqGen/ui/docker/Dockerfile) | The docker image serves as the docsum UI entry point for easy interaction with users, generating FAQs by pasting in question text. |
| [opea/faqgen-react-ui](https://hub.docker.com/r/opea/faqgen-react-ui) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/FaqGen/ui/docker/Dockerfile.react) | The purpose of the docker image is to provide a user interface for Generate FAQs using React. It allows generating FAQs by uploading files or pasting text. |
| [opea/multimodalqna](https://hub.docker.com/r/opea/multimodalqna) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/MultimodalQnA/Dockerfile) | The docker image served as a multimodalqna gateway and dynamically fetches the most relevant multimodal information (frames, transcripts, and/or subtitles) from the user's video collection to solve the problem. |
| [opea/multimodalqna-ui](https://hub.docker.com/r/opea/multimodalqna-ui) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/MultimodalQnA/ui/docker/Dockerfile) | The docker image serves as the docsum UI entry point for easy interaction with users. Answers to questions are generated from videos uploaded by users.. |
| [opea/multimodalqna](https://hub.docker.com/r/opea/multimodalqna) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/MultimodalQnA/Dockerfile) | The docker image served as a multimodalqna gateway and dynamically fetches the most relevant multimodal information (frames, transcripts, and/or subtitles) from the user's video, image, or audio collection to solve the problem. |
| [opea/multimodalqna-ui](https://hub.docker.com/r/opea/multimodalqna-ui) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/MultimodalQnA/ui/docker/Dockerfile) | The docker image serves as the multimodalqna UI entry point for easy interaction with users. Answers to questions are generated from uploaded by users. |
| [opea/productivity-suite-react-ui-server](https://hub.docker.com/r/opea/productivity-suite-react-ui-server) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/ProductivitySuite/ui/docker/Dockerfile.react) | The purpose of the docker image is to provide a user interface for Productivity Suite Application using React. It allows interaction by uploading documents and inputs. |
| [opea/searchqna](https://hub.docker.com/r/opea/searchqna/tags) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/SearchQnA/Dockerfile) | The docker image served as the searchqna gateway to provide service of retrieving accurate and relevant answers to user queries from a knowledge base or dataset |
| [opea/searchqna-ui](https://hub.docker.com/r/opea/searchqna-ui) | [Link](https://github.com/opea-project/GenAIExamples/blob/main/SearchQnA/ui/docker/Dockerfile) | The docker image acted as the searchqna UI entry for facilitating interaction with users for question answering |