Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Search][Recommendation] Introduce Kafka Stream to aggregate all product data. #1149

Open
duylv27 opened this issue Oct 9, 2024 · 0 comments
Assignees

Comments

@duylv27
Copy link
Contributor

duylv27 commented Oct 9, 2024

Description:

The search and recommendation modules listen to the Product CDC topic (created by Debezium) to feed data into Elasticsearch and VectorDB. However, this topic only produces events when there are changes to the product table. It does not generate events when associated tables (such as category, brand, product_attribute, etc.) are updated, which can lead to incomplete or outdated data in Elasticsearch and VectorDB.

Therefore, we need to introduce Kafka Streams to aggregate event changes from all associated tables (such as category, brand, product_attribute, etc.) along with the product table.

PhuocNguyenBa pushed a commit that referenced this issue Oct 16, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 16, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 16, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 16, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 18, 2024
…all product data.

-temp class for kafka test
PhuocNguyenBa pushed a commit that referenced this issue Oct 24, 2024
…all product data.

-temp class for kafka test
PhuocNguyenBa pushed a commit that referenced this issue Oct 24, 2024
PhuocNguyenBa pushed a commit that referenced this issue Oct 24, 2024
PhuocNguyenBa pushed a commit that referenced this issue Oct 24, 2024
PhuocNguyenBa pushed a commit that referenced this issue Oct 24, 2024
PhuocNguyenBa pushed a commit that referenced this issue Oct 25, 2024
PhuocNguyenBa pushed a commit that referenced this issue Oct 25, 2024
PhuocNguyenBa pushed a commit that referenced this issue Oct 25, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 25, 2024
PhuocNguyenBa pushed a commit that referenced this issue Oct 25, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 25, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 28, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 28, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 28, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 28, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 28, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 28, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 28, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 29, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 29, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 29, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
PhuocNguyenBa pushed a commit that referenced this issue Oct 29, 2024
…all product data.

-implement kafka stream application to capture all changes related to product
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants