feat(inventory): add Open Food Facts cache with Deno ingest script #5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: owner <[email protected]>
Co-authored-by: owner <[email protected]>
…; remove estimation artifacts
…nore - Move off_ingest.ts and off_upload_batch.ts to local/openfoodfacts/ folder - Add off_inventory_cache.jsonl to .gitignore (large generated file) - Fix parallel upload logic to prevent memory exhaustion - Add deduplication to handle duplicate barcodes in batches - Successfully uploaded 3M+ products to Supabase inventory_cache
- Removed log_inventory table definition from tables.sql - Updated get_check_history and get_list_items functions to use inventory_cache - Removed background/log_inventory endpoint - Created consolidated getProductFromCache() function as single source of truth - Updated all code to query inventory_cache instead of log_inventory - Removed redundant lookupProduct wrapper function - inventory.ts now checks cache first, falls back to fresh API fetch
- Removed fetchProduct() function and OpenFoodFacts API integration - Removed processOpenFoodFactsProductData() and helper functions - Simplified get() endpoint to only read from inventory_cache - Reduced file from 252 lines to 104 lines (67% reduction) - Now returns 404 if product not found in cache
- Add barcodeToPath() function to convert barcodes to proper OFF path format (e.g., 088/491/237/3946)
- Fix image URL format: full size uses imgId.jpg instead of imgId.WxH.jpg
- Add validateImageUrlExists() function for optional HEAD request validation (currently disabled)
- Update extractDisplayImageUrls() to use correct barcode paths in URLs
- All image URLs now follow format: /images/products/{barcodePath}/{imgId}.{size}.jpg
- Only upsert barcode and images fields during upload - Remove updated_at field from upserts (column may not exist) - Add ignoreDuplicates: false to ensure existing rows are updated - Improve error logging with JSON.stringify for better debugging - This is a temporary change to fix image URLs without reprocessing all data
- Add barcode_matches() SQL function that pads upward only to avoid false matches - Update inventory query to try multiple barcode format variants (EAN-8, UPC-A, EAN-13, ITF-14) - Apply barcode_matches() to JOINs in get_check_history() and get_list_items() - Prevents 8-digit barcodes from matching unrelated 13-digit barcodes - Fixes case where barcode 884912373946 should match 0884912373946
- Remove temporary image-only update logic - Restore full batch upserts with all product data - Will populate fresh inventory_cache table with: - Fixed image URLs with correct barcode paths - All product metadata (name, brand, ingredients, etc.) - Proper last_refreshed_at timestamps
- Change ImageMetadata type to minimal structure (type, language, imgid, sizes[]) - Extract all image types: front, ingredients, nutrition, packaging - Extract images for all languages (en, fr, de, es, it, pt, nl, pl, ru, ja, zh, etc.) - Remove URL construction logic (moved to runtime in inventory.ts) - Remove unused helper functions: isValidImageUrl, validateImageUrlExists, barcodeToPath - Add comprehensive documentation about image storage strategy - Store only language keys, drop numeric keys to reduce storage by ~60% Image URLs will be constructed at runtime with smart selection: - Prefer English, fallback to other languages - Prefer 400px (medium), fallback to next available size - Return one image per type (front, ingredients, nutrition, packaging)
justanotheratom
force-pushed
the
main
branch
from
5b105b7 to
a67ddc7
Compare
justanotheratom
force-pushed
the
main
branch
5 times, most recently
from
cba0124 to
0f645c6
Compare
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a caching layer for Open Food Facts product data to improve barcode lookup performance and reliability.
Changes
Usage
Benefits
Database Impact