Skip to content

feat(inventory): add Open Food Facts cache with Deno ingest script #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
justanotheratom wants to merge 22 commits into
base: main
Choose a base branch
from
Open

feat(inventory): add Open Food Facts cache with Deno ingest script #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
e40f7ed
Add script to estimate OFF payload sizes
cursoragent Sep 28, 2025
a64fa1d
Add off_jsonl_linecount.txt with line count
cursoragent Sep 28, 2025
e213cf0
Add SSE inventory+analysis stream and shared helpers (#4)
justanotheratom Sep 30, 2025
10e72c2
feat(inventory): add Deno OFF ingest script and inventory_cache table…
justanotheratom Sep 30, 2025
62805fd
feat: add Deno OFF ingest script for inventory caching
justanotheratom Sep 30, 2025
028cbb1
chore: remove extraneous estimator files from PR
justanotheratom Sep 30, 2025
c48d7a2
fix: resolve merge conflict in tables.sql
justanotheratom Sep 30, 2025
97505a7
feat: add .env template and load env vars in off_ingest script
justanotheratom Sep 30, 2025
54622bf
refactor: use npm: specifier for standalone script (no import map nee…
justanotheratom Sep 30, 2025
a7ad823
feat: add batch upload with user confirmation every 5 batches
justanotheratom Sep 30, 2025
51b9068
feat: add progress indicators for download and processing
justanotheratom Sep 30, 2025
e414031
feat: add detailed validation statistics to track invalid/empty products
justanotheratom Sep 30, 2025
162c9e7
perf: optimize for speed with larger batches, parallel uploads, and b…
justanotheratom Sep 30, 2025
a2ac1b2
feat: organize OpenFoodFacts scripts and add inventory cache to gitig…
justanotheratom Oct 1, 2025
28c478d
Merge remote-tracking branch 'origin/main' into feature/inventory-cac…
justanotheratom Oct 1, 2025
5181712
Remove log_inventory table and replace with inventory_cache
justanotheratom Oct 1, 2025
111a945
Remove OpenFoodFacts API fallback, make inventory endpoint cache-only
justanotheratom Oct 1, 2025
d4487cf
Fix Open Food Facts image URL generation
justanotheratom Oct 1, 2025
4286619
TEMP: Modify upload to only update images column
justanotheratom Oct 1, 2025
2468a51
Handle barcode matching with or without leading zeros
justanotheratom Oct 1, 2025
3b418e9
Revert to full upserts with all columns
justanotheratom Oct 2, 2025
9ac3c8d
refactor: store raw image metadata instead of constructed URLs
justanotheratom Oct 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
local/finetuning/preferencevalidatordataset/.env
local/.env
local/temp/
76 changes: 76 additions & 0 deletions deno.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 12 additions & 0 deletions local/.env.template
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Supabase Configuration
# Copy this file to .env and fill in your actual values

# Your Supabase project URL
SUPABASE_URL=https://your-project-ref.supabase.co

# Your Supabase secret key (new key type, replaces SUPABASE_SERVICE_ROLE_KEY)
# Get this from your Supabase dashboard > Settings > API
SUPABASE_SECRET_KEY=your_secret_key_here

# Legacy fallback (if you haven't migrated to new keys yet)
# SUPABASE_SERVICE_ROLE_KEY=your_legacy_service_role_key_here
3 changes: 2 additions & 1 deletion local/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
supabase-service.json
datasets/
datasets/
off_inventory_cache.jsonl
Loading