Skip to content

test: add comprehensive tests for scraper package#379

Open
mehallhm wants to merge 10 commits intomainfrom
feature/scraper-tests
Open

test: add comprehensive tests for scraper package#379
mehallhm wants to merge 10 commits intomainfrom
feature/scraper-tests

Conversation

@mehallhm
Copy link
Copy Markdown
Member

Summary

  • Add 318 unit tests across 20 test files for the @sneu/scraper package (previously had zero tests)
  • Tests cover all major modules: data transformation, HTML parsing, fetch infrastructure, schemas, scraping steps, and upload/update utilities
  • Uses Node.js native test runner (node:test) with nock for HTTP mocking

Test breakdown by commit:

Commit Module Tests
events + config + endpoints Core utilities 16
Banner schemas API response validation 68
marshall arrangeCourses, parseMeetingTimes 27
reqs parsePrereqs, parseCoreqs, populatePostReqs 23
scraper schemas Cache + static config schemas 87
sections + terms + campuses Scraping steps (direct fetch) 11
prereqs + coreqs steps Scraping steps (FetchEngine) 9
fetch infrastructure $fetch retry + FetchEngine class 25
upload/update utilities chunk, filterScrapeByPartOfTerm, sendNotifications 20
faculty + names + descriptions Scraping steps (FetchEngine) 24

Not tested (intentionally skipped):

  • generate/main.ts — orchestrator requiring full pipeline mock
  • upload/steps/*.ts — tightly coupled to drizzle DB transactions
  • update/main.ts — coupled to both scraper and DB queries
  • Trivial z.string() schemas (sectionCatalogDetails, sectionCoreqs, etc.)

Test plan

  • All 318 tests pass with node --test --import tsx

🤖 Generated with Claude Code

mehallhm and others added 10 commits March 26, 2026 19:50
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@vercel
Copy link
Copy Markdown

vercel bot commented Mar 26, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
searchneu Canceled Canceled Mar 26, 2026 11:52pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
searchneu-docs Skipped Skipped Mar 26, 2026 11:52pm

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant