fix: lazy import integrations to avoid pyspark bundling error and add deep research - #1
Merged
Merged
Conversation
The CLI binary was failing with "No such file or directory: error-conditions.json" because pyspark was being eagerly imported when any integration was accessed. Changes: - Make integrations/__init__.py use lazy imports via __getattr__ - Update PyInstaller spec to use correct integration paths - Remove library-only integrations (polars, duckdb, spark) from bundle - Only bundle integrations with deploy capability (bigquery, snowflake)
Add `parallel-cli research` command group for running deep research tasks: - `research run` - Create and run research tasks with polling - `research status` - Check task status - `research poll` - Resume polling existing tasks - `research processors` - List available processor tiers Also includes: - New core/research.py module with research API functions - Comprehensive tests with mocking (36 tests) - Refactored shared polling logic to reduce duplication - Added write_json_output() helper for consistent output handling
NormallyGaussian
changed the title
NormallyGaussian
marked this pull request as ready for review
January 23, 2026 20:52
Add assertions after validation to narrow optional types to non-None, fixing 10 pyrefly bad-argument-type errors in enrich_run and enrich_plan.
- Add explicit urllib.error import in auth.py - Use SDK BetaRunInputParam type in batch.py and spark/streaming.py - Add explicit type annotations to fix type inference issues - Use row.asDict() for proper PySpark Row field access - Add type narrowing assertion in test_cli.py - Exclude non-core files from pyrefly (cloud_function, examples, notebooks, scripts) - Add pyrefly pre-commit hook (local, uses uv run)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bug Fix
No such file or directory: error-conditions.jsonwhen runningparallel-cli enrich deploy --system bigqueryintegrations/__init__.pyuse lazy imports via__getattr__to avoid loading pyspark when not neededNew Features
-o, --outputoption tosearchandextractcommands to save results to a JSON fileparallel-cli researchcommand group for deep research tasks:research run- Create and run research tasks with configurable processors and pollingresearch status- Check task status by run IDresearch poll- Resume polling existing tasksresearch processors- List available processor tiers (pro-fast through ultra8x)Code Quality
core/research.pymodule with deep research API functionswrite_json_output()helper for consistent output handlingTest plan
uv run python scripts/build.py --skip-depsparallel-cli enrich deploy --system bigquery --project=test --api-key=testno longer fails with pyspark errorparallel-cli search --helpandparallel-cli extract --helpshow new--outputoptionparallel-cli research --helpshows new research commandsparallel-cli research processorslists all processor tiers