Skip to content

fix: nightly hardening - config input bounds validation#17

Open
mouse-value-add wants to merge 1 commit intobrainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260503-config-bounds-validation
Open

fix: nightly hardening - config input bounds validation#17
mouse-value-add wants to merge 1 commit intobrainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260503-config-bounds-validation

Conversation

@mouse-value-add
Copy link
Copy Markdown
Contributor

Problem\nPromptLens accepted out-of-range runtime/model config values (for example temperature > 2, max_tokens <= 0, empty output formats). Those invalid inputs could pass parsing and fail later during execution with less actionable errors.\n\n## Approach\n- Added pydantic Field constraints to core config models:\n - constrained to \n - constrained to \n - request/timeout/retry execution values constrained to safe non-negative/positive bounds\n - output constrained to non-empty list\n- Added focused tests that verify invalid values now raise .\n\n## Verification\n- Ran: ============================= test session starts ==============================

platform darwin -- Python 3.9.6, pytest-8.4.2, pluggy-1.6.0
rootdir: /Users/mouse/.openclaw/workspace/nightly/PromptLens
configfile: pyproject.toml
testpaths: tests
plugins: anyio-4.12.1, asyncio-1.2.0, cov-7.1.0
asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 12 items

tests/test_cli_hardening.py ... [ 25%]
tests/test_config_validation_hardening.py ... [ 50%]
tests/test_http_provider_response_parsing.py .... [ 83%]
tests/test_loader_top_level_validation.py .. [100%]

=============================== warnings summary ===============================
promptlens/models/tools.py:14
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/tools.py:14: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class ToolParameter(BaseModel):

promptlens/models/test_case.py:10
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/test_case.py:10: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class TestCase(BaseModel):

promptlens/models/test_case.py:66
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/test_case.py:66: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class GoldenSet(BaseModel):

promptlens/models/config.py:100
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/config.py:100: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class RunConfig(BaseModel):

../../../../Library/Python/3.9/lib/python/site-packages/google/api_core/_python_version_support.py:242
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/api_core/_python_version_support.py:242: FutureWarning: You are using a non-supported Python version (3.9.6). Google will not post any further updates to google.api_core supporting this Python version. Please upgrade to the latest Python version, or at least Python 3.10, and then update google.api_core.
warnings.warn(message, FutureWarning)

../../../../Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35
/Users/mouse/Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: urllib3/urllib3#3020
warnings.warn(

../../../../Library/Python/3.9/lib/python/site-packages/google/auth/init.py:54
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/auth/init.py:54: FutureWarning: You are using a Python version 3.9 past its end of life. Google will update google-auth with critical bug fixes on a best-effort basis, but not with any other fixes or features. Please upgrade your Python version, and then update google-auth.
warnings.warn(eol_message.format("3.9"), FutureWarning)

../../../../Library/Python/3.9/lib/python/site-packages/google/oauth2/init.py:40
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/oauth2/init.py:40: FutureWarning: You are using a Python version 3.9 past its end of life. Google will update google-auth with critical bug fixes on a best-effort basis, but not with any other fixes or features. Please upgrade your Python version, and then update google-auth.
warnings.warn(eol_message.format("3.9"), FutureWarning)

promptlens/providers/google.py:8
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/providers/google.py:8: FutureWarning:

All support for the google.generativeai package has ended. It will no longer be receiving
updates or bug fixes. Please switch to the google.genai package as soon as possible.
See README for more details:

https://github.com/google-gemini/deprecated-generative-ai-python/blob/main/README.md

import google.generativeai as genai

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform darwin, python 3.9.6-final-0 ________________

Name Stmts Miss Cover

promptlens/init.py 3 0 100%
promptlens/main.py 3 3 0%
promptlens/cli.py 169 120 29%
promptlens/exporters/init.py 6 0 100%
promptlens/exporters/base.py 15 5 67%
promptlens/exporters/csv_exporter.py 26 14 46%
promptlens/exporters/html_exporter.py 38 25 34%
promptlens/exporters/json_exporter.py 16 6 62%
promptlens/exporters/markdown_exporter.py 77 66 14%
promptlens/judges/init.py 3 0 100%
promptlens/judges/base.py 15 3 80%
promptlens/judges/llm_judge.py 71 50 30%
promptlens/judges/parser.py 107 97 9%
promptlens/judges/prompts.py 30 25 17%
promptlens/loaders/init.py 4 0 100%
promptlens/loaders/base.py 14 3 79%
promptlens/loaders/json_loader.py 23 8 65%
promptlens/loaders/yaml_loader.py 34 17 50%
promptlens/models/init.py 4 0 100%
promptlens/models/config.py 41 0 100%
promptlens/models/result.py 60 13 78%
promptlens/models/test_case.py 25 0 100%
promptlens/models/tools.py 74 34 54%
promptlens/providers/init.py 3 0 100%
promptlens/providers/anthropic.py 52 32 38%
promptlens/providers/base.py 23 6 74%
promptlens/providers/factory.py 21 10 52%
promptlens/providers/google.py 47 28 40%
promptlens/providers/http.py 67 26 61%
promptlens/providers/openai.py 57 36 37%
promptlens/providers/you.py 57 39 32%
promptlens/runners/init.py 2 0 100%
promptlens/runners/runner.py 96 74 23%
promptlens/utils/init.py 1 0 100%
promptlens/utils/cost.py 15 11 27%
promptlens/utils/diff.py 25 25 0%
promptlens/utils/retry.py 21 15 29%
promptlens/utils/timing.py 24 13 46%

TOTAL 1369 804 41%
Coverage HTML written to dir htmlcov
======================== 12 passed, 9 warnings in 1.55s ========================\n- Result: all tests passing (12 passed).\n\n## Risks\n- Configs that previously relied on permissive/invalid values will now fail fast at parse time.\n- This is intentional hardening, but may require users to fix malformed configs.\n\n## Rollback plan\n- Revert this PR commit to restore previous permissive behavior if needed.\n- Alternatively, relax individual constraints in if a specific boundary proves too strict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant