chore: Remove API key usage in CI#39
Conversation
| @nox.session() | ||
| def test_latest_wrappers_novcr(session): | ||
| """Run the latest wrapper tests without vcrpy.""" | ||
| if os.environ.get("CI") or os.environ.get("GITHUB_ACTIONS"): |
There was a problem hiding this comment.
i think if we don't have this, we might want some way of asserting we eventually test reality. maybe if oldest VCR file is X days/weeks/months old, fail the build and force downloading new cassettes. open to whatever idea.
There was a problem hiding this comment.
I agree. let me think about this a bit.
There was a problem hiding this comment.
IMO, maybe we should have a nightly or weekly build that runs without VCR enabled. (Similar to one we might use to check for updated 3rd party package compatibility.)
| cd braintrust-sdk-python | ||
|
|
||
| # Create venv and install all dependencies | ||
| make develop |
There was a problem hiding this comment.
i think we use mise for this now. we should eliminate all of the old ways of creating venvs.
There was a problem hiding this comment.
I can go and clean this up after I merge the PR in. I'll also do a refactor of the readmes then.
There was a problem hiding this comment.
I opened two PRs as a follow up to this:
- meta: Use mise everywhere and clean up contributing docs #52 use mise everywhere for everything (and adjust CI workflows/makefiles accordingly)
- clean up SDK READMEs: chore: Clean up Python SDK READMEs #51
Abhijeet Prasad (AbhiPrasad)
force-pushed
the
abhi-no-api-keys-ci
branch
from
df0400a to
1f45b07
Compare
- Remove ANTHROPIC_API_KEY, OPENAI_API_KEY, and GEMINI_API_KEY secrets from all CI workflows — tests use VCR cassettes with record_mode="none" in CI, so real API keys are never needed - Add dummy ANTHROPIC_API_KEY fallback in conftest.py (OpenAI and Google already had one) to prevent Anthropic client instantiation failures - Skip test_latest_wrappers_novcr nox session in CI since it disables VCR and requires real keys - Add CONTRIBUTING.md with repo setup, structure, testing, and CI docs - Add docs/vcr-testing.md with detailed VCR testing guide covering local vs CI differences and contributor workflow
Abhijeet Prasad (AbhiPrasad)
force-pushed
the
abhi-no-api-keys-ci
branch
from
1f45b07 to
22e2e09
Compare
This reverts commit 22e2e09.
David Elner (delner)
left a comment
David Elner (delner)
left a comment
There was a problem hiding this comment.
Generally in favor of this (removing keys makes things more portable.)
Following on Matt's point, perhaps separate from this implementation, I do think its a good idea to have some kind of automated non-VCR build to check if/when we've drifted from real API calls. That would require some secrets in the env, but I assume that can easily be added for just that build without a lot of code changes here.
|
Going to merge this in to unblock a contributor PR, but I’ll follow up on the automated non-VCR check asap. |
3 participants
The
vcr-testing.mdandCONTRIBUTING.mdwere fully AI generated.