add integration test#31
Open
Zhiyue-Wang wants to merge 23 commits into
Open
Conversation
…ession MemoryClient.get_space/list_spaces/update_space/delete_space referenced self._data_plane._region_name, which does not exist on _DataPlane (it only stores .client), so every control-plane Space call raised AttributeError. Use self.region_name instead, matching create_space. MemorySession.__repr__ referenced self.region_name, but MemorySession only stores self._region_name and exposes no region_name property (the async sibling AsyncMemorySession does), so print/repr raised AttributeError. Use self._region_name. Both were latent: unit tests mock at the method boundary and never exercise these attribute-resolution paths. Surfaced by the non-mock integration suite.
Real-API (no HTTP mocking) integration tests under tests/integration/, gated by environment variables with a three-tier safety model: - default (read-only): list/get + local AgentArtsRuntimeApp via TestClient; zero cloud writes, zero cost. Runs in CI without secrets (cloud tests auto-skip, local tests run). - ALLOW_CREATE=1: create->get->update->delete lifecycle for every resource type; session-scoped resource_registry drains deleters in reverse order so cleanup runs even on mid-test crash. - RUN_BILLABLE=1: code-interpreter sandbox session (via code_session ctx manager) and runtime invoke/exec; paired start/stop, opt-in only. Coverage: IdentityClient (workload identity + api-key/oauth2/sts credential providers, token issuance), MemoryClient + AsyncMemoryClient (space CRUD, session, messages, memories), MCPGatewayClient (gateway + target CRUD; xfail pending SDK trust_policy fix), RuntimeClient (control-plane agent/ endpoint CRUD; skipped pending artifact_source prereq, data-plane session), CodeInterpreter (control-plane CRUD + billable sandbox), AgentArtsRuntimeApp (/ping, /invocations JSON+SSE+error paths, /ws), and require_api_key / require_sts_token decorators end-to-end. All resource names are prefixed aa-it-<run_id> for greppable leak cleanup. Memory suite is self-contained (create_space mints its own data-plane key). Auth decorator bootstrap isolated to a temp dir so .agent_identity.json never pollutes the repo. Depends on the preceding fix(memory) commit (control-plane Space methods were unreachable before the region-attribute fix).
… ops Closes integration-suite coverage gaps (excluding MCP): - AsyncMemoryClient.create_memory_session + add_messages invoked directly in async mode (previously only the sync client was used for setup). - MemorySession + AsyncMemorySession wrapper classes exercised end-to-end (auto-create session, bound add/get/list). Depends on the preceding fix(memory) session-create commit. - CodeInterpreter billable session extended: execute_command, upload_file, download_file (round-trip), clear_context — all inside one code_session. - Runtime billable session extended: upload_files + download_files round-trip. The billable additions are RUN_BILLABLE-gated (skip otherwise). Verified the memory additions against real cloud (ALLOW_CREATE); billable additions are signature-checked and skip-gated pending RUN_BILLABLE + pre-provisioned resources.
Update the Coverage section to reflect the current 69-test suite: async create/add_messages + session wrappers, billable CodeInterpreter/ Runtime file-op extensions, refreshed per-method tables, tier matrix, SDK-bugs-found table, and remaining gaps. Correct the MCP xfail reason (409-masking on accounts where the shared agency already exists).
…re-existing agency
Cover the toolkit (agentarts CLI) layer that the SDK-only suite did not. All invoke the real Typer app end-to-end (no mocking of operations/clients): - Local (no creds, CI-safe): init (4 templates + path/invalid), config CRUD (add/set/get/set-env/list-env/remove-env/set-default/remove), dev (subprocess uvicorn -> /ping + /invocations), --version/--help. - Cloud (ALLOW_CREATE): memory create/list/get/update/delete via subprocess with --output json parsing; mcp-gateway list (read-only). - Billable (RUN_BILLABLE): invoke --mode cloud, runtime start/exec/upload/ download/stop-session -- skip-gated. Two invocation styles: in-process CliRunner (init/config, file assertions) and subprocess (cloud JSON parsing + blocking dev). Completion handling: patch _auto_install_completion in-process; redirect HOME to a temp dir with the marker pre-created for subprocess. Pass every config/init flag explicitly to avoid Prompt.ask aborts under CliRunner. README: new 'Toolkit (CLI) tests' section with invocation styles, completion handling, per-command coverage table, tiers, and not-covered list. Verified: 16 toolkit tests pass; runtime billable skip; mcp lifecycle xfail. Full suite 64 passed / 15 skipped / 7 xfailed / 2 deselected.
The real user journey through the agentarts CLI, non-mock end-to-end. Gated behind Docker availability + cloud_credentials + ALLOW_CREATE + RUN_BILLABLE, so it skips by default. The init-generated .agentarts_config.yaml already contains artifact_source and identity_configuration blocks, so deploy (unlike the bare SDK create_agent test) satisfies the backend's 'artifactSource/identityConfiguration not null' requirements by filling artifact_source.url with the pushed SWR image. Residue: deploy auto-creates an SWR org/repo and pushes an image (no cleanup in the operations layer) — accepted, documented. The cloud agent is destroyed by (also registered with resource_registry as a safety net); the local scaffold lives in tmp_path (auto-cleaned). README: add the full-lifecycle row to the toolkit coverage table and update the not-covered list (deploy cloud-mode + destroy are now covered, gated).
create_agency_with_policy read create_response.agency_id, but CreateAgencyV5Response nests the agency under .agency (the list path already uses matching_agency.agency_id). The create_agency_v5 call succeeded but the attribute access raised AttributeError, which gateway_client treated as a non-409 failure and aborted — so the agency was created but the gateway was not. Use create_response.agency.agency_id. Found by the gateway integration lifecycle test after main's trust_policy fix.
… xfail
main renamed MCPGatewayClient→GatewayClient, the mcp-gateway CLI
subcommand→gateway, and fixed the agency trust_policy (PAP5.0011) via
create_agency_with_policy. Adapt the integration suite:
- conftest: mcp_gateway_client fixture → gateway_client (GatewayClient).
- test_readonly_lists: list_mcp_gateways → list_gateways.
- test_mcp_gateway_lifecycle.py → test_gateway_lifecycle.py: drop the _mcp_
infix, remove the xfail (agency now creates), pass a valid
target_configuration ({mcp_server:{endpoint,server_type}}) and handle the
nested {target:{target_id}} response shape.
- toolkit/test_cli_mcp_gateway.py → test_cli_gateway.py: gateway list/create
subcommands, remove xfail.
- README: gateway rename throughout, gateway lifecycle ✅ (was xfail), bugs
table (#4 fixed upstream, #5 agency_id fix), refreshed counts.
Verified: 8 gateway tests pass (SDK 6 + CLI 2); full suite 71 passed / 16
skipped / 0 xfailed / 2 deselected.
…eploy fixture
The B-class billable CLI tests (invoke, runtime start/exec/upload/
download/stop-session) no longer require a separately pre-provisioned agent
(AGENTARTS_TEST_RUNTIME_AGENT_NAME). Instead a single session-scoped
deployed_runtime_agent fixture runs init->config->deploy (one Docker build +
SWR push + cloud runtime create); the invoke and runtime-session tests reuse
that live agent, and destroy runs as the fixture's session-end teardown
(registered with resource_registry).
- conftest: add deployed_runtime_agent fixture + move docker_available in.
- test_cli_deployed_runtime.py: test_deploy_succeeds / test_invoke_deployed_agent
/ test_runtime_session_on_deployed_agent. start-session's session_id is
parsed from the 'Response: {json}' the CLI prints.
- delete test_cli_deploy_lifecycle.py + test_cli_runtime.py (folded in).
- README: reflect the consolidated rows + shared-fixture note.
Gated behind Docker + ALLOW_CREATE + RUN_BILLABLE (skip by default); verified
to collect + skip cleanly without Docker. SWR org/repo/image residue documented.
test_runtime_agent_lifecycle no longer hard-skips on the artifact_source
prereq. The target agent is supplied two ways (either suffices, so Docker
unavailability does not block):
- standalone: AGENTARTS_TEST_RUNTIME_AGENT_NAME -> use a pre-provisioned
agent (no Docker, no billable);
- reuse: fall back to the shared deployed_runtime_agent fixture (Docker +
RUN_BILLABLE) which deploys one.
The agent itself is not created/deleted by these tests (create_agent needs an
artifact_source, only deploy provides — covered transitively). Tests exercise
find/get/update + endpoint CRUD (endpoint created + cleaned up via
resource_registry). Requires ALLOW_CREATE in both modes (writes).
Move agentarts_cmd / docker_available / deployed_runtime_agent from
toolkit/conftest to the parent conftest so SDK tests under tests/integration/
can reuse the deployed agent. README updated (control-plane table, tier
matrix, gaps).
Verified: 6 runtime-agent tests skip cleanly without env var / Docker; full
suite still 71 passed / 16 skipped / 0 xfailed.
…R + backend constraints) Fixes uncovered by a real Docker+billable run: - conftest.deployed_runtime_agent: drop the step (passing --swr-org flips organization_auto_create to false → SWR org 404); init's default config (auto_create=true) is deploy-ready. - Shield docker-credential-osxkeychain from the subprocess PATH + use a plaintext DOCKER_CONFIG — docker login otherwise hangs on macOS keychain auth (-60006) regardless of credsStore. Only affects the subprocess. - Enable file_transfer_config in the config before deploy (init writes enabled=false; backend rejects enabling it after creation). - test_cli_deployed_runtime: parse start-session session_id with json.loads(strict=False) (response has control chars) and look under top-level OR nested 'data'; --path for upload must be a dir ending '/'; split file-transfer into a best-effort test that skips on 401 (IAM-only agent needs a bearer token for the upload endpoint). - test_runtime_agent_lifecycle: get_agents limit=10 (backend rejects <10); skip endpoint CRUD when the backend requires targetVersionName. Verified: full suite 78 passed / 10 skipped / 0 failed with Docker + ALLOW_CREATE + RUN_BILLABLE; deploy→invoke→session→destroy all run for real.
…irror - deployed_runtime_agent fixture: at session end, force-remove ALL RepoTags of the built image (local <name>:latest AND the <swr-registry>/<org>/<repo> push tag) so 355MB images don't pile up across runs. - Speed up the build's pip install: edit the generated Dockerfile to use a fast PyPI index (AGENTARTS_TEST_PIP_INDEX, default tsinghua mirror). Cold install of agentarts-sdk drops ~160s -> ~30s (5x); Docker layer cache then makes subsequent builds near-instant. - .env.example + README: document AGENTARTS_TEST_PIP_INDEX, the measured speedup, the image cleanup, and alternative acceleration options. Verified: 3 deploy tests pass with the mirror; local + swr image tags cleaned at session end.
Memory extraction is async on the backend — the delete_memory tests soft-skipped because no memories had been extracted yet by the time the retrieval tests ran. Fix per backend behavior (extraction fires ~1min after add_messages): - seeded_messages / async_setup: add_messages with is_force_extract=True, then poll list_memories (wait_for) for extracted memories before the retrieval/delete tests run. - The wait timeout is configurable via AGENTARTS_TEST_MEMORY_WAIT (default 90s); if extraction doesn't fire in time, delete_memory still soft-skips (suite stays green). - .env.example: document AGENTARTS_TEST_MEMORY_WAIT. Verified: 18 memory tests pass; delete soft-skips gracefully when extraction is slow on the account under test.
…memory runs
D-class (delete_memory soft-skip) root cause: create_space didn't tune
the extraction trigger params, and the seeded messages were too trivial to
extract ('hello' has no facts). The backend has extraction ON by default —
it's the params that matter (per backend behavior):
- conftest.memory_space: create_space with memory_extract_idle_seconds=10,
memory_extract_max_tokens=4096, memory_extract_max_messages=100 (was only
the strategy). Dropped the update_space(memory_extract_enabled=True) — the
switch isn't the issue.
- seeded_messages / async_setup: use factual content ('My name is Alice and I
live in Beijing.') so the LLM has something to extract; keep
is_force_extract=True + the configurable wait_for (AGENTARTS_TEST_MEMORY_WAIT).
Verified: 20 memory tests pass, 0 skipped (delete_memory sync+async now run).
docker rmi -f only removes the tagged final image; the build's intermediate layers become dangling <none> images and pile up across runs (reported by the user as local residue). Add Total reclaimed space: 0B after the rmi so dangling intermediates are also reclaimed. Verified: after a deploy test run, no dangling and no aa-it images remain.
… Path usage) CI runs ruff check (strict, non-continue-on-error) which failed on 31 errors. Fixed: - I001: unsorted import blocks (auto-fixed across integration tests + SDK source auth.py/identity_client.py + unit tests + notebooks from the main merge). - RUF100: unused noqa directives (BLE001/ANN001 not enabled). - F401: unused imports (sys, pathlib.Path). - PT018: compound `assert A and B` split into separate asserts. - PTH119/PTH120: os.path.basename/dirname → Path.name/parent. Verified: ruff check . → All checks passed; unit tests still pass.
…oesn't)
The IAM (V11-HMAC-SHA256) upload command returned HTTP 401 "Authentication
failed!" while invoke on the same agent succeeded. Root cause, verified
against the real data-plane gateway: the gateway does NOT include the query
string in the V11 canonical request, but the signer did — so any request
carrying query params (e.g. upload's `path=/home/user/...`) failed
verification, while query-less requests (invoke) passed. Content-Type and
the '/' encoding in the query value were ruled out as causes by direct
experiment on a live IAM agent (aa-it-hmac-98e0df28): query signed -> 401,
empty canonical query -> 200/4xx (auth passes).
- signer_v11: sign an empty canonical query line. The query is still sent
on the wire by the HTTP client; only its participation in the signature
is dropped. `_canonical_query_string` is retained as the standard-V11
reference implementation (documented as unused by sign()).
- conftest: auto-load tests/integration/.env via python-dotenv (already a
dev dependency) so AK/SK / region / tier flags are recognised without
manually sourcing the file; existing env vars always win.
- tests:
* unit (test_signer_v11.py): query does not affect the signature;
Content-Type IS signed; signed-headers sorted/lowercased; basics.
* local-mirror integration (test_v11_upload_signature.py): a local V11
verifier mirroring the gateway (empty query line); fixed signer -> 200,
old query-signed behaviour -> 401, tampered signature -> 401, invoke
(no query) -> 200.
* real-backend E2E (test_v11_upload_signature_real.py): auto-discovers a
deployed IAM (V11-accepting) file-transfer agent from the control plane
and asserts the fixed V11 upload authenticates (non-401), a tampered
signature is rejected (401), and the old query-signed behaviour fails
(401). Non-mutating probe (fake session -> 404); gated by cloud creds.
…_files Two fixes uncovered while verifying the data-plane CLI commands against a real IAM agent end-to-end. 1. _resolve_agent_info (invoke.py): default auth_type to "IAM" when it cannot be resolved from the config (no .agentarts_config.yaml, or the agent has no identity_configuration). Previously the 6 data-plane commands then fell through to SDK mode -> open_ak_sk=False -> unsigned request -> IAM data-plane gateway 401, even though these commands only ever sign with V11. The control plane does not expose the agent's auth type (identity_configuration is null; workload_identity_urn is set for all agents and does not differentiate), so defaulting to IAM is the right call. 2. RuntimeClient._request_stream (runtime_client.py): apply AK/SK signing (V11) when open_ak_sk is set, same as BaseHTTPClient._request. download_files uses _request_stream, which called session.request directly and bypassed signing -> download_files 401'd on IAM agents while upload_files (via _request) signed fine. Verified end-to-end against a real IAM agent (aa-it-curl-gen-03a43b42) with no .agentarts_config.yaml, only --agent/--region + AK/SK: start-session, upload-files, download-files (content round-trip), exec-command, invoke, stop-session — all HTTP 200. Tests: _resolve_agent_info defaulting (no-config / no-identity-config / agent-not-in-config) + _request_stream signs when open_ak_sk.
download-files was the only data-plane command without a -r alias for --region, so 'agentarts runtime download-files -a X -r R' failed with 'No such option: -r' while the other 5 commands accepted -r. Verified end-to-end against a real IAM agent: download-files -r works and the content round-trips.
Many container images don't grant write permission under /home/user, so upload-files with no --path failed there. Default the remote path to /tmp/, which is writable in most images. Users can still override with --path. - runtime_client.upload_files, operations/upload_files, cli/upload_files: default path "/home/user/" -> "/tmp/". - help text + docstrings updated to state the /tmp default and why. - docs/cn/toolkit_user_guide/runtime_cli.md: --path default row, the "默认目录" note, and upload/download/exec examples moved off /home/user. - signer_v11 comment example updated. - tests: default-path assertions updated to /tmp/. Verified end-to-end against a real IAM agent (no --path): file uploaded to /tmp/<name> (success) and download round-trips content.
The AgentArts backend is predominantly arm, so defaulting to x86_64 was the wrong choice when arch can't be detected or is omitted from the config. - AgentArtsRuntimeConfig.arch field default: X86_64 -> ARM64. - BaseConfig.platform field default: linux/amd64 -> linux/arm64 (kept consistent with the arch default). - detect_arch(): keep real-machine detection (arm64/aarch64 -> ARM64, x86_64/amd64 -> X86_64); the unknown-architecture fallback now ARM64. - detect_platform() (init + operations/config): unknown-arch fallback now linux/arm64. Real x86 detection still returns linux/amd64. init/config on a real arm or x86 machine is unchanged (they write the detected arch); only the field defaults and the unrecognised-machine fallback prefer arm now. Tests: detect_arch/detect_platform unknown->arm fallback, and a config omitting arch/platform defaults to arm64 / linux/arm64.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.