Skip to content

Add Agentic AI Security lens#46

Open
agasthik wants to merge 10 commits into
aws-samples:mainfrom
agasthik:agentic-ai-security-lens
Open

Add Agentic AI Security lens#46
agasthik wants to merge 10 commits into
aws-samples:mainfrom
agasthik:agentic-ai-security-lens

Conversation

@agasthik

@agasthik agasthik commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a new always-on Agentic AI Security assessment area (27 AG- checks) derived from the AWS Well-Architected Agentic AI Lens, taking the suite to 161 checks across five areas. The lens is synthesized rather than separately scanned: existing Bedrock (BR-) and AgentCore (AC-) findings are re-mapped into AG- rows with agentic-specific context, plus four native AgentCore gateway checks (AG-24..27).
  • Also folds in IAM-coverage and N/A-semantics.

Changes

New Agentic AI Security area

  • AGENTIC_BEDROCK_CHECK_MAPPINGS (in bedrock_assessments/app.py) re-maps Bedrock findings into AG-01..14, each annotated with agentic context (guardrail association, tool least-privilege, model invocation logging, abuse/token-amplification, session lifetime, etc.).
  • AGENTIC_AGENTCORE_CHECK_MAPPINGS (in agentcore_assessments/app.py) re-maps AgentCore findings into AG-15..23 (network boundaries, principal least-privilege, observability, memory encryption, private connectivity, resource policies).
  • Four native AgentCore gateway checks AG-24..27 (inbound authorization, tool policy enforcement, error-detail exposure, WAF protection).
  • All AG- findings reference the Agentic AI Lens docs and carry per-region N/A rows when source checks can't run.

Report layer

  • New "By Lens" navigation section rendering Agentic AI findings under a dedicated, distinctly-styled nav group with its own service card, filter option, and search/filter wiring.
  • Fixes grouping so BR-22/AG-12 no longer emit Failed findings in regions with no Bedrock footprint.

IAM / status-semantics hardening (review fixes)

  • Synced the FinServ grants + cloudwatch:PutMetricData into the local MemberRole in deployment/2-aiml-security-codebuild.yaml.
  • Added cloudwatch:PutMetricData across all three deployment role files and both SAM templates.
  • AC-02 empty permission cache now resolves to N/A/Informational (was Failed).
  • Distinguished indeterminate (None) from probed-empty (False) Bedrock footprint in N/A finding text via bedrock_footprint_na_detail().

Docs & tests

  • README, docs/SECURITY_CHECKS.md, and docs/DEVELOPER_GUIDE.md updated to 161 checks / five areas, with a full AG-01..27 reference section.
  • 15 new tests across test_agentcore_checks.py, test_bedrock_checks.py, and test_consolidated_report.py covering AG synthesis, gateway checks, and report grouping.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

agasthik added 10 commits June 26, 2026 22:27
@agasthik
feat: Expand and harden Amazon Bedrock security checks (BR-15..32)
0e240bd 
Add 18 new Amazon Bedrock security checks and fix accuracy issues in the
initial BR-15..25 set, all validated against AWS API references and the
boto3/botocore service models.

New checks:
- BR-15..25: cross-account guardrails, guardrail tier, custom-model KMS,
  model evaluation, prompt flow validation, knowledge base KMS, agent action
  group IAM, service-quota throttling, content-filter coverage, automated
  reasoning, RAG evaluation
- BR-26..32: guardrail sensitive-information (PII) filters, contextual
  grounding, agent guardrail association, agent idle session TTL, imported
  model KMS, batch inference output encryption, CloudWatch alarms on
  AWS/Bedrock metrics

Accuracy fixes (verified against AWS docs):
- BR-15 uses BEDROCK_POLICY org policy type
- BR-16 reads contentPolicy.tier.tierName (CLASSIC/STANDARD)
- BR-17 reads modelKmsKeyArn
- BR-19 uses GetFlow validations array
- BR-20 keys off knowledgeBaseConfiguration.type and reads the managed-KB
  serverSideEncryptionConfiguration.kmsKeyArn; reports N/A "indeterminate"
  when the field is absent (older bundled SDK) instead of false-failing
- BR-21 inspects policy names / inline wildcard grants
- BR-23 reads contentPolicy.filters; BR-24 reads automatedReasoningPolicy

Dependencies:
- Pin boto3/botocore to 1.43.32 across Lambda functions and tests (first
  release modeling managedKnowledgeBaseConfiguration); finserv keeps a
  >=1.43.32 floor for security patching
- Add boto3/botocore to functions that imported it without declaring it; add
  missing resolve_regions requirements.txt

IAM (all three CloudFormation templates):
- Grant the read-only actions the new checks need: organizations
  (DescribeOrganization/ListRoots/ListPolicies), servicequotas
  (List/GetServiceQuota), cloudwatch:DescribeAlarms, bedrock action-group /
  imported-model / model-invocation-job / evaluation-job reads, and
  lambda:GetFunction for action-group role inspection

Multi-region:
- BR-26..32 run per scanned region (tagged with the region); BR-15 stays
  global (runs once on the primary region, tagged Global). Added handler-level
  regression tests for primary/non-primary gating

Docs: README and SECURITY_CHECKS.md updated (116 -> 134 total, Bedrock 25 -> 32)
with focus-area summaries and BR-26..32 reference entries.
buildspec: opt out of SAM CLI telemetry.

All 373 tests pass; ruff lint+format and cfn-lint clean.
@agasthik
fix: grant missing Bedrock IAM actions for BR-18/22/30/31 in SAM temp…
385fb69 
…lates

The SAM scanning-Lambda role (BedrockSecurityAssessmentFunction) was missing
read-only actions the newer BR-15..32 checks call, causing AccessDeniedException
('Unable to check') for BR-18, BR-22, BR-30, BR-31 at runtime. The standalone
deployment/*.yaml roles already had these; the SAM templates were not updated.

Add to template.yaml and template-multi-account.yaml:
- bedrock:ListEvaluationJobs (BR-18)
- bedrock:ListImportedModels / GetImportedModel (BR-30)
- bedrock:ListModelInvocationJobs (BR-31)
- servicequotas:ListServiceQuotas / GetServiceQuota (BR-22)
- cloudwatch:DescribeAlarms, bedrock:ListAgentActionGroups / GetAgentActionGroup
  (proactive: same AccessDenied once alarms/action groups exist)

Extend test_core_iam_coverage.py to guard the new actions so this drift
cannot recur silently.
@agasthik
fix: grant Organizations permissions for BR-15 in SAM templates
fca8b38 
BR-15 (Cross-Account Guardrails Enforcement) calls
organizations:DescribeOrganization and organizations:ListPolicies, but the
SAM Bedrock scanning-Lambda role had no Organizations grant, causing
AccessDeniedException ('Unable to check') at runtime. The deployment/*.yaml
roles already had these.

Add an OrganizationsPermissions statement to the BedrockSecurityAssessmentFunction
role in template.yaml and template-multi-account.yaml, and extend
test_core_iam_coverage.py to guard the actions.
@agasthik
fix: report account/feature-gate denials as N/A for BR-18/30/31
7714e17 
Bedrock Custom Model Import, Batch Inference, and Model Evaluation are
account/region feature-gated. When not enabled, the List* APIs raise
AccessDeniedException with 'Your account is not authorized to invoke this API
operation' - a different cause from an IAM-policy gap ('... not authorized to
perform: <action> because no identity-based policy allows ...').

Previously both collapsed to status=Failed with a 'grant <action> permission'
resolution, which is misleading once the IAM grant is in place: no IAM change
enables a feature-gated API.

Add is_account_not_authorized() to distinguish the two, and route the
account-gate case to status=N/A (severity Low) with a feature-not-enabled
resolution for BR-18, BR-30, BR-31. Genuine IAM-gap denials still report Failed.

Add regression tests covering the account-gate branch for each check.
@agasthik
fix: report region-unsupported APIs as N/A for BR-09/20/21/25
4ba21c1 
Audited all 33 Bedrock checks for how region/feature unavailability is handled.
Four checks call region-limited APIs (Knowledge Bases, Agents, RAG/model
evaluation) but had no UnknownOperation branch, so in a region lacking the API
the error skipped the AccessDenied branch and surfaced as a generic ERROR
(BR-20/21/25) or a misleading Failed (BR-09) instead of N/A.

Add a shared is_region_unsupported() helper and a region-unsupported -> N/A
branch to BR-09, BR-20, BR-21, BR-25, matching the existing pattern in
BR-18/24/28/29/30/31. Genuine IAM-gap and validation errors still report as
before.

Add regression tests covering the region-unsupported branch for each, plus a
new BR-25 test class (none existed).
@agasthik
fix: report BR-18 as N/A when no Bedrock footprint in region
46723e4 
check_bedrock_model_evaluations unconditionally reported Failed when
list_evaluation_jobs returned empty, producing a false failure in
regions where Bedrock is not used at all. Gate the empty case on
detect_bedrock_regional_footprint (matching BR-05): N/A when no regional
Bedrock resources exist, Failed only when Bedrock is in use but no
evaluation jobs are configured.

Add a test for the no-footprint N/A path and pin the existing empty-jobs
test to the footprint-present Failed path.
@agasthik
docs: correct stale core check count (52 -> 70)
02fe8e6 
The "52 core checks" figure predated the BR-15..32 Bedrock expansion
(it assumed BR-01..14). The framework now has 70 core checks
(32 Bedrock + 25 SageMaker + 13 AgentCore) and 134 total with the 64
FinServ checks. Update the six stale references so all docs agree with
README.md, SECURITY_CHECKS.md, and the codebase.
@agasthik
style: apply ruff format to test_bedrock_checks.py
97b03d4
@agasthik
Add Agentic AI security lens; harden IAM coverage and N/A semantics
d30253f 
Adds the synthesized Agentic AI (AG-) assessment area across Bedrock and
AgentCore, plus report-layer routing, expanded checks, and tests.

Review fixes folded in:
- Sync FinServ + cloudwatch:PutMetricData grants into the local MemberRole
  in deployment/2-aiml-security-codebuild.yaml (was missing whole module)
- Add cloudwatch:PutMetricData to all three deployment role files
- AC-02 empty permission cache -> N/A/Informational (was Failed)
- Distinguish indeterminate (None) from probed-empty (False) Bedrock
  footprint in N/A finding text via bedrock_footprint_na_detail()
@agasthik
Fix agentic lens report grouping
487cab8 
Render Agentic AI findings under a dedicated By Lens nav section and avoid BR-22/AG-12 failed findings in regions with no Bedrock footprint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

agentcore bedrock deployment documentation Improvements or additions to documentation infrastructure report sagemaker

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@agasthik