Expand and harden Amazon Bedrock security checks#44
Open
agasthik wants to merge 8 commits into
Open
Conversation
Add 18 new Amazon Bedrock security checks and fix accuracy issues in the initial BR-15..25 set, all validated against AWS API references and the boto3/botocore service models. New checks: - BR-15..25: cross-account guardrails, guardrail tier, custom-model KMS, model evaluation, prompt flow validation, knowledge base KMS, agent action group IAM, service-quota throttling, content-filter coverage, automated reasoning, RAG evaluation - BR-26..32: guardrail sensitive-information (PII) filters, contextual grounding, agent guardrail association, agent idle session TTL, imported model KMS, batch inference output encryption, CloudWatch alarms on AWS/Bedrock metrics Accuracy fixes (verified against AWS docs): - BR-15 uses BEDROCK_POLICY org policy type - BR-16 reads contentPolicy.tier.tierName (CLASSIC/STANDARD) - BR-17 reads modelKmsKeyArn - BR-19 uses GetFlow validations array - BR-20 keys off knowledgeBaseConfiguration.type and reads the managed-KB serverSideEncryptionConfiguration.kmsKeyArn; reports N/A "indeterminate" when the field is absent (older bundled SDK) instead of false-failing - BR-21 inspects policy names / inline wildcard grants - BR-23 reads contentPolicy.filters; BR-24 reads automatedReasoningPolicy Dependencies: - Pin boto3/botocore to 1.43.32 across Lambda functions and tests (first release modeling managedKnowledgeBaseConfiguration); finserv keeps a >=1.43.32 floor for security patching - Add boto3/botocore to functions that imported it without declaring it; add missing resolve_regions requirements.txt IAM (all three CloudFormation templates): - Grant the read-only actions the new checks need: organizations (DescribeOrganization/ListRoots/ListPolicies), servicequotas (List/GetServiceQuota), cloudwatch:DescribeAlarms, bedrock action-group / imported-model / model-invocation-job / evaluation-job reads, and lambda:GetFunction for action-group role inspection Multi-region: - BR-26..32 run per scanned region (tagged with the region); BR-15 stays global (runs once on the primary region, tagged Global). Added handler-level regression tests for primary/non-primary gating Docs: README and SECURITY_CHECKS.md updated (116 -> 134 total, Bedrock 25 -> 32) with focus-area summaries and BR-26..32 reference entries. buildspec: opt out of SAM CLI telemetry. All 373 tests pass; ruff lint+format and cfn-lint clean.
…lates
The SAM scanning-Lambda role (BedrockSecurityAssessmentFunction) was missing
read-only actions the newer BR-15..32 checks call, causing AccessDeniedException
('Unable to check') for BR-18, BR-22, BR-30, BR-31 at runtime. The standalone
deployment/*.yaml roles already had these; the SAM templates were not updated.
Add to template.yaml and template-multi-account.yaml:
- bedrock:ListEvaluationJobs (BR-18)
- bedrock:ListImportedModels / GetImportedModel (BR-30)
- bedrock:ListModelInvocationJobs (BR-31)
- servicequotas:ListServiceQuotas / GetServiceQuota (BR-22)
- cloudwatch:DescribeAlarms, bedrock:ListAgentActionGroups / GetAgentActionGroup
(proactive: same AccessDenied once alarms/action groups exist)
Extend test_core_iam_coverage.py to guard the new actions so this drift
cannot recur silently.
BR-15 (Cross-Account Guardrails Enforcement) calls
organizations:DescribeOrganization and organizations:ListPolicies, but the
SAM Bedrock scanning-Lambda role had no Organizations grant, causing
AccessDeniedException ('Unable to check') at runtime. The deployment/*.yaml
roles already had these.
Add an OrganizationsPermissions statement to the BedrockSecurityAssessmentFunction
role in template.yaml and template-multi-account.yaml, and extend
test_core_iam_coverage.py to guard the actions.
Bedrock Custom Model Import, Batch Inference, and Model Evaluation are
account/region feature-gated. When not enabled, the List* APIs raise
AccessDeniedException with 'Your account is not authorized to invoke this API
operation' - a different cause from an IAM-policy gap ('... not authorized to
perform: <action> because no identity-based policy allows ...').
Previously both collapsed to status=Failed with a 'grant <action> permission'
resolution, which is misleading once the IAM grant is in place: no IAM change
enables a feature-gated API.
Add is_account_not_authorized() to distinguish the two, and route the
account-gate case to status=N/A (severity Low) with a feature-not-enabled
resolution for BR-18, BR-30, BR-31. Genuine IAM-gap denials still report Failed.
Add regression tests covering the account-gate branch for each check.
Audited all 33 Bedrock checks for how region/feature unavailability is handled. Four checks call region-limited APIs (Knowledge Bases, Agents, RAG/model evaluation) but had no UnknownOperation branch, so in a region lacking the API the error skipped the AccessDenied branch and surfaced as a generic ERROR (BR-20/21/25) or a misleading Failed (BR-09) instead of N/A. Add a shared is_region_unsupported() helper and a region-unsupported -> N/A branch to BR-09, BR-20, BR-21, BR-25, matching the existing pattern in BR-18/24/28/29/30/31. Genuine IAM-gap and validation errors still report as before. Add regression tests covering the region-unsupported branch for each, plus a new BR-25 test class (none existed).
check_bedrock_model_evaluations unconditionally reported Failed when list_evaluation_jobs returned empty, producing a false failure in regions where Bedrock is not used at all. Gate the empty case on detect_bedrock_regional_footprint (matching BR-05): N/A when no regional Bedrock resources exist, Failed only when Bedrock is in use but no evaluation jobs are configured. Add a test for the no-footprint N/A path and pin the existing empty-jobs test to the footprint-present Failed path.
The "52 core checks" figure predated the BR-15..32 Bedrock expansion (it assumed BR-01..14). The framework now has 70 core checks (32 Bedrock + 25 SageMaker + 13 AgentCore) and 134 total with the 64 FinServ checks. Update the six stale references so all docs agree with README.md, SECURITY_CHECKS.md, and the codebase.
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This branch expands the Amazon Bedrock assessment to 32 checks (18 new checks, BR-15..32), brings the framework
total to 134 checks, and fixes accuracy issues so that "resource/feature absent" conditions are reported as
N/A instead of false failures. It also updates the supporting IAM permissions, dependency pins, CloudFormation
templates, and documentation.
What's included
New Bedrock checks (BR-15..32)
evaluation implementation, prompt flow validation, knowledge base KMS encryption, agent action-group IAM least
privilege, model-invocation throttling limits, guardrail content-filter coverage, automated reasoning policy, RAG
evaluation jobs
idle session TTL, imported-model KMS encryption, batch-inference output encryption, CloudWatch alarms on AWS/Bedrock metrics
Accuracy / false-failure fixes (validated against AWS API references and the boto3/botocore service models)
no Bedrock resources exist in the region, and only reports Failed when Bedrock is in use but no evaluation jobs are
configured (previously it failed unconditionally)
modelKmsKeyArn, GetFlow validations, managed-KB serverSideEncryptionConfiguration.kmsKeyArn)
IAM — all three CloudFormation templates grant the additional read-only actions the new checks need (Organizations
describe/list, Service Quotas, cloudwatch:DescribeAlarms, Bedrock action-group / imported-model / invocation-job /
evaluation-job reads, and lambda:GetFunction).
Dependencies — pin boto3/botocore to 1.43.32 across Lambda functions and tests (first release modeling
managedKnowledgeBaseConfiguration); add the dependency to functions that imported it without declaring it, plus a
missing resolve_regions/requirements.txt.
Multi-region — BR-26..32 run per scanned region (tagged with the region); BR-15 stays global (runs once on the primary
region, tagged Global).
Docs
— README, SECURITY_CHECKS.md, SECURITY_CHECKS_FINSERV.md, DEVELOPER_GUIDE.md, and TROUBLESHOOTING.md updated to the latest counts: 70 core (32 Bedrock + 25 SageMaker + 13 AgentCore) + 64 FinServ = 134 total.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.