From 0e240bdd7c23730fa1bdf2d13bcc30ff39bb06f5 Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Fri, 26 Jun 2026 21:56:37 -0400 Subject: [PATCH 01/10] feat: Expand and harden Amazon Bedrock security checks (BR-15..32) Add 18 new Amazon Bedrock security checks and fix accuracy issues in the initial BR-15..25 set, all validated against AWS API references and the boto3/botocore service models. New checks: - BR-15..25: cross-account guardrails, guardrail tier, custom-model KMS, model evaluation, prompt flow validation, knowledge base KMS, agent action group IAM, service-quota throttling, content-filter coverage, automated reasoning, RAG evaluation - BR-26..32: guardrail sensitive-information (PII) filters, contextual grounding, agent guardrail association, agent idle session TTL, imported model KMS, batch inference output encryption, CloudWatch alarms on AWS/Bedrock metrics Accuracy fixes (verified against AWS docs): - BR-15 uses BEDROCK_POLICY org policy type - BR-16 reads contentPolicy.tier.tierName (CLASSIC/STANDARD) - BR-17 reads modelKmsKeyArn - BR-19 uses GetFlow validations array - BR-20 keys off knowledgeBaseConfiguration.type and reads the managed-KB serverSideEncryptionConfiguration.kmsKeyArn; reports N/A "indeterminate" when the field is absent (older bundled SDK) instead of false-failing - BR-21 inspects policy names / inline wildcard grants - BR-23 reads contentPolicy.filters; BR-24 reads automatedReasoningPolicy Dependencies: - Pin boto3/botocore to 1.43.32 across Lambda functions and tests (first release modeling managedKnowledgeBaseConfiguration); finserv keeps a >=1.43.32 floor for security patching - Add boto3/botocore to functions that imported it without declaring it; add missing resolve_regions requirements.txt IAM (all three CloudFormation templates): - Grant the read-only actions the new checks need: organizations (DescribeOrganization/ListRoots/ListPolicies), servicequotas (List/GetServiceQuota), cloudwatch:DescribeAlarms, bedrock action-group / imported-model / model-invocation-job / evaluation-job reads, and lambda:GetFunction for action-group role inspection Multi-region: - BR-26..32 run per scanned region (tagged with the region); BR-15 stays global (runs once on the primary region, tagged Global). Added handler-level regression tests for primary/non-primary gating Docs: README and SECURITY_CHECKS.md updated (116 -> 134 total, Bedrock 25 -> 32) with focus-area summaries and BR-26..32 reference entries. buildspec: opt out of SAM CLI telemetry. All 373 tests pass; ruff lint+format and cfn-lint clean. --- README.md | 12 +- .../agentcore_assessments/requirements.txt | 3 +- .../security/bedrock_assessments/app.py | 3469 ++++++++++++++++- .../bedrock_assessments/requirements.txt | 4 +- .../security/cleanup_bucket/requirements.txt | 3 +- .../finserv_assessments/requirements.txt | 15 +- .../requirements.txt | 4 +- .../iam_permission_caching/requirements.txt | 4 +- .../security/resolve_regions/requirements.txt | 2 + .../sagemaker_assessments/requirements.txt | 4 +- buildspec.yml | 4 + deployment/1-aiml-security-member-roles.yaml | 24 + deployment/2-aiml-security-codebuild.yaml | 20 + deployment/aiml-security-single-account.yaml | 20 + docs/SECURITY_CHECKS.md | 118 +- tests/requirements.txt | 4 +- tests/test_bedrock_checks.py | 1616 ++++++++ 17 files changed, 5212 insertions(+), 114 deletions(-) create mode 100644 aiml-security-assessment/functions/security/resolve_regions/requirements.txt diff --git a/README.md b/README.md index 807cf7c..ef5eb9d 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ **Open-source automated security scanner for generative AI and machine learning workloads on AWS.** Core checks for Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore are built on the [AWS Well-Architected Framework — Generative AI Lens](https://docs.aws.amazon.com/wellarchitected/latest/generative-ai-lens/generative-ai-lens.html). An optional Financial Services GenAI risk module adds 64 checks aligned to the [AWS User Guide to Governance, Risk, and Compliance for Responsible AI Adoption within Financial Services Industries](https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/whitepapers/compliance/AWS-User-Guide-Governance-Risk-Compliance-for-Responsible-AI-Adoption-Financial-Services.pdf). See the [AWS Security Blog announcement](https://aws.amazon.com/blogs/security/introducing-the-updated-aws-user-guide-to-governance-risk-and-compliance-for-responsible-ai-adoption/) for context on the updated guide. -Run **[116 security checks](docs/SECURITY_CHECKS.md)** across your AWS accounts and regions in one deployment. Surfaces IAM misconfigurations, encryption gaps, network isolation issues, missing guardrails, and governance gaps — with interactive HTML reports, severity ratings, and AWS documentation links for remediation. Single-account or full AWS Organizations multi-account scans; all data stays in your account. +Run **[134 security checks](docs/SECURITY_CHECKS.md)** across your AWS accounts and regions in one deployment. Surfaces IAM misconfigurations, encryption gaps, network isolation issues, missing guardrails, and governance gaps — with interactive HTML reports, severity ratings, and AWS documentation links for remediation. Single-account or full AWS Organizations multi-account scans; all data stays in your account. --- @@ -39,7 +39,7 @@ The framework generates professional, interactive security assessment reports wi - **Executive Summary** with severity counts and service breakdown - **Priority Recommendations** highlighting critical issues requiring immediate attention -- **[116 Security Checks](docs/SECURITY_CHECKS.md)** across Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and Financial Services GenAI Risk +- **[134 Security Checks](docs/SECURITY_CHECKS.md)** across Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and Financial Services GenAI Risk - **Multi-Region Support** for core Bedrock, SageMaker, and AgentCore checks, with per-region risk breakdown - **Interactive Filtering** by account, region, service, severity, and status - **Light/Dark Mode Toggle** with persistent user preference @@ -83,14 +83,14 @@ Designed for workloads using [Amazon Bedrock](https://aws.amazon.com/bedrock/), | Challenge | How This Framework Helps | |-----------|-------------------------| | **Manual security audits are time-consuming** | Fully automated scanning with one-click CloudFormation deployment | -| **Inconsistent security checks across teams** | Standardized 116-check assessment based on AWS Well-Architected Generative AI Lens best practices and AWS Responsible AI governance, risk, and compliance guidance for financial services | +| **Inconsistent security checks across teams** | Standardized 134-check assessment based on AWS Well-Architected Generative AI Lens best practices and AWS Responsible AI governance, risk, and compliance guidance for financial services | | **Difficulty tracking AI/ML security posture** | Interactive HTML dashboards with severity breakdown and per-account visibility | | **Multi-account complexity** | Consolidated reporting across AWS Organizations with cross-account role assumption | | **Compliance and audit support** | Exportable reports to supplement your compliance program, with remediation guidance linked to AWS documentation | | **Generative AI security gaps** | Purpose-built checks for LLM guardrails, model access controls, and prompt injection prevention | **Services Covered:** -- **[Amazon Bedrock](docs/SECURITY_CHECKS.md#amazon-bedrock-security-checks-14)** (14 checks) - Guardrails, encryption, Amazon VPC endpoints, AWS IAM permissions, model invocation logging +- **[Amazon Bedrock](docs/SECURITY_CHECKS.md#amazon-bedrock-security-checks-32)** (32 checks) - Guardrails, cross-account policies, guardrail tiers, content filters, sensitive-information/PII filters, contextual grounding, automated reasoning, encryption (custom, imported, knowledge base, and batch inference output), Amazon VPC endpoints, AWS IAM permissions, agent guardrail association and least privilege, model invocation logging, CloudWatch alarms, model evaluation, prompt flow validation, RAG evaluation, service quotas - **[Amazon SageMaker AI](docs/SECURITY_CHECKS.md#amazon-sagemaker-ai-security-checks-25)** (25 checks) - AWS Security Hub controls (SageMaker.1-5), encryption, network isolation, AWS IAM, MLOps - **[Amazon Bedrock AgentCore](docs/SECURITY_CHECKS.md#amazon-bedrock-agentcore-security-checks-13)** (13 checks) - Amazon VPC configuration, encryption, observability, resource policies - **[Financial Services GenAI Risk](docs/SECURITY_CHECKS.md#financial-services-genai-risk-checks-64-additional-5-upstream-extensions)** (64 checks) - Unbounded consumption, excessive agency, supply chain, training data poisoning, hallucination, prompt injection, PII disclosure, and 8 more FinServ-specific risk categories derived from the [AWS User Guide to Governance, Risk, and Compliance for Responsible AI Adoption within Financial Services Industries](https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/whitepapers/compliance/AWS-User-Guide-Governance-Risk-Compliance-for-Responsible-AI-Adoption-Financial-Services.pdf). See the [AWS Security Blog announcement](https://aws.amazon.com/blogs/security/introducing-the-updated-aws-user-guide-to-governance-risk-and-compliance-for-responsible-ai-adoption/) for context on the updated guide. @@ -115,7 +115,7 @@ This tool operates within the [AWS Shared Responsibility Model](https://aws.amaz **No guarantee of security or compliance.** This framework identifies common misconfigurations based on AWS best practices and the AWS Well-Architected Framework. It does not cover all possible security risks, does not replace formal compliance audits (SOC 2, HIPAA, and similar), and does not guarantee that your workloads are secure. Use the results as one input into your broader security program. -**116 checks across four domains.** The assessment covers Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and optional Financial Services GenAI risk checks. Other AI/ML services (Amazon Comprehend, Amazon Rekognition, Amazon Textract, and others) are not currently assessed. +**134 checks across four domains.** The assessment covers Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and optional Financial Services GenAI risk checks. Other AI/ML services (Amazon Comprehend, Amazon Rekognition, Amazon Textract, and others) are not currently assessed. --- @@ -386,7 +386,7 @@ If you need to reduce scope, review the role policies in: | Document | Description | |----------|-------------| -| [Security Checks Reference](docs/SECURITY_CHECKS.md) | Complete reference for all 116 security checks with severity levels | +| [Security Checks Reference](docs/SECURITY_CHECKS.md) | Complete reference for all 134 security checks with severity levels | | [FinServ GenAI Risk Checks](docs/SECURITY_CHECKS_FINSERV.md) | Complete FS-01..69 reference: shared introduction, severity rubric, upstream-overlap table, compliance framework mapping, and all check definitions (Part 1 infrastructure controls, Part 2 guardrails & content safety, Part 3 app-layer controls & gaps) | | [FinServ Severity Methodology](docs/SECURITY_CHECKS_FINSERV_SEVERITY_METHODOLOGY.md) | Likelihood × Impact → ASFF severity model, disposition rules, and research basis for FS check severities | | [FinServ Severity Register](docs/SECURITY_CHECKS_FINSERV_SEVERITY_REGISTER.md) | Authoritative per-finding severity assignments (the single source of truth enforced by the drift-guard test) | diff --git a/aiml-security-assessment/functions/security/agentcore_assessments/requirements.txt b/aiml-security-assessment/functions/security/agentcore_assessments/requirements.txt index a518cd0..b7f0821 100644 --- a/aiml-security-assessment/functions/security/agentcore_assessments/requirements.txt +++ b/aiml-security-assessment/functions/security/agentcore_assessments/requirements.txt @@ -1,3 +1,4 @@ -boto3>=1.39.8 +boto3==1.43.32 +botocore==1.43.32 pydantic>=2.0 typing-extensions diff --git a/aiml-security-assessment/functions/security/bedrock_assessments/app.py b/aiml-security-assessment/functions/security/bedrock_assessments/app.py index 264dd42..933ad1f 100644 --- a/aiml-security-assessment/functions/security/bedrock_assessments/app.py +++ b/aiml-security-assessment/functions/security/bedrock_assessments/app.py @@ -841,6 +841,43 @@ def has_bedrock_permissions(policy_doc: Any) -> bool: return False +def _policy_grants_wildcard(policy_doc: Any) -> bool: + """ + Return True if the policy document has an Allow statement granting both + Action "*" and Resource "*" (full wildcard access). + """ + try: + if isinstance(policy_doc, str): + policy_doc = json.loads(policy_doc) + + if not policy_doc: + return False + + statements = policy_doc.get("Statement", []) + if isinstance(statements, dict): + statements = [statements] + + for statement in statements: + if statement.get("Effect", "").upper() != "ALLOW": + continue + + actions = statement.get("Action", []) + if isinstance(actions, str): + actions = [actions] + + resources = statement.get("Resource", []) + if isinstance(resources, str): + resources = [resources] + + if "*" in actions and "*" in resources: + return True + + return False + except Exception as e: + logger.warning(f"Error parsing policy document for wildcard access: {str(e)}") + return False + + def handle_aws_throttling(func, *args, **kwargs): """ Handle AWS API throttling with exponential backoff @@ -2657,120 +2694,3278 @@ def check_bedrock_agent_roles(permission_cache, region: str = "") -> Dict[str, A } -def generate_csv_report(findings: List[Dict[str, Any]]) -> str: +def check_bedrock_cross_account_guardrails(region: str = "") -> Dict[str, Any]: """ - Generate CSV report from all security check findings + BR-15: Check if organization-level guardrails are configured using AWS Organizations Bedrock policies + for centralized safety control enforcement (NEW - April 2026 feature) """ - logger.debug("Generating CSV report") - csv_buffer = StringIO() - fieldnames = [ - "Check_ID", - "Finding", - "Finding_Details", - "Resolution", - "Reference", - "Severity", - "Status", - "Region", - ] - writer = csv.DictWriter(csv_buffer, fieldnames=fieldnames) + logger.debug("Starting check for cross-account guardrails enforcement") + try: + findings = { + "check_name": "Cross-Account Guardrails Enforcement Check", + "status": "PASS", + "details": "", + "csv_data": [], + } - writer.writeheader() - for finding in findings: - if finding["csv_data"]: - for row in finding["csv_data"]: - writer.writerow(row) + try: + orgs_client = boto3.client("organizations", config=boto3_config) - return csv_buffer.getvalue() + # Check if running in management account + org_info = orgs_client.describe_organization() + master_account_id = org_info["Organization"]["MasterAccountId"] + # Get current account ID + sts_client = boto3.client("sts", config=boto3_config) + current_account = sts_client.get_caller_identity()["Account"] -def get_current_utc_date(): - return datetime.now(timezone.utc).strftime("%Y/%m/%d") + if current_account != master_account_id: + findings["details"] = ( + "Not running in management account, cannot check organizational policies" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-15", + finding_name="Cross-Account Guardrails Enforcement Check", + finding_details="Check must run in AWS Organizations management account to evaluate organizational policies", + resolution="Run assessment in management account to check cross-account guardrails enforcement", + reference="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_bedrock.html", + severity="Medium", + status="N/A", + region=region, + ) + ) + return findings + # List policy types enabled for the organization. The Amazon Bedrock + # policy type (used to enforce guardrails across accounts) is + # identified as BEDROCK_POLICY in AWS Organizations. + enabled_policy_types = orgs_client.list_roots()["Roots"][0].get( + "PolicyTypes", [] + ) + bedrock_policy_enabled = any( + pt.get("Type") == "BEDROCK_POLICY" and pt.get("Status") == "ENABLED" + for pt in enabled_policy_types + ) -def write_to_s3( - execution_id, csv_content: str, bucket_name: str, region: str = "" -) -> str: - """ - Write CSV report to S3 bucket - """ - logger.debug(f"Writing CSV report to S3 bucket: {bucket_name}") - try: - s3_client = boto3.client("s3", config=boto3_config) - if region: - file_name = f"bedrock_security_report_{execution_id}_{region}.csv" - else: - file_name = f"bedrock_security_report_{execution_id}.csv" + if not bedrock_policy_enabled: + findings["status"] = "WARN" + findings["details"] = ( + "Bedrock Guardrails policy type is not enabled for the organization" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-15", + finding_name="Cross-Account Guardrails Enforcement Check", + finding_details="Bedrock Guardrails policy type is not enabled at the organization level. Cross-account guardrails cannot be enforced without enabling this policy type.", + resolution="Enable Bedrock Guardrails policy type in AWS Organizations to enforce consistent safety controls across all accounts. Use AWS Organizations console or CLI to enable the policy type.", + reference="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_bedrock.html", + severity="High", + status="Failed", + region=region, + ) + ) + return findings - s3_client.put_object( - Bucket=bucket_name, Key=file_name, Body=csv_content, ContentType="text/csv" - ) + # Check for Bedrock policies attached to organization roots/OUs/accounts + policies_attached = False + try: + policies = orgs_client.list_policies(Filter="BEDROCK_POLICY") + policies_attached = len(policies.get("Policies", [])) > 0 + except Exception as e: + if "UnknownOperation" in str(e) or "Unknown operation" in str(e): + logger.info( + "Bedrock Guardrails policy listing not available in this region" + ) + findings["details"] = ( + "Bedrock Guardrails organizational policy feature not available in this region" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-15", + finding_name="Cross-Account Guardrails Enforcement Check", + finding_details="Bedrock Guardrails organizational policy API not available in this region", + resolution="This feature may not be available in all regions. Check AWS documentation for regional availability.", + reference="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_bedrock.html", + severity="Medium", + status="N/A", + region=region, + ) + ) + return findings + raise + + if not policies_attached: + findings["status"] = "WARN" + findings["details"] = ( + "No Bedrock Guardrails policies found at organization level" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-15", + finding_name="Cross-Account Guardrails Enforcement Check", + finding_details="Bedrock Guardrails policy type is enabled but no policies are attached. Cross-account guardrails are not being enforced across the organization.", + resolution="Create and attach Bedrock Guardrails policies to the organization root, OUs, or specific accounts to enforce consistent safety controls across all foundation model interactions.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-enforcements.html", + severity="High", + status="Failed", + region=region, + ) + ) + else: + findings["details"] = ( + "Cross-account guardrails enforcement is configured" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-15", + finding_name="Cross-Account Guardrails Enforcement Check", + finding_details="Bedrock Guardrails policies are configured at organization level, enabling centralized enforcement of safety controls", + resolution="No action required. Continue monitoring guardrail policy coverage and effectiveness.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-enforcements.html", + severity="Medium", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code == "AWSOrganizationsNotInUseException": + findings["details"] = ( + "AWS Organizations is not enabled for this account" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-15", + finding_name="Cross-Account Guardrails Enforcement Check", + finding_details="AWS Organizations is not in use. Cross-account guardrails can only be configured in Organizations-enabled accounts.", + resolution="Enable AWS Organizations and configure Bedrock Guardrails policies for centralized multi-account enforcement, or accept single-account guardrail management.", + reference="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_bedrock.html", + severity="Medium", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: + findings["details"] = ( + "Insufficient permissions to check organizational policies" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-15", + finding_name="Cross-Account Guardrails Enforcement Check", + finding_details=describe_api_error( + e, "Organizations policy check", region + ), + resolution="Grant organizations:DescribeOrganization and organizations:ListPolicies permissions to the assessment role", + reference="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_permissions_overview.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings - s3_url = f"https://{bucket_name}.s3.amazonaws.com/{file_name}" - logger.info(f"Successfully wrote report to S3: {s3_url}") - return s3_url except Exception as e: - logger.error(f"Error writing to S3: {str(e)}", exc_info=True) - raise + logger.error( + f"Error in check_bedrock_cross_account_guardrails: {str(e)}", exc_info=True + ) + return { + "check_name": "Cross-Account Guardrails Enforcement Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-15", + finding_name="Cross-Account Guardrails Enforcement Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-enforcements.html", + severity="High", + status="Failed", + region=region, + ) + ], + } -def lambda_handler(event, context): +def check_bedrock_guardrail_tier(region: str = "") -> Dict[str, Any]: """ - Main Lambda handler + BR-16: Verify guardrails are using Standard tier (vs Express tier) for enhanced protection """ - logger.info("Starting Bedrock security assessment") - all_findings = [] - + logger.debug("Starting check for Bedrock guardrail tier validation") try: - # Extract target region from Step Functions Map state - region = event.get("Region", os.environ.get("AWS_REGION", "us-east-1")) - # IAM is global: only the primary region (Map index 0) runs IAM-only checks. - is_primary_region = int(event.get("RegionIndex", 0)) == 0 - logger.info(f"Scanning region: {region} (primary={is_primary_region})") + findings = { + "check_name": "Guardrail Tier Validation Check", + "status": "PASS", + "details": "", + "csv_data": [], + } - execution_id = event["Execution"]["Name"] + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) - # Initialize permission cache (shared/global IAM data) - logger.info("Initializing IAM permission cache") - permission_cache = get_permissions_cache(execution_id) + try: + # List all guardrails + guardrails_response = bedrock_client.list_guardrails(maxResults=100) + guardrails = guardrails_response.get("guardrails", []) - if not permission_cache: - logger.error( - "Permission cache not found - IAM permission caching may have failed" - ) - permission_cache = {"role_permissions": {}, "user_permissions": {}} + if not guardrails: + findings["details"] = "No Bedrock guardrails found" + findings["csv_data"].append( + create_finding( + check_id="BR-16", + finding_name="Guardrail Tier Validation Check", + finding_details="No Bedrock guardrails configured in this region", + resolution="Create Bedrock guardrails with Standard tier for enhanced content filtering and protection", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-components.html", + severity="Medium", + status="N/A", + region=region, + ) + ) + return findings - # Run global IAM-only checks once (on the primary region) so the same role - # violations are not reported once per scanned region. These run before the - # regional availability gate so they are still emitted even if Bedrock is - # not available in the primary region. - if is_primary_region: - logger.info("Running global AmazonBedrockFullAccess check (BR-01)") - all_findings.append( - check_bedrock_full_access_roles( - permission_cache, region=GLOBAL_REGION_LABEL + suboptimal_guardrails = [] + standard_tier_guardrails = [] + + for guardrail_summary in guardrails: + guardrail_id = guardrail_summary.get("id") + guardrail_name = guardrail_summary.get("name", "unknown") + + if not guardrail_id: + continue + + # Get detailed guardrail configuration + guardrail_detail = bedrock_client.get_guardrail( + guardrailIdentifier=guardrail_id ) - ) + guardrail_config = guardrail_detail.get("guardrail", guardrail_detail) + + # The content-filter tier is reported under + # contentPolicy.tier.tierName. Valid values are CLASSIC and + # STANDARD; STANDARD is the more robust tier. When a guardrail has + # no content policy the field is absent, so default to CLASSIC + # (the baseline) rather than assuming the enhanced tier. + content_policy = guardrail_config.get("contentPolicy", {}) + tier = content_policy.get("tier", {}).get("tierName", "CLASSIC") + + if tier != "STANDARD": + suboptimal_guardrails.append( + {"name": guardrail_name, "id": guardrail_id, "tier": tier} + ) + else: + standard_tier_guardrails.append(guardrail_name) - logger.info("Running global marketplace subscription access check (BR-03)") - all_findings.append( - check_marketplace_subscription_access( - permission_cache, region=GLOBAL_REGION_LABEL + if suboptimal_guardrails: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(suboptimal_guardrails)} guardrails not using STANDARD tier" ) - ) - # logger.info("Running global stale Bedrock access check (BR-14)") - # all_findings.append( - # check_stale_bedrock_access(permission_cache, region=GLOBAL_REGION_LABEL) - # ) + for gr in suboptimal_guardrails: + findings["csv_data"].append( + create_finding( + check_id="BR-16", + finding_name="Guardrail Tier Validation Check", + finding_details=f"Guardrail '{gr['name']}' (ID: {gr['id']}) is using the '{gr['tier']}' content-filter tier instead of 'STANDARD'. The STANDARD tier provides more robust content filtering and broader language support than the CLASSIC tier.", + resolution="Update the guardrail to use the STANDARD content-filter tier for improved contextual understanding, better prompt attack filtering (distinguishing jailbreaks from prompt injection), and broader language support. The STANDARD tier requires cross-Region inference. Review pricing implications before upgrading.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-components.html", + severity="Medium", + status="Failed", + region=region, + ) + ) - # Verify Bedrock is available in this region. A ValidationException here - # (logging simply not configured) still means the service is reachable, - # so only an endpoint connection failure or a region-not-enabled error - # should short-circuit the assessment. - bedrock_unavailable = False - unavailable_detail = "" - try: + if standard_tier_guardrails: + findings["csv_data"].append( + create_finding( + check_id="BR-16", + finding_name="Guardrail Tier Validation Check", + finding_details=f"{len(standard_tier_guardrails)} guardrails are using the STANDARD content-filter tier with enhanced protection capabilities", + resolution="No action required. Continue monitoring guardrail effectiveness.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-16", + finding_name="Guardrail Tier Validation Check", + finding_details=describe_api_error( + e, "Guardrail tier check", region + ), + resolution="Grant bedrock:ListGuardrails and bedrock:GetGuardrail permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error(f"Error in check_bedrock_guardrail_tier: {str(e)}", exc_info=True) + return { + "check_name": "Guardrail Tier Validation Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-16", + finding_name="Guardrail Tier Validation Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html", + severity="Medium", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_custom_model_kms_encryption(region: str = "") -> Dict[str, Any]: + """ + BR-17: Verify fine-tuned/customized models use customer-managed KMS keys instead of AWS-owned keys + Note: This extends the existing check_bedrock_custom_model_encryption (BR-11) to specifically verify KMS key type + """ + logger.debug("Starting check for custom model KMS encryption") + try: + findings = { + "check_name": "Custom Model Customer-Managed KMS Encryption Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) + + try: + # Get custom models using paginator + paginator = bedrock_client.get_paginator("list_custom_models") + custom_models = [] + for page in paginator.paginate(): + custom_models.extend(page.get("modelSummaries", [])) + + if not custom_models: + findings["details"] = "No custom models found" + findings["csv_data"].append( + create_finding( + check_id="BR-17", + finding_name="Custom Model Customer-Managed KMS Encryption Check", + finding_details="No custom (fine-tuned) Bedrock models found in this region", + resolution="When creating custom models, specify a customer-managed KMS key for encryption to maintain control over encryption keys", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-custom-job.html", + severity="High", + status="N/A", + region=region, + ) + ) + return findings + + models_with_aws_keys = [] + models_with_customer_keys = [] + models_unknown = [] + + for model in custom_models: + model_arn = model.get("modelArn") + model_name = model.get("modelName", "unknown") + + # Get model details to check encryption + try: + model_detail = bedrock_client.get_custom_model( + modelIdentifier=model_arn + ) + + # GetCustomModel reports the encryption key as modelKmsKeyArn. + # When absent, the model is encrypted with an AWS-owned key. + kms_key_id = model_detail.get("modelKmsKeyArn") + + if not kms_key_id: + # No KMS key specified = AWS-owned key + models_with_aws_keys.append( + {"name": model_name, "arn": model_arn} + ) + elif kms_key_id.startswith("arn:aws:kms"): + # Customer-managed KMS key + models_with_customer_keys.append(model_name) + else: + # Unknown key format + models_unknown.append(model_name) + + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get("Code", "") + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning( + f"Could not get details for model {model_name}: {error_code}" + ) + models_unknown.append(model_name) + + if models_with_aws_keys: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(models_with_aws_keys)} custom models using AWS-owned keys" + ) + + for model_info in models_with_aws_keys: + findings["csv_data"].append( + create_finding( + check_id="BR-17", + finding_name="Custom Model Customer-Managed KMS Encryption Check", + finding_details=f"Custom model '{model_info['name']}' uses AWS-owned encryption keys instead of customer-managed KMS keys. This limits your control over key rotation, access policies, and audit trail.", + resolution="When creating new custom models, specify a customer-managed KMS key using the customizationConfig.kmsKeyArn parameter. For existing models, consider retraining with customer-managed KMS encryption. Ensure KMS key grants allow Amazon Bedrock service access.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-custom-job.html", + severity="High", + status="Failed", + region=region, + ) + ) + + if models_with_customer_keys: + findings["csv_data"].append( + create_finding( + check_id="BR-17", + finding_name="Custom Model Customer-Managed KMS Encryption Check", + finding_details=f"{len(models_with_customer_keys)} custom models are using customer-managed KMS keys for encryption", + resolution="No action required. Continue using customer-managed keys for new custom models.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-custom-job.html", + severity="Medium", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-17", + finding_name="Custom Model Customer-Managed KMS Encryption Check", + finding_details=describe_api_error( + e, "Custom model encryption check", region + ), + resolution="Grant bedrock:ListCustomModels and bedrock:GetCustomModel permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="High", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_custom_model_kms_encryption: {str(e)}", + exc_info=True, + ) + return { + "check_name": "Custom Model Customer-Managed KMS Encryption Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-17", + finding_name="Custom Model Customer-Managed KMS Encryption Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-custom-job.html", + severity="High", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_model_evaluations(region: str = "") -> Dict[str, Any]: + """ + BR-18: Check if model evaluation jobs exist for foundation models to assess safety metrics + """ + logger.debug("Starting check for Bedrock model evaluation implementation") + try: + findings = { + "check_name": "Model Evaluation Implementation Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) + + try: + # List model evaluation jobs + eval_jobs_response = bedrock_client.list_evaluation_jobs(maxResults=100) + eval_jobs = eval_jobs_response.get("jobSummaries", []) + + if not eval_jobs: + findings["status"] = "WARN" + findings["details"] = "No model evaluation jobs found" + findings["csv_data"].append( + create_finding( + check_id="BR-18", + finding_name="Model Evaluation Implementation Check", + finding_details="No Bedrock model evaluation jobs found. Model evaluation helps assess toxicity, accuracy, semantic robustness, and other safety metrics before production deployment.", + resolution="Create model evaluation jobs using Amazon Bedrock Evaluations to assess foundation model performance against safety and quality metrics. Use built-in datasets or custom test sets. Enable LLM-as-a-judge evaluation for comprehensive assessment.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + return findings + + # Analyze evaluation jobs + recent_evaluations = [] + thirty_days_ago = datetime.now(timezone.utc) - timedelta(days=30) + + for job in eval_jobs: + job_name = job.get("jobName", "unknown") + job_status = job.get("status", "unknown") + creation_time = job.get("creationTime") + + # Check if evaluation is recent (within 30 days) + is_recent = False + if creation_time: + if isinstance(creation_time, str): + try: + creation_time = datetime.fromisoformat( + creation_time.replace("Z", "+00:00") + ) + except ValueError: + pass + if isinstance(creation_time, datetime): + is_recent = creation_time >= thirty_days_ago + + if is_recent and job_status == "Completed": + recent_evaluations.append(job_name) + + findings["details"] = ( + f"Found {len(eval_jobs)} model evaluation jobs, {len(recent_evaluations)} recent" + ) + + if recent_evaluations: + findings["csv_data"].append( + create_finding( + check_id="BR-18", + finding_name="Model Evaluation Implementation Check", + finding_details=f"Found {len(recent_evaluations)} model evaluation jobs completed in the last 30 days. Regular evaluation helps maintain model quality and safety standards.", + resolution="Continue regular model evaluations. Consider implementing automated evaluation pipelines for continuous model validation. Review evaluation results for safety metrics including toxicity and bias.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", + severity="Low", + status="Passed", + region=region, + ) + ) + else: + findings["status"] = "WARN" + findings["csv_data"].append( + create_finding( + check_id="BR-18", + finding_name="Model Evaluation Implementation Check", + finding_details=f"Found {len(eval_jobs)} total model evaluation jobs, but none completed in the last 30 days. Regular evaluation is recommended for production models.", + resolution="Schedule regular model evaluation runs to assess ongoing model performance and safety. Configure evaluations to include responsible AI metrics like toxicity, accuracy, and robustness.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + error_msg = str(e) + + if "UnknownOperation" in error_msg or "Unknown operation" in error_msg: + findings["details"] = ( + "Model evaluation API not available in this region" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-18", + finding_name="Model Evaluation Implementation Check", + finding_details=describe_api_error( + e, "Model evaluation API", region + ), + resolution="Model evaluation may not be available in all regions. Check AWS documentation for regional availability.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-18", + finding_name="Model Evaluation Implementation Check", + finding_details=describe_api_error( + e, "Model evaluation check", region + ), + resolution="Grant bedrock:ListEvaluationJobs permission to assess model evaluation practices", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_model_evaluations: {str(e)}", exc_info=True + ) + return { + "check_name": "Model Evaluation Implementation Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-18", + finding_name="Model Evaluation Implementation Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", + severity="Medium", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_prompt_flow_validation(region: str = "") -> Dict[str, Any]: + """ + BR-19: Verify Bedrock Agents prompt flows are validated before deployment + """ + logger.debug("Starting check for Bedrock prompt flow validation") + try: + findings = { + "check_name": "Prompt Flow Validation Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_agent_client = boto3.client( + "bedrock-agent", config=boto3_config, region_name=region + ) + + try: + # List all flows + flows_response = bedrock_agent_client.list_flows(maxResults=100) + flows = flows_response.get("flowSummaries", []) + + if not flows: + findings["details"] = "No Bedrock prompt flows found" + findings["csv_data"].append( + create_finding( + check_id="BR-19", + finding_name="Prompt Flow Validation Check", + finding_details="No Bedrock prompt flows configured in this region", + resolution="When creating prompt flows, use the ValidateFlowDefinition API to validate flow definitions before deployment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/flows.html", + severity="Medium", + status="N/A", + region=region, + ) + ) + return findings + + unvalidated_flows = [] + validated_flows = [] + + for flow_summary in flows: + flow_id = flow_summary.get("id") + flow_name = flow_summary.get("name", "unknown") + flow_status = flow_summary.get("status", "unknown") + + if not flow_id: + continue + + # Get detailed flow configuration + try: + flow_detail = bedrock_agent_client.get_flow(flowIdentifier=flow_id) + flow_info = flow_detail.get("flow", flow_detail) + + # GetFlow returns a `validations` array describing problems + # found when the flow was last prepared. Treat any validation + # entry with ERROR severity as a failed validation. A flow that + # is Prepared/Published with no error-level validations is + # considered validated. + validations = flow_info.get("validations", []) + error_validations = [ + v + for v in validations + if str(v.get("severity", "")).upper() == "ERROR" + ] + + if error_validations: + messages = "; ".join( + v.get("message", "unknown") for v in error_validations[:5] + ) + unvalidated_flows.append( + { + "name": flow_name, + "id": flow_id, + "status": flow_status, + "errors": messages, + } + ) + elif flow_status in ["Prepared", "Published"]: + validated_flows.append(flow_name) + else: + # No error-level validations, but the flow has not been + # prepared/published, so it has not been validated for + # deployment. + unvalidated_flows.append( + { + "name": flow_name, + "id": flow_id, + "status": flow_status, + "errors": "", + } + ) + + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get("Code", "") + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning( + f"Could not get details for flow {flow_name}: {error_code}" + ) + + if unvalidated_flows: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(unvalidated_flows)} flows that may not be validated" + ) + + for flow in unvalidated_flows: + if flow.get("errors"): + detail = ( + f"Prompt flow '{flow['name']}' (ID: {flow['id']}) has " + f"validation errors: {flow['errors']}. Unvalidated flows " + "can lead to runtime errors or unexpected behavior." + ) + else: + detail = ( + f"Prompt flow '{flow['name']}' (ID: {flow['id']}) has " + f"status '{flow['status']}' and has not been validated for " + "deployment. Unvalidated flows can lead to runtime errors " + "or unexpected behavior." + ) + findings["csv_data"].append( + create_finding( + check_id="BR-19", + finding_name="Prompt Flow Validation Check", + finding_details=detail, + resolution="Use the ValidateFlowDefinition API to validate the flow definition before preparing or publishing. Fix any validation errors before deployment. Ensure all nodes, connections, and configurations are correct.", + reference="https://docs.aws.amazon.com/bedrock/latest/APIReference/API_agent_ValidateFlowDefinition.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + + if validated_flows: + findings["csv_data"].append( + create_finding( + check_id="BR-19", + finding_name="Prompt Flow Validation Check", + finding_details=f"{len(validated_flows)} prompt flows are validated and prepared for deployment", + resolution="No action required. Continue validating flows before deployment.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/flows.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + error_msg = str(e) + + if "UnknownOperation" in error_msg or "Unknown operation" in error_msg: + findings["details"] = "Prompt flows API not available in this region" + findings["csv_data"].append( + create_finding( + check_id="BR-19", + finding_name="Prompt Flow Validation Check", + finding_details=describe_api_error( + e, "Prompt flows API", region + ), + resolution="Bedrock prompt flows may not be available in all regions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/flows.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-19", + finding_name="Prompt Flow Validation Check", + finding_details=describe_api_error( + e, "Prompt flow check", region + ), + resolution="Grant bedrock-agent:ListFlows and bedrock-agent:GetFlow permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_prompt_flow_validation: {str(e)}", exc_info=True + ) + return { + "check_name": "Prompt Flow Validation Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-19", + finding_name="Prompt Flow Validation Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/flows.html", + severity="Medium", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_knowledge_base_kms_encryption(region: str = "") -> Dict[str, Any]: + """ + BR-20: Verify Knowledge Base vector stores use customer-managed KMS keys (extends BR-09) + """ + logger.debug("Starting check for Knowledge Base customer-managed KMS encryption") + try: + findings = { + "check_name": "Knowledge Base Customer-Managed KMS Encryption Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_agent_client = boto3.client( + "bedrock-agent", config=boto3_config, region_name=region + ) + + try: + # List all knowledge bases + kb_response = bedrock_agent_client.list_knowledge_bases(maxResults=100) + knowledge_bases = kb_response.get("knowledgeBaseSummaries", []) + + if not knowledge_bases: + findings["details"] = "No Bedrock knowledge bases found" + findings["csv_data"].append( + create_finding( + check_id="BR-20", + finding_name="Knowledge Base Customer-Managed KMS Encryption Check", + finding_details="No Bedrock knowledge bases found in this region", + resolution="When creating knowledge bases, specify customer-managed KMS keys for both vector store and data source encryption", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html", + severity="High", + status="N/A", + region=region, + ) + ) + return findings + + kbs_with_aws_keys = [] + kbs_with_customer_keys = [] + kbs_storage_layer_review = [] + kbs_indeterminate = [] + + for kb_summary in knowledge_bases: + kb_id = kb_summary.get("knowledgeBaseId") + kb_name = kb_summary.get("name", "unknown") + + if not kb_id: + continue + + # Get detailed KB configuration + try: + kb_detail = bedrock_agent_client.get_knowledge_base( + knowledgeBaseId=kb_id + ) + kb_config = kb_detail.get("knowledgeBase", {}) + + # The KB `type` (VECTOR | KENDRA | SQL | MANAGED) is the + # authoritative discriminator and is present in every SDK + # version. Only a MANAGED knowledge base exposes the + # customer-managed KMS key directly on the KB, under + # knowledgeBaseConfiguration.managedKnowledgeBaseConfiguration. + # serverSideEncryptionConfiguration.kmsKeyArn. For VECTOR / SQL / + # KENDRA stores the encryption key lives on the underlying + # storage resource and cannot be read from this API, so those + # are flagged for manual review rather than reported as failures. + kb_configuration = kb_config.get("knowledgeBaseConfiguration", {}) + kb_type = kb_configuration.get("type", "") + managed_config = kb_configuration.get( + "managedKnowledgeBaseConfiguration" + ) + storage_config = kb_config.get("storageConfiguration", {}) + storage_type = storage_config.get("type") or kb_type or "Unknown" + + is_managed_type = kb_type == "MANAGED" + + if is_managed_type and managed_config is None: + # The KB is MANAGED but the managedKnowledgeBaseConfiguration + # block is absent from the response. This happens when the + # bundled botocore model predates the field (added in + # botocore 1.43.32); botocore silently strips unmodeled + # fields. Surface as indeterminate rather than failing or + # passing on incomplete data. + kbs_indeterminate.append({"name": kb_name, "id": kb_id}) + elif managed_config is not None: + sse_config = managed_config.get( + "serverSideEncryptionConfiguration", {} + ) + kms_key_arn = sse_config.get("kmsKeyArn") + + if kms_key_arn and kms_key_arn.startswith("arn:aws:kms"): + kbs_with_customer_keys.append(kb_name) + else: + kbs_with_aws_keys.append( + { + "name": kb_name, + "id": kb_id, + "storage_type": "Managed (Amazon Bedrock)", + } + ) + else: + # Custom vector store (VECTOR / SQL / KENDRA): encryption is + # managed at the storage layer and cannot be validated from + # the KB API. + kbs_storage_layer_review.append( + { + "name": kb_name, + "id": kb_id, + "storage_type": storage_type, + } + ) + + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get("Code", "") + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning( + f"Could not get details for KB {kb_name}: {error_code}" + ) + + if kbs_with_aws_keys: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(kbs_with_aws_keys)} managed knowledge bases without a customer-managed KMS key" + ) + + for kb in kbs_with_aws_keys: + findings["csv_data"].append( + create_finding( + check_id="BR-20", + finding_name="Knowledge Base Customer-Managed KMS Encryption Check", + finding_details=f"Managed knowledge base '{kb['name']}' (ID: {kb['id']}) does not have a customer-managed KMS key configured and is encrypted with an AWS-owned key. This limits control over key rotation, access policies, and audit trails.", + resolution="When creating or updating a managed knowledge base, specify a customer-managed KMS key ARN under knowledgeBaseConfiguration.managedKnowledgeBaseConfiguration.serverSideEncryptionConfiguration.kmsKeyArn. Ensure the KMS key policy allows Amazon Bedrock service access.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html", + severity="High", + status="Failed", + region=region, + ) + ) + + if kbs_storage_layer_review: + if findings["status"] == "PASS": + findings["status"] = "WARN" + for kb in kbs_storage_layer_review: + findings["csv_data"].append( + create_finding( + check_id="BR-20", + finding_name="Knowledge Base Customer-Managed KMS Encryption Review", + finding_details=f"Knowledge base '{kb['name']}' (ID: {kb['id']}) uses '{kb['storage_type']}' storage. The vector-store encryption key is managed at the storage layer and cannot be validated from the Knowledge Base API. Verify customer-managed KMS encryption on the underlying store.", + resolution="1. For OpenSearch Serverless: verify the collection uses a customer-managed KMS key\n2. For Amazon RDS/Aurora: verify KMS encryption on the database\n3. For third-party stores (Pinecone, Redis, MongoDB): verify the provider's encryption configuration\n4. Verify the customer-managed KMS key used for transient data during ingestion", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + + if kbs_indeterminate: + if findings["status"] == "PASS": + findings["status"] = "WARN" + for kb in kbs_indeterminate: + findings["csv_data"].append( + create_finding( + check_id="BR-20", + finding_name="Knowledge Base Customer-Managed KMS Encryption Review", + finding_details=f"Knowledge base '{kb['name']}' (ID: {kb['id']}) is a MANAGED knowledge base, but its encryption configuration could not be read from the API response. This typically means the deployed AWS SDK (botocore) predates the managedKnowledgeBaseConfiguration field (added in botocore 1.43.32) and silently dropped it. Customer-managed KMS encryption could not be confirmed.", + resolution="Upgrade the function's bundled boto3/botocore to 1.43.32 or later so the managed knowledge base encryption configuration is returned, then re-run the assessment. Meanwhile, verify the customer-managed KMS key in the Amazon Bedrock console.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + + if kbs_with_customer_keys: + findings["csv_data"].append( + create_finding( + check_id="BR-20", + finding_name="Knowledge Base Customer-Managed KMS Encryption Check", + finding_details=f"{len(kbs_with_customer_keys)} managed knowledge bases are using customer-managed KMS keys", + resolution="No action required. Continue using customer-managed keys for new knowledge bases.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html", + severity="Medium", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-20", + finding_name="Knowledge Base Customer-Managed KMS Encryption Check", + finding_details=describe_api_error( + e, "Knowledge base encryption check", region + ), + resolution="Grant bedrock-agent:ListKnowledgeBases and bedrock-agent:GetKnowledgeBase permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="High", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_knowledge_base_kms_encryption: {str(e)}", + exc_info=True, + ) + return { + "check_name": "Knowledge Base Customer-Managed KMS Encryption Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-20", + finding_name="Knowledge Base Customer-Managed KMS Encryption Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html", + severity="High", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_agent_action_group_iam( + region: str = "", permission_cache: Optional[Dict[str, Any]] = None +) -> Dict[str, Any]: + """ + BR-21: Check if Bedrock Agent action groups use scoped Lambda execution roles (extends BR-08) + """ + logger.debug("Starting check for Bedrock Agent action group IAM least privilege") + + if permission_cache is None: + permission_cache = {} + + try: + findings = { + "check_name": "Agent Action Group IAM Least Privilege Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_agent_client = boto3.client( + "bedrock-agent", config=boto3_config, region_name=region + ) + lambda_client = boto3.client("lambda", config=boto3_config, region_name=region) + + try: + # List all agents + agents_response = bedrock_agent_client.list_agents(maxResults=100) + agents = agents_response.get("agentSummaries", []) + + if not agents: + findings["details"] = "No Bedrock agents found" + findings["csv_data"].append( + create_finding( + check_id="BR-21", + finding_name="Agent Action Group IAM Least Privilege Check", + finding_details="No Bedrock agents configured in this region", + resolution="When creating agents with action groups, ensure Lambda execution roles follow least privilege principles", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-security.html", + severity="High", + status="N/A", + region=region, + ) + ) + return findings + + overly_permissive_lambdas = [] + compliant_lambdas = [] + + for agent_summary in agents: + agent_id = agent_summary.get("agentId") + agent_name = agent_summary.get("agentName", "unknown") + + if not agent_id: + continue + + try: + # Get agent action groups + action_groups_response = ( + bedrock_agent_client.list_agent_action_groups( + agentId=agent_id, agentVersion="DRAFT", maxResults=100 + ) + ) + action_groups = action_groups_response.get( + "actionGroupSummaries", [] + ) + + for action_group in action_groups: + # Get action group details + action_group_id = action_group.get("actionGroupId") + action_group_name = action_group.get( + "actionGroupName", "unknown" + ) + + if not action_group_id: + continue + + try: + ag_detail = bedrock_agent_client.get_agent_action_group( + agentId=agent_id, + agentVersion="DRAFT", + actionGroupId=action_group_id, + ) + ag_config = ag_detail.get("agentActionGroup", {}) + + # Get Lambda function ARN if configured + action_group_executor = ag_config.get( + "actionGroupExecutor", {} + ) + lambda_arn = action_group_executor.get("lambda") + + if lambda_arn: + # Extract function name from ARN + function_name = ( + lambda_arn.split(":")[-1] + if ":" in lambda_arn + else lambda_arn + ) + + try: + # Get Lambda function configuration + lambda_config = lambda_client.get_function( + FunctionName=function_name + ) + role_arn = lambda_config.get( + "Configuration", {} + ).get("Role") + + if role_arn: + # Check if role has overly broad permissions + role_name = role_arn.split("/")[-1] + + # Check permission cache for this role. Each + # policy entry is a dict {"name", "document"}, + # so inspect the policy name rather than the + # dict itself. + role_perms = permission_cache.get( + "role_permissions", {} + ).get(role_name, {}) + attached_policies = role_perms.get( + "attached_policies", [] + ) + inline_policies = role_perms.get( + "inline_policies", [] + ) + + # Check for overly permissive managed policies + # by policy name. + has_admin_access = any( + p.get("name") == "AdministratorAccess" + for p in attached_policies + ) + has_full_access = any( + "FullAccess" in (p.get("name") or "") + for p in attached_policies + ) + # Check inline policy documents for an Allow on + # Action "*" / Resource "*" (wildcard access). + has_star_resource = any( + _policy_grants_wildcard(p.get("document")) + for p in inline_policies + ) + + if ( + has_admin_access + or has_full_access + or has_star_resource + ): + if has_admin_access: + issue = "AdministratorAccess" + elif has_full_access: + issue = "a *FullAccess managed policy" + else: + issue = ( + "an inline policy granting wildcard " + 'Action/Resource ("*")' + ) + overly_permissive_lambdas.append( + { + "agent_name": agent_name, + "action_group": action_group_name, + "function_name": function_name, + "role_name": role_name, + "issue": issue, + } + ) + else: + compliant_lambdas.append(function_name) + + except ClientError as lambda_error: + error_code = lambda_error.response.get( + "Error", {} + ).get("Code", "") + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning( + f"Could not get Lambda config for {function_name}: {error_code}" + ) + + except ClientError as ag_error: + error_code = ag_error.response.get("Error", {}).get( + "Code", "" + ) + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning( + f"Could not get action group details: {error_code}" + ) + + except ClientError as list_error: + error_code = list_error.response.get("Error", {}).get("Code", "") + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning( + f"Could not list action groups for agent {agent_name}: {error_code}" + ) + + if overly_permissive_lambdas: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(overly_permissive_lambdas)} Lambda functions with overly permissive IAM roles" + ) + + for lambda_info in overly_permissive_lambdas: + findings["csv_data"].append( + create_finding( + check_id="BR-21", + finding_name="Agent Action Group IAM Least Privilege Check", + finding_details=f"Lambda function '{lambda_info['function_name']}' used by agent '{lambda_info['agent_name']}' action group '{lambda_info['action_group']}' has role '{lambda_info['role_name']}' with {lambda_info['issue']}. This violates least privilege principles.", + resolution="Update the Lambda execution role to use scoped permissions. Remove AdministratorAccess and FullAccess policies. Grant only the specific AWS service permissions needed for the action group's operations. Use Resource-based policies to scope permissions to specific resources.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-security.html", + severity="High", + status="Failed", + region=region, + ) + ) + + if compliant_lambdas: + findings["csv_data"].append( + create_finding( + check_id="BR-21", + finding_name="Agent Action Group IAM Least Privilege Check", + finding_details=f"{len(compliant_lambdas)} Lambda functions are using scoped IAM roles", + resolution="No action required. Continue using least privilege IAM roles for action group Lambda functions.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-security.html", + severity="Medium", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-21", + finding_name="Agent Action Group IAM Least Privilege Check", + finding_details=describe_api_error( + e, "Agent action group IAM check", region + ), + resolution="Grant bedrock-agent:ListAgents, bedrock-agent:ListAgentActionGroups, bedrock-agent:GetAgentActionGroup, and lambda:GetFunction permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="High", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_agent_action_group_iam: {str(e)}", exc_info=True + ) + return { + "check_name": "Agent Action Group IAM Least Privilege Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-21", + finding_name="Agent Action Group IAM Least Privilege Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-security.html", + severity="High", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_service_quotas_throttling(region: str = "") -> Dict[str, Any]: + """ + BR-22: Verify service quotas are configured for model invocation throttling + """ + logger.debug("Starting check for Bedrock service quotas throttling limits") + try: + findings = { + "check_name": "Model Invocation Throttling Limits Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + service_quotas_client = boto3.client( + "service-quotas", config=boto3_config, region_name=region + ) + + try: + # List Bedrock service quotas + quotas_response = service_quotas_client.list_service_quotas( + ServiceCode="bedrock", MaxResults=100 + ) + quotas = quotas_response.get("Quotas", []) + + if not quotas: + findings["details"] = "Could not retrieve Bedrock service quotas" + findings["csv_data"].append( + create_finding( + check_id="BR-22", + finding_name="Model Invocation Throttling Limits Check", + finding_details="Unable to retrieve Bedrock service quotas for this region", + resolution="Verify service quotas access and ensure Bedrock is available in this region", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html", + severity="Medium", + status="N/A", + region=region, + ) + ) + return findings + + # Check for custom quotas (non-default values indicate intentional configuration) + custom_quotas = [] + default_quotas = [] + + for quota in quotas: + quota_name = quota.get("QuotaName", "unknown") + quota_code = quota.get("QuotaCode", "") + + # Check if quota is related to throttling/rate limits + is_throttling_quota = any( + keyword in quota_name.lower() + for keyword in [ + "tokens per minute", + "tpm", + "requests per", + "throughput", + "invocations", + ] + ) + + if is_throttling_quota: + # Check if quota has been customized + default_value = quota.get("Value", 0) + adjustable = quota.get("Adjustable", False) + + # Try to get applied quota (custom value if set) + try: + applied_quota = service_quotas_client.get_service_quota( + ServiceCode="bedrock", QuotaCode=quota_code + ) + applied_value = applied_quota.get("Quota", {}).get( + "Value", default_value + ) + + if applied_value != default_value or not adjustable: + custom_quotas.append( + { + "name": quota_name, + "value": applied_value, + "adjustable": adjustable, + } + ) + else: + default_quotas.append(quota_name) + except ClientError: + # If we can't get applied quota, assume default + if adjustable: + default_quotas.append(quota_name) + + if not custom_quotas and default_quotas: + findings["status"] = "WARN" + findings["details"] = "No custom throttling quotas configured" + findings["csv_data"].append( + create_finding( + check_id="BR-22", + finding_name="Model Invocation Throttling Limits Check", + finding_details="Account is using default Bedrock service quotas for model invocation throttling. Custom quotas help prevent abuse, control costs, and ensure fair resource usage across applications.", + resolution="Review Bedrock service quotas and configure custom limits for model invocation rates (tokens per minute) based on your application requirements. Request quota increases through AWS Service Quotas console if needed. Set up CloudWatch alarms to monitor quota utilization.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + elif custom_quotas: + findings["csv_data"].append( + create_finding( + check_id="BR-22", + finding_name="Model Invocation Throttling Limits Check", + finding_details=f"{len(custom_quotas)} custom throttling quotas are configured. Regular quota review helps maintain appropriate rate limits.", + resolution="Continue monitoring quota utilization. Review and adjust quotas as application requirements change.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + error_msg = str(e) + + if ( + "NoSuchResourceException" in error_msg + or "not found" in error_msg.lower() + ): + findings["details"] = "Bedrock service quotas not available" + findings["csv_data"].append( + create_finding( + check_id="BR-22", + finding_name="Model Invocation Throttling Limits Check", + finding_details="Bedrock service quotas not found in Service Quotas API for this region", + resolution="Bedrock may not be available in this region or quotas may not be published to Service Quotas API", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-22", + finding_name="Model Invocation Throttling Limits Check", + finding_details=describe_api_error( + e, "Service quotas check", region + ), + resolution="Grant servicequotas:ListServiceQuotas and servicequotas:GetServiceQuota permissions", + reference="https://docs.aws.amazon.com/servicequotas/latest/userguide/identity-access-management.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_service_quotas_throttling: {str(e)}", exc_info=True + ) + return { + "check_name": "Model Invocation Throttling Limits Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-22", + finding_name="Model Invocation Throttling Limits Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html", + severity="Medium", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_guardrail_content_filters(region: str = "") -> Dict[str, Any]: + """ + BR-23: Verify guardrails have ALL content filters enabled (extends BR-05) + """ + logger.debug("Starting check for Bedrock guardrail content filter coverage") + try: + findings = { + "check_name": "Guardrail Content Filter Coverage Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) + + try: + # List all guardrails + guardrails_response = bedrock_client.list_guardrails(maxResults=100) + guardrails = guardrails_response.get("guardrails", []) + + if not guardrails: + findings["details"] = "No Bedrock guardrails found" + findings["csv_data"].append( + create_finding( + check_id="BR-23", + finding_name="Guardrail Content Filter Coverage Check", + finding_details="No Bedrock guardrails configured in this region", + resolution="Create guardrails with all content filters enabled (hate, insults, sexual, violence) with appropriate thresholds", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-content-filters.html", + severity="High", + status="N/A", + region=region, + ) + ) + return findings + + incomplete_guardrails = [] + complete_guardrails = [] + + for guardrail_summary in guardrails: + guardrail_id = guardrail_summary.get("id") + guardrail_name = guardrail_summary.get("name", "unknown") + + if not guardrail_id: + continue + + # Get detailed guardrail configuration + guardrail_detail = bedrock_client.get_guardrail( + guardrailIdentifier=guardrail_id + ) + guardrail_config = guardrail_detail.get("guardrail", guardrail_detail) + + # Check content filter configuration. GetGuardrail reports the + # configured filters under contentPolicy.filters (the *Config + # field names are part of the Create/Update request shape, not the + # response). + content_policy = guardrail_config.get("contentPolicy", {}) + filters_config = content_policy.get("filters", []) + + # Required filter types + required_filters = {"HATE", "INSULTS", "SEXUAL", "VIOLENCE"} + configured_filters = set() + missing_filters = [] + + for filter_item in filters_config: + filter_type = filter_item.get("type") + input_strength = filter_item.get("inputStrength", "NONE") + output_strength = filter_item.get("outputStrength", "NONE") + + if filter_type in required_filters: + # Filter is considered configured if it has any strength other than NONE + if input_strength != "NONE" or output_strength != "NONE": + configured_filters.add(filter_type) + + # Find missing filters + missing_filters = required_filters - configured_filters + + if missing_filters: + incomplete_guardrails.append( + { + "name": guardrail_name, + "id": guardrail_id, + "missing": list(missing_filters), + } + ) + else: + complete_guardrails.append(guardrail_name) + + if incomplete_guardrails: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(incomplete_guardrails)} guardrails with incomplete content filter coverage" + ) + + for gr in incomplete_guardrails: + findings["csv_data"].append( + create_finding( + check_id="BR-23", + finding_name="Guardrail Content Filter Coverage Check", + finding_details=f"Guardrail '{gr['name']}' (ID: {gr['id']}) is missing content filters: {', '.join(gr['missing'])}. Complete content filter coverage is essential for comprehensive content safety.", + resolution="Update guardrail to enable all content filters (HATE, INSULTS, SEXUAL, VIOLENCE). Configure appropriate threshold levels (LOW, MEDIUM, HIGH) for both input and output filtering based on your use case. Review AWS documentation for threshold guidance.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-content-filters.html", + severity="High", + status="Failed", + region=region, + ) + ) + + if complete_guardrails: + findings["csv_data"].append( + create_finding( + check_id="BR-23", + finding_name="Guardrail Content Filter Coverage Check", + finding_details=f"{len(complete_guardrails)} guardrails have complete content filter coverage (hate, insults, sexual, violence)", + resolution="No action required. Continue monitoring filter effectiveness and adjust thresholds as needed.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-content-filters.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-23", + finding_name="Guardrail Content Filter Coverage Check", + finding_details=describe_api_error( + e, "Guardrail content filter check", region + ), + resolution="Grant bedrock:ListGuardrails and bedrock:GetGuardrail permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="High", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_guardrail_content_filters: {str(e)}", exc_info=True + ) + return { + "check_name": "Guardrail Content Filter Coverage Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-23", + finding_name="Guardrail Content Filter Coverage Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-content-filters.html", + severity="High", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_automated_reasoning_policy(region: str = "") -> Dict[str, Any]: + """ + BR-24: Check if Automated Reasoning policies are configured on guardrails + """ + logger.debug("Starting check for Bedrock Automated Reasoning policy implementation") + try: + findings = { + "check_name": "Automated Reasoning Policy Implementation Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) + + try: + # List all guardrails + guardrails_response = bedrock_client.list_guardrails(maxResults=100) + guardrails = guardrails_response.get("guardrails", []) + + if not guardrails: + findings["details"] = "No Bedrock guardrails found" + findings["csv_data"].append( + create_finding( + check_id="BR-24", + finding_name="Automated Reasoning Policy Implementation Check", + finding_details="No Bedrock guardrails configured in this region", + resolution="Create guardrails with Automated Reasoning policies for formal verification of model responses", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-automated-reasoning.html", + severity="Medium", + status="N/A", + region=region, + ) + ) + return findings + + without_ar_policy = [] + with_ar_policy = [] + + for guardrail_summary in guardrails: + guardrail_id = guardrail_summary.get("id") + guardrail_name = guardrail_summary.get("name", "unknown") + + if not guardrail_id: + continue + + # Get detailed guardrail configuration + try: + guardrail_detail = bedrock_client.get_guardrail( + guardrailIdentifier=guardrail_id + ) + guardrail_config = guardrail_detail.get( + "guardrail", guardrail_detail + ) + + # GetGuardrail reports Automated Reasoning under + # automatedReasoningPolicy.policies. The guardrail has a policy + # configured when that list is non-empty. + ar_policy = guardrail_config.get("automatedReasoningPolicy") or {} + has_ar_policy = bool(ar_policy.get("policies")) + + if not has_ar_policy: + without_ar_policy.append( + {"name": guardrail_name, "id": guardrail_id} + ) + else: + with_ar_policy.append(guardrail_name) + + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get("Code", "") + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning( + f"Could not get details for guardrail {guardrail_name}: {error_code}" + ) + + if without_ar_policy: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(without_ar_policy)} guardrails without Automated Reasoning policies" + ) + + for gr in without_ar_policy: + findings["csv_data"].append( + create_finding( + check_id="BR-24", + finding_name="Automated Reasoning Policy Implementation Check", + finding_details=f"Guardrail '{gr['name']}' (ID: {gr['id']}) does not have an Automated Reasoning policy configured. Automated Reasoning provides formal verification of model responses against defined policies.", + resolution="Configure Automated Reasoning policies on guardrails to mathematically verify model responses. Define policies that specify allowed and disallowed behaviors. Use for high-assurance use cases where formal verification is required.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-automated-reasoning.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + + if with_ar_policy: + findings["csv_data"].append( + create_finding( + check_id="BR-24", + finding_name="Automated Reasoning Policy Implementation Check", + finding_details=f"{len(with_ar_policy)} guardrails have Automated Reasoning policies configured for formal verification", + resolution="No action required. Continue using Automated Reasoning for high-assurance verification.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-automated-reasoning.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + error_msg = str(e) + + if "UnknownOperation" in error_msg or "Unknown operation" in error_msg: + findings["details"] = ( + "Automated Reasoning feature not available in this region" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-24", + finding_name="Automated Reasoning Policy Implementation Check", + finding_details=describe_api_error( + e, "Automated Reasoning API", region + ), + resolution="Automated Reasoning may not be available in all regions. Check AWS documentation for regional availability.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-automated-reasoning.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-24", + finding_name="Automated Reasoning Policy Implementation Check", + finding_details=describe_api_error( + e, "Automated Reasoning policy check", region + ), + resolution="Grant bedrock:ListGuardrails and bedrock:GetGuardrail permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_automated_reasoning_policy: {str(e)}", + exc_info=True, + ) + return { + "check_name": "Automated Reasoning Policy Implementation Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-24", + finding_name="Automated Reasoning Policy Implementation Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-automated-reasoning.html", + severity="Medium", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_rag_evaluation_jobs(region: str = "") -> Dict[str, Any]: + """ + BR-25: Verify RAG applications have evaluation jobs configured + """ + logger.debug("Starting check for Bedrock RAG evaluation jobs") + try: + findings = { + "check_name": "RAG Evaluation Jobs Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_agent_client = boto3.client( + "bedrock-agent", config=boto3_config, region_name=region + ) + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) + + try: + # List all knowledge bases + kb_response = bedrock_agent_client.list_knowledge_bases(maxResults=100) + knowledge_bases = kb_response.get("knowledgeBaseSummaries", []) + + if not knowledge_bases: + findings["details"] = "No knowledge bases found" + findings["csv_data"].append( + create_finding( + check_id="BR-25", + finding_name="RAG Evaluation Jobs Check", + finding_details="No Bedrock knowledge bases found in this region", + resolution="When implementing RAG applications, configure evaluation jobs to assess context relevance, response correctness, and prevent hallucinations", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-rag.html", + severity="Low", + status="N/A", + region=region, + ) + ) + return findings + + # List evaluation jobs (filter for RAG-related evaluations) + try: + eval_jobs_response = bedrock_client.list_evaluation_jobs(maxResults=100) + eval_jobs = eval_jobs_response.get("jobSummaries", []) + + # Map knowledge bases to evaluation jobs + kbs_with_evals = set() + recent_evaluations = [] + thirty_days_ago = datetime.now(timezone.utc) - timedelta(days=30) + + for job in eval_jobs: + job_name = job.get("jobName", "") + job_status = job.get("status", "unknown") + creation_time = job.get("creationTime") + + # Check if this is a RAG evaluation (simple heuristic: name contains kb id or "rag") + is_rag_eval = "rag" in job_name.lower() or any( + kb["knowledgeBaseId"] in job_name for kb in knowledge_bases + ) + + if is_rag_eval: + # Check if evaluation is recent + is_recent = False + if creation_time: + if isinstance(creation_time, str): + try: + creation_time = datetime.fromisoformat( + creation_time.replace("Z", "+00:00") + ) + except ValueError: + pass + if isinstance(creation_time, datetime): + is_recent = creation_time >= thirty_days_ago + + if is_recent and job_status == "Completed": + recent_evaluations.append(job_name) + # Try to identify which KB this evaluation is for + for kb in knowledge_bases: + if kb["knowledgeBaseId"] in job_name: + kbs_with_evals.add(kb["knowledgeBaseId"]) + + kbs_without_evals = [ + kb + for kb in knowledge_bases + if kb["knowledgeBaseId"] not in kbs_with_evals + ] + + if kbs_without_evals: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(kbs_without_evals)} knowledge bases without recent RAG evaluation jobs" + ) + + for kb in kbs_without_evals: + findings["csv_data"].append( + create_finding( + check_id="BR-25", + finding_name="RAG Evaluation Jobs Check", + finding_details=f"Knowledge base '{kb['name']}' (ID: {kb['knowledgeBaseId']}) does not have recent RAG evaluation jobs. RAG evaluations assess context relevance, response correctness, faithfulness, and harmfulness to prevent hallucinations.", + resolution="Create RAG evaluation jobs for knowledge bases using Amazon Bedrock Model Evaluation. Configure evaluations to test context relevance, answer correctness, and faithfulness metrics. Run evaluations regularly (monthly or after significant KB updates) to maintain quality.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-rag.html", + severity="Low", + status="Failed", + region=region, + ) + ) + + if recent_evaluations: + findings["csv_data"].append( + create_finding( + check_id="BR-25", + finding_name="RAG Evaluation Jobs Check", + finding_details=f"Found {len(recent_evaluations)} recent RAG evaluation jobs. Regular RAG evaluations help maintain response quality and prevent hallucinations.", + resolution="Continue regular RAG evaluations. Review evaluation results and adjust retrieval strategies or knowledge base content as needed.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-rag.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as eval_error: + error_code = eval_error.response.get("Error", {}).get("Code", "") + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning(f"Could not list evaluation jobs: {error_code}") + # Continue check even if evaluation jobs API fails + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-25", + finding_name="RAG Evaluation Jobs Check", + finding_details=describe_api_error( + e, "RAG evaluation check", region + ), + resolution="Grant bedrock-agent:ListKnowledgeBases and bedrock:ListEvaluationJobs permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Low", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_rag_evaluation_jobs: {str(e)}", exc_info=True + ) + return { + "check_name": "RAG Evaluation Jobs Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-25", + finding_name="RAG Evaluation Jobs Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-rag.html", + severity="Low", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_guardrail_pii_filters(region: str = "") -> Dict[str, Any]: + """ + BR-26: Verify guardrails configure sensitive-information (PII) protection + (extends BR-23, which only covers the harmful-content filters). + """ + logger.debug("Starting check for Bedrock guardrail sensitive-information filters") + try: + findings = { + "check_name": "Guardrail Sensitive Information Filter Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) + + try: + guardrails_response = bedrock_client.list_guardrails(maxResults=100) + guardrails = guardrails_response.get("guardrails", []) + + if not guardrails: + findings["details"] = "No Bedrock guardrails found" + findings["csv_data"].append( + create_finding( + check_id="BR-26", + finding_name="Guardrail Sensitive Information Filter Check", + finding_details="No Bedrock guardrails configured in this region", + resolution="Create guardrails with sensitive-information filters (PII entities and/or regex patterns) to detect and redact sensitive data in prompts and model responses", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-sensitive-filters.html", + severity="High", + status="N/A", + region=region, + ) + ) + return findings + + guardrails_without_pii = [] + guardrails_with_pii = [] + + for guardrail_summary in guardrails: + guardrail_id = guardrail_summary.get("id") + guardrail_name = guardrail_summary.get("name", "unknown") + + if not guardrail_id: + continue + + guardrail_detail = bedrock_client.get_guardrail( + guardrailIdentifier=guardrail_id + ) + guardrail_config = guardrail_detail.get("guardrail", guardrail_detail) + + # GetGuardrail reports PII protection under + # sensitiveInformationPolicy.piiEntities and .regexes. + sensitive_policy = guardrail_config.get( + "sensitiveInformationPolicy", {} + ) + pii_entities = sensitive_policy.get("piiEntities", []) + regexes = sensitive_policy.get("regexes", []) + + if pii_entities or regexes: + guardrails_with_pii.append(guardrail_name) + else: + guardrails_without_pii.append( + {"name": guardrail_name, "id": guardrail_id} + ) + + if guardrails_without_pii: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(guardrails_without_pii)} guardrails without sensitive-information filters" + ) + + for gr in guardrails_without_pii: + findings["csv_data"].append( + create_finding( + check_id="BR-26", + finding_name="Guardrail Sensitive Information Filter Check", + finding_details=f"Guardrail '{gr['name']}' (ID: {gr['id']}) has no sensitive-information filters configured (no PII entities or regex patterns). Prompts and model responses are not screened for sensitive data such as PII.", + resolution="Configure sensitive-information filters on the guardrail: add PII entity types (e.g. NAME, EMAIL, SSN, CREDIT_DEBIT_CARD_NUMBER) and/or custom regex patterns, and set the appropriate BLOCK or ANONYMIZE action for input and output.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-sensitive-filters.html", + severity="High", + status="Failed", + region=region, + ) + ) + + if guardrails_with_pii: + findings["csv_data"].append( + create_finding( + check_id="BR-26", + finding_name="Guardrail Sensitive Information Filter Check", + finding_details=f"{len(guardrails_with_pii)} guardrails have sensitive-information (PII) filters configured", + resolution="No action required. Periodically review the PII entity types and regex patterns to ensure coverage matches your data.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-sensitive-filters.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-26", + finding_name="Guardrail Sensitive Information Filter Check", + finding_details=describe_api_error( + e, "Guardrail sensitive information check", region + ), + resolution="Grant bedrock:ListGuardrails and bedrock:GetGuardrail permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="High", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_guardrail_pii_filters: {str(e)}", exc_info=True + ) + return { + "check_name": "Guardrail Sensitive Information Filter Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-26", + finding_name="Guardrail Sensitive Information Filter Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-sensitive-filters.html", + severity="High", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_guardrail_contextual_grounding(region: str = "") -> Dict[str, Any]: + """ + BR-27: Verify guardrails enable contextual grounding checks to detect + hallucinations and irrelevant responses (extends BR-05). + """ + logger.debug("Starting check for Bedrock guardrail contextual grounding") + try: + findings = { + "check_name": "Guardrail Contextual Grounding Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) + + try: + guardrails_response = bedrock_client.list_guardrails(maxResults=100) + guardrails = guardrails_response.get("guardrails", []) + + if not guardrails: + findings["details"] = "No Bedrock guardrails found" + findings["csv_data"].append( + create_finding( + check_id="BR-27", + finding_name="Guardrail Contextual Grounding Check", + finding_details="No Bedrock guardrails configured in this region", + resolution="Create guardrails with contextual grounding checks to detect hallucinations (ungrounded responses) and irrelevant answers, especially for RAG applications", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-contextual-grounding-check.html", + severity="Medium", + status="N/A", + region=region, + ) + ) + return findings + + guardrails_without_grounding = [] + guardrails_with_grounding = [] + + for guardrail_summary in guardrails: + guardrail_id = guardrail_summary.get("id") + guardrail_name = guardrail_summary.get("name", "unknown") + + if not guardrail_id: + continue + + guardrail_detail = bedrock_client.get_guardrail( + guardrailIdentifier=guardrail_id + ) + guardrail_config = guardrail_detail.get("guardrail", guardrail_detail) + + # GetGuardrail reports grounding/relevance checks under + # contextualGroundingPolicy.filters. A filter is active when it + # is enabled (the enabled flag defaults to True when omitted). + grounding_policy = guardrail_config.get("contextualGroundingPolicy", {}) + grounding_filters = grounding_policy.get("filters", []) + active_filters = [ + f for f in grounding_filters if f.get("enabled", True) + ] + + if active_filters: + guardrails_with_grounding.append(guardrail_name) + else: + guardrails_without_grounding.append( + {"name": guardrail_name, "id": guardrail_id} + ) + + if guardrails_without_grounding: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(guardrails_without_grounding)} guardrails without contextual grounding checks" + ) + + for gr in guardrails_without_grounding: + findings["csv_data"].append( + create_finding( + check_id="BR-27", + finding_name="Guardrail Contextual Grounding Check", + finding_details=f"Guardrail '{gr['name']}' (ID: {gr['id']}) does not have contextual grounding checks enabled. Without grounding and relevance checks, the guardrail cannot detect hallucinated (ungrounded) or off-topic model responses.", + resolution="Enable contextual grounding checks (GROUNDING and RELEVANCE filter types) on the guardrail with appropriate thresholds. This is especially important for RAG applications to ensure responses are grounded in the retrieved source material.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-contextual-grounding-check.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + + if guardrails_with_grounding: + findings["csv_data"].append( + create_finding( + check_id="BR-27", + finding_name="Guardrail Contextual Grounding Check", + finding_details=f"{len(guardrails_with_grounding)} guardrails have contextual grounding checks enabled", + resolution="No action required. Review grounding and relevance thresholds periodically to balance hallucination detection against false positives.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-contextual-grounding-check.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-27", + finding_name="Guardrail Contextual Grounding Check", + finding_details=describe_api_error( + e, "Guardrail contextual grounding check", region + ), + resolution="Grant bedrock:ListGuardrails and bedrock:GetGuardrail permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_guardrail_contextual_grounding: {str(e)}", + exc_info=True, + ) + return { + "check_name": "Guardrail Contextual Grounding Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-27", + finding_name="Guardrail Contextual Grounding Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-contextual-grounding-check.html", + severity="Medium", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_agent_guardrail_association(region: str = "") -> Dict[str, Any]: + """ + BR-28: Verify each Bedrock Agent has a guardrail associated so that agent + interactions are subject to safety controls. + """ + logger.debug("Starting check for Bedrock Agent guardrail association") + try: + findings = { + "check_name": "Agent Guardrail Association Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_agent_client = boto3.client( + "bedrock-agent", config=boto3_config, region_name=region + ) + + try: + # list_agents summaries already include guardrailConfiguration, so no + # per-agent get_agent call is required. + agents = [] + paginator = bedrock_agent_client.get_paginator("list_agents") + for page in paginator.paginate(): + agents.extend(page.get("agentSummaries", [])) + + if not agents: + findings["details"] = "No Bedrock agents found" + findings["csv_data"].append( + create_finding( + check_id="BR-28", + finding_name="Agent Guardrail Association Check", + finding_details="No Bedrock agents configured in this region", + resolution="When creating agents, associate a Bedrock guardrail so agent inputs and responses are filtered for harmful content, PII, and denied topics", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-guardrails.html", + severity="High", + status="N/A", + region=region, + ) + ) + return findings + + agents_without_guardrail = [] + agents_with_guardrail = [] + + for agent in agents: + agent_id = agent.get("agentId") + agent_name = agent.get("agentName", agent_id or "unknown") + + guardrail_config = agent.get("guardrailConfiguration") or {} + if guardrail_config.get("guardrailIdentifier"): + agents_with_guardrail.append(agent_name) + else: + agents_without_guardrail.append( + {"name": agent_name, "id": agent_id} + ) + + if agents_without_guardrail: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(agents_without_guardrail)} agents without an associated guardrail" + ) + + for agent in agents_without_guardrail: + findings["csv_data"].append( + create_finding( + check_id="BR-28", + finding_name="Agent Guardrail Association Check", + finding_details=f"Bedrock agent '{agent['name']}' (ID: {agent['id']}) does not have a guardrail associated. Agent interactions are not subject to content filtering, PII protection, or denied-topic controls.", + resolution="Associate a Bedrock guardrail with the agent by setting guardrailConfiguration (guardrailIdentifier and guardrailVersion) on the agent. Prepare the agent after updating so the change takes effect.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-guardrails.html", + severity="High", + status="Failed", + region=region, + ) + ) + + if agents_with_guardrail: + findings["csv_data"].append( + create_finding( + check_id="BR-28", + finding_name="Agent Guardrail Association Check", + finding_details=f"{len(agents_with_guardrail)} agents have an associated guardrail", + resolution="No action required. Continue associating guardrails with new agents.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-guardrails.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + error_msg = str(e) + if "UnknownOperation" in error_msg or "Unknown operation" in error_msg: + findings["details"] = "Bedrock Agents API not available in this region" + findings["csv_data"].append( + create_finding( + check_id="BR-28", + finding_name="Agent Guardrail Association Check", + finding_details=describe_api_error( + e, "Bedrock Agents API", region + ), + resolution="Bedrock Agents may not be available in all regions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-guardrails.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-28", + finding_name="Agent Guardrail Association Check", + finding_details=describe_api_error( + e, "Agent guardrail association check", region + ), + resolution="Grant bedrock-agent:ListAgents permission", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="High", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_agent_guardrail_association: {str(e)}", + exc_info=True, + ) + return { + "check_name": "Agent Guardrail Association Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-28", + finding_name="Agent Guardrail Association Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-guardrails.html", + severity="High", + status="Failed", + region=region, + ) + ], + } + + +# Agents with an idle session TTL longer than this (in seconds) are flagged. +# Default Bedrock value is 600s (10 min); 3600s (1 hour) is a generous ceiling. +AGENT_MAX_IDLE_SESSION_TTL_SECONDS = 3600 + + +def check_bedrock_agent_idle_session_ttl(region: str = "") -> Dict[str, Any]: + """ + BR-29: Verify Bedrock Agents do not use an excessively long idle session TTL, + which widens the window for session/context reuse abuse. + """ + logger.debug("Starting check for Bedrock Agent idle session TTL") + try: + findings = { + "check_name": "Agent Idle Session TTL Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_agent_client = boto3.client( + "bedrock-agent", config=boto3_config, region_name=region + ) + + try: + agents = [] + paginator = bedrock_agent_client.get_paginator("list_agents") + for page in paginator.paginate(): + agents.extend(page.get("agentSummaries", [])) + + if not agents: + findings["details"] = "No Bedrock agents found" + findings["csv_data"].append( + create_finding( + check_id="BR-29", + finding_name="Agent Idle Session TTL Check", + finding_details="No Bedrock agents configured in this region", + resolution="When creating agents, set a conservative idleSessionTTLInSeconds to limit how long an idle session remains resumable", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-create.html", + severity="Low", + status="N/A", + region=region, + ) + ) + return findings + + agents_long_ttl = [] + agents_ok_ttl = [] + + for agent in agents: + agent_id = agent.get("agentId") + agent_name = agent.get("agentName", agent_id or "unknown") + + if not agent_id: + continue + + # idleSessionTTLInSeconds is only returned by GetAgent, not in the + # list summary. + try: + agent_detail = bedrock_agent_client.get_agent(agentId=agent_id) + agent_config = agent_detail.get("agent", agent_detail) + ttl = agent_config.get("idleSessionTTLInSeconds") + + if ttl is None: + continue + + if ttl > AGENT_MAX_IDLE_SESSION_TTL_SECONDS: + agents_long_ttl.append( + {"name": agent_name, "id": agent_id, "ttl": ttl} + ) + else: + agents_ok_ttl.append(agent_name) + + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get("Code", "") + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning( + f"Could not get details for agent {agent_name}: {error_code}" + ) + + if agents_long_ttl: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(agents_long_ttl)} agents with an idle session TTL above {AGENT_MAX_IDLE_SESSION_TTL_SECONDS} seconds" + ) + + for agent in agents_long_ttl: + findings["csv_data"].append( + create_finding( + check_id="BR-29", + finding_name="Agent Idle Session TTL Check", + finding_details=f"Bedrock agent '{agent['name']}' (ID: {agent['id']}) has an idle session TTL of {agent['ttl']} seconds, which exceeds the recommended maximum of {AGENT_MAX_IDLE_SESSION_TTL_SECONDS} seconds. Long-lived idle sessions widen the window for session and conversation-context reuse.", + resolution=f"Reduce idleSessionTTLInSeconds to {AGENT_MAX_IDLE_SESSION_TTL_SECONDS} seconds or less, based on your application's session requirements, so idle sessions expire promptly.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-create.html", + severity="Low", + status="Failed", + region=region, + ) + ) + + if agents_ok_ttl: + findings["csv_data"].append( + create_finding( + check_id="BR-29", + finding_name="Agent Idle Session TTL Check", + finding_details=f"{len(agents_ok_ttl)} agents use an idle session TTL within the recommended bound", + resolution="No action required.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-create.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + error_msg = str(e) + if "UnknownOperation" in error_msg or "Unknown operation" in error_msg: + findings["details"] = "Bedrock Agents API not available in this region" + findings["csv_data"].append( + create_finding( + check_id="BR-29", + finding_name="Agent Idle Session TTL Check", + finding_details=describe_api_error( + e, "Bedrock Agents API", region + ), + resolution="Bedrock Agents may not be available in all regions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-create.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-29", + finding_name="Agent Idle Session TTL Check", + finding_details=describe_api_error( + e, "Agent idle session TTL check", region + ), + resolution="Grant bedrock-agent:ListAgents and bedrock-agent:GetAgent permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Low", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_agent_idle_session_ttl: {str(e)}", exc_info=True + ) + return { + "check_name": "Agent Idle Session TTL Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-29", + finding_name="Agent Idle Session TTL Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents-create.html", + severity="Low", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_imported_model_kms_encryption(region: str = "") -> Dict[str, Any]: + """ + BR-30: Verify imported custom models use customer-managed KMS keys + (complements BR-11/BR-17, which cover fine-tuned custom models). + """ + logger.debug("Starting check for imported model KMS encryption") + try: + findings = { + "check_name": "Imported Model Customer-Managed KMS Encryption Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) + + try: + imported_models = [] + paginator = bedrock_client.get_paginator("list_imported_models") + for page in paginator.paginate(): + imported_models.extend(page.get("modelSummaries", [])) + + if not imported_models: + findings["details"] = "No imported models found" + findings["csv_data"].append( + create_finding( + check_id="BR-30", + finding_name="Imported Model Customer-Managed KMS Encryption Check", + finding_details="No imported custom Bedrock models found in this region", + resolution="When importing models, specify a customer-managed KMS key for encryption to maintain control over encryption keys", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html", + severity="High", + status="N/A", + region=region, + ) + ) + return findings + + models_with_aws_keys = [] + models_with_customer_keys = [] + + for model in imported_models: + model_arn = model.get("modelArn") + model_name = model.get("modelName", "unknown") + + try: + model_detail = bedrock_client.get_imported_model( + modelIdentifier=model_arn + ) + # GetImportedModel reports the encryption key as modelKmsKeyArn. + kms_key_arn = model_detail.get("modelKmsKeyArn") + + if kms_key_arn and kms_key_arn.startswith("arn:aws:kms"): + models_with_customer_keys.append(model_name) + else: + models_with_aws_keys.append( + {"name": model_name, "arn": model_arn} + ) + + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get("Code", "") + if error_code not in ACCESS_DENIED_ERROR_CODES: + logger.warning( + f"Could not get details for imported model {model_name}: {error_code}" + ) + + if models_with_aws_keys: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(models_with_aws_keys)} imported models without a customer-managed KMS key" + ) + + for model_info in models_with_aws_keys: + findings["csv_data"].append( + create_finding( + check_id="BR-30", + finding_name="Imported Model Customer-Managed KMS Encryption Check", + finding_details=f"Imported model '{model_info['name']}' is not encrypted with a customer-managed KMS key. This limits your control over key rotation, access policies, and audit trail.", + resolution="Re-import the model specifying a customer-managed KMS key (modelKmsKeyArn). Ensure the KMS key policy grants Amazon Bedrock service access.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html", + severity="High", + status="Failed", + region=region, + ) + ) + + if models_with_customer_keys: + findings["csv_data"].append( + create_finding( + check_id="BR-30", + finding_name="Imported Model Customer-Managed KMS Encryption Check", + finding_details=f"{len(models_with_customer_keys)} imported models are using customer-managed KMS keys for encryption", + resolution="No action required. Continue using customer-managed keys for imported models.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html", + severity="Medium", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + error_msg = str(e) + if "UnknownOperation" in error_msg or "Unknown operation" in error_msg: + findings["details"] = "Imported models API not available in this region" + findings["csv_data"].append( + create_finding( + check_id="BR-30", + finding_name="Imported Model Customer-Managed KMS Encryption Check", + finding_details=describe_api_error( + e, "Imported models API", region + ), + resolution="Model import may not be available in all regions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-30", + finding_name="Imported Model Customer-Managed KMS Encryption Check", + finding_details=describe_api_error( + e, "Imported model encryption check", region + ), + resolution="Grant bedrock:ListImportedModels and bedrock:GetImportedModel permissions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="High", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_imported_model_kms_encryption: {str(e)}", + exc_info=True, + ) + return { + "check_name": "Imported Model Customer-Managed KMS Encryption Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-30", + finding_name="Imported Model Customer-Managed KMS Encryption Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html", + severity="High", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_batch_inference_output_encryption( + region: str = "", +) -> Dict[str, Any]: + """ + BR-31: Verify batch inference (model invocation) jobs encrypt their S3 output + with a customer-managed KMS key. + """ + logger.debug("Starting check for batch inference output encryption") + try: + findings = { + "check_name": "Batch Inference Output Encryption Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + bedrock_client = boto3.client( + "bedrock", config=boto3_config, region_name=region + ) + + try: + # list_model_invocation_jobs summaries already include + # outputDataConfig.s3OutputDataConfig.s3EncryptionKeyId, so no + # per-job get_model_invocation_job call is required. + jobs = [] + paginator = bedrock_client.get_paginator("list_model_invocation_jobs") + for page in paginator.paginate(): + jobs.extend(page.get("invocationJobSummaries", [])) + + if not jobs: + findings["details"] = "No batch inference jobs found" + findings["csv_data"].append( + create_finding( + check_id="BR-31", + finding_name="Batch Inference Output Encryption Check", + finding_details="No Bedrock batch inference (model invocation) jobs found in this region", + resolution="When creating batch inference jobs, set outputDataConfig.s3OutputDataConfig.s3EncryptionKeyId to a customer-managed KMS key to encrypt the job output", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html", + severity="Medium", + status="N/A", + region=region, + ) + ) + return findings + + jobs_without_cmk = [] + jobs_with_cmk = [] + + for job in jobs: + job_name = job.get("jobName", "unknown") + output_config = job.get("outputDataConfig", {}) + s3_output = output_config.get("s3OutputDataConfig", {}) + encryption_key = s3_output.get("s3EncryptionKeyId") + + if encryption_key: + jobs_with_cmk.append(job_name) + else: + jobs_without_cmk.append(job_name) + + if jobs_without_cmk: + findings["status"] = "WARN" + findings["details"] = ( + f"Found {len(jobs_without_cmk)} batch inference jobs without a customer-managed KMS output key" + ) + + for job_name in jobs_without_cmk: + findings["csv_data"].append( + create_finding( + check_id="BR-31", + finding_name="Batch Inference Output Encryption Check", + finding_details=f"Batch inference job '{job_name}' does not specify a customer-managed KMS key for its S3 output. Job output (model responses) may be encrypted only with the bucket default or an AWS-managed key.", + resolution="When creating batch inference jobs, set outputDataConfig.s3OutputDataConfig.s3EncryptionKeyId to a customer-managed KMS key, and ensure the destination S3 bucket enforces that key.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + + if jobs_with_cmk: + findings["csv_data"].append( + create_finding( + check_id="BR-31", + finding_name="Batch Inference Output Encryption Check", + finding_details=f"{len(jobs_with_cmk)} batch inference jobs specify a customer-managed KMS key for their S3 output", + resolution="No action required. Continue specifying a customer-managed KMS key for batch inference output.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html", + severity="Low", + status="Passed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + error_msg = str(e) + if "UnknownOperation" in error_msg or "Unknown operation" in error_msg: + findings["details"] = "Batch inference API not available in this region" + findings["csv_data"].append( + create_finding( + check_id="BR-31", + finding_name="Batch Inference Output Encryption Check", + finding_details=describe_api_error( + e, "Batch inference API", region + ), + resolution="Batch inference may not be available in all regions", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-31", + finding_name="Batch Inference Output Encryption Check", + finding_details=describe_api_error( + e, "Batch inference output encryption check", region + ), + resolution="Grant bedrock:ListModelInvocationJobs permission", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_batch_inference_output_encryption: {str(e)}", + exc_info=True, + ) + return { + "check_name": "Batch Inference Output Encryption Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-31", + finding_name="Batch Inference Output Encryption Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html", + severity="Medium", + status="Failed", + region=region, + ) + ], + } + + +def check_bedrock_cloudwatch_alarms(region: str = "") -> Dict[str, Any]: + """ + BR-32: Verify CloudWatch alarms exist on Amazon Bedrock runtime metrics + (AWS/Bedrock namespace) to detect abuse, throttling, and cost spikes. + """ + logger.debug("Starting check for CloudWatch alarms on Bedrock metrics") + try: + findings = { + "check_name": "Bedrock CloudWatch Alarm Check", + "status": "PASS", + "details": "", + "csv_data": [], + } + + # Only assess this when the region actually has Bedrock resources, to + # avoid recommending alarms in regions where Bedrock is unused. + bedrock_footprint_found = detect_bedrock_regional_footprint(region=region) + if bedrock_footprint_found is False: + findings["details"] = "No regional Bedrock resources found" + findings["csv_data"].append( + create_finding( + check_id="BR-32", + finding_name="Bedrock CloudWatch Alarm Check", + finding_details="No regional Bedrock resources found to monitor with CloudWatch alarms", + resolution="No action required", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-runtime-metrics.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + return findings + + cloudwatch_client = boto3.client( + "cloudwatch", config=boto3_config, region_name=region + ) + + try: + bedrock_alarms = [] + paginator = cloudwatch_client.get_paginator("describe_alarms") + for page in paginator.paginate(AlarmTypes=["MetricAlarm"]): + for alarm in page.get("MetricAlarms", []): + # A metric alarm targets Bedrock either directly (Namespace) + # or via a metric-math expression referencing AWS/Bedrock. + if alarm.get("Namespace") == "AWS/Bedrock": + bedrock_alarms.append(alarm.get("AlarmName")) + continue + for metric in alarm.get("Metrics", []): + metric_stat = metric.get("MetricStat", {}) + namespace = metric_stat.get("Metric", {}).get("Namespace", "") + if namespace == "AWS/Bedrock": + bedrock_alarms.append(alarm.get("AlarmName")) + break + + if bedrock_alarms: + findings["details"] = ( + f"Found {len(bedrock_alarms)} CloudWatch alarms on Bedrock metrics" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-32", + finding_name="Bedrock CloudWatch Alarm Check", + finding_details=f"Found {len(bedrock_alarms)} CloudWatch alarm(s) monitoring Amazon Bedrock runtime metrics (AWS/Bedrock namespace).", + resolution="No action required. Review alarm thresholds and notification targets periodically to ensure they still detect abuse, throttling, and cost anomalies.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-runtime-metrics.html", + severity="Low", + status="Passed", + region=region, + ) + ) + else: + findings["status"] = "WARN" + findings["details"] = "No CloudWatch alarms found on Bedrock metrics" + findings["csv_data"].append( + create_finding( + check_id="BR-32", + finding_name="Bedrock CloudWatch Alarm Check", + finding_details="No CloudWatch alarms are configured on Amazon Bedrock runtime metrics (AWS/Bedrock namespace). Without alarms, abuse, denial-of-wallet, sustained throttling, and content-filter spikes can go undetected.", + resolution="Create CloudWatch alarms on AWS/Bedrock runtime metrics such as Invocations, InvocationThrottles, InputTokenCount, OutputTokenCount, and ContentFilteredCount, and route them to an Amazon SNS topic for notification.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-runtime-metrics.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + + except ClientError as e: + error_code = e.response.get("Error", {}).get("Code", "") + if error_code in ACCESS_DENIED_ERROR_CODES: + findings["csv_data"].append( + create_finding( + check_id="BR-32", + finding_name="Bedrock CloudWatch Alarm Check", + finding_details=describe_api_error( + e, "CloudWatch alarm check", region + ), + resolution="Grant cloudwatch:DescribeAlarms permission to the assessment role", + reference="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/iam-access-control-overview-cw.html", + severity="Medium", + status="Failed", + region=region, + ) + ) + else: + raise + + return findings + + except Exception as e: + logger.error( + f"Error in check_bedrock_cloudwatch_alarms: {str(e)}", exc_info=True + ) + return { + "check_name": "Bedrock CloudWatch Alarm Check", + "status": "ERROR", + "details": f"Error during check: {str(e)}", + "csv_data": [ + create_finding( + check_id="BR-32", + finding_name="Bedrock CloudWatch Alarm Check", + finding_details=f"Error during check: {str(e)}", + resolution="Investigate error and retry assessment", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-runtime-metrics.html", + severity="Medium", + status="Failed", + region=region, + ) + ], + } + + +def generate_csv_report(findings: List[Dict[str, Any]]) -> str: + """ + Generate CSV report from all security check findings + """ + logger.debug("Generating CSV report") + csv_buffer = StringIO() + fieldnames = [ + "Check_ID", + "Finding", + "Finding_Details", + "Resolution", + "Reference", + "Severity", + "Status", + "Region", + ] + writer = csv.DictWriter(csv_buffer, fieldnames=fieldnames) + + writer.writeheader() + for finding in findings: + if finding["csv_data"]: + for row in finding["csv_data"]: + writer.writerow(row) + + return csv_buffer.getvalue() + + +def get_current_utc_date(): + return datetime.now(timezone.utc).strftime("%Y/%m/%d") + + +def write_to_s3( + execution_id, csv_content: str, bucket_name: str, region: str = "" +) -> str: + """ + Write CSV report to S3 bucket + """ + logger.debug(f"Writing CSV report to S3 bucket: {bucket_name}") + try: + s3_client = boto3.client("s3", config=boto3_config) + if region: + file_name = f"bedrock_security_report_{execution_id}_{region}.csv" + else: + file_name = f"bedrock_security_report_{execution_id}.csv" + + s3_client.put_object( + Bucket=bucket_name, Key=file_name, Body=csv_content, ContentType="text/csv" + ) + + s3_url = f"https://{bucket_name}.s3.amazonaws.com/{file_name}" + logger.info(f"Successfully wrote report to S3: {s3_url}") + return s3_url + except Exception as e: + logger.error(f"Error writing to S3: {str(e)}", exc_info=True) + raise + + +def lambda_handler(event, context): + """ + Main Lambda handler + """ + logger.info("Starting Bedrock security assessment") + all_findings = [] + + try: + # Extract target region from Step Functions Map state + region = event.get("Region", os.environ.get("AWS_REGION", "us-east-1")) + # IAM is global: only the primary region (Map index 0) runs IAM-only checks. + is_primary_region = int(event.get("RegionIndex", 0)) == 0 + logger.info(f"Scanning region: {region} (primary={is_primary_region})") + + execution_id = event["Execution"]["Name"] + + # Initialize permission cache (shared/global IAM data) + logger.info("Initializing IAM permission cache") + permission_cache = get_permissions_cache(execution_id) + + if not permission_cache: + logger.error( + "Permission cache not found - IAM permission caching may have failed" + ) + permission_cache = {"role_permissions": {}, "user_permissions": {}} + + # Run global IAM-only checks once (on the primary region) so the same role + # violations are not reported once per scanned region. These run before the + # regional availability gate so they are still emitted even if Bedrock is + # not available in the primary region. + if is_primary_region: + logger.info("Running global AmazonBedrockFullAccess check (BR-01)") + all_findings.append( + check_bedrock_full_access_roles( + permission_cache, region=GLOBAL_REGION_LABEL + ) + ) + + logger.info("Running global marketplace subscription access check (BR-03)") + all_findings.append( + check_marketplace_subscription_access( + permission_cache, region=GLOBAL_REGION_LABEL + ) + ) + + # logger.info("Running global stale Bedrock access check (BR-14)") + # all_findings.append( + # check_stale_bedrock_access(permission_cache, region=GLOBAL_REGION_LABEL) + # ) + + # Verify Bedrock is available in this region. A ValidationException here + # (logging simply not configured) still means the service is reachable, + # so only an endpoint connection failure or a region-not-enabled error + # should short-circuit the assessment. + bedrock_unavailable = False + unavailable_detail = "" + try: test_client = boto3.client( "bedrock", config=boto3_config, region_name=region ) @@ -2879,6 +6074,102 @@ def lambda_handler(event, context): flows_guardrails_findings = check_bedrock_flows_guardrails(region=region) all_findings.append(flows_guardrails_findings) + # New security checks (BR-15+) + # BR-15 is a global check (runs once on primary region) + if is_primary_region: + logger.info("Running cross-account guardrails enforcement check (BR-15)") + cross_account_guardrails_findings = check_bedrock_cross_account_guardrails( + region=GLOBAL_REGION_LABEL + ) + all_findings.append(cross_account_guardrails_findings) + + # Regional checks (BR-16 through BR-25) + logger.info("Running guardrail tier validation check (BR-16)") + guardrail_tier_findings = check_bedrock_guardrail_tier(region=region) + all_findings.append(guardrail_tier_findings) + + logger.info( + "Running custom model customer-managed KMS encryption check (BR-17)" + ) + custom_model_kms_findings = check_bedrock_custom_model_kms_encryption( + region=region + ) + all_findings.append(custom_model_kms_findings) + + logger.info("Running model evaluation implementation check (BR-18)") + model_eval_findings = check_bedrock_model_evaluations(region=region) + all_findings.append(model_eval_findings) + + logger.info("Running prompt flow validation check (BR-19)") + prompt_flow_findings = check_bedrock_prompt_flow_validation(region=region) + all_findings.append(prompt_flow_findings) + + logger.info( + "Running knowledge base customer-managed KMS encryption check (BR-20)" + ) + kb_kms_findings = check_bedrock_knowledge_base_kms_encryption(region=region) + all_findings.append(kb_kms_findings) + + logger.info("Running agent action group IAM least privilege check (BR-21)") + agent_action_group_iam_findings = check_bedrock_agent_action_group_iam( + region=region, permission_cache=permission_cache + ) + all_findings.append(agent_action_group_iam_findings) + + logger.info("Running service quotas throttling limits check (BR-22)") + service_quotas_findings = check_bedrock_service_quotas_throttling(region=region) + all_findings.append(service_quotas_findings) + + logger.info("Running guardrail content filter coverage check (BR-23)") + content_filter_findings = check_bedrock_guardrail_content_filters(region=region) + all_findings.append(content_filter_findings) + + logger.info("Running automated reasoning policy implementation check (BR-24)") + automated_reasoning_findings = check_bedrock_automated_reasoning_policy( + region=region + ) + all_findings.append(automated_reasoning_findings) + + logger.info("Running RAG evaluation jobs check (BR-25)") + rag_eval_findings = check_bedrock_rag_evaluation_jobs(region=region) + all_findings.append(rag_eval_findings) + + logger.info("Running guardrail sensitive-information filter check (BR-26)") + guardrail_pii_findings = check_bedrock_guardrail_pii_filters(region=region) + all_findings.append(guardrail_pii_findings) + + logger.info("Running guardrail contextual grounding check (BR-27)") + guardrail_grounding_findings = check_bedrock_guardrail_contextual_grounding( + region=region + ) + all_findings.append(guardrail_grounding_findings) + + logger.info("Running agent guardrail association check (BR-28)") + agent_guardrail_findings = check_bedrock_agent_guardrail_association( + region=region + ) + all_findings.append(agent_guardrail_findings) + + logger.info("Running agent idle session TTL check (BR-29)") + agent_ttl_findings = check_bedrock_agent_idle_session_ttl(region=region) + all_findings.append(agent_ttl_findings) + + logger.info("Running imported model KMS encryption check (BR-30)") + imported_model_findings = check_bedrock_imported_model_kms_encryption( + region=region + ) + all_findings.append(imported_model_findings) + + logger.info("Running batch inference output encryption check (BR-31)") + batch_inference_findings = check_bedrock_batch_inference_output_encryption( + region=region + ) + all_findings.append(batch_inference_findings) + + logger.info("Running CloudWatch alarm check (BR-32)") + cloudwatch_alarm_findings = check_bedrock_cloudwatch_alarms(region=region) + all_findings.append(cloudwatch_alarm_findings) + # Generate and upload report logger.info("Generating CSV report") csv_content = generate_csv_report(all_findings) diff --git a/aiml-security-assessment/functions/security/bedrock_assessments/requirements.txt b/aiml-security-assessment/functions/security/bedrock_assessments/requirements.txt index 59cc1e9..3fd7d3a 100644 --- a/aiml-security-assessment/functions/security/bedrock_assessments/requirements.txt +++ b/aiml-security-assessment/functions/security/bedrock_assessments/requirements.txt @@ -1 +1,3 @@ -pydantic \ No newline at end of file +boto3==1.43.32 +botocore==1.43.32 +pydantic diff --git a/aiml-security-assessment/functions/security/cleanup_bucket/requirements.txt b/aiml-security-assessment/functions/security/cleanup_bucket/requirements.txt index 1db657b..646ac48 100644 --- a/aiml-security-assessment/functions/security/cleanup_bucket/requirements.txt +++ b/aiml-security-assessment/functions/security/cleanup_bucket/requirements.txt @@ -1 +1,2 @@ -boto3 \ No newline at end of file +boto3==1.43.32 +botocore==1.43.32 \ No newline at end of file diff --git a/aiml-security-assessment/functions/security/finserv_assessments/requirements.txt b/aiml-security-assessment/functions/security/finserv_assessments/requirements.txt index 183ad4f..c67873e 100644 --- a/aiml-security-assessment/functions/security/finserv_assessments/requirements.txt +++ b/aiml-security-assessment/functions/security/finserv_assessments/requirements.txt @@ -1,13 +1,16 @@ # FinServ Security Assessment Lambda — runtime dependencies # -# These pins are scanned by ASH (Automated Security Helper) via Grype + Syft -# during the pre-PR security scan. Keep versions current; if ASH reports a -# CVE against any pinned dep, bump to the patched version before opening the -# PR. See GIT_WORKFLOW.md Step 5 for the ASH workflow. +# These deps are scanned by ASH (Automated Security Helper) via Grype + Syft +# during the pre-PR security scan. boto3/botocore use >= floors (not exact +# pins) so the Lambda can resolve security patches; the floor must stay at or +# above the minimum required by FS-03/FS-06 (see TestRequirementVersionFloors). +# Keep versions current; if ASH reports a CVE against any dep, bump the floor +# to the patched version before opening the PR. See GIT_WORKFLOW.md Step 5 for +# the ASH workflow. # # Dev tooling (ruff, cfn-lint, ash, sam) is NOT pinned here — install those # in your development environment, not into the Lambda image. -boto3>=1.43.21 -botocore>=1.43.21 +boto3>=1.43.32 +botocore>=1.43.32 pydantic>=2.0.0,<3.0.0 diff --git a/aiml-security-assessment/functions/security/generate_consolidated_report/requirements.txt b/aiml-security-assessment/functions/security/generate_consolidated_report/requirements.txt index 0038bc0..f13bb47 100644 --- a/aiml-security-assessment/functions/security/generate_consolidated_report/requirements.txt +++ b/aiml-security-assessment/functions/security/generate_consolidated_report/requirements.txt @@ -1 +1,3 @@ -beautifulsoup4==4.13.4 \ No newline at end of file +boto3==1.43.32 +botocore==1.43.32 +beautifulsoup4==4.13.4 diff --git a/aiml-security-assessment/functions/security/iam_permission_caching/requirements.txt b/aiml-security-assessment/functions/security/iam_permission_caching/requirements.txt index 59cc1e9..3fd7d3a 100644 --- a/aiml-security-assessment/functions/security/iam_permission_caching/requirements.txt +++ b/aiml-security-assessment/functions/security/iam_permission_caching/requirements.txt @@ -1 +1,3 @@ -pydantic \ No newline at end of file +boto3==1.43.32 +botocore==1.43.32 +pydantic diff --git a/aiml-security-assessment/functions/security/resolve_regions/requirements.txt b/aiml-security-assessment/functions/security/resolve_regions/requirements.txt new file mode 100644 index 0000000..a27be86 --- /dev/null +++ b/aiml-security-assessment/functions/security/resolve_regions/requirements.txt @@ -0,0 +1,2 @@ +boto3==1.43.32 +botocore==1.43.32 diff --git a/aiml-security-assessment/functions/security/sagemaker_assessments/requirements.txt b/aiml-security-assessment/functions/security/sagemaker_assessments/requirements.txt index 59cc1e9..3fd7d3a 100644 --- a/aiml-security-assessment/functions/security/sagemaker_assessments/requirements.txt +++ b/aiml-security-assessment/functions/security/sagemaker_assessments/requirements.txt @@ -1 +1,3 @@ -pydantic \ No newline at end of file +boto3==1.43.32 +botocore==1.43.32 +pydantic diff --git a/buildspec.yml b/buildspec.yml index 5c7eb46..a37f2b7 100644 --- a/buildspec.yml +++ b/buildspec.yml @@ -1,4 +1,8 @@ version: 0.2 +env: + variables: + # Opt out of AWS SAM CLI telemetry collection for all phases. + SAM_CLI_TELEMETRY: "0" phases: install: runtime-versions: diff --git a/deployment/1-aiml-security-member-roles.yaml b/deployment/1-aiml-security-member-roles.yaml index 66b8516..ac14561 100644 --- a/deployment/1-aiml-security-member-roles.yaml +++ b/deployment/1-aiml-security-member-roles.yaml @@ -73,6 +73,13 @@ Resources: - bedrock:GetModelCustomizationJob - bedrock:ListFlows - bedrock:GetFlow + # Model evaluation (BR-18, BR-25) + - bedrock:ListEvaluationJobs + # Imported custom models (BR-30) + - bedrock:ListImportedModels + - bedrock:GetImportedModel + # Batch inference jobs (BR-31) + - bedrock:ListModelInvocationJobs Resource: "*" # Bedrock Agent Permissions (Agents for Amazon Bedrock) - Effect: Allow @@ -83,6 +90,21 @@ Resources: - bedrock:GetAgent - bedrock:ListKnowledgeBases - bedrock:GetKnowledgeBase + # Agent action groups (BR-21) + - bedrock:ListAgentActionGroups + - bedrock:GetAgentActionGroup + Resource: "*" + # Bedrock cross-account / quota / monitoring assessment + # (BR-15 organizational guardrails, BR-22 service quotas, + # BR-32 CloudWatch alarms) + - Effect: Allow + Action: + - organizations:DescribeOrganization + - organizations:ListRoots + - organizations:ListPolicies + - servicequotas:ListServiceQuotas + - servicequotas:GetServiceQuota + - cloudwatch:DescribeAlarms Resource: "*" # Bedrock AgentCore Permissions - Effect: Allow @@ -151,6 +173,8 @@ Resources: - Effect: Allow Action: - lambda:ListFunctions + # Read agent action-group Lambda execution role (BR-21) + - lambda:GetFunction Resource: "*" # ECS Permissions - Effect: Allow diff --git a/deployment/2-aiml-security-codebuild.yaml b/deployment/2-aiml-security-codebuild.yaml index f4a1ee8..6af8f4d 100644 --- a/deployment/2-aiml-security-codebuild.yaml +++ b/deployment/2-aiml-security-codebuild.yaml @@ -213,6 +213,24 @@ Resources: - bedrock:GetFlow - bedrock:ListKnowledgeBases - bedrock:GetKnowledgeBase + # Model evaluation (BR-18, BR-25) + - bedrock:ListEvaluationJobs + # Imported custom models (BR-30) + - bedrock:ListImportedModels + - bedrock:GetImportedModel + # Batch inference jobs (BR-31) + - bedrock:ListModelInvocationJobs + # Agent action groups (BR-21) + - bedrock:ListAgentActionGroups + - bedrock:GetAgentActionGroup + # Cross-account guardrails / quotas / monitoring + # (BR-15, BR-22, BR-32) + - organizations:DescribeOrganization + - organizations:ListRoots + - organizations:ListPolicies + - servicequotas:ListServiceQuotas + - servicequotas:GetServiceQuota + - cloudwatch:DescribeAlarms # SageMaker Permissions - sagemaker:ListNotebookInstances - sagemaker:DescribeNotebookInstance @@ -258,6 +276,8 @@ Resources: - cloudtrail:GetEventSelectors - cloudtrail:GetTrailStatus - lambda:ListFunctions + # Read agent action-group Lambda execution role (BR-21) + - lambda:GetFunction - ecs:ListClusters - ecs:ListTasks - ecs:DescribeTasks diff --git a/deployment/aiml-security-single-account.yaml b/deployment/aiml-security-single-account.yaml index 0997c56..1719183 100644 --- a/deployment/aiml-security-single-account.yaml +++ b/deployment/aiml-security-single-account.yaml @@ -174,9 +174,27 @@ Resources: - bedrock:GetModelCustomizationJob - bedrock:ListFlows - bedrock:GetFlow + # Model evaluation (BR-18, BR-25) + - bedrock:ListEvaluationJobs + # Imported custom models (BR-30) + - bedrock:ListImportedModels + - bedrock:GetImportedModel + # Batch inference jobs (BR-31) + - bedrock:ListModelInvocationJobs # Bedrock Agent Permissions (Knowledge Bases, Flows) - bedrock:ListKnowledgeBases - bedrock:GetKnowledgeBase + # Agent action groups (BR-21) + - bedrock:ListAgentActionGroups + - bedrock:GetAgentActionGroup + # Cross-account guardrails / quotas / monitoring + # (BR-15, BR-22, BR-32) + - organizations:DescribeOrganization + - organizations:ListRoots + - organizations:ListPolicies + - servicequotas:ListServiceQuotas + - servicequotas:GetServiceQuota + - cloudwatch:DescribeAlarms # SageMaker Permissions - sagemaker:ListNotebookInstances - sagemaker:DescribeNotebookInstance @@ -222,6 +240,8 @@ Resources: - cloudtrail:GetEventSelectors - cloudtrail:GetTrailStatus - lambda:ListFunctions + # Read agent action-group Lambda execution role (BR-21) + - lambda:GetFunction - ecs:ListClusters - ecs:ListTasks - ecs:DescribeTasks diff --git a/docs/SECURITY_CHECKS.md b/docs/SECURITY_CHECKS.md index fc732bf..236a22a 100644 --- a/docs/SECURITY_CHECKS.md +++ b/docs/SECURITY_CHECKS.md @@ -1,6 +1,6 @@ # Security Checks Reference -This document provides a comprehensive reference for all 116 security checks performed by the AI/ML Security Assessment framework (52 core checks across Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore, plus 64 Financial Services GenAI Risk checks). +This document provides a comprehensive reference for all 134 security checks performed by the AI/ML Security Assessment framework (70 core checks across Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore, plus 64 Financial Services GenAI Risk checks). ## Table of Contents @@ -9,7 +9,7 @@ This document provides a comprehensive reference for all 116 security checks per - [Severity Levels](#severity-levels) - [Status Values](#status-values) - [Amazon SageMaker AI Security Checks (25)](#amazon-sagemaker-ai-security-checks-25) -- [Amazon Bedrock Security Checks (14)](#amazon-bedrock-security-checks-14) +- [Amazon Bedrock Security Checks (32)](#amazon-bedrock-security-checks-32) - [Amazon Bedrock AgentCore Security Checks (13)](#amazon-bedrock-agentcore-security-checks-13) - [Financial Services GenAI Risk Checks (64)](#financial-services-genai-risk-checks-64-additional-5-upstream-extensions) @@ -22,7 +22,7 @@ The framework evaluates your AI/ML workloads against AWS security best practices | Service | Number of Checks | Focus Areas | |---------|------------------|-------------| | Amazon SageMaker AI | 25 | Security Hub controls, encryption, network isolation, IAM, MLOps | -| Amazon Bedrock | 14 | Guardrails, encryption, VPC endpoints, IAM permissions, logging | +| Amazon Bedrock | 32 | Guardrails, content filters, sensitive-information/PII filters, contextual grounding, automated reasoning, encryption (custom, imported, knowledge base, batch inference output), VPC endpoints, IAM permissions, agent guardrail association and least privilege, logging, CloudWatch alarms, cross-account policies, model evaluation, prompt flow validation, RAG evaluation, service quotas | | Amazon Bedrock AgentCore | 13 | VPC configuration, encryption, observability, resource policies | | Financial Services GenAI Risk | 64 | Unbounded consumption, excessive agency, supply chain, training data poisoning, vector weaknesses, non-compliant output, misinformation, harmful output, biased output, PII disclosure, hallucination, prompt injection, improper output handling, off-topic output, out-of-date training data | @@ -35,7 +35,7 @@ Each security check has a unique identifier with a service prefix: | Prefix | Service | Example | |--------|---------|---------| | **SM-XX** | Amazon SageMaker | SM-01, SM-25 | -| **BR-XX** | Amazon Bedrock | BR-01, BR-14 | +| **BR-XX** | Amazon Bedrock | BR-01, BR-32 | | **AC-XX** | Amazon Bedrock AgentCore | AC-01, AC-13 | | **FS-XX** | Financial Services GenAI Risk | FS-01, FS-69 | @@ -198,7 +198,7 @@ Each security check has a unique identifier with a service prefix: --- -## Amazon Bedrock Security Checks (14) +## Amazon Bedrock Security Checks (32) ### BR-01: AWS IAM Least Privilege @@ -270,6 +270,114 @@ Each security check has a unique identifier with a service prefix: - **Severity:** Medium - **Description:** Detects principals with Bedrock permissions that have not used the service recently, using IAM service-last-accessed data. As an IAM-global check, it runs once per execution and is tagged with the `Global` region in multi-region scans. +### BR-15: Cross-Account Guardrails Enforcement + +- **Severity:** High +- **Type:** Global (runs once) +- **Description:** Verifies organization-level guardrails are configured using AWS Organizations Amazon Bedrock policies (the `BEDROCK_POLICY` policy type) for centralized safety control enforcement across all accounts. Checks if running in the AWS Organizations management account, validates the Bedrock policy type is enabled at the organization root, and verifies that Bedrock policies are attached. + +### BR-16: Guardrail Tier Validation + +- **Severity:** Medium +- **Type:** Regional +- **Description:** Verifies guardrails use the `STANDARD` content-filter tier (vs the `CLASSIC` tier) for enhanced protection and broader language support. Lists all guardrails in the region and inspects each guardrail's `contentPolicy.tier.tierName`. The STANDARD tier requires cross-Region inference. + +### BR-17: Custom Model Customer-Managed KMS Encryption + +- **Severity:** High +- **Type:** Regional +- **Description:** Verifies fine-tuned/customized models use customer-managed KMS keys instead of AWS-owned keys for greater control over encryption. Lists all custom models, retrieves model details to check KMS key configuration, and validates KMS key ARN format. This extends the existing BR-11 check by specifically verifying the type of encryption key used. + +### BR-18: Model Evaluation Implementation + +- **Severity:** Medium +- **Type:** Regional +- **Description:** Checks if model evaluation jobs exist to assess safety metrics (toxicity, accuracy, semantic robustness) before production deployment. Lists all model evaluation jobs, identifies recent evaluations (completed within 30 days), and analyzes evaluation configurations for safety metrics. + +### BR-19: Prompt Flow Validation + +- **Severity:** Medium +- **Type:** Regional +- **Description:** Verifies Bedrock Agents prompt flows are validated using `validate_flow_definition` API before deployment to prevent misconfigured flows. Lists all flows in the region, checks for validation records or status, identifies unvalidated flows, and reports flows deployed without validation. + +### BR-20: Knowledge Base Encryption Enhancement + +- **Severity:** High +- **Type:** Regional +- **Description:** Extends existing BR-09 to verify Knowledge Base encryption uses customer-managed KMS keys. Uses the authoritative knowledge base `type` (`VECTOR | KENDRA | SQL | MANAGED`) to decide how to assess each KB: for `MANAGED` knowledge bases it reads `knowledgeBaseConfiguration.managedKnowledgeBaseConfiguration.serverSideEncryptionConfiguration.kmsKeyArn` and fails KBs encrypted with an AWS-owned key; for custom vector stores (OpenSearch, RDS, Pinecone, etc.) the encryption key lives on the underlying storage resource and cannot be read from the KB API, so those are reported as N/A for manual review. If a `MANAGED` KB's encryption block is missing from the API response (deployed botocore older than 1.43.32, which silently drops the unmodeled field), the KB is reported as N/A "indeterminate" rather than a false-positive failure. + +### BR-21: Agent Action Group IAM Least Privilege + +- **Severity:** High +- **Type:** Regional +- **Description:** Extends existing BR-08 to specifically check if Bedrock Agent action groups use scoped Lambda execution roles with minimal permissions. Enumerates agents and their action groups, retrieves Lambda execution roles for each action group, analyzes IAM policies for overly broad permissions (AdministratorAccess, FullAccess, Resource: "*"), and verifies principle of least privilege. + +### BR-22: Model Invocation Throttling Limits + +- **Severity:** Medium +- **Type:** Regional +- **Description:** Verifies service quotas are configured for model invocation throttling to prevent abuse/DoS and control costs. Queries Service Quotas for Bedrock, checks if custom limits are set for on-demand model invocation TPM (tokens per minute), provisioned throughput limits, and concurrent requests. Reports accounts relying solely on default quotas. + +### BR-23: Guardrail Content Filter Coverage + +- **Severity:** High +- **Type:** Regional +- **Description:** Extends existing BR-05 to verify guardrails have ALL content filters enabled (hate, insults, sexual, violence) with appropriate thresholds. For each guardrail, checks content filter configuration for all four filter types, verifies filter thresholds are configured, and reports missing or misconfigured filters. + +### BR-24: Automated Reasoning Policy Implementation + +- **Severity:** Medium +- **Type:** Regional +- **Description:** Checks if Automated Reasoning policies are configured on guardrails for formal verification of model responses. Enumerates guardrails, checks for Automated Reasoning policy configuration, validates policy syntax and enabled state, and reports guardrails without formal verification capability. + +### BR-25: RAG Evaluation Jobs + +- **Severity:** Low +- **Type:** Regional +- **Description:** Verifies RAG applications have evaluation jobs configured to assess context relevance, response correctness, and prevent hallucinations. Lists Knowledge Bases, checks for associated RAG evaluation jobs for each KB, verifies evaluation metrics include context relevance, response correctness, faithfulness, and harmfulness checks. Reports KBs without evaluation jobs. + +### BR-26: Guardrail Sensitive Information Filter + +- **Severity:** High +- **Type:** Regional +- **Description:** Extends BR-23 (which covers the harmful-content filters) to verify guardrails configure sensitive-information protection. For each guardrail, reads `GetGuardrail.sensitiveInformationPolicy` and reports guardrails that have no PII entity types (`piiEntities`) or custom regex patterns (`regexes`) configured, leaving prompts and responses unscreened for sensitive data. + +### BR-27: Guardrail Contextual Grounding Check + +- **Severity:** Medium +- **Type:** Regional +- **Description:** Verifies guardrails enable contextual grounding checks to detect hallucinated (ungrounded) and off-topic model responses. Reads `GetGuardrail.contextualGroundingPolicy.filters` and reports guardrails with no enabled grounding/relevance filters. Complements BR-25 (RAG evaluation) with a runtime control. + +### BR-28: Agent Guardrail Association + +- **Severity:** High +- **Type:** Regional +- **Description:** Verifies each Bedrock Agent has a guardrail associated so agent interactions are subject to content filtering, PII protection, and denied-topic controls. Reads `guardrailConfiguration` from the agent summaries returned by `ListAgents` and reports agents with no guardrail attached. + +### BR-29: Agent Idle Session TTL + +- **Severity:** Low +- **Type:** Regional +- **Description:** Verifies Bedrock Agents do not use an excessively long idle session TTL, which widens the window for session and conversation-context reuse. Reads `GetAgent.idleSessionTTLInSeconds` and reports agents whose TTL exceeds a conservative ceiling (3600 seconds). + +### BR-30: Imported Model Customer-Managed KMS Encryption + +- **Severity:** High +- **Type:** Regional +- **Description:** Complements BR-11/BR-17 by verifying imported custom models use customer-managed KMS keys. Lists imported models and reads `GetImportedModel.modelKmsKeyArn`, reporting models encrypted with AWS-owned keys instead of a customer-managed key. + +### BR-31: Batch Inference Output Encryption + +- **Severity:** Medium +- **Type:** Regional +- **Description:** Verifies batch inference (model invocation) jobs encrypt their S3 output with a customer-managed KMS key. Reads `outputDataConfig.s3OutputDataConfig.s3EncryptionKeyId` from the job summaries returned by `ListModelInvocationJobs` and reports jobs without a customer-managed output key. + +### BR-32: CloudWatch Alarms on Bedrock Metrics + +- **Severity:** Medium +- **Type:** Regional +- **Description:** Verifies CloudWatch alarms exist on Amazon Bedrock runtime metrics (the `AWS/Bedrock` namespace) to detect abuse, denial-of-wallet, sustained throttling, and content-filter spikes. Uses `DescribeAlarms` and matches alarms that target the `AWS/Bedrock` namespace directly or via a metric-math expression. Only assessed in regions that have Bedrock resources. + --- ## Amazon Bedrock AgentCore Security Checks (13) diff --git a/tests/requirements.txt b/tests/requirements.txt index 7d86f05..ad9162e 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -2,6 +2,6 @@ pytest>=7.4.0 pytest-cov>=4.1.0 pytest-mock>=3.11.0 moto[all]>=4.2.0 -boto3>=1.28.0 -botocore>=1.31.0 +boto3==1.43.32 +botocore==1.43.32 pydantic>=2.0.0 diff --git a/tests/test_bedrock_checks.py b/tests/test_bedrock_checks.py index cd909c7..666d0ac 100644 --- a/tests/test_bedrock_checks.py +++ b/tests/test_bedrock_checks.py @@ -9,6 +9,7 @@ - Output schema validity """ +import contextlib import sys import os import importlib.util @@ -1375,3 +1376,1618 @@ def fake_csv(findings): # checks ran (e.g. BR-04 logging, BR-05 guardrails are present). assert "BR-00" not in check_ids assert len(check_ids) > 3 + + # Regional checks BR-26..32 (function name -> check id) that the handler must + # invoke once per scanned region, passing region=region. + NEW_REGIONAL_CHECKS = { + "check_bedrock_guardrail_pii_filters": "BR-26", + "check_bedrock_guardrail_contextual_grounding": "BR-27", + "check_bedrock_agent_guardrail_association": "BR-28", + "check_bedrock_agent_idle_session_ttl": "BR-29", + "check_bedrock_imported_model_kms_encryption": "BR-30", + "check_bedrock_batch_inference_output_encryption": "BR-31", + "check_bedrock_cloudwatch_alarms": "BR-32", + } + + def _run_handler_with_check_spies(self, event): + """Drive the handler down the full regional path with every check function + replaced by a spy that records the region it was called with. The probe + raises ValidationException (service reachable) so all regional checks run. + Returns {check_function_name: region_passed} plus whether BR-15 ran.""" + recorded = {} + + def make_spy(name): + def spy(*args, region="", **kwargs): + recorded[name] = region + return { + "check_name": name, + "status": "PASS", + "details": "", + "csv_data": [], + } + + return spy + + test_client = MagicMock() + test_client.get_model_invocation_logging_configuration.side_effect = ( + _make_client_error("ValidationException") + ) + + # Spy on every check the handler calls so it runs cleanly end-to-end. + spied = [ + "check_bedrock_full_access_roles", + "check_marketplace_subscription_access", + "check_bedrock_access_and_vpc_endpoints", + "check_bedrock_logging_configuration", + "check_bedrock_guardrails", + "check_bedrock_cloudtrail_logging", + "check_bedrock_prompt_management", + "check_bedrock_agent_roles", + "check_bedrock_knowledge_base_encryption", + "check_bedrock_guardrail_iam_enforcement", + "check_bedrock_custom_model_encryption", + "check_bedrock_invocation_log_encryption", + "check_bedrock_flows_guardrails", + "check_bedrock_cross_account_guardrails", # BR-15 (global) + "check_bedrock_guardrail_tier", + "check_bedrock_custom_model_kms_encryption", + "check_bedrock_model_evaluations", + "check_bedrock_prompt_flow_validation", + "check_bedrock_knowledge_base_kms_encryption", + "check_bedrock_agent_action_group_iam", + "check_bedrock_service_quotas_throttling", + "check_bedrock_guardrail_content_filters", + "check_bedrock_automated_reasoning_policy", + "check_bedrock_rag_evaluation_jobs", + *self.NEW_REGIONAL_CHECKS.keys(), + ] + + with contextlib.ExitStack() as stack: + stack.enter_context( + patch.object(bedrock_app.boto3, "client", return_value=test_client) + ) + stack.enter_context( + patch.object( + bedrock_app, + "get_permissions_cache", + return_value={"role_permissions": {}, "user_permissions": {}}, + ) + ) + stack.enter_context( + patch.object(bedrock_app, "generate_csv_report", return_value="csv") + ) + stack.enter_context( + patch.object(bedrock_app, "write_to_s3", return_value="s3://b/r.csv") + ) + for name in spied: + stack.enter_context( + patch.object(bedrock_app, name, side_effect=make_spy(name)) + ) + resp = bedrock_app.lambda_handler(event, None) + + return resp, recorded + + def test_new_regional_checks_run_per_region_non_primary(self): + # On a non-primary region, BR-26..32 must each run with region=, + # and the global BR-15 check must NOT run. + resp, recorded = self._run_handler_with_check_spies( + _bedrock_event(region="eu-west-3", region_index=2) + ) + assert resp["statusCode"] == 200 + for fn_name in self.NEW_REGIONAL_CHECKS: + assert recorded.get(fn_name) == "eu-west-3", ( + f"{fn_name} not run with scanned region: {recorded.get(fn_name)}" + ) + # BR-15 (cross-account guardrails) is global -> skipped on non-primary. + assert "check_bedrock_cross_account_guardrails" not in recorded + + def test_new_regional_checks_run_per_region_primary(self): + # On the primary region, BR-26..32 run with region= and the + # global BR-15 check runs tagged Global. + resp, recorded = self._run_handler_with_check_spies( + _bedrock_event(region="us-east-1", region_index=0) + ) + assert resp["statusCode"] == 200 + for fn_name in self.NEW_REGIONAL_CHECKS: + assert recorded.get(fn_name) == "us-east-1", ( + f"{fn_name} not run with scanned region: {recorded.get(fn_name)}" + ) + # Global check runs once, tagged Global. + assert recorded.get("check_bedrock_cross_account_guardrails") == "Global" + + +# =================================================================== +# BR-15: check_bedrock_cross_account_guardrails +# =================================================================== +class TestBR15CrossAccountGuardrails: + """BR-15: Check AWS Organizations Bedrock Guardrails policies.""" + + @patch("bedrock_app.boto3.client") + def test_br15_organizations_not_enabled_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_cross_account_guardrails + + org_client = MagicMock() + org_client.describe_organization.side_effect = ClientError( + {"Error": {"Code": "AWSOrganizationsNotInUseException"}}, + "DescribeOrganization", + ) + mock_client.return_value = org_client + + result = check(region="Global") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-15" + assert "not in use" in findings[0]["Finding_Details"] + + @patch("bedrock_app.boto3.client") + def test_br15_policy_type_not_enabled_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_cross_account_guardrails + + org_client = MagicMock() + org_client.describe_organization.return_value = { + "Organization": {"MasterAccountId": "123456789012"} + } + org_client.list_roots.return_value = { + "Roots": [ + { + "Id": "r-abc123", + "Arn": "arn:aws:organizations::123456789012:root/o-xyz/r-abc123", + } + ] + } + # No policies returned = policy type not enabled + org_client.list_policies.return_value = {"Policies": []} + + sts_client = MagicMock() + sts_client.get_caller_identity.return_value = {"Account": "123456789012"} + + def client_factory(service, **kwargs): + if service == "organizations": + return org_client + return sts_client + + mock_client.side_effect = client_factory + + result = check(region="Global") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-15" + assert findings[0]["Severity"] == "High" + + @patch("bedrock_app.boto3.client") + def test_br15_policies_configured_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_cross_account_guardrails + + org_client = MagicMock() + org_client.describe_organization.return_value = { + "Organization": {"MasterAccountId": "123456789012"} + } + org_client.list_roots.return_value = { + "Roots": [ + { + "Id": "r-abc123", + "Arn": "arn:aws:organizations::123456789012:root/o-xyz/r-abc123", + "PolicyTypes": [{"Type": "BEDROCK_POLICY", "Status": "ENABLED"}], + } + ] + } + org_client.list_policies.return_value = { + "Policies": [{"Id": "p-123", "Name": "BedrockGuardrailPolicy"}] + } + + sts_client = MagicMock() + sts_client.get_caller_identity.return_value = {"Account": "123456789012"} + + def client_factory(service, **kwargs): + if service == "organizations": + return org_client + return sts_client + + mock_client.side_effect = client_factory + + result = check(region="Global") + findings = extract_csv_data(result) + assert len(findings) >= 1 + passed_findings = [f for f in findings if f["Status"] == "Passed"] + assert len(passed_findings) >= 1 + assert passed_findings[0]["Check_ID"] == "BR-15" + + @patch("bedrock_app.boto3.client") + def test_br15_access_denied_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_cross_account_guardrails + + org_client = MagicMock() + org_client.describe_organization.side_effect = ClientError( + {"Error": {"Code": "AccessDeniedException"}}, "DescribeOrganization" + ) + mock_client.return_value = org_client + + result = check(region="Global") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-15" + + def test_br15_schema_valid(self): + check = bedrock_app.check_bedrock_cross_account_guardrails + with patch("bedrock_app.boto3.client") as mock_client: + org_client = MagicMock() + org_client.describe_organization.side_effect = ClientError( + {"Error": {"Code": "AWSOrganizationsNotInUseException"}}, + "DescribeOrganization", + ) + mock_client.return_value = org_client + result = check(region="Global") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-16: check_bedrock_guardrail_tier +# =================================================================== +class TestBR16GuardrailTier: + """BR-16: Verify guardrails use Standard tier.""" + + @patch("bedrock_app.boto3.client") + def test_br16_no_guardrails_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_tier + + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-16" + + @patch("bedrock_app.boto3.client") + def test_br16_standard_tier_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_tier + + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr-123", "name": "test-guardrail"}] + } + bedrock_client.get_guardrail.return_value = { + "guardrail": {"contentPolicy": {"tier": {"tierName": "STANDARD"}}} + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + passed_findings = [f for f in findings if f["Status"] == "Passed"] + assert len(passed_findings) >= 1 + assert passed_findings[0]["Check_ID"] == "BR-16" + + @patch("bedrock_app.boto3.client") + def test_br16_non_standard_tier_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_tier + + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr-123", "name": "classic-guardrail"}] + } + bedrock_client.get_guardrail.return_value = { + "guardrail": {"contentPolicy": {"tier": {"tierName": "CLASSIC"}}} + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-16" + assert findings[0]["Severity"] == "Medium" + + @patch("bedrock_app.boto3.client") + def test_br16_access_denied_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_tier + + bedrock_client = MagicMock() + bedrock_client.list_guardrails.side_effect = ClientError( + {"Error": {"Code": "AccessDeniedException"}}, "ListGuardrails" + ) + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-16" + + def test_br16_schema_valid(self): + check = bedrock_app.check_bedrock_guardrail_tier + with patch("bedrock_app.boto3.client") as mock_client: + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-17: check_bedrock_custom_model_kms_encryption +# =================================================================== +class TestBR17CustomModelKMSEncryption: + """BR-17: Verify custom models use customer-managed KMS keys.""" + + @patch("bedrock_app.boto3.client") + def test_br17_no_custom_models_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_custom_model_kms_encryption + + bedrock_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [{"modelSummaries": []}] + bedrock_client.get_paginator.return_value = paginator + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-17" + + @patch("bedrock_app.boto3.client") + def test_br17_customer_managed_kms_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_custom_model_kms_encryption + + bedrock_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [ + { + "modelSummaries": [ + { + "modelArn": "arn:aws:bedrock:us-east-1:123456789012:custom-model/my-model", + "modelName": "my-model", + } + ] + } + ] + bedrock_client.get_paginator.return_value = paginator + bedrock_client.get_custom_model.return_value = { + "modelKmsKeyArn": "arn:aws:kms:us-east-1:123456789012:key/abc-123" + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + passed_findings = [f for f in findings if f["Status"] == "Passed"] + assert len(passed_findings) >= 1 + assert passed_findings[0]["Check_ID"] == "BR-17" + + @patch("bedrock_app.boto3.client") + def test_br17_aws_owned_keys_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_custom_model_kms_encryption + + bedrock_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [ + { + "modelSummaries": [ + { + "modelArn": "arn:aws:bedrock:us-east-1:123456789012:custom-model/my-model", + "modelName": "my-model", + } + ] + } + ] + bedrock_client.get_paginator.return_value = paginator + # No KMS key ID = AWS-owned key + bedrock_client.get_custom_model.return_value = {} + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-17" + assert findings[0]["Severity"] == "High" + + @patch("bedrock_app.boto3.client") + def test_br17_access_denied_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_custom_model_kms_encryption + + bedrock_client = MagicMock() + paginator = MagicMock() + paginator.paginate.side_effect = ClientError( + {"Error": {"Code": "AccessDeniedException"}}, "ListCustomModels" + ) + bedrock_client.get_paginator.return_value = paginator + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-17" + + def test_br17_schema_valid(self): + check = bedrock_app.check_bedrock_custom_model_kms_encryption + with patch("bedrock_app.boto3.client") as mock_client: + bedrock_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [{"modelSummaries": []}] + bedrock_client.get_paginator.return_value = paginator + mock_client.return_value = bedrock_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-18: check_bedrock_model_evaluations +# =================================================================== +class TestBR18ModelEvaluations: + """BR-18: Check if model evaluation jobs exist.""" + + @patch("bedrock_app.boto3.client") + def test_br18_no_evaluations_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_model_evaluations + + bedrock_client = MagicMock() + bedrock_client.list_evaluation_jobs.return_value = {"jobSummaries": []} + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-18" + assert findings[0]["Severity"] == "Medium" + + @patch("bedrock_app.boto3.client") + def test_br18_recent_evaluations_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_model_evaluations + + from datetime import datetime, timezone, timedelta + + recent_time = datetime.now(timezone.utc) - timedelta(days=10) + + bedrock_client = MagicMock() + bedrock_client.list_evaluation_jobs.return_value = { + "jobSummaries": [ + { + "jobName": "eval-job-1", + "status": "Completed", + "creationTime": recent_time, + } + ] + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + passed_findings = [f for f in findings if f["Status"] == "Passed"] + assert len(passed_findings) >= 1 + assert passed_findings[0]["Check_ID"] == "BR-18" + + @patch("bedrock_app.boto3.client") + def test_br18_stale_evaluations_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_model_evaluations + + from datetime import datetime, timezone, timedelta + + stale_time = datetime.now(timezone.utc) - timedelta(days=60) + + bedrock_client = MagicMock() + bedrock_client.list_evaluation_jobs.return_value = { + "jobSummaries": [ + { + "jobName": "eval-job-old", + "status": "Completed", + "creationTime": stale_time, + } + ] + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-18" + assert findings[0]["Severity"] == "Medium" + + @patch("bedrock_app.boto3.client") + def test_br18_unknown_operation_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_model_evaluations + + bedrock_client = MagicMock() + bedrock_client.list_evaluation_jobs.side_effect = ClientError( + {"Error": {"Code": "UnknownOperation", "Message": "Unknown operation"}}, + "ListEvaluationJobs", + ) + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-18" + + @patch("bedrock_app.boto3.client") + def test_br18_access_denied_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_model_evaluations + + bedrock_client = MagicMock() + bedrock_client.list_evaluation_jobs.side_effect = ClientError( + {"Error": {"Code": "AccessDeniedException"}}, "ListEvaluationJobs" + ) + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-18" + + def test_br18_schema_valid(self): + check = bedrock_app.check_bedrock_model_evaluations + with patch("bedrock_app.boto3.client") as mock_client: + bedrock_client = MagicMock() + bedrock_client.list_evaluation_jobs.return_value = {"jobSummaries": []} + mock_client.return_value = bedrock_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-19: check_bedrock_prompt_flow_validation +# =================================================================== +class TestBR19PromptFlowValidation: + """BR-19: Verify prompt flows are validated using GetFlow validations.""" + + @patch("bedrock_app.boto3.client") + def test_br19_no_flows_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_prompt_flow_validation + agent_client = MagicMock() + agent_client.list_flows.return_value = {"flowSummaries": []} + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-19" + + @patch("bedrock_app.boto3.client") + def test_br19_prepared_flow_no_errors_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_prompt_flow_validation + agent_client = MagicMock() + agent_client.list_flows.return_value = { + "flowSummaries": [{"id": "f1", "name": "GoodFlow", "status": "Prepared"}] + } + agent_client.get_flow.return_value = {"validations": []} + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-19" + + @patch("bedrock_app.boto3.client") + def test_br19_flow_with_error_validation_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_prompt_flow_validation + agent_client = MagicMock() + agent_client.list_flows.return_value = { + "flowSummaries": [{"id": "f1", "name": "BadFlow", "status": "Prepared"}] + } + agent_client.get_flow.return_value = { + "validations": [{"severity": "ERROR", "message": "Node X is not connected"}] + } + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-19" + assert "Node X is not connected" in findings[0]["Finding_Details"] + + @patch("bedrock_app.boto3.client") + def test_br19_unprepared_flow_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_prompt_flow_validation + agent_client = MagicMock() + agent_client.list_flows.return_value = { + "flowSummaries": [ + {"id": "f1", "name": "DraftFlow", "status": "NotPrepared"} + ] + } + agent_client.get_flow.return_value = {"validations": []} + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-19" + + def test_br19_schema_valid(self): + check = bedrock_app.check_bedrock_prompt_flow_validation + with patch("bedrock_app.boto3.client") as mock_client: + agent_client = MagicMock() + agent_client.list_flows.return_value = {"flowSummaries": []} + mock_client.return_value = agent_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-20: check_bedrock_knowledge_base_kms_encryption +# =================================================================== +class TestBR20KnowledgeBaseKMS: + """BR-20: Verify managed KB customer-managed KMS encryption.""" + + @patch("bedrock_app.boto3.client") + def test_br20_no_kbs_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_knowledge_base_kms_encryption + agent_client = MagicMock() + agent_client.list_knowledge_bases.return_value = {"knowledgeBaseSummaries": []} + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-20" + + @patch("bedrock_app.boto3.client") + def test_br20_managed_kb_with_cmk_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_knowledge_base_kms_encryption + agent_client = MagicMock() + agent_client.list_knowledge_bases.return_value = { + "knowledgeBaseSummaries": [{"knowledgeBaseId": "kb1", "name": "ManagedKB"}] + } + agent_client.get_knowledge_base.return_value = { + "knowledgeBase": { + "knowledgeBaseConfiguration": { + "type": "MANAGED", + "managedKnowledgeBaseConfiguration": { + "serverSideEncryptionConfiguration": { + "kmsKeyArn": "arn:aws:kms:us-east-1:123:key/abc" + } + }, + } + } + } + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-20" + + @patch("bedrock_app.boto3.client") + def test_br20_managed_kb_without_cmk_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_knowledge_base_kms_encryption + agent_client = MagicMock() + agent_client.list_knowledge_bases.return_value = { + "knowledgeBaseSummaries": [{"knowledgeBaseId": "kb1", "name": "ManagedKB"}] + } + agent_client.get_knowledge_base.return_value = { + "knowledgeBase": { + "knowledgeBaseConfiguration": { + "type": "MANAGED", + "managedKnowledgeBaseConfiguration": {}, + } + } + } + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-20" + assert findings[0]["Severity"] == "High" + + @patch("bedrock_app.boto3.client") + def test_br20_managed_kb_sdk_gap_returns_na(self, mock_client): + # A MANAGED knowledge base whose managedKnowledgeBaseConfiguration block is + # absent (bundled botocore predates the field, < 1.43.32) must surface as + # N/A "indeterminate", not a false-positive Failed. + check = bedrock_app.check_bedrock_knowledge_base_kms_encryption + agent_client = MagicMock() + agent_client.list_knowledge_bases.return_value = { + "knowledgeBaseSummaries": [{"knowledgeBaseId": "kb1", "name": "ManagedKB"}] + } + agent_client.get_knowledge_base.return_value = { + "knowledgeBase": {"knowledgeBaseConfiguration": {"type": "MANAGED"}} + } + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + na = [f for f in findings if f["Status"] == "N/A"] + assert len(na) >= 1 + assert na[0]["Check_ID"] == "BR-20" + assert "1.43.32" in na[0]["Finding_Details"] + # Must NOT be reported as a failure on incomplete data. + assert all(f["Status"] != "Failed" for f in findings) + + @patch("bedrock_app.boto3.client") + def test_br20_custom_vector_store_returns_na_review(self, mock_client): + # Custom vector stores (no managed config) cannot be validated from the + # KB API; they are flagged for manual review, not failed. + check = bedrock_app.check_bedrock_knowledge_base_kms_encryption + agent_client = MagicMock() + agent_client.list_knowledge_bases.return_value = { + "knowledgeBaseSummaries": [{"knowledgeBaseId": "kb1", "name": "VectorKB"}] + } + agent_client.get_knowledge_base.return_value = { + "knowledgeBase": { + "knowledgeBaseConfiguration": { + "type": "VECTOR", + "vectorKnowledgeBaseConfiguration": {}, + }, + "storageConfiguration": {"type": "OPENSEARCH_SERVERLESS"}, + } + } + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + na = [f for f in findings if f["Status"] == "N/A"] + assert len(na) >= 1 + assert na[0]["Check_ID"] == "BR-20" + assert "storage layer" in na[0]["Finding_Details"] + + def test_br20_schema_valid(self): + check = bedrock_app.check_bedrock_knowledge_base_kms_encryption + with patch("bedrock_app.boto3.client") as mock_client: + agent_client = MagicMock() + agent_client.list_knowledge_bases.return_value = { + "knowledgeBaseSummaries": [] + } + mock_client.return_value = agent_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-21: check_bedrock_agent_action_group_iam +# =================================================================== +class TestBR21AgentActionGroupIAM: + """BR-21: Verify action-group Lambda roles follow least privilege.""" + + @staticmethod + def _agent_client_with_lambda_role(role_name): + agent_client = MagicMock() + agent_client.list_agents.return_value = { + "agentSummaries": [{"agentId": "a1", "agentName": "TestAgent"}] + } + agent_client.list_agent_action_groups.return_value = { + "actionGroupSummaries": [ + {"actionGroupId": "ag1", "actionGroupName": "ActionGroup1"} + ] + } + agent_client.get_agent_action_group.return_value = { + "agentActionGroup": { + "actionGroupExecutor": { + "lambda": "arn:aws:lambda:us-east-1:123:function:my-func" + } + } + } + lambda_client = MagicMock() + lambda_client.get_function.return_value = { + "Configuration": {"Role": f"arn:aws:iam::123456789012:role/{role_name}"} + } + return agent_client, lambda_client + + @patch("bedrock_app.boto3.client") + def test_br21_no_agents_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_agent_action_group_iam + agent_client = MagicMock() + agent_client.list_agents.return_value = {"agentSummaries": []} + mock_client.return_value = agent_client + + result = check(region="us-east-1", permission_cache={"role_permissions": {}}) + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-21" + + @patch("bedrock_app.boto3.client") + def test_br21_admin_access_role_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_agent_action_group_iam + agent_client, lambda_client = self._agent_client_with_lambda_role("AdminRole") + + def factory(service, **kwargs): + return lambda_client if service == "lambda" else agent_client + + mock_client.side_effect = factory + + cache = { + "role_permissions": { + "AdminRole": { + "attached_policies": [{"name": "AdministratorAccess"}], + "inline_policies": [], + } + } + } + result = check(region="us-east-1", permission_cache=cache) + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-21" + assert "AdministratorAccess" in findings[0]["Finding_Details"] + + @patch("bedrock_app.boto3.client") + def test_br21_wildcard_inline_policy_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_agent_action_group_iam + agent_client, lambda_client = self._agent_client_with_lambda_role("WildRole") + + def factory(service, **kwargs): + return lambda_client if service == "lambda" else agent_client + + mock_client.side_effect = factory + + cache = { + "role_permissions": { + "WildRole": { + "attached_policies": [], + "inline_policies": [ + { + "name": "inline-wild", + "document": { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": "*", + "Resource": "*", + } + ], + }, + } + ], + } + } + } + result = check(region="us-east-1", permission_cache=cache) + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-21" + + @patch("bedrock_app.boto3.client") + def test_br21_scoped_role_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_agent_action_group_iam + agent_client, lambda_client = self._agent_client_with_lambda_role("ScopedRole") + + def factory(service, **kwargs): + return lambda_client if service == "lambda" else agent_client + + mock_client.side_effect = factory + + cache = { + "role_permissions": { + "ScopedRole": { + "attached_policies": [{"name": "CustomScopedPolicy"}], + "inline_policies": [], + } + } + } + result = check(region="us-east-1", permission_cache=cache) + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-21" + + def test_br21_schema_valid(self): + check = bedrock_app.check_bedrock_agent_action_group_iam + with patch("bedrock_app.boto3.client") as mock_client: + agent_client = MagicMock() + agent_client.list_agents.return_value = {"agentSummaries": []} + mock_client.return_value = agent_client + result = check( + region="us-east-1", permission_cache={"role_permissions": {}} + ) + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-23: check_bedrock_guardrail_content_filters +# =================================================================== +class TestBR23ContentFilters: + """BR-23: Verify all content filters are enabled via contentPolicy.filters.""" + + @staticmethod + def _filters(types): + return [ + {"type": t, "inputStrength": "HIGH", "outputStrength": "HIGH"} + for t in types + ] + + @patch("bedrock_app.boto3.client") + def test_br23_no_guardrails_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_content_filters + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-23" + + @patch("bedrock_app.boto3.client") + def test_br23_all_filters_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_content_filters + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr1", "name": "FullGuardrail"}] + } + bedrock_client.get_guardrail.return_value = { + "guardrail": { + "contentPolicy": { + "filters": self._filters(["HATE", "INSULTS", "SEXUAL", "VIOLENCE"]) + } + } + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-23" + + @patch("bedrock_app.boto3.client") + def test_br23_missing_filters_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_content_filters + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr1", "name": "PartialGuardrail"}] + } + bedrock_client.get_guardrail.return_value = { + "guardrail": { + "contentPolicy": {"filters": self._filters(["HATE", "VIOLENCE"])} + } + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-23" + assert "INSULTS" in findings[0]["Finding_Details"] + + def test_br23_schema_valid(self): + check = bedrock_app.check_bedrock_guardrail_content_filters + with patch("bedrock_app.boto3.client") as mock_client: + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-24: check_bedrock_automated_reasoning_policy +# =================================================================== +class TestBR24AutomatedReasoning: + """BR-24: Verify Automated Reasoning policies via automatedReasoningPolicy.""" + + @patch("bedrock_app.boto3.client") + def test_br24_no_guardrails_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_automated_reasoning_policy + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-24" + + @patch("bedrock_app.boto3.client") + def test_br24_with_ar_policy_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_automated_reasoning_policy + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr1", "name": "VerifiedGuardrail"}] + } + bedrock_client.get_guardrail.return_value = { + "guardrail": { + "automatedReasoningPolicy": { + "policies": [ + "arn:aws:bedrock:us-east-1:123:automated-reasoning-policy/p1" + ] + } + } + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-24" + + @patch("bedrock_app.boto3.client") + def test_br24_without_ar_policy_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_automated_reasoning_policy + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr1", "name": "PlainGuardrail"}] + } + bedrock_client.get_guardrail.return_value = {"guardrail": {}} + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-24" + + def test_br24_schema_valid(self): + check = bedrock_app.check_bedrock_automated_reasoning_policy + with patch("bedrock_app.boto3.client") as mock_client: + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-26: check_bedrock_guardrail_pii_filters +# =================================================================== +class TestBR26GuardrailPIIFilters: + """BR-26: Verify guardrails configure sensitive-information (PII) filters.""" + + @patch("bedrock_app.boto3.client") + def test_br26_no_guardrails_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_pii_filters + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-26" + + @patch("bedrock_app.boto3.client") + def test_br26_pii_configured_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_pii_filters + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr1", "name": "PiiGuardrail"}] + } + bedrock_client.get_guardrail.return_value = { + "guardrail": { + "sensitiveInformationPolicy": { + "piiEntities": [{"type": "EMAIL", "action": "ANONYMIZE"}], + "regexes": [], + } + } + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-26" + + @patch("bedrock_app.boto3.client") + def test_br26_no_pii_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_pii_filters + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr1", "name": "PlainGuardrail"}] + } + bedrock_client.get_guardrail.return_value = { + "guardrail": { + "sensitiveInformationPolicy": {"piiEntities": [], "regexes": []} + } + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-26" + assert findings[0]["Severity"] == "High" + + @patch("bedrock_app.boto3.client") + def test_br26_access_denied_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_pii_filters + bedrock_client = MagicMock() + bedrock_client.list_guardrails.side_effect = ClientError( + {"Error": {"Code": "AccessDeniedException"}}, "ListGuardrails" + ) + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-26" + + def test_br26_schema_valid(self): + check = bedrock_app.check_bedrock_guardrail_pii_filters + with patch("bedrock_app.boto3.client") as mock_client: + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-27: check_bedrock_guardrail_contextual_grounding +# =================================================================== +class TestBR27ContextualGrounding: + """BR-27: Verify guardrails enable contextual grounding checks.""" + + @patch("bedrock_app.boto3.client") + def test_br27_no_guardrails_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_contextual_grounding + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-27" + + @patch("bedrock_app.boto3.client") + def test_br27_grounding_enabled_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_contextual_grounding + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr1", "name": "GroundedGuardrail"}] + } + bedrock_client.get_guardrail.return_value = { + "guardrail": { + "contextualGroundingPolicy": { + "filters": [ + {"type": "GROUNDING", "threshold": 0.75, "enabled": True} + ] + } + } + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-27" + + @patch("bedrock_app.boto3.client") + def test_br27_no_grounding_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_guardrail_contextual_grounding + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = { + "guardrails": [{"id": "gr1", "name": "PlainGuardrail"}] + } + bedrock_client.get_guardrail.return_value = { + "guardrail": {"contextualGroundingPolicy": {"filters": []}} + } + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-27" + + def test_br27_schema_valid(self): + check = bedrock_app.check_bedrock_guardrail_contextual_grounding + with patch("bedrock_app.boto3.client") as mock_client: + bedrock_client = MagicMock() + bedrock_client.list_guardrails.return_value = {"guardrails": []} + mock_client.return_value = bedrock_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-28: check_bedrock_agent_guardrail_association +# =================================================================== +class TestBR28AgentGuardrailAssociation: + """BR-28: Verify agents have an associated guardrail.""" + + @staticmethod + def _agent_client(summaries): + agent_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [{"agentSummaries": summaries}] + agent_client.get_paginator.return_value = paginator + return agent_client + + @patch("bedrock_app.boto3.client") + def test_br28_no_agents_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_agent_guardrail_association + mock_client.return_value = self._agent_client([]) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-28" + + @patch("bedrock_app.boto3.client") + def test_br28_agent_with_guardrail_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_agent_guardrail_association + mock_client.return_value = self._agent_client( + [ + { + "agentId": "a1", + "agentName": "SafeAgent", + "guardrailConfiguration": { + "guardrailIdentifier": "gr-1", + "guardrailVersion": "1", + }, + } + ] + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-28" + + @patch("bedrock_app.boto3.client") + def test_br28_agent_without_guardrail_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_agent_guardrail_association + mock_client.return_value = self._agent_client( + [{"agentId": "a1", "agentName": "OpenAgent"}] + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-28" + assert findings[0]["Severity"] == "High" + + def test_br28_schema_valid(self): + check = bedrock_app.check_bedrock_agent_guardrail_association + with patch("bedrock_app.boto3.client") as mock_client: + mock_client.return_value = self._agent_client([]) + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-29: check_bedrock_agent_idle_session_ttl +# =================================================================== +class TestBR29AgentIdleSessionTTL: + """BR-29: Verify agent idle session TTL is within the recommended bound.""" + + @staticmethod + def _agent_client(summaries, get_agent_return=None): + agent_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [{"agentSummaries": summaries}] + agent_client.get_paginator.return_value = paginator + if get_agent_return is not None: + agent_client.get_agent.return_value = get_agent_return + return agent_client + + @patch("bedrock_app.boto3.client") + def test_br29_no_agents_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_agent_idle_session_ttl + mock_client.return_value = self._agent_client([]) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-29" + + @patch("bedrock_app.boto3.client") + def test_br29_short_ttl_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_agent_idle_session_ttl + mock_client.return_value = self._agent_client( + [{"agentId": "a1", "agentName": "ShortAgent"}], + {"agent": {"idleSessionTTLInSeconds": 600}}, + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-29" + + @patch("bedrock_app.boto3.client") + def test_br29_long_ttl_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_agent_idle_session_ttl + mock_client.return_value = self._agent_client( + [{"agentId": "a1", "agentName": "LongAgent"}], + {"agent": {"idleSessionTTLInSeconds": 7200}}, + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-29" + + def test_br29_schema_valid(self): + check = bedrock_app.check_bedrock_agent_idle_session_ttl + with patch("bedrock_app.boto3.client") as mock_client: + mock_client.return_value = self._agent_client([]) + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-30: check_bedrock_imported_model_kms_encryption +# =================================================================== +class TestBR30ImportedModelKMS: + """BR-30: Verify imported models use customer-managed KMS keys.""" + + @staticmethod + def _bedrock_client(summaries, get_return=None, list_side_effect=None): + bedrock_client = MagicMock() + paginator = MagicMock() + if list_side_effect is not None: + paginator.paginate.side_effect = list_side_effect + else: + paginator.paginate.return_value = [{"modelSummaries": summaries}] + bedrock_client.get_paginator.return_value = paginator + if get_return is not None: + bedrock_client.get_imported_model.return_value = get_return + return bedrock_client + + @patch("bedrock_app.boto3.client") + def test_br30_no_models_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_imported_model_kms_encryption + mock_client.return_value = self._bedrock_client([]) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-30" + + @patch("bedrock_app.boto3.client") + def test_br30_customer_key_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_imported_model_kms_encryption + mock_client.return_value = self._bedrock_client( + [{"modelArn": "arn:model:1", "modelName": "imported-1"}], + {"modelKmsKeyArn": "arn:aws:kms:us-east-1:123:key/abc"}, + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-30" + + @patch("bedrock_app.boto3.client") + def test_br30_aws_owned_key_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_imported_model_kms_encryption + mock_client.return_value = self._bedrock_client( + [{"modelArn": "arn:model:1", "modelName": "imported-1"}], + {}, + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-30" + assert findings[0]["Severity"] == "High" + + @patch("bedrock_app.boto3.client") + def test_br30_access_denied_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_imported_model_kms_encryption + mock_client.return_value = self._bedrock_client( + [], + list_side_effect=ClientError( + {"Error": {"Code": "AccessDeniedException"}}, "ListImportedModels" + ), + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-30" + + def test_br30_schema_valid(self): + check = bedrock_app.check_bedrock_imported_model_kms_encryption + with patch("bedrock_app.boto3.client") as mock_client: + mock_client.return_value = self._bedrock_client([]) + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-31: check_bedrock_batch_inference_output_encryption +# =================================================================== +class TestBR31BatchInferenceOutputEncryption: + """BR-31: Verify batch inference jobs encrypt output with customer KMS.""" + + @staticmethod + def _bedrock_client(summaries, list_side_effect=None): + bedrock_client = MagicMock() + paginator = MagicMock() + if list_side_effect is not None: + paginator.paginate.side_effect = list_side_effect + else: + paginator.paginate.return_value = [{"invocationJobSummaries": summaries}] + bedrock_client.get_paginator.return_value = paginator + return bedrock_client + + @patch("bedrock_app.boto3.client") + def test_br31_no_jobs_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_batch_inference_output_encryption + mock_client.return_value = self._bedrock_client([]) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-31" + + @patch("bedrock_app.boto3.client") + def test_br31_job_with_cmk_returns_passed(self, mock_client): + check = bedrock_app.check_bedrock_batch_inference_output_encryption + mock_client.return_value = self._bedrock_client( + [ + { + "jobName": "batch-1", + "outputDataConfig": { + "s3OutputDataConfig": { + "s3Uri": "s3://out/", + "s3EncryptionKeyId": "arn:aws:kms:us-east-1:123:key/abc", + } + }, + } + ] + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + passed = [f for f in findings if f["Status"] == "Passed"] + assert len(passed) >= 1 + assert passed[0]["Check_ID"] == "BR-31" + + @patch("bedrock_app.boto3.client") + def test_br31_job_without_cmk_returns_failed(self, mock_client): + check = bedrock_app.check_bedrock_batch_inference_output_encryption + mock_client.return_value = self._bedrock_client( + [ + { + "jobName": "batch-1", + "outputDataConfig": {"s3OutputDataConfig": {"s3Uri": "s3://out/"}}, + } + ] + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-31" + assert findings[0]["Severity"] == "Medium" + + def test_br31_schema_valid(self): + check = bedrock_app.check_bedrock_batch_inference_output_encryption + with patch("bedrock_app.boto3.client") as mock_client: + mock_client.return_value = self._bedrock_client([]) + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + +# =================================================================== +# BR-32: check_bedrock_cloudwatch_alarms +# =================================================================== +class TestBR32CloudWatchAlarms: + """BR-32: Verify CloudWatch alarms exist on AWS/Bedrock metrics.""" + + @patch("bedrock_app.detect_bedrock_regional_footprint", return_value=False) + @patch("bedrock_app.boto3.client") + def test_br32_no_footprint_returns_na(self, mock_client, mock_footprint): + check = bedrock_app.check_bedrock_cloudwatch_alarms + result = check(region="eu-west-3") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-32" + + @patch("bedrock_app.detect_bedrock_regional_footprint", return_value=True) + @patch("bedrock_app.boto3.client") + def test_br32_bedrock_alarm_returns_passed(self, mock_client, mock_footprint): + check = bedrock_app.check_bedrock_cloudwatch_alarms + cw_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [ + { + "MetricAlarms": [ + {"AlarmName": "bedrock-throttle", "Namespace": "AWS/Bedrock"} + ] + } + ] + cw_client.get_paginator.return_value = paginator + mock_client.return_value = cw_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Passed" + assert findings[0]["Check_ID"] == "BR-32" + + @patch("bedrock_app.detect_bedrock_regional_footprint", return_value=True) + @patch("bedrock_app.boto3.client") + def test_br32_metric_math_alarm_returns_passed(self, mock_client, mock_footprint): + check = bedrock_app.check_bedrock_cloudwatch_alarms + cw_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [ + { + "MetricAlarms": [ + { + "AlarmName": "bedrock-tpm", + "Metrics": [ + {"MetricStat": {"Metric": {"Namespace": "AWS/Bedrock"}}} + ], + } + ] + } + ] + cw_client.get_paginator.return_value = paginator + mock_client.return_value = cw_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Passed" + assert findings[0]["Check_ID"] == "BR-32" + + @patch("bedrock_app.detect_bedrock_regional_footprint", return_value=True) + @patch("bedrock_app.boto3.client") + def test_br32_no_bedrock_alarm_returns_failed(self, mock_client, mock_footprint): + check = bedrock_app.check_bedrock_cloudwatch_alarms + cw_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [ + {"MetricAlarms": [{"AlarmName": "ec2-cpu", "Namespace": "AWS/EC2"}]} + ] + cw_client.get_paginator.return_value = paginator + mock_client.return_value = cw_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "Failed" + assert findings[0]["Check_ID"] == "BR-32" + assert findings[0]["Severity"] == "Medium" + + @patch("bedrock_app.detect_bedrock_regional_footprint", return_value=True) + @patch("bedrock_app.boto3.client") + def test_br32_schema_valid(self, mock_client, mock_footprint): + check = bedrock_app.check_bedrock_cloudwatch_alarms + cw_client = MagicMock() + paginator = MagicMock() + paginator.paginate.return_value = [{"MetricAlarms": []}] + cw_client.get_paginator.return_value = paginator + mock_client.return_value = cw_client + + result = check(region="us-east-1") + for f in extract_csv_data(result): + assert_finding_schema(f) From 385fb69dc97ff4472f072c57b379cfad03375c4a Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Fri, 26 Jun 2026 22:38:10 -0400 Subject: [PATCH 02/10] fix: grant missing Bedrock IAM actions for BR-18/22/30/31 in SAM templates The SAM scanning-Lambda role (BedrockSecurityAssessmentFunction) was missing read-only actions the newer BR-15..32 checks call, causing AccessDeniedException ('Unable to check') for BR-18, BR-22, BR-30, BR-31 at runtime. The standalone deployment/*.yaml roles already had these; the SAM templates were not updated. Add to template.yaml and template-multi-account.yaml: - bedrock:ListEvaluationJobs (BR-18) - bedrock:ListImportedModels / GetImportedModel (BR-30) - bedrock:ListModelInvocationJobs (BR-31) - servicequotas:ListServiceQuotas / GetServiceQuota (BR-22) - cloudwatch:DescribeAlarms, bedrock:ListAgentActionGroups / GetAgentActionGroup (proactive: same AccessDenied once alarms/action groups exist) Extend test_core_iam_coverage.py to guard the new actions so this drift cannot recur silently. --- .../template-multi-account.yaml | 16 ++++++++++++++++ aiml-security-assessment/template.yaml | 17 +++++++++++++++++ tests/test_core_iam_coverage.py | 14 ++++++++++++++ 3 files changed, 47 insertions(+) diff --git a/aiml-security-assessment/template-multi-account.yaml b/aiml-security-assessment/template-multi-account.yaml index 34b054e..8666e54 100644 --- a/aiml-security-assessment/template-multi-account.yaml +++ b/aiml-security-assessment/template-multi-account.yaml @@ -221,6 +221,22 @@ Resources: - bedrock:GetFlow - bedrock:ListKnowledgeBases - bedrock:GetKnowledgeBase + - bedrock:ListEvaluationJobs # BR-18 + - bedrock:ListImportedModels # BR-30 + - bedrock:GetImportedModel # BR-30 + - bedrock:ListAgentActionGroups + - bedrock:GetAgentActionGroup + Resource: '*' + - Sid: ServiceQuotasPermissions + Effect: Allow + Action: + - servicequotas:ListServiceQuotas # BR-22 + - servicequotas:GetServiceQuota # BR-22 + Resource: '*' + - Sid: CloudWatchPermissions + Effect: Allow + Action: + - cloudwatch:DescribeAlarms Resource: '*' - Sid: S3BucketEncryptionPermissions Effect: Allow diff --git a/aiml-security-assessment/template.yaml b/aiml-security-assessment/template.yaml index c6bd63d..d75a149 100644 --- a/aiml-security-assessment/template.yaml +++ b/aiml-security-assessment/template.yaml @@ -219,6 +219,23 @@ Resources: - bedrock:GetFlow - bedrock:ListKnowledgeBases - bedrock:GetKnowledgeBase + - bedrock:ListEvaluationJobs # BR-18 + - bedrock:ListImportedModels # BR-30 + - bedrock:GetImportedModel # BR-30 + - bedrock:ListModelInvocationJobs # BR-31 + - bedrock:ListAgentActionGroups + - bedrock:GetAgentActionGroup + Resource: '*' + - Sid: ServiceQuotasPermissions + Effect: Allow + Action: + - servicequotas:ListServiceQuotas # BR-22 + - servicequotas:GetServiceQuota # BR-22 + Resource: '*' + - Sid: CloudWatchPermissions + Effect: Allow + Action: + - cloudwatch:DescribeAlarms Resource: '*' - Sid: S3BucketEncryptionPermissions Effect: Allow diff --git a/tests/test_core_iam_coverage.py b/tests/test_core_iam_coverage.py index 575cddc..22d2488 100644 --- a/tests/test_core_iam_coverage.py +++ b/tests/test_core_iam_coverage.py @@ -16,6 +16,13 @@ "bedrock:GetModelInvocationLoggingConfiguration", "bedrock:ListKnowledgeBases", "bedrock:GetKnowledgeBase", + "bedrock:ListEvaluationJobs", # BR-18 + "bedrock:ListImportedModels", # BR-30 + "bedrock:GetImportedModel", # BR-30 + "bedrock:ListModelInvocationJobs", # BR-31 + "servicequotas:ListServiceQuotas", # BR-22 + "servicequotas:GetServiceQuota", # BR-22 + "cloudwatch:DescribeAlarms", }, }, { @@ -34,6 +41,13 @@ "bedrock:GetModelInvocationLoggingConfiguration", "bedrock:ListKnowledgeBases", "bedrock:GetKnowledgeBase", + "bedrock:ListEvaluationJobs", # BR-18 + "bedrock:ListImportedModels", # BR-30 + "bedrock:GetImportedModel", # BR-30 + "bedrock:ListModelInvocationJobs", # BR-31 + "servicequotas:ListServiceQuotas", # BR-22 + "servicequotas:GetServiceQuota", # BR-22 + "cloudwatch:DescribeAlarms", }, }, { From fca8b386ad40b7a903a210aac2efda815b96c60d Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Fri, 26 Jun 2026 22:48:01 -0400 Subject: [PATCH 03/10] fix: grant Organizations permissions for BR-15 in SAM templates BR-15 (Cross-Account Guardrails Enforcement) calls organizations:DescribeOrganization and organizations:ListPolicies, but the SAM Bedrock scanning-Lambda role had no Organizations grant, causing AccessDeniedException ('Unable to check') at runtime. The deployment/*.yaml roles already had these. Add an OrganizationsPermissions statement to the BedrockSecurityAssessmentFunction role in template.yaml and template-multi-account.yaml, and extend test_core_iam_coverage.py to guard the actions. --- aiml-security-assessment/template-multi-account.yaml | 6 ++++++ aiml-security-assessment/template.yaml | 6 ++++++ tests/test_core_iam_coverage.py | 4 ++++ 3 files changed, 16 insertions(+) diff --git a/aiml-security-assessment/template-multi-account.yaml b/aiml-security-assessment/template-multi-account.yaml index 8666e54..6e35b3e 100644 --- a/aiml-security-assessment/template-multi-account.yaml +++ b/aiml-security-assessment/template-multi-account.yaml @@ -238,6 +238,12 @@ Resources: Action: - cloudwatch:DescribeAlarms Resource: '*' + - Sid: OrganizationsPermissions + Effect: Allow + Action: + - organizations:DescribeOrganization # BR-15 + - organizations:ListPolicies # BR-15 + Resource: '*' - Sid: S3BucketEncryptionPermissions Effect: Allow Action: diff --git a/aiml-security-assessment/template.yaml b/aiml-security-assessment/template.yaml index d75a149..9eb767c 100644 --- a/aiml-security-assessment/template.yaml +++ b/aiml-security-assessment/template.yaml @@ -237,6 +237,12 @@ Resources: Action: - cloudwatch:DescribeAlarms Resource: '*' + - Sid: OrganizationsPermissions + Effect: Allow + Action: + - organizations:DescribeOrganization # BR-15 + - organizations:ListPolicies # BR-15 + Resource: '*' - Sid: S3BucketEncryptionPermissions Effect: Allow Action: diff --git a/tests/test_core_iam_coverage.py b/tests/test_core_iam_coverage.py index 22d2488..99a2e63 100644 --- a/tests/test_core_iam_coverage.py +++ b/tests/test_core_iam_coverage.py @@ -23,6 +23,8 @@ "servicequotas:ListServiceQuotas", # BR-22 "servicequotas:GetServiceQuota", # BR-22 "cloudwatch:DescribeAlarms", + "organizations:DescribeOrganization", # BR-15 + "organizations:ListPolicies", # BR-15 }, }, { @@ -48,6 +50,8 @@ "servicequotas:ListServiceQuotas", # BR-22 "servicequotas:GetServiceQuota", # BR-22 "cloudwatch:DescribeAlarms", + "organizations:DescribeOrganization", # BR-15 + "organizations:ListPolicies", # BR-15 }, }, { From 7714e176a95a17645f6bcf5696f833b5139f8c76 Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Fri, 26 Jun 2026 23:45:02 -0400 Subject: [PATCH 04/10] fix: report account/feature-gate denials as N/A for BR-18/30/31 Bedrock Custom Model Import, Batch Inference, and Model Evaluation are account/region feature-gated. When not enabled, the List* APIs raise AccessDeniedException with 'Your account is not authorized to invoke this API operation' - a different cause from an IAM-policy gap ('... not authorized to perform: because no identity-based policy allows ...'). Previously both collapsed to status=Failed with a 'grant permission' resolution, which is misleading once the IAM grant is in place: no IAM change enables a feature-gated API. Add is_account_not_authorized() to distinguish the two, and route the account-gate case to status=N/A (severity Low) with a feature-not-enabled resolution for BR-18, BR-30, BR-31. Genuine IAM-gap denials still report Failed. Add regression tests covering the account-gate branch for each check. --- .../security/bedrock_assessments/app.py | 75 +++++++++++++++++++ tests/test_bedrock_checks.py | 68 +++++++++++++++++ 2 files changed, 143 insertions(+) diff --git a/aiml-security-assessment/functions/security/bedrock_assessments/app.py b/aiml-security-assessment/functions/security/bedrock_assessments/app.py index 933ad1f..57f4562 100644 --- a/aiml-security-assessment/functions/security/bedrock_assessments/app.py +++ b/aiml-security-assessment/functions/security/bedrock_assessments/app.py @@ -48,6 +48,27 @@ } +def is_account_not_authorized(error: Exception) -> bool: + """ + Distinguish an account/feature-gate denial from an IAM-policy denial. + + Both surface as AccessDeniedException, but the cause differs: + - IAM gap: "... is not authorized to perform: because no + identity-based policy allows ..." -> grant the action. + - Account gate: "Your account is not authorized to invoke this API + operation." -> the Bedrock feature (e.g. Custom Model + Import, Batch Inference, Model Evaluation) is not enabled + or allow-listed for this account/region. No IAM change + fixes it, so the check is Not Applicable rather than a + finding. + """ + text = str(error) + return ( + "not authorized to invoke this API operation" in text + or "account is not authorized" in text + ) + + def describe_api_error(error: Exception, api_label: str, region: str = "") -> str: """ Build a report-friendly description for an API error raised by a regional @@ -3316,6 +3337,24 @@ def check_bedrock_model_evaluations(region: str = "") -> Dict[str, Any]: region=region, ) ) + elif is_account_not_authorized(e): + findings["details"] = ( + "Model evaluation not enabled for this account/region" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-18", + finding_name="Model Evaluation Implementation Check", + finding_details=describe_api_error( + e, "Model evaluation check", region + ), + resolution="Amazon Bedrock model evaluation is not enabled or available for this account in this region. No IAM change is required; enable the feature to assess model evaluation practices.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", + severity="Low", + status="N/A", + region=region, + ) + ) elif error_code in ACCESS_DENIED_ERROR_CODES: findings["csv_data"].append( create_finding( @@ -5521,6 +5560,24 @@ def check_bedrock_imported_model_kms_encryption(region: str = "") -> Dict[str, A region=region, ) ) + elif is_account_not_authorized(e): + findings["details"] = ( + "Custom model import not enabled for this account/region" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-30", + finding_name="Imported Model Customer-Managed KMS Encryption Check", + finding_details=describe_api_error( + e, "Imported model encryption check", region + ), + resolution="Amazon Bedrock Custom Model Import is not enabled or available for this account in this region. No IAM change is required; the check applies only once model import is in use.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-customization-import-model.html", + severity="Low", + status="N/A", + region=region, + ) + ) elif error_code in ACCESS_DENIED_ERROR_CODES: findings["csv_data"].append( create_finding( @@ -5677,6 +5734,24 @@ def check_bedrock_batch_inference_output_encryption( region=region, ) ) + elif is_account_not_authorized(e): + findings["details"] = ( + "Batch inference not enabled for this account/region" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-31", + finding_name="Batch Inference Output Encryption Check", + finding_details=describe_api_error( + e, "Batch inference output encryption check", region + ), + resolution="Amazon Bedrock batch inference is not enabled or available for this account in this region. No IAM change is required; the check applies only once batch inference is in use.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html", + severity="Low", + status="N/A", + region=region, + ) + ) elif error_code in ACCESS_DENIED_ERROR_CODES: findings["csv_data"].append( create_finding( diff --git a/tests/test_bedrock_checks.py b/tests/test_bedrock_checks.py index 666d0ac..1edab3d 100644 --- a/tests/test_bedrock_checks.py +++ b/tests/test_bedrock_checks.py @@ -1932,6 +1932,29 @@ def test_br18_access_denied_returns_failed(self, mock_client): assert findings[0]["Status"] == "Failed" assert findings[0]["Check_ID"] == "BR-18" + @patch("bedrock_app.boto3.client") + def test_br18_account_not_authorized_returns_na(self, mock_client): + # Account/feature-gate denial (model evaluation not enabled) must be N/A. + check = bedrock_app.check_bedrock_model_evaluations + + bedrock_client = MagicMock() + bedrock_client.list_evaluation_jobs.side_effect = ClientError( + { + "Error": { + "Code": "AccessDeniedException", + "Message": "Your account is not authorized to invoke this API operation.", + } + }, + "ListEvaluationJobs", + ) + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-18" + def test_br18_schema_valid(self): check = bedrock_app.check_bedrock_model_evaluations with patch("bedrock_app.boto3.client") as mock_client: @@ -2810,6 +2833,29 @@ def test_br30_access_denied_returns_failed(self, mock_client): assert findings[0]["Status"] == "Failed" assert findings[0]["Check_ID"] == "BR-30" + @patch("bedrock_app.boto3.client") + def test_br30_account_not_authorized_returns_na(self, mock_client): + # Account/feature-gate denial (Custom Model Import not enabled) must be + # N/A, not a Failed finding telling the user to grant IAM permissions. + check = bedrock_app.check_bedrock_imported_model_kms_encryption + mock_client.return_value = self._bedrock_client( + [], + list_side_effect=ClientError( + { + "Error": { + "Code": "AccessDeniedException", + "Message": "Your account is not authorized to invoke this API operation.", + } + }, + "ListImportedModels", + ), + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-30" + def test_br30_schema_valid(self): check = bedrock_app.check_bedrock_imported_model_kms_encryption with patch("bedrock_app.boto3.client") as mock_client: @@ -2888,6 +2934,28 @@ def test_br31_job_without_cmk_returns_failed(self, mock_client): assert findings[0]["Check_ID"] == "BR-31" assert findings[0]["Severity"] == "Medium" + @patch("bedrock_app.boto3.client") + def test_br31_account_not_authorized_returns_na(self, mock_client): + # Account/feature-gate denial (batch inference not enabled) must be N/A. + check = bedrock_app.check_bedrock_batch_inference_output_encryption + mock_client.return_value = self._bedrock_client( + [], + list_side_effect=ClientError( + { + "Error": { + "Code": "AccessDeniedException", + "Message": "Your account is not authorized to invoke this API operation.", + } + }, + "ListModelInvocationJobs", + ), + ) + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-31" + def test_br31_schema_valid(self): check = bedrock_app.check_bedrock_batch_inference_output_encryption with patch("bedrock_app.boto3.client") as mock_client: From 4ba21c197196122dcfa687eed8a70df740380f51 Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Fri, 26 Jun 2026 23:59:55 -0400 Subject: [PATCH 05/10] fix: report region-unsupported APIs as N/A for BR-09/20/21/25 Audited all 33 Bedrock checks for how region/feature unavailability is handled. Four checks call region-limited APIs (Knowledge Bases, Agents, RAG/model evaluation) but had no UnknownOperation branch, so in a region lacking the API the error skipped the AccessDenied branch and surfaced as a generic ERROR (BR-20/21/25) or a misleading Failed (BR-09) instead of N/A. Add a shared is_region_unsupported() helper and a region-unsupported -> N/A branch to BR-09, BR-20, BR-21, BR-25, matching the existing pattern in BR-18/24/28/29/30/31. Genuine IAM-gap and validation errors still report as before. Add regression tests covering the region-unsupported branch for each, plus a new BR-25 test class (none existed). --- .../security/bedrock_assessments/app.py | 87 ++++++++++++- tests/test_bedrock_checks.py | 121 ++++++++++++++++++ 2 files changed, 205 insertions(+), 3 deletions(-) diff --git a/aiml-security-assessment/functions/security/bedrock_assessments/app.py b/aiml-security-assessment/functions/security/bedrock_assessments/app.py index 57f4562..0acc90e 100644 --- a/aiml-security-assessment/functions/security/bedrock_assessments/app.py +++ b/aiml-security-assessment/functions/security/bedrock_assessments/app.py @@ -69,6 +69,20 @@ def is_account_not_authorized(error: Exception) -> bool: ) +def is_region_unsupported(error: Exception) -> bool: + """ + Detect a "this API/feature is not available in this region" error. + + Several Bedrock features (Knowledge Bases, Agents, Flows, Model/RAG + evaluation, ...) are not in every region. boto3 surfaces an unsupported + operation as an UnknownOperation/"Unknown operation" error. When a check + calls such an API in a region that lacks it, that is Not Applicable rather + than a security finding or a hard error. + """ + text = str(error) + return "UnknownOperation" in text or "Unknown operation" in text + + def describe_api_error(error: Exception, api_label: str, region: str = "") -> str: """ Build a report-friendly description for an API error raised by a regional @@ -1746,6 +1760,23 @@ def check_bedrock_knowledge_base_encryption(region: str = "") -> Dict[str, Any]: region=region, ) ) + elif is_region_unsupported(e): + findings["status"] = "N/A" + findings["details"] = "Knowledge Bases API not available in this region" + findings["csv_data"].append( + create_finding( + check_id="BR-09", + finding_name="Bedrock Knowledge Base Encryption Check", + finding_details=describe_api_error( + e, "Knowledge Bases API", region + ), + resolution="Amazon Bedrock Knowledge Bases are not available in this region. No action required.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html", + severity="Low", + status="N/A", + region=region, + ) + ) else: error_code = e.response.get("Error", {}).get("Code") if error_code == "ValidationException": @@ -3797,7 +3828,23 @@ def check_bedrock_knowledge_base_kms_encryption(region: str = "") -> Dict[str, A except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") - if error_code in ACCESS_DENIED_ERROR_CODES: + if is_region_unsupported(e): + findings["details"] = "Knowledge Bases API not available in this region" + findings["csv_data"].append( + create_finding( + check_id="BR-20", + finding_name="Knowledge Base Customer-Managed KMS Encryption Check", + finding_details=describe_api_error( + e, "Knowledge Bases API", region + ), + resolution="Amazon Bedrock Knowledge Bases are not available in this region. No action required.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: findings["csv_data"].append( create_finding( check_id="BR-20", @@ -4070,7 +4117,23 @@ def check_bedrock_agent_action_group_iam( except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") - if error_code in ACCESS_DENIED_ERROR_CODES: + if is_region_unsupported(e): + findings["details"] = "Bedrock Agents API not available in this region" + findings["csv_data"].append( + create_finding( + check_id="BR-21", + finding_name="Agent Action Group IAM Least Privilege Check", + finding_details=describe_api_error( + e, "Bedrock Agents API", region + ), + resolution="Amazon Bedrock Agents are not available in this region. No action required.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/agents.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: findings["csv_data"].append( create_finding( check_id="BR-21", @@ -4771,7 +4834,25 @@ def check_bedrock_rag_evaluation_jobs(region: str = "") -> Dict[str, Any]: except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") - if error_code in ACCESS_DENIED_ERROR_CODES: + if is_region_unsupported(e): + findings["details"] = ( + "Knowledge Bases / evaluation API not available in this region" + ) + findings["csv_data"].append( + create_finding( + check_id="BR-25", + finding_name="RAG Evaluation Jobs Check", + finding_details=describe_api_error( + e, "RAG evaluation API", region + ), + resolution="Amazon Bedrock Knowledge Bases or RAG evaluation are not available in this region. No action required.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-evaluation-rag.html", + severity="Low", + status="N/A", + region=region, + ) + ) + elif error_code in ACCESS_DENIED_ERROR_CODES: findings["csv_data"].append( create_finding( check_id="BR-25", diff --git a/tests/test_bedrock_checks.py b/tests/test_bedrock_checks.py index 1edab3d..214e970 100644 --- a/tests/test_bedrock_checks.py +++ b/tests/test_bedrock_checks.py @@ -745,6 +745,29 @@ def test_br09_access_denied_returns_na(self, mock_client): assert findings[0]["Status"] == "N/A" assert findings[0]["Severity"] == "Informational" + @patch("boto3.client") + def test_br09_region_unsupported_returns_na(self, mock_client): + # Knowledge Bases API absent in the region -> N/A, not ERROR/Failed. + check = bedrock_app.check_bedrock_knowledge_base_encryption + mock_agent = MagicMock() + mock_client.return_value = mock_agent + paginator = MagicMock() + mock_agent.get_paginator.return_value = paginator + paginator.paginate.side_effect = ClientError( + { + "Error": { + "Code": "UnknownOperationException", + "Message": "Unknown operation ListKnowledgeBases", + } + }, + "ListKnowledgeBases", + ) + result = check(region="ap-southeast-3") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-09" + assert "not available" in findings[0]["Finding_Details"] + @patch("boto3.client") def test_br09_schema_valid(self, mock_client): check = bedrock_app.check_bedrock_knowledge_base_encryption @@ -2168,6 +2191,28 @@ def test_br20_custom_vector_store_returns_na_review(self, mock_client): assert na[0]["Check_ID"] == "BR-20" assert "storage layer" in na[0]["Finding_Details"] + @patch("bedrock_app.boto3.client") + def test_br20_region_unsupported_returns_na(self, mock_client): + # Knowledge Bases API absent in the region -> N/A, not ERROR. + check = bedrock_app.check_bedrock_knowledge_base_kms_encryption + agent_client = MagicMock() + agent_client.list_knowledge_bases.side_effect = ClientError( + { + "Error": { + "Code": "UnknownOperationException", + "Message": "Unknown operation ListKnowledgeBases", + } + }, + "ListKnowledgeBases", + ) + mock_client.return_value = agent_client + + result = check(region="ap-southeast-3") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-20" + assert "not available" in findings[0]["Finding_Details"] + def test_br20_schema_valid(self): check = bedrock_app.check_bedrock_knowledge_base_kms_encryption with patch("bedrock_app.boto3.client") as mock_client: @@ -2309,6 +2354,28 @@ def factory(service, **kwargs): assert len(passed) >= 1 assert passed[0]["Check_ID"] == "BR-21" + @patch("bedrock_app.boto3.client") + def test_br21_region_unsupported_returns_na(self, mock_client): + # Bedrock Agents API absent in the region -> N/A, not ERROR. + check = bedrock_app.check_bedrock_agent_action_group_iam + agent_client = MagicMock() + agent_client.list_agents.side_effect = ClientError( + { + "Error": { + "Code": "UnknownOperationException", + "Message": "Unknown operation ListAgents", + } + }, + "ListAgents", + ) + mock_client.return_value = agent_client + + result = check(region="ap-southeast-3", permission_cache={"role_permissions": {}}) + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-21" + assert "not available" in findings[0]["Finding_Details"] + def test_br21_schema_valid(self): check = bedrock_app.check_bedrock_agent_action_group_iam with patch("bedrock_app.boto3.client") as mock_client: @@ -2471,6 +2538,60 @@ def test_br24_schema_valid(self): assert_finding_schema(f) +# =================================================================== +# BR-25: check_bedrock_rag_evaluation_jobs +# =================================================================== +class TestBR25RAGEvaluationJobs: + """BR-25: Verify RAG applications have evaluation jobs configured.""" + + @patch("bedrock_app.boto3.client") + def test_br25_no_kbs_returns_na(self, mock_client): + check = bedrock_app.check_bedrock_rag_evaluation_jobs + agent_client = MagicMock() + agent_client.list_knowledge_bases.return_value = {"knowledgeBaseSummaries": []} + mock_client.return_value = agent_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-25" + + @patch("bedrock_app.boto3.client") + def test_br25_region_unsupported_returns_na(self, mock_client): + # Knowledge Bases / evaluation API absent in the region -> N/A, not ERROR. + check = bedrock_app.check_bedrock_rag_evaluation_jobs + agent_client = MagicMock() + agent_client.list_knowledge_bases.side_effect = ClientError( + { + "Error": { + "Code": "UnknownOperationException", + "Message": "Unknown operation ListKnowledgeBases", + } + }, + "ListKnowledgeBases", + ) + mock_client.return_value = agent_client + + result = check(region="ap-southeast-3") + findings = extract_csv_data(result) + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-25" + assert "not available" in findings[0]["Finding_Details"] + + def test_br25_schema_valid(self): + check = bedrock_app.check_bedrock_rag_evaluation_jobs + with patch("bedrock_app.boto3.client") as mock_client: + agent_client = MagicMock() + agent_client.list_knowledge_bases.return_value = { + "knowledgeBaseSummaries": [] + } + mock_client.return_value = agent_client + result = check(region="us-east-1") + + for f in extract_csv_data(result): + assert_finding_schema(f) + + # =================================================================== # BR-26: check_bedrock_guardrail_pii_filters # =================================================================== From 46723e4aa0e5ee2e5f3fc0602e4c0a01c4c653bd Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Sat, 27 Jun 2026 11:41:08 -0400 Subject: [PATCH 06/10] fix: report BR-18 as N/A when no Bedrock footprint in region check_bedrock_model_evaluations unconditionally reported Failed when list_evaluation_jobs returned empty, producing a false failure in regions where Bedrock is not used at all. Gate the empty case on detect_bedrock_regional_footprint (matching BR-05): N/A when no regional Bedrock resources exist, Failed only when Bedrock is in use but no evaluation jobs are configured. Add a test for the no-footprint N/A path and pin the existing empty-jobs test to the footprint-present Failed path. --- .../security/bedrock_assessments/app.py | 45 +++++++++++++------ tests/test_bedrock_checks.py | 22 ++++++++- 2 files changed, 53 insertions(+), 14 deletions(-) diff --git a/aiml-security-assessment/functions/security/bedrock_assessments/app.py b/aiml-security-assessment/functions/security/bedrock_assessments/app.py index 0acc90e..f8be450 100644 --- a/aiml-security-assessment/functions/security/bedrock_assessments/app.py +++ b/aiml-security-assessment/functions/security/bedrock_assessments/app.py @@ -3273,20 +3273,39 @@ def check_bedrock_model_evaluations(region: str = "") -> Dict[str, Any]: eval_jobs = eval_jobs_response.get("jobSummaries", []) if not eval_jobs: - findings["status"] = "WARN" - findings["details"] = "No model evaluation jobs found" - findings["csv_data"].append( - create_finding( - check_id="BR-18", - finding_name="Model Evaluation Implementation Check", - finding_details="No Bedrock model evaluation jobs found. Model evaluation helps assess toxicity, accuracy, semantic robustness, and other safety metrics before production deployment.", - resolution="Create model evaluation jobs using Amazon Bedrock Evaluations to assess foundation model performance against safety and quality metrics. Use built-in datasets or custom test sets. Enable LLM-as-a-judge evaluation for comprehensive assessment.", - reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", - severity="Medium", - status="Failed", - region=region, - ) + bedrock_footprint_found = detect_bedrock_regional_footprint( + region=region ) + + if bedrock_footprint_found is False: + findings["details"] = "No regional Bedrock resources found" + findings["csv_data"].append( + create_finding( + check_id="BR-18", + finding_name="Model Evaluation Implementation Check", + finding_details="No regional Bedrock resources found to assess with model evaluation jobs", + resolution="No action required", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + else: + findings["status"] = "WARN" + findings["details"] = "No model evaluation jobs found" + findings["csv_data"].append( + create_finding( + check_id="BR-18", + finding_name="Model Evaluation Implementation Check", + finding_details="No Bedrock model evaluation jobs found. Model evaluation helps assess toxicity, accuracy, semantic robustness, and other safety metrics before production deployment.", + resolution="Create model evaluation jobs using Amazon Bedrock Evaluations to assess foundation model performance against safety and quality metrics. Use built-in datasets or custom test sets. Enable LLM-as-a-judge evaluation for comprehensive assessment.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", + severity="Medium", + status="Failed", + region=region, + ) + ) return findings # Analyze evaluation jobs diff --git a/tests/test_bedrock_checks.py b/tests/test_bedrock_checks.py index 214e970..f89c543 100644 --- a/tests/test_bedrock_checks.py +++ b/tests/test_bedrock_checks.py @@ -1853,8 +1853,11 @@ def test_br17_schema_valid(self): class TestBR18ModelEvaluations: """BR-18: Check if model evaluation jobs exist.""" + @patch("bedrock_app.detect_bedrock_regional_footprint", return_value=True) @patch("bedrock_app.boto3.client") - def test_br18_no_evaluations_returns_failed(self, mock_client): + def test_br18_no_evaluations_with_footprint_returns_failed( + self, mock_client, mock_footprint + ): check = bedrock_app.check_bedrock_model_evaluations bedrock_client = MagicMock() @@ -1868,6 +1871,23 @@ def test_br18_no_evaluations_returns_failed(self, mock_client): assert findings[0]["Check_ID"] == "BR-18" assert findings[0]["Severity"] == "Medium" + @patch("bedrock_app.detect_bedrock_regional_footprint", return_value=False) + @patch("bedrock_app.boto3.client") + def test_br18_no_evaluations_no_footprint_returns_na( + self, mock_client, mock_footprint + ): + check = bedrock_app.check_bedrock_model_evaluations + + bedrock_client = MagicMock() + bedrock_client.list_evaluation_jobs.return_value = {"jobSummaries": []} + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + assert len(findings) >= 1 + assert findings[0]["Status"] == "N/A" + assert findings[0]["Check_ID"] == "BR-18" + @patch("bedrock_app.boto3.client") def test_br18_recent_evaluations_returns_passed(self, mock_client): check = bedrock_app.check_bedrock_model_evaluations From 02fe8e6172c84b8bae79fbe7258f93a588d05840 Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Sat, 27 Jun 2026 12:00:25 -0400 Subject: [PATCH 07/10] docs: correct stale core check count (52 -> 70) The "52 core checks" figure predated the BR-15..32 Bedrock expansion (it assumed BR-01..14). The framework now has 70 core checks (32 Bedrock + 25 SageMaker + 13 AgentCore) and 134 total with the 64 FinServ checks. Update the six stale references so all docs agree with README.md, SECURITY_CHECKS.md, and the codebase. --- docs/DEVELOPER_GUIDE.md | 6 +++--- docs/SECURITY_CHECKS_FINSERV.md | 4 ++-- docs/TROUBLESHOOTING.md | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/DEVELOPER_GUIDE.md b/docs/DEVELOPER_GUIDE.md index 010be26..fa343e5 100644 --- a/docs/DEVELOPER_GUIDE.md +++ b/docs/DEVELOPER_GUIDE.md @@ -44,7 +44,7 @@ ## Architecture Overview -The AI/ML Security Assessment Framework is a serverless, multi-account security assessment solution for AWS AI/ML workloads. It performs 52 core security checks across Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore, with an optional 64-check Financial Services GenAI risk assessment, generating interactive HTML reports with findings and remediation guidance. +The AI/ML Security Assessment Framework is a serverless, multi-account security assessment solution for AWS AI/ML workloads. It performs 70 core security checks across Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore, with an optional 64-check Financial Services GenAI risk assessment, generating interactive HTML reports with findings and remediation guidance. ### Security Design Principles @@ -211,7 +211,7 @@ sample-aiml-security-assessment/ ## Assessment Structure -The framework includes **52 core security checks** across three AI/ML services, plus **64 optional Financial Services GenAI risk checks** when `EnableFinServAssessment` is enabled. For the complete list of checks with descriptions, see the [Security Checks Reference](SECURITY_CHECKS.md). +The framework includes **70 core security checks** across three AI/ML services, plus **64 optional Financial Services GenAI risk checks** when `EnableFinServAssessment` is enabled. For the complete list of checks with descriptions, see the [Security Checks Reference](SECURITY_CHECKS.md). ### AWS Lambda Functions @@ -560,7 +560,7 @@ For detailed troubleshooting guidance, common issues, and debugging tips, see th ## Development Roadmap ### Current Status -- **AI/ML Assessment**: 52 core checks across three services, plus 64 optional Financial Services GenAI risk checks (see [Security Checks Reference](SECURITY_CHECKS.md)) +- **AI/ML Assessment**: 70 core checks across three services, plus 64 optional Financial Services GenAI risk checks (see [Security Checks Reference](SECURITY_CHECKS.md)) ### Potential Additions - **Amazon Comprehend**: Data privacy, access controls, entity recognition security diff --git a/docs/SECURITY_CHECKS_FINSERV.md b/docs/SECURITY_CHECKS_FINSERV.md index 0227eea..433b598 100644 --- a/docs/SECURITY_CHECKS_FINSERV.md +++ b/docs/SECURITY_CHECKS_FINSERV.md @@ -136,7 +136,7 @@ Because `aws-samples` is an OSPO-managed organization, pushes to your personal f ## Relationship to upstream SM/BR/AC checks The upstream [sample-aiml-security-assessment](https://github.com/aws-samples/sample-aiml-security-assessment) -framework already provides 52 core security checks (SM-01 to SM-25, BR-01 to BR-14, AC-01 to AC-13). +framework already provides 70 core security checks (SM-01 to SM-25, BR-01 to BR-32, AC-01 to AC-13). The 69 FS checks in this document are **additive**: they enhance the upstream with FinServ-specific detection and remediation guidance drawn from the Responsible AI GRC guide. A few FS checks overlap with upstream checks — in those cases, the FS check adds FinServ-specific depth @@ -179,7 +179,7 @@ whether the FS check adds FinServ-specific regulatory specificity, (3) severity - **Extend upstream (5 FS checks merged into 5 upstream checks):** FS-17 → SM-07; FS-18 → SM-23; FS-19 → SM-22; FS-23 → BR-06; FS-64 → BR-04. These checks are replaced by upstream-extension notes in Parts 1 and 3 and are removed from `finserv_assessments/app.py`. - **Keep separate (64 FS checks):** All other FS checks ship as standalone entries. This includes FS-20, FS-22, FS-25, FS-26, FS-39, FS-41, all Guardrail-policy-level checks (FS-27, FS-28, FS-36, FS-38, FS-45, FS-47, FS-50, FS-51, FS-59), and all FS checks that have no upstream overlap at all. -After consolidation the combined framework contains **52 upstream + 64 FS = 116 distinct checks** (down from 52 + 69 = 121 before merging). The consolidation reduces duplication without losing FinServ-specific regulatory depth. +After consolidation the combined framework contains **70 upstream + 64 FS = 134 distinct checks** (down from 70 + 69 = 139 before merging). The consolidation reduces duplication without losing FinServ-specific regulatory depth. --- diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 09e29bb..786eb6f 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -323,7 +323,7 @@ A: Minimal ongoing costs: **Q: Can I customize which security checks are included?** -A: Currently, all 52 core checks run by default to provide comprehensive coverage. If `EnableFinServAssessment` is enabled, the 64 optional Financial Services GenAI risk checks also run. You can filter results in the generated HTML reports by severity, status, service, industry, or region. Future versions may support selective check execution. +A: Currently, all 70 core checks run by default to provide comprehensive coverage. If `EnableFinServAssessment` is enabled, the 64 optional Financial Services GenAI risk checks also run. You can filter results in the generated HTML reports by severity, status, service, industry, or region. Future versions may support selective check execution. **Q: Can I add custom security checks?** From 97b03d48fd18e86661e52252f4e3798dcbf8c87d Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Sat, 27 Jun 2026 12:09:17 -0400 Subject: [PATCH 08/10] style: apply ruff format to test_bedrock_checks.py --- tests/test_bedrock_checks.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/test_bedrock_checks.py b/tests/test_bedrock_checks.py index f89c543..e63485a 100644 --- a/tests/test_bedrock_checks.py +++ b/tests/test_bedrock_checks.py @@ -2390,7 +2390,9 @@ def test_br21_region_unsupported_returns_na(self, mock_client): ) mock_client.return_value = agent_client - result = check(region="ap-southeast-3", permission_cache={"role_permissions": {}}) + result = check( + region="ap-southeast-3", permission_cache={"role_permissions": {}} + ) findings = extract_csv_data(result) assert findings[0]["Status"] == "N/A" assert findings[0]["Check_ID"] == "BR-21" From d30253f24006d138f469594a75ccd48a6a8f9f0d Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Sun, 28 Jun 2026 00:48:29 -0400 Subject: [PATCH 09/10] Add Agentic AI security lens; harden IAM coverage and N/A semantics Adds the synthesized Agentic AI (AG-) assessment area across Bedrock and AgentCore, plus report-layer routing, expanded checks, and tests. Review fixes folded in: - Sync FinServ + cloudwatch:PutMetricData grants into the local MemberRole in deployment/2-aiml-security-codebuild.yaml (was missing whole module) - Add cloudwatch:PutMetricData to all three deployment role files - AC-02 empty permission cache -> N/A/Informational (was Failed) - Distinguish indeterminate (None) from probed-empty (False) Bedrock footprint in N/A finding text via bedrock_footprint_na_detail() --- .gitignore | 3 + README.md | 11 +- .../security/agentcore_assessments/app.py | 442 ++++++++++++- .../security/bedrock_assessments/app.py | 625 +++++++++++++++--- .../generate_consolidated_report/app.py | 31 +- .../report_template.py | 136 +++- .../test_generate_report.py | 60 ++ .../template-multi-account.yaml | 2 + aiml-security-assessment/template.yaml | 2 + consolidate_html_reports.py | 11 +- deployment/1-aiml-security-member-roles.yaml | 3 + deployment/2-aiml-security-codebuild.yaml | 49 ++ deployment/aiml-security-single-account.yaml | 3 + docs/DEVELOPER_GUIDE.md | 22 +- docs/SECURITY_CHECKS.md | 213 +++++- docs/SECURITY_CHECKS_FINSERV.md | 4 +- docs/TROUBLESHOOTING.md | 2 +- sample-reports/README.md | 3 + tests/test_agentcore_checks.py | 201 +++++- tests/test_bedrock_checks.py | 175 ++++- tests/test_consolidated_report.py | 31 + tests/test_core_iam_coverage.py | 2 + 22 files changed, 1901 insertions(+), 130 deletions(-) diff --git a/.gitignore b/.gitignore index d6df932..5618451 100644 --- a/.gitignore +++ b/.gitignore @@ -5,6 +5,9 @@ .codex/ .mcp.json +AGENTS.md +CLAUDE.md + # AWS SAM build artifacts aiml-security-assessment/.aws-sam/ diff --git a/README.md b/README.md index ef5eb9d..302bab3 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ **Open-source automated security scanner for generative AI and machine learning workloads on AWS.** Core checks for Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore are built on the [AWS Well-Architected Framework — Generative AI Lens](https://docs.aws.amazon.com/wellarchitected/latest/generative-ai-lens/generative-ai-lens.html). An optional Financial Services GenAI risk module adds 64 checks aligned to the [AWS User Guide to Governance, Risk, and Compliance for Responsible AI Adoption within Financial Services Industries](https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/whitepapers/compliance/AWS-User-Guide-Governance-Risk-Compliance-for-Responsible-AI-Adoption-Financial-Services.pdf). See the [AWS Security Blog announcement](https://aws.amazon.com/blogs/security/introducing-the-updated-aws-user-guide-to-governance-risk-and-compliance-for-responsible-ai-adoption/) for context on the updated guide. -Run **[134 security checks](docs/SECURITY_CHECKS.md)** across your AWS accounts and regions in one deployment. Surfaces IAM misconfigurations, encryption gaps, network isolation issues, missing guardrails, and governance gaps — with interactive HTML reports, severity ratings, and AWS documentation links for remediation. Single-account or full AWS Organizations multi-account scans; all data stays in your account. +Run **[161 security checks](docs/SECURITY_CHECKS.md)** across your AWS accounts and regions in one deployment. Surfaces IAM misconfigurations, encryption gaps, network isolation issues, missing guardrails, and governance gaps — with interactive HTML reports, severity ratings, and AWS documentation links for remediation. Single-account or full AWS Organizations multi-account scans; all data stays in your account. --- @@ -39,7 +39,7 @@ The framework generates professional, interactive security assessment reports wi - **Executive Summary** with severity counts and service breakdown - **Priority Recommendations** highlighting critical issues requiring immediate attention -- **[134 Security Checks](docs/SECURITY_CHECKS.md)** across Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and Financial Services GenAI Risk +- **[161 Security Checks](docs/SECURITY_CHECKS.md)** across Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, Agentic AI Security, and Financial Services GenAI Risk - **Multi-Region Support** for core Bedrock, SageMaker, and AgentCore checks, with per-region risk breakdown - **Interactive Filtering** by account, region, service, severity, and status - **Light/Dark Mode Toggle** with persistent user preference @@ -83,7 +83,7 @@ Designed for workloads using [Amazon Bedrock](https://aws.amazon.com/bedrock/), | Challenge | How This Framework Helps | |-----------|-------------------------| | **Manual security audits are time-consuming** | Fully automated scanning with one-click CloudFormation deployment | -| **Inconsistent security checks across teams** | Standardized 134-check assessment based on AWS Well-Architected Generative AI Lens best practices and AWS Responsible AI governance, risk, and compliance guidance for financial services | +| **Inconsistent security checks across teams** | Standardized 161-check assessment based on AWS Well-Architected Generative AI Lens and Agentic AI Lens best practices and AWS Responsible AI governance, risk, and compliance guidance for financial services | | **Difficulty tracking AI/ML security posture** | Interactive HTML dashboards with severity breakdown and per-account visibility | | **Multi-account complexity** | Consolidated reporting across AWS Organizations with cross-account role assumption | | **Compliance and audit support** | Exportable reports to supplement your compliance program, with remediation guidance linked to AWS documentation | @@ -93,6 +93,7 @@ Designed for workloads using [Amazon Bedrock](https://aws.amazon.com/bedrock/), - **[Amazon Bedrock](docs/SECURITY_CHECKS.md#amazon-bedrock-security-checks-32)** (32 checks) - Guardrails, cross-account policies, guardrail tiers, content filters, sensitive-information/PII filters, contextual grounding, automated reasoning, encryption (custom, imported, knowledge base, and batch inference output), Amazon VPC endpoints, AWS IAM permissions, agent guardrail association and least privilege, model invocation logging, CloudWatch alarms, model evaluation, prompt flow validation, RAG evaluation, service quotas - **[Amazon SageMaker AI](docs/SECURITY_CHECKS.md#amazon-sagemaker-ai-security-checks-25)** (25 checks) - AWS Security Hub controls (SageMaker.1-5), encryption, network isolation, AWS IAM, MLOps - **[Amazon Bedrock AgentCore](docs/SECURITY_CHECKS.md#amazon-bedrock-agentcore-security-checks-13)** (13 checks) - Amazon VPC configuration, encryption, observability, resource policies +- **[Agentic AI Security](docs/SECURITY_CHECKS.md#agentic-ai-security-checks-27)** (27 always-on checks) - Bounded autonomy, agent identity, tool authorization, guardrail enforcement, prompt/input protection, memory privacy, auditability, and abuse protection, aligned to the [AWS Well-Architected Agentic AI Lens](https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentic-ai-lens.html). Synthesized by re-mapping Amazon Bedrock and AgentCore findings, with native AgentCore gateway checks - **[Financial Services GenAI Risk](docs/SECURITY_CHECKS.md#financial-services-genai-risk-checks-64-additional-5-upstream-extensions)** (64 checks) - Unbounded consumption, excessive agency, supply chain, training data poisoning, hallucination, prompt injection, PII disclosure, and 8 more FinServ-specific risk categories derived from the [AWS User Guide to Governance, Risk, and Compliance for Responsible AI Adoption within Financial Services Industries](https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/whitepapers/compliance/AWS-User-Guide-Governance-Risk-Compliance-for-Responsible-AI-Adoption-Financial-Services.pdf). See the [AWS Security Blog announcement](https://aws.amazon.com/blogs/security/introducing-the-updated-aws-user-guide-to-governance-risk-and-compliance-for-responsible-ai-adoption/) for context on the updated guide. **Deployment Options:** @@ -115,7 +116,7 @@ This tool operates within the [AWS Shared Responsibility Model](https://aws.amaz **No guarantee of security or compliance.** This framework identifies common misconfigurations based on AWS best practices and the AWS Well-Architected Framework. It does not cover all possible security risks, does not replace formal compliance audits (SOC 2, HIPAA, and similar), and does not guarantee that your workloads are secure. Use the results as one input into your broader security program. -**134 checks across four domains.** The assessment covers Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, and optional Financial Services GenAI risk checks. Other AI/ML services (Amazon Comprehend, Amazon Rekognition, Amazon Textract, and others) are not currently assessed. +**161 checks across five areas.** The assessment covers Amazon Bedrock, Amazon SageMaker AI, Amazon Bedrock AgentCore, always-on Agentic AI Security, and optional Financial Services GenAI risk checks. Other AI/ML services (Amazon Comprehend, Amazon Rekognition, Amazon Textract, and others) are not currently assessed. --- @@ -386,7 +387,7 @@ If you need to reduce scope, review the role policies in: | Document | Description | |----------|-------------| -| [Security Checks Reference](docs/SECURITY_CHECKS.md) | Complete reference for all 134 security checks with severity levels | +| [Security Checks Reference](docs/SECURITY_CHECKS.md) | Complete reference for all 161 security checks with severity levels | | [FinServ GenAI Risk Checks](docs/SECURITY_CHECKS_FINSERV.md) | Complete FS-01..69 reference: shared introduction, severity rubric, upstream-overlap table, compliance framework mapping, and all check definitions (Part 1 infrastructure controls, Part 2 guardrails & content safety, Part 3 app-layer controls & gaps) | | [FinServ Severity Methodology](docs/SECURITY_CHECKS_FINSERV_SEVERITY_METHODOLOGY.md) | Likelihood × Impact → ASFF severity model, disposition rules, and research basis for FS check severities | | [FinServ Severity Register](docs/SECURITY_CHECKS_FINSERV_SEVERITY_REGISTER.md) | Authoritative per-finding severity assignments (the single source of truth enforced by the drift-guard test) | diff --git a/aiml-security-assessment/functions/security/agentcore_assessments/app.py b/aiml-security-assessment/functions/security/agentcore_assessments/app.py index 595f9be..71a16da 100644 --- a/aiml-security-assessment/functions/security/agentcore_assessments/app.py +++ b/aiml-security-assessment/functions/security/agentcore_assessments/app.py @@ -47,6 +47,85 @@ # duplicate findings when scanning multiple regions. GLOBAL_REGION_LABEL = "Global" +AGENTIC_AI_LENS_URL = ( + "https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/" + "agentic-ai-lens.html" +) +AGENTCORE_GATEWAY_API_REFERENCE_URL = ( + "https://docs.aws.amazon.com/bedrock-agentcore-control/latest/APIReference/" + "API_GetGateway.html" +) +AGENTCORE_POLICY_ENGINE_REFERENCE_URL = ( + "https://docs.aws.amazon.com/bedrock-agentcore-control/latest/APIReference/" + "API_GatewayPolicyEngineConfiguration.html" +) + +AGENTIC_AGENTCORE_CHECK_MAPPINGS = { + "AC-01": { + "check_id": "AG-15", + "finding": "Agentic AI Runtime Network Boundary", + "lens_domain": "Bounded Autonomy", + "agentic_context": "Agent runtimes should execute inside explicit network boundaries to reduce unintended external reachability.", + "resolution": "Configure AgentCore runtimes with appropriate VPC settings and restrict network paths to required services.", + }, + "AC-02": { + "check_id": "AG-16", + "finding": "Agentic AI AgentCore Least Privilege", + "lens_domain": "Agent Identity & Access", + "agentic_context": "Over-permissive AgentCore principals can let agents or operators bypass intended autonomy and tool boundaries.", + "resolution": "Replace full-access AgentCore permissions with least-privilege IAM policies scoped to required resources and actions.", + }, + "AC-03": { + "check_id": "AG-17", + "finding": "Agentic AI Stale AgentCore Access", + "lens_domain": "Agent Identity & Access", + "agentic_context": "Unused AgentCore permissions increase the blast radius of compromised principals.", + "resolution": "Remove or restrict stale AgentCore permissions for principals that no longer need access.", + }, + "AC-04": { + "check_id": "AG-18", + "finding": "Agentic AI AgentCore Observability", + "lens_domain": "Auditability & Observability", + "agentic_context": "AgentCore observability provides the telemetry needed to investigate runtime, tool, memory, and gateway behavior.", + "resolution": "Enable CloudWatch Logs, tracing, and AgentCore observability for runtime and gateway resources where supported.", + }, + "AC-07": { + "check_id": "AG-19", + "finding": "Agentic AI Memory Data Protection", + "lens_domain": "Memory & Data Privacy", + "agentic_context": "Agent memory can contain sensitive user or business context and should use customer-controlled encryption where required.", + "resolution": "Configure AgentCore memory resources with customer-managed KMS keys and review memory access permissions.", + }, + "AC-08": { + "check_id": "AG-20", + "finding": "Agentic AI Private AgentCore Connectivity", + "lens_domain": "Bounded Autonomy", + "agentic_context": "Private service connectivity reduces exposure for agents that access AgentCore control or runtime services.", + "resolution": "Create required VPC endpoints for AgentCore services and validate endpoint availability.", + }, + "AC-10": { + "check_id": "AG-21", + "finding": "Agentic AI Resource Policy Boundary", + "lens_domain": "Agent Identity & Access", + "agentic_context": "Resource-based policies add a second authorization boundary for AgentCore runtimes and gateways.", + "resolution": "Attach resource-based policies to AgentCore resources to constrain principals, accounts, and network sources.", + }, + "AC-11": { + "check_id": "AG-22", + "finding": "Agentic AI Policy Engine Data Protection", + "lens_domain": "Tool Authorization", + "agentic_context": "Policy engines contain authorization logic for tool calls and should be protected with appropriate encryption controls.", + "resolution": "Configure policy engines with customer-managed KMS keys where enhanced key control is required.", + }, + "AC-12": { + "check_id": "AG-23", + "finding": "Agentic AI Gateway Data Protection", + "lens_domain": "Tool Authorization", + "agentic_context": "Gateway configuration can include tool schemas, target definitions, and integration metadata.", + "resolution": "Configure AgentCore gateways with customer-managed KMS keys where enhanced key control is required.", + }, +} + # Error codes returned when a region exists but is not enabled/usable for the # account (opt-in regions, disabled regions). The availability probe treats # these the same as an endpoint connection failure. @@ -255,6 +334,72 @@ def generate_csv_report(findings: List[Dict[str, Any]]) -> str: return csv_content +def build_agentic_agentcore_security_findings( + findings: List[Dict[str, Any]], +) -> List[Dict[str, Any]]: + """Create AG-* rows from AgentCore checks that already prove agentic controls.""" + agentic_findings = [] + for row in findings: + source_check_id = row.get("Check_ID", "") + mapping = AGENTIC_AGENTCORE_CHECK_MAPPINGS.get(source_check_id) + if not mapping: + continue + + status = row.get("Status", "N/A") + severity = row.get("Severity", "Informational") + if status == "N/A": + severity = "Informational" + + agentic_findings.append( + create_finding( + check_id=mapping["check_id"], + finding_name=mapping["finding"], + finding_details=( + f"Agentic AI security domain: {mapping['lens_domain']}. " + f"{mapping['agentic_context']} " + f"Source check {source_check_id}: {row.get('Finding_Details', '')}" + ), + resolution=mapping["resolution"], + reference=AGENTIC_AI_LENS_URL, + severity=severity, + status=status, + region=row.get("Region", ""), + ) + ) + return agentic_findings + + +def build_agentic_agentcore_unavailable_findings( + region: str, existing_findings: List[Dict[str, Any]] +) -> List[Dict[str, Any]]: + """Create N/A AG-* rows for AgentCore-derived checks that could not run.""" + existing_check_ids = {finding.get("Check_ID") for finding in existing_findings} + unavailable_findings = [] + + for mapping in AGENTIC_AGENTCORE_CHECK_MAPPINGS.values(): + if mapping["check_id"] in existing_check_ids: + continue + + unavailable_findings.append( + create_finding( + check_id=mapping["check_id"], + finding_name=mapping["finding"], + finding_details=( + f"Agentic AI security domain: {mapping['lens_domain']}. " + f"This AgentCore-derived control could not be assessed because " + f"Amazon Bedrock AgentCore is not available in region {region}." + ), + resolution="No action required unless AgentCore workloads are expected in this region.", + reference=AGENTIC_AI_LENS_URL, + severity=SeverityEnum.INFORMATIONAL, + status=StatusEnum.NA, + region=region, + ) + ) + + return unavailable_findings + + def write_to_s3( execution_id: str, csv_content: str, bucket_name: str, region: str = "" ) -> str: @@ -979,8 +1124,8 @@ def check_stale_agentcore_access( finding_details=f"The following principals have AgentCore permissions but have never accessed the service: {never_accessed_details}", resolution="Review and remove unused AgentCore permissions following least privilege principle", reference="https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_last-accessed.html", - severity=SeverityEnum.MEDIUM, - status=StatusEnum.FAILED, + severity=SeverityEnum.INFORMATIONAL, + status=StatusEnum.NA, ) ) @@ -2449,6 +2594,284 @@ def check_agentcore_gateway_configuration() -> List[Dict[str, Any]]: return findings +def check_agentcore_gateway_agentic_security() -> List[Dict[str, Any]]: + """ + Check API-provable AgentCore Gateway controls for agentic tool execution. + + Validates: + - Inbound gateway authorization is enabled + - Policy engine is attached in ENFORCE mode + - Debug exception detail is not exposed + - AWS WAF web ACL is associated + """ + findings = [] + + if agentcore_client is None: + for check_id, finding_name in [ + ("AG-24", "Agentic AI Gateway Inbound Authorization"), + ("AG-25", "Agentic AI Gateway Tool Policy Enforcement"), + ("AG-26", "Agentic AI Gateway Error Detail Exposure"), + ("AG-27", "Agentic AI Gateway WAF Protection"), + ]: + findings.append( + create_finding( + check_id=check_id, + finding_name=finding_name, + finding_details="AgentCore client not available in this region", + resolution="Deploy in a region where Amazon Bedrock AgentCore is available", + reference=AGENTIC_AI_LENS_URL, + severity=SeverityEnum.INFORMATIONAL, + status=StatusEnum.NA, + ) + ) + return findings + + try: + gateways = _agentcore_list_all("list_gateways", ["items", "gateways"]) + except AttributeError: + return [ + create_finding( + check_id="AG-24", + finding_name="Agentic AI Gateway Security Controls", + finding_details="Gateway APIs not yet available in bedrock-agentcore-control client", + resolution="Upgrade the AWS SDK/runtime when AgentCore Gateway APIs are available", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.INFORMATIONAL, + status=StatusEnum.NA, + ) + ] + except ClientError as e: + status = ( + StatusEnum.NA if _is_access_denied_client_error(e) else StatusEnum.FAILED + ) + severity = ( + SeverityEnum.INFORMATIONAL + if status == StatusEnum.NA + else SeverityEnum.MEDIUM + ) + return [ + create_finding( + check_id="AG-24", + finding_name="Agentic AI Gateway Security Controls", + finding_details=f"Unable to list AgentCore Gateways: {str(e)}", + resolution="Grant bedrock-agentcore:ListGateways and retry the assessment", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=severity, + status=status, + ) + ] + + if not gateways: + return [ + create_finding( + check_id="AG-24", + finding_name="Agentic AI Gateway Inbound Authorization", + finding_details="No AgentCore Gateways found", + resolution="No action required", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.INFORMATIONAL, + status=StatusEnum.NA, + ), + create_finding( + check_id="AG-25", + finding_name="Agentic AI Gateway Tool Policy Enforcement", + finding_details="No AgentCore Gateways found", + resolution="No action required", + reference=AGENTCORE_POLICY_ENGINE_REFERENCE_URL, + severity=SeverityEnum.INFORMATIONAL, + status=StatusEnum.NA, + ), + create_finding( + check_id="AG-26", + finding_name="Agentic AI Gateway Error Detail Exposure", + finding_details="No AgentCore Gateways found", + resolution="No action required", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.INFORMATIONAL, + status=StatusEnum.NA, + ), + create_finding( + check_id="AG-27", + finding_name="Agentic AI Gateway WAF Protection", + finding_details="No AgentCore Gateways found", + resolution="No action required", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.INFORMATIONAL, + status=StatusEnum.NA, + ), + ] + + for gateway in gateways: + gateway_id = gateway.get("gatewayId", "unknown") + gateway_name = gateway.get("name", gateway_id) + + try: + gateway_details = agentcore_client.get_gateway(gatewayIdentifier=gateway_id) + except ClientError as e: + findings.append( + create_finding( + check_id="AG-24", + finding_name="Agentic AI Gateway Security Controls", + finding_details=f"Unable to retrieve Gateway '{gateway_name}' ({gateway_id}): {str(e)}", + resolution="Grant bedrock-agentcore:GetGateway and retry the assessment", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.INFORMATIONAL, + status=StatusEnum.NA, + ) + ) + continue + + authorizer_type = gateway_details.get("authorizerType") or gateway.get( + "authorizerType" + ) + policy_engine_config = gateway_details.get("policyEngineConfiguration") or {} + policy_engine_mode = policy_engine_config.get("mode") + policy_engine_arn = policy_engine_config.get("arn") + + if authorizer_type in {"AWS_IAM", "CUSTOM_JWT"}: + findings.append( + create_finding( + check_id="AG-24", + finding_name="Agentic AI Gateway Inbound Authorization", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) uses authorizerType {authorizer_type}.", + resolution="No action required", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.HIGH, + status=StatusEnum.PASSED, + ) + ) + elif ( + authorizer_type == "AUTHENTICATE_ONLY" + and policy_engine_mode == "ENFORCE" + and policy_engine_arn + ): + findings.append( + create_finding( + check_id="AG-24", + finding_name="Agentic AI Gateway Inbound Authorization", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) uses authorizerType AUTHENTICATE_ONLY and delegates authorization to policy engine {policy_engine_arn} in ENFORCE mode.", + resolution="No action required. Continue validating policy coverage for all exposed gateway targets.", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.HIGH, + status=StatusEnum.PASSED, + ) + ) + elif authorizer_type == "AUTHENTICATE_ONLY": + findings.append( + create_finding( + check_id="AG-24", + finding_name="Agentic AI Gateway Authenticate-Only Authorization", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) uses authorizerType AUTHENTICATE_ONLY without an attached policy engine in ENFORCE mode. AgentCore Gateway authenticates the SigV4 caller but does not make an authorization decision for this authorizer type.", + resolution="Use AWS_IAM or CUSTOM_JWT for gateway-enforced authorization, or attach an AgentCore policy engine in ENFORCE mode before using AUTHENTICATE_ONLY.", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.HIGH, + status=StatusEnum.FAILED, + ) + ) + else: + findings.append( + create_finding( + check_id="AG-24", + finding_name="Agentic AI Gateway Inbound Authorization Disabled", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) uses authorizerType {authorizer_type or 'unspecified'}. Agent tool endpoints must use an explicit gateway authorizer.", + resolution="Configure the gateway authorizerType as AWS_IAM or CUSTOM_JWT and provide the required authorizer configuration.", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.HIGH, + status=StatusEnum.FAILED, + ) + ) + + if not policy_engine_config: + findings.append( + create_finding( + check_id="AG-25", + finding_name="Agentic AI Gateway Tool Policy Enforcement Missing", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) does not have a policy engine configuration. Tool calls are not evaluated by AgentCore policy enforcement.", + resolution="Attach an AgentCore policy engine to the gateway and use ENFORCE mode for production tool authorization.", + reference=AGENTCORE_POLICY_ENGINE_REFERENCE_URL, + severity=SeverityEnum.HIGH, + status=StatusEnum.FAILED, + ) + ) + elif policy_engine_mode != "ENFORCE": + findings.append( + create_finding( + check_id="AG-25", + finding_name="Agentic AI Gateway Tool Policy Not Enforced", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) policy engine {policy_engine_arn or 'unknown'} is in {policy_engine_mode or 'unknown'} mode. LOG_ONLY mode records decisions but does not block denied tool calls.", + resolution="Change the gateway policyEngineConfiguration mode to ENFORCE after validating policies in LOG_ONLY mode.", + reference=AGENTCORE_POLICY_ENGINE_REFERENCE_URL, + severity=SeverityEnum.HIGH, + status=StatusEnum.FAILED, + ) + ) + else: + findings.append( + create_finding( + check_id="AG-25", + finding_name="Agentic AI Gateway Tool Policy Enforcement", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) has policy engine {policy_engine_arn or 'unknown'} in ENFORCE mode.", + resolution="No action required", + reference=AGENTCORE_POLICY_ENGINE_REFERENCE_URL, + severity=SeverityEnum.HIGH, + status=StatusEnum.PASSED, + ) + ) + + if gateway_details.get("exceptionLevel") == "DEBUG": + findings.append( + create_finding( + check_id="AG-26", + finding_name="Agentic AI Gateway Debug Error Detail Enabled", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) returns DEBUG-level exception detail. Detailed errors can disclose tool, target, or policy implementation details to callers.", + resolution="Remove DEBUG exceptionLevel for production gateways so callers receive generic gateway errors.", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.MEDIUM, + status=StatusEnum.FAILED, + ) + ) + else: + findings.append( + create_finding( + check_id="AG-26", + finding_name="Agentic AI Gateway Error Detail Exposure", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) does not expose DEBUG-level exception detail.", + resolution="No action required", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.MEDIUM, + status=StatusEnum.PASSED, + ) + ) + + web_acl_arn = gateway_details.get("webAclArn") + if web_acl_arn: + findings.append( + create_finding( + check_id="AG-27", + finding_name="Agentic AI Gateway WAF Protection", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) is associated with WAF web ACL {web_acl_arn}.", + resolution="No action required", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.LOW, + status=StatusEnum.PASSED, + ) + ) + else: + findings.append( + create_finding( + check_id="AG-27", + finding_name="Agentic AI Gateway WAF Protection Missing", + finding_details=f"Gateway '{gateway_name}' ({gateway_id}) is not associated with an AWS WAF web ACL.", + resolution="Associate an AWS WAF web ACL with internet-facing AgentCore gateways to add request filtering and abuse protection.", + reference=AGENTCORE_GATEWAY_API_REFERENCE_URL, + severity=SeverityEnum.LOW, + status=StatusEnum.FAILED, + ) + ) + + return findings + + def lambda_handler(event, context): """ Lambda handler for AgentCore security assessment. @@ -2599,6 +3022,14 @@ def lambda_handler(event, context): region=region, ) ) + for finding in all_findings: + if not finding.get("Region"): + finding["Region"] = region + all_findings.extend(check_agentcore_gateway_agentic_security()) + all_findings.extend(build_agentic_agentcore_security_findings(all_findings)) + all_findings.extend( + build_agentic_agentcore_unavailable_findings(region, all_findings) + ) for finding in all_findings: if not finding.get("Region"): finding["Region"] = region @@ -2632,6 +3063,7 @@ def lambda_handler(event, context): ("Resource-Based Policies", check_agentcore_resource_based_policies), ("Policy Engine Encryption", check_agentcore_policy_engine_encryption), ("Gateway Encryption", check_agentcore_gateway_encryption), + ("Agentic Gateway Security", check_agentcore_gateway_agentic_security), ] for check_name, check_func in checks: @@ -2673,6 +3105,12 @@ def lambda_handler(event, context): if not finding.get("Region"): finding["Region"] = region + logger.info("Building Agentic AI Security findings from AgentCore results") + all_findings.extend(build_agentic_agentcore_security_findings(all_findings)) + for finding in all_findings: + if not finding.get("Region"): + finding["Region"] = region + # Generate CSV report logger.info(f"Generating CSV report with {len(all_findings)} total findings") csv_content = generate_csv_report(all_findings) diff --git a/aiml-security-assessment/functions/security/bedrock_assessments/app.py b/aiml-security-assessment/functions/security/bedrock_assessments/app.py index f8be450..ee5301f 100644 --- a/aiml-security-assessment/functions/security/bedrock_assessments/app.py +++ b/aiml-security-assessment/functions/security/bedrock_assessments/app.py @@ -9,6 +9,7 @@ from botocore.config import Config from botocore.exceptions import ClientError, EndpointConnectionError import random +import re import json from schema import create_finding @@ -31,6 +32,112 @@ # duplicate findings when scanning multiple regions. GLOBAL_REGION_LABEL = "Global" +AGENTIC_AI_LENS_URL = ( + "https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/" + "agentic-ai-lens.html" +) + +AGENTIC_BEDROCK_CHECK_MAPPINGS = { + "BR-04": { + "check_id": "AG-07", + "finding": "Agentic AI Model Invocation Logging", + "lens_domain": "Auditability & Observability", + "agentic_context": "Agents can take multi-step actions, so prompt, response, and guardrail traces need to be available for investigation.", + "resolution": "Enable Amazon Bedrock model invocation logging and retain logs according to your incident response and data governance requirements.", + }, + "BR-06": { + "check_id": "AG-08", + "finding": "Agentic AI API Audit Trail", + "lens_domain": "Auditability & Observability", + "agentic_context": "Agent activity must be attributable through CloudTrail events for Bedrock control plane and runtime operations.", + "resolution": "Enable CloudTrail trails with management event logging and validate that Bedrock API activity is captured.", + }, + "BR-15": { + "check_id": "AG-09", + "finding": "Agentic AI Guardrail Enforcement Boundary", + "lens_domain": "Guardrail Enforcement", + "agentic_context": "Organization-level guardrail enforcement helps prevent agents from bypassing required safety controls across accounts.", + "resolution": "Use IAM and organization controls to require approved guardrails for model and agent invocations where supported.", + }, + "BR-18": { + "check_id": "AG-10", + "finding": "Agentic AI Adversarial Evaluation Coverage", + "lens_domain": "Prompt & Input Protection", + "agentic_context": "Agentic applications should be tested for adversarial prompts and unsafe behaviors before production use.", + "resolution": "Configure model or application evaluations that include adversarial, safety, and security-relevant test cases.", + }, + "BR-19": { + "check_id": "AG-11", + "finding": "Agentic AI Prompt Flow Validation", + "lens_domain": "Prompt & Input Protection", + "agentic_context": "Validated prompt flows reduce the risk that malformed orchestration logic causes unsafe agent behavior.", + "resolution": "Validate Bedrock flow definitions before deployment and remediate validation findings before publishing new versions.", + }, + "BR-21": { + "check_id": "AG-06", + "finding": "Agentic AI Tool Execution Least Privilege", + "lens_domain": "Tool Authorization", + "agentic_context": "Agent action groups are tool execution boundaries; over-permissive roles can let an agent perform unintended operations.", + "resolution": "Restrict action group Lambda roles and referenced IAM permissions to the specific tools, resources, and actions required.", + }, + "BR-22": { + "check_id": "AG-12", + "finding": "Agentic AI Invocation Abuse Controls", + "lens_domain": "Abuse & Cost Protection", + "agentic_context": "Autonomous agents can amplify token usage through retries, loops, or high-volume tool workflows.", + "resolution": "Configure service quotas, throttling limits, and alerting to detect and limit abnormal model invocation patterns.", + }, + "BR-23": { + "check_id": "AG-02", + "finding": "Agentic AI Harmful Content Guardrail Coverage", + "lens_domain": "Guardrail Enforcement", + "agentic_context": "Agents should use guardrails that filter harmful content in both intermediate and final responses.", + "resolution": "Configure guardrails with appropriate content filters and thresholds for all agent-facing workloads.", + }, + "BR-24": { + "check_id": "AG-04", + "finding": "Agentic AI Automated Reasoning Guardrails", + "lens_domain": "Guardrail Enforcement", + "agentic_context": "Automated reasoning policies help verify agent responses against deterministic business or safety rules.", + "resolution": "Configure automated reasoning policies on guardrails where formal response validation is required.", + }, + "BR-26": { + "check_id": "AG-03", + "finding": "Agentic AI Sensitive Information Protection", + "lens_domain": "Memory & Data Privacy", + "agentic_context": "Agents can receive and produce sensitive data across conversations, tool calls, and retrieved context.", + "resolution": "Configure guardrail sensitive-information filters for PII entities and custom sensitive-data patterns.", + }, + "BR-27": { + "check_id": "AG-05", + "finding": "Agentic AI Grounding Controls", + "lens_domain": "Prompt & Input Protection", + "agentic_context": "Grounding checks reduce the chance that an agent acts on hallucinated or irrelevant context.", + "resolution": "Enable contextual grounding checks on guardrails for RAG and tool-using agent workflows.", + }, + "BR-28": { + "check_id": "AG-01", + "finding": "Agentic AI Agent Guardrail Association", + "lens_domain": "Guardrail Enforcement", + "agentic_context": "Bedrock agents should have guardrails associated so autonomous interactions are filtered consistently.", + "resolution": "Associate an approved Bedrock guardrail with each Bedrock agent and prepare the agent after updating.", + }, + "BR-29": { + "check_id": "AG-13", + "finding": "Agentic AI Session Boundary", + "lens_domain": "Bounded Autonomy", + "agentic_context": "Long-lived idle sessions widen the window for session reuse and unintended continuation of agent context.", + "resolution": "Set a conservative idleSessionTTLInSeconds value for agents based on application session requirements.", + }, + "BR-32": { + "check_id": "AG-14", + "finding": "Agentic AI Operational Abuse Alarms", + "lens_domain": "Abuse & Cost Protection", + "agentic_context": "CloudWatch alarms help detect anomalous invocation errors, throttling, or volume caused by autonomous workflows.", + "resolution": "Configure CloudWatch alarms for Bedrock invocation errors, throttles, latency, and token or request volume where metrics are available.", + }, +} + # Error codes returned when a region exists but is not enabled/usable for the # account (opt-in regions, disabled regions). The availability probe treats # these the same as an endpoint connection failure. @@ -159,6 +266,48 @@ def _first_page_items(paginator, result_key: str) -> List[Dict[str, Any]]: return [] +def _list_all_items( + client, + operation_name: str, + result_key: str, + *, + max_results_param: str = "maxResults", + token_param: str = "nextToken", + token_response_keys: tuple = ("nextToken", "NextToken"), + max_results: int = 100, + **kwargs, +) -> List[Dict[str, Any]]: + """Collect all items from list APIs that expose explicit next-token fields.""" + items = [] + next_token = None + operation = getattr(client, operation_name) + + while True: + request = dict(kwargs) + request[max_results_param] = max_results + if next_token: + request[token_param] = next_token + + response = operation(**request) + if not isinstance(response, dict): + raise TypeError( + f"{operation_name} returned unexpected response type: " + f"{type(response).__name__}" + ) + items.extend(response.get(result_key, [])) + + next_token = None + for token_key in token_response_keys: + candidate_token = response.get(token_key) + if isinstance(candidate_token, str) and candidate_token: + next_token = candidate_token + break + if not next_token: + break + + return items + + def detect_bedrock_regional_footprint(region: str = "") -> Optional[bool]: """ Detect whether a region has Bedrock-managed resources that justify regional findings. @@ -184,7 +333,7 @@ def detect_bedrock_regional_footprint(region: str = "") -> Optional[bool]: ), ( "Bedrock Agents", - lambda: bedrock_agent_client.list_agents().get("agents", []), + lambda: bedrock_agent_client.list_agents().get("agentSummaries", []), ), ( "Bedrock Knowledge Bases", @@ -226,6 +375,25 @@ def detect_bedrock_regional_footprint(region: str = "") -> Optional[bool]: return None if indeterminate else False +def bedrock_footprint_na_detail( + footprint_found: Optional[bool], suffix: str = "" +) -> str: + """Build N/A finding text that distinguishes a probed-empty region from an + indeterminate one. + + ``detect_bedrock_regional_footprint`` returns ``False`` when the APIs were + reachable and reported no resources, but ``None`` when every probe was + inconclusive (e.g. access denied). Reporting "No regional Bedrock resources + found" in the ``None`` case asserts an absence that was never established, so + use distinct phrasing for it. + """ + if footprint_found is None: + base = "Bedrock footprint could not be determined in this region (access may be restricted)" + else: + base = "No regional Bedrock resources found" + return f"{base} {suffix}".rstrip() if suffix else base + + def _extract_s3_bucket_name(s3_config: Optional[Dict[str, Any]]) -> Optional[str]: """Support both the documented field name and the legacy test fixture key.""" if not s3_config: @@ -963,13 +1131,18 @@ def check_bedrock_access_and_vpc_endpoints( if bedrock_access_found: bedrock_footprint_found = detect_bedrock_regional_footprint(region=region) - if bedrock_footprint_found is False: - findings["details"] = "No regional Bedrock resources found" + if bedrock_footprint_found is not True: + findings["details"] = bedrock_footprint_na_detail( + bedrock_footprint_found + ) findings["csv_data"].append( create_finding( check_id="BR-02", finding_name="Amazon Bedrock private connectivity check", - finding_details="No regional Bedrock resources found to assess private connectivity", + finding_details=bedrock_footprint_na_detail( + bedrock_footprint_found, + "to assess private connectivity", + ), resolution="No action required", reference="https://docs.aws.amazon.com/bedrock/latest/userguide/vpc-interface-endpoints.html", severity="Informational", @@ -1072,12 +1245,12 @@ def check_bedrock_guardrails(region: str = "") -> Dict[str, Any]: try: # List all guardrails - response = bedrock_client.list_guardrails() + guardrails = _list_all_items( + bedrock_client, "list_guardrails", "guardrails" + ) - if response.get("guardrails", []): - guardrail_names = [ - guardrail["name"] for guardrail in response["guardrails"] - ] + if guardrails: + guardrail_names = [guardrail["name"] for guardrail in guardrails] findings["details"] = ( f"Found {len(guardrail_names)} Bedrock guardrails configured" ) @@ -1098,13 +1271,18 @@ def check_bedrock_guardrails(region: str = "") -> Dict[str, Any]: region=region ) - if bedrock_footprint_found is False: - findings["details"] = "No regional Bedrock resources found" + if bedrock_footprint_found is not True: + findings["details"] = bedrock_footprint_na_detail( + bedrock_footprint_found + ) findings["csv_data"].append( create_finding( check_id="BR-05", finding_name="Bedrock Guardrails Check", - finding_details="No regional Bedrock resources found to protect with guardrails", + finding_details=bedrock_footprint_na_detail( + bedrock_footprint_found, + "to protect with guardrails", + ), resolution="No action required", reference="https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html", severity="Informational", @@ -1187,13 +1365,16 @@ def check_bedrock_logging_configuration(region: str = "") -> Dict[str, Any]: } bedrock_footprint_found = detect_bedrock_regional_footprint(region=region) - if bedrock_footprint_found is False: - findings["details"] = "No regional Bedrock resources found" + if bedrock_footprint_found is not True: + findings["details"] = bedrock_footprint_na_detail(bedrock_footprint_found) findings["csv_data"].append( create_finding( check_id="BR-04", finding_name="Bedrock Model Invocation Logging Check", - finding_details="No regional Bedrock resources found to monitor with invocation logging", + finding_details=bedrock_footprint_na_detail( + bedrock_footprint_found, + "to monitor with invocation logging", + ), resolution="No action required", reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html", severity="Informational", @@ -1321,13 +1502,16 @@ def check_bedrock_cloudtrail_logging(region: str = "") -> Dict[str, Any]: } bedrock_footprint_found = detect_bedrock_regional_footprint(region=region) - if bedrock_footprint_found is False: - findings["details"] = "No regional Bedrock resources found" + if bedrock_footprint_found is not True: + findings["details"] = bedrock_footprint_na_detail(bedrock_footprint_found) findings["csv_data"].append( create_finding( check_id="BR-06", finding_name="Bedrock CloudTrail Logging Check", - finding_details="No regional Bedrock resources found to audit with Bedrock-specific CloudTrail coverage", + finding_details=bedrock_footprint_na_detail( + bedrock_footprint_found, + "to audit with Bedrock-specific CloudTrail coverage", + ), resolution="No action required", reference="https://docs.aws.amazon.com/bedrock/latest/userguide/logging-using-cloudtrail.html", severity="Informational", @@ -1791,8 +1975,8 @@ def check_bedrock_knowledge_base_encryption(region: str = "") -> Dict[str, Any]: finding_details=f"Error checking Knowledge Base encryption: {str(e)}", resolution="Verify your AWS credentials and permissions to access Bedrock Knowledge Bases", reference="https://docs.aws.amazon.com/bedrock/latest/userguide/encryption-kb.html", - severity="High", - status="Failed", + severity="Informational", + status="N/A", region=region, ) ) @@ -1846,8 +2030,9 @@ def check_bedrock_guardrail_iam_enforcement( # First check if any guardrails exist try: - guardrails_response = bedrock_client.list_guardrails() - guardrails = guardrails_response.get("guardrails", []) + guardrails = _list_all_items( + bedrock_client, "list_guardrails", "guardrails" + ) if not guardrails: findings["csv_data"].append( @@ -2287,8 +2472,8 @@ def check_bedrock_invocation_log_encryption(region: str = "") -> Dict[str, Any]: finding_details=f"S3 bucket '{bucket_name}' for invocation logs has NO encryption configured. Logs containing prompts and responses are stored unencrypted.", resolution="Enable SSE-KMS encryption with a customer-managed key on the S3 bucket immediately", reference="https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html", - severity="High", - status="Failed", + severity="Informational", + status="N/A", region=region, ) ) @@ -2913,7 +3098,7 @@ def check_bedrock_cross_account_guardrails(region: str = "") -> Dict[str, Any]: resolution="Grant organizations:DescribeOrganization and organizations:ListPolicies permissions to the assessment role", reference="https://docs.aws.amazon.com/organizations/latest/userguide/orgs_permissions_overview.html", severity="Medium", - status="Failed", + status="N/A", region=region, ) ) @@ -2964,8 +3149,9 @@ def check_bedrock_guardrail_tier(region: str = "") -> Dict[str, Any]: try: # List all guardrails - guardrails_response = bedrock_client.list_guardrails(maxResults=100) - guardrails = guardrails_response.get("guardrails", []) + guardrails = _list_all_items( + bedrock_client, "list_guardrails", "guardrails" + ) if not guardrails: findings["details"] = "No Bedrock guardrails found" @@ -2985,6 +3171,7 @@ def check_bedrock_guardrail_tier(region: str = "") -> Dict[str, Any]: suboptimal_guardrails = [] standard_tier_guardrails = [] + unassessed_guardrails = [] for guardrail_summary in guardrails: guardrail_id = guardrail_summary.get("id") @@ -2994,9 +3181,25 @@ def check_bedrock_guardrail_tier(region: str = "") -> Dict[str, Any]: continue # Get detailed guardrail configuration - guardrail_detail = bedrock_client.get_guardrail( - guardrailIdentifier=guardrail_id - ) + try: + guardrail_detail = bedrock_client.get_guardrail( + guardrailIdentifier=guardrail_id + ) + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get( + "Code", "Unknown" + ) + logger.warning( + f"Could not get details for guardrail {guardrail_name}: {error_code}" + ) + unassessed_guardrails.append( + { + "name": guardrail_name, + "id": guardrail_id, + "error": error_code, + } + ) + continue guardrail_config = guardrail_detail.get("guardrail", guardrail_detail) # The content-filter tier is reported under @@ -3048,6 +3251,22 @@ def check_bedrock_guardrail_tier(region: str = "") -> Dict[str, Any]: ) ) + if unassessed_guardrails: + findings["status"] = "WARN" + for gr in unassessed_guardrails: + findings["csv_data"].append( + create_finding( + check_id="BR-16", + finding_name="Guardrail Tier Validation Check", + finding_details=f"Guardrail '{gr['name']}' (ID: {gr['id']}) could not be assessed because GetGuardrail returned {gr['error']}.", + resolution="Retry the assessment. If the error persists, grant bedrock:GetGuardrail and verify the guardrail still exists.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") if error_code in ACCESS_DENIED_ERROR_CODES: @@ -3269,21 +3488,27 @@ def check_bedrock_model_evaluations(region: str = "") -> Dict[str, Any]: try: # List model evaluation jobs - eval_jobs_response = bedrock_client.list_evaluation_jobs(maxResults=100) - eval_jobs = eval_jobs_response.get("jobSummaries", []) + eval_jobs = _list_all_items( + bedrock_client, "list_evaluation_jobs", "jobSummaries" + ) if not eval_jobs: bedrock_footprint_found = detect_bedrock_regional_footprint( region=region ) - if bedrock_footprint_found is False: - findings["details"] = "No regional Bedrock resources found" + if bedrock_footprint_found is not True: + findings["details"] = bedrock_footprint_na_detail( + bedrock_footprint_found + ) findings["csv_data"].append( create_finding( check_id="BR-18", finding_name="Model Evaluation Implementation Check", - finding_details="No regional Bedrock resources found to assess with model evaluation jobs", + finding_details=bedrock_footprint_na_detail( + bedrock_footprint_found, + "to assess with model evaluation jobs", + ), resolution="No action required", reference="https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation.html", severity="Informational", @@ -3467,8 +3692,7 @@ def check_bedrock_prompt_flow_validation(region: str = "") -> Dict[str, Any]: try: # List all flows - flows_response = bedrock_agent_client.list_flows(maxResults=100) - flows = flows_response.get("flowSummaries", []) + flows = _list_all_items(bedrock_agent_client, "list_flows", "flowSummaries") if not flows: findings["details"] = "No Bedrock prompt flows found" @@ -3677,8 +3901,11 @@ def check_bedrock_knowledge_base_kms_encryption(region: str = "") -> Dict[str, A try: # List all knowledge bases - kb_response = bedrock_agent_client.list_knowledge_bases(maxResults=100) - knowledge_bases = kb_response.get("knowledgeBaseSummaries", []) + knowledge_bases = _list_all_items( + bedrock_agent_client, + "list_knowledge_bases", + "knowledgeBaseSummaries", + ) if not knowledge_bases: findings["details"] = "No Bedrock knowledge bases found" @@ -3933,8 +4160,9 @@ def check_bedrock_agent_action_group_iam( try: # List all agents - agents_response = bedrock_agent_client.list_agents(maxResults=100) - agents = agents_response.get("agentSummaries", []) + agents = _list_all_items( + bedrock_agent_client, "list_agents", "agentSummaries" + ) if not agents: findings["details"] = "No Bedrock agents found" @@ -3964,13 +4192,12 @@ def check_bedrock_agent_action_group_iam( try: # Get agent action groups - action_groups_response = ( - bedrock_agent_client.list_agent_action_groups( - agentId=agent_id, agentVersion="DRAFT", maxResults=100 - ) - ) - action_groups = action_groups_response.get( - "actionGroupSummaries", [] + action_groups = _list_all_items( + bedrock_agent_client, + "list_agent_action_groups", + "actionGroupSummaries", + agentId=agent_id, + agentVersion="DRAFT", ) for action_group in action_groups: @@ -4214,10 +4441,15 @@ def check_bedrock_service_quotas_throttling(region: str = "") -> Dict[str, Any]: try: # List Bedrock service quotas - quotas_response = service_quotas_client.list_service_quotas( - ServiceCode="bedrock", MaxResults=100 + quotas = _list_all_items( + service_quotas_client, + "list_service_quotas", + "Quotas", + max_results_param="MaxResults", + token_param="NextToken", + token_response_keys=("NextToken",), + ServiceCode="bedrock", ) - quotas = quotas_response.get("Quotas", []) if not quotas: findings["details"] = "Could not retrieve Bedrock service quotas" @@ -4235,9 +4467,13 @@ def check_bedrock_service_quotas_throttling(region: str = "") -> Dict[str, Any]: ) return findings - # Check for custom quotas (non-default values indicate intentional configuration) + # Check for custom quotas by comparing account-applied values with + # AWS default quota values. list_service_quotas and get_service_quota + # both return the applied account quota, so get_aws_default_service_quota + # is required to detect actual customization. custom_quotas = [] default_quotas = [] + indeterminate_quotas = [] for quota in quotas: quota_name = quota.get("QuotaName", "unknown") @@ -4256,33 +4492,36 @@ def check_bedrock_service_quotas_throttling(region: str = "") -> Dict[str, Any]: ) if is_throttling_quota: - # Check if quota has been customized - default_value = quota.get("Value", 0) - adjustable = quota.get("Adjustable", False) - - # Try to get applied quota (custom value if set) + # Try to get applied and AWS default quotas. try: applied_quota = service_quotas_client.get_service_quota( ServiceCode="bedrock", QuotaCode=quota_code ) applied_value = applied_quota.get("Quota", {}).get( - "Value", default_value + "Value", quota.get("Value", 0) + ) + default_quota = ( + service_quotas_client.get_aws_default_service_quota( + ServiceCode="bedrock", QuotaCode=quota_code + ) + ) + default_value = default_quota.get("Quota", {}).get( + "Value", applied_value ) - if applied_value != default_value or not adjustable: + if applied_value != default_value: custom_quotas.append( { "name": quota_name, "value": applied_value, - "adjustable": adjustable, + "default": default_value, } ) else: default_quotas.append(quota_name) except ClientError: - # If we can't get applied quota, assume default - if adjustable: - default_quotas.append(quota_name) + # If either quota cannot be read, do not infer pass/fail. + indeterminate_quotas.append(quota_name) if not custom_quotas and default_quotas: findings["status"] = "WARN" @@ -4312,6 +4551,35 @@ def check_bedrock_service_quotas_throttling(region: str = "") -> Dict[str, Any]: region=region, ) ) + elif indeterminate_quotas: + findings["csv_data"].append( + create_finding( + check_id="BR-22", + finding_name="Model Invocation Throttling Limits Check", + finding_details=f"Could not determine whether {len(indeterminate_quotas)} Bedrock throttling quotas differ from AWS defaults.", + resolution="Grant servicequotas:GetServiceQuota and servicequotas:GetAWSDefaultServiceQuota permissions, then re-run the assessment.", + reference="https://docs.aws.amazon.com/servicequotas/latest/userguide/identity-access-management.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + else: + # Quotas were returned but none matched the throttling keyword + # set, so there is nothing to compare against AWS defaults. Emit + # an explicit N/A so the check never silently disappears. + findings["csv_data"].append( + create_finding( + check_id="BR-22", + finding_name="Model Invocation Throttling Limits Check", + finding_details="No model invocation throttling quotas were found in the Bedrock Service Quotas for this region.", + resolution="Review Bedrock service quotas in the AWS Service Quotas console and confirm throttling limits (tokens per minute, requests per minute) are published for this region.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html", + severity="Informational", + status="N/A", + region=region, + ) + ) except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") @@ -4342,7 +4610,7 @@ def check_bedrock_service_quotas_throttling(region: str = "") -> Dict[str, Any]: finding_details=describe_api_error( e, "Service quotas check", region ), - resolution="Grant servicequotas:ListServiceQuotas and servicequotas:GetServiceQuota permissions", + resolution="Grant servicequotas:ListServiceQuotas, servicequotas:GetServiceQuota, and servicequotas:GetAWSDefaultServiceQuota permissions", reference="https://docs.aws.amazon.com/servicequotas/latest/userguide/identity-access-management.html", severity="Medium", status="Failed", @@ -4396,8 +4664,9 @@ def check_bedrock_guardrail_content_filters(region: str = "") -> Dict[str, Any]: try: # List all guardrails - guardrails_response = bedrock_client.list_guardrails(maxResults=100) - guardrails = guardrails_response.get("guardrails", []) + guardrails = _list_all_items( + bedrock_client, "list_guardrails", "guardrails" + ) if not guardrails: findings["details"] = "No Bedrock guardrails found" @@ -4417,6 +4686,7 @@ def check_bedrock_guardrail_content_filters(region: str = "") -> Dict[str, Any]: incomplete_guardrails = [] complete_guardrails = [] + unassessed_guardrails = [] for guardrail_summary in guardrails: guardrail_id = guardrail_summary.get("id") @@ -4426,9 +4696,25 @@ def check_bedrock_guardrail_content_filters(region: str = "") -> Dict[str, Any]: continue # Get detailed guardrail configuration - guardrail_detail = bedrock_client.get_guardrail( - guardrailIdentifier=guardrail_id - ) + try: + guardrail_detail = bedrock_client.get_guardrail( + guardrailIdentifier=guardrail_id + ) + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get( + "Code", "Unknown" + ) + logger.warning( + f"Could not get details for guardrail {guardrail_name}: {error_code}" + ) + unassessed_guardrails.append( + { + "name": guardrail_name, + "id": guardrail_id, + "error": error_code, + } + ) + continue guardrail_config = guardrail_detail.get("guardrail", guardrail_detail) # Check content filter configuration. GetGuardrail reports the @@ -4501,6 +4787,22 @@ def check_bedrock_guardrail_content_filters(region: str = "") -> Dict[str, Any]: ) ) + if unassessed_guardrails: + findings["status"] = "WARN" + for gr in unassessed_guardrails: + findings["csv_data"].append( + create_finding( + check_id="BR-23", + finding_name="Guardrail Content Filter Coverage Check", + finding_details=f"Guardrail '{gr['name']}' (ID: {gr['id']}) could not be assessed because GetGuardrail returned {gr['error']}.", + resolution="Retry the assessment. If the error persists, grant bedrock:GetGuardrail and verify the guardrail still exists.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") if error_code in ACCESS_DENIED_ERROR_CODES: @@ -4565,8 +4867,9 @@ def check_bedrock_automated_reasoning_policy(region: str = "") -> Dict[str, Any] try: # List all guardrails - guardrails_response = bedrock_client.list_guardrails(maxResults=100) - guardrails = guardrails_response.get("guardrails", []) + guardrails = _list_all_items( + bedrock_client, "list_guardrails", "guardrails" + ) if not guardrails: findings["details"] = "No Bedrock guardrails found" @@ -4745,8 +5048,11 @@ def check_bedrock_rag_evaluation_jobs(region: str = "") -> Dict[str, Any]: try: # List all knowledge bases - kb_response = bedrock_agent_client.list_knowledge_bases(maxResults=100) - knowledge_bases = kb_response.get("knowledgeBaseSummaries", []) + knowledge_bases = _list_all_items( + bedrock_agent_client, + "list_knowledge_bases", + "knowledgeBaseSummaries", + ) if not knowledge_bases: findings["details"] = "No knowledge bases found" @@ -4766,22 +5072,36 @@ def check_bedrock_rag_evaluation_jobs(region: str = "") -> Dict[str, Any]: # List evaluation jobs (filter for RAG-related evaluations) try: - eval_jobs_response = bedrock_client.list_evaluation_jobs(maxResults=100) - eval_jobs = eval_jobs_response.get("jobSummaries", []) + eval_jobs = _list_all_items( + bedrock_client, "list_evaluation_jobs", "jobSummaries" + ) # Map knowledge bases to evaluation jobs kbs_with_evals = set() recent_evaluations = [] thirty_days_ago = datetime.now(timezone.utc) - timedelta(days=30) + def _job_references_kb(name: str, kb_id: str) -> bool: + # Match the KB id as a whole token so a short id that happens + # to be a substring of an unrelated job name is not counted + # as an evaluation for that knowledge base. + return bool( + re.search( + r"(? Dict[str, Any]: recent_evaluations.append(job_name) # Try to identify which KB this evaluation is for for kb in knowledge_bases: - if kb["knowledgeBaseId"] in job_name: + if _job_references_kb(job_name, kb["knowledgeBaseId"]): kbs_with_evals.add(kb["knowledgeBaseId"]) kbs_without_evals = [ @@ -4933,8 +5253,9 @@ def check_bedrock_guardrail_pii_filters(region: str = "") -> Dict[str, Any]: ) try: - guardrails_response = bedrock_client.list_guardrails(maxResults=100) - guardrails = guardrails_response.get("guardrails", []) + guardrails = _list_all_items( + bedrock_client, "list_guardrails", "guardrails" + ) if not guardrails: findings["details"] = "No Bedrock guardrails found" @@ -4954,6 +5275,7 @@ def check_bedrock_guardrail_pii_filters(region: str = "") -> Dict[str, Any]: guardrails_without_pii = [] guardrails_with_pii = [] + unassessed_guardrails = [] for guardrail_summary in guardrails: guardrail_id = guardrail_summary.get("id") @@ -4962,9 +5284,25 @@ def check_bedrock_guardrail_pii_filters(region: str = "") -> Dict[str, Any]: if not guardrail_id: continue - guardrail_detail = bedrock_client.get_guardrail( - guardrailIdentifier=guardrail_id - ) + try: + guardrail_detail = bedrock_client.get_guardrail( + guardrailIdentifier=guardrail_id + ) + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get( + "Code", "Unknown" + ) + logger.warning( + f"Could not get details for guardrail {guardrail_name}: {error_code}" + ) + unassessed_guardrails.append( + { + "name": guardrail_name, + "id": guardrail_id, + "error": error_code, + } + ) + continue guardrail_config = guardrail_detail.get("guardrail", guardrail_detail) # GetGuardrail reports PII protection under @@ -5016,6 +5354,22 @@ def check_bedrock_guardrail_pii_filters(region: str = "") -> Dict[str, Any]: ) ) + if unassessed_guardrails: + findings["status"] = "WARN" + for gr in unassessed_guardrails: + findings["csv_data"].append( + create_finding( + check_id="BR-26", + finding_name="Guardrail Sensitive Information Filter Check", + finding_details=f"Guardrail '{gr['name']}' (ID: {gr['id']}) could not be assessed because GetGuardrail returned {gr['error']}.", + resolution="Retry the assessment. If the error persists, grant bedrock:GetGuardrail and verify the guardrail still exists.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") if error_code in ACCESS_DENIED_ERROR_CODES: @@ -5080,8 +5434,9 @@ def check_bedrock_guardrail_contextual_grounding(region: str = "") -> Dict[str, ) try: - guardrails_response = bedrock_client.list_guardrails(maxResults=100) - guardrails = guardrails_response.get("guardrails", []) + guardrails = _list_all_items( + bedrock_client, "list_guardrails", "guardrails" + ) if not guardrails: findings["details"] = "No Bedrock guardrails found" @@ -5101,6 +5456,7 @@ def check_bedrock_guardrail_contextual_grounding(region: str = "") -> Dict[str, guardrails_without_grounding = [] guardrails_with_grounding = [] + unassessed_guardrails = [] for guardrail_summary in guardrails: guardrail_id = guardrail_summary.get("id") @@ -5109,9 +5465,25 @@ def check_bedrock_guardrail_contextual_grounding(region: str = "") -> Dict[str, if not guardrail_id: continue - guardrail_detail = bedrock_client.get_guardrail( - guardrailIdentifier=guardrail_id - ) + try: + guardrail_detail = bedrock_client.get_guardrail( + guardrailIdentifier=guardrail_id + ) + except ClientError as detail_error: + error_code = detail_error.response.get("Error", {}).get( + "Code", "Unknown" + ) + logger.warning( + f"Could not get details for guardrail {guardrail_name}: {error_code}" + ) + unassessed_guardrails.append( + { + "name": guardrail_name, + "id": guardrail_id, + "error": error_code, + } + ) + continue guardrail_config = guardrail_detail.get("guardrail", guardrail_detail) # GetGuardrail reports grounding/relevance checks under @@ -5164,6 +5536,22 @@ def check_bedrock_guardrail_contextual_grounding(region: str = "") -> Dict[str, ) ) + if unassessed_guardrails: + findings["status"] = "WARN" + for gr in unassessed_guardrails: + findings["csv_data"].append( + create_finding( + check_id="BR-27", + finding_name="Guardrail Contextual Grounding Check", + finding_details=f"Guardrail '{gr['name']}' (ID: {gr['id']}) could not be assessed because GetGuardrail returned {gr['error']}.", + resolution="Retry the assessment. If the error persists, grant bedrock:GetGuardrail and verify the guardrail still exists.", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_id-based-policy-examples.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") if error_code in ACCESS_DENIED_ERROR_CODES: @@ -5913,13 +6301,16 @@ def check_bedrock_cloudwatch_alarms(region: str = "") -> Dict[str, Any]: # Only assess this when the region actually has Bedrock resources, to # avoid recommending alarms in regions where Bedrock is unused. bedrock_footprint_found = detect_bedrock_regional_footprint(region=region) - if bedrock_footprint_found is False: - findings["details"] = "No regional Bedrock resources found" + if bedrock_footprint_found is not True: + findings["details"] = bedrock_footprint_na_detail(bedrock_footprint_found) findings["csv_data"].append( create_finding( check_id="BR-32", finding_name="Bedrock CloudWatch Alarm Check", - finding_details="No regional Bedrock resources found to monitor with CloudWatch alarms", + finding_details=bedrock_footprint_na_detail( + bedrock_footprint_found, + "to monitor with CloudWatch alarms", + ), resolution="No action required", reference="https://docs.aws.amazon.com/bedrock/latest/userguide/monitoring-runtime-metrics.html", severity="Informational", @@ -6054,6 +6445,55 @@ def generate_csv_report(findings: List[Dict[str, Any]]) -> str: return csv_buffer.getvalue() +def _flatten_csv_rows(findings: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + rows = [] + for finding in findings: + rows.extend(finding.get("csv_data") or []) + return rows + + +def build_agentic_bedrock_security_findings( + findings: List[Dict[str, Any]], +) -> Dict[str, Any]: + """Create AG-* rows from Bedrock checks that already prove agentic controls.""" + agentic_rows = [] + for row in _flatten_csv_rows(findings): + source_check_id = row.get("Check_ID", "") + mapping = AGENTIC_BEDROCK_CHECK_MAPPINGS.get(source_check_id) + if not mapping: + continue + + source_details = row.get("Finding_Details", "") + status = row.get("Status", "N/A") + severity = row.get("Severity", "Informational") + if status == "N/A": + severity = "Informational" + + agentic_rows.append( + create_finding( + check_id=mapping["check_id"], + finding_name=mapping["finding"], + finding_details=( + f"Agentic AI security domain: {mapping['lens_domain']}. " + f"{mapping['agentic_context']} " + f"Source check {source_check_id}: {source_details}" + ), + resolution=mapping["resolution"], + reference=AGENTIC_AI_LENS_URL, + severity=severity, + status=status, + region=row.get("Region", ""), + ) + ) + + return { + "check_name": "Agentic AI Security - Bedrock", + "status": "Completed", + "details": f"Generated {len(agentic_rows)} Agentic AI Security findings from Bedrock checks", + "csv_data": agentic_rows, + } + + def get_current_utc_date(): return datetime.now(timezone.utc).strftime("%Y/%m/%d") @@ -6345,6 +6785,9 @@ def lambda_handler(event, context): cloudwatch_alarm_findings = check_bedrock_cloudwatch_alarms(region=region) all_findings.append(cloudwatch_alarm_findings) + logger.info("Building Agentic AI Security findings from Bedrock results") + all_findings.append(build_agentic_bedrock_security_findings(all_findings)) + # Generate and upload report logger.info("Generating CSV report") csv_content = generate_csv_report(all_findings) diff --git a/aiml-security-assessment/functions/security/generate_consolidated_report/app.py b/aiml-security-assessment/functions/security/generate_consolidated_report/app.py index f20fc9c..dd97f13 100644 --- a/aiml-security-assessment/functions/security/generate_consolidated_report/app.py +++ b/aiml-security-assessment/functions/security/generate_consolidated_report/app.py @@ -94,6 +94,7 @@ def get_assessment_results(execution_id: str, account_id: str = None) -> Dict[st "bedrock": {}, "sagemaker": {}, "agentcore": {}, + "agentic": {}, "finserv": {}, } @@ -153,10 +154,11 @@ def get_assessment_results(execution_id: str, account_id: str = None) -> Dict[st "total_files_processed": len(assessment_results["bedrock"]) + len(assessment_results["sagemaker"]) + len(assessment_results["agentcore"]) + + len(assessment_results["agentic"]) + len(assessment_results["finserv"]), "categories_found": [ cat - for cat in ["bedrock", "sagemaker", "agentcore", "finserv"] + for cat in ["bedrock", "sagemaker", "agentcore", "agentic", "finserv"] if assessment_results[cat] ], "rows": assessment_results["bedrock"], @@ -164,6 +166,7 @@ def get_assessment_results(execution_id: str, account_id: str = None) -> Dict[st "bedrock": list(assessment_results["bedrock"].keys()), "sagemaker": list(assessment_results["sagemaker"].keys()), "agentcore": list(assessment_results["agentcore"].keys()), + "agentic": list(assessment_results["agentic"].keys()), "finserv": list(assessment_results["finserv"].keys()), }, } @@ -204,9 +207,16 @@ def generate_html_report(assessment_results: Dict[str, Any]) -> str: "bedrock": {"passed": 0, "failed": 0, "na": 0}, "sagemaker": {"passed": 0, "failed": 0, "na": 0}, "agentcore": {"passed": 0, "failed": 0, "na": 0}, + "agentic": {"passed": 0, "failed": 0, "na": 0}, "finserv": {"passed": 0, "failed": 0, "na": 0}, } - service_findings = {"bedrock": [], "sagemaker": [], "agentcore": [], "finserv": []} + service_findings = { + "bedrock": [], + "sagemaker": [], + "agentcore": [], + "agentic": [], + "finserv": [], + } regions = set() # Global/IAM findings (Region == "Global", e.g. BR-01, SM-02, AC-09) are @@ -222,9 +232,14 @@ def generate_html_report(assessment_results: Dict[str, Any]) -> str: if service in assessment_results: for report_type, findings in assessment_results[service].items(): for finding in findings: + output_service = ( + "agentic" + if finding.get("Check_ID", "").upper().startswith("AG-") + else service + ) dedup_key = ( finding.get("Account_ID", ""), - service, + output_service, finding.get("Check_ID", ""), finding.get("Region", ""), finding.get("Finding_Details", ""), @@ -233,16 +248,16 @@ def generate_html_report(assessment_results: Dict[str, Any]) -> str: continue seen_findings.add(dedup_key) - finding["_service"] = service + finding["_service"] = output_service all_findings.append(finding) - service_findings[service].append(finding) + service_findings[output_service].append(finding) status = finding.get("Status", "").lower() if status == "passed": - service_stats[service]["passed"] += 1 + service_stats[output_service]["passed"] += 1 elif status == "failed": - service_stats[service]["failed"] += 1 + service_stats[output_service]["failed"] += 1 elif status == "n/a": - service_stats[service]["na"] += 1 + service_stats[output_service]["na"] += 1 region = finding.get("Region", "") # "Global" tags IAM-only findings; it is not a scanned region # and must not inflate the region count / multi-region UI. diff --git a/aiml-security-assessment/functions/security/generate_consolidated_report/report_template.py b/aiml-security-assessment/functions/security/generate_consolidated_report/report_template.py index 27cf527..b05836a 100644 --- a/aiml-security-assessment/functions/security/generate_consolidated_report/report_template.py +++ b/aiml-security-assessment/functions/security/generate_consolidated_report/report_template.py @@ -23,10 +23,25 @@ '' ) +AGENTIC_ICON = ( + '' + '' + '' +) +AGENTIC_ICON_SMALL = ( + '' + '' + '' + '' +) GENAI_LENS_URL = ( "https://docs.aws.amazon.com/wellarchitected/latest/generative-ai-lens/" "generative-ai-lens.html" ) +AGENTIC_AI_LENS_URL = ( + "https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/" + "agentic-ai-lens.html" +) FINSERV_GUIDE_URL = ( "https://aws.amazon.com/blogs/security/" "introducing-the-updated-aws-user-guide-to-governance-risk-and-compliance-for-responsible-ai-adoption/" @@ -317,6 +332,7 @@ def get_html_template() -> str: AgentCore {agentcore_total} + {agentic_nav} {industry_nav}
@@ -364,7 +380,7 @@ def get_html_template() -> str:
{account_filter} {region_filter} -
+
@@ -382,11 +398,12 @@ def get_html_template() -> str:
{account_risk_section} {region_risk_section} -

Findings by Service

+

Findings by Assessment Area

Bedrock
{bedrock_total}
{bedrock_failed} Failed · {bedrock_passed} Passed
SageMaker
{sagemaker_total}
{sagemaker_failed} Failed · {sagemaker_passed} Passed
AgentCore
{agentcore_total}
{agentcore_failed} Failed · {agentcore_passed} Passed
+ {agentic_service_card} {finserv_service_card}
@@ -426,6 +443,7 @@ def get_html_template() -> str:
{agentcore_rows}
Account IDRegionCheck IDFindingDetailsResolutionReferenceSeverityStatus
+ {agentic_section} {finserv_section}
Assessment Methodology
@@ -612,6 +630,7 @@ def get_html_template() -> str: createServiceFilter('bedrockTable', 'bedrockSearchInput', 'bedrockAccountFilter', 'bedrockRegionFilter', 'bedrockSeverityFilter', 'bedrockStatusFilter', 'bedrockResetFilters'); createServiceFilter('sagemakerTable', 'sagemakerSearchInput', 'sagemakerAccountFilter', 'sagemakerRegionFilter', 'sagemakerSeverityFilter', 'sagemakerStatusFilter', 'sagemakerResetFilters'); createServiceFilter('agentcoreTable', 'agentcoreSearchInput', 'agentcoreAccountFilter', 'agentcoreRegionFilter', 'agentcoreSeverityFilter', 'agentcoreStatusFilter', 'agentcoreResetFilters'); + createServiceFilter('agenticTable', 'agenticSearchInput', 'agenticAccountFilter', 'agenticRegionFilter', 'agenticSeverityFilter', 'agenticStatusFilter', 'agenticResetFilters'); createServiceFilter('finservTable', 'finservSearchInput', 'finservAccountFilter', 'finservRegionFilter', 'finservSeverityFilter', 'finservStatusFilter', 'finservResetFilters'); // Apply initial filters for main table @@ -737,6 +756,8 @@ def generate_html_report( if f.get("_service") == "bedrock" else "SageMaker" if f.get("_service") == "sagemaker" + else "Agentic AI" + if f.get("_service") == "agentic" else "FinServ" if f.get("_service") == "finserv" else "AgentCore" @@ -755,6 +776,8 @@ def generate_html_report( if f.get("_service") == "bedrock" else "SageMaker" if f.get("_service") == "sagemaker" + else "Agentic AI" + if f.get("_service") == "agentic" else "FinServ" if f.get("_service") == "finserv" else "AgentCore" @@ -781,6 +804,9 @@ def generate_html_report( agentcore_rows = generate_table_rows( service_findings.get("agentcore", []), include_data_attrs=True ) + agentic_rows = generate_table_rows( + service_findings.get("agentic", []), include_data_attrs=True + ) finserv_rows = generate_table_rows( service_findings.get("finserv", []), include_data_attrs=True ) @@ -805,11 +831,13 @@ def generate_html_report( bedrock_region_filter = f'
' sagemaker_region_filter = f'
' agentcore_region_filter = f'
' + agentic_region_filter = f'
' else: region_filter = "" bedrock_region_filter = "" sagemaker_region_filter = "" agentcore_region_filter = "" + agentic_region_filter = "" # Mode-specific content num_accounts = len(account_ids) if account_ids else 1 @@ -835,6 +863,7 @@ def generate_html_report( bedrock_account_filter = f'
' sagemaker_account_filter = f'
' agentcore_account_filter = f'
' + agentic_account_filter = f'
' finserv_account_filter = f'
' # Calculate per-account risk metrics @@ -898,6 +927,7 @@ def generate_html_report( bedrock_account_filter = "" sagemaker_account_filter = "" agentcore_account_filter = "" + agentic_account_filter = "" finserv_account_filter = "" account_risk_section = "" @@ -945,6 +975,84 @@ def generate_html_report( else: region_risk_section = "" + # Agentic AI Security (AG-*) — security-focused lens mapping rendered when AG rows exist. + agentic_total = ( + service_stats.get("agentic", {}).get("passed", 0) + + service_stats.get("agentic", {}).get("failed", 0) + + service_stats.get("agentic", {}).get("na", 0) + ) + agentic_failed = service_stats.get("agentic", {}).get("failed", 0) + agentic_passed = service_stats.get("agentic", {}).get("passed", 0) + if agentic_total > 0: + agentic_regions = sorted( + { + f.get("region", f.get("Region", "")) + for f in service_findings.get("agentic", []) + if f.get("region", f.get("Region", "")) + } + ) + agentic_region_options = "".join( + [f'' for r in agentic_regions] + ) + agentic_region_filter = ( + '
' + '
' + if agentic_regions + else agentic_region_filter + ) + agentic_nav = ( + '' + + AGENTIC_ICON + + " Agentic AI Security" + + f'{agentic_total}' + ) + agentic_filter_option = '' + agentic_service_card = ( + '
' + + AGENTIC_ICON_SMALL + + f' Agentic AI Security
{agentic_total}
' + + f'
{agentic_failed} Failed · {agentic_passed} Passed
' + ) + agentic_section = ( + '
' + '
' + + AGENTIC_ICON + + "Agentic AI Security Findings
" + '
Scope: API-provable Agentic AI security controls mapped to the AWS Well-Architected Agentic AI Lens security guidance. ' + "Human-in-the-loop governance is referenced in methodology but not scored automatically unless an AWS API can prove the control.
" + '
' + '
' + + agentic_account_filter + + agentic_region_filter + + '
' + '
' + '' + "
" + '
' + + agentic_rows + + "
Account IDRegionCheck IDFindingDetailsResolutionReferenceSeverityStatus
" + "
" + ) + agentic_scope_block = ( + '
' + '
Agentic AI Security
' + '
' + + AGENTIC_ICON_SMALL + + 'Agentic AI Security Lens Mapping
' + ) + agentic_scope_source = ( + " Controls that cannot be proven using AWS APIs, including semantic" + " human-in-the-loop workflow quality, are not automatically scored." + ) + else: + agentic_nav = "" + agentic_filter_option = "" + agentic_service_card = "" + agentic_section = "" + agentic_scope_block = "" + agentic_scope_source = "" + # FinServ (FS-*) — first-class industry assessment, rendered only when findings exist # (so non-FinServ accounts and EnableFinServAssessment=false deploys stay clean). finserv_total = ( @@ -1075,17 +1183,29 @@ def generate_html_report( + service_stats.get("agentcore", {}).get("na", 0), agentcore_failed=service_stats.get("agentcore", {}).get("failed", 0), agentcore_passed=service_stats.get("agentcore", {}).get("passed", 0), + agentic_total=service_stats.get("agentic", {}).get("passed", 0) + + service_stats.get("agentic", {}).get("failed", 0) + + service_stats.get("agentic", {}).get("na", 0), + agentic_failed=service_stats.get("agentic", {}).get("failed", 0), + agentic_passed=service_stats.get("agentic", {}).get("passed", 0), alerts=alerts_html, all_rows=all_rows, bedrock_rows=bedrock_rows, sagemaker_rows=sagemaker_rows, agentcore_rows=agentcore_rows, + agentic_rows=agentic_rows, bedrock_account_filter=bedrock_account_filter, sagemaker_account_filter=sagemaker_account_filter, agentcore_account_filter=agentcore_account_filter, + agentic_account_filter=agentic_account_filter, bedrock_region_filter=bedrock_region_filter, sagemaker_region_filter=sagemaker_region_filter, agentcore_region_filter=agentcore_region_filter, + agentic_region_filter=agentic_region_filter, + agentic_nav=agentic_nav, + agentic_filter_option=agentic_filter_option, + agentic_service_card=agentic_service_card, + agentic_section=agentic_section, industry_nav=industry_nav, finserv_filter_option=finserv_filter_option, finserv_service_card=finserv_service_card, @@ -1095,25 +1215,29 @@ def generate_html_report( ) base_scope_source = ( f"Bedrock, SageMaker, and AgentCore checks are based on the " - f'AWS Well-Architected Framework Generative AI Lens.' + f'AWS Well-Architected Framework Generative AI Lens. ' + f'Agentic AI Security references the AWS Well-Architected Agentic AI Lens.' ) rendered_html = rendered_html.replace( "Based on AWS Well-Architected Framework (Generative AI Lens) and service-specific security documentation.", base_scope_source, 1, ) - if finserv_scope_industry_block: + scope_extension_blocks = agentic_scope_block + finserv_scope_industry_block + if scope_extension_blocks: rendered_html = rendered_html.replace( 'Amazon Bedrock AgentCore

Amazon Bedrock AgentCore' + "" - + finserv_scope_industry_block + + scope_extension_blocks + "

Agentic AI Security', html) + self.assertIn("AG-01", html) + self.assertIn('data-service="agentic"', html) + self.assertIn("Agentic AI Security Findings", html) + self.assertIn( + "wellarchitected/latest/agentic-ai-lens/agentic-ai-lens.html", html + ) + # The Agentic AI Lens hyperlink in the methodology must appear exactly + # once even when agentic findings are present (no duplicate link). + self.assertEqual( + html.count('target="_blank">AWS Well-Architected Agentic AI Lens'), + 1, + ) + self.assertIn("Human-in-the-loop governance", html) + + def test_agentic_security_omitted_when_absent(self): + """With no AG-* data the Agentic section is omitted cleanly.""" + html = generate_html_report(self.test_assessment_results) + + self.assertNotIn('id="agentic"', html) + self.assertNotIn('id="agenticTable"', html) + self.assertNotIn('', html) + self.assertIn( + "wellarchitected/latest/agentic-ai-lens/agentic-ai-lens.html", html + ) + def tearDown(self): """Clean up test files after running tests""" pass diff --git a/aiml-security-assessment/template-multi-account.yaml b/aiml-security-assessment/template-multi-account.yaml index 6e35b3e..b29ee2c 100644 --- a/aiml-security-assessment/template-multi-account.yaml +++ b/aiml-security-assessment/template-multi-account.yaml @@ -232,6 +232,7 @@ Resources: Action: - servicequotas:ListServiceQuotas # BR-22 - servicequotas:GetServiceQuota # BR-22 + - servicequotas:GetAWSDefaultServiceQuota # BR-22 Resource: '*' - Sid: CloudWatchPermissions Effect: Allow @@ -242,6 +243,7 @@ Resources: Effect: Allow Action: - organizations:DescribeOrganization # BR-15 + - organizations:ListRoots # BR-15 - organizations:ListPolicies # BR-15 Resource: '*' - Sid: S3BucketEncryptionPermissions diff --git a/aiml-security-assessment/template.yaml b/aiml-security-assessment/template.yaml index 9eb767c..935cb8c 100644 --- a/aiml-security-assessment/template.yaml +++ b/aiml-security-assessment/template.yaml @@ -231,6 +231,7 @@ Resources: Action: - servicequotas:ListServiceQuotas # BR-22 - servicequotas:GetServiceQuota # BR-22 + - servicequotas:GetAWSDefaultServiceQuota # BR-22 Resource: '*' - Sid: CloudWatchPermissions Effect: Allow @@ -241,6 +242,7 @@ Resources: Effect: Allow Action: - organizations:DescribeOrganization # BR-15 + - organizations:ListRoots # BR-15 - organizations:ListPolicies # BR-15 Resource: '*' - Sid: S3BucketEncryptionPermissions diff --git a/consolidate_html_reports.py b/consolidate_html_reports.py index e0037bf..105bb21 100644 --- a/consolidate_html_reports.py +++ b/consolidate_html_reports.py @@ -84,9 +84,16 @@ def consolidate_html_reports(): "bedrock": {"passed": 0, "failed": 0, "na": 0}, "sagemaker": {"passed": 0, "failed": 0, "na": 0}, "agentcore": {"passed": 0, "failed": 0, "na": 0}, + "agentic": {"passed": 0, "failed": 0, "na": 0}, "finserv": {"passed": 0, "failed": 0, "na": 0}, } - service_findings = {"bedrock": [], "sagemaker": [], "agentcore": [], "finserv": []} + service_findings = { + "bedrock": [], + "sagemaker": [], + "agentcore": [], + "agentic": [], + "finserv": [], + } for account_dir in glob.glob(os.path.join(_account_files_dir(), "*/")): account_id = os.path.basename(account_dir.rstrip("/")) @@ -138,6 +145,8 @@ def consolidate_html_reports(): service = "sagemaker" elif check_id.startswith("AC-"): service = "agentcore" + elif check_id.startswith("AG-"): + service = "agentic" elif check_id.startswith("FS-"): service = "finserv" else: diff --git a/deployment/1-aiml-security-member-roles.yaml b/deployment/1-aiml-security-member-roles.yaml index ac14561..aabe7d2 100644 --- a/deployment/1-aiml-security-member-roles.yaml +++ b/deployment/1-aiml-security-member-roles.yaml @@ -104,7 +104,10 @@ Resources: - organizations:ListPolicies - servicequotas:ListServiceQuotas - servicequotas:GetServiceQuota + - servicequotas:GetAWSDefaultServiceQuota - cloudwatch:DescribeAlarms + # AgentCore assessment-duration custom metric + - cloudwatch:PutMetricData Resource: "*" # Bedrock AgentCore Permissions - Effect: Allow diff --git a/deployment/2-aiml-security-codebuild.yaml b/deployment/2-aiml-security-codebuild.yaml index 6af8f4d..36abbb1 100644 --- a/deployment/2-aiml-security-codebuild.yaml +++ b/deployment/2-aiml-security-codebuild.yaml @@ -230,7 +230,10 @@ Resources: - organizations:ListPolicies - servicequotas:ListServiceQuotas - servicequotas:GetServiceQuota + - servicequotas:GetAWSDefaultServiceQuota - cloudwatch:DescribeAlarms + # AgentCore assessment-duration custom metric + - cloudwatch:PutMetricData # SageMaker Permissions - sagemaker:ListNotebookInstances - sagemaker:DescribeNotebookInstance @@ -324,6 +327,52 @@ Resources: - s3:GetBucketVersioning - s3:GetBucketTagging - s3:HeadBucket + # FinServ GenAI Risk Assessment Permissions (FS-01 to FS-69) + # Read-only list/describe/get actions required by the FinServ + # security checks. Kept in sync with the canonical member role + # in deployment/1-aiml-security-member-roles.yaml. + # WAF / Shield (FS-01) + - wafv2:ListWebACLs + - wafv2:GetWebACL + - shield:DescribeSubscription + # API Gateway (FS-02, FS-68, FS-69) + - apigateway:GET + # Service Quotas (FS-03) + - servicequotas:ListAWSDefaultServiceQuotas + # Cost Explorer / Budgets (FS-04, FS-06) + - ce:GetAnomalyMonitors + - budgets:ViewBudget + # Logs account policies (FS-05, FS-11, FS-43) + - logs:DescribeAccountPolicies + # Bedrock — models, data sources, reasoning (FS-07, FS-15, FS-27..50) + - bedrock:ListFoundationModels + - bedrock:ListDataSources + - bedrock:GetDataSource + - bedrock:ListAutomatedReasoningPolicies + - bedrock:ListTagsForResource + # Lambda concurrency / agent transaction checks (FS-09, FS-67, FS-69) + - lambda:GetFunctionConcurrency + # Step Functions — human-in-the-loop / rate checks (FS-10, FS-11) + - states:ListStateMachines + - states:DescribeStateMachine + # Organizations — SCP model access restrictions (FS-12) + - organizations:DescribePolicy + # SageMaker — model cards / tags (FS-13, FS-61, FS-63) + - sagemaker:ListModelCards + - sagemaker:ListTags + # S3 — bucket metadata (FS-20, FS-21, FS-46, FS-65) + - s3:ListAllMyBuckets + - s3:GetBucketNotification + # OpenSearch Serverless — KB vector store (FS-25, FS-26) + - aoss:ListSecurityPolicies + # Macie — PII detection on training data (FS-44) + - macie2:GetMacieSession + # EventBridge — KB integrity monitoring (FS-33, FS-65) + - events:ListRules + # EventBridge Scheduler — KB sync schedules (FS-61) + - scheduler:ListSchedules + # Config — model onboarding governance (FS-14) + - config:DescribeConfigRules Resource: "*" # S3 bucket for assessment results diff --git a/deployment/aiml-security-single-account.yaml b/deployment/aiml-security-single-account.yaml index 1719183..0b3d0c4 100644 --- a/deployment/aiml-security-single-account.yaml +++ b/deployment/aiml-security-single-account.yaml @@ -194,7 +194,10 @@ Resources: - organizations:ListPolicies - servicequotas:ListServiceQuotas - servicequotas:GetServiceQuota + - servicequotas:GetAWSDefaultServiceQuota - cloudwatch:DescribeAlarms + # AgentCore assessment-duration custom metric + - cloudwatch:PutMetricData # SageMaker Permissions - sagemaker:ListNotebookInstances - sagemaker:DescribeNotebookInstance diff --git a/docs/DEVELOPER_GUIDE.md b/docs/DEVELOPER_GUIDE.md index fa343e5..3195cdd 100644 --- a/docs/DEVELOPER_GUIDE.md +++ b/docs/DEVELOPER_GUIDE.md @@ -44,7 +44,7 @@ ## Architecture Overview -The AI/ML Security Assessment Framework is a serverless, multi-account security assessment solution for AWS AI/ML workloads. It performs 70 core security checks across Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore, with an optional 64-check Financial Services GenAI risk assessment, generating interactive HTML reports with findings and remediation guidance. +The AI/ML Security Assessment Framework is a serverless, multi-account security assessment solution for AWS AI/ML workloads. It performs 70 core security checks across Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore, plus 27 always-on Agentic AI Security checks, with an optional 64-check Financial Services GenAI risk assessment, generating interactive HTML reports with findings and remediation guidance. ### Security Design Principles @@ -211,7 +211,7 @@ sample-aiml-security-assessment/ ## Assessment Structure -The framework includes **70 core security checks** across three AI/ML services, plus **64 optional Financial Services GenAI risk checks** when `EnableFinServAssessment` is enabled. For the complete list of checks with descriptions, see the [Security Checks Reference](SECURITY_CHECKS.md). +The framework includes **70 core security checks** across three AI/ML services, plus **27 always-on Agentic AI Security checks** and **64 optional Financial Services GenAI risk checks** when `EnableFinServAssessment` is enabled. For the complete list of checks with descriptions, see the [Security Checks Reference](SECURITY_CHECKS.md). ### AWS Lambda Functions @@ -481,7 +481,7 @@ sam local invoke ComprehendSecurityAssessmentFunction --event testfile.json - **Handle Exceptions**: Implement proper error handling and logging - **Follow Least Privilege**: Only request necessary permissions - **Standardize Findings**: Use the `create_finding()` function for consistent output -- **Check ID Convention**: Use service prefixes for check IDs (BR-XX for Amazon Bedrock, SM-XX for Amazon SageMaker AI, AC-XX for Amazon Bedrock AgentCore, FS-XX for Financial Services GenAI risk checks) +- **Check ID Convention**: Use service prefixes for check IDs (BR-XX for Amazon Bedrock, SM-XX for Amazon SageMaker AI, AC-XX for Amazon Bedrock AgentCore, AG-XX for Agentic AI Security, FS-XX for Financial Services GenAI risk checks) - **Status Semantics**: Use correct status values: - `Passed`: Resources were checked and met the assessed best practice - `Failed`: Resources were checked and found non-compliant @@ -505,16 +505,18 @@ try: # Assessment logic result = aws_client.describe_service() except ClientError as e: - if e.response["Error"]["Code"] == "AccessDenied": - # Handle permission issues + # Access-denied and region-unsupported paths resolve to N/A, not Failed: + # the check could not run, which is not a confirmed misconfiguration. + if e.response["Error"]["Code"] in ACCESS_DENIED_ERROR_CODES: logger.warning(f"Access denied for service check: {str(e)}") return create_finding( finding_name="Permission Check", - finding_details="Insufficient permissions to assess service", + finding_details=describe_api_error(e, "Service check", region), resolution="Grant required permissions to assessment role", reference="https://docs.aws.amazon.com/service/permissions", - severity="High", - status="Failed", + severity="Medium", + status="N/A", + region=region, ) else: # Handle other AWS errors @@ -560,7 +562,7 @@ For detailed troubleshooting guidance, common issues, and debugging tips, see th ## Development Roadmap ### Current Status -- **AI/ML Assessment**: 70 core checks across three services, plus 64 optional Financial Services GenAI risk checks (see [Security Checks Reference](SECURITY_CHECKS.md)) +- **AI/ML Assessment**: 70 core checks across three services, 27 always-on Agentic AI Security checks, plus 64 optional Financial Services GenAI risk checks (see [Security Checks Reference](SECURITY_CHECKS.md)) ### Potential Additions - **Amazon Comprehend**: Data privacy, access controls, entity recognition security @@ -716,7 +718,7 @@ After generating new screenshots, update the README to reference them: **Preview:** ![Executive Dashboard](sample-reports/dashboard-overview-light.png) -*Executive summary with severity counts and service breakdown* +*Executive summary with severity counts and assessment-area breakdown* ![Findings Table](sample-reports/findings-table.png) *Interactive findings table with filtering capabilities* diff --git a/docs/SECURITY_CHECKS.md b/docs/SECURITY_CHECKS.md index 236a22a..2b85c52 100644 --- a/docs/SECURITY_CHECKS.md +++ b/docs/SECURITY_CHECKS.md @@ -1,6 +1,6 @@ # Security Checks Reference -This document provides a comprehensive reference for all 134 security checks performed by the AI/ML Security Assessment framework (70 core checks across Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore, plus 64 Financial Services GenAI Risk checks). +This document provides a comprehensive reference for all 161 security checks performed by the AI/ML Security Assessment framework (70 core checks across Amazon Bedrock, Amazon SageMaker AI, and Amazon Bedrock AgentCore, 27 Agentic AI Security checks, plus 64 Financial Services GenAI Risk checks). ## Table of Contents @@ -11,6 +11,7 @@ This document provides a comprehensive reference for all 134 security checks per - [Amazon SageMaker AI Security Checks (25)](#amazon-sagemaker-ai-security-checks-25) - [Amazon Bedrock Security Checks (32)](#amazon-bedrock-security-checks-32) - [Amazon Bedrock AgentCore Security Checks (13)](#amazon-bedrock-agentcore-security-checks-13) +- [Agentic AI Security Checks (27)](#agentic-ai-security-checks-27) - [Financial Services GenAI Risk Checks (64)](#financial-services-genai-risk-checks-64-additional-5-upstream-extensions) --- @@ -24,6 +25,7 @@ The framework evaluates your AI/ML workloads against AWS security best practices | Amazon SageMaker AI | 25 | Security Hub controls, encryption, network isolation, IAM, MLOps | | Amazon Bedrock | 32 | Guardrails, content filters, sensitive-information/PII filters, contextual grounding, automated reasoning, encryption (custom, imported, knowledge base, batch inference output), VPC endpoints, IAM permissions, agent guardrail association and least privilege, logging, CloudWatch alarms, cross-account policies, model evaluation, prompt flow validation, RAG evaluation, service quotas | | Amazon Bedrock AgentCore | 13 | VPC configuration, encryption, observability, resource policies | +| Agentic AI Security | 27 | Bounded autonomy, agent identity, tool authorization, guardrail enforcement, prompt/input protection, memory privacy, auditability, abuse protection | | Financial Services GenAI Risk | 64 | Unbounded consumption, excessive agency, supply chain, training data poisoning, vector weaknesses, non-compliant output, misinformation, harmful output, biased output, PII disclosure, hallucination, prompt injection, improper output handling, off-topic output, out-of-date training data | --- @@ -37,6 +39,7 @@ Each security check has a unique identifier with a service prefix: | **SM-XX** | Amazon SageMaker | SM-01, SM-25 | | **BR-XX** | Amazon Bedrock | BR-01, BR-32 | | **AC-XX** | Amazon Bedrock AgentCore | AC-01, AC-13 | +| **AG-XX** | Agentic AI Security | AG-01, AG-27 | | **FS-XX** | Financial Services GenAI Risk | FS-01, FS-69 | --- @@ -449,10 +452,218 @@ Each security check has a unique identifier with a service prefix: --- +## Agentic AI Security Checks (27) + +Agentic AI Security checks use the `AG-XX` namespace and are included with the +default assessment. They follow a hybrid model: + +- Reused API-backed controls from Amazon Bedrock and Amazon Bedrock AgentCore + are mapped into agentic security domains. +- New checks are added only where AWS APIs can prove the control state. +- Controls that cannot be proven by AWS APIs are not scored. Human-in-the-loop + governance is therefore documented as a methodology note, not emitted as an + automated pass/fail finding. + +These checks reference the +[AWS Well-Architected Agentic AI Lens](https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentic-ai-lens.html), +with scope limited to the Security pillar. + +### AG-01: Agent Guardrail Association + +- **Severity:** High +- **Source:** BR-28 +- **Domain:** Guardrail Enforcement +- **Description:** Maps Bedrock agent guardrail association into the Agentic AI Security view. + +### AG-02: Harmful Content Guardrail Coverage + +- **Severity:** Source check severity +- **Source:** BR-23 +- **Domain:** Guardrail Enforcement +- **Description:** Maps guardrail content filter coverage for agent-facing workloads. + +### AG-03: Sensitive Information Protection + +- **Severity:** Source check severity +- **Source:** BR-26 +- **Domain:** Memory & Data Privacy +- **Description:** Maps guardrail sensitive-information and PII protection controls. + +### AG-04: Automated Reasoning Guardrails + +- **Severity:** Source check severity +- **Source:** BR-24 +- **Domain:** Guardrail Enforcement +- **Description:** Maps automated reasoning policies used to verify responses against deterministic rules. + +### AG-05: Grounding Controls + +- **Severity:** Source check severity +- **Source:** BR-27 +- **Domain:** Prompt & Input Protection +- **Description:** Maps contextual grounding checks for RAG and tool-using agents. + +### AG-06: Tool Execution Least Privilege + +- **Severity:** Source check severity +- **Source:** BR-21 +- **Domain:** Tool Authorization +- **Description:** Maps Bedrock agent action group IAM least-privilege findings. + +### AG-07: Model Invocation Logging + +- **Severity:** Source check severity +- **Source:** BR-04 +- **Domain:** Auditability & Observability +- **Description:** Maps model invocation logging for agent prompts, responses, and guardrail traces. + +### AG-08: API Audit Trail + +- **Severity:** Source check severity +- **Source:** BR-06 +- **Domain:** Auditability & Observability +- **Description:** Maps CloudTrail coverage for Bedrock activity. + +### AG-09: Guardrail Enforcement Boundary + +- **Severity:** Source check severity +- **Source:** BR-15 +- **Domain:** Guardrail Enforcement +- **Description:** Maps organization-level guardrail enforcement controls. + +### AG-10: Adversarial Evaluation Coverage + +- **Severity:** Source check severity +- **Source:** BR-18 +- **Domain:** Prompt & Input Protection +- **Description:** Maps model/application evaluation coverage for adversarial and safety testing. + +### AG-11: Prompt Flow Validation + +- **Severity:** Source check severity +- **Source:** BR-19 +- **Domain:** Prompt & Input Protection +- **Description:** Maps Bedrock flow validation before deployment. + +### AG-12: Invocation Abuse Controls + +- **Severity:** Source check severity +- **Source:** BR-22 +- **Domain:** Abuse & Cost Protection +- **Description:** Maps Bedrock service quota and throttling controls. + +### AG-13: Session Boundary + +- **Severity:** Source check severity +- **Source:** BR-29 +- **Domain:** Bounded Autonomy +- **Description:** Maps Bedrock agent idle session TTL controls. + +### AG-14: Operational Abuse Alarms + +- **Severity:** Source check severity +- **Source:** BR-32 +- **Domain:** Abuse & Cost Protection +- **Description:** Maps CloudWatch alarms for Bedrock invocation abuse and operational anomalies. + +### AG-15: Runtime Network Boundary + +- **Severity:** Source check severity +- **Source:** AC-01 +- **Domain:** Bounded Autonomy +- **Description:** Maps AgentCore runtime VPC configuration. + +### AG-16: AgentCore Least Privilege + +- **Severity:** Source check severity +- **Source:** AC-02 +- **Domain:** Agent Identity & Access +- **Description:** Maps AgentCore full-access IAM findings. + +### AG-17: Stale AgentCore Access + +- **Severity:** Source check severity +- **Source:** AC-03 +- **Domain:** Agent Identity & Access +- **Description:** Maps stale AgentCore permissions. + +### AG-18: AgentCore Observability + +- **Severity:** Source check severity +- **Source:** AC-04 +- **Domain:** Auditability & Observability +- **Description:** Maps AgentCore logging, tracing, and observability coverage. + +### AG-19: Memory Data Protection + +- **Severity:** Source check severity +- **Source:** AC-07 +- **Domain:** Memory & Data Privacy +- **Description:** Maps AgentCore memory encryption controls. + +### AG-20: Private AgentCore Connectivity + +- **Severity:** Source check severity +- **Source:** AC-08 +- **Domain:** Bounded Autonomy +- **Description:** Maps VPC endpoint coverage for AgentCore services. + +### AG-21: Resource Policy Boundary + +- **Severity:** Source check severity +- **Source:** AC-10 +- **Domain:** Agent Identity & Access +- **Description:** Maps AgentCore runtime and gateway resource-based policy controls. + +### AG-22: Policy Engine Data Protection + +- **Severity:** Source check severity +- **Source:** AC-11 +- **Domain:** Tool Authorization +- **Description:** Maps AgentCore policy engine encryption controls. + +### AG-23: Gateway Data Protection + +- **Severity:** Source check severity +- **Source:** AC-12 +- **Domain:** Tool Authorization +- **Description:** Maps AgentCore gateway encryption controls. + +### AG-24: Gateway Inbound Authorization + +- **Severity:** High +- **Source:** AgentCore `ListGateways` and `GetGateway` +- **Domain:** Tool Authorization +- **Description:** Fails gateways with missing, unknown, or `NONE` authorizers. Passes `AWS_IAM` and `CUSTOM_JWT`. `AUTHENTICATE_ONLY` passes only when an AgentCore policy engine is attached in `ENFORCE` mode, because the gateway authenticates the SigV4 caller but does not make an authorization decision for that authorizer type. + +### AG-25: Gateway Tool Policy Enforcement + +- **Severity:** High +- **Source:** AgentCore `GetGateway.policyEngineConfiguration` +- **Domain:** Tool Authorization +- **Description:** Fails gateways without a policy engine or with policy engine mode other than `ENFORCE`. + +### AG-26: Gateway Error Detail Exposure + +- **Severity:** Medium +- **Source:** AgentCore `GetGateway.exceptionLevel` +- **Domain:** Auditability & Observability +- **Description:** Fails gateways configured to return `DEBUG`-level exception detail. + +### AG-27: Gateway WAF Protection + +- **Severity:** Low +- **Source:** AgentCore `GetGateway.webAclArn` +- **Domain:** Abuse & Cost Protection +- **Description:** Fails AgentCore gateways without an associated AWS WAF web ACL. + +--- + ## Additional Resources - [Amazon SageMaker Security Best Practices](https://docs.aws.amazon.com/sagemaker/latest/dg/security.html) - [Amazon Bedrock Security](https://docs.aws.amazon.com/bedrock/latest/userguide/security.html) +- [AWS Well-Architected Agentic AI Lens](https://docs.aws.amazon.com/wellarchitected/latest/agentic-ai-lens/agentic-ai-lens.html) - [AWS Security Hub SageMaker Controls](https://docs.aws.amazon.com/securityhub/latest/userguide/sagemaker-controls.html) - [AWS Well-Architected Framework - Security Pillar](https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/welcome.html) diff --git a/docs/SECURITY_CHECKS_FINSERV.md b/docs/SECURITY_CHECKS_FINSERV.md index 433b598..7ee559a 100644 --- a/docs/SECURITY_CHECKS_FINSERV.md +++ b/docs/SECURITY_CHECKS_FINSERV.md @@ -136,7 +136,7 @@ Because `aws-samples` is an OSPO-managed organization, pushes to your personal f ## Relationship to upstream SM/BR/AC checks The upstream [sample-aiml-security-assessment](https://github.com/aws-samples/sample-aiml-security-assessment) -framework already provides 70 core security checks (SM-01 to SM-25, BR-01 to BR-32, AC-01 to AC-13). +framework already provides 70 core security checks (SM-01 to SM-25, BR-01 to BR-32, AC-01 to AC-13) and 27 always-on Agentic AI Security checks (AG-01 to AG-27). The 69 FS checks in this document are **additive**: they enhance the upstream with FinServ-specific detection and remediation guidance drawn from the Responsible AI GRC guide. A few FS checks overlap with upstream checks — in those cases, the FS check adds FinServ-specific depth @@ -179,7 +179,7 @@ whether the FS check adds FinServ-specific regulatory specificity, (3) severity - **Extend upstream (5 FS checks merged into 5 upstream checks):** FS-17 → SM-07; FS-18 → SM-23; FS-19 → SM-22; FS-23 → BR-06; FS-64 → BR-04. These checks are replaced by upstream-extension notes in Parts 1 and 3 and are removed from `finserv_assessments/app.py`. - **Keep separate (64 FS checks):** All other FS checks ship as standalone entries. This includes FS-20, FS-22, FS-25, FS-26, FS-39, FS-41, all Guardrail-policy-level checks (FS-27, FS-28, FS-36, FS-38, FS-45, FS-47, FS-50, FS-51, FS-59), and all FS checks that have no upstream overlap at all. -After consolidation the combined framework contains **70 upstream + 64 FS = 134 distinct checks** (down from 70 + 69 = 139 before merging). The consolidation reduces duplication without losing FinServ-specific regulatory depth. +After consolidation the combined framework contains **70 upstream + 27 AG + 64 FS = 161 distinct checks** (down from 70 + 27 AG + 69 FS = 166 before merging FinServ overlaps). The consolidation reduces duplication without losing FinServ-specific regulatory depth. --- diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 786eb6f..cb14d08 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -323,7 +323,7 @@ A: Minimal ongoing costs: **Q: Can I customize which security checks are included?** -A: Currently, all 70 core checks run by default to provide comprehensive coverage. If `EnableFinServAssessment` is enabled, the 64 optional Financial Services GenAI risk checks also run. You can filter results in the generated HTML reports by severity, status, service, industry, or region. Future versions may support selective check execution. +A: Currently, all 70 core checks and 27 Agentic AI Security checks run by default to provide comprehensive coverage. If `EnableFinServAssessment` is enabled, the 64 optional Financial Services GenAI risk checks also run. You can filter results in the generated HTML reports by severity, status, assessment area, industry, or region. Future versions may support selective check execution. **Q: Can I add custom security checks?** diff --git a/sample-reports/README.md b/sample-reports/README.md index 047aeb7..882534b 100644 --- a/sample-reports/README.md +++ b/sample-reports/README.md @@ -10,6 +10,8 @@ Interactive HTML reports demonstrating the assessment output: - **[security_assessment_single_account.html](security_assessment_single_account.html)** - Example report for a single AWS account showing 7 findings across Bedrock, SageMaker, and AgentCore - **[security_assessment_multi_account.html](security_assessment_multi_account.html)** - Example consolidated report for 3 AWS accounts showing 73 findings +- **[security_assessment_multi_account_agentic_prototype.html](security_assessment_multi_account_agentic_prototype.html)** - Prototype based on the existing multi-account report with an Agentic AI security overlay added to the same UI +- **[agentic-ai-lens-prototype.html](agentic-ai-lens-prototype.html)** - Prototype report showing how security-scoped Agentic AI check and control-domain metadata could be added to the HTML experience **Features:** - Executive dashboard with severity breakdown @@ -17,6 +19,7 @@ Interactive HTML reports demonstrating the assessment output: - Filterable findings table - Light/dark mode toggle - Direct links to AWS documentation +- Agentic AI security prototype view with security control summaries, question mapping, and lens-aware filters **How to view:** Download the HTML file and open it in your web browser. diff --git a/tests/test_agentcore_checks.py b/tests/test_agentcore_checks.py index 7e42c26..4fa4b82 100644 --- a/tests/test_agentcore_checks.py +++ b/tests/test_agentcore_checks.py @@ -773,6 +773,205 @@ def test_ac13_schema_valid(self): assert_finding_schema(f) +# =================================================================== +# AG-24..AG-27: check_agentcore_gateway_agentic_security +# =================================================================== +class TestAgenticGatewaySecurity: + """Agentic AI Gateway security checks.""" + + @patch("agentcore_app.agentcore_client") + def test_gateway_policy_controls_fail_when_not_enforced(self, mock_ac): + mock_ac.list_gateways.return_value = { + "items": [{"gatewayId": "gw-1", "name": "TestGateway"}] + } + mock_ac.get_gateway.return_value = { + "gatewayId": "gw-1", + "name": "TestGateway", + "authorizerType": "NONE", + "policyEngineConfiguration": { + "arn": "arn:aws:bedrock-agentcore:us-east-1:123456789012:policy-engine/TestEngine-abcdefghij", + "mode": "LOG_ONLY", + }, + "exceptionLevel": "DEBUG", + } + + findings = agentcore_app.check_agentcore_gateway_agentic_security() + statuses = {f["Check_ID"]: f["Status"] for f in findings} + + assert statuses["AG-24"] == "Failed" + assert statuses["AG-25"] == "Failed" + assert statuses["AG-26"] == "Failed" + assert statuses["AG-27"] == "Failed" + for finding in findings: + assert_finding_schema(finding) + + @patch("agentcore_app.agentcore_client") + def test_gateway_authorizer_unspecified_fails_closed(self, mock_ac): + mock_ac.list_gateways.return_value = { + "items": [{"gatewayId": "gw-1", "name": "TestGateway"}] + } + mock_ac.get_gateway.return_value = { + "gatewayId": "gw-1", + "name": "TestGateway", + "policyEngineConfiguration": { + "arn": "arn:aws:bedrock-agentcore:us-east-1:123456789012:policy-engine/TestEngine-abcdefghij", + "mode": "ENFORCE", + }, + "webAclArn": "arn:aws:wafv2:us-east-1:123456789012:regional/webacl/test/abc", + } + + findings = agentcore_app.check_agentcore_gateway_agentic_security() + ag24 = [f for f in findings if f["Check_ID"] == "AG-24"] + + assert ag24 + assert ag24[0]["Status"] == "Failed" + assert "unspecified" in ag24[0]["Finding_Details"] + + @patch("agentcore_app.agentcore_client") + def test_gateway_authenticate_only_without_enforced_policy_fails_closed( + self, mock_ac + ): + mock_ac.list_gateways.return_value = { + "items": [{"gatewayId": "gw-1", "name": "TestGateway"}] + } + mock_ac.get_gateway.return_value = { + "gatewayId": "gw-1", + "name": "TestGateway", + "authorizerType": "AUTHENTICATE_ONLY", + "policyEngineConfiguration": { + "arn": "arn:aws:bedrock-agentcore:us-east-1:123456789012:policy-engine/TestEngine-abcdefghij", + "mode": "LOG_ONLY", + }, + "webAclArn": "arn:aws:wafv2:us-east-1:123456789012:regional/webacl/test/abc", + } + + findings = agentcore_app.check_agentcore_gateway_agentic_security() + ag24 = [f for f in findings if f["Check_ID"] == "AG-24"] + + assert ag24 + assert ag24[0]["Status"] == "Failed" + assert "AUTHENTICATE_ONLY" in ag24[0]["Finding_Details"] + + @patch("agentcore_app.agentcore_client") + def test_gateway_authenticate_only_with_enforced_policy_passes(self, mock_ac): + mock_ac.list_gateways.return_value = { + "items": [{"gatewayId": "gw-1", "name": "TestGateway"}] + } + mock_ac.get_gateway.return_value = { + "gatewayId": "gw-1", + "name": "TestGateway", + "authorizerType": "AUTHENTICATE_ONLY", + "policyEngineConfiguration": { + "arn": "arn:aws:bedrock-agentcore:us-east-1:123456789012:policy-engine/TestEngine-abcdefghij", + "mode": "ENFORCE", + }, + "webAclArn": "arn:aws:wafv2:us-east-1:123456789012:regional/webacl/test/abc", + } + + findings = agentcore_app.check_agentcore_gateway_agentic_security() + ag24 = [f for f in findings if f["Check_ID"] == "AG-24"] + + assert ag24 + assert ag24[0]["Status"] == "Passed" + assert "policy engine" in ag24[0]["Finding_Details"] + + @patch("agentcore_app.agentcore_client") + def test_gateway_detail_access_denied_returns_na(self, mock_ac): + mock_ac.list_gateways.return_value = { + "items": [{"gatewayId": "gw-1", "name": "TestGateway"}] + } + mock_ac.get_gateway.side_effect = _make_client_error( + "AccessDeniedException", "Denied" + ) + + findings = agentcore_app.check_agentcore_gateway_agentic_security() + + assert len(findings) == 1 + assert findings[0]["Check_ID"] == "AG-24" + assert findings[0]["Status"] == "N/A" + assert findings[0]["Severity"] == "Informational" + assert "Unable to retrieve Gateway" in findings[0]["Finding_Details"] + assert_finding_schema(findings[0]) + + @patch("agentcore_app.agentcore_client") + def test_gateway_policy_controls_pass_when_enforced(self, mock_ac): + mock_ac.list_gateways.return_value = { + "items": [{"gatewayId": "gw-1", "name": "TestGateway"}] + } + mock_ac.get_gateway.return_value = { + "gatewayId": "gw-1", + "name": "TestGateway", + "authorizerType": "AWS_IAM", + "policyEngineConfiguration": { + "arn": "arn:aws:bedrock-agentcore:us-east-1:123456789012:policy-engine/TestEngine-abcdefghij", + "mode": "ENFORCE", + }, + "webAclArn": "arn:aws:wafv2:us-east-1:123456789012:regional/webacl/test/abc", + } + + findings = agentcore_app.check_agentcore_gateway_agentic_security() + statuses = {f["Check_ID"]: f["Status"] for f in findings} + + assert statuses["AG-24"] == "Passed" + assert statuses["AG-25"] == "Passed" + assert statuses["AG-26"] == "Passed" + assert statuses["AG-27"] == "Passed" + + +class TestAgenticAgentCoreMapping: + """Agentic AI AG-* rows are generated from API-backed AgentCore checks.""" + + EXPECTED_AGENTIC_MAPPINGS = { + "AC-01": "AG-15", + "AC-02": "AG-16", + "AC-03": "AG-17", + "AC-04": "AG-18", + "AC-07": "AG-19", + "AC-08": "AG-20", + "AC-10": "AG-21", + "AC-11": "AG-22", + "AC-12": "AG-23", + } + + def test_all_agentcore_agentic_mappings_emit_expected_rows(self): + source_findings = [] + for source_check_id in self.EXPECTED_AGENTIC_MAPPINGS: + source_findings.append( + { + "Account_ID": "123456789012", + "Check_ID": source_check_id, + "Finding": f"{source_check_id} source finding", + "Finding_Details": f"{source_check_id} source details", + "Resolution": "No action required.", + "Reference": "https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/security.html", + "Severity": "Medium", + "Status": "Passed", + "Region": "us-east-1", + } + ) + + findings = agentcore_app.build_agentic_agentcore_security_findings( + source_findings + ) + + assert len(findings) == len(self.EXPECTED_AGENTIC_MAPPINGS) + actual_by_source = {} + for finding in findings: + details = finding["Finding_Details"] + source_check_id = details.split("Source check ", 1)[1].split(":", 1)[0] + actual_by_source[source_check_id] = finding + + assert finding["Status"] == "Passed" + assert finding["Severity"] == "Medium" + assert finding["Region"] == "us-east-1" + assert f"Source check {source_check_id}" in details + assert_finding_schema(finding) + + assert set(actual_by_source) == set(self.EXPECTED_AGENTIC_MAPPINGS) + for source_check_id, expected_ag_id in self.EXPECTED_AGENTIC_MAPPINGS.items(): + assert actual_by_source[source_check_id]["Check_ID"] == expected_ag_id + + # =================================================================== # lambda_handler: multi-region gating and availability probe # =================================================================== @@ -885,7 +1084,7 @@ def test_non_primary_region_skips_global_iam_checks(self): assert "AC-02" not in check_ids assert "AC-03" not in check_ids assert "AC-09" not in check_ids - assert check_ids == {"AC-00"} + assert check_ids == {"AC-00"} | {f"AG-{i:02d}" for i in range(15, 28)} def test_optin_region_error_treated_as_unavailable(self): # A region-not-enabled ClientError code makes agentcore_client None, so diff --git a/tests/test_bedrock_checks.py b/tests/test_bedrock_checks.py index e63485a..3c51816 100644 --- a/tests/test_bedrock_checks.py +++ b/tests/test_bedrock_checks.py @@ -1618,7 +1618,7 @@ def client_factory(service, **kwargs): assert passed_findings[0]["Check_ID"] == "BR-15" @patch("bedrock_app.boto3.client") - def test_br15_access_denied_returns_failed(self, mock_client): + def test_br15_access_denied_returns_na(self, mock_client): check = bedrock_app.check_bedrock_cross_account_guardrails org_client = MagicMock() @@ -1630,7 +1630,9 @@ def test_br15_access_denied_returns_failed(self, mock_client): result = check(region="Global") findings = extract_csv_data(result) assert len(findings) >= 1 - assert findings[0]["Status"] == "Failed" + # Access-denied paths resolve to N/A (CLAUDE.md status semantics), + # not Failed — a permission gap is not a security misconfiguration. + assert findings[0]["Status"] == "N/A" assert findings[0]["Check_ID"] == "BR-15" def test_br15_schema_valid(self): @@ -1724,6 +1726,44 @@ def test_br16_access_denied_returns_failed(self, mock_client): assert findings[0]["Status"] == "Failed" assert findings[0]["Check_ID"] == "BR-16" + @patch("bedrock_app.boto3.client") + def test_br16_paginates_and_continues_when_one_guardrail_detail_fails( + self, mock_client + ): + check = bedrock_app.check_bedrock_guardrail_tier + + bedrock_client = MagicMock() + bedrock_client.list_guardrails.side_effect = [ + { + "guardrails": [{"id": "gr-error", "name": "DeletedGuardrail"}], + "nextToken": "page-2", + }, + { + "guardrails": [{"id": "gr-ok", "name": "StandardGuardrail"}], + }, + ] + bedrock_client.get_guardrail.side_effect = [ + ClientError( + {"Error": {"Code": "ResourceNotFoundException"}}, + "GetGuardrail", + ), + {"guardrail": {"contentPolicy": {"tier": {"tierName": "STANDARD"}}}}, + ] + mock_client.return_value = bedrock_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + statuses = [f["Status"] for f in findings] + + assert bedrock_client.list_guardrails.call_count == 2 + assert "N/A" in statuses + assert "Passed" in statuses + unassessed = [ + f for f in findings if "DeletedGuardrail" in f["Finding_Details"] + ] + assert unassessed + assert unassessed[0]["Severity"] == "Informational" + def test_br16_schema_valid(self): check = bedrock_app.check_bedrock_guardrail_tier with patch("bedrock_app.boto3.client") as mock_client: @@ -3202,3 +3242,134 @@ def test_br32_schema_valid(self, mock_client, mock_footprint): result = check(region="us-east-1") for f in extract_csv_data(result): assert_finding_schema(f) + + +class TestBR22ServiceQuotas: + """BR-22: Verify throttling quotas are compared against AWS defaults.""" + + @patch("bedrock_app.boto3.client") + def test_br22_custom_quota_uses_aws_default_comparison(self, mock_client): + check = bedrock_app.check_bedrock_service_quotas_throttling + + quotas_client = MagicMock() + quotas_client.list_service_quotas.side_effect = [ + { + "Quotas": [ + { + "QuotaName": "Tokens per minute for model invocations", + "QuotaCode": "L-123", + "Value": 200, + } + ], + "NextToken": "page-2", + }, + {"Quotas": []}, + ] + quotas_client.get_service_quota.return_value = { + "Quota": {"QuotaName": "Tokens per minute", "Value": 200} + } + quotas_client.get_aws_default_service_quota.return_value = { + "Quota": {"QuotaName": "Tokens per minute", "Value": 100} + } + mock_client.return_value = quotas_client + + result = check(region="us-east-1") + findings = extract_csv_data(result) + + assert quotas_client.list_service_quotas.call_count == 2 + assert findings[0]["Check_ID"] == "BR-22" + assert findings[0]["Status"] == "Passed" + assert "custom throttling quotas" in findings[0]["Finding_Details"] + + +class TestAgenticBedrockMapping: + """Agentic AI AG-* rows are generated from API-backed Bedrock checks.""" + + EXPECTED_AGENTIC_MAPPINGS = { + "BR-04": "AG-07", + "BR-06": "AG-08", + "BR-15": "AG-09", + "BR-18": "AG-10", + "BR-19": "AG-11", + "BR-21": "AG-06", + "BR-22": "AG-12", + "BR-23": "AG-02", + "BR-24": "AG-04", + "BR-26": "AG-03", + "BR-27": "AG-05", + "BR-28": "AG-01", + "BR-29": "AG-13", + "BR-32": "AG-14", + } + + def test_all_bedrock_agentic_mappings_emit_expected_rows(self): + source_rows = [] + for source_check_id in self.EXPECTED_AGENTIC_MAPPINGS: + source_rows.append( + { + "Check_ID": source_check_id, + "Finding": f"{source_check_id} source finding", + "Finding_Details": f"{source_check_id} source details", + "Resolution": "No action required.", + "Reference": "https://docs.aws.amazon.com/bedrock/latest/userguide/security.html", + "Severity": "Medium", + "Status": "Passed", + "Region": "us-east-1", + } + ) + + source_findings = [ + { + "check_name": "Bedrock Agentic Mapping Sources", + "csv_data": source_rows, + } + ] + + result = bedrock_app.build_agentic_bedrock_security_findings(source_findings) + findings = extract_csv_data(result) + + assert len(findings) == len(self.EXPECTED_AGENTIC_MAPPINGS) + actual_by_source = {} + for finding in findings: + details = finding["Finding_Details"] + source_check_id = details.split("Source check ", 1)[1].split(":", 1)[0] + actual_by_source[source_check_id] = finding + + assert finding["Status"] == "Passed" + assert finding["Severity"] == "Medium" + assert finding["Region"] == "us-east-1" + assert f"Source check {source_check_id}" in details + assert_finding_schema(finding) + + assert set(actual_by_source) == set(self.EXPECTED_AGENTIC_MAPPINGS) + for source_check_id, expected_ag_id in self.EXPECTED_AGENTIC_MAPPINGS.items(): + assert actual_by_source[source_check_id]["Check_ID"] == expected_ag_id + + def test_br28_generates_ag01_mapping(self): + source_findings = [ + { + "check_name": "Bedrock Agent Guardrail Association", + "csv_data": [ + { + "Check_ID": "BR-28", + "Finding": "Bedrock Agent Guardrail Association", + "Finding_Details": "Agent has a guardrail associated.", + "Resolution": "No action required.", + "Reference": "https://docs.aws.amazon.com/bedrock/latest/userguide/agents-guardrails.html", + "Severity": "High", + "Status": "Passed", + "Region": "us-east-1", + } + ], + } + ] + + result = bedrock_app.build_agentic_bedrock_security_findings(source_findings) + findings = extract_csv_data(result) + + assert len(findings) == 1 + assert findings[0]["Check_ID"] == "AG-01" + assert findings[0]["Status"] == "Passed" + assert findings[0]["Region"] == "us-east-1" + assert "Source check BR-28" in findings[0]["Finding_Details"] + assert_finding_schema(findings[0]) diff --git a/tests/test_consolidated_report.py b/tests/test_consolidated_report.py index 6d83d40..f6cfb2f 100644 --- a/tests/test_consolidated_report.py +++ b/tests/test_consolidated_report.py @@ -207,3 +207,34 @@ def test_global_only_yields_no_regions(self, monkeypatch): consolidated_app.generate_html_report(results) assert captured["regions"] is None + + +class TestAgenticFindingClassification: + """AG-* rows are classified into the Agentic assessment area.""" + + def test_ag_rows_from_bedrock_csv_move_to_agentic_bucket(self, monkeypatch): + captured = {} + monkeypatch.setattr( + consolidated_app, + "generate_report_from_template", + lambda **kwargs: captured.update(kwargs) or "", + ) + + results = _build_results( + { + "bedrock_security_report_exec-123_us-east-1": [ + _finding( + "AG-01", + "us-east-1", + status="Passed", + name="Agentic AI Agent Guardrail Association", + ), + ], + } + ) + + consolidated_app.generate_html_report(results) + + assert captured["service_stats"]["agentic"]["passed"] == 1 + assert captured["service_stats"]["bedrock"]["passed"] == 0 + assert captured["service_findings"]["agentic"][0]["_service"] == "agentic" diff --git a/tests/test_core_iam_coverage.py b/tests/test_core_iam_coverage.py index 99a2e63..a0a4547 100644 --- a/tests/test_core_iam_coverage.py +++ b/tests/test_core_iam_coverage.py @@ -22,6 +22,7 @@ "bedrock:ListModelInvocationJobs", # BR-31 "servicequotas:ListServiceQuotas", # BR-22 "servicequotas:GetServiceQuota", # BR-22 + "servicequotas:GetAWSDefaultServiceQuota", # BR-22 "cloudwatch:DescribeAlarms", "organizations:DescribeOrganization", # BR-15 "organizations:ListPolicies", # BR-15 @@ -49,6 +50,7 @@ "bedrock:ListModelInvocationJobs", # BR-31 "servicequotas:ListServiceQuotas", # BR-22 "servicequotas:GetServiceQuota", # BR-22 + "servicequotas:GetAWSDefaultServiceQuota", # BR-22 "cloudwatch:DescribeAlarms", "organizations:DescribeOrganization", # BR-15 "organizations:ListPolicies", # BR-15 From 487cab8fa5f3002bcef3d56c3527f4b94b865523 Mon Sep 17 00:00:00 2001 From: Agasthi Kothurkar Date: Sun, 28 Jun 2026 11:12:06 -0400 Subject: [PATCH 10/10] Fix agentic lens report grouping Render Agentic AI findings under a dedicated By Lens nav section and avoid BR-22/AG-12 failed findings in regions with no Bedrock footprint. --- .../security/bedrock_assessments/app.py | 22 ++++++++ .../report_template.py | 14 ++++- .../test_generate_report.py | 10 ++++ tests/test_bedrock_checks.py | 56 +++++++++++++++++-- 4 files changed, 97 insertions(+), 5 deletions(-) diff --git a/aiml-security-assessment/functions/security/bedrock_assessments/app.py b/aiml-security-assessment/functions/security/bedrock_assessments/app.py index ee5301f..993ed1e 100644 --- a/aiml-security-assessment/functions/security/bedrock_assessments/app.py +++ b/aiml-security-assessment/functions/security/bedrock_assessments/app.py @@ -4435,6 +4435,28 @@ def check_bedrock_service_quotas_throttling(region: str = "") -> Dict[str, Any]: "csv_data": [], } + # Default quotas are only actionable when Bedrock is actually in use in + # the region. Otherwise, the check creates false regional failures. + bedrock_footprint_found = detect_bedrock_regional_footprint(region=region) + if bedrock_footprint_found is not True: + findings["details"] = bedrock_footprint_na_detail(bedrock_footprint_found) + findings["csv_data"].append( + create_finding( + check_id="BR-22", + finding_name="Model Invocation Throttling Limits Check", + finding_details=bedrock_footprint_na_detail( + bedrock_footprint_found, + "to assess model invocation throttling quotas", + ), + resolution="No action required", + reference="https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html", + severity="Informational", + status="N/A", + region=region, + ) + ) + return findings + service_quotas_client = boto3.client( "service-quotas", config=boto3_config, region_name=region ) diff --git a/aiml-security-assessment/functions/security/generate_consolidated_report/report_template.py b/aiml-security-assessment/functions/security/generate_consolidated_report/report_template.py index b05836a..a6688e7 100644 --- a/aiml-security-assessment/functions/security/generate_consolidated_report/report_template.py +++ b/aiml-security-assessment/functions/security/generate_consolidated_report/report_template.py @@ -188,6 +188,11 @@ def get_html_template() -> str: .section-title .service-icon svg {{ border-radius: 8px; }} .nav-item .count {{ margin-left: auto; font-size: 12px; font-weight: 600; background: var(--surface-2); padding: 2px 8px; border-radius: 10px; }} .nav-item.active .count {{ background: var(--accent); color: #fff; }} + .nav-section.lens-nav {{ border-top: 1px solid var(--border); padding-top: 16px; margin-top: -8px; }} + .lens-nav .nav-item {{ background: var(--warning-soft); color: var(--text); box-shadow: inset 3px 0 0 var(--warning); }} + .lens-nav .nav-item:hover {{ background: var(--warning-soft); color: var(--warning); }} + .lens-nav .nav-item.active {{ background: var(--warning-soft); color: var(--warning); }} + .lens-nav .nav-item .count {{ background: var(--warning); color: #fff; }} .nav-section.industry-nav {{ border-top: 1px solid var(--border); padding-top: 16px; margin-top: -8px; }} .industry-nav .nav-item {{ background: var(--accent-soft); color: var(--text); box-shadow: inset 3px 0 0 var(--accent); }} .industry-nav .nav-item:hover {{ background: var(--accent-soft); color: var(--accent); }} @@ -332,8 +337,8 @@ def get_html_template() -> str: AgentCore {agentcore_total} - {agentic_nav} + {lens_nav} {industry_nav}

Generated: {date_display}

@@ -1007,6 +1012,11 @@ def generate_html_report( + " Agentic AI Security" + f'{agentic_total}' ) + lens_nav = ( + '" + ) agentic_filter_option = '' agentic_service_card = ( '
' @@ -1047,6 +1057,7 @@ def generate_html_report( ) else: agentic_nav = "" + lens_nav = "" agentic_filter_option = "" agentic_service_card = "" agentic_section = "" @@ -1203,6 +1214,7 @@ def generate_html_report( agentcore_region_filter=agentcore_region_filter, agentic_region_filter=agentic_region_filter, agentic_nav=agentic_nav, + lens_nav=lens_nav, agentic_filter_option=agentic_filter_option, agentic_service_card=agentic_service_card, agentic_section=agentic_section, diff --git a/aiml-security-assessment/functions/security/generate_consolidated_report/test_generate_report.py b/aiml-security-assessment/functions/security/generate_consolidated_report/test_generate_report.py index 353f194..4f1c5be 100644 --- a/aiml-security-assessment/functions/security/generate_consolidated_report/test_generate_report.py +++ b/aiml-security-assessment/functions/security/generate_consolidated_report/test_generate_report.py @@ -431,6 +431,8 @@ def test_agentic_security_renders_when_present(self): self.assertIn('id="agentic"', html) self.assertIn('id="agenticTable"', html) self.assertIn('', html) + self.assertIn("

By Lens

", html) + self.assertIn('class="nav-section lens-nav"', html) self.assertIn("AG-01", html) self.assertIn('data-service="agentic"', html) self.assertIn("Agentic AI Security Findings", html) @@ -444,6 +446,12 @@ def test_agentic_security_renders_when_present(self): 1, ) self.assertIn("Human-in-the-loop governance", html) + by_service_nav = html.split("

By Service

", 1)[1].split( + "

By Lens

", 1 + )[0] + self.assertNotIn("Agentic AI Security", by_service_nav) + by_lens_nav = html.split("

By Lens

", 1)[1].split("", 1)[0] + self.assertIn("Agentic AI Security", by_lens_nav) def test_agentic_security_omitted_when_absent(self): """With no AG-* data the Agentic section is omitted cleanly.""" @@ -452,6 +460,8 @@ def test_agentic_security_omitted_when_absent(self): self.assertNotIn('id="agentic"', html) self.assertNotIn('id="agenticTable"', html) self.assertNotIn('', html) + self.assertNotIn("

By Lens

", html) + self.assertNotIn('class="nav-section lens-nav"', html) self.assertIn( "wellarchitected/latest/agentic-ai-lens/agentic-ai-lens.html", html ) diff --git a/tests/test_bedrock_checks.py b/tests/test_bedrock_checks.py index 3c51816..e139190 100644 --- a/tests/test_bedrock_checks.py +++ b/tests/test_bedrock_checks.py @@ -1758,9 +1758,7 @@ def test_br16_paginates_and_continues_when_one_guardrail_detail_fails( assert bedrock_client.list_guardrails.call_count == 2 assert "N/A" in statuses assert "Passed" in statuses - unassessed = [ - f for f in findings if "DeletedGuardrail" in f["Finding_Details"] - ] + unassessed = [f for f in findings if "DeletedGuardrail" in f["Finding_Details"]] assert unassessed assert unassessed[0]["Severity"] == "Informational" @@ -3247,8 +3245,27 @@ def test_br32_schema_valid(self, mock_client, mock_footprint): class TestBR22ServiceQuotas: """BR-22: Verify throttling quotas are compared against AWS defaults.""" + @patch("bedrock_app.detect_bedrock_regional_footprint", return_value=False) @patch("bedrock_app.boto3.client") - def test_br22_custom_quota_uses_aws_default_comparison(self, mock_client): + def test_br22_no_regional_footprint_returns_na(self, mock_client, mock_footprint): + check = bedrock_app.check_bedrock_service_quotas_throttling + + result = check(region="sa-east-1") + findings = extract_csv_data(result) + + assert findings[0]["Check_ID"] == "BR-22" + assert findings[0]["Status"] == "N/A" + assert findings[0]["Severity"] == "Informational" + assert findings[0]["Finding_Details"] == ( + "No regional Bedrock resources found to assess model invocation throttling quotas" + ) + mock_client.assert_not_called() + + @patch("bedrock_app.detect_bedrock_regional_footprint", return_value=True) + @patch("bedrock_app.boto3.client") + def test_br22_custom_quota_uses_aws_default_comparison( + self, mock_client, mock_footprint + ): check = bedrock_app.check_bedrock_service_quotas_throttling quotas_client = MagicMock() @@ -3345,6 +3362,37 @@ def test_all_bedrock_agentic_mappings_emit_expected_rows(self): for source_check_id, expected_ag_id in self.EXPECTED_AGENTIC_MAPPINGS.items(): assert actual_by_source[source_check_id]["Check_ID"] == expected_ag_id + def test_br22_na_generates_ag12_informational_na(self): + source_findings = [ + { + "check_name": "Model Invocation Throttling Limits Check", + "csv_data": [ + { + "Check_ID": "BR-22", + "Finding": "Model Invocation Throttling Limits Check", + "Finding_Details": "No regional Bedrock resources found to assess model invocation throttling quotas", + "Resolution": "No action required", + "Reference": "https://docs.aws.amazon.com/bedrock/latest/userguide/quotas.html", + "Severity": "Informational", + "Status": "N/A", + "Region": "sa-east-1", + } + ], + } + ] + + result = bedrock_app.build_agentic_bedrock_security_findings(source_findings) + findings = extract_csv_data(result) + + assert len(findings) == 1 + assert findings[0]["Check_ID"] == "AG-12" + assert findings[0]["Status"] == "N/A" + assert findings[0]["Severity"] == "Informational" + assert findings[0]["Region"] == "sa-east-1" + assert "Source check BR-22" in findings[0]["Finding_Details"] + assert "No regional Bedrock resources found" in findings[0]["Finding_Details"] + assert_finding_schema(findings[0]) + def test_br28_generates_ag01_mapping(self): source_findings = [ {