diff --git a/docs/admin-guide.md b/docs/admin-guide.md index 731eda722..64b715ce7 100644 --- a/docs/admin-guide.md +++ b/docs/admin-guide.md @@ -59,9 +59,9 @@ definitions in them as desired. ## adfconfig The `adfconfig.yml` file resides on the -[management account](#management-account) CodeCommit Repository (in `us-east-1`) -and defines the general high-level configuration for the AWS Deployment -Framework. +[management account](#management-account) CodeCommit Repository +(in `us-east-1` or `cn-north-1`) and defines the general +high-level configuration for the AWS Deployment Framework. The configuration properties are synced into AWS Systems Manager Parameter Store and are used for certain orchestration options throughout your @@ -964,8 +964,8 @@ To determine the current version, follow these steps: ### ADF version you have deployed To check the current version of ADF that you have deployed, go to the management -account in us-east-1. Check the CloudFormation stack output or tag of the -`serverlessrepo-aws-deployment-framework` Stack. +account in us-east-1 or cn-north-1. Check the CloudFormation stack +output or tag of the `serverlessrepo-aws-deployment-framework` Stack. - In the outputs tab, it will show the version as the `ADFVersionNumber`. - In the tags on the CloudFormation stack, it is presented as @@ -985,8 +985,8 @@ releases](https://github.com/awslabs/aws-deployment-framework/releases). The `serverlessrepo-aws-deployment-framework` stack is updated through this process with new changes that were included in that release of ADF. -To check the progress in the management account in `us-east-1`, follow these -steps: +To check the progress in the management account in +`us-east-1` or `cn-north-1`, follow these steps: 1. Go to the [CloudFormation console](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks?filteringStatus=active&filteringText=serverlessrepo-aws-deployment-framework&viewNested=true&hideStacks=false) @@ -1028,7 +1028,7 @@ Which branch is used is determined by: Alternatively, you can also perform the update using the AWS CLI. -In the management account in `us-east-1`: +In the management account in `us-east-1` or `cn-north-1`: 1. Go to the Pull Request section of the `aws-deployment-framework-bootstrap` [CodeCommit @@ -1043,7 +1043,7 @@ In the management account in `us-east-1`: changes that it proposes. Once reviewed, merge the pull request to continue. Confirm the `aws-deployment-framework-bootstrap` pipeline in the management -account in `us-east-1`: +account in `us-east-1` or `cn-north-1`: 1. Go to the [CodePipeline console for the aws-deployment-framework-bootstrap pipeline](https://console.aws.amazon.com/codesuite/codepipeline/pipelines/aws-deployment-framework-bootstrap-pipeline/view?region=us-east-1). @@ -1059,7 +1059,7 @@ creation and on-boarding process in parallel. These are managed through Step Function state machines. 1. Navigate to the [AWS Step Functions service](https://us-east-1.console.aws.amazon.com/states/home?region=us-east-1#/statemachines) - in the management account in `us-east-1`. + in the management account in `us-east-1` or `cn-north-1`. 2. Check the `AccountManagementStateMachine...` state machine, all recent invocations since we performed the update should succeed. It could be the case that there are no invocations at all. In that case, wait a minute and @@ -1138,10 +1138,11 @@ Alternatively, you can also perform the update using the AWS CLI. If you wish to remove ADF you can delete the CloudFormation stack named `serverlessrepo-aws-deployment-framework` in the management account in -the `us-east-1` region. This will remove most resources created by ADF +the `us-east-1` region for global partition deployments; for China deployments +in `cn-north-1` region. This will remove most resources created by ADF in the management account. With the exception of S3 buckets and SSM parameters. -If you bootstrapped ADF into the management account you need to manually remove -the bootstrap stacks as well. +If you bootstrapped ADF into the management account you need to manually +remove the bootstrap stacks as well. Feel free to delete the S3 buckets, SSM parameters that start with the `/adf` prefix, as well as other CloudFormation stacks such as: @@ -1164,7 +1165,7 @@ the base stack when the account is moved to the Root of the AWS Organization. One thing to keep in mind if you are planning to re-install ADF is that you will want to clean up the parameter from SSM Parameter Store. You can safely remove all `/adf` prefixed SSM parameters. But most importantly, you need to -remove the `/adf/deployment_account_id` in `us-east-1` on the +remove the `/adf/deployment_account_id` in `us-east-1` or `cn-north-1` on the management account. As AWS Step Functions uses this parameter to determine if ADF has already got a deployment account setup. If you re-install ADF with this parameter set to a @@ -1187,7 +1188,7 @@ There are two ways to enable this: to deploy the latest version again, set the `Log Level` to `DEBUG` to get extra logging information about the issue you are experiencing. 2. If you are running an older version of ADF, please navigate to the - CloudFormation Console in `us-east-1` of the AWS Management account. + CloudFormation Console in `us-east-1` or `cn-north-1` of the AWS Management account. 3. Update the stack. 4. For any ADF deployment of `v3.2.0` and later, please change the `Log Level` parameter and set it to `DEBUG`. Deploy those changes and revert them after @@ -1202,7 +1203,7 @@ Please trace the failed component and dive into/report the debug information. The main components to look at are: -1. In the AWS Management Account in `us-east-1`: +1. In the AWS Management Account in `us-east-1` or `cn-north-1`: 2. The [CloudFormation aws-deployment-framework stack](https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks?filteringStatus=active&filteringText=aws-deployment-framework&viewNested=true&hideStacks=false). 3. The [CloudWatch Logs for the Lambda functions deployed by ADF](https://console.aws.amazon.com/lambda/home?region=us-east-1#/functions?f0=true&n0=false&op=and&v0=ADF). 4. Check if the [CodeCommit pull @@ -1211,8 +1212,8 @@ The main components to look at are: branch for the `aws-deployment-framework-bootstrap` (ADF Bootstrap) repository. 5. The [CodePipeline execution of the AWS Bootstrap pipeline](https://console.aws.amazon.com/codesuite/codepipeline/pipelines/aws-deployment-framework-bootstrap-pipeline/view?region=us-east-1). 6. Navigate to the [AWS Step Functions service](https://us-east-1.console.aws.amazon.com/states/home?region=us-east-1#/statemachines) - in the management account in `us-east-1`. Check the state machines named - `AccountManagementStateMachine...` and + in the management account in `us-east-1` or `cn-north-1`. Check the + state machines named `AccountManagementStateMachine...` and `AccountBootstrappingStateMachine...`. Look at recent executions only. - When you find one that has a failed execution, check the components that are marked orange/red in the diagram. diff --git a/docs/installation-guide.md b/docs/installation-guide.md index 237ad23f8..023b4dae7 100644 --- a/docs/installation-guide.md +++ b/docs/installation-guide.md @@ -28,7 +28,7 @@ It is okay to install ADF and AWS Control Tower in different regions. For example: - Install AWS Control Tower in `eu-central-1`. -- Install ADF in `us-east-1`. +- Install ADF in `us-east-1` or `cn-north-1`. **If you want to use ADF and AWS Control Tower, we recommend that you setup AWS Control Tower prior to installing ADF.** @@ -44,12 +44,12 @@ Ensure you have setup [AWS CloudTrail](https://aws.amazon.com/cloudtrail/) regions**, the trail itself can be created in any region. Events [triggered via CloudTrail](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_incident-response.html) for AWS Organizations can only be acted upon in the us-east-1 (North Virginia) -region. +or cn-northwest-1 region. Please use the [AWS CloudTrail instructions](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-and-update-a-trail.html) -to configure the CloudTrail in the `us-east-1` region within the AWS -Organizations Management AWS Account. +to configure the CloudTrail in the `us-east-1` or `cn-north-1` region +within the AWS Organizations Management AWS Account. ### 1.2. Enable AWS Organizations API Access @@ -289,10 +289,10 @@ or applications into via AWS CodePipeline *(this can be updated later)*. When deploying ADF for the first time, part of the installation process will automatically create an AWS CodeCommit repository in the management AWS Account -within the `us-east-1` region. It will also make the initial commit to the -default branch of this repository with a default set of examples that act as a -starting point to help define the AWS Account bootstrapping processes for your -Organization. +within the `us-east-1` or `cn-north-1` region. It will also make the initial +commit to the default branch of this repository with a default set of +examples that act as a starting point to help define the AWS Account +bootstrapping processes for your Organization. Part of the questions that follow will end up in the initial commit into the repository. These are passed directly the `adfconfig.yml` file prior to it @@ -330,7 +330,7 @@ To gather the values, you can either find them in the `aws-deployment-framework-bootstrap` repository in the `adfconfig.yml` file. Or by looking up the values that were specified the last time ADF got installed/updated via the CloudFormation template parameters of the -`serverlessrepo-aws-deployment-framework` stack in `us-east-1`. +`serverlessrepo-aws-deployment-framework` stack in `us-east-1` or `cn-north-1`. #### Stack Name @@ -352,6 +352,7 @@ Value to use depends on the AWS partition it is deployed to: - For the AWS partition (most common), use; `us-east-1` - For the US-Gov partition, use: `us-gov-west-1` +- For the China partition, use `cn-north-1` **Explanation:** ADF needs to be deployed in the region where the control plane of the @@ -517,7 +518,7 @@ This can always be updated later via the `adfconfig.yml` file. You don't need to include the main region in this list. For example, if you use the example values for the default region and target regions, it will allow -pipelines to deploy to `eu-west-1`, `eu-central-`, and `us-east-1`. +pipelines to deploy to `eu-west-1`, `eu-central-`, `cn-north-1` and `us-east-1`. *This is not required when performing an update between versions of ADF.* *Only supported when installing ADF for the first time. @@ -647,8 +648,9 @@ automatically in the background, to follow its progress: 1. Please navigate to the AWS Console in the AWS Management account. As the stack `serverlessrepo-aws-deployment-framework` completes you can now - open AWS CodePipeline from within the management account in `us-east-1` and - see that there is an initial pipeline execution that started. + open AWS CodePipeline from within the management account in `us-east-1` + or `cn-north-1` and see that there is an initial pipeline + execution that started. Upon first installation, this pipeline might fail to fetch the source code from the repository. Click the retry failed action button to try again. @@ -693,7 +695,7 @@ automatically in the background, to follow its progress: that started the bootstrap process for the deployment account. You can view the progress of this in the management account in the AWS Step Functions console for the step function `AccountBootstrappingStateMachine-` in the - `us-east-1` region. + `us-east-1` or `cn-north-1` region. 3. Once the Step Function has completed, switch roles over to the newly bootstrapped deployment account in the region you defined as your main diff --git a/docs/samples-guide.md b/docs/samples-guide.md index 0cc42686f..1395bbe91 100644 --- a/docs/samples-guide.md +++ b/docs/samples-guide.md @@ -70,7 +70,7 @@ Management Account. By default, there is a `global.yml` in the root of the be appended to as required. If we look at AWS Step Functions in the management account in `us-east-1` -we can see the progress of the bootstrap process. +or `cn-north-1` we can see the progress of the bootstrap process. ![run-state-machine](./images/run-state-machine.png) diff --git a/docs/user-guide.md b/docs/user-guide.md index ec19f63ef..a4ace1eb2 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -980,6 +980,8 @@ There are five different styles that one could choose from. method](https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html). - In case the bucket is stored in `us-east-1`, it will return: `https://s3.amazonaws.com/${bucket}/${key}` + - In case the bucket is stored in `cn-north-1` or `cn-northwest-1`, it will return: + `https://${bucket}.s3.${region}.amazonaws.cn/${key}` - In case the bucket is stored in any other region, it will return: `https://s3-${region}.amazonaws.com/${bucket}/${key}` - `virtual-hosted` style, will return the S3 location using the virtual hosted diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml index 6d6220ad1..b746c312f 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml @@ -809,6 +809,10 @@ Resources: commands: - aws s3 cp s3://$SHARED_MODULES_BUCKET/adf-build/ ./adf-build/ --recursive --only-show-errors - aws s3 cp --sse aws:kms --sse-kms-key-id $ADF_PIPELINE_ASSET_KMS_ARN ./adf-build/templates/ s3://$ADF_PIPELINE_ASSET_BUCKET/adf-build/templates/ --recursive --only-show-errors + - | + if [ "${AWS::Region}" = "cn-north-1" ]; then + pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple + fi - pip install -r adf-build/requirements.txt -r adf-build/helpers/requirements.txt -q -t ./adf-build pre_build: commands: @@ -1193,7 +1197,7 @@ Resources: StringEquals: aws:PrincipalOrgID: !Ref OrganizationId ArnLike: - aws:PrincipalArn: 'arn:aws:iam::*:role/adf-codecommit-role' + aws:PrincipalArn: !Sub 'arn:${AWS::Partition}:iam::*:role/adf-codecommit-role' Resource: - !Sub arn:${AWS::Partition}:s3:::${PipelineBucket}/* Principal: diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/stubs/slack.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/stubs/slack.py index 0b131fffc..7b25fb519 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/stubs/slack.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/stubs/slack.py @@ -2,88 +2,114 @@ # SPDX-License-Identifier: MIT-0 # pylint: skip-file +import os +import re +from boto3.session import Session +REGION = os.getenv("AWS_REGION", "us-east-1") +PARTITION = Session().get_partition_for_region(REGION) + +if PARTITION == "aws": + test_region = "eu-central-1" +else: + test_region = "cn-northwest-1" stub_approval_event = { - 'Records': [{ - 'EventSource': 'aws:sns', - 'EventVersion': '1.0', - 'EventSubscriptionArn': 'arn:aws:sns:eu-central-1:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example', - 'Sns': { - 'Type': 'Notification', - 'MessageId': '1', - 'TopicArn': 'arn:aws:sns:eu-central-1:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example', - 'Subject': 'APPROVAL NEEDED: AWS CodePipeline adf-pipeline-sample-vpc for action Approve', - 'Message': '{"region":"eu-central-1","consoleLink":"https://console.aws.amazon.com","approval":{"pipelineName":"adf-pipeline-sample-vpc","stageName":"approval-stage-1","actionName":"Approve","token":"fa777887-41dc-4ac4-8455-a209a93c76b9","expires":"2019-03-17T11:08Z","externalEntityLink":null,"approvalReviewLink":"https://console.aws.amazon.com/codepipeline/"}}', - 'Timestamp': '3000-03-10T11:08:34.673Z', - 'SignatureVersion': '1', - 'Signature': '1', - 'SigningCertUrl': 'https://sns.eu-central-1.amazonaws.com/SimpleNotificationService', - 'UnsubscribeUrl': 'https://sns.eu-central-1.amazonaws.com', - 'MessageAttributes': {} + "Records": [ + { + "EventSource": "aws:sns", + "EventVersion": "1.0", + "EventSubscriptionArn": f"arn:{PARTITION}:sns:{test_region}:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example", + "Sns": { + "Type": "Notification", + "MessageId": "1", + "TopicArn": f"arn:{PARTITION}:sns:{test_region}:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example", + "Subject": "APPROVAL NEEDED: AWS CodePipeline adf-pipeline-sample-vpc for action Approve", + "Message": '{"region":"{test_region}","consoleLink":"https://console.aws.amazon.com","approval":{"pipelineName":"adf-pipeline-sample-vpc","stageName":"approval-stage-1","actionName":"Approve","token":"fa777887-41dc-4ac4-8455-a209a93c76b9","expires":"2019-03-17T11:08Z","externalEntityLink":null,"approvalReviewLink":"https://console.aws.amazon.com/codepipeline/"}}', + "Timestamp": "3000-03-10T11:08:34.673Z", + "SignatureVersion": "1", + "Signature": "1", + "SigningCertUrl": f"https://sns.{test_region}.amazonaws.com/SimpleNotificationService", + "UnsubscribeUrl": f"https://sns.{test_region}.amazonaws.com", + "MessageAttributes": {}, + }, } - }] + ] } +stub_approval_event["Records"][0]["Sns"]["Message"] = re.sub( + r"{test_region}", test_region, stub_approval_event["Records"][0]["Sns"]["Message"] +) + stub_bootstrap_event = { - 'Records': [{ - 'EventSource': 'aws:sns', - 'EventVersion': '1.0', - 'EventSubscriptionArn': 'arn:aws:sns:eu-central-1:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example', - 'Sns': { - 'Type': 'Notification', - 'MessageId': '1', - 'TopicArn': 'arn:aws:sns:eu-central-1:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example', - 'Subject': 'AWS Deployment Framework Bootstrap', - 'Message': 'Account 1111111 has now been bootstrapped into banking/production', - 'Timestamp': '3000-03-10T11:08:34.673Z', - 'SignatureVersion': '1', - 'Signature': '1', - 'SigningCertUrl': 'https://sns.eu-central-1.amazonaws.com/SimpleNotificationService', - 'UnsubscribeUrl': 'https://sns.eu-central-1.amazonaws.com', - 'MessageAttributes': {} + "Records": [ + { + "EventSource": "aws:sns", + "EventVersion": "1.0", + "EventSubscriptionArn": f"arn:{PARTITION}:sns:{test_region}:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example", + "Sns": { + "Type": "Notification", + "MessageId": "1", + "TopicArn": f"arn:{PARTITION}:sns:{test_region}:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example", + "Subject": "AWS Deployment Framework Bootstrap", + "Message": "Account 1111111 has now been bootstrapped into banking/production", + "Timestamp": "3000-03-10T11:08:34.673Z", + "SignatureVersion": "1", + "Signature": "1", + "SigningCertUrl": f"https://sns.{test_region}.amazonaws.com/SimpleNotificationService", + "UnsubscribeUrl": f"https://sns.{test_region}.amazonaws.com", + "MessageAttributes": {}, + }, } - }] + ] } stub_failed_pipeline_event = { - 'Records': [{ - 'EventSource': 'aws:sns', - 'EventVersion': '1.0', - 'EventSubscriptionArn': 'arn:aws:sns:eu-central-1:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example', - 'Sns': { - 'Type': 'Notification', - 'MessageId': '1', - 'TopicArn': 'arn:aws:sns:eu-central-1:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example', - 'Subject': None, - 'Message': '{"version":"0","id":"1","detail-type":"CodePipeline Pipeline Execution State Change","source":"aws.codepipeline","account":"2","time":"3000-03-10T11:09:38Z","region":"eu-central-1","resources":["arn:aws:codepipeline:eu-central-1:999999:adf-pipeline-sample-vpc"],"detail":{"pipeline":"adf-pipeline-sample-vpc","execution-id":"1","state":"FAILED","version":9.0}}', - 'Timestamp': '2019-03-10T11:09:49.953Z', - 'SignatureVersion': '1', - 'Signature': '2', - 'SigningCertUrl': 'https://sns.eu-central-1.amazonaws.com/SimpleNotificationService', - 'UnsubscribeUrl': 'https://sns.eu-central-1.amazonaws.com', - 'MessageAttributes': {} + "Records": [ + { + "EventSource": "aws:sns", + "EventVersion": "1.0", + "EventSubscriptionArn": f"arn:{PARTITION}:sns:{test_region}:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example", + "Sns": { + "Type": "Notification", + "MessageId": "1", + "TopicArn": f"arn:{PARTITION}:sns:{test_region}:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example", + "Subject": None, + "Message": '{"version":"0","id":"1","detail-type":"CodePipeline Pipeline Execution State Change","source":"aws.codepipeline","account":"2","time":"3000-03-10T11:09:38Z","region":"{test_region}","resources":["arn:aws:codepipeline:{test_region}:999999:adf-pipeline-sample-vpc"],"detail":{"pipeline":"adf-pipeline-sample-vpc","execution-id":"1","state":"FAILED","version":9.0}}', + "Timestamp": "2019-03-10T11:09:49.953Z", + "SignatureVersion": "1", + "Signature": "2", + "SigningCertUrl": f"https://sns.{test_region}.amazonaws.com/SimpleNotificationService", + "UnsubscribeUrl": f"https://sns.{test_region}.amazonaws.com", + "MessageAttributes": {}, + }, } - }] + ] } stub_failed_bootstrap_event = { - 'Records': [{ - 'EventSource': 'aws:sns', - 'EventVersion': '1.0', - 'EventSubscriptionArn': 'arn:aws:sns:eu-central-1:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example', - 'Sns': { - 'Type': 'Notification', - 'MessageId': '1', - 'TopicArn': 'arn:aws:sns:eu-central-1:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example', - 'Subject': 'Failure - AWS Deployment Framework Bootstrap', - 'Message': '{"Error":"Exception","Cause":"{\\"errorMessage\\": \\"CloudFormation Stack Failed - Account: 111 Region: eu-central-1 Status: ROLLBACK_IN_PROGRESS\\", \\"errorType\\": \\"Exception\\", \\"stackTrace\\": [[\\"/var/task/wait_until_complete.py\\", 99, \\"lambda_handler\\", \\"status))\\"]]}"}', - 'Timestamp': '2019-03-10T11:09:49.953Z', - 'SignatureVersion': '1', - 'Signature': '2', - 'SigningCertUrl': 'https://sns.eu-central-1.amazonaws.com/SimpleNotificationService', - 'UnsubscribeUrl': 'https://sns.eu-central-1.amazonaws.com', - 'MessageAttributes': {} + "Records": [ + { + "EventSource": "aws:sns", + "EventVersion": "1.0", + "EventSubscriptionArn": f"arn:{PARTITION}:sns:{test_region}:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example", + "Sns": { + "Type": "Notification", + "MessageId": "1", + "TopicArn": f"arn:{PARTITION}:sns:{test_region}:9999999:adf-pipeline-sample-vpc-PipelineSNSTopic-example", + "Subject": "Failure - AWS Deployment Framework Bootstrap", + "Message": '{"Error":"Exception","Cause":"{\\"errorMessage\\": \\"CloudFormation Stack Failed - Account: 111 Region: {test_region} Status: ROLLBACK_IN_PROGRESS\\", \\"errorType\\": \\"Exception\\", \\"stackTrace\\": [[\\"/var/task/wait_until_complete.py\\", 99, \\"lambda_handler\\", \\"status))\\"]]}"}', + "Timestamp": "2019-03-10T11:09:49.953Z", + "SignatureVersion": "1", + "Signature": "2", + "SigningCertUrl": f"https://sns.{test_region}.amazonaws.com/SimpleNotificationService", + "UnsubscribeUrl": f"https://sns.{test_region}.amazonaws.com", + "MessageAttributes": {}, + }, } - }] + ] } + +stub_failed_bootstrap_event["Records"][0]["Sns"]["Message"] = re.sub( + r"{test_region}", test_region, stub_failed_bootstrap_event["Records"][0]["Sns"]["Message"] +) diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/stubs/stub_iam.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/stubs/stub_iam.py index f7f994397..116dafebf 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/stubs/stub_iam.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/stubs/stub_iam.py @@ -2,40 +2,48 @@ # SPDX-License-Identifier: MIT-0 # pylint: skip-file +import os +from boto3.session import Session + +REGION = os.getenv("AWS_REGION", "us-east-1") +PARTITION = Session().get_partition_for_region(REGION) + +if PARTITION == "aws": + test_region = "eu-west-1" +else: + test_region = "cn-northwest-1" """ Stubs for testing iam.py """ get_role_policy = { - 'RoleName': 'string', - 'PolicyName': 'string', - 'PolicyDocument': { + "RoleName": "string", + "PolicyName": "string", + "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "KMS", "Effect": "Allow", "Action": ["iam:ChangePassword"], - "Resource": ( - "arn:aws:kms:eu-west-1:111111111111:key/existing_key" - ), + "Resource": (f"arn:{PARTITION}:kms:{test_region}:111111111111:key/existing_key"), }, { "Sid": "S3", "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": [ - "arn:aws:s3:::existing_bucket", - "arn:aws:s3:::existing_bucket/*", + f"arn:{PARTITION}:s3:::existing_bucket", + f"arn:{PARTITION}:s3:::existing_bucket/*", ], }, { "Sid": "AssumeRole", "Effect": "Allow", "Action": "sts:AssumeRole", - "Resource": ['something'], + "Resource": ["something"], }, - ] - } + ], + }, } diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/test_iam_cfn_deploy_role_policy.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/test_iam_cfn_deploy_role_policy.py index be5c1eb66..55d7d0681 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/test_iam_cfn_deploy_role_policy.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/test_iam_cfn_deploy_role_policy.py @@ -4,12 +4,16 @@ # pylint: skip-file import json +import os +from boto3.session import Session from pytest import fixture, raises from mock import call, Mock from copy import deepcopy from .stubs import stub_iam from lambda_codebase.iam_cfn_deploy_role_policy import IAMCfnDeployRolePolicy +REGION = os.getenv("AWS_REGION", "us-east-1") +PARTITION = Session().get_partition_for_region(REGION) @fixture def iam_client(): @@ -114,8 +118,8 @@ def test_grant_access_to_s3_buckets_new_bucket_single_resource(iam_client): ) assert instance.policy_document['Statement'][1]['Resource'] == [ policy_doc_before['Statement'][1]['Resource'], - 'arn:aws:s3:::new_bucket', - 'arn:aws:s3:::new_bucket/*', + f'arn:{PARTITION}:s3:::new_bucket', + f'arn:{PARTITION}:s3:::new_bucket/*', ] assert instance.policy_document['Statement'][2] == ( policy_doc_before['Statement'][2] @@ -149,10 +153,10 @@ def test_grant_access_to_s3_buckets_new_buckets(iam_client): assert instance.policy_document['Statement'][1]['Resource'] == [ policy_doc_before['Statement'][1]['Resource'][0], policy_doc_before['Statement'][1]['Resource'][1], - 'arn:aws:s3:::new_bucket', - 'arn:aws:s3:::new_bucket/*', - 'arn:aws:s3:::another_new_bucket', - 'arn:aws:s3:::another_new_bucket/*', + f'arn:{PARTITION}:s3:::new_bucket', + f'arn:{PARTITION}:s3:::new_bucket/*', + f'arn:{PARTITION}:s3:::another_new_bucket', + f'arn:{PARTITION}:s3:::another_new_bucket/*', ] assert instance.policy_document['Statement'][2] == ( policy_doc_before['Statement'][2] @@ -187,8 +191,8 @@ def test_grant_access_to_kms_keys_new_key_single_resource(iam_client): instance.policy_document['Statement'][1]['Resource'][0] ) policy_doc_before = deepcopy(instance.policy_document) - - new_key_arn = 'arn:aws:kms:eu-west-1:111111111111:key/new_key' + test_region = "cn-north-1" if PARTITION == "aws-cn" else "eu-west-1" + new_key_arn = f'arn:{PARTITION}:kms:{test_region}:111111111111:key/new_key' instance.grant_access_to_kms_keys([ new_key_arn, ]) @@ -226,8 +230,8 @@ def test_grant_access_to_kms_keys_new_keys(iam_client): ] policy_doc_before = deepcopy(instance.policy_document) - new_key_arn_1 = 'arn:aws:kms:eu-west-1:111111111111:key/new_key_no_1' - new_key_arn_2 = 'arn:aws:kms:eu-west-1:111111111111:key/new_key_no_2' + new_key_arn_1 = f'arn:{PARTITION}:kms:eu-west-1:111111111111:key/new_key_no_1' + new_key_arn_2 = f'arn:{PARTITION}:kms:eu-west-1:111111111111:key/new_key_no_2' instance.grant_access_to_kms_keys([ new_key_arn_1, existing_key_arn_1, @@ -350,8 +354,8 @@ def test_update_iam_role_policies_updated(iam_client): policy_doc['Statement'][1]['Resource'] = [ policy_doc['Statement'][1]['Resource'][0], policy_doc['Statement'][1]['Resource'][1], - 'arn:aws:s3:::new_bucket', - 'arn:aws:s3:::new_bucket/*', + f'arn:{PARTITION}:s3:::new_bucket', + f'arn:{PARTITION}:s3:::new_bucket/*', ] policy_doc_json = json.dumps(policy_doc) diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/test_slack.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/test_slack.py index 60c44b81d..1b3415062 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/test_slack.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/lambda_codebase/tests/test_slack.py @@ -2,9 +2,11 @@ # SPDX-License-Identifier: MIT-0 # pylint: skip-file - +import os +from boto3.session import Session from pytest import fixture from ..slack import * + from .stubs.slack import ( stub_approval_event, stub_failed_pipeline_event, @@ -12,6 +14,9 @@ stub_failed_bootstrap_event, ) +REGION = os.getenv("AWS_REGION", "us-east-1") +PARTITION = Session().get_partition_for_region(REGION) + @fixture def stubs(): os.environ["ADF_PIPELINE_PREFIX"] = 'adf-pipeline-' diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/pipeline_management.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/pipeline_management.yml index c523fe9f6..646b40772 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/pipeline_management.yml +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/pipeline_management.yml @@ -667,6 +667,11 @@ Resources: python: 3.12 nodejs: 20 commands: + - | + if [ "${AWS::Region}" = "cn-north-1" ]; then + pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple + npm config set registry https://registry.npmmirror.com + fi - npm install aws-cdk@2.136.0 -g -y --quiet --no-progress - aws s3 cp s3://$SHARED_MODULES_BUCKET/adf-build/ ./adf-build/ --recursive --only-show-errors - pip install -r adf-build/requirements.txt -q -t ./adf-build diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/__init__.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/__init__.py new file mode 100644 index 000000000..014883ae9 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/__init__.py @@ -0,0 +1,4 @@ +# Copyright Amazon.com Inc. or its affiliates. +# SPDX-License-Identifier: MIT-0 + +# pylint: skip-file diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/__init__.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/__init__.py new file mode 100644 index 000000000..014883ae9 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/__init__.py @@ -0,0 +1,4 @@ +# Copyright Amazon.com Inc. or its affiliates. +# SPDX-License-Identifier: MIT-0 + +# pylint: skip-file diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/handler.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/handler.py new file mode 100644 index 000000000..ee8946e1f --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/handler.py @@ -0,0 +1,29 @@ +# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 + +""" +The forward function will forward events to the target SFN. +""" + +import logging +import os + +import boto3 +from stepfunction_helper import Stepfunction + +LOGGER = logging.getLogger(__name__) +LOGGER.setLevel(os.environ.get("ADF_LOG_LEVEL", logging.INFO)) +SFN_ARN = os.getenv("SFN_ARN", "") +sfn_name = SFN_ARN.split(":")[-1] + + +def lambda_handler(event, _): + LOGGER.debug(event) + if "source" in event and event["source"] == "aws.organizations": + session = boto3.session.Session(region_name="cn-north-1") + sfn_instance = Stepfunction(session, LOGGER) + _, state_name = sfn_instance.invoke_sfn_execution( + sfn_arn=SFN_ARN, + input_data=event, + ) + LOGGER.info("Successfully invoke sfn %s with statemachine name %s.", sfn_name, state_name) diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/pytest.ini b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/pytest.ini new file mode 100644 index 000000000..ac18618ea --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/pytest.ini @@ -0,0 +1,5 @@ +# Copyright Amazon.com Inc. or its affiliates. +# SPDX-License-Identifier: MIT-0 + +[pytest] +testpaths = tests diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/requirements.txt b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/requirements.txt new file mode 100644 index 000000000..e69de29bb diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/stepfunction_helper.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/stepfunction_helper.py new file mode 100644 index 000000000..3ea11cfb3 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/stepfunction_helper.py @@ -0,0 +1,49 @@ +# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 + +""" +Helper module for AWS Step Functions operations. +Provides utilities for invoking Step Functions and handling decimal conversions. +""" + +import json +import uuid +from decimal import Decimal + + +def convert_decimals(obj): + if isinstance(obj, Decimal): + return str(obj) + if isinstance(obj, list): + return [convert_decimals(item) for item in obj] + if isinstance(obj, dict): + return {key: convert_decimals(value) for key, value in obj.items()} + return obj + + +class Stepfunction: + """Class to handle Custom Stepfunction methods""" + + def __init__(self, session, logger): + self.logger = logger + self.session = session + + def get_stepfunction_client(self): + return self.session.client("stepfunctions") + + def invoke_sfn_execution(self, sfn_arn, input_data: dict, execution_name=None): + try: + state_machine_arn = sfn_arn + sfn_client = self.get_stepfunction_client() + + if not execution_name: + execution_name = str(uuid.uuid4()) + event_body = json.dumps(convert_decimals(input_data), indent=2) + response = sfn_client.start_execution( + stateMachineArn=state_machine_arn, name=execution_name, input=event_body + ) + except Exception as e: + msg = f"Couldn't invoke stepfunction {sfn_arn}, error: {e}." + self.logger.error(msg) + raise + return response, execution_name diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/__init__.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/__init__.py new file mode 100644 index 000000000..014883ae9 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/__init__.py @@ -0,0 +1,4 @@ +# Copyright Amazon.com Inc. or its affiliates. +# SPDX-License-Identifier: MIT-0 + +# pylint: skip-file diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/conftest.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/conftest.py new file mode 100644 index 000000000..0c665fd24 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/conftest.py @@ -0,0 +1,24 @@ +""" +conftest file +""" +import pytest + +# Global variables for AWS configuration +AWS_REGIONS = { + "china": "cn-north-1", + "us": "us-east-1", +} + +AWS_PARTITIONS = { + "china": "aws-cn", + "us": "aws", +} + + +@pytest.fixture(scope="session") +def aws_settings(): + """Provide AWS settings for the current test environment.""" + # You could determine this from environment or other factors + environment = "china" + + return {"region": AWS_REGIONS[environment], "partition": AWS_PARTITIONS[environment]} diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/test_handler.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/test_handler.py new file mode 100644 index 000000000..304f91b40 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/test_handler.py @@ -0,0 +1,131 @@ +""" +Tests for the AWS Lambda handler that forwards events to Step Functions. +""" +import os +import importlib +from unittest.mock import patch, MagicMock + +import pytest +import handler + +# pylint: disable=redefined-outer-name,no-member,unused-argument + + +@pytest.fixture +def mock_env_variables(aws_settings): + """ + Fixture to mock the SFN_ARN constant in the handler module. + """ + region = aws_settings["region"] + partition = aws_settings["partition"] + sfn_arn = f"arn:{partition}:states:{region}:123456789012:stateMachine:test-sfn" + with patch.object(handler, "SFN_ARN", sfn_arn): + yield sfn_arn + + +@pytest.fixture +def mock_boto3_session(): + """ + Fixture to mock boto3.session.Session and provide access to the mock session object. + """ + # Create a mock session + mock_session = MagicMock() + + # Create a mock client that the session will return + mock_client = MagicMock() + + # Configure the session to return our mock client when requested + mock_session.client.return_value = mock_client + + # Create a mock for the Session constructor + mock_session_constructor = MagicMock(return_value=mock_session) + + # Patch boto3.session.Session to use our mock constructor + with patch("boto3.session.Session", mock_session_constructor): + # Yield both the session constructor and client mocks + yield { + "session_constructor": mock_session_constructor, + "session": mock_session, + "client": mock_client + } + + +@pytest.fixture +def mock_stepfunction(): + """ + Fixture to mock the Stepfunction class in the handler module. + """ + mock_sfn = MagicMock() + mock_sfn.invoke_sfn_execution.return_value = ({"executionArn": "test-arn"}, "test-state-name") + with patch("handler.Stepfunction", return_value=mock_sfn): + yield mock_sfn + + +class TestLambdaHandler: + """Tests for the lambda_handler function.""" + + def test_lambda_handler_with_organizations_event( + self, mock_env_variables, mock_boto3_session, mock_stepfunction, aws_settings + ): + """Test handling of an AWS Organizations event.""" + region = aws_settings["region"] + partition = aws_settings["partition"] + + event = { + "source": "aws.organizations", + "detail-type": "AWS API Call via CloudTrail", + "detail": {"eventName": "CreateAccount"}, + } + + handler.lambda_handler(event, {}) + + # Check that boto3 session was created with correct region + mock_boto3_session["session_constructor"].assert_called_with(region_name=region) + + # Check that Stepfunction was instantiated correctly + handler.Stepfunction.assert_called_once() + + # Check that invoke_sfn_execution was called with correct parameters + expected_arn = f"arn:{partition}:states:{region}:123456789012:stateMachine:test-sfn" + mock_stepfunction.invoke_sfn_execution.assert_called_once_with( + sfn_arn=expected_arn, + input_data=event + ) + + def test_lambda_handler_with_non_organizations_event( + self, mock_env_variables, mock_boto3_session, mock_stepfunction + ): + """Test handling of a non-AWS Organizations event.""" + event = {"source": "aws.ec2", "detail-type": "EC2 Instance State-change Notification"} + + handler.lambda_handler(event, {}) + + # Check that Stepfunction was not instantiated + handler.Stepfunction.assert_not_called() + mock_stepfunction.invoke_sfn_execution.assert_not_called() + + def test_lambda_handler_with_missing_source( + self, + mock_env_variables, + mock_boto3_session, + mock_stepfunction + ): + """Test handling of an event missing the source field.""" + event = {"detail-type": "Some Event", "detail": {}} + + handler.lambda_handler(event, {}) + + # Check that Stepfunction was not instantiated + handler.Stepfunction.assert_not_called() + mock_stepfunction.invoke_sfn_execution.assert_not_called() + + +def test_sfn_name_extraction(aws_settings): + """Test extraction of Step Function name from ARN.""" + sfn_name = "test-sfn" + region = aws_settings["region"] + partition = aws_settings["partition"] + sfn_arn = f"arn:{partition}:states:{region}:123456789012:stateMachine:{sfn_name}" + with patch.dict(os.environ, {"SFN_ARN": sfn_arn}): + importlib.reload(handler) + assert handler.sfn_name == sfn_name diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/test_stepfunction_helper.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/test_stepfunction_helper.py new file mode 100644 index 000000000..53e5288c6 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/china-forward-function/tests/test_stepfunction_helper.py @@ -0,0 +1,157 @@ +""" +test for stepfunction +""" +import json +from decimal import Decimal +from unittest.mock import MagicMock, patch + +import pytest +from stepfunction_helper import Stepfunction, convert_decimals + + +class TestConvertDecimals: + def test_convert_decimal_to_string(self): + assert convert_decimals(Decimal("10.5")) == "10.5" + + def test_convert_list_of_decimals(self): + result = convert_decimals([Decimal("10.5"), Decimal("20.7")]) + assert result == ["10.5", "20.7"] + + def test_convert_dict_with_decimals(self): + data = {"price": Decimal("10.5"), "quantity": Decimal("2")} + result = convert_decimals(data) + assert result == {"price": "10.5", "quantity": "2"} + + def test_convert_nested_structure(self): + data = { + "items": [ + {"price": Decimal("10.5"), "quantity": 2}, + {"price": Decimal("20.7"), "quantity": 1} + ], + "total": Decimal("41.7"), + } + result = convert_decimals(data) + expected = { + "items": [ + {"price": "10.5", "quantity": 2}, + {"price": "20.7", "quantity": 1} + ], + "total": "41.7" + } + assert result == expected + + def test_non_decimal_values_unchanged(self): + data = { + "name": "Test", + "active": True, + "count": 5, + "items": ["a", "b", "c"] + } + assert convert_decimals(data) == data + + +class TestStepfunction: + @pytest.fixture + def mock_session(self): + session = MagicMock() + mock_client = MagicMock() + session.client.return_value = mock_client + return session, mock_client + + @pytest.fixture + def stepfunction(self, mock_session): + session, _ = mock_session + logger = MagicMock() + return Stepfunction(session, logger) + + def test_get_stepfunction_client(self, stepfunction, mock_session): + session, _ = mock_session + stepfunction.get_stepfunction_client() + session.client.assert_called_once_with("stepfunctions") + + def test_invoke_sfn_execution_with_default_name(self, stepfunction, mock_session, aws_settings): + region = aws_settings["region"] + partition = aws_settings["partition"] + _, mock_client = mock_session + execution_arn = ( + f"arn:{partition}:states:{region}:" + f"123456789012:execution:test-sfn:test-execution" + ) + mock_response = { + "executionArn": execution_arn + } + mock_client.start_execution.return_value = mock_response + + with patch("uuid.uuid4", return_value="mocked-uuid"): + response, name = stepfunction.invoke_sfn_execution( + sfn_arn=f"arn:{partition}:states:{region}:123456789012:stateMachine:test-sfn", + input_data={"key": "value"}, + ) + + assert response == mock_response + assert name == "mocked-uuid" + mock_client.start_execution.assert_called_once() + + def test_invoke_sfn_execution_with_custom_name(self, stepfunction, mock_session, aws_settings): + region = aws_settings["region"] + partition = aws_settings["partition"] + _, mock_client = mock_session + mock_response = { + "executionArn": ( + f"arn:{partition}:states:{region}:" + f"123456789012:execution:test-sfn:custom-name" + ) + } + mock_client.start_execution.return_value = mock_response + + response, name = stepfunction.invoke_sfn_execution( + sfn_arn=f"arn:{partition}:states:{region}:123456789012:stateMachine:test-sfn", + input_data={"key": "value"}, + execution_name="custom-name", + ) + + assert response == mock_response + assert name == "custom-name" + mock_client.start_execution.assert_called_once() + + def test_invoke_sfn_execution_with_decimal_data(self, stepfunction, mock_session, aws_settings): + region = aws_settings["region"] + partition = aws_settings["partition"] + _, mock_client = mock_session + mock_response = { + "executionArn": ( + f"arn:{partition}:states:{region}:" + f"123456789012:execution:test-sfn:test-execution" + ) + } + mock_client.start_execution.return_value = mock_response + + input_data = {"amount": Decimal("123.45")} + expected_input = json.dumps({"amount": "123.45"}, indent=2) + + with patch("uuid.uuid4", return_value="mocked-uuid"): + stepfunction.invoke_sfn_execution( + sfn_arn=f"arn:{partition}:states:{region}:123456789012:stateMachine:test-sfn", + input_data=input_data + ) + + mock_client.start_execution.assert_called_once_with( + stateMachineArn=f"arn:{partition}:states:{region}:123456789012:stateMachine:test-sfn", + name="mocked-uuid", + input=expected_input, + ) + + def test_invoke_sfn_execution_exception(self, stepfunction, mock_session, aws_settings): + region = aws_settings["region"] + partition = aws_settings["partition"] + _, mock_client = mock_session + mock_client.start_execution.side_effect = Exception("Test exception") + + with pytest.raises(Exception) as excinfo: + stepfunction.invoke_sfn_execution( + sfn_arn=f"arn:{partition}:states:{region}:123456789012:stateMachine:test-sfn", + input_data={"key": "value"}, + ) + + assert "Test exception" in str(excinfo.value) + stepfunction.logger.error.assert_called_once() diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/cn_northwest_bucket.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/cn_northwest_bucket.yml new file mode 100644 index 000000000..ae0d2bbc5 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/cn_northwest_bucket.yml @@ -0,0 +1,26 @@ +AWSTemplateFormatVersion: "2010-09-09" +Description: >- + ADF CloudFormation Template - Create S3 bucket for cn-northwest-1 + +Parameters: + BucketName: + Type: String + +Resources: + BootstrapArtifactStorageBucket: + DeletionPolicy: Delete + Type: AWS::S3::Bucket + Properties: + BucketName: !Ref BucketName + AccessControl: BucketOwnerFullControl + BucketEncryption: + ServerSideEncryptionConfiguration: + - ServerSideEncryptionByDefault: + SSEAlgorithm: AES256 + VersioningConfiguration: + Status: Enabled + PublicAccessBlockConfiguration: + BlockPublicAcls: true + BlockPublicPolicy: true + IgnorePublicAcls: true + RestrictPublicBuckets: true diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/cn_northwest_deploy.yml b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/cn_northwest_deploy.yml new file mode 100644 index 000000000..733077480 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/cn_northwest_deploy.yml @@ -0,0 +1,83 @@ +# // Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# // SPDX-License-Identifier: Apache-2.0 + +AWSTemplateFormatVersion: '2010-09-09' +Transform: 'AWS::Serverless-2016-10-31' +Description: ADF CloudFormation Stack for deploy extra resources in China cn-northwest-1. + +Parameters: + AccountBootstrapingStateMachineArn: + Type: String + AdfLogLevel: + Type: String + +Globals: + Function: + Architectures: + - arm64 + CodeUri: china-forward-function + Runtime: python3.12 + Timeout: 300 + Tracing: Active + +Resources: + ForwardStateMachineFunctionRole: + Type: "AWS::IAM::Role" + Properties: + Path: "/adf-china-extra/" + AssumeRolePolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Principal: + Service: + - "lambda.amazonaws.com" + Action: + - "sts:AssumeRole" + + ForwardStateMachineFunctionRolePolicy: + Type: AWS::IAM::Policy + Properties: + PolicyName: "forward-state-machine-function-policy" + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Action: "states:StartExecution" + Resource: !Ref AccountBootstrapingStateMachineArn + - Effect: Allow + Action: + - "logs:CreateLogGroup" + - "logs:CreateLogStream" + - "logs:PutLogEvents" + - "xray:PutTelemetryRecords" + - "xray:PutTraceSegments" + - "cloudwatch:PutMetricData" + Resource: "*" + Roles: + - !Ref ForwardStateMachineFunctionRole + + + ForwardStateMachineFunction: + Type: 'AWS::Serverless::Function' + Properties: + Handler: handler.lambda_handler + Description: "ADF Lambda Function - Forward events to statemachine" + Environment: + Variables: + SFN_ARN: !Ref AccountBootstrapingStateMachineArn + ADF_LOG_LEVEL: !Ref AdfLogLevel + FunctionName: ForwardStateMachineFunction + Role: !GetAtt ForwardStateMachineFunctionRole.Arn + Events: + RuleEvent: + Type: EventBridgeRule + Properties: + Pattern: + source: + - aws.organizations + detail: + eventSource: + - organizations.amazonaws.com + eventName: + - MoveAccount diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/create_s3_cn.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/create_s3_cn.py new file mode 100644 index 000000000..f2d2d23d6 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/create_s3_cn.py @@ -0,0 +1,54 @@ +# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: MIT-0 + +""" +Main entry point for create_s3_cn.py execution which +is executed from within AWS CodeBuild in the management account +""" +import os +import sys +import boto3 +from logger import configure_logger +from cloudformation import CloudFormation + +REGION_DEFAULT = os.environ["AWS_REGION"] +MANAGEMENT_ACCOUNT_ID = os.environ["MANAGEMENT_ACCOUNT_ID"] +LOGGER = configure_logger(__name__) +ADF_GLOBAL_BOOTSTRAP_CHINA_BUCKET_STACK_NAME = "adf-regional-base-china-bucket" + +def _create_s3_bucket(bucket_name): + try: + LOGGER.info("Deploy S3 bucket %s...", bucket_name) + extra_deploy_region = "cn-northwest-1" + template_path = "adf-build/china-support/cn_northwest_bucket.yml" + stack_name = ADF_GLOBAL_BOOTSTRAP_CHINA_BUCKET_STACK_NAME + parameters = [ + { + "ParameterKey": "BucketName", + "ParameterValue": bucket_name, + "UsePreviousValue": False, + }, + ] + cloudformation = CloudFormation( + region=extra_deploy_region, + deployment_account_region=extra_deploy_region, + role=boto3, + wait=True, + stack_name=stack_name, + account_id=MANAGEMENT_ACCOUNT_ID, + parameters=parameters, + local_template_path=template_path, + ) + cloudformation.create_stack() + except Exception as error: + LOGGER.error("Failed to process _create_s3_bucket, error:\n %s", error) + sys.exit(1) + + +def main(): + bucket_name = f"adf-china-bootstrap-cn-northwest-1-{MANAGEMENT_ACCOUNT_ID}" + _create_s3_bucket(bucket_name) + + +if __name__ == "__main__": + main() diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/pytest.ini b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/pytest.ini new file mode 100644 index 000000000..ed987b426 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/pytest.ini @@ -0,0 +1,6 @@ +# Copyright Amazon.com Inc. or its affiliates. +# SPDX-License-Identifier: MIT-0 + +[pytest] +testpaths = tests +norecursedirs = adf-build/china-support/china-forward-function diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/tests/__init__.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/tests/__init__.py new file mode 100644 index 000000000..014883ae9 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/tests/__init__.py @@ -0,0 +1,4 @@ +# Copyright Amazon.com Inc. or its affiliates. +# SPDX-License-Identifier: MIT-0 + +# pylint: skip-file diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/tests/test_create_s3_cn.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/tests/test_create_s3_cn.py new file mode 100644 index 000000000..1630426e6 --- /dev/null +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/china-support/tests/test_create_s3_cn.py @@ -0,0 +1,124 @@ +""" +s3 bucket creation script +""" +from unittest.mock import patch, MagicMock + +import pytest +import create_s3_cn + +# pylint: disable=redefined-outer-name,no-member,unused-argument,,protected-access + +@pytest.fixture +def mock_env_vars(): + """ + Fixture to directly patch the module variables that are derived from environment variables. + """ + # Save original values + original_region = create_s3_cn.REGION_DEFAULT + original_account_id = create_s3_cn.MANAGEMENT_ACCOUNT_ID + + # Patch the module variables directly + create_s3_cn.REGION_DEFAULT = "cn-northwest-1" + create_s3_cn.MANAGEMENT_ACCOUNT_ID = "123456789012" + + yield + + # Restore original values + create_s3_cn.REGION_DEFAULT = original_region + create_s3_cn.MANAGEMENT_ACCOUNT_ID = original_account_id + + +@pytest.fixture +def mock_cloudformation(): + """Mock the CloudFormation class.""" + with patch("create_s3_cn.CloudFormation") as mock_cf: + # Configure the mock to return a mock instance + mock_instance = MagicMock() + mock_cf.return_value = mock_instance + yield mock_instance + + +@pytest.fixture +def mock_logger(): + """Mock the logger.""" + with patch("create_s3_cn.LOGGER") as mock_logger: + yield mock_logger + + +def test_create_s3_bucket_china(mock_env_vars, mock_cloudformation, mock_logger): + """Test creating an S3 bucket in China region.""" + # Arrange + bucket_name = "adf-china-bootstrap-cn-northwest-1-123456789012" + + # Act + create_s3_cn._create_s3_bucket(bucket_name) + + # Assert + # Remove this line as it's incorrect - we're not calling the mock directly + # mock_cloudformation.assert_called_once() + + # Instead, check that create_stack was called on the mock instance + mock_cloudformation.create_stack.assert_called_once() + + # Verify CloudFormation was initialized with correct parameters + # This needs to be updated to check the CloudFormation class instantiation + args = create_s3_cn.CloudFormation.call_args + if args: + _, kwargs = args + assert kwargs["region"] == "cn-northwest-1" + assert kwargs["deployment_account_region"] == "cn-northwest-1" + assert kwargs["stack_name"] == "adf-regional-base-china-bucket" + assert kwargs["account_id"] == "123456789012" + + # Verify parameters passed to CloudFormation + parameters = kwargs["parameters"] + assert len(parameters) == 1 + assert parameters[0]["ParameterKey"] == "BucketName" + assert parameters[0]["ParameterValue"] == bucket_name + + # Verify logger was called + mock_logger.info.assert_called_with("Deploy S3 bucket %s...", bucket_name) + + +def test_create_s3_bucket_exception(mock_env_vars, mock_cloudformation, mock_logger): + """Test exception handling when creating an S3 bucket fails.""" + # Arrange + bucket_name = "adf-china-bootstrap-cn-northwest-1-123456789012" + mock_cloudformation.create_stack.side_effect = Exception("Mocked error") + + # Act & Assert + with pytest.raises(SystemExit) as excinfo: + create_s3_cn._create_s3_bucket(bucket_name) + + assert excinfo.value.code == 1 + mock_logger.error.assert_called_once() + assert "Failed to process _create_s3_bucket" in mock_logger.error.call_args[0][0] + + +def test_main_function(mock_env_vars, monkeypatch): + """Test the main function calls _create_s3_bucket with correct bucket name.""" + # Arrange + mock_create_bucket = MagicMock() + monkeypatch.setattr(create_s3_cn, "_create_s3_bucket", mock_create_bucket) + + # Update expected bucket name to match actual implementation + expected_bucket_name = "adf-china-bootstrap-cn-northwest-1-123456789012" + + # Act + create_s3_cn.main() + + # Assert + mock_create_bucket.assert_called_once_with(expected_bucket_name) + + +def test_cloudformation_template_path(mock_env_vars, mock_cloudformation): + """Test the CloudFormation template path is correct.""" + # Arrange + bucket_name = "test-bucket" + + # Act + create_s3_cn._create_s3_bucket(bucket_name) + + # Assert + _, kwargs = create_s3_cn.CloudFormation.call_args + assert kwargs["local_template_path"] == "adf-build/china-support/cn_northwest_bucket.yml" diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/main.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/main.py index a4cd8eddb..0ac92fa37 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/main.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/main.py @@ -24,7 +24,7 @@ from errors import GenericAccountConfigureError, ParameterNotFoundError, Error from sts import STS from s3 import S3 -from partition import get_partition +from partition import get_partition, get_aws_domain from config import Config from organization_policy import OrganizationPolicy @@ -57,6 +57,10 @@ ADF_DEFAULT_SCM_FALLBACK_BRANCH = 'main' ADF_DEFAULT_DEPLOYMENT_MAPS_ALLOW_EMPTY_TARGET = 'disabled' ADF_DEFAULT_ORG_STAGE = "none" +CHINA_PRIMARY_REGION = "cn-north-1" +CHINA_SECONDARY_REGION = "cn-northwest-1" +ADF_REGIONAL_BASE_CHINA_EXTRA_STACK_NAME = "adf-regional-base-china-extra" +CHINA_SECONDARY_REGION_DEPLOY_TEMP = "china-support/cn_northwest_deploy" LOGGER = configure_logger(__name__) @@ -369,9 +373,10 @@ def await_sfn_executions(sfn_client): "Account Management State Machine encountered a failed, " "timed out, or aborted execution. Please look into this problem " "before retrying the bootstrap pipeline. You can navigate to: " - "https://%s.console.aws.amazon.com/states/home" + "https://%s.console.%s/states/home" "?region=%s#/statemachines/view/%s ", REGION_DEFAULT, + get_aws_domain(REGION_DEFAULT), REGION_DEFAULT, ACCOUNT_MANAGEMENT_STATE_MACHINE_ARN, ) @@ -401,10 +406,11 @@ def await_sfn_executions(sfn_client): "Account Bootstrapping State Machine encountered a failed, " "timed out, or aborted execution. Please look into this problem " "before retrying the bootstrap pipeline. You can navigate to: " - "https://%(region)s.console.aws.amazon.com/states/home" + "https://%(region)s.console.%(domain)s/states/home" "?region=%(region)s#/statemachines/view/%(sfn_arn)s", { "region": REGION_DEFAULT, + "domain": get_aws_domain(REGION_DEFAULT), "sfn_arn": ACCOUNT_BOOTSTRAPPING_STATE_MACHINE_ARN, }, ) @@ -459,6 +465,49 @@ def _sfn_execution_exists_with( return False +def _china_region_extra_deploy(): + if REGION_DEFAULT == CHINA_PRIMARY_REGION: + parameters = [ + { + 'ParameterKey': 'AccountBootstrapingStateMachineArn', + 'ParameterValue': ACCOUNT_BOOTSTRAPPING_STATE_MACHINE_ARN, + 'UsePreviousValue': False, + }, + { + 'ParameterKey': 'AdfLogLevel', + 'ParameterValue': ADF_LOG_LEVEL, + 'UsePreviousValue': False, + }, + ] + try: + s3_china = S3( + region=REGION_DEFAULT, + bucket=S3_BUCKET_NAME + ) + cloudformation = CloudFormation( + region=CHINA_SECONDARY_REGION, + deployment_account_region=CHINA_SECONDARY_REGION, + role=boto3, + wait=True, + stack_name=ADF_REGIONAL_BASE_CHINA_EXTRA_STACK_NAME, + s3=s3_china, + s3_key_path='adf-build', + account_id=MANAGEMENT_ACCOUNT_ID, + template_file_prefix=CHINA_SECONDARY_REGION_DEPLOY_TEMP, + parameters=parameters + + ) + cloudformation.create_stack() + except Exception as error: + LOGGER.error( + "China extra stack adf-regional-base-china-extra deployment failed in region " + "%(region)s, please check following error: %(error)s", + { + "region": CHINA_SECONDARY_REGION, + "error": str(error), + }, + ) + sys.exit(2) def main(): # pylint: disable=R0915 LOGGER.info("ADF Version %s", ADF_VERSION) @@ -469,7 +518,8 @@ def main(): # pylint: disable=R0915 policies = OrganizationPolicy() config = Config() cache = Cache() - + # fix the china org service endpoint issue + _china_region_extra_deploy() try: parameter_store = ParameterStore(REGION_DEFAULT, boto3) deployment_account_id = parameter_store.fetch_parameter( diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codebuild.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codebuild.py index b6112c10e..43c758a86 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codebuild.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codebuild.py @@ -27,6 +27,22 @@ ADF_DEFAULT_BUILD_ROLE_NAME = 'adf-codebuild-role' ADF_DEFAULT_BUILD_TIMEOUT = 20 +def get_partition(region_name: str) -> str: + """Given the region, this function will return the appropriate partition. + + :param region_name: The name of the region (us-east-1, us-gov-west-1) + :return: Returns the partition name as a string. + """ + + if region_name.startswith('us-gov'): + return 'aws-us-gov' + if region_name.startswith("cn-north"): + return "aws-cn" + return 'aws' + + +ADF_DEPLOYMENT_PARTITION = get_partition(ADF_DEPLOYMENT_REGION) + class CodeBuild(Construct): # pylint: disable=no-value-for-parameter, too-many-locals @@ -379,8 +395,8 @@ def determine_build_image(codebuild_id, scope, target, map_params): if repository_name: repository_arn = ( - f"arn:aws:ecr:{ADF_DEPLOYMENT_REGION}:" - f"{ADF_DEPLOYMENT_ACCOUNT_ID}:{repository_name}" + f"arn:{ADF_DEPLOYMENT_PARTITION}:ecr:{ADF_DEPLOYMENT_REGION}:" + f"{ADF_DEPLOYMENT_ACCOUNT_ID}:repository/{repository_name}" ) ecr_repo = _ecr.Repository.from_repository_arn( diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codepipeline.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codepipeline.py index 61a77553e..faf29e667 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codepipeline.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/cdk/cdk_constructs/adf_codepipeline.py @@ -39,7 +39,8 @@ def get_partition(region_name: str) -> str: if region_name.startswith('us-gov'): return 'aws-us-gov' - + if region_name.startswith("cn-north"): + return "aws-cn" return 'aws' diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/retrieve_organization_accounts.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/retrieve_organization_accounts.py index 048d60327..dd8e1ff5c 100755 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/retrieve_organization_accounts.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/helpers/retrieve_organization_accounts.py @@ -166,7 +166,8 @@ def _get_partition(region_name: str) -> str: if region_name.startswith("us-gov"): return "aws-us-gov" - + if region_name.startswith("cn-north"): + return "aws-cn" return "aws" diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py index 4111ebb99..07990fc39 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py @@ -16,6 +16,8 @@ from errors import InvalidTemplateError, GenericAccountConfigureError from logger import configure_logger from paginator import paginator +from partition import get_partition + LOGGER = configure_logger(__name__) STACK_TERMINATION_PROTECTION = os.environ.get('TERMINATION_PROTECTION', False) @@ -29,7 +31,8 @@ CFN_UNACCEPTED_CHARS = re.compile(r"[^-a-zA-Z0-9]") ADF_GLOBAL_IAM_STACK_NAME = 'adf-global-base-iam' ADF_GLOBAL_BOOTSTRAP_STACK_NAME = 'adf-global-base-bootstrap' - +ADF_GLOBAL_BOOTSTRAP_CHINA_BUCKET_STACK_NAME = "adf-regional-base-china-bucket" +ADF_GLOBAL_BOOTSTRAP_CHINA_EXTRA_STACK_NAME = "adf-regional-base-china-extra" class StackProperties: clean_stack_status = [ @@ -105,6 +108,7 @@ def __init__( else None ) self.s3 = s3 + self.partition = get_partition(region) self.stack_name = stack_name or self._get_stack_name() def _get_geo_prefix(self): @@ -148,6 +152,10 @@ def _get_valid_stack_names(self): valid_stack_names.append(ADF_GLOBAL_IAM_STACK_NAME) valid_stack_names.append(ADF_GLOBAL_BOOTSTRAP_STACK_NAME) + if self.partition == "aws-cn": + valid_stack_names.append(ADF_GLOBAL_BOOTSTRAP_CHINA_BUCKET_STACK_NAME) + valid_stack_names.append(ADF_GLOBAL_BOOTSTRAP_CHINA_EXTRA_STACK_NAME) + return valid_stack_names @@ -170,6 +178,8 @@ def __init__( parameters=None, account_id=None, # Used for logging visibility role_arn=None, + template_file_prefix=None, # define a custom template file + local_template_path=None, # support local tempplate path ): self.client = role.client( 'cloudformation', @@ -189,6 +199,12 @@ def __init__( s3=s3, s3_key_path=s3_key_path ) + self.template_url_from_template_file_prefix = self.s3.fetch_s3_url( + self._create_template_path(self.s3_key_path, template_file_prefix) + ) \ + if template_file_prefix else None + self.template_url = template_url or self.template_url_from_template_file_prefix + self.local_template_path = local_template_path def validate_template(self): try: @@ -205,6 +221,20 @@ def validate_template(self): f"{self.template_url}: {error}", ) from None + def _handle_template_path( + self, + template_path + ): + try: + # Read the CloudFormation template from a file + with open(template_path, 'r', encoding='utf-8') as template_file: + template_body = template_file.read() + + return template_body + except Exception as error: + LOGGER.error("Process _handle_template_path function error:\n %s", error) + return None + def _wait_if_in_progress(self): status = self.get_stack_status() if status not in StackProperties.in_progress_state_waiters: @@ -358,16 +388,26 @@ def _create_change_set(self): self.stack_name, ) try: - self.template_url = ( - self.template_url - if self.template_url is not None - else self.get_template_url() - ) - if self.template_url: - self.validate_template() + # Add local template ability + cfn_template_map = None + if self.local_template_path: + cfn_template_map = { + "TemplateBody": self._handle_template_path(self.local_template_path) + } + else: + self.template_url = ( + self.template_url + if self.template_url is not None + else self.get_template_url() + ) + if self.template_url: + self.validate_template() + cfn_template_map = { + "TemplateURL": self.template_url + } + if cfn_template_map: change_set_params = { "StackName": self.stack_name, - "TemplateURL": self.template_url, "Parameters": ( self.parameters if self.parameters is not None @@ -384,6 +424,7 @@ def _create_change_set(self): "ChangeSetName": self.stack_name, "ChangeSetType": self._get_change_set_type() } + change_set_params.update(cfn_template_map) if self.role_arn: change_set_params["RoleARN"] = self.role_arn self._clean_up_when_required() diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/partition.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/partition.py index 30050db40..6aabbf5e3 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/partition.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/partition.py @@ -10,39 +10,47 @@ """ from boto3.session import Session +from botocore.exceptions import UnknownRegionError -COMPATIBLE_PARTITIONS = ['aws-us-gov', 'aws'] +class IncompatibleRegionError(Exception): + """Raised in case the regions is not supported.""" -class IncompatiblePartitionError(Exception): - """Raised in case the partition is not supported.""" def get_partition(region_name: str) -> str: """Given the region, this function will return the appropriate partition. - :param region_name: The name of the region (us-east-1, us-gov-west-1) + :param region_name: The name of the region (us-east-1, us-gov-west-1, cn-north-1) + :raises IncompatibleRegionError: If the provided region is not supported. :return: Returns the partition name as a string. """ - partition = Session().get_partition_for_region(region_name) - if partition not in COMPATIBLE_PARTITIONS: - raise IncompatiblePartitionError( - f'The {partition} partition is not supported by this version of ' - 'ADF yet.' - ) - + try: + partition = Session().get_partition_for_region(region_name) + except UnknownRegionError as e: + raise IncompatibleRegionError(f"The region {region_name} is not supported.") from e return partition def get_organization_api_region(region_name: str) -> str: """ Given the current region, it will determine the partition and use - that to return the Organizations API region (us-east-1 or us-gov-west-1) + that to return the Organizations API region (us-east-1 or us-gov-west-1 or cn-northwest-1) - :param region_name: The name of the region (eu-west-1, us-gov-east-1) + :param region_name: The name of the region (eu-west-1, us-gov-east-1 or cn-northwest-1) :return: Returns the AWS Organizations API region to use as a string. """ - if get_partition(region_name) == 'aws-us-gov': - return 'us-gov-west-1' + if get_partition(region_name) == "aws-us-gov": + return "us-gov-west-1" + if get_partition(region_name) == "aws-cn": + return "cn-northwest-1" + return "us-east-1" + - return 'us-east-1' +def get_aws_domain(region_name: str) -> str: + """ + Get AWS domain suffix + """ + if region_name.startswith("cn-north"): + return f"amazonaws.com.{region_name.split('-')[0]}" + return "amazonaws.com" diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/s3.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/s3.py index 2ebd6fbfd..315988a25 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/s3.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/s3.py @@ -9,6 +9,10 @@ from logger import configure_logger +from partition import ( + get_aws_domain, + get_partition +) LOGGER = configure_logger(__name__) @@ -24,7 +28,8 @@ def __init__(self, region, bucket, kms_key_arn=None): self.resource = boto3.resource('s3', region_name=region) self.bucket = bucket self.kms_key_arn = kms_key_arn - + self.domain_suffix = get_aws_domain(region) + self.partition = get_partition(region) @staticmethod def supported_path_styles(): """ @@ -50,8 +55,8 @@ def build_pathing_style(self, style, key): 's3-url' returns: 's3://{bucket}/{key}' 's3-uri' returns: '{bucket}/{key}' 's3-key-only' return: '{key}' - 'path': returns: 'https://{s3-region}.amazonaws.com/{bucket}/{key}' - 'virtual-hosted' returns: 'https://{buycket}.{s3-region}.amazonaws.com/{key}' + 'path': returns: 'https://{s3-region}.{self.domain_suffix}/{bucket}/{key}' + 'virtual-hosted' returns: 'https://{buycket}.{s3-region}.{self.domain_suffix}/{key}' key (str): The object key to include in the path. @@ -70,9 +75,11 @@ def build_pathing_style(self, style, key): s3_region_name = f"s3-{self.region}" if style == 'path': - return f"https://{s3_region_name}.amazonaws.com/{self.bucket}/{key}" + if self.partition == "aws-cn": + return f"https://{self.bucket}.s3.{self.region}.{self.domain_suffix}/{key}" + return f"https://{s3_region_name}.{self.domain_suffix}/{self.bucket}/{key}" if style == 'virtual-hosted': - return f"https://{self.bucket}.{s3_region_name}.amazonaws.com/{key}" + return f"https://{self.bucket}.{s3_region_name}.{self.domain_suffix}/{key}" raise ValueError( f"Unknown upload style syntax: {style}. " @@ -195,8 +202,10 @@ def fetch_s3_url(self, key): s3_object.get() LOGGER.debug('Found Template at: %s', s3_object.key) if self.region == 'us-east-1': - return f"https://s3.amazonaws.com/{self.bucket}/{key}" - return f"https://s3-{self.region}.amazonaws.com/{self.bucket}/{key}" + return f"https://s3.{self.domain_suffix}/{self.bucket}/{key}" + if self.partition == 'aws-cn': + return self.build_pathing_style("path", key) + return f"https://s3-{self.region}.{self.domain_suffix}/{self.bucket}/{key}" except self.client.exceptions.NoSuchKey: # Split the path to remove the last key entry from the string key_level_up = key.split('/') diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_partition.py b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_partition.py index 34af6b5e3..5deb4d934 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_partition.py +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_partition.py @@ -5,39 +5,36 @@ import pytest -from partition import get_partition, IncompatiblePartitionError +from partition import get_partition, IncompatibleRegionError -_us_commercial_regions = [ - 'us-east-1', - 'us-west-1', - 'us-west-2' -] +_us_commercial_regions = ["us-east-1", "us-west-1", "us-west-2"] -_govcloud_regions = [ - 'us-gov-west-1', - 'us-gov-east-1' -] +_govcloud_regions = ["us-gov-west-1", "us-gov-east-1"] -_incompatible_regions = [ - 'cn-north-1', - 'cn-northwest-1' -] +_china_region = ["cn-north-1", "cn-northwest-1"] +_incompatible_regions = ["cp-noexist-1"] -@pytest.mark.parametrize('region', _govcloud_regions) + +@pytest.mark.parametrize("region", _govcloud_regions) def test_partition_govcloud_regions(region): - assert get_partition(region) == 'aws-us-gov' + assert get_partition(region) == "aws-us-gov" -@pytest.mark.parametrize('region', _us_commercial_regions) +@pytest.mark.parametrize("region", _us_commercial_regions) def test_partition_us_commercial_regions(region): - assert get_partition(region) == 'aws' + assert get_partition(region) == "aws" + + +@pytest.mark.parametrize("region", _china_region) +def test_partition_china_regions(region): + assert get_partition(region) == "aws-cn" -@pytest.mark.parametrize('region', _incompatible_regions) -def test_partition_incompatible_regions(region): - with pytest.raises(IncompatiblePartitionError) as excinfo: +@pytest.mark.parametrize("region", _incompatible_regions) +def test_partition_unknown_regions(region): + with pytest.raises(IncompatibleRegionError) as excinfo: get_partition(region) error_message = str(excinfo.value) - assert error_message.find("partition is not supported") >= 0 + assert error_message.find(f"The region {region} is not supported") >= 0 diff --git a/src/lambda_codebase/initial_commit/bootstrap_repository/pytest.ini b/src/lambda_codebase/initial_commit/bootstrap_repository/pytest.ini index 88cf3750c..e44547e53 100644 --- a/src/lambda_codebase/initial_commit/bootstrap_repository/pytest.ini +++ b/src/lambda_codebase/initial_commit/bootstrap_repository/pytest.ini @@ -5,4 +5,4 @@ env = ACCOUNT_ID="123456789012" testpaths = adf-build/tests adf-bootstrap/deployment/lambda_codebase/tests adf-build/shared/python/tests/ -norecursedirs = adf-bootstrap/deployment/lambda_codebase/initial_commit adf-bootstrap/deployment/lambda_codebase/determine_default_branch adf-build/shared +norecursedirs = adf-bootstrap/deployment/lambda_codebase/initial_commit adf-bootstrap/deployment/lambda_codebase/determine_default_branch adf-build/shared adf-build/china-support diff --git a/src/lambda_codebase/jump_role_manager/main.py b/src/lambda_codebase/jump_role_manager/main.py index 92937c9b4..b861b23a6 100644 --- a/src/lambda_codebase/jump_role_manager/main.py +++ b/src/lambda_codebase/jump_role_manager/main.py @@ -354,7 +354,7 @@ def _generate_policy_document(non_bootstrapped_account_ids): "sts:AssumeRole" ], "Resource": [ - f"arn:aws:iam::*:role/{CROSS_ACCOUNT_ACCESS_ROLE_NAME}", + f"arn:{AWS_PARTITION}:iam::*:role/{CROSS_ACCOUNT_ACCESS_ROLE_NAME}", ], "Condition": { "DateLessThan": { diff --git a/src/lambda_codebase/jump_role_manager/tests/test_main.py b/src/lambda_codebase/jump_role_manager/tests/test_main.py index 4a87f4169..0977c8bbd 100644 --- a/src/lambda_codebase/jump_role_manager/tests/test_main.py +++ b/src/lambda_codebase/jump_role_manager/tests/test_main.py @@ -12,6 +12,7 @@ from aws_xray_sdk import global_sdk_config from main import ( + AWS_PARTITION, ADF_JUMP_MANAGED_POLICY_ARN, ADF_TEST_BOOTSTRAP_ROLE_NAME, CROSS_ACCOUNT_ACCESS_ROLE_NAME, @@ -691,7 +692,7 @@ def test_generate_policy_document(get_mock): "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": [ - f"arn:aws:iam::*:role/{CROSS_ACCOUNT_ACCESS_ROLE_NAME}", + f"arn:{AWS_PARTITION}:iam::*:role/{CROSS_ACCOUNT_ACCESS_ROLE_NAME}", ], "Condition": { "DateLessThan": { diff --git a/src/lambda_codebase/organization/main.py b/src/lambda_codebase/organization/main.py index 6b37ae662..097982361 100644 --- a/src/lambda_codebase/organization/main.py +++ b/src/lambda_codebase/organization/main.py @@ -72,7 +72,8 @@ def as_cfn_response(self) -> Tuple[PhysicalResourceId, Data]: def create_(_event: Mapping[str, Any], _context: Any) -> CloudFormationResponse: approved_regions = [ 'us-east-1', - 'us-gov-west-1' + 'us-gov-west-1', + 'cn-north-1' ] region = os.getenv('AWS_REGION') @@ -80,6 +81,7 @@ def create_(_event: Mapping[str, Any], _context: Any) -> CloudFormationResponse: raise ValueError( "Deployment of ADF is only available via the us-east-1 " "and us-gov-west-1 regions." + "and cn-north-1 regions." ) organization_id, created = ensure_organization() organization_root_id = get_organization_root_id() diff --git a/src/template.yml b/src/template.yml index f1118633c..900579085 100644 --- a/src/template.yml +++ b/src/template.yml @@ -3,7 +3,7 @@ AWSTemplateFormatVersion: "2010-09-09" Transform: "AWS::Serverless-2016-10-31" -Description: ADF CloudFormation Initial Base Stack for the Management Account in the us-east-1 region. +Description: ADF CloudFormation Initial Base Stack for the Management Account in the base region of the partition (us-east-1, us-gov-west-1 or cn-north-1). Metadata: AWS::ServerlessRepo::Application: @@ -77,9 +77,9 @@ Parameters: DeploymentAccountMainRegion: Type: String - AllowedPattern: "(us(-gov)?|ap|ca|eu|sa)-(central|(north|south)?(east|west)?)-\\d" + AllowedPattern: "(us(-gov)?|ap|ca|eu|sa|cn)-(central|(north|south)?(east|west)?)-\\d" MinLength: 6 - Description: "Example -> us-east-1, us-gov-west-1, eu-west-1" + Description: "Example -> us-east-1, us-gov-west-1, eu-west-1, cn-north-1" DeploymentAccountTargetRegions: Type: CommaDelimitedList @@ -171,6 +171,13 @@ Conditions: CreateCrossAccountAccessRole: !Equals - !Ref AllowBootstrappingOfManagementAccount - "Yes" + IsChinaMainRegion: + "Fn::Equals": + - !Sub "${AWS::Region}" + - "cn-north-1" + NotChinaMainRegion: + "Fn::Not": + - Condition: IsChinaMainRegion Resources: BootstrapTemplatesBucketPolicy: @@ -1448,6 +1455,7 @@ Resources: AccountOUMoveEventsRule: Type: "AWS::Events::Rule" + Condition: NotChinaMainRegion Properties: Name: "adf-account-bootstrapping-account-ou-move" Description: >- @@ -1568,6 +1576,59 @@ Resources: - !GetAtt "BootstrapArtifactStorageBucket.Arn" - !Sub "${BootstrapArtifactStorageBucket.Arn}/*" + CodeBuildPolicyChina: + Type: "AWS::IAM::ManagedPolicy" + Condition: IsChinaMainRegion + Properties: + Description: "Policy to allow codebuild to perform actions for china region" + PolicyDocument: + Version: "2012-10-17" + Statement: + - Effect: "Allow" + Action: + - "lambda:*" + - "iam:PassRole" + - "iam:CreatePolicy" + - "iam:CreateRole" + - "iam:DeleteRole" + - "iam:DeleteRolePolicy" + - "iam:GetRole" + - "iam:PutRolePolicy" + - "iam:UpdateAssumeRolePolicy" + - "iam:TagRole" + - "iam:DeleteRole" + + Resource: + - !Sub "arn:${AWS::Partition}:lambda:*:${AWS::AccountId}:function:ForwardStateMachineFunction" + - !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/adf-china-extra/adf-regional-base-*" + - !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/adf-regional-base-*" + - Effect: "Allow" + Action: + - "s3:*" + Resource: + - !Sub "arn:${AWS::Partition}:s3:::adf-china-bootstrap-cn-northwest-1-${AWS::AccountId}" + - !Sub "arn:${AWS::Partition}:s3:::adf-china-bootstrap-cn-northwest-1-${AWS::AccountId}/*" + - Effect: "Allow" + Action: + - "events:*" + Resource: + - !Sub "arn:${AWS::Partition}:events:cn-northwest-1:${AWS::AccountId}:rule/adf-regional-base-china*" + - Effect: "Allow" + Action: + - "cloudformation:*" + Resource: + - !Sub "arn:${AWS::Partition}:cloudformation:cn-northwest-1:${AWS::AccountId}:stack/adf-regional-base-china-bucket/*" + - !Sub "arn:${AWS::Partition}:cloudformation:cn-northwest-1:${AWS::AccountId}:stack/adf-regional-base-china-extra/*" + - !Sub "arn:${AWS::Partition}:cloudformation:cn-northwest-1:aws:transform/Serverless-2016-10-31" + - Effect: "Allow" + Action: + - "cloudformation:ValidateTemplate" + - "cloudformation:List*" + - "cloudformation:Describe*" + Resource: "*" + Roles: + - !Ref BootstrapCodeBuildRole + OrganizationsReadonlyRole: Type: AWS::IAM::Role DependsOn: CleanupLegacyStacks @@ -1601,9 +1662,7 @@ Resources: - organizations:ListAccounts - organizations:ListAccountsForParent - organizations:DescribeAccount - - organizations:ListOrganizationalUnitsForParent - - organizations:ListRoots - - organizations:ListChildren + - organizations:List* - tag:GetResources Resource: "*" @@ -1663,6 +1722,10 @@ Resources: python: 3.12 pre_build: commands: + - | + if [ "${AWS_REGION}" = "cn-north-1" ]; then + pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple + fi - >- pip install -r requirements-dev.txt @@ -1691,6 +1754,12 @@ Resources: --s3-prefix adf-bootstrap/deployment --s3-bucket $SHARED_MODULES_BUCKET - python adf-build/store_config.py + - | + if [ "${AWS_REGION}" = "cn-north-1" ]; then + python adf-build/china-support/create_s3_cn.py + sam build -t adf-build/china-support/cn_northwest_deploy.yml --region cn-northwest-1 + sam package --output-template-file adf-build/china-support/cn_northwest_deploy.yml --s3-prefix adf-bootstrap --s3-bucket adf-china-bootstrap-cn-northwest-1-${MANAGEMENT_ACCOUNT_ID} --region cn-northwest-1 + fi # Shared Modules to be used with AWS CodeBuild: - >- aws s3 sync @@ -2382,7 +2451,7 @@ Resources: - ssm:GetParameters - ssm:GetParameter Resource: - - !Sub "arn:${AWS::Partition}:ssm:*:${AWS::AccountId}:parameter/adf/*" + - !Sub "arn:${AWS::Partition}:ssm:*:${AWS::AccountId}:parameter/*" - Effect: Allow Action: - iam:CreateRole @@ -2426,6 +2495,11 @@ Resources: - iam:UntagRole Resource: - !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:role/adf-update-cross-account-access-role" + - Effect: "Allow" + Action: + - iam:* + Resource: + - "*" Roles: - !Ref OrganizationsRole