Initial commit

.gitignore

@@ -0,0 +1,12 @@
+*.js
+!jest.config.js
+!test/*.js
+*.d.ts
+node_modules
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out
+
+# Parcel default cache directory
+.parcel-cache

.npmignore

@@ -0,0 +1,6 @@
+*.ts
+!*.d.ts
+
+# CDK asset staging directory
+.cdk.staging
+cdk.out

README.md

@@ -1,17 +1,151 @@
-## My Project
+# Audit Service Sample <!-- omit in toc -->
 
-TODO: Fill this README out!
+This repository contains a sample audit service where its components are decoupled through the use of [Amazon EventBridge](https://docs.aws.amazon.com/eventbridge/latest/userguide/what-is-amazon-eventbridge.html). When an audit event happens, it is put into a custom bus where different rules evaluate if their corresponding targets should be notified of the event based on the patterns their are interested to.
 
-Be sure to:
+It uses the [AWS CDK framework](https://docs.aws.amazon.com/cdk/latest/guide/home.html) for defining the required infrastructure while relying on [CDK Pipelines](https://docs.aws.amazon.com/cdk/latest/guide/cdk_pipeline.html) (still on [Developer Preview](https://docs.aws.amazon.com/cdk/api/latest/docs/pipelines-readme.html) as of December 2020) for implementing CI/CD.
 
-* Change the title in this README
-* Edit your repository description on GitHub
+**Table of contents:**
 
-## Security
+- [Architecture](#architecture)
+  - [Schema](#schema)
+- [Requirements](#requirements)
+- [Deployment](#deployment)
+- [CI/CD](#cicd)
+- [Clean up](#clean-up)
 
-See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
+## Architecture
+
+Custom bus defines three rules:
+
+- Rule for audit events (`detail-type='Object State Change'` - See [Schema](#schema) section for more details). These events contain information about changes that are performed over entities in different systems. They are sent to an [AWS Step Function State Machine](https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html) that:
+  - Stores the actual entity into an [Amazon S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#BasicsBucket) through an [AWS Lambda Function](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html).
+  - Saves metadata about the actual change (author, timestamp or type of the operation among others) into an [Amazon DynamoDB table](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html#HowItWorks.CoreComponents.TablesItemsAttributes) so we can later perform queries to obtain all actions that were taken by a given user or see historical versions of a particular entity.
+- Publish a message to an [Amazon SNS topic](https://docs.aws.amazon.com/sns/latest/dg/welcome.html) when an entity is deleted (`detail-type='Object State Change'` and `detail.operation='delete'`). This allows to have other processes notified upon removal actions, such as sending an email to an administrator.
+- Log all events going through the bus into an [Amazon CloudWatch Log Group](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatchLogsConcepts.html) for debugging purposes (could be only for dev/test environments).
+
+![Architecture](images/architecture.png)
+
+NOTE: Although CDK Pipelines offer great help to deploy the same stack to multiple [enviroments](https://docs.aws.amazon.com/cdk/latest/guide/environments.html) (account - region pair), for demo purposes and thus simplicity, this repository assumes only two environments (`staging` and `production`) which live within the same AWS Account.
+
+Given the above, resources are prefixed with the environment they belong to which is just author's preference to keep things tidy in such scenario.
+
+### Schema
+
+Events going through the bus must comply with the following schema:
+
+```javascript
+{
+  "detail-type": "Object State Change",
+  "source": "<system that generated event>",
+  "detail": {
+    "entity-type": "<entity type in source system>",
+    "entity-id": "<entity id in source system>",
+    "operation": "insert | update | delete",
+    "ts": "<timestamp when change happened (ms)>",
+    "author": "<user who triggered the change>",
+    "data": {
+      // entity body
+    }
+  }
+}
+```
+
+There are some sample events in the `events` folder that you can use to manually test resources provisioned by this project:
+
+```sh
+aws events put-events --entries file://./events/book-insert.json
+```
+
+## Requirements
+
+- Node.js 12 or above
+- npm 6 or above
+- [AWS CDK Toolkit](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) 1.77.0 or above.
+- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html). You must have run `aws configure` to set up your terminal.
+
+## Deployment
+
+To set up your environment for the first time, run the following commands:
+
+```sh
+# create the required parameters in your AWS account so
+# AWS CodePipeline can connect to Github and pull source code
+aws ssm put-parameter --name github_username --value <YOUR_GITHUB_USERNAME>
+aws secretsmanager create-secret --name github_token
+aws secretsmanager put-secret-value --secret-id github_token --secret-string '{"github_token": "<YOUR_GITHUB_TOKEN>"}'
+
+# install aws cdk
+npm install -g aws-cdk
+
+# install dependencies
+npm i
+
+# bootstrap cdk for the target accont and region
+# (profile may be different based on your terminal setup)
+export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text)
+export AWS_REGION=$(aws configure get region)
 
-## License
+cdk bootstrap aws://$ACCOUNT_ID/$AWS_REGION
+```
+
+Then, run the following to deploy the app:
+
+```sh
+npm run build
+cdk deploy
+```
+
+After confirming that there will be security-related changes, the provisioning of the CloudFormation will start. After successful completion, pipeline is automatically triggered. You can go to [AWS CodePipeline](https://docs.aws.amazon.com/codepipeline/latest/userguide/welcome.html) to monitor its progress.
+
+## CI/CD
+
+Pipeline generated by the CDK application contains the following stages:
+
+- **Source**
+  - Connects to Github using SSM Parameter and Secret generated as part of [Deployment](#deployment) steps
+- **Build**
+  - Through npm scripts, we provide a custom command for building our application (`build.sh`):
+    - Compile our application from Typescript to ES2018
+    - Run [fine-grained assertions](https://docs.aws.amazon.com/cdk/latest/guide/testing.html) to ensure resources generated by our CDK stack meet our requirements
+    - As our application contains an AWS Lambda Function, we equally build it and run unit tests to validate behaviour
+  - Similarly, synthesize our application once we have removed non-production npm modules (`synth.sh`)
+- **Update Pipeline**
+  - Stage provided by CDK pipelines which modifies the pipeline in case its definition has changed as part of the last changes
+- **Assets**
+  - Stage provided by CDK pipelines that prepares and publishes our lambda function assets to Amazon S3.
+- **Staging**
+  - Deploy CDK application to staging environment
+  - Once provisioned, it runs a suite of end-to-end tests in this environment using [AWS CodeBuild](https://docs.aws.amazon.com/codebuild/latest/userguide/welcome.html). Their goal is to ensure that given a input (an event on the bus in our case), we obtain the expected output (logs in CloudWatch, object in S3 bucket or new row in DynamoDB)
+- **Production**
+  - Before doing the actual deployment to this environment, we add a *Manual Approval* action after having generated the corresponding [Amazon CloudFormation Change set](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets.html), so we can validate that those changes is what we expect to deploy to production
+
+NOTE: You can find further information about CDK Pipelines in this [post](https://aws.amazon.com/blogs/developer/cdk-pipelines-continuous-delivery-for-aws-cdk-applications/) of our [AWS Developer Blog](https://aws.amazon.com/blogs/developer/) where some areas are covered in more detail such as deploying the app to different accounts and regions.
+
+## Clean up
+
+Run the following command in your terminal to destroy the pipeline stack:
+
+```bash
+cdk destroy 
+```
+
+Then, remove stacks for each environment through:
+
+```bash
+aws cloudformation delete-stack --stack--name Staging-AuditService
+aws cloudformation delete-stack --stack--name Production-AuditService
+```
+
+Be aware some resources will not be removed as part of the deletion operation and they will have to be deleted manually: DynamoDB table, S3 bucket and CloudWatch log group.
+
+## Want to contribute? <!-- omit in toc -->
+
+Check our [contribution guidelines](CONTRIBUTING.md) before submitting a pull request. Any contribution must be done to the `develop` branch.
+
+## Security <!-- omit in toc -->
+
+See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more information.
 
-This library is licensed under the MIT-0 License. See the LICENSE file.
+## License <!-- omit in toc -->
 
+This library is licensed under the MIT-0 License. See the [LICENSE](LICENSE.md) file.

bin/audit-service-sample.ts

@@ -0,0 +1,18 @@
+#!/usr/bin/env node
+
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: MIT-0
+
+import 'source-map-support/register';
+import { App } from '@aws-cdk/core';
+import { PipelineStack } from '../lib/pipeline-stack';
+
+const app = new App();
+new PipelineStack(app, 'AuditServicePipeline', {
+    env: {
+        account: process.env.CDK_DEFAULT_ACCOUNT,
+        region: process.env.CDK_DEFAULT_REGION
+    }
+});
+
+app.synth();

build.sh

@@ -0,0 +1,38 @@
+#!/bin/bash
+
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: MIT-0
+
+set -euo pipefail
+
+echo "********************************************"
+echo "* Executing build for Audit Service sample *"
+echo "********************************************"
+echo ;
+
+echo "Building main project..."
+tsc
+echo -e "Done.\n"
+
+echo "Executing unit tests..."
+npm test
+echo -e "Done.\n"
+
+echo "Building AWS Lambda functions"
+cd lib/lambda/
+
+cd save-to-s3
+echo "- save-to-s3"
+echo "  - Installing dependencies..."
+npm ci
+echo "  - Done."
+echo "  - Compiling Typescript files..."
+npm run build
+echo "  - Done."
+echo "  - Executing unit tests..."
+npm test
+
+
+echo "********************************************"
+echo "* Success                                  *"
+echo "********************************************"

cdk.json

@@ -0,0 +1,9 @@
+{
+  "app": "npx ts-node bin/audit-service-sample.ts",
+  "context": {
+    "@aws-cdk/core:enableStackNameDuplicates": "true",
+    "aws-cdk:enableDiffNoFail": "true",
+    "@aws-cdk/core:stackRelativeExports": "true",
+    "@aws-cdk/core:newStyleStackSynthesis": true
+  }
+}

events/book-delete.json

@@ -0,0 +1,6 @@
+[{
+    "Source": "custom.books-api",
+    "DetailType": "Object State Change",
+    "Detail": "{\"entity-type\":\"book\",\"entity-id\":\"c56b33f1-0e42-45d3-b895-4014963cefd7\",\"operation\":\"delete\",\"author\":\"[email protected]\",\"ts\":\"1603493121000\"}",
+    "EventBusName": "staging-audit-event-bus"
+}]

events/book-insert.json

@@ -0,0 +1,6 @@
+[{
+    "Source": "custom.books-api",
+    "DetailType": "Object State Change",
+    "Detail": "{\"entity-type\":\"book\",\"entity-id\":\"ebad79e3-abe1-450a-b505-6fd0aa466cbf\",\"operation\":\"insert\",\"author\":\"[email protected]\",\"ts\":\"1603042460000\",\"data\": {\"name\":\"bob\"}}",
+    "EventBusName": "staging-audit-event-bus"
+}]

events/book-update.json

@@ -0,0 +1,6 @@
+[{
+    "Source": "custom.books-api",
+    "DetailType": "Object State Change",
+    "Detail": "{\"entity-type\":\"book\",\"entity-id\":\"03650f4e-dfd5-4584-97f7-44af7d2465b3\",\"operation\":\"update\",\"author\":\"[email protected]\",\"ts\":\"1603293123000\",\"data\": {\"name\":\"sponge bob\"}}",
+    "EventBusName": "staging-audit-event-bus"
+}]

jest.config.js

@@ -0,0 +1,15 @@
+module.exports = {
+  roots: [
+    '<rootDir>/lib/'
+  ],
+  testMatch: [
+    '**/*.spec.ts'
+  ],
+  testPathIgnorePatterns: [
+    "/node_modules/",
+    "/lib/lambda"
+  ],
+  transform: {
+    '^.+\\.tsx?$': 'ts-jest'
+  }
+};

lib/audit-service-sample-stack.spec.ts

@@ -0,0 +1,84 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: MIT-0
+
+import { Stack } from "@aws-cdk/core";
+import { AuditServiceStack } from "./audit-service-sample-stack";
+
+import '@aws-cdk/assert/jest';
+
+let stack: Stack;
+
+beforeEach(() => {
+  stack = new AuditServiceStack(stack, 'stateMachine', {
+    logicalEnv: 'test'
+  });
+});
+
+test('should create a CloudWatch log group', () => {
+  expect(stack).toHaveResource('AWS::Logs::LogGroup', {
+    LogGroupName: '/aws/events/test-audit-events',
+    RetentionInDays: 1
+  });
+});
+
+test('should create a SNS topic', () => {
+  expect(stack).toHaveResource('AWS::SNS::Topic', {
+    TopicName: 'test-deleted-entities'
+  });
+});
+
+test('should create an EventBridge bus', () => {
+  expect(stack).toHaveResource('AWS::Events::EventBus', {
+    Name: 'test-audit-event-bus'
+  });
+});
+
+test('should create rule for audit events going to Step Function state machine', () => {
+  expect(stack).toHaveResourceLike('AWS::Events::Rule', {
+    Name: 'test-audit-events-rule',
+    Description: 'Rule matching audit events',
+    EventBusName: {Ref: 'AuditEventBus4CA9BCB2'},
+    EventPattern: {
+      'detail-type': ['Object State Change']
+    },
+    Targets: [{
+      Arn: {Ref: 'StateMachineTargetLogAuditEventEE46E9C7'}
+    }]
+  });
+});
+
+test('should create rule for all events going to CloudWatch log group', () => {
+  expect(stack).toHaveResourceLike('AWS::Events::Rule', {
+    Name: 'test-all-events-rule',
+    Description: 'Rule matching all events',
+    EventBusName: {Ref: 'AuditEventBus4CA9BCB2'},
+    EventPattern: {
+      source: [{prefix: ''}]
+    },
+    Targets: [{
+      Id: 'test-all-events-cw-logs'
+    }]
+  });
+});
+
+test('should create rule for deleted entities going to SNS topic', () => {
+  expect(stack).toHaveResourceLike('AWS::Events::Rule', {
+    Name: 'test-deleted-entities-rule',
+    Description: 'Rule matching audit events for delete operations',
+    EventBusName: {Ref: 'AuditEventBus4CA9BCB2'},
+    EventPattern: {
+      'detail-type': ['Object State Change'],
+      detail: {operation: ['delete']}
+    },
+    Targets: [{
+      Arn: {Ref: 'DeletedEntitiesTopic8CC38689'},
+      InputTransformer: {
+        InputPathsMap: {
+          'detail-entity-id': '$.detail.entity-id',
+          "detail-author": '$.detail.author'
+        },
+        InputTemplate: '\"Entity with id <detail-entity-id> has been deleted by <detail-author>\"'
+      }
+    }]
+  });
+});