AWS Secret Manager env vars pull and merge (#48)

* Initial commit. Missing order definition.

* Made tf file not fail if empty value passed

* Added option to define Repo and AWS env files

* Updated README

Author: LeoDiazL

README.md

@@ -8,20 +8,52 @@ The action will copy this repo to the VM and then run `docker-compose up`.
 Your app needs a `Dockerfile` and a `docker-compose.yaml` file.
 
 > For more details on setting up Docker and Docker Compose, check out Bitovi's Academy Course: [Learn Docker](https://www.bitovi.com/academy/learn-docker.html)
+>
+## Environment variables
+
+For envirnoment variables in your app, you can provide a `repo_env` file in your repo, a `.env` file in GitHub Secrets named `DOT_ENV`, or an AWS Secret. Then hook it up in your `docker-compose.yaml` file like:
 
-For envirnoment variables in your app, provide a `.env` file in GitHub Secrets named `DOT_ENV` and hook it up in your `docker-compose.yaml` file like:
 ```
 version: '3.9'
 services:
   app:
     env_file: .env
 ```
 
+These environment variables are merged to the .env file quoted in the following order:
+ - Terraform passed env vars ( This is not optional nor customizable )
+ - Repository checked-in env vars - repo_env file as default. (KEY=VALUE style)
+ - Github Secret - Create a secret named DOT_ENV - (KEY=VALUE style)
+ - AWS Secret - JSON style like '{"key":"value"}'
+
 ## Example usage
 
 Create `.github/workflow/deploy.yaml` with the following to build on push.
 
+### Basic example
+```yaml
+name: Basic deploy
+on:
+  push:
+    branches: [ main ]
+
+jobs:
+  EC2-Deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - id: deploy
+        uses: bitovi/[email protected]
+        with:
+          aws_access_key_id: ${{ secrets.AWS_ACCESS_KEY_ID_PTO}}
+          aws_secret_access_key: ${{ secrets.AWS_SECRET_ACCESS_KEY_PTO}}
+          aws_default_region: us-east-1
+          dot_env: ${{ secrets.DOT_ENV }}
+```
+
+### Advanced example
+
 ```yaml
+name: Advanced deploy
 on:
   push:
     branches: [ main ]
@@ -79,15 +111,17 @@ The following inputs can be used as `step.with` keys
 | `domain_name` | String | Define the root domain name for the application. e.g. bitovi.com' |
 | `sub_domain` | String | Define the sub-domain part of the URL. Defaults to `${org}-${repo}-{branch}` |
 | `tf_state_bucket` | String | AWS S3 bucket to use for Terraform state. Will be deleted if stack_destroy set to true |
-| `dot_env` | String | `.env` file to be used with the app |
+| `repo_env` | String | `.env` file containing environment variables to be used with the app. Name defaults to `repo_env`. Check **SEnvironment variables** note |
+| `dot_env` | String | `.env` file to be used with the app. This is the name of the [Github secret](https://docs.github.com/es/actions/security-guides/encrypted-secrets). Check **SEnvironment variables** note |
+| `aws_secret_env` | String | Secret name to pull environment variables from AWS Secret Manager. Check **SEnvironment variables** note |
 | `app_port` | String | port to expose for the app |
 | `lb_port` | String | Load balancer listening port. Defaults to 80 if NO FQDN provided, 443 if FQDN provided |
 | `lb_healthcheck` | String | Load balancer health check string. Defaults to HTTP:app_port |
 | `ec2_instance_profile` | String | The AWS IAM instance profile to use for the EC2 instance. Default is `${GITHUB_ORG_NAME}-${GITHUB_REPO_NAME}-${GITHUB_BRANCH_NAME}` |
 | `ec2_instance_type` | String | The AWS IAM instance type to use. Default is t2.small. See [this list](https://aws.amazon.com/ec2/instance-types/) for reference |
-| `stack_destroy` | String | Set to `true` to destroy the stack. Default is `""` - Will delete the tf_state_bucket after destroy. |
+| `stack_destroy` | String | Set to `true` to destroy the stack. Default is `""` - Will delete the tf_state and elb_logs bucket after the destroy action runs. |
 | `aws_resource_identifier` | String | Set to override the AWS resource identifier for the deployment.  Defaults to `${org}-{repo}-{branch}`.  Use with destroy to destroy specific resources. |
-| `app_directory` | String | Relative path for the directory of the app (i.e. where `Dockerfile` and `docker-compose.yaml` files are located). This is the directory that is copied to the EC2 instance.  Default is the root of the repo. |
+| `app_directory` | String | Relative path for the directory of the app (i.e. where `Dockerfile` and `docker-compose.yaml` files are located). This is the directory that is copied to the EC2 instance. Default is the root of the repo. |
 | `additional_tags` | JSON | Add additional tags to the terraform [default tags](https://www.hashicorp.com/blog/default-tags-in-the-terraform-aws-provider), any tags put here will be added to all provisioned resources.|
 
 ## Note about resource identifiers
@@ -101,7 +135,7 @@ For some specific resources, we have a 32 characters limit. If the identifier le
 
 ### S3 buckets naming
 
-Buckets name can be made of up to 63 characters. If the length allows us to add -tf-state, we will do so. If not, a simple -tf will be added.
+Buckets names can be made of up to 63 characters. If the length allows us to add -tf-state, we will do so. If not, a simple -tf will be added.
 
 ## Made with BitOps
 [BitOps](https://bitops.sh) allows you to define Infrastructure-as-Code for multiple tools in a central place.  This action uses a BitOps [Operations Repository](https://bitops.sh/operations-repo-structure/) to set up the necessary Terraform and Ansible to create infrastructure and deploy to it.

action.yaml

@@ -24,9 +24,17 @@ inputs:
   tf_state_bucket:
     description: 'AWS S3 bucket to use for Terraform state. Defaults to `${org}-${repo}-{branch}-tf-state`. Will be deleted with stack_destroy.'
     required: false
+  repo_env:
+    description: 'File containing environment variables to be used with the app'
+    required: false
+    default: 'repo_env'
   dot_env:
     description: '`.env` file to be used with the app'
     required: false
+  aws_secret_env:
+    description: 'Secret name to pull env variables from AWS Secret Manager'
+    required: false
+    default: ''
   app_port:
     description: 'Port to expose for the app'
     required: false
@@ -81,7 +89,9 @@ runs:
         AWS_SESSION_TOKEN: ${{ inputs.aws_session_token }}
         AWS_DEFAULT_REGION: ${{ inputs.aws_default_region }}
         TF_STATE_BUCKET: ${{ inputs.tf_state_bucket }}
+        REPO_ENV: ${{ inputs.repo_env }}
         DOT_ENV: ${{ inputs.dot_env }}
+        AWS_SECRET_ENV: ${{ inputs.aws_secret_env }}
         APP_PORT: ${{ inputs.app_port }}
         LB_PORT: ${{ inputs.lb_port }}
         LB_HEALTHCHECK: ${{ inputs.lb_healthcheck }}

operations/_scripts/generate/generate_app_repo.sh

@@ -16,3 +16,7 @@ if [ -n "$APP_DIRECTORY" ]; then
 fi
 
 cp -rf "$TARGET_PATH"/* "${GITHUB_ACTION_PATH}/operations/deployment/ansible/app/${GITHUB_REPO_NAME}/"
+
+echo "Copying checked in env file from repo to Ansible deployment path"
+
+cp "$TARGET_PATH/$REPO_ENV" "${GITHUB_ACTION_PATH}/operations/deployment/ansible/repo.env"

operations/_scripts/generate/generate_dot_env.sh

@@ -5,4 +5,4 @@ set -e
 
 echo "In generate_dot_env.sh"
 
-echo "$DOT_ENV" >> "${GITHUB_ACTION_PATH}/operations/deployment/ansible/app.env"
+echo "$DOT_ENV" >> "${GITHUB_ACTION_PATH}/operations/deployment/ansible/ghs.env"

operations/_scripts/generate/generate_tf_vars.sh

@@ -67,6 +67,8 @@ aws_resource_identifier = \"${GITHUB_IDENTIFIER}\"
 
 aws_resource_identifier_supershort = \"${GITHUB_IDENTIFIER_SS}\"
 
+aws_secret_env = \"${AWS_SECRET_ENV}\"
+
 sub_domain_name = \"${SUB_DOMAIN}\"
 
 domain_name = \"${DOMAIN_NAME}\"

operations/deployment/ansible/bitops.before-deploy.d/merge-tf-env.sh

@@ -1,21 +1,76 @@
 #!/bin/bash
+
+set -e
+
+
 echo "BitOps Ansible before script: Merge Terraform Enviornment Variables..."
 
-# dotenv file
+# Merging order
+order=tf,repo,ghs,aws
+
+# Ansible dotenv file -> The final destination of all
 DOTENV_FILE="${BITOPS_ENVROOT}/ansible/app.env"
 
-# tf dotenv file
+# TF dotenv file
 TF_DOTENV_FILE="${BITOPS_ENVROOT}/terraform/tf.env"
 
-echo "cat DOTENV_FILE ($DOTENV_FILE)"
-cat $DOTENV_FILE
+# Repo env file
+REPO_ENV_FILE="${BITOPS_ENVROOT}/ansible/repo.env"
+
+# GH Secrets env file
+GHS_ENV_FILE="${BITOPS_ENVROOT}/ansible/ghs.env"
+
+# TF AWS dotenv file
+AWS_SECRET_FILE="${BITOPS_ENVROOT}/terraform/aws.env"
+
+# Make sure app.env is empty, if not, delete it and create one.
+
+if [ -f $DOTENV_FILE ]; then 
+  rm -rf $DOTENV_FILE
+fi 
+touch $DOTENV_FILE
+
+# Function to merge to destination
 
+function merge {
+  if [ -s $1 ]; then
+    echo "Merging $2 envs"
+    cat $1 >> $DOTENV_FILE
+  else
+    echo "Nothing to merge from $2"
+  fi
+}
 
-TEMP_DOTENV_FILE="${BITOPS_ENVROOT}/ansible/tmp.env"
+# Function to be called based on the input string
+function process {
+  case $1 in
+    aws)
+      # Code to be executed for option1
+      merge $AWS_SECRET_FILE "AWS Secret"
+      ;;
+    repo)
+      # Code to be executed for option2
+      merge $REPO_ENV_FILE "checked-in"
+      ;;
+    ghs)
+      # Code to be executed for option3
+      merge $GHS_ENV_FILE "GH-Secret"
+      ;;
+    tf)
+      # Code to be executed for option3
+      merge $TF_DOTENV_FILE "Terraform"
+      ;;
+    *)
+      # Code to be executed if no matching option is found
+      echo "Invalid option"
+      ;;
+  esac
+}
 
-cat $TF_DOTENV_FILE >> $TEMP_DOTENV_FILE
-cat $DOTENV_FILE >> $TEMP_DOTENV_FILE
-cat $TEMP_DOTENV_FILE > $DOTENV_FILE
+# Read the input string and split it into an array
+IFS=',' read -r -a options <<< "$order"
 
-echo "cat TEMP_DOTENV_FILE ($TEMP_DOTENV_FILE)"
-cat $TEMP_DOTENV_FILE
+# Loop through the array and call the process function for each element
+for option in "${options[@]}"; do
+  process "$option"
+done

operations/deployment/terraform/tf-secretdotenv.tf

@@ -0,0 +1,19 @@
+data "aws_secretsmanager_secret_version" "env_secret" {
+  count = local.secret_provided ? 1 : 0
+  secret_id = var.aws_secret_env
+}
+
+locals {
+  s3_secret_raw    = local.secret_provided ? nonsensitive(jsondecode(data.aws_secretsmanager_secret_version.env_secret[0].secret_string)) : {}
+  s3_secret_string = local.secret_provided ? join("\n", [for k, v in local.s3_secret_raw : "${k}=\"${v}\""]) : ""
+}
+
+resource "local_file" "tf-secretdotenv" {
+  count = local.secret_provided ? 1 : 0
+  filename = format("%s/%s", abspath(path.root), "aws.env")
+  content  = local.secret_provided ? "${local.s3_secret_string}\n" : ""
+}
+
+locals {
+  secret_provided = ( var.aws_secret_env != "" ? true : false )
+}

operations/deployment/terraform/variables.tf

@@ -78,6 +78,11 @@ variable "aws_resource_identifier_supershort" {
   description = "Identifier to use for AWS resources (defaults to GITHUB_ORG-GITHUB_REPO-GITHUB_BRANCH) shortened to 30 chars"
 }
 
+variable "aws_secret_env" {
+  type = string
+  description = "Secret name to pull env variables from AWS Secret Manager"
+}
+
 variable "sub_domain_name" {
   type = string
   description = "Subdomain name for DNS record"