AWS Automation Projects — Terraform + Lambda

A project that deploys multiple personal automations on AWS, fully managed with Terraform and organized into independent modules.

New to Terraform or AWS? Check the companion tutorial: TUTORIAL.md

Project Overview

1. Project Architecture

This repository holds two independent automations under a single Terraform deployment, organized as modules:

Module 1: Real Estate Scraper

A scheduled EventBridge rule triggers a Lambda every hour. The Lambda scrapes latorrecase.it, compares results against DynamoDB, and sends an email via SNS when new listings appear.

EventBridge (schedule) ──> Lambda (Python)
                              ├── HTTP GET latorrecase.it
                              ├── DynamoDB (compare known listings)
                              └── SNS ──> email (if new listings found)

Module 2: YouTube Summarizer

An on-demand Lambda that receives a YouTube URL, analyzes the video using Google Gemini (with OpenAI GPT-4o as fallback), and sends a structured summary to your inbox via SES.

You invoke ──> Lambda (Python)
                  ├── SSM Parameter Store (read API keys)
                  ├── Gemini API (native video analysis)
                  │   └── fallback: YouTube Transcript API + OpenAI GPT-4o
                  └── SES ──> email with summary

2. Your PC vs AWS — Who Does What?

YOUR PC                                    AWS (Ireland, eu-west-1)
───────                                    ────────────────────────

terraform/                                 ┌─────────────────────────────────┐
  *.tf files ── terraform apply ──────>    │ Creates all resources:          │
                                           │  DynamoDB table                 │
lambda/                                    │  SNS topic + email subscription │
  *.py files ── ZIP + upload ─────────>    │  IAM roles (permissions)        │
                                           │  Lambda functions               │
deploy.ps1                                 │  EventBridge schedule           │
  (automates both steps above)             │  SSM parameters                 │
                                           │  CloudWatch log groups          │
                                           └─────────────────────────────────┘

At this point your PC                      Every hour, WITHOUT your PC:
can be TURNED OFF.
                                           EventBridge ──> Scraper Lambda
                                                            ├── latorrecase.it
                                                            ├── DynamoDB
                                                            └── SNS ──> email

                                           On demand (when you invoke it):

                                           YouTube Summarizer Lambda
                                             ├── SSM (read API key)
                                             ├── Gemini/OpenAI API
                                             └── SES ──> email

Key takeaway: Your PC only contains instruction files (.tf and .py). After terraform apply, everything runs on Amazon's servers. Your PC can be turned off.

3. Execution Flows

Real Estate Scraper — What happens every hour

EventBridge fires the scheduled rule and invokes the Lambda
Lambda starts, reads TARGET_URL from its environment variables
Lambda sends an HTTP GET to latorrecase.it and parses the HTML with BeautifulSoup
Lambda calls DynamoDB Scan to read all previously known listing IDs
Lambda compares the two sets: new = found on website − already in database
If there are new listings:
- DynamoDB BatchWriteItem to save the new listings
- SNS Publish to send an email notification
Lambda exits. AWS destroys the execution environment

YouTube Summarizer — What happens when you invoke it

You invoke the Lambda (via CLI or AWS Console) with {"youtube_url": "..."}
Lambda starts, reads the SSM parameter paths from its environment variables
Lambda calls SSM GetParameter to fetch the Gemini API key (encrypted)
Phase 1 — Gemini: Lambda sends the YouTube URL to Google Gemini API via Part.from_uri(). Gemini analyzes the video natively and returns a summary
If Gemini fails → Phase 2 — OpenAI: Lambda calls SSM for the OpenAI key, fetches the video transcript via youtube_transcript_api, and sends the text to GPT-4o for summarization
Lambda calls SES SendEmail to deliver the summary to your inbox
Lambda exits

4. File Structure

aws-automation-projects/
├── deploy.ps1                         # One-click build & deploy script
├── help.ps1                           # Interactive command reference (run .\help.ps1)
├── README.md                          # This guide
├── TUTORIAL.md                        # Full AWS & Terraform deep-dive lesson
│
├── lambda/                            # Lambda Python code
│   ├── real_estate_scraper/
│   │   ├── scraper.py                 # Web scraping + DynamoDB + SNS logic
│   │   └── requirements.txt           # Python dependencies (requests, beautifulsoup4)
│   └── youtube_summarizer/
│       ├── lambda_function.py         # Gemini/OpenAI video analysis + SES email
│       └── requirements.txt           # Python dependencies (google-genai, openai, youtube-transcript-api)
│
└── terraform/                         # Infrastructure as Code
    ├── main.tf                        # Provider config + module calls
    ├── variables.tf                   # All configurable variables (root level)
    ├── outputs.tf                     # Values printed after deployment
    ├── terraform.tfvars.example       # Template — copy to terraform.tfvars and fill in
    └── modules/                       # Each module is an independent unit
        ├── real_estate_scraper/       # lambda.tf, dynamodb.tf, sns.tf, eventbridge.tf
        │   ├── variables.tf           #   Module inputs (project_name, email, schedule, etc.)
        │   └── outputs.tf             #   Module outputs (function name, ARNs)
        └── youtube_summarizer/        # main.tf (SSM + Lambda), iam.tf
            └── variables.tf           #   Module inputs (project_name, SES emails)

Why three variables.tf files? Each module is isolated — like a function with its own parameters. The root variables.tf declares what the user can configure. Each module's variables.tf declares what that module accepts. The root main.tf wires them together.

Getting Started

5. Prerequisites and Installation

1. AWS Account

Create one at aws.amazon.com. The Free Tier covers this project for 12 months.

2. AWS CLI

winget install Amazon.AWSCLI
aws --version
# Expected: aws-cli/2.x.x ...

3. Terraform

winget install HashiCorp.Terraform
terraform -version
# Expected: Terraform v1.x.x

4. Python 3.11+

python --version
# Expected: Python 3.11.x or higher

6. AWS Credentials Configuration

Create an IAM User (recommended for beginners)

Go to the AWS IAM console
Create a new user with programmatic access
Attach the AdministratorAccess policy (for development/testing only!)
Save the Access Key ID and Secret Access Key

Configure the CLI

aws configure

It will prompt you for:

AWS Access Key ID [None]: AKIA...YOUR_ACCESS_KEY
AWS Secret Access Key [None]: wJal...YOUR_SECRET_KEY
Default region name [None]: eu-west-1
Default output format [None]: json

Verify it works:

aws sts get-caller-identity
# You should see your Account ID and ARN

7. Step-by-Step Deployment

Step 1: Configure Variables

cd terraform
Copy-Item terraform.tfvars.example terraform.tfvars

Edit terraform.tfvars and fill in your emails:

notification_email          = "your@email.com"
youtube_ses_sender_email    = "your@email.com"
youtube_ses_recipient_email = "your@email.com"

Step 2: Deploy Everything

From the project root:

.\deploy.ps1

What this does: Packages both Lambdas (pip install + zip with Linux-compatible binaries), then runs terraform init, terraform validate, and terraform apply. Type yes when prompted.

Step 3: Post-Deploy Setup

Confirm SNS subscription: Check your inbox for an email from AWS titled "AWS Notification - Subscription Confirmation" and click the link. Without this, the scraper cannot send you emails.

Verify SES email identity: Required for the YouTube Summarizer to send emails (SES sandbox mode requires both sender and recipient to be verified).

aws ses verify-email-identity --email-address "your@email.com"
# Then click the verification link in your inbox

Store your Gemini API key (get one free at aistudio.google.com):

aws ssm put-parameter `
  --name "/scraper/youtube_summarizer/gemini_api_key" `
  --value "YOUR_GEMINI_API_KEY" --type "SecureString" --overwrite

(Optional) Store your OpenAI API key (used as fallback if Gemini fails):

aws ssm put-parameter `
  --name "/scraper/youtube_summarizer/openai_api_key" `
  --value "YOUR_OPENAI_API_KEY" --type "SecureString" --overwrite

8. Testing the Lambdas

Test the Real Estate Scraper

aws lambda invoke --function-name scraper-function `
  --payload '{}' --cli-binary-format raw-in-base64-out response.json
Get-Content response.json

Test the YouTube Summarizer

aws lambda invoke --function-name scraper-youtube-summarizer-v2 `
  --payload '{"youtube_url": "https://www.youtube.com/watch?v=VIDEO_ID"}' `
  --cli-binary-format raw-in-base64-out response.json
Get-Content response.json

View Logs

# Scraper logs (last hour)
aws logs tail /aws/lambda/scraper-function --since 1h

# YouTube Summarizer logs (last hour)
aws logs tail /aws/lambda/scraper-youtube-summarizer-v2 --since 1h

# Follow logs in real time (CTRL+C to exit)
aws logs tail /aws/lambda/scraper-youtube-summarizer-v2 --follow

9. Scheduling

The Real Estate Scraper is scheduled automatically via EventBridge. To change the frequency, edit terraform.tfvars:

schedule_expression = "rate(1 hour)"          # every hour
schedule_expression = "rate(30 minutes)"      # every 30 minutes
schedule_expression = "cron(0 9 * * ? *)"     # daily at 9:00 AM UTC
schedule_expression = "cron(0 9,18 * * ? *)"  # daily at 9:00 AM and 6:00 PM UTC

Then apply the change:

cd terraform
terraform apply

Useful EventBridge commands:

# Check the current schedule
aws events describe-rule --name scraper-schedule

# Temporarily disable (without destroying the rule)
aws events disable-rule --name scraper-schedule

# Re-enable
aws events enable-rule --name scraper-schedule

10. Tearing Down the Infrastructure

Delete ALL resources with a single command:

cd terraform
terraform destroy

Type yes to confirm. Terraform deletes everything in the correct dependency order — no forgotten resources running up costs. This is one of IaC's greatest advantages.

Reference

11. Estimated Costs

With the AWS Free Tier active, this project costs virtually nothing:

Service	Free Tier	Estimated use/month
Lambda	1M requests + 400,000 GB-sec	~720 requests
DynamoDB	25 GB storage + 25 WCU/RCU	< 1 MB, few ops
EventBridge	All rules are free	1 rule
SNS	1,000 free emails	~720 emails (max)
SES	62,000 emails/month (from Lambda)	< 100 emails
SSM	Standard parameters are free	2 parameters
CloudWatch	5 GB logs, 10 metrics, 10 alarms	< 100 MB logs

After the Free Tier expires: approximately $0.50 – $1.00 / month.

12. Troubleshooting

"Email address is not verified" (SES)

Your AWS account is in SES sandbox mode. Both sender and recipient must be verified:

aws ses verify-email-identity --email-address "your@email.com"

Then click the link in the confirmation email.

"No module named 'requests'" or similar import errors

The Lambda package was not built correctly. Re-run:

.\deploy.ps1

"Subscription is pending confirmation" (SNS)

Check your inbox (including spam) for the AWS subscription confirmation email and click the link.

"Access Denied" on DynamoDB, SNS, SSM, or SES

The Lambda's IAM role may be missing permissions. Verify that terraform apply completed successfully and check the IAM policy in the relevant module's .tf files.

The Scraper finds no listings

The website may have changed its HTML structure. Check CloudWatch logs:

aws logs tail /aws/lambda/scraper-function --since 1h

YouTube Summarizer returns "Could not generate summary"

Check that the Gemini API key is set in SSM (not the dummy placeholder)
Check CloudWatch logs for the specific error from Gemini or OpenAI
Some videos may have transcripts disabled or be geo-restricted

`terraform plan` shows errors

Make sure you have run terraform init and that AWS credentials are configured (aws configure).

Quick command reference

Run the interactive help script:

.\help.ps1

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
lambda		lambda
terraform		terraform
.gitignore		.gitignore
README.md		README.md
TUTORIAL.md		TUTORIAL.md
deploy.ps1		deploy.ps1
help.ps1		help.ps1

Folders and files

Latest commit

History

Repository files navigation