Data Ingestion Service

Serverless data ingestion system built with AWS Lambda, DynamoDB, and S3. Provides a REST API for storing arbitrary JSON data and automated weekly summary generation.

• REST API endpoint for ingesting JSON data
• Automatic data persistence to DynamoDB with UUID and timestamp
• Weekly scheduled summary generation
• Automated S3 storage for summary reports
• Infrastructure fully defined as code with AWS CDK

Architecture

Setup

Quick Start

1. Configure AWS credentials:

aws configure

2. Run setup script:

./setup.sh

Deployment

Deploy to AWS

./deploy.sh

This script will:

1. Validate AWS credentials
2. Bootstrap CDK (if needed)
3. Synthesize template
4. Deploy all resources

Validate Deployment

Run automated validation tests:

./test-deployment.sh

This script will:

1. Retrieve CDK stack outputs
2. POST test data to the API
3. Verify data in DynamoDB
4. Trigger summary Lambda manually
5. Validate S3 summary file creation
6. Test API error handling

Usage

Ingest Data

Send JSON data to the API endpoint:

curl -X POST ${endpoint}/data \
  -H "Content-Type: application/json" \
  -d '{
    "sensor_id": "sensor-001",
    "temperature": 23.5,
    "humidity": 60,
    "timestamp": "2025-10-29T10:00:00Z"
  }'

Response:

{
  "success": true,
  "message": "Data saved successfully",
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": 1730217600000
}

Verify Data in DynamoDB

aws dynamodb scan --table-name data-table --limit 5

Trigger Summary Manually

aws lambda invoke \
  --function-name cce-summary-function \
  --payload '{}' \
  response.json

cat response.json

Summary format:

{
  "periodStart": "2025-10-22T00:00:00Z",
  "periodEnd": "2025-10-29T00:00:00Z",
  "totalItems": 150,
  "generatedAt": 1730217600000
}

API Documentation

POST /data

Ingest JSON data into the system.

Response (Success):

{
  "success": true,
  "message": "Data saved successfully",
  "id": "uuid-string",
  "timestamp": 1730217600000
}

Response (Error):

{
  "success": false,
  "error": "Error message"
}

CI/CD

CI-CD.md

Cleanup

To remove all AWS resources

./destroy.sh

Current implementation is for demonstration. For production deployment, consider:

• Add API authentication
• Implement request throttling and rate limiting
• Add input validation
• Enable DynamoDB encryption if required
• Enable AWS WAF for API Gateway
• Implement proper logging and monitoring
• Use AWS Secrets Manager for sensitive configuration