ecs-deepagent

Disclaimer: Personal learning project — not production-ready. Use at your own risk.

A demo system that runs an AI agent in a per-user isolated container on AWS ECS Fargate, with a WebSocket gateway for real-time streaming. Each agent invocation gets its own container with an isolated filesystem and OS. A warm pool of pre-started containers prevents cold-start latency from being visible to users.

---

Architecture

User
 |
 | WebSocket (ws://alb-dns/ws)
 |
[ALB]
 |
[Gateway — ECS Fargate, long-lived]
 |  - Accepts user WebSocket connection
 |  - Calls ECS RunTask to spin up a worker container
 |  - Polls ECS DescribeTasks until worker is RUNNING (gets private IP)
 |  - Polls worker /health until app is ready
 |  - Opens WebSocket to worker
 |  - Proxies messages bidirectionally between user and worker
 |
[Worker — ECS Fargate, ephemeral, one per user]
   - Private subnet, not internet-accessible
   - Accepts WebSocket connection from gateway only
   - Runs the agent, streams responses back
   - Destroyed when user disconnects

Key design decisions

• Per-user containers — each user gets their own Fargate task with an isolated filesystem and OS, providing a sandbox for agent execution
• Fargate — chosen over EC2 to avoid managing host instances; Fargate uses Firecracker microVMs for stronger isolation than plain Docker
• awsvpc networking — each task gets its own ENI and private IP, enabling direct task-to-task communication within the VPC
• Private workers — worker tasks run in a private subnet: no inbound internet access (the security group only allows traffic from the gateway), but outbound access via a NAT gateway so the agent can reach external APIs
• VPC endpoints — AWS API traffic from the private subnet (ECR pulls, Secrets Manager, CloudWatch Logs, S3 layers) goes through Interface/Gateway endpoints rather than the NAT gateway — cheaper, lower-latency, and keeps that traffic on AWS's private network

---

Connecting

The gateway is the only public entry point, sitting behind the ALB on HTTP port 80 (ws://, no TLS).

Find the URL

terraform -chdir=infra/app-infra output -raw gateway_url
# ws://ecs-deepagent-gateway-123456789.us-east-1.elb.amazonaws.com/ws

Or fetch the ALB DNS name directly:

aws elbv2 describe-load-balancers \
  --names ecs-deepagent-gateway \
  --region us-east-1 \
  --query 'LoadBalancers[0].DNSName' \
  --output text

Health check

curl http://<alb-dns>/health
# {"status":"ok"}

A 200 means the gateway is up. The warm pool may still be seeding on first boot — the first connection can fall back to a cold start (~30s) if so.

Message protocol

Client → gateway: plain text, one user prompt per message.

Gateway → client: a stream of JSON frames, one per agent event:

type	Fields	Meaning
TEXT	content	Agent reply text
TOOL_CALL	tool_name, args	A tool was invoked; args is the full argument dict
TOOL_RESULT	tool_name, result_content	Tool completed successfully
TOOL_ERROR	tool_name, error	Tool failed

TOOL_CALL is emitted immediately when the tool is triggered, before it finishes — so the client can show progress while the tool runs. TOOL_RESULT / TOOL_ERROR follow once it completes.

Connect

Using websocat:

websocat ws://<alb-dns>/ws
what's the weather in London?

Or wscat:

wscat -c ws://<alb-dns>/ws
> what's the weather in London?

Or Python:

import asyncio, json, websockets

async def main():
    async with websockets.connect("ws://<alb-dns>/ws") as ws:
        await ws.send("What's the weather in London?")
        async for raw in ws:
            msg = json.loads(raw)
            match msg["type"]:
                case "TEXT":
                    print(msg["content"])
                case "TOOL_CALL":
                    print(f"→ {msg['tool_name']}({msg['args']})")
                case "TOOL_RESULT":
                    print(f"← {msg['result_content']}")
                case "TOOL_ERROR":
                    print(f"✗ {msg['error']}")

asyncio.run(main())

When you connect, the gateway hands you a pre-warmed worker from the pool (or cold-starts one if the pool is empty), proxies the socket both ways, and StopTasks the worker when you disconnect. There is no authentication on the endpoint — anyone who can reach the ALB can connect.

---

Warm pool

The problem

A freshly-launched Fargate task takes ~30s to become usable across four serial steps:

1. Task provisioning — Fargate finds capacity (~2–5s, fixed)
2. ENI attachment — awsvpc assigns a network interface (~5–10s, fixed)
3. Image pull — Fargate caches nothing between tasks
4. App boot — the worker process imports its dependencies

Steps 1 and 2 are unavoidable on Fargate. The only way to serve requests in under a second is for the worker to be running and healthy before the user arrives.

The solution

The gateway maintains an in-memory pool of pre-started, health-checked workers. When a user connects, it pops a ready worker off the pool — the 30s was paid earlier, off the user's clock. The pool is an asyncio.Queue of (task_arn, ip) pairs, sized to IDLE_TASK_POOL_SIZE. The queue is single-threaded so get() is atomic — two simultaneous connections can never receive the same worker.

Lifecycle

Phase	What happens
Seed (startup)	The lifespan hook warms IDLE_TASK_POOL_SIZE workers concurrently and blocks until the pool is full before serving traffic.
Take (per connection)	/ws pops a warm worker and proxies to it immediately.
Reconcile (background loop)	Every TOPUP_INTERVAL seconds a loop tops the pool back up to N, replacing workers consumed by connections or lost to failures.
Drain (shutdown)	All producers are stopped and awaited, then every pooled worker is StopTasked.

Each worker is health-checked before it enters the queue, so the queue only ever holds ready-to-use workers.

Failure handling

• Bounded warm-up — create_warm_worker wraps the RUNNING + health wait in asyncio.wait_for(..., timeout=WORKER_READY_TIMEOUT). A worker that never becomes healthy is StopTasked and the attempt fails cleanly.
• Empty pool — if the pool is exhausted when a user connects, the endpoint falls back to a cold start. If that also fails, the WebSocket closes with 1013 ("try again later").
• Reconcile as safety net — reactive top-ups are fire-and-forget; the reconcile loop is what guarantees the pool eventually returns to N regardless of individual failures. It subtracts in-flight spawns from its deficit so it never double-spawns.
• Race-free shutdown — the lifespan cancels and awaits every background producer before draining the queue, so no producer can put() after the drain has passed it.

Tunable constants (gateway.py)

Constant	Meaning
IDLE_TASK_POOL_SIZE	Warm workers kept ready. Higher = absorbs bigger bursts, costs more idle compute.
WORKER_READY_TIMEOUT	How long a warm-up may take before it's abandoned.
QUEUE_CONSUMER_TIMEOUT	How long a connection waits for a pooled worker before falling back to a cold start.
TOPUP_INTERVAL	How often the reconcile loop checks and refills the pool.

The pool is topped up only by the reconcile loop, so after a burst it can stay depleted for up to one TOPUP_INTERVAL before connections stop falling back to cold start. A reactive refill on the take path would tighten this, at the cost of more in-flight spawns.

---

Components

Component	Description
src/gateway.py	FastAPI app — WebSocket entry point, warm-pool management, ECS orchestration, message proxy
src/worker.py	FastAPI app — WebSocket endpoint, agent execution
src/config.py	Config loaded from environment variables
infra/bootstrap/	Terraform to create S3 bucket for remote state
infra/app-infra/	Terraform for all AWS infrastructure
Dockerfile	Single image used for both gateway and worker (different CMD per task definition)

---

Infrastructure

All managed with Terraform. Resources created:

• VPC with public and private subnets across two AZs
• Internet Gateway + route tables for the public subnet
• NAT Gateway — outbound internet for the private subnet, no inbound
• VPC Endpoints for ECR (Interface), Secrets Manager (Interface), CloudWatch Logs (Interface), and S3 (Gateway)
• ECR repository for the Docker image
• ECS Cluster (Fargate)
• ECS Task Definitions — gateway and worker (same image, different CMD)
• ECS Service — keeps one gateway task running
• ALB — stable DNS entry point for the gateway
• IAM roles — execution role (ECR pull) and task role (ECS RunTask/DescribeTasks/StopTask)
• Security Groups — ALB → gateway → worker, locked down by layer

---

CI/CD

GitHub Actions on push to main:

1. infra job — runs terraform apply on infra/app-infra/
2. app job — lints, formats, tests, builds Docker image, pushes to ECR, forces new ECS deployment

A separate build-bootstrap.yml workflow (manual trigger) provisions the Terraform state bucket.

---

Known gaps

• Orphaned workers on hard kill — graceful shutdown drain can't run when ECS SIGKILLs the gateway on deploy. Fix: tag workers on RunTask, then list and reap tagged tasks on gateway startup.
• Dead pooled workers — a worker can crash while sitting idle in the queue; the next user is handed a dead IP. Fix: treat a failed websockets.connect as a dead worker, discard it, and pop/spawn another.
• Blocking boto3 on the event loop — RunTask / DescribeTasks / StopTask are synchronous calls; under load they stutter WebSocket proxying. Fix: wrap each in asyncio.to_thread(...).
• No reactive refill — the pool stays depleted for up to one TOPUP_INTERVAL after a burst (see note above).
• Region is hardcoded to us-east-1 in several places.
• No authentication on the WebSocket endpoint.