Setup monitoring of ECS nodes

[hellais]

Currently we don't have observability into the container host of the ECS cluster. Moreover we are only able to scrape aggregate metrics from the services that are behind the balancer, which means we end up with the metrics "flapping".

Ideally we would have a way of scraping metrics for the container host, but also the per-servicer docker containers.

In summary we would like to collect two classes of metrics:

• Host container metrics (the ec2 nodes that run docker and we deploy docker containers to), using node_exporter
• Docker container application metrics, which are exposed using the instrumentator and we would like to scrape independently per each host container

[hellais]

Leaving here some links for things to look at:

• https://github.com/santosh-at-github/ecs-monitoring-with-prometheus-grafana
• https://github.com/google/cadvisor
• https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-ECS.html

[LDiazN]

I think this might be the way: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#ec2_sd_config

The problem I'm seeing right now is that the monitor server is not in AWS, so we have some issues with the connection between that server and the ec2 instances:

1. We have to set up IAM credentials for the server, as mentioned in the link above
2. The prometheus server needs a way to reach the ec2 instances (not the load balancer), but they're probably not open to internet traffic (and I don't think they should), what can we do about this?

[LDiazN]

To solve 2), I had some talk with @hellais , and we had the idea of using nginx as proxy. You send scrape requests to the nginx proxy and it knows to which host to send it based on the path of the request.

For example, if you want to request metrics from the node foo:

1. The prometheus server would send the request: get nginx.proxy.com/foo/metrics/
2. nginx would then perform the get foo/metrics using proxy_pass and forward it to the prometheus server

After testing this with a docker compose with nginx and two go servers, I think it's possible using the following nginx configuration:

events { }

http {
    server {
        listen 80;

        location ~ /([a-zA-Z0-9_\.]+)/([a-zA-Z0-9_]*) {
            proxy_pass http://$1:8080;
        }
    }
}

As a side note, you can't use the hostname of the docker service you want to reach, you have to use the local ip address