diff --git a/README.md b/README.md index a636cab..fbe1e9b 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ This repository grew organically based on tests that I needed to perform in orde # 🥢 Tools used +- Custom Collector images from [my collection of distributions](https://github.com/jpkrohling/otelcol-distributions). Anywhere a custom image is being used, you can use contrib if you prefer. - [`telemetrygen`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen#installing) is used a lot in this repository to send telemetry data to our Collector instance - [`otelcol-contrib`](https://github.com/open-telemetry/opentelemetry-collector-releases/releases) is used as well, both in binary format for local examples, and as container image in examples using Kubernetes - [`k3d`](https://k3d.io) is used in the Kubernetes recipes, in order to create a local Kubernetes cluster diff --git a/client-side-load-balancing/README.md b/client-side-load-balancing/README.md index fb24c0d..0f1bb68 100644 --- a/client-side-load-balancing/README.md +++ b/client-side-load-balancing/README.md @@ -2,6 +2,8 @@ This recipe shows how to configure a Collector to load balance its exporter requests to another layer of Collectors. While the solution itself is very simple, comprising of two lines of configuration, verifying that it works is trickier: we want to start with a fixed number of server instances, likely higher than 1, and scale up. If it works, we'll see that all instances will have equivalent load over time, with the rate of requests going down as more server collectors are available. +For a successful load-balancing, we need clients that are able to retrieve a list of available endpoints, and we need the server to force connections to be closed at a specific amount of time. While the client configuration can be reused as is, the server configuration in this example requires some attention before being used in production: make sure to adjust the max connection age to a value that makes sense to you. + We are discarding the telemetry data that we are generating, as we are only interested in assessing this behavior by observing the Collector's own metrics. ## 🧄 Ingredients @@ -51,9 +53,9 @@ We are discarding the telemetry data that we are generating, as we are only inte telemetrygen traces --rate 100 --duration 10m --otlp-insecure --otlp-attributes='recipe="client-side-load-balancing"' ``` -9. After a few minutes, update the `otelcol-server.yaml` to have 10 replicas instead of the original 5. +9. After a few minutes, update the `otelcol-server.yaml` to have 10 replicas instead of the original 5. -10. Open your Grafana instance, go to Explore, and select the metrics datasource ("...-prom") and a run a query like: `rate(receiver_accepted_spans_total[$__rate_interval])` +10. Open your Grafana instance, go to Explore, and select the metrics datasource ("...-prom") and a run a query like: `rate(receiver_accepted_spans_total[$__rate_interval])`. You can also remove the `rate` function to see the totals, which should show the first collectors having similar numbers among themselves, while the newer collectors would have lower numbers, but still equivalent among each other. ## 😋 Executed last time with these versions diff --git a/client-side-load-balancing/otelcol-server.yaml b/client-side-load-balancing/otelcol-server.yaml index 11e14e6..f03315e 100644 --- a/client-side-load-balancing/otelcol-server.yaml +++ b/client-side-load-balancing/otelcol-server.yaml @@ -3,13 +3,16 @@ kind: OpenTelemetryCollector metadata: name: otelcol-server spec: - image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.101.0 + image: ghcr.io/jpkrohling/otelcol-distributions/otelcol-otlp:0.101.5 replicas: 5 config: receivers: otlp: protocols: - grpc: {} + grpc: + keepalive: + server_parameters: + max_connection_age: 1m exporters: nop: {} diff --git a/grafana-cloud-from-kubernetes/otelcol-cr.yaml b/grafana-cloud-from-kubernetes/otelcol-cr.yaml index 93979c2..59290d5 100644 --- a/grafana-cloud-from-kubernetes/otelcol-cr.yaml +++ b/grafana-cloud-from-kubernetes/otelcol-cr.yaml @@ -3,7 +3,7 @@ kind: OpenTelemetryCollector metadata: name: grafana-cloud-from-kubernetes spec: - image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.101.0 + image: ghcr.io/jpkrohling/otelcol-distributions/otelcol-otlp:0.101.5 envFrom: - secretRef: name: grafana-cloud-credentials diff --git a/kafka-on-kubernetes/otelcol-pub.yaml b/kafka-on-kubernetes/otelcol-pub.yaml index 4028c13..b774f37 100644 --- a/kafka-on-kubernetes/otelcol-pub.yaml +++ b/kafka-on-kubernetes/otelcol-pub.yaml @@ -3,7 +3,7 @@ kind: OpenTelemetryCollector metadata: name: otelcol-pub spec: - image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.101.0 + image: ghcr.io/jpkrohling/otelcol-distributions/otelcol-kafka:0.101.5 config: receivers: otlp: diff --git a/kafka-on-kubernetes/otelcol-sub.yaml b/kafka-on-kubernetes/otelcol-sub.yaml index d6af99c..03dcbf2 100644 --- a/kafka-on-kubernetes/otelcol-sub.yaml +++ b/kafka-on-kubernetes/otelcol-sub.yaml @@ -3,7 +3,7 @@ kind: OpenTelemetryCollector metadata: name: otelcol-sub spec: - image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.101.0 + image: ghcr.io/jpkrohling/otelcol-distributions/otelcol-kafka:0.101.5 envFrom: - secretRef: name: grafana-cloud-credentials diff --git a/profiling-the-collector/README.md b/profiling-the-collector/README.md index c5f91e4..0aa1016 100644 --- a/profiling-the-collector/README.md +++ b/profiling-the-collector/README.md @@ -5,6 +5,8 @@ This recipe shows how to use Pyroscope to profile the OpenTelemetry Collector ru - ["grafana-cloud"](../grafana-cloud/) - ["grafana-cloud-from-kubernetes"](../grafana-cloud-from-kubernetes/) +We are discarding the telemetry data that we are generating, as we are only interested in assessing this behavior by observing the Collector's profiles. + ## 🧄 Ingredients - OpenTelemetry Operator, see the main [`README.md`](../README.md) for instructions @@ -25,47 +27,40 @@ This recipe shows how to use Pyroscope to profile the OpenTelemetry Collector ru kubens profiling-the-collector ``` -2. Create a secret with the credentials: - ```terminal - kubectl create secret generic grafana-cloud-credentials --from-literal=GRAFANA_CLOUD_USER="$GRAFANA_CLOUD_USER" --from-literal=GRAFANA_CLOUD_TOKEN="$GRAFANA_CLOUD_TOKEN" - ``` - -3. Change the `endpoint` parameter for the `otlphttp` exporter to point to your stack's endpoint - -4. Install the OTel Collector custom resource +2. Install the OTel Collector custom resource ```terminal kubectl apply -f otelcol-cr.yaml ``` -5. Open a port-forward to the Collector +3. Open a port-forward to the Collector ```terminal kubectl port-forward svc/profiling-the-collector-collector 4317 ``` -6. Send 10 traces per second for 30 minutes to our Collector +4. Send 500 traces per second for 30 minutes to our Collector ```terminal - telemetrygen traces --duration 30m --rate 10 --otlp-insecure --otlp-attributes='recipe="profiling-the-collector"' + telemetrygen traces --duration 30m --rate 500 --otlp-insecure --otlp-attributes='recipe="profiling-the-collector"' ``` -7. Install a custom service exposing the `pprof` port +5. Install a custom service exposing the `pprof` port ```terminal kubectl apply -f otelcol-pprof-svc.yaml ``` -8. Add the Grafana Helm repository +6. Add the Grafana Helm repository ```terminal helm repo add grafana https://grafana.github.io/helm-charts helm repo update ``` -9. Replace the user, token, and URL on the values.yaml to reflect the values from your environment variables +7. Replace the URL on the values.yaml to reflect your Pyroscope connection details -10. Install Grafana Agent with the Pyroscope eBPF Profiler +8. Install Grafana Agent with the Pyroscope eBPF Profiler ```terminal helm install pyroscope-ebpf grafana/grafana-agent -f values.yaml --set username=$GRAFANA_CLOUD_PROFILES_USER --set password=$GRAFANA_CLOUD_PROFILES_TOKEN ``` -11. Open your Grafana instance, go to Explore, and select the profiles datasource. +9. Open your Grafana instance, go to Explore, and select the profiles datasource. ## 😋 Executed last time with these versions diff --git a/profiling-the-collector/otelcol-cr.yaml b/profiling-the-collector/otelcol-cr.yaml index 4c5381a..b7bead1 100644 --- a/profiling-the-collector/otelcol-cr.yaml +++ b/profiling-the-collector/otelcol-cr.yaml @@ -3,18 +3,11 @@ kind: OpenTelemetryCollector metadata: name: profiling-the-collector spec: - image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.101.0 - envFrom: - - secretRef: - name: grafana-cloud-credentials + image: ghcr.io/jpkrohling/otelcol-distributions/otelcol-otlp:0.101.5 config: extensions: pprof: endpoint: 0.0.0.0:1777 - basicauth: - client_auth: - username: "${env:GRAFANA_CLOUD_USER}" - password: "${env:GRAFANA_CLOUD_TOKEN}" receivers: otlp: @@ -22,23 +15,20 @@ spec: grpc: {} exporters: - otlphttp: - endpoint: https://otlp-gateway-prod-eu-west-2.grafana.net/otlp - auth: - authenticator: basicauth + nop: {} service: - extensions: [ pprof, basicauth ] + extensions: [ pprof ] pipelines: traces: receivers: [ otlp ] processors: [ ] - exporters: [ otlphttp ] + exporters: [ nop ] logs: receivers: [ otlp ] processors: [ ] - exporters: [ otlphttp ] + exporters: [ nop ] metrics: receivers: [ otlp ] processors: [ ] - exporters: [ otlphttp ] + exporters: [ nop ]