When you hear about statsd exporter, you might confuse it with statsd. Its okay, you are not the only person who does it. Statsd run as a daemon and it receives metrics from statsd clients and sends aggregates to backends like graphite for storage and future visualisation. On other hand, statsd exporter receives metrics from statsd clients or statsd server itself and exports them as prometheus metrics. You can configure your prometheus or victoria metrics to scrape metrics on path /metrics of statsd exporter.
In this article, I will explain the steps to run multiple statsd exporters in kubernetes as a deployment.
Deploy Statsd exporter in kubernetes:
We need to create three kubernetes objects:
- A deployment to run three statsd exporter pods
- A ClusterIP service endpoint which accepts metrics in both UDP and TCP formats
- A configmap which contains Statsd exporter config file.
apiVersion: apps/v1 kind: Deployment metadata: name: statsd-exporter namespace: <namespace> spec: strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate replicas: 3 selector: matchLabels: app: statsd-exporter template: metadata: annotations: prometheus.io/port: "9102" prometheus.io/scrape: "true" labels: app: statsd-exporter spec: containers: - args: - --statsd.mapping-config=/etc/statsd.yml - --statsd.listen-udp=:9125 - --statsd.listen-tcp=:9125 - --web.listen-address=:9102 - --log.level=info volumeMounts: - mountPath: /etc/statsd.yml subPath: statsd.yml name: statsd-yaml readOnly: true image: prom/statsd-exporter:v0.17.0 name: statsd-exporter lifecycle: preStop: exec: command: ["sleep", "240"] livenessProbe: failureThreshold: 1 exec: command: - wget - --no-verbose - --tries=1 - --spider - http://localhost:9102/metrics initialDelaySeconds: 90 periodSeconds: 60 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 1 exec: command: - wget - --no-verbose - --tries=1 - --spider - http://localhost:9102/metrics initialDelaySeconds: 5 periodSeconds: 3 successThreshold: 1 timeoutSeconds: 10 ports: - containerPort: 9125 protocol: UDP - containerPort: 9125 protocol: TCP - containerPort: 9102 protocol: TCP resources: requests: cpu: 200m memory: 200Mi terminationGracePeriodSeconds: 300 volumes: - name: statsd-yaml configMap: name: statsd --- apiVersion: v1 kind: Service metadata: name: statsd-exporter namespace: <namespace> spec: ports: - name: ingress-tcp port: 9125 protocol: TCP targetPort: 9125 - name: ingress-udp port: 9125 protocol: UDP targetPort: 9125 selector: app: statsd-exporter type: ClusterIP --- apiVersion: v1 data: statsd.yml: | defaults: # By default all histogram use following buckets timer_type: histogram ttl: 20m buckets: [5, 30, 50, 80, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000] mappings: # Example of a particular metric using probably different bucket(not in this case, having just for sample) - match: sample_metrics.* timer_type: histogram buckets: [5, 30, 50, 80, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10000, 15000, 25000, 35000, 45000, 60000] name: "sample_metrics" kind: ConfigMap metadata: name: statsd namespace: <namespace>
Apply the YAML file to create the three K8s object we need. The newly created service endpoint statsd-exporter.the_name_space.svc.cluster.local
and port 9125 would be used by the application to send metrics to statsd exporter.
Configure statsd exporter to be scraped by prometheus:
Now we need to scrap these metrics exported by statsd exporter using prometheus or Victoria metrics. Statsd exporter exposes the metrics on path /metrics on port 9102.
If you noticed, I added following annotations in the statsd-exporter deployment yaml file, which instructs prometheus to scrap the metrics on port 9102 and path /metrics.
annotations: prometheus.io/port: "9102" prometheus.io/scrape: "true"
Go to Prometheus console and make sure statsd exporter is recognised as targets.
Push metrics to statsd exporter:
Once you make sure the prometheus scraps statsd exporter metrics, you can go ahead and configure your application to push metrics to the statsd exporter. You can also test pushing metrics using curl command. One example is,
echo "example_metrics:1|h" | nc -u -w0 statsd-exporter.the_name_space.svc.cluster.local 9125
While plotting the metrics in Prometheus or Grafana, make sure to use aggregator like SUM, AVG ..etc to include metrics from all statsd exporter pods.
Good day.