learn-metrics.git
4 months agoReorganize configs main
Martin Pitt [Fri, 9 Jul 2021 07:32:39 +0000 (09:32 +0200)]
Reorganize configs

- Rename directories like the volumes.
- Add Grafana dashboard provisioning, and split off config. It was weird
  to mount the same directory under two different paths.
- Now arbitrary dashboards can be put into
  grafana-provisioning/dashboards/*.json and they will be automatically
  deployed.

5 months agoMove Grafana config to directory volume
Martin Pitt [Wed, 16 Jun 2021 05:23:14 +0000 (07:23 +0200)]
Move Grafana config to directory volume

This more closely resembles what one would do on Kubernetes with a
`ConfigMap`.

5 months agoGrafana: Enable anonymous access
Martin Pitt [Wed, 16 Jun 2021 05:05:50 +0000 (07:05 +0200)]
Grafana: Enable anonymous access

6 months agoPull prometheus image from quay.io
Martin Pitt [Wed, 26 May 2021 08:13:45 +0000 (10:13 +0200)]
Pull prometheus image from quay.io

To avoid docker.io pull limits

6 months agoMove Prometheus config to directory volume
Martin Pitt [Wed, 26 May 2021 06:52:23 +0000 (08:52 +0200)]
Move Prometheus config to directory volume

This more closely resembles what one would do on Kubernetes with a
`ConfigMap`.

6 months agoAdd histrogram example
Martin Pitt [Mon, 24 May 2021 10:03:35 +0000 (12:03 +0200)]
Add histrogram example

6 months agoAdd script to generate time series
Martin Pitt [Fri, 21 May 2021 11:26:23 +0000 (13:26 +0200)]
Add script to generate time series

The text import format supports specifying a time stamp [1]. Use this to
generate a time series for thing_count and thing_failures.

    ./things-series  > http/metrics
    podman pod rm -f learn-metrics || true
    podman play kube learn-metrics.yaml

First non-trivial query from "indefinitely long lookback" counters: "How
many tests happened in the last 5 minutes?

    rate(thing_count{job="host-http"}[5m]) * 5 * 60

The re-scaling is because the natural unit is "1/s" and we want the unit
to be the same as the original counter.

[1] https://prometheus.io/docs/instrumenting/exposition_formats/

6 months agoAdd custom prometheus data source from host
Martin Pitt [Fri, 21 May 2021 08:40:32 +0000 (10:40 +0200)]
Add custom prometheus data source from host

Export http/metrics into prometheus via an intermediate web server in
the pod, and configuring it as an additional source.

You can manually provide metrics by copying http/metrics.template to
http/metrics and edit the numbers while prometheus is running.

6 months agoReplace launch script with kubernetes resource
Martin Pitt [Fri, 21 May 2021 06:56:59 +0000 (08:56 +0200)]
Replace launch script with kubernetes resource

This can be deployed locally with

    podman play kube learn-metrics.yaml

Generated with

    podman generate kube learn-metrics > learn-metrics.yaml

and heavy editing/manual cleanup.

6 months agoPre-configure prometheus data source in Grafana
Martin Pitt [Fri, 21 May 2021 06:34:06 +0000 (08:34 +0200)]
Pre-configure prometheus data source in Grafana

This mysteriously fixes the data source test "not found" error.

6 months agoRun unconfigured grafana and prometheus containers
Martin Pitt [Fri, 21 May 2021 06:08:03 +0000 (08:08 +0200)]
Run unconfigured grafana and prometheus containers

Start with `./launch.sh`, tear down with `podman pod rm -f learn-metrics`.

Prometheus explorer at http://localhost:3001 (to avoid port conflict
with Cockpit). It can list or graph simple queries like
`go_memstats_heap_objects` or `go_memstats_heap_objects[5min]`.

Metrics source at <http://localhost:3001/metrics>
<http://metrics-prometheus:9090/metrics> (in grafana container).

Grafana UI at <http://localhost:3000>, log in as admin:foobar. Manually
add Prometheus data source at <http://metrics-prometheus:9090/metrics>.

This fails with "HTTP Error Not Found" with "Server" access (which is
usually what you want -- "Browser" access fails on CORS).