Observability

This directory contains the observability stack for the application. The stack is composed of the following components:

Prometheus v3.2.1
Grafana 11.6.0
Loki 3.4.2
Jaeger 2.4.0
Otel Collector 0.120.0 # 0.121.0 remove loki

Prometheus

Prometheus is a monitoring and alerting toolkit. It scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be used to visualize the collected data.

Grafana

Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts, graphs, and alerts for the web when connected to supported data sources.

Loki

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Jaeger

Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems, including:

Distributed context propagation
Distributed transaction monitoring
Root cause analysis
Service dependency analysis
Performance / latency optimization

Otel Collector

The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends.

How to use

To deploy the observability stack, run the following command:

docker latest version

docker compose -f docker-compose.yml -f docker-compose.override.yml up -d

docker compose v2.0.0 or before

docke-compose -f docker-compose.yml -f docker-compose.override.yml up -d

To access the Grafana dashboard, navigate to http://localhost:3000 in your browser. The default username and password are admin and admin, respectively.

To access the Jaeger dashboard, navigate to http://localhost:16686 in your browser.

To access the Prometheus dashboard, navigate to http://localhost:9090 in your browser.

How to stop

To stop the observability stack, run the following command:

docker compose -f docker-compose.yml -f docker-compose.override.yml down

How to remove data

To remove the data generated by the observability stack, run the following command:

docker compose -f docker-compose.yml -f docker-compose.override.yml down -v

How to configure

To configure the observability stack, modify the docker-compose.override.yml file. The file contains the following

services:
  prometheus:
    environment:
      - PROMETHEUS_CONFIG_FILE=/etc/prometheus/prometheus.yml
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning

The prometheus service mounts the prometheus.yml file to /etc/prometheus/prometheus.yml. The grafana service mounts the grafana/provisioning directory to /etc/grafana/provisioning. You can modify these files to configure the observability stack.

3.5 KiB Raw Blame History