# Observability

This directory contains the observability stack for the application. The stack is composed of the following components:

- Prometheus v3.2.1
- Grafana 11.6.0
- Loki 3.4.2
- Jaeger 2.4.0
- Otel Collector 0.120.0 # 0.121.0 remove loki

## Prometheus

Prometheus is a monitoring and alerting toolkit. It scrapes metrics from instrumented jobs, either directly or via an
intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to
either aggregate and record new time series from existing data or generate alerts. Grafana or other API consumers can be
used to visualize the collected data.

## Grafana

Grafana is a multi-platform open-source analytics and interactive visualization web application. It provides charts,
graphs, and alerts for the web when connected to supported data sources.

## Loki

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is
designed to be very cost-effective and easy to operate. It does not index the contents of the logs, but rather a set of
labels for each log stream.

## Jaeger

Jaeger is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and
troubleshooting microservices-based distributed systems, including:

- Distributed context propagation
- Distributed transaction monitoring
- Root cause analysis
- Service dependency analysis
- Performance / latency optimization

## Otel Collector

The OpenTelemetry Collector offers a vendor-agnostic implementation on how to receive, process, and export telemetry
data. It removes the need to run, operate, and maintain multiple agents/collectors in order to support open-source
observability data formats (e.g. Jaeger, Prometheus, etc.) sending to one or more open-source or commercial back-ends.

## How to use

To deploy the observability stack, run the following command:

- docker latest version

```bash
docker compose -f docker-compose.yml -f docker-compose.override.yml up -d
```

- docker compose v2.0.0 or before

```bash
docke-compose -f docker-compose.yml -f docker-compose.override.yml up -d
```

To access the Grafana dashboard, navigate to `http://localhost:3000` in your browser. The default username and password
are `admin` and `admin`, respectively.

To access the Jaeger dashboard, navigate to `http://localhost:16686` in your browser.

To access the Prometheus dashboard, navigate to `http://localhost:9090` in your browser.

## How to stop

To stop the observability stack, run the following command:

```bash
docker compose -f docker-compose.yml -f docker-compose.override.yml down
```

## How to remove data

To remove the data generated by the observability stack, run the following command:

```bash
docker compose -f docker-compose.yml -f docker-compose.override.yml down -v
```

## How to configure

To configure the observability stack, modify the `docker-compose.override.yml` file. The file contains the following

```yaml
services:
  prometheus:
    environment:
      - PROMETHEUS_CONFIG_FILE=/etc/prometheus/prometheus.yml
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  grafana:
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
```

The `prometheus` service mounts the `prometheus.yml` file to `/etc/prometheus/prometheus.yml`. The `grafana` service
mounts the `grafana/provisioning` directory to `/etc/grafana/provisioning`. You can modify these files to configure the
observability stack.