diff --git a/.docker/openobserve-otel/README.md b/.docker/openobserve-otel/README.md new file mode 100644 index 00000000..5993c2fc --- /dev/null +++ b/.docker/openobserve-otel/README.md @@ -0,0 +1,75 @@ +# OpenObserve + OpenTelemetry Collector + +[![OpenObserve](https://img.shields.io/badge/OpenObserve-OpenSource-blue.svg)](https://openobserve.org) +[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-Collector-green.svg)](https://opentelemetry.io/) + +English | [中文](README_ZH.md) + +This directory contains the configuration files for setting up an observability stack with OpenObserve and OpenTelemetry +Collector. + +### Overview + +This setup provides a complete observability solution for your applications: + +- **OpenObserve**: A modern, open-source observability platform for logs, metrics, and traces. +- **OpenTelemetry Collector**: Collects and processes telemetry data before sending it to OpenObserve. + +### Setup Instructions + +1. **Prerequisites**: + - Docker and Docker Compose installed + - Sufficient memory resources (minimum 2GB recommended) + +2. **Starting the Services**: + ```bash + cd .docker/openobserve-otel + docker compose -f docker-compose.yml up -d + ``` + +3. **Accessing the Dashboard**: + - OpenObserve UI: http://localhost:5080 + - Default credentials: + - Username: root@rustfs.com + - Password: rustfs123 + +### Configuration + +#### OpenObserve Configuration + +The OpenObserve service is configured with: + +- Root user credentials +- Data persistence through a volume mount +- Memory cache enabled +- Health checks +- Exposed ports: + - 5080: HTTP API and UI + - 5081: OTLP gRPC + +#### OpenTelemetry Collector Configuration + +The collector is configured to: + +- Receive telemetry data via OTLP (HTTP and gRPC) +- Collect logs from files +- Process data in batches +- Export data to OpenObserve +- Manage memory usage + +### Integration with Your Application + +To send telemetry data from your application, configure your OpenTelemetry SDK to send data to: + +- OTLP gRPC: `localhost:4317` +- OTLP HTTP: `localhost:4318` + +For example, in a Rust application using the `rustfs-obs` library: + +```bash +export RUSTFS_OBS_ENDPOINT=http://localhost:4317 +export RUSTFS_OBS_SERVICE_NAME=yourservice +export RUSTFS_OBS_SERVICE_VERSION=1.0.0 +export RUSTFS_OBS_ENVIRONMENT=development +``` + diff --git a/.docker/openobserve-otel/README_ZH.md b/.docker/openobserve-otel/README_ZH.md new file mode 100644 index 00000000..2e2e80c9 --- /dev/null +++ b/.docker/openobserve-otel/README_ZH.md @@ -0,0 +1,75 @@ +# OpenObserve + OpenTelemetry Collector + +[![OpenObserve](https://img.shields.io/badge/OpenObserve-OpenSource-blue.svg)](https://openobserve.org) +[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-Collector-green.svg)](https://opentelemetry.io/) + +[English](README.md) | 中文 + +## 中文 + +本目录包含搭建 OpenObserve 和 OpenTelemetry Collector 可观测性栈的配置文件。 + +### 概述 + +此设置为应用程序提供了完整的可观测性解决方案： + +- **OpenObserve**：现代化、开源的可观测性平台，用于日志、指标和追踪。 +- **OpenTelemetry Collector**：收集和处理遥测数据，然后将其发送到 OpenObserve。 + +### 设置说明 + +1. **前提条件**： + - 已安装 Docker 和 Docker Compose + - 足够的内存资源（建议至少 2GB） + +2. **启动服务**： + ```bash + cd .docker/openobserve-otel + docker compose -f docker-compose.yml up -d + ``` + +3. **访问仪表板**： + - OpenObserve UI：http://localhost:5080 + - 默认凭据： + - 用户名：root@rustfs.com + - 密码：rustfs123 + +### 配置 + +#### OpenObserve 配置 + +OpenObserve 服务配置： + +- 根用户凭据 +- 通过卷挂载实现数据持久化 +- 启用内存缓存 +- 健康检查 +- 暴露端口： + - 5080：HTTP API 和 UI + - 5081：OTLP gRPC + +#### OpenTelemetry Collector 配置 + +收集器配置为： + +- 通过 OTLP（HTTP 和 gRPC）接收遥测数据 +- 从文件中收集日志 +- 批处理数据 +- 将数据导出到 OpenObserve +- 管理内存使用 + +### 与应用程序集成 + +要从应用程序发送遥测数据，将 OpenTelemetry SDK 配置为发送数据到： + +- OTLP gRPC:`localhost:4317` +- OTLP HTTP:`localhost:4318` + +例如，在使用 `rustfs-obs` 库的 Rust 应用程序中： + +```bash +export RUSTFS_OBS_ENDPOINT=http://localhost:4317 +export RUSTFS_OBS_SERVICE_NAME=yourservice +export RUSTFS_OBS_SERVICE_VERSION=1.0.0 +export RUSTFS_OBS_ENVIRONMENT=development +``` \ No newline at end of file diff --git a/README.md b/README.md index 9bbb8be9..8cb48943 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ export RUSTFS_CONSOLE_ENABLE=true export RUSTFS_CONSOLE_ADDRESS="0.0.0.0:9001" # Observability config -export RUSTFS_OBS_CONFIG="./deploy/config/obs.toml" +export RUSTFS_OBS_ENDPOINT="http://localhost:4317" # Event message configuration #export RUSTFS_EVENT_CONFIG="./deploy/config/event.toml" @@ -92,25 +92,3 @@ export RUSTFS_OBS_CONFIG="./deploy/config/obs.toml" - Grafana: `http://localhost:3000` (credentials: `admin`/`admin`) - Jaeger: `http://localhost:16686` - Prometheus: `http://localhost:9090` - -#### Configuring Observability - -1. Copy the example configuration: - ```bash - cd deploy/config - cp obs.toml.example obs.toml - ``` - -2. Edit `obs.toml` with the following parameters: - -| Parameter | Description | Example | -|----------------------|-----------------------------------|-----------------------| -| endpoint | OpenTelemetry Collector address | http://localhost:4317 | -| service_name | Service name | rustfs | -| service_version | Service version | 1.0.0 | -| environment | Runtime environment | production | -| meter_interval | Metrics export interval (seconds) | 30 | -| sample_ratio | Sampling ratio | 1.0 | -| use_stdout | Output to console | true/false | -| logger_level | Log level | info | -| local_logging_enable | stdout | true/false | diff --git a/README_ZH.md b/README_ZH.md index 89fe320a..7362e9e7 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -60,7 +60,7 @@ export RUSTFS_CONSOLE_ENABLE=true export RUSTFS_CONSOLE_ADDRESS="0.0.0.0:9001" # 可观测性配置 -export RUSTFS_OBS_CONFIG="./deploy/config/obs.toml" +export RUSTFS_OBS_ENDPOINT="http://localhost:4317" # 事件消息配置 #export RUSTFS_EVENT_CONFIG="./deploy/config/event.toml" @@ -94,24 +94,6 @@ export RUSTFS_OBS_CONFIG="./deploy/config/obs.toml" #### 配置可观测性 -1. 复制示例配置： - ```bash - cd deploy/config - cp obs.toml.example obs.toml - ``` - -2. 编辑 `obs.toml` 配置文件，参数如下： - -| 配置项 | 说明 | 示例值 | -|----------------------|----------------------------|-----------------------| -| endpoint | OpenTelemetry Collector 地址 | http://localhost:4317 | -| service_name | 服务名称 | rustfs | -| service_version | 服务版本 | 1.0.0 | -| environment | 运行环境 | production | -| meter_interval | 指标导出间隔 (秒) | 30 | -| sample_ratio | 采样率 | 1.0 | -| use_stdout | 是否输出到控制台 | true/false | -| logger_level | 日志级别 | info | -| local_logging_enable | 控制台是否答应日志 | true/false | - +``` +OpenTelemetry Collector 地址(endpoint): http://localhost:4317 ``` \ No newline at end of file diff --git a/crates/obs/src/config.rs b/crates/obs/src/config.rs index b1f4850d..5284e301 100644 --- a/crates/obs/src/config.rs +++ b/crates/obs/src/config.rs @@ -37,14 +37,14 @@ impl OtelConfig { pub fn extract_otel_config_from_env(endpoint: Option) -> OtelConfig { let endpoint = if let Some(endpoint) = endpoint { if endpoint.is_empty() { - env::var("RUSTFS_OBSERVABILITY_ENDPOINT").unwrap_or_else(|_| "".to_string()) + env::var("RUSTFS_OBS_ENDPOINT").unwrap_or_else(|_| "".to_string()) } else { endpoint } } else { - env::var("RUSTFS_OBSERVABILITY_ENDPOINT").unwrap_or_else(|_| "".to_string()) + env::var("RUSTFS_OBS_ENDPOINT").unwrap_or_else(|_| "".to_string()) }; - let mut use_stdout = env::var("RUSTFS_OBSERVABILITY_USE_STDOUT") + let mut use_stdout = env::var("RUSTFS_OBS_USE_STDOUT") .ok() .and_then(|v| v.parse().ok()) .or(Some(USE_STDOUT)); @@ -55,51 +55,51 @@ impl OtelConfig { OtelConfig { endpoint, use_stdout, - sample_ratio: env::var("RUSTFS_OBSERVABILITY_SAMPLE_RATIO") + sample_ratio: env::var("RUSTFS_OBS_SAMPLE_RATIO") .ok() .and_then(|v| v.parse().ok()) .or(Some(SAMPLE_RATIO)), - meter_interval: env::var("RUSTFS_OBSERVABILITY_METER_INTERVAL") + meter_interval: env::var("RUSTFS_OBS_METER_INTERVAL") .ok() .and_then(|v| v.parse().ok()) .or(Some(METER_INTERVAL)), - service_name: env::var("RUSTFS_OBSERVABILITY_SERVICE_NAME") + service_name: env::var("RUSTFS_OBS_SERVICE_NAME") .ok() .and_then(|v| v.parse().ok()) .or(Some(APP_NAME.to_string())), - service_version: env::var("RUSTFS_OBSERVABILITY_SERVICE_VERSION") + service_version: env::var("RUSTFS_OBS_SERVICE_VERSION") .ok() .and_then(|v| v.parse().ok()) .or(Some(SERVICE_VERSION.to_string())), - environment: env::var("RUSTFS_OBSERVABILITY_ENVIRONMENT") + environment: env::var("RUSTFS_OBS_ENVIRONMENT") .ok() .and_then(|v| v.parse().ok()) .or(Some(ENVIRONMENT.to_string())), - logger_level: env::var("RUSTFS_OBSERVABILITY_LOGGER_LEVEL") + logger_level: env::var("RUSTFS_OBS_LOGGER_LEVEL") .ok() .and_then(|v| v.parse().ok()) .or(Some(DEFAULT_LOG_LEVEL.to_string())), - local_logging_enabled: env::var("RUSTFS_OBSERVABILITY_LOCAL_LOGGING_ENABLED") + local_logging_enabled: env::var("RUSTFS_OBS_LOCAL_LOGGING_ENABLED") .ok() .and_then(|v| v.parse().ok()) .or(Some(false)), - log_directory: env::var("RUSTFS_OBSERVABILITY_LOG_DIRECTORY") + log_directory: env::var("RUSTFS_OBS_LOG_DIRECTORY") .ok() .and_then(|v| v.parse().ok()) .or(Some(DEFAULT_LOG_DIR.to_string())), - log_filename: env::var("RUSTFS_OBSERVABILITY_LOG_FILENAME") + log_filename: env::var("RUSTFS_OBS_LOG_FILENAME") .ok() .and_then(|v| v.parse().ok()) .or(Some(DEFAULT_LOG_FILENAME.to_string())), - log_rotation_size_mb: env::var("RUSTFS_OBSERVABILITY_LOG_ROTATION_SIZE_MB") + log_rotation_size_mb: env::var("RUSTFS_OBS_LOG_ROTATION_SIZE_MB") .ok() .and_then(|v| v.parse().ok()) .or(Some(DEFAULT_LOG_ROTATION_SIZE_MB)), // Default to 100 MB - log_rotation_time: env::var("RUSTFS_OBSERVABILITY_LOG_ROTATION_TIME") + log_rotation_time: env::var("RUSTFS_OBS_LOG_ROTATION_TIME") .ok() .and_then(|v| v.parse().ok()) .or(Some(DEFAULT_LOG_ROTATION_TIME.to_string())), // Default to "Day" - log_keep_files: env::var("RUSTFS_OBSERVABILITY_LOG_KEEP_FILES") + log_keep_files: env::var("RUSTFS_OBS_LOG_KEEP_FILES") .ok() .and_then(|v| v.parse().ok()) .or(Some(DEFAULT_LOG_KEEP_FILES)), // Default to keeping 30 log files diff --git a/deploy/config/.example.obs.env b/deploy/config/.example.obs.env deleted file mode 100644 index edbf3e88..00000000 --- a/deploy/config/.example.obs.env +++ /dev/null @@ -1,28 +0,0 @@ -#RUSTFS__OBSERVABILITY__ENDPOINT=http://localhost:4317 -#RUSTFS__OBSERVABILITY__USE_STDOUT=true -#RUSTFS__OBSERVABILITY__SAMPLE_RATIO=2.0 -#RUSTFS__OBSERVABILITY__METER_INTERVAL=30 -#RUSTFS__OBSERVABILITY__SERVICE_NAME=rustfs -#RUSTFS__OBSERVABILITY__SERVICE_VERSION=0.1.0 -#RUSTFS__OBSERVABILITY__ENVIRONMENT=develop -#RUSTFS__OBSERVABILITY__LOGGER_LEVEL=debug -# -#RUSTFS__SINKS_0__type=Kakfa -#RUSTFS__SINKS_0__brokers=localhost:9092 -#RUSTFS__SINKS_0__topic=logs -#RUSTFS__SINKS_0__batch_size=100 -#RUSTFS__SINKS_0__batch_timeout_ms=1000 -# -#RUSTFS__SINKS_1__type=Webhook -#RUSTFS__SINKS_1__endpoint=http://localhost:8080/webhook -#RUSTFS__SINKS_1__auth_token=you-auth-token -#RUSTFS__SINKS_1__batch_size=100 -#RUSTFS__SINKS_1__batch_timeout_ms=1000 -# -#RUSTFS__SINKS_2__type=File -#RUSTFS__SINKS_2__path=./deploy/logs/rustfs.log -#RUSTFS__SINKS_2__buffer_size=10 -#RUSTFS__SINKS_2__flush_interval_ms=1000 -#RUSTFS__SINKS_2__flush_threshold=100 -# -#RUSTFS__LOGGER__QUEUE_CAPACITY=10 \ No newline at end of file diff --git a/deploy/config/event.example.toml b/deploy/config/event.example.toml deleted file mode 100644 index 6deb67a2..00000000 --- a/deploy/config/event.example.toml +++ /dev/null @@ -1,29 +0,0 @@ -# config.toml -store_path = "./deploy/logs/event_store" -channel_capacity = 5000 - -[[adapters]] -type = "Webhook" -endpoint = "http://127.0.0.1:3020/webhook" -auth_token = "your-auth-token" -max_retries = 3 -timeout = 50 - -[adapters.custom_headers] -custom_server = "value_server" -custom_client = "value_client" - -#[[adapters]] -#type = "Kafka" -#brokers = "localhost:9092" -#topic = "notifications" -#max_retries = 3 -#timeout = 60 -# -#[[adapters]] -#type = "Mqtt" -#broker = "mqtt.example.com" -#port = 1883 -#client_id = "event-notifier" -#topic = "events" -#max_retries = 3 \ No newline at end of file diff --git a/deploy/config/obs-zh.example.toml b/deploy/config/obs-zh.example.toml deleted file mode 100644 index 0474391b..00000000 --- a/deploy/config/obs-zh.example.toml +++ /dev/null @@ -1,34 +0,0 @@ -[observability] -endpoint = "http://localhost:4317" # 可观测性数据上报的终端地址，默认为"http://localhost:4317" -use_stdout = true # 是否将日志输出到标准输出 -sample_ratio = 2.0 # 采样率，表示每 2 条数据采样 1 条 -meter_interval = 30 # 指标收集间隔，单位为秒 -service_name = "rustfs" # 服务名称，用于标识当前服务 -service_version = "0.1.0" # 服务版本号 -environments = "develop" # 运行环境，如开发环境 (develop) -logger_level = "debug" # 日志级别，可选 debug/info/warn/error 等 -local_logging_enabled = true # 是否启用本地 stdout 日志记录，true 表示启用，false 表示禁用 - -#[[sinks]] # Kafka 接收器配置 -#type = "Kafka" # 是否启用 Kafka 接收器，默认禁用 -#brokers = "localhost:9092" # Kafka 服务器地址 -#topic = "logs" # Kafka 主题名称 -#batch_size = 100 # 批处理大小，每次发送的消息数量 -#batch_timeout_ms = 1000 # 批处理超时时间，单位为毫秒 - -#[[sinks]] # Webhook 接收器配置 -#type = "Webhook" # 是否启用 Webhook 接收器 -#endpoint = "http://localhost:8080/webhook" # Webhook 接收地址 -#auth_token = "" # 认证令牌 -#batch_size = 100 # 批处理大小 -#batch_timeout_ms = 1000 # 批处理超时时间，单位为毫秒 - -[[sinks]] # 文件接收器配置 -type = "File" # 是否启用文件接收器 -path = "./deploy/logs/rustfs.log" # 日志文件路径 -buffer_size = 10 # 缓冲区大小，表示每次写入的字节数 -flush_interval_ms = 100 # 批处理超时时间，单位为毫秒 -flush_threshold = 100 # 刷新阈值，表示在达到该数量时刷新日志 - -[logger] # 日志器配置 -queue_capacity = 10 # 日志队列容量，表示可以缓存的日志条数 \ No newline at end of file diff --git a/deploy/config/obs.example.toml b/deploy/config/obs.example.toml deleted file mode 100644 index 0733b229..00000000 --- a/deploy/config/obs.example.toml +++ /dev/null @@ -1,34 +0,0 @@ -[observability] -endpoint = "http://localhost:4317" # Default is "http://localhost:4317" if not specified -use_stdout = false # Output with stdout, true output, false no output -sample_ratio = 2.0 -meter_interval = 30 -service_name = "rustfs" -service_version = "0.0.1" -environment = "production" # Default is "production" if not specified -logger_level = "info" -local_logging_enabled = true - -#[[sinks]] -#type = "Kafka" -#brokers = "localhost:9092" -#topic = "logs" -#batch_size = 100 # Default is 100 if not specified -#batch_timeout_ms = 100 # Default is 1000ms if not specified -# -#[[sinks]] -#type = "Webhook" -#endpoint = "http://localhost:8080/webhook" -#auth_token = "" -#batch_size = 100 # Default is 3 if not specified -#batch_timeout_ms = 100 # Default is 100ms if not specified - -[[sinks]] -type = "File" -path = "deploy/logs/rustfs.log" -buffer_size = 101 # Default is 8192 bytes if not specified -flush_interval_ms = 1000 -flush_threshold = 100 - -[logger] -queue_capacity = 10000 diff --git a/deploy/config/rustfs-zh.env b/deploy/config/rustfs-zh.env index fd2a5178..37f20e5c 100644 --- a/deploy/config/rustfs-zh.env +++ b/deploy/config/rustfs-zh.env @@ -20,8 +20,8 @@ RUSTFS_SERVER_ENDPOINT="http://127.0.0.1:9000" RUSTFS_SERVER_DOMAINS=127.0.0.1:9002 # RustFS 许可证内容 RUSTFS_LICENSE="license content" -# 可观测性配置文件路径：deploy/config/obs.example.toml -RUSTFS_OBS_CONFIG=/etc/default/obs.toml +# 可观测性配置Endpoint：http://localhost:4317 +RUSTFS_OBS_ENDPOINT=http://localhost:4317 # TLS 证书目录路径：deploy/certs RUSTFS_TLS_PATH=/etc/default/tls # 事件通知配置文件路径：deploy/config/event.example.toml diff --git a/deploy/config/rustfs.env b/deploy/config/rustfs.env index 3f50033f..a8a0d853 100644 --- a/deploy/config/rustfs.env +++ b/deploy/config/rustfs.env @@ -20,8 +20,8 @@ RUSTFS_SERVER_ENDPOINT="http://127.0.0.1:9000" RUSTFS_SERVER_DOMAINS=127.0.0.1:9002 # RustFS license content RUSTFS_LICENSE="license content" -# Observability configuration file path: deploy/config/obs.example.toml -RUSTFS_OBS_CONFIG=/etc/default/obs.toml +# Observability configuration endpoint: RUSTFS_OBS_ENDPOINT +RUSTFS_OBS_ENDPOINT=http://localhost:4317 # TLS certificates directory path: deploy/certs RUSTFS_TLS_PATH=/etc/default/tls # event notification configuration file path: deploy/config/event.example.toml diff --git a/scripts/run.sh b/scripts/run.sh index aa8258b5..aebca09b 100755 --- a/scripts/run.sh +++ b/scripts/run.sh @@ -36,22 +36,19 @@ export RUSTFS_CONSOLE_ADDRESS=":9001" # HTTPS 证书目录 # export RUSTFS_TLS_PATH="./deploy/certs" -# 具体路径修改为配置文件真实路径，obs.example.toml 仅供参考 其中 `RUSTFS_OBS_CONFIG` 和下面变量二选一 -# export RUSTFS_OBS_ENDPOINT=http://localhost:4317 - -# 如下变量需要必须参数都有值才可以，以及会覆盖配置文件中的值 -#export RUSTFS_OBSERVABILITY_ENDPOINT=http://localhost:4317 # OpenTelemetry Collector 的地址 -#export RUSTFS_OBSERVABILITY_USE_STDOUT=false # 是否使用标准输出 -#export RUSTFS_OBSERVABILITY_SAMPLE_RATIO=2.0 # 采样率，0.0-1.0之间，0.0表示不采样，1.0表示全部采样 -#export RUSTFS_OBSERVABILITY_METER_INTERVAL=1 # 采样间隔，单位为秒 -#export RUSTFS_OBSERVABILITY_SERVICE_NAME=rustfs # 服务名称 -#export RUSTFS_OBSERVABILITY_SERVICE_VERSION=0.1.0 # 服务版本 -#export RUSTFS_OBSERVABILITY_ENVIRONMENT=develop # 环境名称 -#export RUSTFS_OBSERVABILITY_LOGGER_LEVEL=debug # 日志级别，支持 trace, debug, info, warn, error -export RUSTFS_OBSERVABILITY_LOCAL_LOGGING_ENABLED=true # 是否启用本地日志记录 -export RUSTFS_OBSERVABILITY_LOG_DIRECTORY="./deploy/logs" # Log directory -export RUSTFS_OBSERVABILITY_LOG_ROTATION_TIME="minute" # Log rotation time unit, can be "second", "minute", "hour", "day" -export RUSTFS_OBSERVABILITY_LOG_ROTATION_SIZE_MB=1 # Log rotation size in MB +# 可观测性 相关配置信息 +#export RUSTFS_OBS_ENDPOINT=http://localhost:4317 # OpenTelemetry Collector 的地址 +#export RUSTFS_OBS_USE_STDOUT=false # 是否使用标准输出 +#export RUSTFS_OBS_SAMPLE_RATIO=2.0 # 采样率，0.0-1.0之间，0.0表示不采样，1.0表示全部采样 +#export RUSTFS_OBS_METER_INTERVAL=1 # 采样间隔，单位为秒 +#export RUSTFS_OBS_SERVICE_NAME=rustfs # 服务名称 +#export RUSTFS_OBS_SERVICE_VERSION=0.1.0 # 服务版本 +#export RUSTFS_OBS_ENVIRONMENT=develop # 环境名称 +#export RUSTFS_OBS_LOGGER_LEVEL=debug # 日志级别，支持 trace, debug, info, warn, error +export RUSTFS_OBS_LOCAL_LOGGING_ENABLED=true # 是否启用本地日志记录 +export RUSTFS_OBS_LOG_DIRECTORY="./deploy/logs" # Log directory +export RUSTFS_OBS_LOG_ROTATION_TIME="minute" # Log rotation time unit, can be "second", "minute", "hour", "day" +export RUSTFS_OBS_LOG_ROTATION_SIZE_MB=1 # Log rotation size in MB # #export RUSTFS_SINKS_FILE_PATH=./deploy/logs/rustfs.log