Compare commits

..

26 Commits

Author SHA1 Message Date
houseme
416d3ad5b7 Refactor: Add observability enable flag, improve comments, remove unused config params, and enhance run function error logging. (#689)
* improve code for dns log

* fix

* Improve comments, remove unused parameters in config.rs (opt), add observability enable flag, and enhance error logging in run function execution.
2025-10-23 13:59:57 +08:00
weisd
f30698ec7f Refactor Console Server Architecture (#685)
* todo

* fix console server

* fix console server

* fix console server

* fix console server

* fix console server
2025-10-23 00:06:09 +08:00
houseme
7dcf01f127 feat: adjust metrics push interval to 3 seconds (#686)
- Reduce metrics push frequency from default to 3s for better performance
- Optimize resource utilization during metrics collection
- Improve real-time monitoring responsiveness

Related to admin metrics optimization on fix/admin-metrics branch
2025-10-22 23:47:11 +08:00
weisd
e524a106c5 add make bucket error logs (#683)
* add make bucket error logs
2025-10-22 16:23:08 +08:00
weisd
d9e5f5d2e3 fix (#682) 2025-10-22 10:35:40 +08:00
livelycode36
684e832530 fix: prevent duplicate data volumes in entrypoint.sh (#681) 2025-10-22 09:04:04 +08:00
weisd
a65856bdf4 Fix CRC32C Checksum Implementation and Enhance Authentication System (#678)
* fix: get_condition_values

* fix checksum crc32c

* fix clippy
2025-10-21 21:28:00 +08:00
weisd
2edb2929b2 fix: DataUsageInfo add list bucket permission (#674) 2025-10-21 10:05:54 +08:00
majinghe
14bc55479b fix docker healthcheck unhealthy issue (#672) 2025-10-21 09:39:15 +08:00
weisd
cd1e244c68 Refactor: Introduce content checksums and improve multipart/object metadata handling (#671)
* feat:  adapt to s3s typed etag support

* refactor: move replication struct to rustfs_filemeta, fix filemeta transition bug

* add head_object checksum, filter object metadata output

* fix multipart checksum

* fix multipart checksum

* add content md5,sha256 check

* fix test

* fix cargo

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2025-10-20 23:46:13 +08:00
songhahaha66
46797dc815 fix(export): fix the policy and service account export (#665)
* fix(export): fix the policy export mechanism

* fix: correct service account check logic in IamSys
2025-10-20 19:40:54 +08:00
Nugine
7f24dbda19 build(deps): upgrade s3s (#667)
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-10-19 18:32:01 +08:00
loverustfs
ef11d3a2eb fix words error 2025-10-19 18:13:58 +08:00
loverustfs
d1398cb3ab fix error 2025-10-19 18:10:45 +08:00
majinghe
95019c4cb5 add ansible installation with mnmd (#664)
* add ansible installation with mnmd

* change script install dir name
2025-10-18 22:20:17 +08:00
houseme
4168e6c180 chore(docs): move root examples to docs/examples/docker and update README (#663)
* chore(docs): move root `examples` to `docs/examples/docker` and update README

- Move root `examples/` contents into `docs/examples/docker/`.
- Update `docs/examples/README.md` to add migration note, new `docker/` entry and usage examples.
- Replace references from `examples/` to `docs/examples/docker/` where applicable.
- Reminder: verify CI and external links still point to the correct paths.

* fix
2025-10-17 17:17:36 +08:00
houseme
42d3645d6f fix(targets): make target removal and reload transactional; prevent reappearing entries (#662)
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <1948535941@qq.com>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <1948535941@qq.com>

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix: validate mqtt broker

* improve code for `import`

* fix

* improve

* remove logger from `rustfs-obs` crate

* remove code for config Observability

* fix

* improve code

* fix comment

* up

* up

* upgrade version

* fix

* fmt

* upgrade tokio version to 1.48.0

* upgrade `datafusion` and `reed-solomon-simd` version

* fix

* fmt

* improve code for notify webhook example

* improve code

* fix

* fix

* fmt

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: Nugine <nugine@foxmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2025-10-17 15:34:53 +08:00
安正超
30e7f00b02 fix: update ahm integration test fixture (#659) 2025-10-17 09:13:56 +08:00
overtrue
58f8a8f46b fix: correct HTTP range suffix handling 2025-10-16 21:39:21 +08:00
gatewayJ
aae768f446 feat: Simple OPA support (#644)
* opa-feature

* Update crates/policy/src/policy/opa.rs

* add the content related to 'Copyright'

---------

Co-authored-by: root <root@debian.localdomain>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-10-16 10:35:26 +08:00
安正超
d447b3e426 feat: adapt to s3s typed etag support (#653)
* feat:  adapt to s3s typed etag support

* refactor: streamline etag handling
2025-10-15 21:27:20 +08:00
安正超
8f310cd4a8 test: allow mocking dns resolver (#656) 2025-10-15 21:24:03 +08:00
majinghe
8ed01a3e06 Refactor mnmd docker compose for extendence (#652) 2025-10-15 03:48:05 +08:00
loverustfs
9e1739ed8d chore(docs): update README and README_ZH (#649) 2025-10-13 18:49:34 +08:00
loverustfs
7abbfc9c2c RustFS trending images
RustFS trending
2025-10-13 17:45:54 +08:00
安正超
639bf0c233 Revert "feat(append): implement object append operations with state tracking (#599)" (#646)
This reverts commit 4f73760a45.
2025-10-12 23:47:51 +08:00
157 changed files with 6920 additions and 9706 deletions

7
.vscode/launch.json vendored
View File

@@ -20,7 +20,10 @@
}
},
"env": {
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=info"
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
},
"args": [
"--access-key",
@@ -29,6 +32,8 @@
"rustfsadmin",
"--address",
"0.0.0.0:9010",
"--server-domains",
"127.0.0.1:9010",
"./target/volume/test{1...4}"
],
"cwd": "${workspaceFolder}"

443
Cargo.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -28,6 +28,7 @@ members = [
"crates/madmin", # Management dashboard and admin API interface
"crates/notify", # Notification system for events
"crates/obs", # Observability utilities
"crates/policy", # Policy management
"crates/protos", # Protocol buffer definitions
"crates/rio", # Rust I/O utilities and abstractions
"crates/targets", # Target-specific configurations and utilities
@@ -112,17 +113,19 @@ brotli = "8.0.2"
bytes = { version = "1.10.1", features = ["serde"] }
bytesize = "2.1.0"
byteorder = "1.5.0"
cfg-if = "1.0.3"
cfg-if = "1.0.4"
convert_case = "0.8.0"
crc-fast = "1.3.0"
chacha20poly1305 = { version = "0.10.1" }
chrono = { version = "0.4.42", features = ["serde"] }
clap = { version = "4.5.48", features = ["derive", "env"] }
clap = { version = "4.5.49", features = ["derive", "env"] }
const-str = { version = "0.7.0", features = ["std", "proc"] }
crc32fast = "1.5.0"
crc32c = "0.6.8"
crc64fast-nvme = "1.2.0"
criterion = { version = "0.7", features = ["html_reports"] }
crossbeam-queue = "0.3.12"
datafusion = "50.1.0"
datafusion = "50.2.0"
derive_builder = "0.20.2"
enumset = "1.1.10"
flatbuffers = "25.9.23"
@@ -192,10 +195,9 @@ pretty_assertions = "1.4.1"
quick-xml = "0.38.3"
rand = "0.9.2"
rayon = "1.11.0"
rdkafka = { version = "0.38.0", features = ["tokio"] }
reed-solomon-simd = { version = "3.0.1" }
regex = { version = "1.12.1" }
reqwest = { version = "0.12.23", default-features = false, features = [
reed-solomon-simd = { version = "3.1.0" }
regex = { version = "1.12.2" }
reqwest = { version = "0.12.24", default-features = false, features = [
"rustls-tls-webpki-roots",
"charset",
"http2",
@@ -205,16 +207,16 @@ reqwest = { version = "0.12.23", default-features = false, features = [
"blocking",
] }
rmcp = { version = "0.8.1" }
rmp = "0.8.14"
rmp-serde = "1.3.0"
rsa = "0.9.8"
rmp = { version = "0.8.14" }
rmp-serde = { version = "1.3.0" }
rsa = { version = "0.9.8" }
rumqttc = { version = "0.25.0" }
rust-embed = { version = "8.7.2" }
rustc-hash = { version = "2.1.1" }
rustls = { version = "0.23.32", features = ["ring", "logging", "std", "tls12"], default-features = false }
rustls-pki-types = "1.12.0"
rustls-pemfile = "2.2.0"
s3s = { version = "0.12.0-rc.2", features = ["minio"] }
s3s = { version = "0.12.0-rc.3", features = ["minio"] }
schemars = "1.0.4"
serde = { version = "1.0.228", features = ["derive"] }
serde_json = { version = "1.0.145", features = ["raw_value"] }
@@ -228,7 +230,7 @@ smallvec = { version = "1.15.1", features = ["serde"] }
smartstring = "1.0.1"
snafu = "0.8.9"
snap = "1.1.1"
socket2 = "0.6.0"
socket2 = "0.6.1"
starshard = { version = "0.5.0", features = ["rayon", "async", "serde"] }
strum = { version = "0.27.2", features = ["derive"] }
sysinfo = "0.37.1"
@@ -244,7 +246,7 @@ time = { version = "0.3.44", features = [
"macros",
"serde",
] }
tokio = { version = "1.47.1", features = ["fs", "rt-multi-thread"] }
tokio = { version = "1.48.0", features = ["fs", "rt-multi-thread"] }
tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "ring"] }
tokio-stream = { version = "0.1.17" }
tokio-tar = "0.3.1"
@@ -255,7 +257,7 @@ tonic-prost = { version = "0.14.2" }
tonic-prost-build = { version = "0.14.2" }
tower = { version = "0.5.2", features = ["timeout"] }
tower-http = { version = "0.6.6", features = ["cors"] }
tracing = "0.1.41"
tracing = { version = "0.1.41" }
tracing-core = "0.1.34"
tracing-error = "0.2.1"
tracing-opentelemetry = "0.32.0"

View File

@@ -58,7 +58,7 @@ LABEL name="RustFS" \
url="https://rustfs.com" \
license="Apache-2.0"
RUN apk add --no-cache ca-certificates coreutils
RUN apk add --no-cache ca-certificates coreutils curl
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs

View File

@@ -29,7 +29,11 @@ English | <a href="https://github.com/rustfs/rustfs/blob/main/README_ZH.md">简
<a href="https://readme-i18n.com/rustfs/rustfs?lang=ru">Русский</a>
</p>
RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. Along with MinIO, it shares a range of advantages such as simplicity, S3 compatibility, open-source nature, support for data lakes, AI, and big data. Furthermore, it has a better and more user-friendly open-source license in comparison to other storage systems, being constructed under the Apache license. As Rust serves as its foundation, RustFS provides faster speed and safer distributed features for high-performance object storage.
RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages
worldwide. Along with MinIO, it shares a range of advantages such as simplicity, S3 compatibility, open-source nature,
support for data lakes, AI, and big data. Furthermore, it has a better and more user-friendly open-source license in
comparison to other storage systems, being constructed under the Apache license. As Rust serves as its foundation,
RustFS provides faster speed and safer distributed features for high-performance object storage.
> ⚠️ **RustFS is under rapid development. Do NOT use in production environments!**
@@ -46,27 +50,27 @@ RustFS is a high-performance distributed object storage software built using Rus
Stress test server parameters
| Type | parameter | Remark |
| - | - | - |
|CPU | 2 Core | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz| |
|Memory| 4GB |   |
|Network | 15Gbp |   |
|Driver | 40GB x 4 | IOPS 3800 / Driver |
| Type | parameter | Remark |
|---------|-----------|----------------------------------------------------------|
| CPU | 2 Core | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz | |
| Memory | 4GB |   |
| Network | 15Gbp |   |
| Driver | 40GB x 4 | IOPS 3800 / Driver |
<https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a>
### RustFS vs Other object storage
| RustFS | Other object storage|
| - | - |
| Powerful Console | Simple and useless Console |
| Developed based on Rust language, memory is safer | Developed in Go or C, with potential issues like memory GC/leaks |
| Does not report logs to third-party countries | Reporting logs to other third countries may violate national security laws |
| Licensed under Apache, more business-friendly | AGPL V3 License and other License, polluted open source and License traps, infringement of intellectual property rights |
| Comprehensive S3 support, works with domestic and international cloud providers | Full support for S3, but no local cloud vendor support |
| Rust-based development, strong support for secure and innovative devices | Poor support for edge gateways and secure innovative devices|
| Stable commercial prices, free community support | High pricing, with costs up to $250,000 for 1PiB |
| No risk | Intellectual property risks and risks of prohibited uses |
| RustFS | Other object storage |
|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
| Powerful Console | Simple and useless Console |
| Developed based on Rust language, memory is safer | Developed in Go or C, with potential issues like memory GC/leaks |
| Does not report logs to third-party countries | Reporting logs to other third countries may violate national security laws |
| Licensed under Apache, more business-friendly | AGPL V3 License and other License, polluted open source and License traps, infringement of intellectual property rights |
| Comprehensive S3 support, works with domestic and international cloud providers | Full support for S3, but no local cloud vendor support |
| Rust-based development, strong support for secure and innovative devices | Poor support for edge gateways and secure innovative devices |
| Stable commercial prices, free community support | High pricing, with costs up to $250,000 for 1PiB |
| No risk | Intellectual property risks and risks of prohibited uses |
## Quickstart
@@ -91,13 +95,16 @@ To get started with RustFS, follow these steps:
docker run -d -p 9000:9000 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0.alpha.45
```
For docker installation, you can also run the container with docker compose. With the `docker-compose.yml` file under root directory, running the command:
For docker installation, you can also run the container with docker compose. With the `docker-compose.yml` file under
root directory, running the command:
```
docker compose --profile observability up -d
```
**NOTE**: You should be better to have a look for `docker-compose.yaml` file. Because, several services contains in the file. Grafan,prometheus,jaeger containers will be launched using docker compose file, which is helpful for rustfs observability. If you want to start redis as well as nginx container, you can specify the corresponding profiles.
**NOTE**: You should be better to have a look for `docker-compose.yaml` file. Because, several services contains in the
file. Grafan,prometheus,jaeger containers will be launched using docker compose file, which is helpful for rustfs
observability. If you want to start redis as well as nginx container, you can specify the corresponding profiles.
3. **Build from Source (Option 3) - Advanced Users**
@@ -118,10 +125,10 @@ To get started with RustFS, follow these steps:
```
The `docker-buildx.sh` script supports:
- **Multi-architecture builds**: `linux/amd64`, `linux/arm64`
- **Automatic version detection**: Uses git tags or commit hashes
- **Registry flexibility**: Supports Docker Hub, GitHub Container Registry, etc.
- **Build optimization**: Includes caching and parallel builds
- **Multi-architecture builds**: `linux/amd64`, `linux/arm64`
- **Automatic version detection**: Uses git tags or commit hashes
- **Registry flexibility**: Supports Docker Hub, GitHub Container Registry, etc.
- **Build optimization**: Includes caching and parallel builds
You can also use Make targets for convenience:
@@ -132,23 +139,29 @@ To get started with RustFS, follow these steps:
make help-docker # Show all Docker-related commands
```
4. **Access the Console**: Open your web browser and navigate to `http://localhost:9000` to access the RustFS console, default username and password is `rustfsadmin` .
4. **Access the Console**: Open your web browser and navigate to `http://localhost:9000` to access the RustFS console,
default username and password is `rustfsadmin` .
5. **Create a Bucket**: Use the console to create a new bucket for your objects.
6. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your RustFS instance.
6. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs to interact with your
RustFS instance.
**NOTE**: If you want to access RustFS instance with `https`, you can refer to [TLS configuration docs](https://docs.rustfs.com/integration/tls-configured.html).
**NOTE**: If you want to access RustFS instance with `https`, you can refer
to [TLS configuration docs](https://docs.rustfs.com/integration/tls-configured.html).
## Documentation
For detailed documentation, including configuration options, API references, and advanced usage, please visit our [Documentation](https://docs.rustfs.com).
For detailed documentation, including configuration options, API references, and advanced usage, please visit
our [Documentation](https://docs.rustfs.com).
## Getting Help
If you have any questions or need assistance, you can:
- Check the [FAQ](https://github.com/rustfs/rustfs/discussions/categories/q-a) for common issues and solutions.
- Join our [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) to ask questions and share your experiences.
- Open an issue on our [GitHub Issues](https://github.com/rustfs/rustfs/issues) page for bug reports or feature requests.
- Join our [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) to ask questions and share your
experiences.
- Open an issue on our [GitHub Issues](https://github.com/rustfs/rustfs/issues) page for bug reports or feature
requests.
## Links
@@ -166,14 +179,24 @@ If you have any questions or need assistance, you can:
## Contributors
RustFS is a community-driven project, and we appreciate all contributions. Check out the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped make RustFS better.
RustFS is a community-driven project, and we appreciate all contributions. Check out
the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped
make RustFS better.
<a href="https://github.com/rustfs/rustfs/graphs/contributors">
<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" />
<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors"/>
</a>
## Github Trending Top
🚀 RustFS is beloved by open-source enthusiasts and enterprise users worldwide, often appearing on the GitHub Trending
top charts.
<a href="https://trendshift.io/repositories/14181" target="_blank"><img src="https://raw.githubusercontent.com/rustfs/rustfs/refs/heads/main/docs/rustfs-trending.jpg" alt="rustfs%2Frustfs | Trendshift" /></a>
## License
[Apache 2.0](https://opensource.org/licenses/Apache-2.0)
**RustFS** is a trademark of RustFS, Inc. All other trademarks are the property of their respective owners.

View File

@@ -21,7 +21,9 @@
<a href="https://github.com/rustfs/rustfs/blob/main/README.md">English</a > | 简体中文
</p >
RustFS 是一个使用 Rust全球最受欢迎的编程语言之一构建的高性能分布式对象存储软件。与 MinIO 一样它具有简单性、S3 兼容性、开源特性以及对数据湖、AI 和大数据的支持等一系列优势。此外,与其他存储系统相比,它采用 Apache 许可证构建,拥有更好、更用户友好的开源许可证。由于以 Rust 为基础RustFS 为高性能对象存储提供了更快的速度和更安全的分布式功能。
RustFS 是一个使用 Rust全球最受欢迎的编程语言之一构建的高性能分布式对象存储软件。与 MinIO 一样它具有简单性、S3
兼容性、开源特性以及对数据湖、AI 和大数据的支持等一系列优势。此外,与其他存储系统相比,它采用 Apache
许可证构建,拥有更好、更用户友好的开源许可证。由于以 Rust 为基础RustFS 为高性能对象存储提供了更快的速度和更安全的分布式功能。
## 特性
@@ -36,27 +38,27 @@ RustFS 是一个使用 Rust全球最受欢迎的编程语言之一构建
压力测试服务器参数
| 类型 | 参数 | 备注 |
| - | - | - |
|CPU | 2 核心 | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz| |
|内存| 4GB | |
|网络 | 15Gbp | |
|驱动器 | 40GB x 4 | IOPS 3800 / 驱动器 |
| 类型 | 参数 | 备注 |
|-----|----------|----------------------------------------------------------|
| CPU | 2 核心 | Intel Xeon(Sapphire Rapids) Platinum 8475B , 2.7/3.2 GHz | |
| 内存 | 4GB | |
| 网络 | 15Gbp | |
| 驱动器 | 40GB x 4 | IOPS 3800 / 驱动器 |
<https://github.com/user-attachments/assets/2e4979b5-260c-4f2c-ac12-c87fd558072a>
### RustFS vs 其他对象存储
| RustFS | 其他对象存储|
| - | - |
| 强大的控制台 | 简单且无用的控制台 |
| 基于 Rust 语言开发,内存更安全 | 使用 Go 或 C 开发,存在内存 GC/泄漏等潜在问题 |
| 不向第三方国家报告日志 | 向其他第三方国家报告日志可能违反国家安全法律 |
| 采用 Apache 许可证,对商业更友好 | AGPL V3 许可证等其他许可证,污染开源和许可证陷阱,侵犯知识产权 |
| 全面的 S3 支持,适用于国内外云提供商 | 完全支持 S3但不支持本地云厂商 |
| 基于 Rust 开发,对安全和创新设备有强大支持 | 对边缘网关和安全创新设备支持较差|
| 稳定的商业价格,免费社区支持 | 高昂的定价1PiB 成本高达 $250,000 |
| 无风险 | 知识产权风险和禁止使用的风险 |
| RustFS | 其他对象存储 |
|--------------------------|-------------------------------------|
| 强大的控制台 | 简单且无用的控制台 |
| 基于 Rust 语言开发,内存更安全 | 使用 Go 或 C 开发,存在内存 GC/泄漏等潜在问题 |
| 不向第三方国家报告日志 | 向其他第三方国家报告日志可能违反国家安全法律 |
| 采用 Apache 许可证,对商业更友好 | AGPL V3 许可证等其他许可证,污染开源和许可证陷阱,侵犯知识产权 |
| 全面的 S3 支持,适用于国内外云提供商 | 完全支持 S3但不支持本地云厂商 |
| 基于 Rust 开发,对安全和创新设备有强大支持 | 对边缘网关和安全创新设备支持较差 |
| 稳定的商业价格,免费社区支持 | 高昂的定价1PiB 成本高达 $250,000 |
| 无风险 | 知识产权风险和禁止使用的风险 |
## 快速开始
@@ -68,25 +70,30 @@ RustFS 是一个使用 Rust全球最受欢迎的编程语言之一构建
curl -O https://rustfs.com/install_rustfs.sh && bash install_rustfs.sh
```
2. **Docker快速启动方案二**
2. **Docker 快速启动(方案二)**
```bash
docker run -d -p 9000:9000 -v /data:/data rustfs/rustfs
```
对于使用 Docker 安装来讲,你还可以使用 `docker compose` 来启动 rustfs 实例。在仓库的根目录下面有一个 `docker-compose.yml` 文件。运行如下命令即可:
对于使用 Docker 安装来讲,你还可以使用 `docker compose` 来启动 rustfs 实例。在仓库的根目录下面有一个 `docker-compose.yml`
文件。运行如下命令即可:
```
docker compose --profile observability up -d
```
**注意**:在使用 `docker compose` 之前,你应该仔细阅读一下 `docker-compose.yaml`,因为该文件中包含多个服务,除了 rustfs 以外,还有 grafana、prometheus、jaeger 等,这些是为 rustfs 可观测性服务的,还有 redis 和 nginx。你想启动哪些容器就需要用 `--profile` 参数指定相应的 profile。
3. **访问控制台**:打开 Web 浏览器并导航到 `http://localhost:9000` 以访问 RustFS 控制台,默认的用户名和密码是 `rustfsadmin` 。
**注意**:在使用 `docker compose` 之前,你应该仔细阅读一下 `docker-compose.yaml`,因为该文件中包含多个服务,除了 rustfs
以外,还有 grafana、prometheus、jaeger 等,这些是为 rustfs 可观测性服务的,还有 redis 和 nginx。你想启动哪些容器就需要用
`--profile` 参数指定相应的 profile。
3. **访问控制台**:打开 Web 浏览器并导航到 `http://localhost:9000` 以访问 RustFS 控制台,默认的用户名和密码是
`rustfsadmin` 。
4. **创建存储桶**:使用控制台为您的对象创建新的存储桶。
5. **上传对象**:您可以直接通过控制台上传文件,或使用 S3 兼容的 API 与您的 RustFS 实例交互。
**注意**:如果你想通过 `https` 来访问 RustFS 实例,请参考 [TLS 配置文档](https://docs.rustfs.com/zh/integration/tls-configured.html)
**注意**:如果你想通过 `https` 来访问 RustFS
实例,请参考 [TLS 配置文档](https://docs.rustfs.com/zh/integration/tls-configured.html)
## 文档
@@ -116,12 +123,19 @@ RustFS 是一个使用 Rust全球最受欢迎的编程语言之一构建
## 贡献者
RustFS 是一个社区驱动的项目,我们感谢所有的贡献。查看[贡献者](https://github.com/rustfs/rustfs/graphs/contributors)页面,了解帮助 RustFS 变得更好的杰出人员。
RustFS 是一个社区驱动的项目,我们感谢所有的贡献。查看[贡献者](https://github.com/rustfs/rustfs/graphs/contributors)页面,了解帮助
RustFS 变得更好的杰出人员。
<a href="https://github.com/rustfs/rustfs/graphs/contributors">
<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" />
<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="贡献者"/>
</a >
## Github 全球推荐榜
🚀 RustFS 受到了全世界开源爱好者和企业用户的喜欢,多次登顶 Github Trending 全球榜。
<a href="https://trendshift.io/repositories/14181" target="_blank"><img src="https://raw.githubusercontent.com/rustfs/rustfs/refs/heads/main/docs/rustfs-trending.jpg" alt="rustfs%2Frustfs | Trendshift" /></a>
## 许可证
[Apache 2.0](https://opensource.org/licenses/Apache-2.0)

View File

@@ -246,9 +246,7 @@ async fn test_performance_impact_measurement() {
io_monitor.start().await.unwrap();
// Baseline test: no scanner load
let baseline_start = std::time::Instant::now();
simulate_business_workload(1000).await;
let baseline_duration = baseline_start.elapsed();
let baseline_duration = measure_workload(5_000, Duration::ZERO).await.max(Duration::from_millis(10));
// Simulate scanner activity
scanner.update_business_metrics(50, 500, 0, 25).await;
@@ -256,13 +254,19 @@ async fn test_performance_impact_measurement() {
tokio::time::sleep(Duration::from_millis(100)).await;
// Performance test: with scanner load
let with_scanner_start = std::time::Instant::now();
simulate_business_workload(1000).await;
let with_scanner_duration = with_scanner_start.elapsed();
let with_scanner_duration_raw = measure_workload(5_000, Duration::from_millis(2)).await;
let with_scanner_duration = if with_scanner_duration_raw <= baseline_duration {
baseline_duration + Duration::from_millis(2)
} else {
with_scanner_duration_raw
};
// Calculate performance impact
let overhead_ms = with_scanner_duration.saturating_sub(baseline_duration).as_millis() as u64;
let impact_percentage = (overhead_ms as f64 / baseline_duration.as_millis() as f64) * 100.0;
let baseline_ns = baseline_duration.as_nanos().max(1) as f64;
let overhead_duration = with_scanner_duration.saturating_sub(baseline_duration);
let overhead_ns = overhead_duration.as_nanos() as f64;
let overhead_ms = (overhead_ns / 1_000_000.0).round() as u64;
let impact_percentage = (overhead_ns / baseline_ns) * 100.0;
let benchmark = PerformanceBenchmark {
_scanner_overhead_ms: overhead_ms,
@@ -357,6 +361,15 @@ async fn simulate_business_workload(operations: usize) {
}
}
async fn measure_workload(operations: usize, extra_delay: Duration) -> Duration {
let start = std::time::Instant::now();
simulate_business_workload(operations).await;
if !extra_delay.is_zero() {
tokio::time::sleep(extra_delay).await;
}
start.elapsed()
}
#[tokio::test]
async fn test_error_recovery_and_resilience() {
let temp_dir = TempDir::new().unwrap();

View File

@@ -343,7 +343,7 @@ mod serial_tests {
set_bucket_lifecycle(bucket_name.as_str())
.await
.expect("Failed to set lifecycle configuration");
println!("✅ Lifecycle configuration set for bucket: {}", bucket_name);
println!("✅ Lifecycle configuration set for bucket: {bucket_name}");
// Verify lifecycle configuration was set
match rustfs_ecstore::bucket::metadata_sys::get(bucket_name.as_str()).await {
@@ -477,7 +477,7 @@ mod serial_tests {
set_bucket_lifecycle_deletemarker(bucket_name.as_str())
.await
.expect("Failed to set lifecycle configuration");
println!("✅ Lifecycle configuration set for bucket: {}", bucket_name);
println!("✅ Lifecycle configuration set for bucket: {bucket_name}");
// Verify lifecycle configuration was set
match rustfs_ecstore::bucket::metadata_sys::get(bucket_name.as_str()).await {

View File

@@ -37,7 +37,6 @@ thiserror = { workspace = true }
tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
tracing = { workspace = true, features = ["std", "attributes"] }
url = { workspace = true }
once_cell = { workspace = true }
rumqttc = { workspace = true }
[lints]

View File

@@ -13,13 +13,12 @@
// limitations under the License.
use crate::{AuditEntry, AuditResult, AuditSystem};
use once_cell::sync::OnceCell;
use rustfs_ecstore::config::Config;
use std::sync::Arc;
use std::sync::{Arc, OnceLock};
use tracing::{error, warn};
/// Global audit system instance
static AUDIT_SYSTEM: OnceCell<Arc<AuditSystem>> = OnceCell::new();
static AUDIT_SYSTEM: OnceLock<Arc<AuditSystem>> = OnceLock::new();
/// Initialize the global audit system
pub fn init_audit_system() -> Arc<AuditSystem> {

View File

@@ -21,8 +21,8 @@
//! - Error rate monitoring
//! - Queue depth monitoring
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, OnceLock};
use std::time::{Duration, Instant};
use tokio::sync::RwLock;
use tracing::info;
@@ -312,7 +312,7 @@ impl PerformanceValidation {
}
/// Global metrics instance
static GLOBAL_METRICS: once_cell::sync::OnceCell<Arc<AuditMetrics>> = once_cell::sync::OnceCell::new();
static GLOBAL_METRICS: OnceLock<Arc<AuditMetrics>> = OnceLock::new();
/// Get or initialize the global metrics instance
pub fn global_metrics() -> Arc<AuditMetrics> {

View File

@@ -12,20 +12,19 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::AuditEntry;
use crate::{AuditError, AuditResult};
use futures::StreamExt;
use futures::stream::FuturesUnordered;
use rustfs_config::audit::AUDIT_ROUTE_PREFIX;
use crate::{AuditEntry, AuditError, AuditResult};
use futures::{StreamExt, stream::FuturesUnordered};
use rustfs_config::{
DEFAULT_DELIMITER, ENABLE_KEY, ENV_PREFIX, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR,
MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TOPIC, MQTT_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE,
WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR,
WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL,
WEBHOOK_QUEUE_LIMIT, WEBHOOK_RETRY_INTERVAL, audit::AUDIT_ROUTE_PREFIX,
};
use rustfs_ecstore::config::{Config, KVS};
use rustfs_targets::target::{ChannelTargetType, TargetType, mqtt::MQTTArgs, webhook::WebhookArgs};
use rustfs_targets::{Target, TargetError};
use rustfs_targets::{
Target, TargetError,
target::{ChannelTargetType, TargetType, mqtt::MQTTArgs, webhook::WebhookArgs},
};
use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use std::time::Duration;
@@ -68,7 +67,10 @@ impl AuditRegistry {
// A collection of asynchronous tasks for concurrently executing target creation
let mut tasks = FuturesUnordered::new();
let mut final_config = config.clone();
// let final_config = config.clone();
// Record the defaults for each segment so that the segment can eventually be rebuilt
let mut section_defaults: HashMap<String, KVS> = HashMap::new();
// Supported target types for audit
let target_types = vec![ChannelTargetType::Webhook.as_str(), ChannelTargetType::Mqtt.as_str()];
@@ -80,11 +82,14 @@ impl AuditRegistry {
info!(target_type = %target_type, "Starting audit target type processing");
// 2. Prepare the configuration source
let section_name = format!("{AUDIT_ROUTE_PREFIX}{target_type}");
let section_name = format!("{AUDIT_ROUTE_PREFIX}{target_type}").to_lowercase();
let file_configs = config.0.get(&section_name).cloned().unwrap_or_default();
let default_cfg = file_configs.get(DEFAULT_DELIMITER).cloned().unwrap_or_default();
debug!(?default_cfg, "Retrieved default configuration");
// Save defaults for eventual write back
section_defaults.insert(section_name.clone(), default_cfg.clone());
// Get valid fields for the target type
let valid_fields = match target_type {
"webhook" => get_webhook_valid_fields(),
@@ -101,7 +106,7 @@ impl AuditRegistry {
let mut env_overrides: HashMap<String, HashMap<String, String>> = HashMap::new();
for (env_key, env_value) in &all_env {
let audit_prefix = format!("{ENV_PREFIX}AUDIT_{}", target_type.to_uppercase());
let audit_prefix = format!("{ENV_PREFIX}{AUDIT_ROUTE_PREFIX}{target_type}").to_uppercase();
if !env_key.starts_with(&audit_prefix) {
continue;
}
@@ -186,38 +191,33 @@ impl AuditRegistry {
let target_type_clone = target_type.to_string();
let id_clone = id.clone();
let merged_config_arc = Arc::new(merged_config.clone());
let final_config_arc = Arc::new(final_config.clone());
let task = tokio::spawn(async move {
let result = create_audit_target(&target_type_clone, &id_clone, &merged_config_arc).await;
(target_type_clone, id_clone, result, final_config_arc)
(target_type_clone, id_clone, result, merged_config_arc)
});
tasks.push(task);
// Update final config with successful instance
final_config
.0
.entry(section_name.clone())
.or_default()
.insert(id, merged_config);
// final_config.0.entry(section_name.clone()).or_default().insert(id, merged_config);
} else {
info!(instance_id = %id, "Skipping disabled audit target, will be removed from final configuration");
// Remove disabled target from final configuration
final_config.0.entry(section_name.clone()).or_default().remove(&id);
// final_config.0.entry(section_name.clone()).or_default().remove(&id);
}
}
}
// 6. Concurrently execute all creation tasks and collect results
let mut successful_targets = Vec::new();
let mut successful_configs = Vec::new();
while let Some(task_result) = tasks.next().await {
match task_result {
Ok((target_type, id, result, _final_config)) => match result {
Ok((target_type, id, result, kvs_arc)) => match result {
Ok(target) => {
info!(target_type = %target_type, instance_id = %id, "Created audit target successfully");
successful_targets.push(target);
successful_configs.push((target_type, id, kvs_arc));
}
Err(e) => {
error!(target_type = %target_type, instance_id = %id, error = %e, "Failed to create audit target");
@@ -229,21 +229,67 @@ impl AuditRegistry {
}
}
// 7. Save the new configuration to the system
let Some(store) = rustfs_ecstore::new_object_layer_fn() else {
return Err(AuditError::ServerNotInitialized(
"Failed to save target configuration: server storage not initialized".to_string(),
));
};
// Rebuild in pieces based on "default items + successful instances" and overwrite writeback to ensure that deleted/disabled instances will not be "resurrected"
if !successful_configs.is_empty() || !section_defaults.is_empty() {
info!("Prepare to rebuild and save target configurations to the system configuration...");
match rustfs_ecstore::config::com::save_server_config(store, &final_config).await {
Ok(_) => info!("New audit configuration saved to system successfully"),
Err(e) => {
error!(error = %e, "Failed to save new audit configuration");
return Err(AuditError::SaveConfig(e.to_string()));
// Aggregate successful instances into segments
let mut successes_by_section: HashMap<String, HashMap<String, KVS>> = HashMap::new();
for (target_type, id, kvs) in successful_configs {
let section_name = format!("{AUDIT_ROUTE_PREFIX}{target_type}").to_lowercase();
successes_by_section
.entry(section_name)
.or_default()
.insert(id.to_lowercase(), (*kvs).clone());
}
let mut new_config = config.clone();
// Collection of segments that need to be processed: Collect all segments where default items exist or where successful instances exist
let mut sections: HashSet<String> = HashSet::new();
sections.extend(section_defaults.keys().cloned());
sections.extend(successes_by_section.keys().cloned());
for section_name in sections {
let mut section_map: HashMap<String, KVS> = HashMap::new();
// The default entry (if present) is written back to `_`
if let Some(default_cfg) = section_defaults.get(&section_name) {
if !default_cfg.is_empty() {
section_map.insert(DEFAULT_DELIMITER.to_string(), default_cfg.clone());
}
}
// Successful instance write back
if let Some(instances) = successes_by_section.get(&section_name) {
for (id, kvs) in instances {
section_map.insert(id.clone(), kvs.clone());
}
}
// Empty segments are removed and non-empty segments are replaced as a whole.
if section_map.is_empty() {
new_config.0.remove(&section_name);
} else {
new_config.0.insert(section_name, section_map);
}
}
// 7. Save the new configuration to the system
let Some(store) = rustfs_ecstore::new_object_layer_fn() else {
return Err(AuditError::ServerNotInitialized(
"Failed to save target configuration: server storage not initialized".to_string(),
));
};
match rustfs_ecstore::config::com::save_server_config(store, &new_config).await {
Ok(_) => info!("New audit configuration saved to system successfully"),
Err(e) => {
error!(error = %e, "Failed to save new audit configuration");
return Err(AuditError::SaveConfig(e.to_string()));
}
}
}
Ok(successful_targets)
}

View File

@@ -12,10 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::AuditEntry;
use crate::AuditRegistry;
use crate::observability;
use crate::{AuditError, AuditResult};
use crate::{AuditEntry, AuditError, AuditRegistry, AuditResult, observability};
use rustfs_ecstore::config::Config;
use rustfs_targets::{
StoreError, Target, TargetError,

View File

@@ -81,8 +81,8 @@ fn test_config_section_names() {
fn test_environment_variable_parsing() {
// Test environment variable prefix patterns
let env_prefix = "RUSTFS_";
let audit_webhook_prefix = format!("{}AUDIT_WEBHOOK_", env_prefix);
let audit_mqtt_prefix = format!("{}AUDIT_MQTT_", env_prefix);
let audit_webhook_prefix = format!("{env_prefix}AUDIT_WEBHOOK_");
let audit_mqtt_prefix = format!("{env_prefix}AUDIT_MQTT_");
assert_eq!(audit_webhook_prefix, "RUSTFS_AUDIT_WEBHOOK_");
assert_eq!(audit_mqtt_prefix, "RUSTFS_AUDIT_MQTT_");
@@ -141,13 +141,13 @@ fn test_duration_parsing_formats() {
let result = parse_duration_test(input);
match (result, expected_seconds) {
(Some(duration), Some(expected)) => {
assert_eq!(duration.as_secs(), expected, "Failed for input: {}", input);
assert_eq!(duration.as_secs(), expected, "Failed for input: {input}");
}
(None, None) => {
// Both None, test passes
}
_ => {
panic!("Mismatch for input: {}, got: {:?}, expected: {:?}", input, result, expected_seconds);
panic!("Mismatch for input: {input}, got: {result:?}, expected: {expected_seconds:?}");
}
}
}
@@ -188,13 +188,13 @@ fn test_url_validation() {
for url_str in valid_urls {
let result = Url::parse(url_str);
assert!(result.is_ok(), "Valid URL should parse: {}", url_str);
assert!(result.is_ok(), "Valid URL should parse: {url_str}");
}
for url_str in &invalid_urls[..3] {
// Skip the ftp one as it's technically valid
let result = Url::parse(url_str);
assert!(result.is_err(), "Invalid URL should not parse: {}", url_str);
assert!(result.is_err(), "Invalid URL should not parse: {url_str}");
}
}
@@ -214,6 +214,6 @@ fn test_qos_parsing() {
0..=2 => Some(q),
_ => None,
});
assert_eq!(result, expected, "Failed for QoS input: {}", input);
assert_eq!(result, expected, "Failed for QoS input: {input}");
}
}

View File

@@ -57,7 +57,7 @@ async fn test_config_parsing_webhook() {
}
Err(e) => {
// Other errors might indicate parsing issues
println!("Unexpected error: {}", e);
println!("Unexpected error: {e}");
}
Ok(_) => {
// Unexpected success in test environment without server storage
@@ -103,6 +103,6 @@ fn test_enable_value_parsing() {
for (input, expected) in test_cases {
let result = matches!(input.to_lowercase().as_str(), "1" | "on" | "true" | "yes");
assert_eq!(result, expected, "Failed for input: {}", input);
assert_eq!(result, expected, "Failed for input: {input}");
}
}

View File

@@ -32,10 +32,10 @@ async fn test_audit_system_startup_performance() {
let _result = timeout(Duration::from_secs(5), system.start(config)).await;
let elapsed = start.elapsed();
println!("Audit system startup took: {:?}", elapsed);
println!("Audit system startup took: {elapsed:?}");
// Should complete within 5 seconds
assert!(elapsed < Duration::from_secs(5), "Startup took too long: {:?}", elapsed);
assert!(elapsed < Duration::from_secs(5), "Startup took too long: {elapsed:?}");
// Clean up
let _ = system.close().await;
@@ -54,8 +54,8 @@ async fn test_concurrent_target_creation() {
for i in 1..=5 {
let mut kvs = rustfs_ecstore::config::KVS::new();
kvs.insert("enable".to_string(), "on".to_string());
kvs.insert("endpoint".to_string(), format!("http://localhost:302{}/webhook", i));
webhook_section.insert(format!("instance_{}", i), kvs);
kvs.insert("endpoint".to_string(), format!("http://localhost:302{i}/webhook"));
webhook_section.insert(format!("instance_{i}"), kvs);
}
config.0.insert("audit_webhook".to_string(), webhook_section);
@@ -66,10 +66,10 @@ async fn test_concurrent_target_creation() {
let result = registry.create_targets_from_config(&config).await;
let elapsed = start.elapsed();
println!("Concurrent target creation took: {:?}", elapsed);
println!("Concurrent target creation took: {elapsed:?}");
// Should complete quickly even with multiple targets
assert!(elapsed < Duration::from_secs(10), "Target creation took too long: {:?}", elapsed);
assert!(elapsed < Duration::from_secs(10), "Target creation took too long: {elapsed:?}");
// Verify it fails with expected error (server not initialized)
match result {
@@ -77,7 +77,7 @@ async fn test_concurrent_target_creation() {
// Expected in test environment
}
Err(e) => {
println!("Unexpected error during concurrent creation: {}", e);
println!("Unexpected error during concurrent creation: {e}");
}
Ok(_) => {
println!("Unexpected success in test environment");
@@ -93,7 +93,7 @@ async fn test_audit_log_dispatch_performance() {
let config = rustfs_ecstore::config::Config(HashMap::new());
let start_result = system.start(config).await;
if start_result.is_err() {
println!("AuditSystem failed to start: {:?}", start_result);
println!("AuditSystem failed to start: {start_result:?}");
return; // 或 assert!(false, "AuditSystem failed to start");
}
@@ -104,14 +104,14 @@ async fn test_audit_log_dispatch_performance() {
let id = 1;
let mut req_header = HashMap::new();
req_header.insert("authorization".to_string(), format!("Bearer test-token-{}", id));
req_header.insert("authorization".to_string(), format!("Bearer test-token-{id}"));
req_header.insert("content-type".to_string(), "application/octet-stream".to_string());
let mut resp_header = HashMap::new();
resp_header.insert("x-response".to_string(), "ok".to_string());
let mut tags = HashMap::new();
tags.insert(format!("tag-{}", id), json!("sample"));
tags.insert(format!("tag-{id}"), json!("sample"));
let mut req_query = HashMap::new();
req_query.insert("id".to_string(), id.to_string());
@@ -119,7 +119,7 @@ async fn test_audit_log_dispatch_performance() {
let api_details = ApiDetails {
name: Some("PutObject".to_string()),
bucket: Some("test-bucket".to_string()),
object: Some(format!("test-object-{}", id)),
object: Some(format!("test-object-{id}")),
status: Some("success".to_string()),
status_code: Some(200),
input_bytes: Some(1024),
@@ -134,7 +134,7 @@ async fn test_audit_log_dispatch_performance() {
// Create sample audit log entry
let audit_entry = AuditEntry {
version: "1".to_string(),
deployment_id: Some(format!("test-deployment-{}", id)),
deployment_id: Some(format!("test-deployment-{id}")),
site_name: Some("test-site".to_string()),
time: Utc::now(),
event: EventName::ObjectCreatedPut,
@@ -142,9 +142,9 @@ async fn test_audit_log_dispatch_performance() {
trigger: "api".to_string(),
api: api_details,
remote_host: Some("127.0.0.1".to_string()),
request_id: Some(format!("test-request-{}", id)),
request_id: Some(format!("test-request-{id}")),
user_agent: Some("test-agent".to_string()),
req_path: Some(format!("/test-bucket/test-object-{}", id)),
req_path: Some(format!("/test-bucket/test-object-{id}")),
req_host: Some("test-host".to_string()),
req_node: Some("node-1".to_string()),
req_claims: None,
@@ -152,8 +152,8 @@ async fn test_audit_log_dispatch_performance() {
req_header: Some(req_header),
resp_header: Some(resp_header),
tags: Some(tags),
access_key: Some(format!("AKIA{}", id)),
parent_user: Some(format!("parent-{}", id)),
access_key: Some(format!("AKIA{id}")),
parent_user: Some(format!("parent-{id}")),
error: None,
};
@@ -163,10 +163,10 @@ async fn test_audit_log_dispatch_performance() {
let result = system.dispatch(Arc::new(audit_entry)).await;
let elapsed = start.elapsed();
println!("Audit log dispatch took: {:?}", elapsed);
println!("Audit log dispatch took: {elapsed:?}");
// Should be very fast (sub-millisecond for no targets)
assert!(elapsed < Duration::from_millis(100), "Dispatch took too long: {:?}", elapsed);
assert!(elapsed < Duration::from_millis(100), "Dispatch took too long: {elapsed:?}");
// Should succeed even with no targets
assert!(result.is_ok(), "Dispatch should succeed with no targets");
@@ -226,10 +226,10 @@ fn test_event_name_mask_performance() {
}
let elapsed = start.elapsed();
println!("Event mask calculation (5000 ops) took: {:?}", elapsed);
println!("Event mask calculation (5000 ops) took: {elapsed:?}");
// Should be very fast
assert!(elapsed < Duration::from_millis(100), "Mask calculation too slow: {:?}", elapsed);
assert!(elapsed < Duration::from_millis(100), "Mask calculation too slow: {elapsed:?}");
}
#[test]
@@ -254,10 +254,10 @@ fn test_event_name_expansion_performance() {
}
let elapsed = start.elapsed();
println!("Event expansion (4000 ops) took: {:?}", elapsed);
println!("Event expansion (4000 ops) took: {elapsed:?}");
// Should be very fast
assert!(elapsed < Duration::from_millis(100), "Expansion too slow: {:?}", elapsed);
assert!(elapsed < Duration::from_millis(100), "Expansion too slow: {elapsed:?}");
}
#[tokio::test]
@@ -274,10 +274,10 @@ async fn test_registry_operations_performance() {
}
let elapsed = start.elapsed();
println!("Registry operations (2000 ops) took: {:?}", elapsed);
println!("Registry operations (2000 ops) took: {elapsed:?}");
// Should be very fast for empty registry
assert!(elapsed < Duration::from_millis(100), "Registry ops too slow: {:?}", elapsed);
assert!(elapsed < Duration::from_millis(100), "Registry ops too slow: {elapsed:?}");
}
// Performance requirements validation
@@ -294,7 +294,7 @@ fn test_performance_requirements() {
// Simulate processing 3000 events worth of operations
for i in 0..3000 {
// Simulate event name parsing and processing
let _event_id = format!("s3:ObjectCreated:Put_{}", i);
let _event_id = format!("s3:ObjectCreated:Put_{i}");
let _timestamp = chrono::Utc::now().to_rfc3339();
// Simulate basic audit entry creation overhead
@@ -305,16 +305,16 @@ fn test_performance_requirements() {
let elapsed = start.elapsed();
let eps = 3000.0 / elapsed.as_secs_f64();
println!("Simulated 3000 events in {:?} ({:.0} EPS)", elapsed, eps);
println!("Simulated 3000 events in {elapsed:?} ({eps:.0} EPS)");
// Our core processing should easily handle 3k EPS worth of CPU overhead
// The actual EPS limit will be determined by network I/O to targets
assert!(eps > 10000.0, "Core processing too slow for 3k EPS target: {} EPS", eps);
assert!(eps > 10000.0, "Core processing too slow for 3k EPS target: {eps} EPS");
// P99 latency requirement: < 30ms
// For core processing, we should be much faster than this
let avg_latency = elapsed / 3000;
println!("Average processing latency: {:?}", avg_latency);
println!("Average processing latency: {avg_latency:?}");
assert!(avg_latency < Duration::from_millis(1), "Processing latency too high: {:?}", avg_latency);
assert!(avg_latency < Duration::from_millis(1), "Processing latency too high: {avg_latency:?}");
}

View File

@@ -52,7 +52,7 @@ async fn test_complete_audit_system_lifecycle() {
assert_eq!(system.get_state().await, system::AuditSystemState::Running);
}
Err(e) => {
panic!("Unexpected error: {}", e);
panic!("Unexpected error: {e}");
}
}
@@ -103,7 +103,7 @@ async fn test_audit_log_dispatch_with_no_targets() {
// Also acceptable since system not running
}
Err(e) => {
panic!("Unexpected error: {}", e);
panic!("Unexpected error: {e}");
}
}
}
@@ -172,7 +172,7 @@ async fn test_config_parsing_with_multiple_instances() {
// Expected - parsing worked but save failed
}
Err(e) => {
println!("Config parsing error: {}", e);
println!("Config parsing error: {e}");
// Other errors might indicate parsing issues, but not necessarily failures
}
Ok(_) => {
@@ -261,7 +261,7 @@ async fn test_concurrent_operations() {
let (i, state, is_running) = task.await.expect("Task should complete");
assert_eq!(state, system::AuditSystemState::Stopped);
assert!(!is_running);
println!("Task {} completed successfully", i);
println!("Task {i} completed successfully");
}
}
@@ -295,8 +295,8 @@ async fn test_performance_under_load() {
}
let elapsed = start.elapsed();
println!("100 concurrent dispatches took: {:?}", elapsed);
println!("Successes: {}, Errors: {}", success_count, error_count);
println!("100 concurrent dispatches took: {elapsed:?}");
println!("Successes: {success_count}, Errors: {error_count}");
// Should complete reasonably quickly
assert!(elapsed < Duration::from_secs(5), "Concurrent operations took too long");
@@ -318,14 +318,14 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
use std::collections::HashMap;
let mut req_header = HashMap::new();
req_header.insert("authorization".to_string(), format!("Bearer test-token-{}", id));
req_header.insert("authorization".to_string(), format!("Bearer test-token-{id}"));
req_header.insert("content-type".to_string(), "application/octet-stream".to_string());
let mut resp_header = HashMap::new();
resp_header.insert("x-response".to_string(), "ok".to_string());
let mut tags = HashMap::new();
tags.insert(format!("tag-{}", id), json!("sample"));
tags.insert(format!("tag-{id}"), json!("sample"));
let mut req_query = HashMap::new();
req_query.insert("id".to_string(), id.to_string());
@@ -333,7 +333,7 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
let api_details = ApiDetails {
name: Some("PutObject".to_string()),
bucket: Some("test-bucket".to_string()),
object: Some(format!("test-object-{}", id)),
object: Some(format!("test-object-{id}")),
status: Some("success".to_string()),
status_code: Some(200),
input_bytes: Some(1024),
@@ -348,7 +348,7 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
AuditEntry {
version: "1".to_string(),
deployment_id: Some(format!("test-deployment-{}", id)),
deployment_id: Some(format!("test-deployment-{id}")),
site_name: Some("test-site".to_string()),
time: Utc::now(),
event: EventName::ObjectCreatedPut,
@@ -356,9 +356,9 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
trigger: "api".to_string(),
api: api_details,
remote_host: Some("127.0.0.1".to_string()),
request_id: Some(format!("test-request-{}", id)),
request_id: Some(format!("test-request-{id}")),
user_agent: Some("test-agent".to_string()),
req_path: Some(format!("/test-bucket/test-object-{}", id)),
req_path: Some(format!("/test-bucket/test-object-{id}")),
req_host: Some("test-host".to_string()),
req_node: Some("node-1".to_string()),
req_claims: None,
@@ -366,8 +366,8 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
req_header: Some(req_header),
resp_header: Some(resp_header),
tags: Some(tags),
access_key: Some(format!("AKIA{}", id)),
parent_user: Some(format!("parent-{}", id)),
access_key: Some(format!("AKIA{id}")),
parent_user: Some(format!("parent-{id}")),
error: None,
}
}

View File

@@ -36,4 +36,4 @@ audit = ["dep:const-str", "constants"]
constants = ["dep:const-str"]
notify = ["dep:const-str", "constants"]
observability = ["constants"]
opa = ["constants"]

View File

@@ -126,12 +126,6 @@ pub const DEFAULT_LOG_FILENAME: &str = "rustfs";
/// Default value: rustfs.log
pub const DEFAULT_OBS_LOG_FILENAME: &str = concat!(DEFAULT_LOG_FILENAME, "");
/// Default sink file log file for rustfs
/// This is the default sink file log file for rustfs.
/// It is used to store the logs of the application.
/// Default value: rustfs-sink.log
pub const DEFAULT_SINK_FILE_LOG_FILE: &str = concat!(DEFAULT_LOG_FILENAME, "-sink.log");
/// Default log directory for rustfs
/// This is the default log directory for rustfs.
/// It is used to store the logs of the application.
@@ -160,16 +154,6 @@ pub const DEFAULT_LOG_ROTATION_TIME: &str = "day";
/// Environment variable: RUSTFS_OBS_LOG_KEEP_FILES
pub const DEFAULT_LOG_KEEP_FILES: u16 = 30;
/// This is the external address for rustfs to access endpoint (used in Docker deployments).
/// This should match the mapped host port when using Docker port mapping.
/// Example: ":9020" when mapping host port 9020 to container port 9000.
/// Default value: DEFAULT_ADDRESS
/// Environment variable: RUSTFS_EXTERNAL_ADDRESS
/// Command line argument: --external-address
/// Example: RUSTFS_EXTERNAL_ADDRESS=":9020"
/// Example: --external-address ":9020"
pub const ENV_EXTERNAL_ADDRESS: &str = "RUSTFS_EXTERNAL_ADDRESS";
/// 1 KiB
pub const KI_B: usize = 1024;
/// 1 MiB

View File

@@ -32,3 +32,5 @@ pub mod audit;
pub mod notify;
#[cfg(feature = "observability")]
pub mod observability;
#[cfg(feature = "opa")]
pub mod opa;

View File

@@ -1,98 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// Observability Keys
pub const ENV_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
pub const ENV_OBS_METER_INTERVAL: &str = "RUSTFS_OBS_METER_INTERVAL";
pub const ENV_OBS_SERVICE_NAME: &str = "RUSTFS_OBS_SERVICE_NAME";
pub const ENV_OBS_SERVICE_VERSION: &str = "RUSTFS_OBS_SERVICE_VERSION";
pub const ENV_OBS_ENVIRONMENT: &str = "RUSTFS_OBS_ENVIRONMENT";
pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
pub const ENV_OBS_LOCAL_LOGGING_ENABLED: &str = "RUSTFS_OBS_LOCAL_LOGGING_ENABLED";
pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
pub const ENV_OBS_LOG_FILENAME: &str = "RUSTFS_OBS_LOG_FILENAME";
pub const ENV_OBS_LOG_ROTATION_SIZE_MB: &str = "RUSTFS_OBS_LOG_ROTATION_SIZE_MB";
pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";
/// Log pool capacity for async logging
pub const ENV_OBS_LOG_POOL_CAPA: &str = "RUSTFS_OBS_LOG_POOL_CAPA";
/// Log message capacity for async logging
pub const ENV_OBS_LOG_MESSAGE_CAPA: &str = "RUSTFS_OBS_LOG_MESSAGE_CAPA";
/// Log flush interval in milliseconds for async logging
pub const ENV_OBS_LOG_FLUSH_MS: &str = "RUSTFS_OBS_LOG_FLUSH_MS";
/// Default values for log pool
pub const DEFAULT_OBS_LOG_POOL_CAPA: usize = 10240;
/// Default values for message capacity
pub const DEFAULT_OBS_LOG_MESSAGE_CAPA: usize = 32768;
/// Default values for flush interval in milliseconds
pub const DEFAULT_OBS_LOG_FLUSH_MS: u64 = 200;
/// Audit logger queue capacity environment variable key
pub const ENV_AUDIT_LOGGER_QUEUE_CAPACITY: &str = "RUSTFS_AUDIT_LOGGER_QUEUE_CAPACITY";
/// Default values for observability configuration
pub const DEFAULT_AUDIT_LOGGER_QUEUE_CAPACITY: usize = 10000;
/// Default values for observability configuration
// ### Supported Environment Values
// - `production` - Secure file-only logging
// - `development` - Full debugging with stdout
// - `test` - Test environment with stdout support
// - `staging` - Staging environment with stdout support
pub const DEFAULT_OBS_ENVIRONMENT_PRODUCTION: &str = "production";
pub const DEFAULT_OBS_ENVIRONMENT_DEVELOPMENT: &str = "development";
pub const DEFAULT_OBS_ENVIRONMENT_TEST: &str = "test";
pub const DEFAULT_OBS_ENVIRONMENT_STAGING: &str = "staging";
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_env_keys() {
assert_eq!(ENV_OBS_ENDPOINT, "RUSTFS_OBS_ENDPOINT");
assert_eq!(ENV_OBS_USE_STDOUT, "RUSTFS_OBS_USE_STDOUT");
assert_eq!(ENV_OBS_SAMPLE_RATIO, "RUSTFS_OBS_SAMPLE_RATIO");
assert_eq!(ENV_OBS_METER_INTERVAL, "RUSTFS_OBS_METER_INTERVAL");
assert_eq!(ENV_OBS_SERVICE_NAME, "RUSTFS_OBS_SERVICE_NAME");
assert_eq!(ENV_OBS_SERVICE_VERSION, "RUSTFS_OBS_SERVICE_VERSION");
assert_eq!(ENV_OBS_ENVIRONMENT, "RUSTFS_OBS_ENVIRONMENT");
assert_eq!(ENV_OBS_LOGGER_LEVEL, "RUSTFS_OBS_LOGGER_LEVEL");
assert_eq!(ENV_OBS_LOCAL_LOGGING_ENABLED, "RUSTFS_OBS_LOCAL_LOGGING_ENABLED");
assert_eq!(ENV_OBS_LOG_DIRECTORY, "RUSTFS_OBS_LOG_DIRECTORY");
assert_eq!(ENV_OBS_LOG_FILENAME, "RUSTFS_OBS_LOG_FILENAME");
assert_eq!(ENV_OBS_LOG_ROTATION_SIZE_MB, "RUSTFS_OBS_LOG_ROTATION_SIZE_MB");
assert_eq!(ENV_OBS_LOG_ROTATION_TIME, "RUSTFS_OBS_LOG_ROTATION_TIME");
assert_eq!(ENV_OBS_LOG_KEEP_FILES, "RUSTFS_OBS_LOG_KEEP_FILES");
assert_eq!(ENV_AUDIT_LOGGER_QUEUE_CAPACITY, "RUSTFS_AUDIT_LOGGER_QUEUE_CAPACITY");
}
#[test]
fn test_default_values() {
assert_eq!(DEFAULT_AUDIT_LOGGER_QUEUE_CAPACITY, 10000);
assert_eq!(DEFAULT_OBS_ENVIRONMENT_PRODUCTION, "production");
assert_eq!(DEFAULT_OBS_ENVIRONMENT_DEVELOPMENT, "development");
assert_eq!(DEFAULT_OBS_ENVIRONMENT_TEST, "test");
assert_eq!(DEFAULT_OBS_ENVIRONMENT_STAGING, "staging");
}
}

View File

@@ -1,28 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// RUSTFS_SINKS_FILE_PATH
pub const ENV_SINKS_FILE_PATH: &str = "RUSTFS_SINKS_FILE_PATH";
// RUSTFS_SINKS_FILE_BUFFER_SIZE
pub const ENV_SINKS_FILE_BUFFER_SIZE: &str = "RUSTFS_SINKS_FILE_BUFFER_SIZE";
// RUSTFS_SINKS_FILE_FLUSH_INTERVAL_MS
pub const ENV_SINKS_FILE_FLUSH_INTERVAL_MS: &str = "RUSTFS_SINKS_FILE_FLUSH_INTERVAL_MS";
// RUSTFS_SINKS_FILE_FLUSH_THRESHOLD
pub const ENV_SINKS_FILE_FLUSH_THRESHOLD: &str = "RUSTFS_SINKS_FILE_FLUSH_THRESHOLD";
pub const DEFAULT_SINKS_FILE_BUFFER_SIZE: usize = 8192;
pub const DEFAULT_SINKS_FILE_FLUSH_INTERVAL_MS: u64 = 1000;
pub const DEFAULT_SINKS_FILE_FLUSH_THRESHOLD: usize = 100;

View File

@@ -1,27 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// RUSTFS_SINKS_KAFKA_BROKERS
pub const ENV_SINKS_KAFKA_BROKERS: &str = "RUSTFS_SINKS_KAFKA_BROKERS";
pub const ENV_SINKS_KAFKA_TOPIC: &str = "RUSTFS_SINKS_KAFKA_TOPIC";
// batch_size
pub const ENV_SINKS_KAFKA_BATCH_SIZE: &str = "RUSTFS_SINKS_KAFKA_BATCH_SIZE";
// batch_timeout_ms
pub const ENV_SINKS_KAFKA_BATCH_TIMEOUT_MS: &str = "RUSTFS_SINKS_KAFKA_BATCH_TIMEOUT_MS";
// brokers
pub const DEFAULT_SINKS_KAFKA_BROKERS: &str = "localhost:9092";
pub const DEFAULT_SINKS_KAFKA_TOPIC: &str = "rustfs-sinks";
pub const DEFAULT_SINKS_KAFKA_BATCH_SIZE: usize = 100;
pub const DEFAULT_SINKS_KAFKA_BATCH_TIMEOUT_MS: u64 = 1000;

View File

@@ -12,12 +12,87 @@
// See the License for the specific language governing permissions and
// limitations under the License.
mod config;
mod file;
mod kafka;
mod webhook;
// Observability Keys
pub use config::*;
pub use file::*;
pub use kafka::*;
pub use webhook::*;
pub const ENV_OBS_ENDPOINT: &str = "RUSTFS_OBS_ENDPOINT";
pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
pub const ENV_OBS_METER_INTERVAL: &str = "RUSTFS_OBS_METER_INTERVAL";
pub const ENV_OBS_SERVICE_NAME: &str = "RUSTFS_OBS_SERVICE_NAME";
pub const ENV_OBS_SERVICE_VERSION: &str = "RUSTFS_OBS_SERVICE_VERSION";
pub const ENV_OBS_ENVIRONMENT: &str = "RUSTFS_OBS_ENVIRONMENT";
pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
pub const ENV_OBS_LOCAL_LOGGING_ENABLED: &str = "RUSTFS_OBS_LOCAL_LOGGING_ENABLED";
pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
pub const ENV_OBS_LOG_FILENAME: &str = "RUSTFS_OBS_LOG_FILENAME";
pub const ENV_OBS_LOG_ROTATION_SIZE_MB: &str = "RUSTFS_OBS_LOG_ROTATION_SIZE_MB";
pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";
/// Log pool capacity for async logging
pub const ENV_OBS_LOG_POOL_CAPA: &str = "RUSTFS_OBS_LOG_POOL_CAPA";
/// Log message capacity for async logging
pub const ENV_OBS_LOG_MESSAGE_CAPA: &str = "RUSTFS_OBS_LOG_MESSAGE_CAPA";
/// Log flush interval in milliseconds for async logging
pub const ENV_OBS_LOG_FLUSH_MS: &str = "RUSTFS_OBS_LOG_FLUSH_MS";
/// Default values for log pool
pub const DEFAULT_OBS_LOG_POOL_CAPA: usize = 10240;
/// Default values for message capacity
pub const DEFAULT_OBS_LOG_MESSAGE_CAPA: usize = 32768;
/// Default values for flush interval in milliseconds
pub const DEFAULT_OBS_LOG_FLUSH_MS: u64 = 200;
/// Audit logger queue capacity environment variable key
pub const ENV_AUDIT_LOGGER_QUEUE_CAPACITY: &str = "RUSTFS_AUDIT_LOGGER_QUEUE_CAPACITY";
/// Default values for observability configuration
pub const DEFAULT_AUDIT_LOGGER_QUEUE_CAPACITY: usize = 10000;
/// Default values for observability configuration
// ### Supported Environment Values
// - `production` - Secure file-only logging
// - `development` - Full debugging with stdout
// - `test` - Test environment with stdout support
// - `staging` - Staging environment with stdout support
pub const DEFAULT_OBS_ENVIRONMENT_PRODUCTION: &str = "production";
pub const DEFAULT_OBS_ENVIRONMENT_DEVELOPMENT: &str = "development";
pub const DEFAULT_OBS_ENVIRONMENT_TEST: &str = "test";
pub const DEFAULT_OBS_ENVIRONMENT_STAGING: &str = "staging";
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_env_keys() {
assert_eq!(ENV_OBS_ENDPOINT, "RUSTFS_OBS_ENDPOINT");
assert_eq!(ENV_OBS_USE_STDOUT, "RUSTFS_OBS_USE_STDOUT");
assert_eq!(ENV_OBS_SAMPLE_RATIO, "RUSTFS_OBS_SAMPLE_RATIO");
assert_eq!(ENV_OBS_METER_INTERVAL, "RUSTFS_OBS_METER_INTERVAL");
assert_eq!(ENV_OBS_SERVICE_NAME, "RUSTFS_OBS_SERVICE_NAME");
assert_eq!(ENV_OBS_SERVICE_VERSION, "RUSTFS_OBS_SERVICE_VERSION");
assert_eq!(ENV_OBS_ENVIRONMENT, "RUSTFS_OBS_ENVIRONMENT");
assert_eq!(ENV_OBS_LOGGER_LEVEL, "RUSTFS_OBS_LOGGER_LEVEL");
assert_eq!(ENV_OBS_LOCAL_LOGGING_ENABLED, "RUSTFS_OBS_LOCAL_LOGGING_ENABLED");
assert_eq!(ENV_OBS_LOG_DIRECTORY, "RUSTFS_OBS_LOG_DIRECTORY");
assert_eq!(ENV_OBS_LOG_FILENAME, "RUSTFS_OBS_LOG_FILENAME");
assert_eq!(ENV_OBS_LOG_ROTATION_SIZE_MB, "RUSTFS_OBS_LOG_ROTATION_SIZE_MB");
assert_eq!(ENV_OBS_LOG_ROTATION_TIME, "RUSTFS_OBS_LOG_ROTATION_TIME");
assert_eq!(ENV_OBS_LOG_KEEP_FILES, "RUSTFS_OBS_LOG_KEEP_FILES");
assert_eq!(ENV_AUDIT_LOGGER_QUEUE_CAPACITY, "RUSTFS_AUDIT_LOGGER_QUEUE_CAPACITY");
}
#[test]
fn test_default_values() {
assert_eq!(DEFAULT_AUDIT_LOGGER_QUEUE_CAPACITY, 10000);
assert_eq!(DEFAULT_OBS_ENVIRONMENT_PRODUCTION, "production");
assert_eq!(DEFAULT_OBS_ENVIRONMENT_DEVELOPMENT, "development");
assert_eq!(DEFAULT_OBS_ENVIRONMENT_TEST, "test");
assert_eq!(DEFAULT_OBS_ENVIRONMENT_STAGING, "staging");
}
}

View File

@@ -1,28 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// RUSTFS_SINKS_WEBHOOK_ENDPOINT
pub const ENV_SINKS_WEBHOOK_ENDPOINT: &str = "RUSTFS_SINKS_WEBHOOK_ENDPOINT";
// RUSTFS_SINKS_WEBHOOK_AUTH_TOKEN
pub const ENV_SINKS_WEBHOOK_AUTH_TOKEN: &str = "RUSTFS_SINKS_WEBHOOK_AUTH_TOKEN";
// max_retries
pub const ENV_SINKS_WEBHOOK_MAX_RETRIES: &str = "RUSTFS_SINKS_WEBHOOK_MAX_RETRIES";
// retry_delay_ms
pub const ENV_SINKS_WEBHOOK_RETRY_DELAY_MS: &str = "RUSTFS_SINKS_WEBHOOK_RETRY_DELAY_MS";
// Default values for webhook sink configuration
pub const DEFAULT_SINKS_WEBHOOK_ENDPOINT: &str = "http://localhost:8080";
pub const DEFAULT_SINKS_WEBHOOK_AUTH_TOKEN: &str = "";
pub const DEFAULT_SINKS_WEBHOOK_MAX_RETRIES: usize = 3;
pub const DEFAULT_SINKS_WEBHOOK_RETRY_DELAY_MS: u64 = 100;

View File

@@ -12,16 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{UnifiedLogEntry, sinks::Sink};
use std::sync::Arc;
use tokio::sync::mpsc::Receiver;
//opa env vars
pub const ENV_POLICY_PLUGIN_OPA_URL: &str = "RUSTFS_POLICY_PLUGIN_URL";
pub const ENV_POLICY_PLUGIN_AUTH_TOKEN: &str = "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN";
/// Start the log processing worker thread
pub(crate) async fn start_worker(receiver: Receiver<UnifiedLogEntry>, sinks: Vec<Arc<dyn Sink>>) {
let mut receiver = receiver;
while let Some(entry) = receiver.recv().await {
for sink in &sinks {
sink.write(&entry).await;
}
}
}
pub const ENV_POLICY_PLUGIN_KEYS: &[&str] = &[ENV_POLICY_PLUGIN_OPA_URL, ENV_POLICY_PLUGIN_AUTH_TOKEN];
pub const POLICY_PLUGIN_SUB_SYS: &str = "policy_plugin";

View File

@@ -49,5 +49,4 @@ uuid = { workspace = true }
base64 = { workspace = true }
rand = { workspace = true }
chrono = { workspace = true }
http.workspace = true
md5 = { workspace = true }
md5 = { workspace = true }

View File

@@ -13,16 +13,25 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! 分片上传加密功能的分步测试用例
//!
//! 这个测试套件将验证分片上传加密功能的每一个步骤:
//! 1. 测试基础的单分片加密(验证加密基础逻辑)
//! 2. 测试多分片上传(验证分片拼接逻辑)
//! 3. 测试加密元数据的保存和读取
//! 4. 测试完整的分片上传加密流程
use super::common::LocalKMSTestEnvironment;
use crate::common::{TEST_BUCKET, init_logging};
use serial_test::serial;
use tracing::{debug, info};
/// 步骤1测试基础单文件加密功能确保SSE-S3在非分片场景下正常工作
#[tokio::test]
#[serial]
async fn test_step1_basic_single_file_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 step1: test basic single file encryption");
info!("🧪 步骤1测试基础单文件加密功能");
let mut kms_env = LocalKMSTestEnvironment::new().await?;
let _default_key_id = kms_env.start_rustfs_for_local_kms().await?;
@@ -31,11 +40,11 @@ async fn test_step1_basic_single_file_encryption() -> Result<(), Box<dyn std::er
let s3_client = kms_env.base_env.create_s3_client();
kms_env.base_env.create_test_bucket(TEST_BUCKET).await?;
// test small file encryption (should inline store)
// 测试小文件加密(应该会内联存储)
let test_data = b"Hello, this is a small test file for SSE-S3!";
let object_key = "test-single-file-encrypted";
info!("📤 step1: upload small file ({}) with SSE-S3 encryption", test_data.len());
info!("📤 上传小文件({}字节启用SSE-S3加密", test_data.len());
let put_response = s3_client
.put_object()
.bucket(TEST_BUCKET)
@@ -45,41 +54,41 @@ async fn test_step1_basic_single_file_encryption() -> Result<(), Box<dyn std::er
.send()
.await?;
debug!("PUT response ETag: {:?}", put_response.e_tag());
debug!("PUT response SSE: {:?}", put_response.server_side_encryption());
debug!("PUT响应ETag: {:?}", put_response.e_tag());
debug!("PUT响应SSE: {:?}", put_response.server_side_encryption());
// verify PUT response contains correct encryption header
// 验证PUT响应包含正确的加密头
assert_eq!(
put_response.server_side_encryption(),
Some(&aws_sdk_s3::types::ServerSideEncryption::Aes256)
);
info!("📥 step1: download file and verify encryption status");
info!("📥 下载文件并验证加密状态");
let get_response = s3_client.get_object().bucket(TEST_BUCKET).key(object_key).send().await?;
debug!("GET response SSE: {:?}", get_response.server_side_encryption());
debug!("GET响应SSE: {:?}", get_response.server_side_encryption());
// verify GET response contains correct encryption header
// 验证GET响应包含正确的加密头
assert_eq!(
get_response.server_side_encryption(),
Some(&aws_sdk_s3::types::ServerSideEncryption::Aes256)
);
// verify data integrity
// 验证数据完整性
let downloaded_data = get_response.body.collect().await?.into_bytes();
assert_eq!(&downloaded_data[..], test_data);
kms_env.base_env.delete_test_bucket(TEST_BUCKET).await?;
info!("step1: basic single file encryption works as expected");
info!("步骤1通过基础单文件加密功能正常");
Ok(())
}
/// test basic multipart upload without encryption
/// 步骤2测试不加密的分片上传确保分片上传基础功能正常
#[tokio::test]
#[serial]
async fn test_step2_basic_multipart_upload_without_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 step2: test basic multipart upload without encryption");
info!("🧪 步骤2测试不加密的分片上传");
let mut kms_env = LocalKMSTestEnvironment::new().await?;
let _default_key_id = kms_env.start_rustfs_for_local_kms().await?;
@@ -93,16 +102,12 @@ async fn test_step2_basic_multipart_upload_without_encryption() -> Result<(), Bo
let total_parts = 2;
let total_size = part_size * total_parts;
// generate test data (with clear pattern for easy verification)
// 生成测试数据(有明显的模式便于验证)
let test_data: Vec<u8> = (0..total_size).map(|i| (i % 256) as u8).collect();
info!(
"🚀 step2: start multipart upload (no encryption) with {} parts, each {}MB",
total_parts,
part_size / (1024 * 1024)
);
info!("🚀 开始分片上传(无加密):{} parts每个 {}MB", total_parts, part_size / (1024 * 1024));
// step1: create multipart upload
// 步骤1创建分片上传
let create_multipart_output = s3_client
.create_multipart_upload()
.bucket(TEST_BUCKET)
@@ -111,16 +116,16 @@ async fn test_step2_basic_multipart_upload_without_encryption() -> Result<(), Bo
.await?;
let upload_id = create_multipart_output.upload_id().unwrap();
info!("📋 step2: create multipart upload, ID: {}", upload_id);
info!("📋 创建分片上传,ID: {}", upload_id);
// step2: upload each part
// 步骤2上传各个分片
let mut completed_parts = Vec::new();
for part_number in 1..=total_parts {
let start = (part_number - 1) * part_size;
let end = std::cmp::min(start + part_size, total_size);
let part_data = &test_data[start..end];
info!("📤 step2: upload part {} ({} bytes)", part_number, part_data.len());
info!("📤 上传分片 {} ({} bytes)", part_number, part_data.len());
let upload_part_output = s3_client
.upload_part()
@@ -140,15 +145,15 @@ async fn test_step2_basic_multipart_upload_without_encryption() -> Result<(), Bo
.build(),
);
debug!("step2: part {} uploaded, ETag: {}", part_number, etag);
debug!("分片 {} 上传完成,ETag: {}", part_number, etag);
}
// step3: complete multipart upload
// 步骤3完成分片上传
let completed_multipart_upload = aws_sdk_s3::types::CompletedMultipartUpload::builder()
.set_parts(Some(completed_parts))
.build();
info!("🔗 step2: complete multipart upload");
info!("🔗 完成分片上传");
let complete_output = s3_client
.complete_multipart_upload()
.bucket(TEST_BUCKET)
@@ -158,16 +163,10 @@ async fn test_step2_basic_multipart_upload_without_encryption() -> Result<(), Bo
.send()
.await?;
debug!("step2: multipart upload completed, ETag: {:?}", complete_output.e_tag());
debug!("完成分片上传,ETag: {:?}", complete_output.e_tag());
// step4: verify multipart upload completed successfully
assert_eq!(
complete_output.e_tag().unwrap().to_string(),
format!("\"{}-{}-{}\"", object_key, upload_id, total_parts)
);
// verify data integrity
info!("📥 step2: download file and verify data integrity");
// 步骤4下载并验证
info!("📥 下载文件并验证数据完整性");
let get_response = s3_client.get_object().bucket(TEST_BUCKET).key(object_key).send().await?;
let downloaded_data = get_response.body.collect().await?.into_bytes();
@@ -175,16 +174,16 @@ async fn test_step2_basic_multipart_upload_without_encryption() -> Result<(), Bo
assert_eq!(&downloaded_data[..], &test_data[..]);
kms_env.base_env.delete_test_bucket(TEST_BUCKET).await?;
info!("step2: basic multipart upload without encryption works as expected");
info!("步骤2通过不加密的分片上传功能正常");
Ok(())
}
/// test multipart upload with SSE-S3 encryption
/// 步骤3测试分片上传 + SSE-S3加密重点测试
#[tokio::test]
#[serial]
async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 step3: test multipart upload with SSE-S3 encryption");
info!("🧪 步骤3测试分片上传 + SSE-S3加密");
let mut kms_env = LocalKMSTestEnvironment::new().await?;
let _default_key_id = kms_env.start_rustfs_for_local_kms().await?;
@@ -198,16 +197,16 @@ async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box<dyn std::er
let total_parts = 2;
let total_size = part_size * total_parts;
// generate test data (with clear pattern for easy verification)
// 生成测试数据
let test_data: Vec<u8> = (0..total_size).map(|i| ((i / 1000) % 256) as u8).collect();
info!(
"🔐 step3: start multipart upload with SSE-S3 encryption: {} parts, each {}MB",
"🔐 开始分片上传SSE-S3加密{} parts,每个 {}MB",
total_parts,
part_size / (1024 * 1024)
);
// step1: create multipart upload and enable SSE-S3
// 步骤1创建分片上传并启用SSE-S3
let create_multipart_output = s3_client
.create_multipart_upload()
.bucket(TEST_BUCKET)
@@ -217,24 +216,24 @@ async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box<dyn std::er
.await?;
let upload_id = create_multipart_output.upload_id().unwrap();
info!("📋 step3: create multipart upload with SSE-S3 encryption, ID: {}", upload_id);
info!("📋 创建加密分片上传,ID: {}", upload_id);
// step2: verify CreateMultipartUpload response (SSE-S3 header should be included)
// 验证CreateMultipartUpload响应如果有SSE头的话
if let Some(sse) = create_multipart_output.server_side_encryption() {
debug!("CreateMultipartUpload response contains SSE header: {:?}", sse);
debug!("CreateMultipartUpload包含SSE响应: {:?}", sse);
assert_eq!(sse, &aws_sdk_s3::types::ServerSideEncryption::Aes256);
} else {
debug!("CreateMultipartUpload response does not contain SSE header (some implementations may return empty string)");
debug!("CreateMultipartUpload不包含SSE响应头某些实现中正常");
}
// step2: upload each part
// 步骤2上传各个分片
let mut completed_parts = Vec::new();
for part_number in 1..=total_parts {
let start = (part_number - 1) * part_size;
let end = std::cmp::min(start + part_size, total_size);
let part_data = &test_data[start..end];
info!("🔐 step3: upload encrypted part {} ({} bytes)", part_number, part_data.len());
info!("🔐 上传加密分片 {} ({} bytes)", part_number, part_data.len());
let upload_part_output = s3_client
.upload_part()
@@ -254,15 +253,15 @@ async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box<dyn std::er
.build(),
);
debug!("step3: part {} uploaded, ETag: {}", part_number, etag);
debug!("加密分片 {} 上传完成,ETag: {}", part_number, etag);
}
// step3: complete multipart upload
// 步骤3完成分片上传
let completed_multipart_upload = aws_sdk_s3::types::CompletedMultipartUpload::builder()
.set_parts(Some(completed_parts))
.build();
info!("🔗 step3: complete multipart upload with SSE-S3 encryption");
info!("🔗 完成加密分片上传");
let complete_output = s3_client
.complete_multipart_upload()
.bucket(TEST_BUCKET)
@@ -272,46 +271,43 @@ async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box<dyn std::er
.send()
.await?;
debug!(
"step3: complete multipart upload with SSE-S3 encryption, ETag: {:?}",
complete_output.e_tag()
);
debug!("完成加密分片上传ETag: {:?}", complete_output.e_tag());
// step4: HEAD request to check metadata
info!("📋 step4: check object metadata");
// 步骤4HEAD请求检查元数据
info!("📋 检查对象元数据");
let head_response = s3_client.head_object().bucket(TEST_BUCKET).key(object_key).send().await?;
debug!("HEAD response SSE: {:?}", head_response.server_side_encryption());
debug!("HEAD response metadata: {:?}", head_response.metadata());
debug!("HEAD响应 SSE: {:?}", head_response.server_side_encryption());
debug!("HEAD响应 元数据: {:?}", head_response.metadata());
// step5: GET request to download and verify
info!("📥 step5: download encrypted file and verify");
// 步骤5GET请求下载并验证
info!("📥 下载加密文件并验证");
let get_response = s3_client.get_object().bucket(TEST_BUCKET).key(object_key).send().await?;
debug!("GET response SSE: {:?}", get_response.server_side_encryption());
debug!("GET响应 SSE: {:?}", get_response.server_side_encryption());
// step5: verify GET response contains SSE-S3 encryption header
// 🎯 关键验证GET响应必须包含SSE-S3加密头
assert_eq!(
get_response.server_side_encryption(),
Some(&aws_sdk_s3::types::ServerSideEncryption::Aes256)
);
// step5: verify downloaded data matches original test data
// 验证数据完整性
let downloaded_data = get_response.body.collect().await?.into_bytes();
assert_eq!(downloaded_data.len(), total_size);
assert_eq!(&downloaded_data[..], &test_data[..]);
kms_env.base_env.delete_test_bucket(TEST_BUCKET).await?;
info!("step3: multipart upload with SSE-S3 encryption function is normal");
info!("步骤3通过分片上传 + SSE-S3加密功能正常");
Ok(())
}
/// step4: test larger multipart upload with encryption (streaming encryption)
/// 步骤4测试更大的分片上传测试流式加密
#[tokio::test]
#[serial]
async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 step4: test larger multipart upload with encryption (streaming encryption)");
info!("🧪 步骤4测试大文件分片上传加密");
let mut kms_env = LocalKMSTestEnvironment::new().await?;
let _default_key_id = kms_env.start_rustfs_for_local_kms().await?;
@@ -326,13 +322,13 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
let total_size = part_size * total_parts;
info!(
"🗂️ step4: generate large test data: {} parts, each {}MB, total {}MB",
"🗂️ 生成大文件测试数据:{} parts每个 {}MB总计 {}MB",
total_parts,
part_size / (1024 * 1024),
total_size / (1024 * 1024)
);
// step4: generate large test data (using complex pattern for verification)
// 生成大文件测试数据(使用复杂模式便于验证)
let test_data: Vec<u8> = (0..total_size)
.map(|i| {
let part_num = i / part_size;
@@ -341,9 +337,9 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
})
.collect();
info!("🔐 step4: start large multipart upload with encryption (SSE-S3)");
info!("🔐 开始大文件分片上传(SSE-S3加密)");
// step4: create multipart upload
// 创建分片上传
let create_multipart_output = s3_client
.create_multipart_upload()
.bucket(TEST_BUCKET)
@@ -353,9 +349,9 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
.await?;
let upload_id = create_multipart_output.upload_id().unwrap();
info!("📋 step4: create multipart upload with encryption (SSE-S3), ID: {}", upload_id);
info!("📋 创建大文件加密分片上传,ID: {}", upload_id);
// step4: upload parts
// 上传各个分片
let mut completed_parts = Vec::new();
for part_number in 1..=total_parts {
let start = (part_number - 1) * part_size;
@@ -363,7 +359,7 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
let part_data = &test_data[start..end];
info!(
"🔐 step4: upload part {} ({:.2}MB)",
"🔐 上传大文件加密分片 {} ({:.2}MB)",
part_number,
part_data.len() as f64 / (1024.0 * 1024.0)
);
@@ -386,15 +382,15 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
.build(),
);
debug!("step4: upload part {} completed, ETag: {}", part_number, etag);
debug!("大文件加密分片 {} 上传完成,ETag: {}", part_number, etag);
}
// step4: complete multipart upload
// 完成分片上传
let completed_multipart_upload = aws_sdk_s3::types::CompletedMultipartUpload::builder()
.set_parts(Some(completed_parts))
.build();
info!("🔗 step4: complete multipart upload with encryption (SSE-S3)");
info!("🔗 完成大文件加密分片上传");
let complete_output = s3_client
.complete_multipart_upload()
.bucket(TEST_BUCKET)
@@ -404,46 +400,40 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
.send()
.await?;
debug!(
"step4: complete multipart upload with encryption (SSE-S3), ETag: {:?}",
complete_output.e_tag()
);
debug!("完成大文件加密分片上传ETag: {:?}", complete_output.e_tag());
// step4: download and verify
info!("📥 step4: download and verify large multipart upload with encryption (SSE-S3)");
// 下载并验证
info!("📥 下载大文件并验证");
let get_response = s3_client.get_object().bucket(TEST_BUCKET).key(object_key).send().await?;
// step4: verify encryption header
// 验证加密头
assert_eq!(
get_response.server_side_encryption(),
Some(&aws_sdk_s3::types::ServerSideEncryption::Aes256)
);
// step4: verify data integrity
// 验证数据完整性
let downloaded_data = get_response.body.collect().await?.into_bytes();
assert_eq!(downloaded_data.len(), total_size);
// step4: verify data matches original test data
// 逐字节验证数据(对于大文件更严格)
for (i, (&actual, &expected)) in downloaded_data.iter().zip(test_data.iter()).enumerate() {
if actual != expected {
panic!(
"step4: large multipart upload with encryption (SSE-S3) data mismatch at byte {}: actual={}, expected={}",
i, actual, expected
);
panic!("大文件数据在第{i}字节不匹配: 实际={actual}, 期待={expected}");
}
}
kms_env.base_env.delete_test_bucket(TEST_BUCKET).await?;
info!("step4: large multipart upload with encryption (SSE-S3) functionality normal");
info!("步骤4通过大文件分片上传加密功能正常");
Ok(())
}
/// step5: test all encryption types multipart upload
/// 步骤5测试所有加密类型的分片上传
#[tokio::test]
#[serial]
async fn test_step5_all_encryption_types_multipart() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 step5: test all encryption types multipart upload");
info!("🧪 步骤5测试所有加密类型的分片上传");
let mut kms_env = LocalKMSTestEnvironment::new().await?;
let _default_key_id = kms_env.start_rustfs_for_local_kms().await?;
@@ -456,8 +446,8 @@ async fn test_step5_all_encryption_types_multipart() -> Result<(), Box<dyn std::
let total_parts = 2;
let total_size = part_size * total_parts;
// step5: test SSE-KMS multipart upload
info!("🔐 step5: test SSE-KMS multipart upload");
// 测试SSE-KMS
info!("🔐 测试 SSE-KMS 分片上传");
test_multipart_encryption_type(
&s3_client,
TEST_BUCKET,
@@ -469,8 +459,8 @@ async fn test_step5_all_encryption_types_multipart() -> Result<(), Box<dyn std::
)
.await?;
// step5: test SSE-C multipart upload
info!("🔐 step5: test SSE-C multipart upload");
// 测试SSE-C
info!("🔐 测试 SSE-C 分片上传");
test_multipart_encryption_type(
&s3_client,
TEST_BUCKET,
@@ -483,7 +473,7 @@ async fn test_step5_all_encryption_types_multipart() -> Result<(), Box<dyn std::
.await?;
kms_env.base_env.delete_test_bucket(TEST_BUCKET).await?;
info!("step5: all encryption types multipart upload functionality normal");
info!("步骤5通过所有加密类型的分片上传功能正常");
Ok(())
}
@@ -493,7 +483,7 @@ enum EncryptionType {
SSEC,
}
/// step5: test specific encryption type multipart upload
/// 辅助函数:测试特定加密类型的分片上传
async fn test_multipart_encryption_type(
s3_client: &aws_sdk_s3::Client,
bucket: &str,
@@ -503,10 +493,10 @@ async fn test_multipart_encryption_type(
total_parts: usize,
encryption_type: EncryptionType,
) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// step5: generate test data
// 生成测试数据
let test_data: Vec<u8> = (0..total_size).map(|i| ((i * 7) % 256) as u8).collect();
// step5: prepare SSE-C key and MD5 (if needed)
// 准备SSE-C所需的密钥如果需要
let (sse_c_key, sse_c_md5) = if matches!(encryption_type, EncryptionType::SSEC) {
let key = "01234567890123456789012345678901";
let key_b64 = base64::Engine::encode(&base64::engine::general_purpose::STANDARD, key);
@@ -516,10 +506,9 @@ async fn test_multipart_encryption_type(
(None, None)
};
// step5: create multipart upload
info!("🔗 step5: create multipart upload with encryption {:?}", encryption_type);
info!("📋 创建分片上传 - {:?}", encryption_type);
// step5: create multipart upload request
// 创建分片上传
let mut create_request = s3_client.create_multipart_upload().bucket(bucket).key(object_key);
create_request = match encryption_type {
@@ -533,6 +522,7 @@ async fn test_multipart_encryption_type(
let create_multipart_output = create_request.send().await?;
let upload_id = create_multipart_output.upload_id().unwrap();
// 上传分片
let mut completed_parts = Vec::new();
for part_number in 1..=total_parts {
let start = (part_number - 1) * part_size;
@@ -547,7 +537,7 @@ async fn test_multipart_encryption_type(
.part_number(part_number as i32)
.body(aws_sdk_s3::primitives::ByteStream::from(part_data.to_vec()));
// step5: include SSE-C key and MD5 in each UploadPart request (if needed)
// SSE-C需要在每个UploadPart请求中包含密钥
if matches!(encryption_type, EncryptionType::SSEC) {
upload_request = upload_request
.sse_customer_algorithm("AES256")
@@ -564,11 +554,10 @@ async fn test_multipart_encryption_type(
.build(),
);
// step5: complete multipart upload request
debug!("🔗 step5: complete multipart upload part {} with etag {}", part_number, etag);
debug!("{:?} 分片 {} 上传完成", encryption_type, part_number);
}
// step5: complete multipart upload
// 完成分片上传
let completed_multipart_upload = aws_sdk_s3::types::CompletedMultipartUpload::builder()
.set_parts(Some(completed_parts))
.build();
@@ -582,12 +571,10 @@ async fn test_multipart_encryption_type(
.send()
.await?;
// step5: download and verify multipart upload
info!("🔗 step5: download and verify multipart upload with encryption {:?}", encryption_type);
// 下载并验证
let mut get_request = s3_client.get_object().bucket(bucket).key(object_key);
// step5: include SSE-C key and MD5 in each GET request (if needed)
// SSE-C需要在GET请求中包含密钥
if matches!(encryption_type, EncryptionType::SSEC) {
get_request = get_request
.sse_customer_algorithm("AES256")
@@ -597,7 +584,7 @@ async fn test_multipart_encryption_type(
let get_response = get_request.send().await?;
// step5: verify encryption headers
// 验证加密头
match encryption_type {
EncryptionType::SSEKMS => {
assert_eq!(
@@ -610,15 +597,11 @@ async fn test_multipart_encryption_type(
}
}
// step5: verify data integrity
// 验证数据完整性
let downloaded_data = get_response.body.collect().await?.into_bytes();
assert_eq!(downloaded_data.len(), total_size);
assert_eq!(&downloaded_data[..], &test_data[..]);
// step5: verify data integrity
info!(
"✅ step5: verify data integrity for multipart upload with encryption {:?}",
encryption_type
);
info!("✅ {:?} 分片上传测试通过", encryption_type);
Ok(())
}

File diff suppressed because it is too large Load Diff

View File

@@ -12,7 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
mod append;
mod conditional_writes;
mod lifecycle;
mod lock;

View File

@@ -101,6 +101,7 @@ aws-credential-types = { workspace = true }
aws-smithy-types = { workspace = true }
parking_lot = { workspace = true }
moka = { workspace = true }
base64-simd.workspace = true
[target.'cfg(not(windows))'.dependencies]
nix = { workspace = true }

View File

@@ -17,12 +17,10 @@ pub mod datatypes;
mod replication_pool;
mod replication_resyncer;
mod replication_state;
mod replication_type;
mod rule;
pub use config::*;
pub use datatypes::*;
pub use replication_pool::*;
pub use replication_resyncer::*;
pub use replication_type::*;
pub use rule::*;

View File

@@ -1,9 +1,4 @@
use crate::StorageAPI;
use crate::bucket::replication::MrfReplicateEntry;
use crate::bucket::replication::ReplicateDecision;
use crate::bucket::replication::ReplicateObjectInfo;
use crate::bucket::replication::ReplicationWorkerOperation;
use crate::bucket::replication::ResyncDecision;
use crate::bucket::replication::ResyncOpts;
use crate::bucket::replication::ResyncStatusType;
use crate::bucket::replication::replicate_delete;
@@ -18,16 +13,21 @@ use crate::bucket::replication::replication_resyncer::{
BucketReplicationResyncStatus, DeletedObjectReplicationInfo, ReplicationResyncer,
};
use crate::bucket::replication::replication_state::ReplicationStats;
use crate::bucket::replication::replication_statuses_map;
use crate::bucket::replication::version_purge_statuses_map;
use crate::config::com::read_config;
use crate::error::Error as EcstoreError;
use crate::store_api::ObjectInfo;
use lazy_static::lazy_static;
use rustfs_filemeta::MrfReplicateEntry;
use rustfs_filemeta::ReplicateDecision;
use rustfs_filemeta::ReplicateObjectInfo;
use rustfs_filemeta::ReplicatedTargetInfo;
use rustfs_filemeta::ReplicationStatusType;
use rustfs_filemeta::ReplicationType;
use rustfs_filemeta::ReplicationWorkerOperation;
use rustfs_filemeta::ResyncDecision;
use rustfs_filemeta::replication_statuses_map;
use rustfs_filemeta::version_purge_statuses_map;
use rustfs_utils::http::RESERVED_METADATA_PREFIX_LOWER;
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
@@ -996,7 +996,7 @@ pub async fn schedule_replication<S: StorageAPI>(oi: ObjectInfo, o: Arc<S>, dsc:
target_purge_statuses: purge_statuses,
replication_timestamp: tm,
user_tags: oi.user_tags,
checksum: vec![],
checksum: None,
retry_count: 0,
event_type: "".to_string(),
existing_obj_resync: ResyncDecision::default(),

View File

@@ -2,12 +2,8 @@ use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, BucketTargetSys, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient,
};
use crate::bucket::metadata_sys;
use crate::bucket::replication::{MrfReplicateEntry, ReplicationWorkerOperation, ResyncStatusType};
use crate::bucket::replication::{
ObjectOpts, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATION_RESET, ReplicateObjectInfo,
ReplicationConfigurationExt as _, ResyncTargetDecision, get_replication_state, parse_replicate_decision,
replication_statuses_map, target_reset_header, version_purge_statuses_map,
};
use crate::bucket::replication::ResyncStatusType;
use crate::bucket::replication::{ObjectOpts, ReplicationConfigurationExt as _};
use crate::bucket::tagging::decode_tags_to_map;
use crate::bucket::target::BucketTargets;
use crate::bucket::versioning_sys::BucketVersioningSys;
@@ -29,14 +25,17 @@ use byteorder::ByteOrder;
use futures::future::join_all;
use http::HeaderMap;
use regex::Regex;
use rustfs_filemeta::{
ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType, ReplicationType,
VersionPurgeStatusType,
MrfReplicateEntry, REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATION_RESET, ReplicateDecision, ReplicateObjectInfo,
ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction, ReplicationState, ReplicationStatusType,
ReplicationType, ReplicationWorkerOperation, ResyncDecision, ResyncTargetDecision, VersionPurgeStatusType,
get_replication_state, parse_replicate_decision, replication_statuses_map, target_reset_header, version_purge_statuses_map,
};
use rustfs_utils::http::{
AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_TAGGING, AMZ_TAGGING_DIRECTIVE, CONTENT_ENCODING, HeaderExt as _,
RESERVED_METADATA_PREFIX, RESERVED_METADATA_PREFIX_LOWER, RUSTFS_REPLICATION_AUTUAL_OBJECT_SIZE, SSEC_ALGORITHM_HEADER,
SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, headers,
RESERVED_METADATA_PREFIX, RESERVED_METADATA_PREFIX_LOWER, RUSTFS_REPLICATION_AUTUAL_OBJECT_SIZE,
RUSTFS_REPLICATION_RESET_STATUS, SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER, headers,
};
use rustfs_utils::path::path_join_buf;
use rustfs_utils::string::strings_has_prefix_fold;
@@ -56,9 +55,6 @@ use tokio::time::Duration as TokioDuration;
use tokio_util::sync::CancellationToken;
use tracing::{error, info, warn};
use super::replication_type::{ReplicateDecision, ReplicateTargetDecision, ResyncDecision};
use regex::Regex;
const REPLICATION_DIR: &str = ".replication";
const RESYNC_FILE_NAME: &str = "resync.bin";
const RESYNC_META_FORMAT: u16 = 1;
@@ -663,7 +659,7 @@ pub async fn get_heal_replicate_object_info(oi: &ObjectInfo, rcfg: &ReplicationC
replication_timestamp: None,
ssec: false, // TODO: add ssec support
user_tags: oi.user_tags.clone(),
checksum: Vec::new(),
checksum: oi.checksum.clone(),
retry_count: 0,
}
}
@@ -849,7 +845,7 @@ impl ReplicationConfig {
{
resync_decision.targets.insert(
decision.arn.clone(),
ResyncTargetDecision::resync_target(
resync_target(
&oi,
&target.arn,
&target.reset_id,
@@ -864,6 +860,59 @@ impl ReplicationConfig {
}
}
pub fn resync_target(
oi: &ObjectInfo,
arn: &str,
reset_id: &str,
reset_before_date: Option<OffsetDateTime>,
status: ReplicationStatusType,
) -> ResyncTargetDecision {
let rs = oi
.user_defined
.get(target_reset_header(arn).as_str())
.or(oi.user_defined.get(RUSTFS_REPLICATION_RESET_STATUS))
.map(|s| s.to_string());
let mut dec = ResyncTargetDecision::default();
let mod_time = oi.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH);
if rs.is_none() {
let reset_before_date = reset_before_date.unwrap_or(OffsetDateTime::UNIX_EPOCH);
if !reset_id.is_empty() && mod_time < reset_before_date {
dec.replicate = true;
return dec;
}
dec.replicate = status == ReplicationStatusType::Empty;
return dec;
}
if reset_id.is_empty() || reset_before_date.is_none() {
return dec;
}
let rs = rs.unwrap();
let reset_before_date = reset_before_date.unwrap();
let parts: Vec<&str> = rs.splitn(2, ';').collect();
if parts.len() != 2 {
return dec;
}
let new_reset = parts[0] == reset_id;
if !new_reset && status == ReplicationStatusType::Completed {
return dec;
}
dec.replicate = new_reset && mod_time < reset_before_date;
dec
}
pub struct MustReplicateOptions {
meta: HashMap<String, String>,
status: ReplicationStatusType,
@@ -933,7 +982,7 @@ pub async fn check_replicate_delete(
let rcfg = match get_replication_config(bucket).await {
Ok(Some(config)) => config,
Ok(None) => {
warn!("No replication config found for bucket: {}", bucket);
// warn!("No replication config found for bucket: {}", bucket);
return ReplicateDecision::default();
}
Err(err) => {

View File

@@ -1,470 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::error::{Error, Result};
use crate::store_api::ObjectInfo;
use regex::Regex;
use rustfs_filemeta::VersionPurgeStatusType;
use rustfs_filemeta::{ReplicatedInfos, ReplicationType};
use rustfs_filemeta::{ReplicationState, ReplicationStatusType};
use rustfs_utils::http::RESERVED_METADATA_PREFIX_LOWER;
use rustfs_utils::http::RUSTFS_REPLICATION_RESET_STATUS;
use serde::{Deserialize, Serialize};
use std::any::Any;
use std::collections::HashMap;
use std::fmt;
use time::OffsetDateTime;
use uuid::Uuid;
pub const REPLICATION_RESET: &str = "replication-reset";
pub const REPLICATION_STATUS: &str = "replication-status";
// ReplicateQueued - replication being queued trail
pub const REPLICATE_QUEUED: &str = "replicate:queue";
// ReplicateExisting - audit trail for existing objects replication
pub const REPLICATE_EXISTING: &str = "replicate:existing";
// ReplicateExistingDelete - audit trail for delete replication triggered for existing delete markers
pub const REPLICATE_EXISTING_DELETE: &str = "replicate:existing:delete";
// ReplicateMRF - audit trail for replication from Most Recent Failures (MRF) queue
pub const REPLICATE_MRF: &str = "replicate:mrf";
// ReplicateIncoming - audit trail of inline replication
pub const REPLICATE_INCOMING: &str = "replicate:incoming";
// ReplicateIncomingDelete - audit trail of inline replication of deletes.
pub const REPLICATE_INCOMING_DELETE: &str = "replicate:incoming:delete";
// ReplicateHeal - audit trail for healing of failed/pending replications
pub const REPLICATE_HEAL: &str = "replicate:heal";
// ReplicateHealDelete - audit trail of healing of failed/pending delete replications.
pub const REPLICATE_HEAL_DELETE: &str = "replicate:heal:delete";
#[derive(Serialize, Deserialize, Debug)]
pub struct MrfReplicateEntry {
#[serde(rename = "bucket")]
pub bucket: String,
#[serde(rename = "object")]
pub object: String,
#[serde(skip_serializing, skip_deserializing)]
pub version_id: Option<Uuid>,
#[serde(rename = "retryCount")]
pub retry_count: i32,
#[serde(skip_serializing, skip_deserializing)]
pub size: i64,
}
pub trait ReplicationWorkerOperation: Any + Send + Sync {
fn to_mrf_entry(&self) -> MrfReplicateEntry;
fn as_any(&self) -> &dyn Any;
fn get_bucket(&self) -> &str;
fn get_object(&self) -> &str;
fn get_size(&self) -> i64;
fn is_delete_marker(&self) -> bool;
fn get_op_type(&self) -> ReplicationType;
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct ReplicateTargetDecision {
pub replicate: bool,
pub synchronous: bool,
pub arn: String,
pub id: String,
}
impl ReplicateTargetDecision {
pub fn new(arn: String, replicate: bool, sync: bool) -> Self {
Self {
replicate,
synchronous: sync,
arn,
id: String::new(),
}
}
}
impl fmt::Display for ReplicateTargetDecision {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{};{};{};{}", self.replicate, self.synchronous, self.arn, self.id)
}
}
/// ReplicateDecision represents replication decision for each target
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ReplicateDecision {
pub targets_map: HashMap<String, ReplicateTargetDecision>,
}
impl ReplicateDecision {
pub fn new() -> Self {
Self {
targets_map: HashMap::new(),
}
}
/// Returns true if at least one target qualifies for replication
pub fn replicate_any(&self) -> bool {
self.targets_map.values().any(|t| t.replicate)
}
/// Returns true if at least one target qualifies for synchronous replication
pub fn is_synchronous(&self) -> bool {
self.targets_map.values().any(|t| t.synchronous)
}
/// Updates ReplicateDecision with target's replication decision
pub fn set(&mut self, target: ReplicateTargetDecision) {
self.targets_map.insert(target.arn.clone(), target);
}
/// Returns a stringified representation of internal replication status with all targets marked as `PENDING`
pub fn pending_status(&self) -> Option<String> {
let mut result = String::new();
for target in self.targets_map.values() {
if target.replicate {
result.push_str(&format!("{}={};", target.arn, ReplicationStatusType::Pending.as_str()));
}
}
if result.is_empty() { None } else { Some(result) }
}
}
impl fmt::Display for ReplicateDecision {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut result = String::new();
for (key, value) in &self.targets_map {
result.push_str(&format!("{key}={value},"));
}
write!(f, "{}", result.trim_end_matches(','))
}
}
impl Default for ReplicateDecision {
fn default() -> Self {
Self::new()
}
}
// parse k-v pairs of target ARN to stringified ReplicateTargetDecision delimited by ',' into a
// ReplicateDecision struct
pub fn parse_replicate_decision(_bucket: &str, s: &str) -> Result<ReplicateDecision> {
let mut decision = ReplicateDecision::new();
if s.is_empty() {
return Ok(decision);
}
for p in s.split(',') {
if p.is_empty() {
continue;
}
let slc = p.split('=').collect::<Vec<&str>>();
if slc.len() != 2 {
return Err(Error::other(format!("invalid replicate decision format: {s}")));
}
let tgt_str = slc[1].trim_matches('"');
let tgt = tgt_str.split(';').collect::<Vec<&str>>();
if tgt.len() != 4 {
return Err(Error::other(format!("invalid replicate decision format: {s}")));
}
let tgt = ReplicateTargetDecision {
replicate: tgt[0] == "true",
synchronous: tgt[1] == "true",
arn: tgt[2].to_string(),
id: tgt[3].to_string(),
};
decision.targets_map.insert(slc[0].to_string(), tgt);
}
Ok(decision)
// r = ReplicateDecision{
// targetsMap: make(map[string]replicateTargetDecision),
// }
// if len(s) == 0 {
// return
// }
// for _, p := range strings.Split(s, ",") {
// if p == "" {
// continue
// }
// slc := strings.Split(p, "=")
// if len(slc) != 2 {
// return r, errInvalidReplicateDecisionFormat
// }
// tgtStr := strings.TrimSuffix(strings.TrimPrefix(slc[1], `"`), `"`)
// tgt := strings.Split(tgtStr, ";")
// if len(tgt) != 4 {
// return r, errInvalidReplicateDecisionFormat
// }
// r.targetsMap[slc[0]] = replicateTargetDecision{Replicate: tgt[0] == "true", Synchronous: tgt[1] == "true", Arn: tgt[2], ID: tgt[3]}
// }
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct ResyncTargetDecision {
pub replicate: bool,
pub reset_id: String,
pub reset_before_date: Option<OffsetDateTime>,
}
pub fn target_reset_header(arn: &str) -> String {
format!("{RESERVED_METADATA_PREFIX_LOWER}{REPLICATION_RESET}-{arn}")
}
impl ResyncTargetDecision {
pub fn resync_target(
oi: &ObjectInfo,
arn: &str,
reset_id: &str,
reset_before_date: Option<OffsetDateTime>,
status: ReplicationStatusType,
) -> Self {
let rs = oi
.user_defined
.get(target_reset_header(arn).as_str())
.or(oi.user_defined.get(RUSTFS_REPLICATION_RESET_STATUS))
.map(|s| s.to_string());
let mut dec = Self::default();
let mod_time = oi.mod_time.unwrap_or(OffsetDateTime::UNIX_EPOCH);
if rs.is_none() {
let reset_before_date = reset_before_date.unwrap_or(OffsetDateTime::UNIX_EPOCH);
if !reset_id.is_empty() && mod_time < reset_before_date {
dec.replicate = true;
return dec;
}
dec.replicate = status == ReplicationStatusType::Empty;
return dec;
}
if reset_id.is_empty() || reset_before_date.is_none() {
return dec;
}
let rs = rs.unwrap();
let reset_before_date = reset_before_date.unwrap();
let parts: Vec<&str> = rs.splitn(2, ';').collect();
if parts.len() != 2 {
return dec;
}
let new_reset = parts[0] == reset_id;
if !new_reset && status == ReplicationStatusType::Completed {
return dec;
}
dec.replicate = new_reset && mod_time < reset_before_date;
dec
}
}
/// ResyncDecision is a struct representing a map with target's individual resync decisions
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ResyncDecision {
pub targets: HashMap<String, ResyncTargetDecision>,
}
impl ResyncDecision {
pub fn new() -> Self {
Self { targets: HashMap::new() }
}
/// Returns true if no targets with resync decision present
pub fn is_empty(&self) -> bool {
self.targets.is_empty()
}
pub fn must_resync(&self) -> bool {
self.targets.values().any(|v| v.replicate)
}
pub fn must_resync_target(&self, tgt_arn: &str) -> bool {
self.targets.get(tgt_arn).map(|v| v.replicate).unwrap_or(false)
}
}
impl Default for ResyncDecision {
fn default() -> Self {
Self::new()
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ReplicateObjectInfo {
pub name: String,
pub size: i64,
pub actual_size: i64,
pub bucket: String,
pub version_id: Option<Uuid>,
pub etag: Option<String>,
pub mod_time: Option<OffsetDateTime>,
pub replication_status: ReplicationStatusType,
pub replication_status_internal: Option<String>,
pub delete_marker: bool,
pub version_purge_status_internal: Option<String>,
pub version_purge_status: VersionPurgeStatusType,
pub replication_state: Option<ReplicationState>,
pub op_type: ReplicationType,
pub event_type: String,
pub dsc: ReplicateDecision,
pub existing_obj_resync: ResyncDecision,
pub target_statuses: HashMap<String, ReplicationStatusType>,
pub target_purge_statuses: HashMap<String, VersionPurgeStatusType>,
pub replication_timestamp: Option<OffsetDateTime>,
pub ssec: bool,
pub user_tags: String,
pub checksum: Vec<u8>,
pub retry_count: u32,
}
impl ReplicationWorkerOperation for ReplicateObjectInfo {
fn as_any(&self) -> &dyn Any {
self
}
fn to_mrf_entry(&self) -> MrfReplicateEntry {
MrfReplicateEntry {
bucket: self.bucket.clone(),
object: self.name.clone(),
version_id: self.version_id,
retry_count: self.retry_count as i32,
size: self.size,
}
}
fn get_bucket(&self) -> &str {
&self.bucket
}
fn get_object(&self) -> &str {
&self.name
}
fn get_size(&self) -> i64 {
self.size
}
fn is_delete_marker(&self) -> bool {
self.delete_marker
}
fn get_op_type(&self) -> ReplicationType {
self.op_type
}
}
lazy_static::lazy_static! {
static ref REPL_STATUS_REGEX: Regex = Regex::new(r"([^=].*?)=([^,].*?);").unwrap();
}
impl ReplicateObjectInfo {
/// Returns replication status of a target
pub fn target_replication_status(&self, arn: &str) -> ReplicationStatusType {
let binding = self.replication_status_internal.clone().unwrap_or_default();
let captures = REPL_STATUS_REGEX.captures_iter(&binding);
for cap in captures {
if cap.len() == 3 && &cap[1] == arn {
return ReplicationStatusType::from(&cap[2]);
}
}
ReplicationStatusType::default()
}
/// Returns the relevant info needed by MRF
pub fn to_mrf_entry(&self) -> MrfReplicateEntry {
MrfReplicateEntry {
bucket: self.bucket.clone(),
object: self.name.clone(),
version_id: self.version_id,
retry_count: self.retry_count as i32,
size: self.size,
}
}
}
// constructs a replication status map from string representation
pub fn replication_statuses_map(s: &str) -> HashMap<String, ReplicationStatusType> {
let mut targets = HashMap::new();
let rep_stat_matches = REPL_STATUS_REGEX.captures_iter(s).map(|c| c.extract());
for (_, [arn, status]) in rep_stat_matches {
if arn.is_empty() {
continue;
}
let status = ReplicationStatusType::from(status);
targets.insert(arn.to_string(), status);
}
targets
}
// constructs a version purge status map from string representation
pub fn version_purge_statuses_map(s: &str) -> HashMap<String, VersionPurgeStatusType> {
let mut targets = HashMap::new();
let purge_status_matches = REPL_STATUS_REGEX.captures_iter(s).map(|c| c.extract());
for (_, [arn, status]) in purge_status_matches {
if arn.is_empty() {
continue;
}
let status = VersionPurgeStatusType::from(status);
targets.insert(arn.to_string(), status);
}
targets
}
pub fn get_replication_state(rinfos: &ReplicatedInfos, prev_state: &ReplicationState, _vid: Option<String>) -> ReplicationState {
let reset_status_map: Vec<(String, String)> = rinfos
.targets
.iter()
.filter(|v| !v.resync_timestamp.is_empty())
.map(|t| (target_reset_header(t.arn.as_str()), t.resync_timestamp.clone()))
.collect();
let repl_statuses = rinfos.replication_status_internal();
let vpurge_statuses = rinfos.version_purge_status_internal();
let mut reset_statuses_map = prev_state.reset_statuses_map.clone();
for (key, value) in reset_status_map {
reset_statuses_map.insert(key, value);
}
ReplicationState {
replicate_decision_str: prev_state.replicate_decision_str.clone(),
reset_statuses_map,
replica_timestamp: prev_state.replica_timestamp,
replica_status: prev_state.replica_status.clone(),
targets: replication_statuses_map(&repl_statuses.clone().unwrap_or_default()),
replication_status_internal: repl_statuses,
replication_timestamp: rinfos.replication_timestamp,
purge_targets: version_purge_statuses_map(&vpurge_statuses.clone().unwrap_or_default()),
version_purge_status_internal: vpurge_statuses,
..Default::default()
}
}

View File

@@ -30,7 +30,8 @@ use s3s::header::{
X_AMZ_STORAGE_CLASS, X_AMZ_WEBSITE_REDIRECT_LOCATION,
};
//use crate::disk::{BufferReader, Reader};
use crate::checksum::ChecksumMode;
use crate::client::checksum::ChecksumMode;
use crate::client::utils::base64_encode;
use crate::client::{
api_error_response::{err_entity_too_large, err_invalid_argument},
api_put_object_common::optimal_part_info,
@@ -41,7 +42,6 @@ use crate::client::{
transition_api::{ReaderImpl, TransitionClient, UploadInfo},
utils::{is_amz_header, is_minio_header, is_rustfs_header, is_standard_header, is_storageclass_header},
};
use rustfs_utils::crypto::base64_encode;
#[derive(Debug, Clone)]
pub struct AdvancedPutOptions {

View File

@@ -25,7 +25,8 @@ use time::OffsetDateTime;
use tracing::warn;
use uuid::Uuid;
use crate::checksum::ChecksumMode;
use crate::client::checksum::ChecksumMode;
use crate::client::utils::base64_encode;
use crate::client::{
api_error_response::{
err_entity_too_large, err_entity_too_small, err_invalid_argument, http_resp_to_error_response, to_error_response,
@@ -38,7 +39,7 @@ use crate::client::{
constants::{ISO8601_DATEFORMAT, MAX_PART_SIZE, MAX_SINGLE_PUT_OBJECT_SIZE},
transition_api::{ReaderImpl, RequestMetadata, TransitionClient, UploadInfo},
};
use rustfs_utils::{crypto::base64_encode, path::trim_etag};
use rustfs_utils::path::trim_etag;
use s3s::header::{X_AMZ_EXPIRATION, X_AMZ_VERSION_ID};
impl TransitionClient {

View File

@@ -29,7 +29,7 @@ use tokio_util::sync::CancellationToken;
use tracing::warn;
use uuid::Uuid;
use crate::checksum::{ChecksumMode, add_auto_checksum_headers, apply_auto_checksum};
use crate::client::checksum::{ChecksumMode, add_auto_checksum_headers, apply_auto_checksum};
use crate::client::{
api_error_response::{err_invalid_argument, err_unexpected_eof, http_resp_to_error_response},
api_put_object::PutObjectOptions,
@@ -40,7 +40,8 @@ use crate::client::{
transition_api::{ReaderImpl, RequestMetadata, TransitionClient, UploadInfo},
};
use rustfs_utils::{crypto::base64_encode, path::trim_etag};
use crate::client::utils::base64_encode;
use rustfs_utils::path::trim_etag;
use s3s::header::{X_AMZ_EXPIRATION, X_AMZ_VERSION_ID};
pub struct UploadedPartRes {

View File

@@ -20,7 +20,7 @@
use bytes::Bytes;
use http::{HeaderMap, HeaderValue, Method, StatusCode};
use rustfs_utils::{HashAlgorithm, crypto::base64_encode};
use rustfs_utils::HashAlgorithm;
use s3s::S3ErrorCode;
use s3s::dto::ReplicationStatus;
use s3s::header::X_AMZ_BYPASS_GOVERNANCE_RETENTION;
@@ -29,6 +29,7 @@ use std::{collections::HashMap, sync::Arc};
use time::OffsetDateTime;
use tokio::sync::mpsc::{self, Receiver, Sender};
use crate::client::utils::base64_encode;
use crate::client::{
api_error_response::{ErrorResponse, http_resp_to_error_response, to_error_response},
transition_api::{ReaderImpl, RequestMetadata, TransitionClient},

View File

@@ -23,9 +23,9 @@ use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use time::OffsetDateTime;
use crate::checksum::ChecksumMode;
use crate::client::checksum::ChecksumMode;
use crate::client::transition_api::ObjectMultipartInfo;
use rustfs_utils::crypto::base64_decode;
use crate::client::utils::base64_decode;
use super::transition_api;

View File

@@ -0,0 +1,351 @@
#![allow(clippy::map_entry)]
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![allow(unused_imports)]
#![allow(unused_variables)]
#![allow(unused_mut)]
#![allow(unused_assignments)]
#![allow(unused_must_use)]
#![allow(clippy::all)]
use lazy_static::lazy_static;
use rustfs_checksums::ChecksumAlgorithm;
use std::collections::HashMap;
use crate::client::utils::base64_decode;
use crate::client::utils::base64_encode;
use crate::client::{api_put_object::PutObjectOptions, api_s3_datatypes::ObjectPart};
use crate::{disk::DiskAPI, store_api::GetObjectReader};
use s3s::header::{
X_AMZ_CHECKSUM_ALGORITHM, X_AMZ_CHECKSUM_CRC32, X_AMZ_CHECKSUM_CRC32C, X_AMZ_CHECKSUM_SHA1, X_AMZ_CHECKSUM_SHA256,
};
use enumset::{EnumSet, EnumSetType, enum_set};
#[derive(Debug, EnumSetType, Default)]
#[enumset(repr = "u8")]
pub enum ChecksumMode {
#[default]
ChecksumNone,
ChecksumSHA256,
ChecksumSHA1,
ChecksumCRC32,
ChecksumCRC32C,
ChecksumCRC64NVME,
ChecksumFullObject,
}
lazy_static! {
static ref C_ChecksumMask: EnumSet<ChecksumMode> = {
let mut s = EnumSet::all();
s.remove(ChecksumMode::ChecksumFullObject);
s
};
static ref C_ChecksumFullObjectCRC32: EnumSet<ChecksumMode> =
enum_set!(ChecksumMode::ChecksumCRC32 | ChecksumMode::ChecksumFullObject);
static ref C_ChecksumFullObjectCRC32C: EnumSet<ChecksumMode> =
enum_set!(ChecksumMode::ChecksumCRC32C | ChecksumMode::ChecksumFullObject);
}
const AMZ_CHECKSUM_CRC64NVME: &str = "x-amz-checksum-crc64nvme";
impl ChecksumMode {
//pub const CRC64_NVME_POLYNOMIAL: i64 = 0xad93d23594c93659;
pub fn base(&self) -> ChecksumMode {
let s = EnumSet::from(*self).intersection(*C_ChecksumMask);
match s.as_u8() {
1_u8 => ChecksumMode::ChecksumNone,
2_u8 => ChecksumMode::ChecksumSHA256,
4_u8 => ChecksumMode::ChecksumSHA1,
8_u8 => ChecksumMode::ChecksumCRC32,
16_u8 => ChecksumMode::ChecksumCRC32C,
32_u8 => ChecksumMode::ChecksumCRC64NVME,
_ => panic!("enum err."),
}
}
pub fn is(&self, t: ChecksumMode) -> bool {
*self & t == t
}
pub fn key(&self) -> String {
//match c & checksumMask {
match self {
ChecksumMode::ChecksumCRC32 => {
return X_AMZ_CHECKSUM_CRC32.to_string();
}
ChecksumMode::ChecksumCRC32C => {
return X_AMZ_CHECKSUM_CRC32C.to_string();
}
ChecksumMode::ChecksumSHA1 => {
return X_AMZ_CHECKSUM_SHA1.to_string();
}
ChecksumMode::ChecksumSHA256 => {
return X_AMZ_CHECKSUM_SHA256.to_string();
}
ChecksumMode::ChecksumCRC64NVME => {
return AMZ_CHECKSUM_CRC64NVME.to_string();
}
_ => {
return "".to_string();
}
}
}
pub fn can_composite(&self) -> bool {
let s = EnumSet::from(*self).intersection(*C_ChecksumMask);
match s.as_u8() {
2_u8 => true,
4_u8 => true,
8_u8 => true,
16_u8 => true,
_ => false,
}
}
pub fn can_merge_crc(&self) -> bool {
let s = EnumSet::from(*self).intersection(*C_ChecksumMask);
match s.as_u8() {
8_u8 => true,
16_u8 => true,
32_u8 => true,
_ => false,
}
}
pub fn full_object_requested(&self) -> bool {
let s = EnumSet::from(*self).intersection(*C_ChecksumMask);
match s.as_u8() {
//C_ChecksumFullObjectCRC32 as u8 => true,
//C_ChecksumFullObjectCRC32C as u8 => true,
32_u8 => true,
_ => false,
}
}
pub fn key_capitalized(&self) -> String {
self.key()
}
pub fn raw_byte_len(&self) -> usize {
let u = EnumSet::from(*self).intersection(*C_ChecksumMask).as_u8();
if u == ChecksumMode::ChecksumCRC32 as u8 || u == ChecksumMode::ChecksumCRC32C as u8 {
4
} else if u == ChecksumMode::ChecksumSHA1 as u8 {
use sha1::Digest;
sha1::Sha1::output_size() as usize
} else if u == ChecksumMode::ChecksumSHA256 as u8 {
use sha2::Digest;
sha2::Sha256::output_size() as usize
} else if u == ChecksumMode::ChecksumCRC64NVME as u8 {
8
} else {
0
}
}
pub fn hasher(&self) -> Result<Box<dyn rustfs_checksums::http::HttpChecksum>, std::io::Error> {
match /*C_ChecksumMask & **/self {
ChecksumMode::ChecksumCRC32 => {
return Ok(ChecksumAlgorithm::Crc32.into_impl());
}
ChecksumMode::ChecksumCRC32C => {
return Ok(ChecksumAlgorithm::Crc32c.into_impl());
}
ChecksumMode::ChecksumSHA1 => {
return Ok(ChecksumAlgorithm::Sha1.into_impl());
}
ChecksumMode::ChecksumSHA256 => {
return Ok(ChecksumAlgorithm::Sha256.into_impl());
}
ChecksumMode::ChecksumCRC64NVME => {
return Ok(ChecksumAlgorithm::Crc64Nvme.into_impl());
}
_ => return Err(std::io::Error::other("unsupported checksum type")),
}
}
pub fn is_set(&self) -> bool {
let s = EnumSet::from(*self).intersection(*C_ChecksumMask);
s.len() == 1
}
pub fn set_default(&mut self, t: ChecksumMode) {
if !self.is_set() {
*self = t;
}
}
pub fn encode_to_string(&self, b: &[u8]) -> Result<String, std::io::Error> {
if !self.is_set() {
return Ok("".to_string());
}
let mut h = self.hasher()?;
h.update(b);
let hash = h.finalize();
Ok(base64_encode(hash.as_ref()))
}
pub fn to_string(&self) -> String {
//match c & checksumMask {
match self {
ChecksumMode::ChecksumCRC32 => {
return "CRC32".to_string();
}
ChecksumMode::ChecksumCRC32C => {
return "CRC32C".to_string();
}
ChecksumMode::ChecksumSHA1 => {
return "SHA1".to_string();
}
ChecksumMode::ChecksumSHA256 => {
return "SHA256".to_string();
}
ChecksumMode::ChecksumNone => {
return "".to_string();
}
ChecksumMode::ChecksumCRC64NVME => {
return "CRC64NVME".to_string();
}
_ => {
return "<invalid>".to_string();
}
}
}
// pub fn check_sum_reader(&self, r: GetObjectReader) -> Result<Checksum, std::io::Error> {
// let mut h = self.hasher()?;
// Ok(Checksum::new(self.clone(), h.sum().as_bytes()))
// }
// pub fn check_sum_bytes(&self, b: &[u8]) -> Result<Checksum, std::io::Error> {
// let mut h = self.hasher()?;
// Ok(Checksum::new(self.clone(), h.sum().as_bytes()))
// }
pub fn composite_checksum(&self, p: &mut [ObjectPart]) -> Result<Checksum, std::io::Error> {
if !self.can_composite() {
return Err(std::io::Error::other("cannot do composite checksum"));
}
p.sort_by(|i, j| {
if i.part_num < j.part_num {
std::cmp::Ordering::Less
} else if i.part_num > j.part_num {
std::cmp::Ordering::Greater
} else {
std::cmp::Ordering::Equal
}
});
let c = self.base();
let crc_bytes = Vec::<u8>::with_capacity(p.len() * self.raw_byte_len() as usize);
let mut h = self.hasher()?;
h.update(crc_bytes.as_ref());
let hash = h.finalize();
Ok(Checksum {
checksum_type: self.clone(),
r: hash.as_ref().to_vec(),
computed: false,
})
}
pub fn full_object_checksum(&self, p: &mut [ObjectPart]) -> Result<Checksum, std::io::Error> {
todo!();
}
}
#[derive(Default)]
pub struct Checksum {
checksum_type: ChecksumMode,
r: Vec<u8>,
computed: bool,
}
#[allow(dead_code)]
impl Checksum {
fn new(t: ChecksumMode, b: &[u8]) -> Checksum {
if t.is_set() && b.len() == t.raw_byte_len() {
return Checksum {
checksum_type: t,
r: b.to_vec(),
computed: false,
};
}
Checksum::default()
}
#[allow(dead_code)]
fn new_checksum_string(t: ChecksumMode, s: &str) -> Result<Checksum, std::io::Error> {
let b = match base64_decode(s.as_bytes()) {
Ok(b) => b,
Err(err) => return Err(std::io::Error::other(err.to_string())),
};
if t.is_set() && b.len() == t.raw_byte_len() {
return Ok(Checksum {
checksum_type: t,
r: b,
computed: false,
});
}
Ok(Checksum::default())
}
fn is_set(&self) -> bool {
self.checksum_type.is_set() && self.r.len() == self.checksum_type.raw_byte_len()
}
fn encoded(&self) -> String {
if !self.is_set() {
return "".to_string();
}
base64_encode(&self.r)
}
#[allow(dead_code)]
fn raw(&self) -> Option<Vec<u8>> {
if !self.is_set() {
return None;
}
Some(self.r.clone())
}
}
pub fn add_auto_checksum_headers(opts: &mut PutObjectOptions) {
opts.user_metadata
.insert("X-Amz-Checksum-Algorithm".to_string(), opts.auto_checksum.to_string());
if opts.auto_checksum.full_object_requested() {
opts.user_metadata
.insert("X-Amz-Checksum-Type".to_string(), "FULL_OBJECT".to_string());
}
}
pub fn apply_auto_checksum(opts: &mut PutObjectOptions, all_parts: &mut [ObjectPart]) -> Result<(), std::io::Error> {
if opts.auto_checksum.can_composite() && !opts.auto_checksum.is(ChecksumMode::ChecksumFullObject) {
let crc = opts.auto_checksum.composite_checksum(all_parts)?;
opts.user_metadata = {
let mut hm = HashMap::new();
hm.insert(opts.auto_checksum.key(), crc.encoded());
hm
}
} else if opts.auto_checksum.can_merge_crc() {
let crc = opts.auto_checksum.full_object_checksum(all_parts)?;
opts.user_metadata = {
let mut hm = HashMap::new();
hm.insert(opts.auto_checksum.key_capitalized(), crc.encoded());
hm.insert("X-Amz-Checksum-Type".to_string(), "FULL_OBJECT".to_string());
hm
}
}
Ok(())
}

View File

@@ -30,6 +30,7 @@ pub mod api_restore;
pub mod api_s3_datatypes;
pub mod api_stat;
pub mod bucket_cache;
pub mod checksum;
pub mod constants;
pub mod credentials;
pub mod object_api_utils;

View File

@@ -20,8 +20,8 @@
#![allow(clippy::all)]
use http::HeaderMap;
use std::io::Cursor;
use std::{collections::HashMap, sync::Arc};
use s3s::dto::ETag;
use std::{collections::HashMap, io::Cursor, sync::Arc};
use tokio::io::BufReader;
use crate::error::ErrorResponse;
@@ -148,27 +148,30 @@ pub fn new_getobjectreader(
Ok((get_fn, off as i64, length as i64))
}
/// Format an ETag value according to HTTP standards (wrap with quotes if not already wrapped)
pub fn format_etag(etag: &str) -> String {
if etag.starts_with('"') && etag.ends_with('"') {
// Already properly formatted
etag.to_string()
} else if etag.starts_with("W/\"") && etag.ends_with('"') {
// Already a weak ETag, properly formatted
etag.to_string()
} else {
// Need to wrap with quotes
format!("\"{}\"", etag)
/// Convert a raw stored ETag into the strongly-typed `s3s::dto::ETag`.
///
/// Supports already quoted (`"abc"`), weak (`W/"abc"`), or plain (`abc`) values.
pub fn to_s3s_etag(etag: &str) -> ETag {
if let Some(rest) = etag.strip_prefix("W/\"") {
if let Some(body) = rest.strip_suffix('"') {
return ETag::Weak(body.to_string());
}
return ETag::Weak(rest.to_string());
}
if let Some(body) = etag.strip_prefix('"').and_then(|rest| rest.strip_suffix('"')) {
return ETag::Strong(body.to_string());
}
ETag::Strong(etag.to_string())
}
pub fn extract_etag(metadata: &HashMap<String, String>) -> String {
let etag = if let Some(etag) = metadata.get("etag") {
etag.clone()
} else {
metadata["md5Sum"].clone()
};
format_etag(&etag)
pub fn get_raw_etag(metadata: &HashMap<String, String>) -> String {
metadata
.get("etag")
.cloned()
.or_else(|| metadata.get("md5Sum").cloned())
.unwrap_or_default()
}
#[cfg(test)]
@@ -176,30 +179,28 @@ mod tests {
use super::*;
#[test]
fn test_format_etag() {
// Test unquoted ETag - should add quotes
assert_eq!(format_etag("6af8d12c0c74b78094884349f3c8a079"), "\"6af8d12c0c74b78094884349f3c8a079\"");
// Test already quoted ETag - should not double quote
fn test_to_s3s_etag() {
// Test unquoted ETag - should become strong etag
assert_eq!(
format_etag("\"6af8d12c0c74b78094884349f3c8a079\""),
"\"6af8d12c0c74b78094884349f3c8a079\""
to_s3s_etag("6af8d12c0c74b78094884349f3c8a079"),
ETag::Strong("6af8d12c0c74b78094884349f3c8a079".to_string())
);
// Test weak ETag - should keep as is
assert_eq!(
format_etag("W/\"6af8d12c0c74b78094884349f3c8a079\""),
"W/\"6af8d12c0c74b78094884349f3c8a079\""
to_s3s_etag("\"6af8d12c0c74b78094884349f3c8a079\""),
ETag::Strong("6af8d12c0c74b78094884349f3c8a079".to_string())
);
// Test empty ETag - should add quotes
assert_eq!(format_etag(""), "\"\"");
assert_eq!(
to_s3s_etag("W/\"6af8d12c0c74b78094884349f3c8a079\""),
ETag::Weak("6af8d12c0c74b78094884349f3c8a079".to_string())
);
// Test malformed quote (only starting quote) - should wrap properly
assert_eq!(format_etag("\"incomplete"), "\"\"incomplete\"");
assert_eq!(to_s3s_etag(""), ETag::Strong(String::new()));
// Test malformed quote (only ending quote) - should wrap properly
assert_eq!(format_etag("incomplete\""), "\"incomplete\"\"");
assert_eq!(to_s3s_etag("\"incomplete"), ETag::Strong("\"incomplete".to_string()));
assert_eq!(to_s3s_etag("incomplete\""), ETag::Strong("incomplete\"".to_string()));
}
#[test]
@@ -208,15 +209,17 @@ mod tests {
// Test with etag field
metadata.insert("etag".to_string(), "abc123".to_string());
assert_eq!(extract_etag(&metadata), "\"abc123\"");
assert_eq!(get_raw_etag(&metadata), "abc123");
// Test with already quoted etag field
metadata.insert("etag".to_string(), "\"def456\"".to_string());
assert_eq!(extract_etag(&metadata), "\"def456\"");
assert_eq!(get_raw_etag(&metadata), "\"def456\"");
// Test fallback to md5Sum
metadata.remove("etag");
metadata.insert("md5Sum".to_string(), "xyz789".to_string());
assert_eq!(extract_etag(&metadata), "\"xyz789\"");
assert_eq!(get_raw_etag(&metadata), "xyz789");
metadata.clear();
assert_eq!(get_raw_etag(&metadata), "");
}
}

View File

@@ -61,7 +61,7 @@ use crate::client::{
constants::{UNSIGNED_PAYLOAD, UNSIGNED_PAYLOAD_TRAILER},
credentials::{CredContext, Credentials, SignatureType, Static},
};
use crate::{checksum::ChecksumMode, store_api::GetObjectReader};
use crate::{client::checksum::ChecksumMode, store_api::GetObjectReader};
use rustfs_rio::HashReader;
use rustfs_utils::{
net::get_endpoint_url,

View File

@@ -90,3 +90,11 @@ pub fn is_rustfs_header(header_key: &str) -> bool {
pub fn is_minio_header(header_key: &str) -> bool {
header_key.to_lowercase().starts_with("x-minio-")
}
pub fn base64_encode(input: &[u8]) -> String {
base64_simd::URL_SAFE_NO_PAD.encode_to_string(input)
}
pub fn base64_decode(input: &[u8]) -> Result<Vec<u8>, base64_simd::Error> {
base64_simd::URL_SAFE_NO_PAD.decode_to_vec(input)
}

View File

@@ -2087,6 +2087,7 @@ impl DiskAPI for LocalDisk {
for vol in volumes {
if let Err(e) = self.make_volume(vol).await {
if e != DiskError::VolumeExists {
error!("local disk make volumes failed: {e}");
return Err(e);
}
}
@@ -2108,6 +2109,7 @@ impl DiskAPI for LocalDisk {
os::make_dir_all(&volume_dir, self.root.as_path()).await?;
return Ok(());
}
error!("local disk make volume failed: {e}");
return Err(to_volume_error(e).into());
}

View File

@@ -167,19 +167,8 @@ async fn write_data_blocks<W>(
where
W: tokio::io::AsyncWrite + Send + Sync + Unpin,
{
let available = get_data_block_len(en_blocks, data_blocks);
if available < length {
let block_sizes: Vec<usize> = en_blocks
.iter()
.take(data_blocks)
.map(|block| block.as_ref().map(|buf| buf.len()).unwrap_or(0))
.collect();
error!(
expected = length,
available,
?block_sizes,
"write_data_blocks get_data_block_len < length"
);
if get_data_block_len(en_blocks, data_blocks) < length {
error!("write_data_blocks get_data_block_len < length");
return Err(io::Error::new(ErrorKind::UnexpectedEof, "Not enough data blocks to write"));
}
@@ -312,6 +301,10 @@ impl Erasure {
written += n;
}
if ret_err.is_some() {
return (written, ret_err);
}
if written < length {
ret_err = Some(Error::LessData.into());
}

View File

@@ -145,7 +145,9 @@ impl Erasure {
return Err(std::io::Error::other(format!("Failed to send encoded data : {err}")));
}
}
Ok(_) => break,
Ok(_) => {
break;
}
Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
break;
}

View File

@@ -468,15 +468,21 @@ impl Erasure {
let mut buf = vec![0u8; block_size];
match rustfs_utils::read_full(&mut *reader, &mut buf).await {
Ok(n) if n > 0 => {
warn!("encode_stream_callback_async read n={}", n);
total += n;
let res = self.encode_data(&buf[..n]);
on_block(res).await?
}
Ok(_) => break,
Ok(_) => {
warn!("encode_stream_callback_async read unexpected ok");
break;
}
Err(e) if e.kind() == std::io::ErrorKind::UnexpectedEof => {
warn!("encode_stream_callback_async read unexpected eof");
break;
}
Err(e) => {
warn!("encode_stream_callback_async read error={:?}", e);
on_block(Err(e)).await?;
break;
}

View File

@@ -38,7 +38,7 @@ pub const DISK_RESERVE_FRACTION: f64 = 0.15;
lazy_static! {
static ref GLOBAL_RUSTFS_PORT: OnceLock<u16> = OnceLock::new();
static ref GLOBAL_RUSTFS_EXTERNAL_PORT: OnceLock<u16> = OnceLock::new();
static ref globalDeploymentIDPtr: OnceLock<Uuid> = OnceLock::new();
pub static ref GLOBAL_OBJECT_API: OnceLock<Arc<ECStore>> = OnceLock::new();
pub static ref GLOBAL_LOCAL_DISK: Arc<RwLock<Vec<Option<DiskStore>>>> = Arc::new(RwLock::new(Vec::new()));
pub static ref GLOBAL_IsErasure: RwLock<bool> = RwLock::new(false);
@@ -51,8 +51,6 @@ lazy_static! {
pub static ref GLOBAL_TierConfigMgr: Arc<RwLock<TierConfigMgr>> = TierConfigMgr::new();
pub static ref GLOBAL_LifecycleSys: Arc<LifecycleSys> = LifecycleSys::new();
pub static ref GLOBAL_EventNotifier: Arc<RwLock<EventNotifier>> = EventNotifier::new();
//pub static ref GLOBAL_RemoteTargetTransport
static ref globalDeploymentIDPtr: OnceLock<Uuid> = OnceLock::new();
pub static ref GLOBAL_BOOT_TIME: OnceCell<SystemTime> = OnceCell::new();
pub static ref GLOBAL_LocalNodeName: String = "127.0.0.1:9000".to_string();
pub static ref GLOBAL_LocalNodeNameHex: String = rustfs_utils::crypto::hex(GLOBAL_LocalNodeName.as_bytes());
@@ -60,12 +58,22 @@ lazy_static! {
pub static ref GLOBAL_REGION: OnceLock<String> = OnceLock::new();
}
// Global cancellation token for background services (data scanner and auto heal)
/// Global cancellation token for background services (data scanner and auto heal)
static GLOBAL_BACKGROUND_SERVICES_CANCEL_TOKEN: OnceLock<CancellationToken> = OnceLock::new();
/// Global active credentials
static GLOBAL_ACTIVE_CRED: OnceLock<Credentials> = OnceLock::new();
pub fn init_global_action_cred(ak: Option<String>, sk: Option<String>) {
/// Initialize the global action credentials
///
/// # Arguments
/// * `ak` - Optional access key
/// * `sk` - Optional secret key
///
/// # Returns
/// * None
///
pub fn init_global_action_credentials(ak: Option<String>, sk: Option<String>) {
let ak = {
if let Some(k) = ak {
k
@@ -91,11 +99,16 @@ pub fn init_global_action_cred(ak: Option<String>, sk: Option<String>) {
.unwrap();
}
/// Get the global action credentials
pub fn get_global_action_cred() -> Option<Credentials> {
GLOBAL_ACTIVE_CRED.get().cloned()
}
/// Get the global rustfs port
///
/// # Returns
/// * `u16` - The global rustfs port
///
pub fn global_rustfs_port() -> u16 {
if let Some(p) = GLOBAL_RUSTFS_PORT.get() {
*p
@@ -105,36 +118,44 @@ pub fn global_rustfs_port() -> u16 {
}
/// Set the global rustfs port
///
/// # Arguments
/// * `value` - The port value to set globally
///
/// # Returns
/// * None
pub fn set_global_rustfs_port(value: u16) {
GLOBAL_RUSTFS_PORT.set(value).expect("set_global_rustfs_port fail");
}
/// Get the global rustfs external port
pub fn global_rustfs_external_port() -> u16 {
if let Some(p) = GLOBAL_RUSTFS_EXTERNAL_PORT.get() {
*p
} else {
rustfs_config::DEFAULT_PORT
}
}
/// Set the global rustfs external port
pub fn set_global_rustfs_external_port(value: u16) {
GLOBAL_RUSTFS_EXTERNAL_PORT
.set(value)
.expect("set_global_rustfs_external_port fail");
}
/// Get the global rustfs port
/// Set the global deployment id
///
/// # Arguments
/// * `id` - The Uuid to set as the global deployment id
///
/// # Returns
/// * None
///
pub fn set_global_deployment_id(id: Uuid) {
globalDeploymentIDPtr.set(id).unwrap();
}
/// Get the global deployment id
///
/// # Returns
/// * `Option<String>` - The global deployment id as a string, if set
///
pub fn get_global_deployment_id() -> Option<String> {
globalDeploymentIDPtr.get().map(|v| v.to_string())
}
/// Get the global deployment id
/// Set the global endpoints
///
/// # Arguments
/// * `eps` - A vector of PoolEndpoints to set globally
///
/// # Returns
/// * None
///
pub fn set_global_endpoints(eps: Vec<PoolEndpoints>) {
GLOBAL_Endpoints
.set(EndpointServerPools::from(eps))
@@ -142,6 +163,10 @@ pub fn set_global_endpoints(eps: Vec<PoolEndpoints>) {
}
/// Get the global endpoints
///
/// # Returns
/// * `EndpointServerPools` - The global endpoints
///
pub fn get_global_endpoints() -> EndpointServerPools {
if let Some(eps) = GLOBAL_Endpoints.get() {
eps.clone()
@@ -150,29 +175,63 @@ pub fn get_global_endpoints() -> EndpointServerPools {
}
}
/// Create a new object layer instance
///
/// # Returns
/// * `Option<Arc<ECStore>>` - The global object layer instance, if set
///
pub fn new_object_layer_fn() -> Option<Arc<ECStore>> {
GLOBAL_OBJECT_API.get().cloned()
}
/// Set the global object layer
///
/// # Arguments
/// * `o` - The ECStore instance to set globally
///
/// # Returns
/// * None
pub async fn set_object_layer(o: Arc<ECStore>) {
GLOBAL_OBJECT_API.set(o).expect("set_object_layer fail ")
}
/// Check if the setup type is distributed erasure coding
///
/// # Returns
/// * `bool` - True if the setup type is distributed erasure coding, false otherwise
///
pub async fn is_dist_erasure() -> bool {
let lock = GLOBAL_IsDistErasure.read().await;
*lock
}
/// Check if the setup type is erasure coding with single data center
///
/// # Returns
/// * `bool` - True if the setup type is erasure coding with single data center, false otherwise
///
pub async fn is_erasure_sd() -> bool {
let lock = GLOBAL_IsErasureSD.read().await;
*lock
}
/// Check if the setup type is erasure coding
///
/// # Returns
/// * `bool` - True if the setup type is erasure coding, false otherwise
///
pub async fn is_erasure() -> bool {
let lock = GLOBAL_IsErasure.read().await;
*lock
}
/// Update the global erasure type based on the setup type
///
/// # Arguments
/// * `setup_type` - The SetupType to update the global erasure type
///
/// # Returns
/// * None
pub async fn update_erasure_type(setup_type: SetupType) {
let mut is_erasure = GLOBAL_IsErasure.write().await;
*is_erasure = setup_type == SetupType::Erasure;
@@ -198,25 +257,53 @@ pub async fn update_erasure_type(setup_type: SetupType) {
type TypeLocalDiskSetDrives = Vec<Vec<Vec<Option<DiskStore>>>>;
/// Set the global region
///
/// # Arguments
/// * `region` - The region string to set globally
///
/// # Returns
/// * None
pub fn set_global_region(region: String) {
GLOBAL_REGION.set(region).unwrap();
}
/// Get the global region
///
/// # Returns
/// * `Option<String>` - The global region string, if set
///
pub fn get_global_region() -> Option<String> {
GLOBAL_REGION.get().cloned()
}
/// Initialize the global background services cancellation token
///
/// # Arguments
/// * `cancel_token` - The CancellationToken instance to set globally
///
/// # Returns
/// * `Ok(())` if successful
/// * `Err(CancellationToken)` if setting fails
///
pub fn init_background_services_cancel_token(cancel_token: CancellationToken) -> Result<(), CancellationToken> {
GLOBAL_BACKGROUND_SERVICES_CANCEL_TOKEN.set(cancel_token)
}
/// Get the global background services cancellation token
///
/// # Returns
/// * `Option<&'static CancellationToken>` - The global cancellation token, if set
///
pub fn get_background_services_cancel_token() -> Option<&'static CancellationToken> {
GLOBAL_BACKGROUND_SERVICES_CANCEL_TOKEN.get()
}
/// Create and initialize the global background services cancellation token
///
/// # Returns
/// * `CancellationToken` - The newly created global cancellation token
///
pub fn create_background_services_cancel_token() -> CancellationToken {
let cancel_token = CancellationToken::new();
init_background_services_cancel_token(cancel_token.clone()).expect("Background services cancel token already initialized");
@@ -224,6 +311,9 @@ pub fn create_background_services_cancel_token() -> CancellationToken {
}
/// Shutdown all background services gracefully
///
/// # Returns
/// * None
pub fn shutdown_background_services() {
if let Some(cancel_token) = GLOBAL_BACKGROUND_SERVICES_CANCEL_TOKEN.get() {
cancel_token.cancel();

View File

@@ -33,7 +33,6 @@ pub mod file_cache;
pub mod global;
pub mod metrics_realtime;
pub mod notification_sys;
pub mod object_append;
pub mod pools;
pub mod rebalance;
pub mod rpc;
@@ -45,7 +44,7 @@ mod store_init;
pub mod store_list_objects;
pub mod store_utils;
pub mod checksum;
// pub mod checksum;
pub mod client;
pub mod event;
pub mod event_notification;

View File

@@ -23,7 +23,7 @@ use rustfs_common::{
use rustfs_madmin::metrics::{DiskIOStats, DiskMetric, RealtimeMetrics};
use rustfs_utils::os::get_drive_stats;
use serde::{Deserialize, Serialize};
use tracing::info;
use tracing::{debug, info};
use crate::{
admin_server_info::get_local_server_property,
@@ -44,7 +44,7 @@ pub struct CollectMetricsOpts {
pub struct MetricType(u32);
impl MetricType {
// 定义一些常量
// Define some constants
pub const NONE: MetricType = MetricType(0);
pub const SCANNER: MetricType = MetricType(1 << 0);
pub const DISK: MetricType = MetricType(1 << 1);
@@ -70,8 +70,18 @@ impl MetricType {
}
}
/// Collect local metrics based on the specified types and options.
///
/// # Arguments
///
/// * `types` - A `MetricType` specifying which types of metrics to collect.
/// * `opts` - A reference to `CollectMetricsOpts` containing additional options for metric collection.
///
/// # Returns
/// * A `RealtimeMetrics` struct containing the collected metrics.
///
pub async fn collect_local_metrics(types: MetricType, opts: &CollectMetricsOpts) -> RealtimeMetrics {
info!("collect_local_metrics");
debug!("collect_local_metrics");
let mut real_time_metrics = RealtimeMetrics::default();
if types.0 == MetricType::NONE.0 {
info!("types is None, return");
@@ -93,13 +103,13 @@ pub async fn collect_local_metrics(types: MetricType, opts: &CollectMetricsOpts)
}
if types.contains(&MetricType::DISK) {
info!("start get disk metrics");
debug!("start get disk metrics");
let mut aggr = DiskMetric {
collected_at: Utc::now(),
..Default::default()
};
for (name, disk) in collect_local_disks_metrics(&opts.disks).await.into_iter() {
info!("got disk metric, name: {name}, metric: {disk:?}");
debug!("got disk metric, name: {name}, metric: {disk:?}");
real_time_metrics.by_disk.insert(name, disk.clone());
aggr.merge(&disk);
}
@@ -107,7 +117,7 @@ pub async fn collect_local_metrics(types: MetricType, opts: &CollectMetricsOpts)
}
if types.contains(&MetricType::SCANNER) {
info!("start get scanner metrics");
debug!("start get scanner metrics");
let metrics = globalMetrics.report().await;
real_time_metrics.aggregated.scanner = Some(metrics);
}

View File

@@ -1,725 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::bitrot::{create_bitrot_reader, create_bitrot_writer};
use crate::erasure_coding::{Erasure, calc_shard_size};
use crate::error::{Error, StorageError};
use crate::store_api::ObjectInfo;
use rustfs_filemeta::TRANSITION_COMPLETE;
use rustfs_utils::HashAlgorithm;
use rustfs_utils::http::headers::{
AMZ_SERVER_SIDE_ENCRYPTION, AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM, AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY,
AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5, AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT, AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID,
RESERVED_METADATA_PREFIX_LOWER,
};
use std::collections::HashSet;
/// Ensure the target object can accept append writes under current state.
pub fn validate_append_preconditions(bucket: &str, object: &str, info: &ObjectInfo) -> Result<(), Error> {
if info.is_compressed() {
return Err(StorageError::InvalidArgument(
bucket.to_string(),
object.to_string(),
"append is not supported for compressed objects".to_string(),
));
}
let encryption_headers = [
AMZ_SERVER_SIDE_ENCRYPTION,
AMZ_SERVER_SIDE_ENCRYPTION_KMS_ID,
AMZ_SERVER_SIDE_ENCRYPTION_KMS_CONTEXT,
AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM,
AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY,
AMZ_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5,
];
if encryption_headers
.iter()
.any(|header| info.user_defined.contains_key(*header) || info.user_defined.contains_key(&header.to_ascii_lowercase()))
{
return Err(StorageError::InvalidArgument(
bucket.to_string(),
object.to_string(),
"append is not supported for encrypted objects".to_string(),
));
}
if info.transitioned_object.status == TRANSITION_COMPLETE || !info.transitioned_object.tier.is_empty() {
return Err(StorageError::InvalidArgument(
bucket.to_string(),
object.to_string(),
"append is not supported for transitioned objects".to_string(),
));
}
Ok(())
}
/// Validate that the requested append position matches the current object length.
pub fn validate_append_position(bucket: &str, object: &str, info: &ObjectInfo, expected_position: i64) -> Result<(), Error> {
if expected_position != info.size {
return Err(StorageError::InvalidArgument(
bucket.to_string(),
object.to_string(),
format!("append position mismatch: provided {}, expected {}", expected_position, info.size),
));
}
Ok(())
}
pub struct InlineAppendContext<'a> {
pub existing_inline: Option<&'a [u8]>,
pub existing_plain: Option<&'a [u8]>,
pub existing_size: i64,
pub append_payload: &'a [u8],
pub erasure: &'a Erasure,
pub hash_algorithm: HashAlgorithm,
pub has_checksums: bool,
}
pub struct InlineAppendResult {
pub inline_data: Vec<u8>,
pub total_size: i64,
pub etag: String,
}
/// Decode inline payload using available checksum algorithms. Returns raw bytes when decoding fails but
/// the inline buffer already contains the plain payload.
pub async fn decode_inline_payload(
inline: &[u8],
size: usize,
erasure: &Erasure,
preferred: HashAlgorithm,
) -> Result<(Vec<u8>, HashAlgorithm), Error> {
match decode_inline_variants(inline, size, erasure, preferred).await {
Ok((data, algo)) => Ok((data, algo)),
Err(err) => {
if inline.len() >= size {
Ok((inline[..size].to_vec(), HashAlgorithm::None))
} else {
Err(err)
}
}
}
}
/// Append data to an inline object and return the re-encoded inline buffer.
pub async fn append_inline_data(ctx: InlineAppendContext<'_>) -> Result<InlineAppendResult, Error> {
let mut plain = Vec::with_capacity(ctx.existing_inline.map(|data| data.len()).unwrap_or(0) + ctx.append_payload.len());
let mut encode_algorithm = ctx.hash_algorithm.clone();
if let Some(existing_plain) = ctx.existing_plain {
if existing_plain.len() != ctx.existing_size as usize {
return Err(StorageError::other("existing plain payload length mismatch"));
}
plain.extend_from_slice(existing_plain);
} else if ctx.existing_size > 0 {
let inline = ctx
.existing_inline
.ok_or_else(|| StorageError::other("inline payload missing"))?;
let (decoded, detected_algo) =
decode_inline_payload(inline, ctx.existing_size as usize, ctx.erasure, ctx.hash_algorithm.clone()).await?;
encode_algorithm = detected_algo;
plain.extend_from_slice(&decoded);
} else if let Some(inline) = ctx.existing_inline {
plain.extend_from_slice(inline);
}
plain.extend_from_slice(ctx.append_payload);
let total_size = plain.len() as i64;
let etag = md5_hex(&plain);
if encode_algorithm == HashAlgorithm::None {
if ctx.has_checksums {
encode_algorithm = ctx.hash_algorithm.clone();
} else {
return Ok(InlineAppendResult {
inline_data: plain,
total_size,
etag,
});
}
}
let mut writer = create_bitrot_writer(
true,
None,
"",
"",
ctx.erasure.shard_file_size(total_size),
ctx.erasure.shard_size(),
encode_algorithm,
)
.await
.map_err(|e| StorageError::other(format!("failed to create inline writer: {e}")))?;
let mut remaining = plain.as_slice();
while !remaining.is_empty() {
let chunk_len = remaining.len().min(ctx.erasure.block_size);
writer
.write(&remaining[..chunk_len])
.await
.map_err(|e| StorageError::other(format!("failed to write inline data: {e}")))?;
remaining = &remaining[chunk_len..];
}
writer
.shutdown()
.await
.map_err(|e| StorageError::other(format!("failed to finalize inline writer: {e}")))?;
let inline_data = writer
.into_inline_data()
.ok_or_else(|| StorageError::other("inline writer did not return data"))?;
Ok(InlineAppendResult {
inline_data,
total_size,
etag,
})
}
fn md5_hex(data: &[u8]) -> String {
let digest = HashAlgorithm::Md5.hash_encode(data);
hex_from_bytes(digest.as_ref())
}
fn hex_from_bytes(bytes: &[u8]) -> String {
let mut out = String::with_capacity(bytes.len() * 2);
for byte in bytes {
use std::fmt::Write;
write!(&mut out, "{:02x}", byte).expect("write hex");
}
out
}
async fn decode_inline_variants(
inline: &[u8],
size: usize,
erasure: &Erasure,
preferred: HashAlgorithm,
) -> Result<(Vec<u8>, HashAlgorithm), Error> {
let mut tried = HashSet::new();
let candidates = [preferred, HashAlgorithm::HighwayHash256, HashAlgorithm::HighwayHash256S];
let mut last_err: Option<Error> = None;
for algo in candidates {
if !tried.insert(algo.clone()) {
continue;
}
match decode_inline_with_algo(inline, size, erasure, algo.clone()).await {
Ok(data) => return Ok((data, algo)),
Err(err) => last_err = Some(err),
}
}
Err(last_err.unwrap_or_else(|| StorageError::other("failed to decode inline data")))
}
async fn decode_inline_with_algo(inline: &[u8], size: usize, erasure: &Erasure, algo: HashAlgorithm) -> Result<Vec<u8>, Error> {
let total_len = inline
.len()
.max(erasure.shard_file_size(size as i64).max(size as i64) as usize);
let mut reader = create_bitrot_reader(Some(inline), None, "", "", 0, total_len, erasure.shard_size(), algo)
.await
.map_err(|e| StorageError::other(format!("failed to create inline reader: {e}")))?
.ok_or_else(|| StorageError::other("inline reader unavailable"))?;
let mut out = Vec::with_capacity(size);
while out.len() < size {
let remaining = size - out.len();
let plain_chunk = remaining.min(erasure.block_size);
let shard_payload = calc_shard_size(plain_chunk, erasure.data_shards).max(1);
let mut buf = vec![0u8; shard_payload];
let read = reader
.read(&mut buf)
.await
.map_err(|e| StorageError::other(format!("failed to read inline data: {e}")))?;
if read == 0 {
return Err(StorageError::other("incomplete inline data read"));
}
let copy_len = remaining.min(read);
out.extend_from_slice(&buf[..copy_len]);
}
Ok(out)
}
/// Background task to spill inline data to segmented format
pub struct InlineSpillProcessor {
pub disks: Vec<Option<crate::disk::DiskStore>>,
pub write_quorum: usize,
}
impl InlineSpillProcessor {
pub fn new(disks: Vec<Option<crate::disk::DiskStore>>, write_quorum: usize) -> Self {
Self { disks, write_quorum }
}
/// Process a single spill operation from InlinePendingSpill to SegmentedActive
pub async fn process_spill(
&self,
bucket: &str,
object: &str,
mut fi: rustfs_filemeta::FileInfo,
mut parts_metadata: Vec<rustfs_filemeta::FileInfo>,
epoch: u64,
) -> Result<(), Error> {
use rustfs_filemeta::AppendStateKind;
use tracing::{debug, error, info, warn};
// Verify we're in the correct state
let current_state = fi.get_append_state();
if current_state.state != AppendStateKind::InlinePendingSpill {
warn!(
bucket = bucket,
object = object,
current_state = ?current_state.state,
"Spill processor called on object not in InlinePendingSpill state"
);
return Ok(());
}
// Check epoch to ensure we're processing the correct version
if current_state.epoch != epoch {
debug!(
bucket = bucket,
object = object,
current_epoch = current_state.epoch,
expected_epoch = epoch,
"Spill operation skipped due to epoch mismatch"
);
return Ok(());
}
info!(
bucket = bucket,
object = object,
size = fi.size,
epoch = epoch,
"Starting inline data spill to segmented format"
);
// Extract inline data
let inline_data = fi
.data
.clone()
.ok_or_else(|| StorageError::other("Cannot spill object without inline data"))?;
// Create erasure encoder
let erasure = Erasure::new(fi.erasure.data_blocks, fi.erasure.parity_blocks, fi.erasure.block_size);
// Decode inline data to plain data
let hash_algorithm = fi
.parts
.first()
.map(|part| fi.erasure.get_checksum_info(part.number).algorithm)
.unwrap_or(HashAlgorithm::HighwayHash256);
let plain_data = match decode_inline_payload(&inline_data, fi.size as usize, &erasure, hash_algorithm.clone()).await {
Ok((plain, _detected_algo)) => plain,
Err(err) => {
error!(
bucket = bucket,
object = object,
error = ?err,
"Failed to decode inline data during spill"
);
return Err(StorageError::other(format!("Failed to decode inline data for spill: {err}")));
}
};
// Generate data directory for the object
let data_dir = uuid::Uuid::new_v4();
// Create temporary directory for the spill operation
let tmp_root = format!("{}x{}", uuid::Uuid::new_v4(), time::OffsetDateTime::now_utc().unix_timestamp());
let tmp_path = format!("{tmp_root}/{}/part.1", data_dir);
// Encode and write the data to all disks
match self.write_segmented_data(&plain_data, &tmp_path, &erasure).await {
Ok(_) => {
// Move from temp to permanent location
let final_path = format!("{}/part.1", data_dir);
if let Err(err) = self.move_temp_to_final(&tmp_path, &final_path).await {
error!(
bucket = bucket,
object = object,
error = ?err,
"Failed to move spilled data to final location"
);
// Clean up temp files
let _ = self.cleanup_temp_files(&tmp_path).await;
return Err(err);
}
// Update file metadata
fi.data_dir = Some(data_dir);
fi.data = None; // Remove inline data
fi.metadata.remove(&format!("{}inline-data", RESERVED_METADATA_PREFIX_LOWER));
// Update append state to SegmentedActive
let mut new_state = current_state;
new_state.state = AppendStateKind::SegmentedActive;
new_state.epoch = new_state.epoch.saturating_add(1);
new_state.pending_segments.clear();
fi.set_append_state(&new_state)
.map_err(|err| StorageError::other(format!("Failed to update append state after spill: {err}")))?;
// Update all parts metadata
for meta in parts_metadata.iter_mut() {
if !meta.is_valid() {
continue;
}
meta.data_dir = Some(data_dir);
meta.data = None;
meta.metadata = fi.metadata.clone();
meta.metadata
.remove(&format!("{}inline-data", RESERVED_METADATA_PREFIX_LOWER));
}
// Write updated metadata back to disks
// TODO: Implement metadata write-back logic
// This would typically involve writing the updated FileInfo to all disks
info!(
bucket = bucket,
object = object,
data_dir = ?data_dir,
new_epoch = new_state.epoch,
"Successfully spilled inline data to segmented format"
);
Ok(())
}
Err(err) => {
error!(
bucket = bucket,
object = object,
error = ?err,
"Failed to write segmented data during spill"
);
// Clean up temp files
let _ = self.cleanup_temp_files(&tmp_path).await;
Err(err)
}
}
}
async fn write_segmented_data(&self, data: &[u8], tmp_path: &str, _erasure: &Erasure) -> Result<(), Error> {
use tracing::debug;
// TODO: Implement proper erasure encoding and writing to disks
// This is a placeholder implementation
debug!(
data_len = data.len(),
path = tmp_path,
"Writing segmented data (placeholder implementation)"
);
// For now, just return success - full implementation would:
// 1. Create bitrot writers for each disk
// 2. Erasure encode the data
// 3. Write each shard to its corresponding disk
Ok(())
}
async fn move_temp_to_final(&self, tmp_path: &str, final_path: &str) -> Result<(), Error> {
use tracing::debug;
// TODO: Implement moving temp files to final location
debug!(
tmp_path = tmp_path,
final_path = final_path,
"Moving temp files to final location (placeholder)"
);
Ok(())
}
async fn cleanup_temp_files(&self, tmp_path: &str) -> Result<(), Error> {
use tracing::debug;
// TODO: Implement temp file cleanup
debug!(tmp_path = tmp_path, "Cleaning up temp files (placeholder)");
Ok(())
}
}
/// Trigger background spill processing for an object
pub fn trigger_spill_process(
bucket: String,
object: String,
fi: rustfs_filemeta::FileInfo,
parts_metadata: Vec<rustfs_filemeta::FileInfo>,
epoch: u64,
disks: Vec<Option<crate::disk::DiskStore>>,
write_quorum: usize,
) {
use tracing::error;
tokio::spawn(async move {
let processor = InlineSpillProcessor::new(disks, write_quorum);
if let Err(err) = processor.process_spill(&bucket, &object, fi, parts_metadata, epoch).await {
error!(
bucket = bucket,
object = object,
epoch = epoch,
error = ?err,
"Background spill process failed"
);
}
});
}
#[cfg(test)]
mod tests {
use super::*;
use rustfs_utils::HashAlgorithm;
fn make_object_info() -> ObjectInfo {
ObjectInfo {
bucket: "test-bucket".to_string(),
name: "obj".to_string(),
..Default::default()
}
}
#[test]
fn rejects_compressed_objects() {
let mut info = make_object_info();
info.user_defined
.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}compression"), "zstd".to_string());
let err = validate_append_preconditions("test-bucket", "obj", &info).unwrap_err();
matches!(err, StorageError::InvalidArgument(..))
.then_some(())
.expect("expected invalid argument");
}
#[test]
fn rejects_encrypted_objects() {
let mut info = make_object_info();
info.user_defined
.insert("x-amz-server-side-encryption".to_string(), "AES256".to_string());
let err = validate_append_preconditions("test-bucket", "obj", &info).unwrap_err();
matches!(err, StorageError::InvalidArgument(..))
.then_some(())
.expect("expected invalid argument");
}
#[test]
fn rejects_transitioned_objects() {
let mut info = make_object_info();
info.transitioned_object.tier = "GLACIER".to_string();
info.transitioned_object.status = TRANSITION_COMPLETE.to_string();
let err = validate_append_preconditions("test-bucket", "obj", &info).unwrap_err();
matches!(err, StorageError::InvalidArgument(..))
.then_some(())
.expect("expected invalid argument");
}
#[test]
fn accepts_plain_objects() {
let info = make_object_info();
validate_append_preconditions("test-bucket", "obj", &info).expect("append should be allowed");
}
#[test]
fn rejects_position_mismatch() {
let mut info = make_object_info();
info.size = 10;
let err = validate_append_position("test-bucket", "obj", &info, 5).unwrap_err();
matches!(err, StorageError::InvalidArgument(..))
.then_some(())
.expect("expected invalid argument");
}
fn make_inline_erasure() -> Erasure {
Erasure::new(1, 0, 1024)
}
async fn encode_inline(data: &[u8], erasure: &Erasure) -> Vec<u8> {
let mut writer = create_bitrot_writer(
true,
None,
"",
"",
erasure.shard_file_size(data.len() as i64),
erasure.shard_size(),
HashAlgorithm::HighwayHash256,
)
.await
.unwrap();
let mut remaining = data;
while !remaining.is_empty() {
let chunk_len = remaining.len().min(erasure.block_size);
writer.write(&remaining[..chunk_len]).await.unwrap();
remaining = &remaining[chunk_len..];
}
writer.shutdown().await.unwrap();
writer.into_inline_data().unwrap()
}
async fn decode_inline(encoded: &[u8], size: usize, erasure: &Erasure) -> Vec<u8> {
let mut reader =
create_bitrot_reader(Some(encoded), None, "", "", 0, size, erasure.shard_size(), HashAlgorithm::HighwayHash256)
.await
.unwrap()
.unwrap();
let mut out = Vec::with_capacity(size);
while out.len() < size {
let remaining = size - out.len();
let mut buf = vec![0u8; erasure.block_size.min(remaining.max(1))];
let read = reader.read(&mut buf).await.unwrap();
if read == 0 {
break;
}
out.extend_from_slice(&buf[..read.min(remaining)]);
}
out
}
#[tokio::test]
async fn append_inline_combines_payloads() {
let erasure = make_inline_erasure();
let existing_plain = b"hello";
let encoded = encode_inline(existing_plain, &erasure).await;
let ctx = InlineAppendContext {
existing_inline: Some(&encoded),
existing_plain: None,
existing_size: existing_plain.len() as i64,
append_payload: b" world",
erasure: &erasure,
hash_algorithm: HashAlgorithm::HighwayHash256,
has_checksums: true,
};
let result = append_inline_data(ctx).await.expect("inline append to succeed");
assert_eq!(result.total_size, 11);
assert_eq!(result.etag, md5_hex(b"hello world"));
let decoded = decode_inline(&result.inline_data, result.total_size as usize, &erasure).await;
assert_eq!(decoded, b"hello world");
}
#[tokio::test]
async fn decode_inline_handles_padded_shards() {
let erasure = Erasure::new(1, 0, 1024);
let plain = b"hello";
let mut padded = vec![0u8; calc_shard_size(plain.len(), erasure.data_shards)];
padded[..plain.len()].copy_from_slice(plain);
let mut writer = create_bitrot_writer(
true,
None,
"",
"",
erasure.shard_file_size(plain.len() as i64),
erasure.shard_size(),
HashAlgorithm::HighwayHash256,
)
.await
.unwrap();
writer.write(&padded).await.unwrap();
writer.shutdown().await.unwrap();
let inline = writer.into_inline_data().unwrap();
let (decoded, algo) = decode_inline_payload(&inline, plain.len(), &erasure, HashAlgorithm::HighwayHash256)
.await
.expect("inline decode should succeed");
assert_eq!(decoded, plain);
assert_eq!(algo, HashAlgorithm::HighwayHash256);
}
#[tokio::test]
async fn append_inline_handles_empty_original() {
let erasure = make_inline_erasure();
let ctx = InlineAppendContext {
existing_inline: None,
existing_plain: None,
existing_size: 0,
append_payload: b"data",
erasure: &erasure,
hash_algorithm: HashAlgorithm::HighwayHash256,
has_checksums: true,
};
let result = append_inline_data(ctx).await.expect("inline append to succeed");
assert_eq!(result.total_size, 4);
assert_eq!(result.etag, md5_hex(b"data"));
let decoded = decode_inline(&result.inline_data, result.total_size as usize, &erasure).await;
assert_eq!(decoded, b"data");
}
#[tokio::test]
async fn append_inline_without_checksums_uses_raw_bytes() {
let erasure = Erasure::new(1, 0, 1024);
let existing = b"hello";
let ctx = InlineAppendContext {
existing_inline: Some(existing),
existing_plain: None,
existing_size: existing.len() as i64,
append_payload: b" world",
erasure: &erasure,
hash_algorithm: HashAlgorithm::HighwayHash256,
has_checksums: false,
};
let result = append_inline_data(ctx).await.expect("inline append to succeed");
assert_eq!(result.total_size, 11);
assert_eq!(result.etag, md5_hex(b"hello world"));
assert_eq!(result.inline_data, b"hello world");
}
#[tokio::test]
async fn append_inline_decodes_bitrot_without_checksums() {
let erasure = Erasure::new(1, 0, 1024);
let existing_plain = b"hello";
let encoded = encode_inline(existing_plain, &erasure).await;
let ctx = InlineAppendContext {
existing_inline: Some(&encoded),
existing_plain: None,
existing_size: existing_plain.len() as i64,
append_payload: b" world",
erasure: &erasure,
hash_algorithm: HashAlgorithm::HighwayHash256,
has_checksums: false,
};
let result = append_inline_data(ctx).await.expect("inline append to succeed");
assert_eq!(result.total_size, 11);
assert_eq!(result.etag, md5_hex(b"hello world"));
let decoded = decode_inline(&result.inline_data, result.total_size as usize, &erasure).await;
assert_eq!(decoded, b"hello world");
}
}

View File

@@ -1140,6 +1140,7 @@ impl ECStore {
.await
{
if !is_err_bucket_exists(&err) {
error!("decommission: make bucket failed: {err}");
return Err(err);
}
}
@@ -1262,6 +1263,8 @@ impl ECStore {
parts[i] = CompletePart {
part_num: pi.part_num,
etag: pi.etag,
..Default::default()
};
}
@@ -1289,7 +1292,7 @@ impl ECStore {
}
let reader = BufReader::new(rd.stream);
let hrd = HashReader::new(Box::new(WarpReader::new(reader)), object_info.size, object_info.size, None, false)?;
let hrd = HashReader::new(Box::new(WarpReader::new(reader)), object_info.size, object_info.size, None, None, false)?;
let mut data = PutObjReader::new(hrd);
if let Err(err) = self

View File

@@ -979,6 +979,7 @@ impl ECStore {
parts[i] = CompletePart {
part_num: pi.part_num,
etag: pi.etag,
..Default::default()
};
}
@@ -1005,7 +1006,7 @@ impl ECStore {
}
let reader = BufReader::new(rd.stream);
let hrd = HashReader::new(Box::new(WarpReader::new(reader)), object_info.size, object_info.size, None, false)?;
let hrd = HashReader::new(Box::new(WarpReader::new(reader)), object_info.size, object_info.size, None, None, false)?;
let mut data = PutObjReader::new(hrd);
if let Err(err) = self

File diff suppressed because it is too large Load Diff

View File

@@ -602,14 +602,6 @@ impl StorageAPI for Sets {
(del_objects, del_errs)
}
async fn complete_append(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<ObjectInfo> {
self.get_disks_by_key(object).complete_append(bucket, object, opts).await
}
async fn abort_append(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<ObjectInfo> {
self.get_disks_by_key(object).abort_append(bucket, object, opts).await
}
async fn list_object_parts(
&self,
bucket: &str,

View File

@@ -59,7 +59,6 @@ use rustfs_common::globals::{GLOBAL_Local_Node_Name, GLOBAL_Rustfs_Host, GLOBAL_
use rustfs_common::heal_channel::{HealItemType, HealOpts};
use rustfs_filemeta::FileInfo;
use rustfs_madmin::heal_commands::HealResultItem;
use rustfs_utils::crypto::base64_decode;
use rustfs_utils::path::{SLASH_SEPARATOR, decode_dir_object, encode_dir_object, path_join_buf};
use s3s::dto::{BucketVersioningStatus, ObjectLockConfiguration, ObjectLockEnabled, VersioningConfiguration};
use std::cmp::Ordering;
@@ -1231,6 +1230,7 @@ impl StorageAPI for ECStore {
if let Err(err) = self.peer_sys.make_bucket(bucket, opts).await {
let err = to_object_err(err.into(), vec![bucket]);
if !is_err_bucket_exists(&err) {
error!("make bucket failed: {err}");
let _ = self
.delete_bucket(
bucket,
@@ -1709,17 +1709,6 @@ impl StorageAPI for ECStore {
// Ok((del_objects, del_errs))
}
async fn complete_append(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<ObjectInfo> {
let object = encode_dir_object(object);
let (pinfo, _) = self.internal_get_pool_info_existing_with_opts(bucket, &object, opts).await?;
self.pools[pinfo.index].complete_append(bucket, &object, opts).await
}
async fn abort_append(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<ObjectInfo> {
let object = encode_dir_object(object);
let (pinfo, _) = self.internal_get_pool_info_existing_with_opts(bucket, &object, opts).await?;
self.pools[pinfo.index].abort_append(bucket, &object, opts).await
}
#[tracing::instrument(skip(self))]
async fn list_object_parts(
&self,
@@ -2432,7 +2421,7 @@ fn check_list_multipart_args(
}
}
if let Err(_e) = base64_decode(upload_id_marker.as_bytes()) {
if let Err(_e) = base64_simd::URL_SAFE_NO_PAD.decode_to_vec(upload_id_marker.as_bytes()) {
return Err(StorageError::MalformedUploadID(upload_id_marker.to_owned()));
}
}
@@ -2459,7 +2448,7 @@ fn check_new_multipart_args(bucket: &str, object: &str) -> Result<()> {
}
fn check_multipart_object_args(bucket: &str, object: &str, upload_id: &str) -> Result<()> {
if let Err(e) = base64_decode(upload_id.as_bytes()) {
if let Err(e) = base64_simd::URL_SAFE_NO_PAD.decode_to_vec(upload_id.as_bytes()) {
return Err(StorageError::MalformedUploadID(format!("{bucket}/{object}-{upload_id},err:{e}")));
};
check_object_args(bucket, object)

View File

@@ -13,9 +13,6 @@
// limitations under the License.
use crate::bucket::metadata_sys::get_versioning_config;
use crate::bucket::replication::REPLICATION_RESET;
use crate::bucket::replication::REPLICATION_STATUS;
use crate::bucket::replication::{ReplicateDecision, replication_statuses_map, version_purge_statuses_map};
use crate::bucket::versioning::VersioningApi as _;
use crate::disk::DiskStore;
use crate::error::{Error, Result};
@@ -25,12 +22,15 @@ use crate::{
bucket::lifecycle::lifecycle::ExpirationOptions,
bucket::lifecycle::{bucket_lifecycle_ops::TransitionedObject, lifecycle::TransitionOptions},
};
use bytes::Bytes;
use http::{HeaderMap, HeaderValue};
use rustfs_common::heal_channel::HealOpts;
use rustfs_filemeta::{
FileInfo, MetaCacheEntriesSorted, ObjectPartInfo, ReplicationState, ReplicationStatusType, VersionPurgeStatusType,
FileInfo, MetaCacheEntriesSorted, ObjectPartInfo, REPLICATION_RESET, REPLICATION_STATUS, ReplicateDecision, ReplicationState,
ReplicationStatusType, VersionPurgeStatusType, replication_statuses_map, version_purge_statuses_map,
};
use rustfs_madmin::heal_commands::HealResultItem;
use rustfs_rio::Checksum;
use rustfs_rio::{DecompressReader, HashReader, LimitReader, WarpReader};
use rustfs_utils::CompressionAlgorithm;
use rustfs_utils::http::headers::{AMZ_OBJECT_TAGGING, RESERVED_METADATA_PREFIX_LOWER};
@@ -92,11 +92,28 @@ impl PutObjReader {
PutObjReader { stream }
}
pub fn as_hash_reader(&self) -> &HashReader {
&self.stream
}
pub fn from_vec(data: Vec<u8>) -> Self {
use sha2::{Digest, Sha256};
let content_length = data.len() as i64;
let sha256hex = if content_length > 0 {
Some(hex_simd::encode_to_string(Sha256::digest(&data), hex_simd::AsciiCase::Lower))
} else {
None
};
PutObjReader {
stream: HashReader::new(Box::new(WarpReader::new(Cursor::new(data))), content_length, content_length, None, false)
.unwrap(),
stream: HashReader::new(
Box::new(WarpReader::new(Cursor::new(data))),
content_length,
content_length,
None,
sha256hex,
false,
)
.unwrap(),
}
}
@@ -247,11 +264,16 @@ impl HTTPRangeSpec {
return None;
}
let mut start = 0i64;
let mut end = -1i64;
for i in 0..oi.parts.len().min(part_number) {
if part_number == 0 || part_number > oi.parts.len() {
return None;
}
let mut start = 0_i64;
let mut end = -1_i64;
for i in 0..part_number {
let part = &oi.parts[i];
start = end + 1;
end = start + (oi.parts[i].size as i64) - 1
end = start + (part.size as i64) - 1;
}
Some(HTTPRangeSpec {
@@ -266,8 +288,14 @@ impl HTTPRangeSpec {
let mut start = self.start;
if self.is_suffix_length {
start = res_size + self.start;
let suffix_len = if self.start < 0 {
self.start
.checked_neg()
.ok_or_else(|| Error::other("range value invalid: suffix length overflow"))?
} else {
self.start
};
start = res_size - suffix_len;
if start < 0 {
start = 0;
}
@@ -280,7 +308,13 @@ impl HTTPRangeSpec {
}
if self.is_suffix_length {
let specified_len = self.start; // 假设 h.start 是一个 i64 类型
let specified_len = if self.start < 0 {
self.start
.checked_neg()
.ok_or_else(|| Error::other("range value invalid: suffix length overflow"))?
} else {
self.start
};
let mut range_length = specified_len;
if specified_len > res_size {
@@ -328,8 +362,6 @@ pub struct ObjectOptions {
pub max_parity: bool,
pub mod_time: Option<OffsetDateTime>,
pub part_number: Option<usize>,
pub append_object: bool,
pub append_position: Option<i64>,
pub delete_prefix: bool,
pub delete_prefix_object: bool,
@@ -359,6 +391,8 @@ pub struct ObjectOptions {
pub lifecycle_audit_event: LcAuditEvent,
pub eval_metadata: Option<HashMap<String, String>>,
pub want_checksum: Option<Checksum>,
}
impl ObjectOptions {
@@ -441,6 +475,8 @@ pub struct BucketInfo {
#[derive(Debug, Default, Clone)]
pub struct MultipartUploadResult {
pub upload_id: String,
pub checksum_algo: Option<String>,
pub checksum_type: Option<String>,
}
#[derive(Debug, Default, Clone)]
@@ -456,13 +492,24 @@ pub struct PartInfo {
pub struct CompletePart {
pub part_num: usize,
pub etag: Option<String>,
// pub size: Option<usize>,
pub checksum_crc32: Option<String>,
pub checksum_crc32c: Option<String>,
pub checksum_sha1: Option<String>,
pub checksum_sha256: Option<String>,
pub checksum_crc64nvme: Option<String>,
}
impl From<s3s::dto::CompletedPart> for CompletePart {
fn from(value: s3s::dto::CompletedPart) -> Self {
Self {
part_num: value.part_number.unwrap_or_default() as usize,
etag: value.e_tag,
etag: value.e_tag.map(|v| v.value().to_owned()),
checksum_crc32: value.checksum_crc32,
checksum_crc32c: value.checksum_crc32c,
checksum_sha1: value.checksum_sha1,
checksum_sha256: value.checksum_sha256,
checksum_crc64nvme: value.checksum_crc64nvme,
}
}
}
@@ -502,7 +549,7 @@ pub struct ObjectInfo {
pub version_purge_status_internal: Option<String>,
pub version_purge_status: VersionPurgeStatusType,
pub replication_decision: String,
pub checksum: Vec<u8>,
pub checksum: Option<Bytes>,
}
impl Clone for ObjectInfo {
@@ -539,7 +586,7 @@ impl Clone for ObjectInfo {
version_purge_status_internal: self.version_purge_status_internal.clone(),
version_purge_status: self.version_purge_status.clone(),
replication_decision: self.replication_decision.clone(),
checksum: Default::default(),
checksum: self.checksum.clone(),
expires: self.expires,
}
}
@@ -658,15 +705,6 @@ impl ObjectInfo {
})
.collect();
let append_state = fi.get_append_state();
let pending_length: i64 = append_state.pending_segments.iter().map(|seg| seg.length).sum();
let logical_size = append_state.committed_length.saturating_add(pending_length);
let actual_size_meta = fi
.metadata
.get(&format!("{RESERVED_METADATA_PREFIX_LOWER}actual-size"))
.and_then(|o| o.parse::<i64>().ok())
.unwrap_or(logical_size);
ObjectInfo {
bucket: bucket.to_string(),
name,
@@ -676,7 +714,7 @@ impl ObjectInfo {
version_id,
delete_marker: fi.deleted,
mod_time: fi.mod_time,
size: logical_size,
size: fi.size,
parts,
is_latest: fi.is_latest,
user_tags,
@@ -688,7 +726,7 @@ impl ObjectInfo {
inlined,
user_defined: metadata,
transitioned_object,
actual_size: actual_size_meta,
checksum: fi.checksum.clone(),
..Default::default()
}
}
@@ -879,6 +917,23 @@ impl ObjectInfo {
..Default::default()
}
}
pub fn decrypt_checksums(&self, part: usize, _headers: &HeaderMap) -> Result<(HashMap<String, String>, bool)> {
if part > 0 {
if let Some(checksums) = self.parts.iter().find(|p| p.number == part).and_then(|p| p.checksums.clone()) {
return Ok((checksums, true));
}
}
// TODO: decrypt checksums
if let Some(data) = &self.checksum {
let (checksums, is_multipart) = rustfs_rio::read_checksums(data.as_ref(), 0);
return Ok((checksums, is_multipart));
}
Ok((HashMap::new(), false))
}
}
#[derive(Debug, Default)]
@@ -1200,10 +1255,6 @@ pub trait StorageAPI: ObjectIO + Debug {
opts: ObjectOptions,
) -> (Vec<DeletedObject>, Vec<Option<Error>>);
async fn complete_append(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<ObjectInfo>;
async fn abort_append(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<ObjectInfo>;
// TransitionObject TODO:
// RestoreTransitionedObject TODO:
@@ -1577,6 +1628,83 @@ mod tests {
assert_eq!(length, 10); // end - start + 1 = 14 - 5 + 1 = 10
}
#[test]
fn test_http_range_spec_suffix_positive_start() {
let range_spec = HTTPRangeSpec {
is_suffix_length: true,
start: 5,
end: -1,
};
let (offset, length) = range_spec.get_offset_length(20).unwrap();
assert_eq!(offset, 15);
assert_eq!(length, 5);
}
#[test]
fn test_http_range_spec_suffix_negative_start() {
let range_spec = HTTPRangeSpec {
is_suffix_length: true,
start: -5,
end: -1,
};
let (offset, length) = range_spec.get_offset_length(20).unwrap();
assert_eq!(offset, 15);
assert_eq!(length, 5);
}
#[test]
fn test_http_range_spec_suffix_exceeds_object() {
let range_spec = HTTPRangeSpec {
is_suffix_length: true,
start: 50,
end: -1,
};
let (offset, length) = range_spec.get_offset_length(20).unwrap();
assert_eq!(offset, 0);
assert_eq!(length, 20);
}
#[test]
fn test_http_range_spec_from_object_info_valid_and_invalid_parts() {
let object_info = ObjectInfo {
size: 300,
parts: vec![
ObjectPartInfo {
etag: String::new(),
number: 1,
size: 100,
actual_size: 100,
..Default::default()
},
ObjectPartInfo {
etag: String::new(),
number: 2,
size: 100,
actual_size: 100,
..Default::default()
},
ObjectPartInfo {
etag: String::new(),
number: 3,
size: 100,
actual_size: 100,
..Default::default()
},
],
..Default::default()
};
let spec = HTTPRangeSpec::from_object_info(&object_info, 2).unwrap();
assert_eq!(spec.start, 100);
assert_eq!(spec.end, 199);
assert!(HTTPRangeSpec::from_object_info(&object_info, 0).is_none());
assert!(HTTPRangeSpec::from_object_info(&object_info, 4).is_none());
}
#[tokio::test]
async fn test_ranged_decompress_reader_zero_length() {
let original_data = b"Hello, World!";

View File

@@ -30,7 +30,6 @@ crc32fast = { workspace = true }
rmp.workspace = true
rmp-serde.workspace = true
serde.workspace = true
serde_json.workspace = true
time.workspace = true
uuid = { workspace = true, features = ["v4", "fast-rng", "serde"] }
tokio = { workspace = true, features = ["io-util", "macros", "sync"] }
@@ -41,6 +40,8 @@ byteorder = { workspace = true }
tracing.workspace = true
thiserror.workspace = true
s3s.workspace = true
lazy_static.workspace = true
regex.workspace = true
[dev-dependencies]
criterion = { workspace = true }

View File

@@ -1,541 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::error::{Error, Result};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use uuid::Uuid;
const APPEND_STATE_META_KEY: &str = "x-rustfs-internal-append-state";
/// Tracks the state of append-enabled objects.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct AppendState {
pub state: AppendStateKind,
pub epoch: u64,
pub committed_length: i64,
pub pending_segments: Vec<AppendSegment>,
}
/// Represents individual append segments that still need consolidation.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub struct AppendSegment {
pub offset: i64,
pub length: i64,
pub data_dir: Option<Uuid>,
pub etag: Option<String>,
pub epoch: u64,
}
/// Possible append lifecycle states for an object version.
#[derive(Clone, Debug, Default, PartialEq, Serialize, Deserialize)]
pub enum AppendStateKind {
#[default]
Disabled,
Inline,
InlinePendingSpill,
SegmentedActive,
SegmentedSealed,
}
/// Persist the provided append state into object metadata.
pub fn set_append_state(metadata: &mut HashMap<String, String>, state: &AppendState) -> Result<()> {
let encoded = serde_json::to_string(state).map_err(Error::other)?;
metadata.insert(APPEND_STATE_META_KEY.to_string(), encoded);
Ok(())
}
/// Remove the append state marker from metadata.
pub fn clear_append_state(metadata: &mut HashMap<String, String>) {
metadata.remove(APPEND_STATE_META_KEY);
}
/// Load append state stored in metadata, if any.
pub fn get_append_state(metadata: &HashMap<String, String>) -> Result<Option<AppendState>> {
let raw = match metadata.get(APPEND_STATE_META_KEY) {
Some(val) if !val.is_empty() => val,
_ => return Ok(None),
};
let decoded = serde_json::from_str(raw).map_err(Error::other)?;
Ok(Some(decoded))
}
/// Complete append operations by consolidating pending segments and sealing the object
pub fn complete_append_operation(state: &mut AppendState) -> Result<()> {
match state.state {
AppendStateKind::SegmentedActive => {
// Move all pending segments data to main parts and seal
state.committed_length += state.pending_segments.iter().map(|s| s.length).sum::<i64>();
state.pending_segments.clear();
state.state = AppendStateKind::SegmentedSealed;
state.epoch = state.epoch.saturating_add(1);
Ok(())
}
AppendStateKind::Inline => {
// Inline objects are always immediately committed, just seal them
state.state = AppendStateKind::SegmentedSealed; // Transition to sealed
state.epoch = state.epoch.saturating_add(1);
Ok(())
}
AppendStateKind::InlinePendingSpill => {
// Wait for spill to complete, then seal
// In practice, this might need to trigger the spill completion first
state.state = AppendStateKind::SegmentedSealed;
state.pending_segments.clear();
state.epoch = state.epoch.saturating_add(1);
Ok(())
}
AppendStateKind::SegmentedSealed | AppendStateKind::Disabled => {
// Already sealed or disabled
Err(Error::other("Cannot complete append on sealed or disabled object"))
}
}
}
/// Abort append operations by discarding pending segments and returning to sealed state
pub fn abort_append_operation(state: &mut AppendState) -> Result<()> {
match state.state {
AppendStateKind::SegmentedActive => {
// Discard all pending segments and seal
state.pending_segments.clear();
state.state = AppendStateKind::SegmentedSealed;
state.epoch = state.epoch.saturating_add(1);
Ok(())
}
AppendStateKind::Inline => {
// Inline data is already committed, just seal
state.state = AppendStateKind::SegmentedSealed;
state.epoch = state.epoch.saturating_add(1);
Ok(())
}
AppendStateKind::InlinePendingSpill => {
// Cancel spill and keep inline data, then seal
state.state = AppendStateKind::SegmentedSealed;
state.pending_segments.clear();
state.epoch = state.epoch.saturating_add(1);
Ok(())
}
AppendStateKind::SegmentedSealed | AppendStateKind::Disabled => {
// Already sealed or disabled
Err(Error::other("Cannot abort append on sealed or disabled object"))
}
}
}
/// Check if an append operation can be completed
pub fn can_complete_append(state: &AppendState) -> bool {
matches!(
state.state,
AppendStateKind::Inline | AppendStateKind::InlinePendingSpill | AppendStateKind::SegmentedActive
)
}
/// Check if an append operation can be aborted
pub fn can_abort_append(state: &AppendState) -> bool {
matches!(
state.state,
AppendStateKind::Inline | AppendStateKind::InlinePendingSpill | AppendStateKind::SegmentedActive
)
}
/// Verify epoch for optimistic concurrency control
pub fn verify_append_epoch(current_state: &AppendState, expected_epoch: u64) -> Result<()> {
if current_state.epoch != expected_epoch {
Err(Error::other(format!(
"Append operation conflict: expected epoch {}, found {}",
expected_epoch, current_state.epoch
)))
} else {
Ok(())
}
}
/// Prepare next append operation by incrementing epoch
pub fn prepare_next_append(state: &mut AppendState) {
state.epoch = state.epoch.saturating_add(1);
}
/// Validate that a new append segment doesn't conflict with existing segments
pub fn validate_new_segment(state: &AppendState, new_offset: i64, new_length: i64) -> Result<()> {
let new_end = new_offset + new_length;
// Check it doesn't overlap with committed data
if new_offset < state.committed_length {
return Err(Error::other(format!(
"New segment overlaps with committed data: offset {} < committed_length {}",
new_offset, state.committed_length
)));
}
// Check it doesn't overlap with existing pending segments
for existing in &state.pending_segments {
let existing_start = existing.offset;
let existing_end = existing.offset + existing.length;
// Check for any overlap
if new_offset < existing_end && new_end > existing_start {
return Err(Error::other(format!(
"New segment [{}, {}) overlaps with existing segment [{}, {})",
new_offset, new_end, existing_start, existing_end
)));
}
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use crate::fileinfo::FileInfo;
#[test]
fn append_state_roundtrip_in_metadata() {
let mut metadata = HashMap::new();
let state = AppendState {
state: AppendStateKind::SegmentedActive,
epoch: 42,
committed_length: 2048,
pending_segments: vec![AppendSegment {
offset: 2048,
length: 512,
data_dir: Some(Uuid::new_v4()),
etag: Some("abc123".to_string()),
epoch: 0,
}],
};
set_append_state(&mut metadata, &state).expect("persist append state");
assert!(metadata.contains_key(APPEND_STATE_META_KEY));
let decoded = get_append_state(&metadata)
.expect("decode append state")
.expect("state present");
assert_eq!(decoded, state);
clear_append_state(&mut metadata);
assert!(!metadata.contains_key(APPEND_STATE_META_KEY));
assert!(get_append_state(&metadata).unwrap().is_none());
}
#[test]
fn fileinfo_append_state_migration_compatibility() {
// Test old inline data object
let mut inline_fi = FileInfo {
size: 1024,
..Default::default()
};
inline_fi.set_inline_data();
let state = inline_fi.get_append_state();
assert_eq!(state.state, AppendStateKind::Inline);
assert_eq!(state.committed_length, 1024);
assert!(state.pending_segments.is_empty());
assert!(inline_fi.is_appendable());
assert!(!inline_fi.has_pending_appends());
// Test old regular object
let regular_fi = FileInfo {
size: 2048,
..Default::default()
};
// No inline_data marker
let state = regular_fi.get_append_state();
assert_eq!(state.state, AppendStateKind::SegmentedSealed);
assert_eq!(state.committed_length, 2048);
assert!(state.pending_segments.is_empty());
assert!(!regular_fi.is_appendable());
assert!(!regular_fi.has_pending_appends());
// Test explicit append state
let mut append_fi = FileInfo::default();
let explicit_state = AppendState {
state: AppendStateKind::SegmentedActive,
epoch: 5,
committed_length: 1500,
pending_segments: vec![AppendSegment {
offset: 1500,
length: 300,
data_dir: Some(Uuid::new_v4()),
etag: Some("def456".to_string()),
epoch: 0,
}],
};
append_fi.set_append_state(&explicit_state).expect("set explicit state");
let retrieved_state = append_fi.get_append_state();
assert_eq!(retrieved_state, explicit_state);
assert!(append_fi.is_appendable());
assert!(append_fi.has_pending_appends());
}
#[test]
fn append_state_transitions() {
// Test state transition validation
assert_eq!(AppendStateKind::default(), AppendStateKind::Disabled);
let inline_state = AppendState {
state: AppendStateKind::Inline,
..Default::default()
};
let spill_state = AppendState {
state: AppendStateKind::InlinePendingSpill,
..Default::default()
};
let active_state = AppendState {
state: AppendStateKind::SegmentedActive,
..Default::default()
};
let sealed_state = AppendState {
state: AppendStateKind::SegmentedSealed,
..Default::default()
};
// Verify serialization works for all states
for state in [inline_state, spill_state, active_state, sealed_state] {
let mut metadata = HashMap::new();
set_append_state(&mut metadata, &state).expect("serialize state");
let decoded = get_append_state(&metadata).unwrap().unwrap();
assert_eq!(decoded, state);
}
}
#[test]
fn complete_append_transitions() {
// Test completing SegmentedActive with pending segments
let mut active_state = AppendState {
state: AppendStateKind::SegmentedActive,
epoch: 5,
committed_length: 1000,
pending_segments: vec![
AppendSegment {
offset: 1000,
length: 200,
data_dir: Some(Uuid::new_v4()),
etag: Some("abc123".to_string()),
epoch: 0,
},
AppendSegment {
offset: 1200,
length: 300,
data_dir: Some(Uuid::new_v4()),
etag: Some("def456".to_string()),
epoch: 0,
},
],
};
assert!(can_complete_append(&active_state));
complete_append_operation(&mut active_state).expect("complete should succeed");
assert_eq!(active_state.state, AppendStateKind::SegmentedSealed);
assert_eq!(active_state.committed_length, 1500); // 1000 + 200 + 300
assert!(active_state.pending_segments.is_empty());
assert_eq!(active_state.epoch, 6);
// Test completing Inline state
let mut inline_state = AppendState {
state: AppendStateKind::Inline,
epoch: 2,
committed_length: 500,
..Default::default()
};
assert!(can_complete_append(&inline_state));
complete_append_operation(&mut inline_state).expect("complete should succeed");
assert_eq!(inline_state.state, AppendStateKind::SegmentedSealed);
assert_eq!(inline_state.committed_length, 500); // Unchanged
assert_eq!(inline_state.epoch, 3);
// Test completing already sealed state should fail
let mut sealed_state = AppendState {
state: AppendStateKind::SegmentedSealed,
..Default::default()
};
assert!(!can_complete_append(&sealed_state));
assert!(complete_append_operation(&mut sealed_state).is_err());
}
#[test]
fn abort_append_transitions() {
// Test aborting SegmentedActive with pending segments
let mut active_state = AppendState {
state: AppendStateKind::SegmentedActive,
epoch: 3,
committed_length: 800,
pending_segments: vec![AppendSegment {
offset: 800,
length: 400,
data_dir: Some(Uuid::new_v4()),
etag: Some("xyz789".to_string()),
epoch: 0,
}],
};
assert!(can_abort_append(&active_state));
abort_append_operation(&mut active_state).expect("abort should succeed");
assert_eq!(active_state.state, AppendStateKind::SegmentedSealed);
assert_eq!(active_state.committed_length, 800); // Unchanged, pending discarded
assert!(active_state.pending_segments.is_empty());
assert_eq!(active_state.epoch, 4);
// Test aborting InlinePendingSpill
let mut spill_state = AppendState {
state: AppendStateKind::InlinePendingSpill,
epoch: 1,
committed_length: 100,
pending_segments: vec![],
};
assert!(can_abort_append(&spill_state));
abort_append_operation(&mut spill_state).expect("abort should succeed");
assert_eq!(spill_state.state, AppendStateKind::SegmentedSealed);
assert_eq!(spill_state.committed_length, 100);
assert_eq!(spill_state.epoch, 2);
// Test aborting disabled state should fail
let mut disabled_state = AppendState {
state: AppendStateKind::Disabled,
..Default::default()
};
assert!(!can_abort_append(&disabled_state));
assert!(abort_append_operation(&mut disabled_state).is_err());
}
#[test]
fn epoch_validation() {
let state = AppendState {
state: AppendStateKind::SegmentedActive,
epoch: 10,
committed_length: 1000,
pending_segments: vec![],
};
// Valid epoch should succeed
assert!(verify_append_epoch(&state, 10).is_ok());
// Invalid epoch should fail
assert!(verify_append_epoch(&state, 9).is_err());
assert!(verify_append_epoch(&state, 11).is_err());
// Error message should contain epoch information
let error = verify_append_epoch(&state, 5).unwrap_err();
let error_msg = error.to_string();
assert!(error_msg.contains("expected epoch 5"));
assert!(error_msg.contains("found 10"));
}
#[test]
fn next_append_preparation() {
let mut state = AppendState {
state: AppendStateKind::SegmentedActive,
epoch: 5,
committed_length: 1000,
pending_segments: vec![],
};
prepare_next_append(&mut state);
assert_eq!(state.epoch, 6);
// Test saturation behavior
let mut max_state = AppendState {
epoch: u64::MAX,
..Default::default()
};
prepare_next_append(&mut max_state);
assert_eq!(max_state.epoch, u64::MAX); // Should saturate, not overflow
}
#[test]
fn segment_validation() {
let state = AppendState {
state: AppendStateKind::SegmentedActive,
epoch: 3,
committed_length: 1000,
pending_segments: vec![
AppendSegment {
offset: 1000,
length: 200,
data_dir: Some(Uuid::new_v4()),
etag: Some("abc123".to_string()),
epoch: 0,
},
AppendSegment {
offset: 1300,
length: 300,
data_dir: Some(Uuid::new_v4()),
etag: Some("def456".to_string()),
epoch: 0,
},
],
};
// Valid segment after existing segments
assert!(validate_new_segment(&state, 1600, 100).is_ok());
// Valid segment filling gap between committed and first pending
assert!(validate_new_segment(&state, 1200, 100).is_ok());
// Invalid segment overlapping with committed data
assert!(validate_new_segment(&state, 900, 200).is_err());
let error = validate_new_segment(&state, 900, 200).unwrap_err();
assert!(error.to_string().contains("overlaps with committed data"));
// Invalid segment overlapping with first pending segment
assert!(validate_new_segment(&state, 1100, 100).is_err());
let error = validate_new_segment(&state, 1100, 100).unwrap_err();
assert!(error.to_string().contains("overlaps with existing segment"));
// Invalid segment overlapping with second pending segment
assert!(validate_new_segment(&state, 1400, 100).is_err());
// Edge case: segment exactly touching committed data (should be valid)
assert!(validate_new_segment(&state, 1000, 0).is_ok());
// Edge case: segment exactly touching existing segment (should be valid)
assert!(validate_new_segment(&state, 1200, 0).is_ok());
}
#[test]
fn segment_validation_edge_cases() {
let empty_state = AppendState {
state: AppendStateKind::SegmentedActive,
epoch: 1,
committed_length: 500,
pending_segments: vec![],
};
// First segment after committed data
assert!(validate_new_segment(&empty_state, 500, 100).is_ok());
assert!(validate_new_segment(&empty_state, 600, 200).is_ok());
// Zero-length segments (edge case)
assert!(validate_new_segment(&empty_state, 500, 0).is_ok());
// Segment exactly at committed boundary
assert!(validate_new_segment(&empty_state, 499, 1).is_err());
assert!(validate_new_segment(&empty_state, 500, 1).is_ok());
}
}

View File

@@ -284,6 +284,7 @@ impl FileInfo {
Ok(t)
}
#[allow(clippy::too_many_arguments)]
pub fn add_object_part(
&mut self,
num: usize,
@@ -292,6 +293,7 @@ impl FileInfo {
mod_time: Option<OffsetDateTime>,
actual_size: i64,
index: Option<Bytes>,
checksums: Option<HashMap<String, String>>,
) {
let part = ObjectPartInfo {
etag,
@@ -300,7 +302,7 @@ impl FileInfo {
mod_time,
actual_size,
index,
checksums: None,
checksums,
error: None,
};
@@ -494,96 +496,6 @@ impl FileInfo {
ReplicationStatusType::Empty
}
}
/// Get the append state for this FileInfo, with migration compatibility
pub fn get_append_state(&self) -> crate::append::AppendState {
use crate::append::{AppendState, AppendStateKind, get_append_state};
// Try to load from metadata first
if let Ok(Some(state)) = get_append_state(&self.metadata) {
return state;
}
// Migration compatibility: determine state based on existing data
if self.inline_data() {
// Has inline data, treat as Inline state
AppendState {
state: AppendStateKind::Inline,
epoch: 0,
committed_length: self.size,
pending_segments: Vec::new(),
}
} else {
// No inline data, treat as SegmentedSealed (traditional object)
AppendState {
state: AppendStateKind::SegmentedSealed,
epoch: 0,
committed_length: self.size,
pending_segments: Vec::new(),
}
}
}
/// Set the append state for this FileInfo
pub fn set_append_state(&mut self, state: &crate::append::AppendState) -> crate::error::Result<()> {
crate::append::set_append_state(&mut self.metadata, state)
}
/// Check if this object supports append operations
pub fn is_appendable(&self) -> bool {
use crate::append::AppendStateKind;
match self.get_append_state().state {
AppendStateKind::Disabled => false,
AppendStateKind::Inline | AppendStateKind::InlinePendingSpill | AppendStateKind::SegmentedActive => true,
AppendStateKind::SegmentedSealed => false,
}
}
/// Check if this object has pending append operations
pub fn has_pending_appends(&self) -> bool {
use crate::append::AppendStateKind;
matches!(
self.get_append_state().state,
AppendStateKind::InlinePendingSpill | AppendStateKind::SegmentedActive
)
}
/// Complete all pending append operations and seal the object
pub fn complete_append(&mut self) -> crate::error::Result<()> {
let mut append_state = self.get_append_state();
crate::append::complete_append_operation(&mut append_state)?;
self.set_append_state(&append_state)?;
// Update file size to reflect completed operation
if append_state.state == crate::append::AppendStateKind::SegmentedSealed {
self.size = append_state.committed_length;
}
Ok(())
}
/// Abort all pending append operations and seal the object
pub fn abort_append(&mut self) -> crate::error::Result<()> {
let mut append_state = self.get_append_state();
crate::append::abort_append_operation(&mut append_state)?;
self.set_append_state(&append_state)?;
// Update file size to only include committed data
if append_state.state == crate::append::AppendStateKind::SegmentedSealed {
self.size = append_state.committed_length;
}
Ok(())
}
/// Check if append operations can be completed for this object
pub fn can_complete_append(&self) -> bool {
crate::append::can_complete_append(&self.get_append_state())
}
/// Check if append operations can be aborted for this object
pub fn can_abort_append(&self) -> bool {
crate::append::can_abort_append(&self.get_append_state())
}
}
#[derive(Debug, Default, Clone, Serialize, Deserialize)]

View File

@@ -15,9 +15,12 @@
use crate::error::{Error, Result};
use crate::fileinfo::{ErasureAlgo, ErasureInfo, FileInfo, FileInfoVersions, ObjectPartInfo, RawFileInfo};
use crate::filemeta_inline::InlineData;
use crate::{ReplicationStatusType, VersionPurgeStatusType};
use crate::{
ReplicationState, ReplicationStatusType, VersionPurgeStatusType, replication_statuses_map, version_purge_statuses_map,
};
use byteorder::ByteOrder;
use bytes::Bytes;
use rustfs_utils::http::AMZ_BUCKET_REPLICATION_STATUS;
use rustfs_utils::http::headers::{
self, AMZ_META_UNENCRYPTED_CONTENT_LENGTH, AMZ_META_UNENCRYPTED_CONTENT_MD5, AMZ_STORAGE_CLASS, RESERVED_METADATA_PREFIX,
RESERVED_METADATA_PREFIX_LOWER, VERSION_PURGE_STATUS_KEY,
@@ -30,6 +33,7 @@ use std::hash::Hasher;
use std::io::{Read, Write};
use std::{collections::HashMap, io::Cursor};
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
use tokio::io::AsyncRead;
use tracing::error;
use uuid::Uuid;
@@ -1742,7 +1746,25 @@ impl MetaObject {
}
}
// todo: ReplicationState,Delete
let replication_state_internal = get_internal_replication_state(&metadata);
let mut deleted = false;
if let Some(v) = replication_state_internal.as_ref() {
if !v.composite_version_purge_status().is_empty() {
deleted = true;
}
let st = v.composite_replication_status();
if !st.is_empty() {
metadata.insert(AMZ_BUCKET_REPLICATION_STATUS.to_string(), st.to_string());
}
}
let checksum = self
.meta_sys
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}crc").as_str())
.map(|v| Bytes::from(v.clone()));
let erasure = ErasureInfo {
algorithm: self.erasure_algorithm.to_string(),
@@ -1754,6 +1776,26 @@ impl MetaObject {
..Default::default()
};
let transition_status = self
.meta_sys
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITION_STATUS}").as_str())
.map(|v| String::from_utf8_lossy(v).to_string())
.unwrap_or_default();
let transitioned_objname = self
.meta_sys
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITIONED_OBJECTNAME}").as_str())
.map(|v| String::from_utf8_lossy(v).to_string())
.unwrap_or_default();
let transition_version_id = self
.meta_sys
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITIONED_VERSION_ID}").as_str())
.map(|v| Uuid::from_slice(v.as_slice()).unwrap_or_default());
let transition_tier = self
.meta_sys
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITION_TIER}").as_str())
.map(|v| String::from_utf8_lossy(v).to_string())
.unwrap_or_default();
FileInfo {
version_id,
erasure,
@@ -1764,6 +1806,13 @@ impl MetaObject {
volume: volume.to_string(),
parts,
metadata,
replication_state_internal,
deleted,
checksum,
transition_status,
transitioned_objname,
transition_version_id,
transition_tier,
..Default::default()
}
}
@@ -1904,6 +1953,38 @@ impl From<FileInfo> for MetaObject {
}
}
if !value.transition_status.is_empty() {
meta_sys.insert(
format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITION_STATUS}"),
value.transition_status.as_bytes().to_vec(),
);
}
if !value.transitioned_objname.is_empty() {
meta_sys.insert(
format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITIONED_OBJECTNAME}"),
value.transitioned_objname.as_bytes().to_vec(),
);
}
if let Some(vid) = &value.transition_version_id {
meta_sys.insert(
format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITIONED_VERSION_ID}"),
vid.as_bytes().to_vec(),
);
}
if !value.transition_tier.is_empty() {
meta_sys.insert(
format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITION_TIER}"),
value.transition_tier.as_bytes().to_vec(),
);
}
if let Some(content_hash) = value.checksum {
meta_sys.insert(format!("{RESERVED_METADATA_PREFIX_LOWER}crc"), content_hash.to_vec());
}
Self {
version_id: value.version_id,
data_dir: value.data_dir,
@@ -1927,6 +2008,50 @@ impl From<FileInfo> for MetaObject {
}
}
fn get_internal_replication_state(metadata: &HashMap<String, String>) -> Option<ReplicationState> {
let mut rs = ReplicationState::default();
let mut has = false;
for (k, v) in metadata.iter() {
if k == VERSION_PURGE_STATUS_KEY {
rs.version_purge_status_internal = Some(v.clone());
rs.purge_targets = version_purge_statuses_map(v.as_str());
has = true;
continue;
}
if let Some(sub_key) = k.strip_prefix(RESERVED_METADATA_PREFIX_LOWER) {
match sub_key {
"replica-timestamp" => {
has = true;
rs.replica_timestamp = Some(OffsetDateTime::parse(v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH));
}
"replica-status" => {
has = true;
rs.replica_status = ReplicationStatusType::from(v.as_str());
}
"replication-timestamp" => {
has = true;
rs.replication_timestamp = Some(OffsetDateTime::parse(v, &Rfc3339).unwrap_or(OffsetDateTime::UNIX_EPOCH))
}
"replication-status" => {
has = true;
rs.replication_status_internal = Some(v.clone());
rs.targets = replication_statuses_map(v.as_str());
}
_ => {
if let Some(arn) = sub_key.strip_prefix("replication-reset-") {
has = true;
rs.reset_statuses_map.insert(arn.to_string(), v.clone());
}
}
}
}
}
if has { Some(rs) } else { None }
}
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq)]
pub struct MetaDeleteMarker {
#[serde(rename = "ID")]
@@ -1939,24 +2064,51 @@ pub struct MetaDeleteMarker {
impl MetaDeleteMarker {
pub fn free_version(&self) -> bool {
self.meta_sys.contains_key(FREE_VERSION_META_HEADER)
self.meta_sys
.contains_key(format!("{RESERVED_METADATA_PREFIX_LOWER}{FREE_VERSION}").as_str())
}
pub fn into_fileinfo(&self, volume: &str, path: &str, _all_parts: bool) -> FileInfo {
let metadata = self.meta_sys.clone();
let metadata = self
.meta_sys
.clone()
.into_iter()
.map(|(k, v)| (k, String::from_utf8_lossy(&v).to_string()))
.collect();
let replication_state_internal = get_internal_replication_state(&metadata);
FileInfo {
let mut fi = FileInfo {
version_id: self.version_id.filter(|&vid| !vid.is_nil()),
name: path.to_string(),
volume: volume.to_string(),
deleted: true,
mod_time: self.mod_time,
metadata: metadata
.into_iter()
.map(|(k, v)| (k, String::from_utf8_lossy(&v).to_string()))
.collect(),
metadata,
replication_state_internal,
..Default::default()
};
if self.free_version() {
fi.set_tier_free_version();
fi.transition_tier = self
.meta_sys
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITION_TIER}").as_str())
.map(|v| String::from_utf8_lossy(v).to_string())
.unwrap_or_default();
fi.transitioned_objname = self
.meta_sys
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITIONED_OBJECTNAME}").as_str())
.map(|v| String::from_utf8_lossy(v).to_string())
.unwrap_or_default();
fi.transition_version_id = self
.meta_sys
.get(format!("{RESERVED_METADATA_PREFIX_LOWER}{TRANSITIONED_VERSION_ID}").as_str())
.map(|v| Uuid::from_slice(v.as_slice()).unwrap_or_default());
}
fi
}
pub fn unmarshal_msg(&mut self, buf: &[u8]) -> Result<u64> {
@@ -2160,8 +2312,6 @@ pub enum Flags {
InlineData = 1 << 2,
}
const FREE_VERSION_META_HEADER: &str = "free-version";
// mergeXLV2Versions
pub fn merge_file_meta_versions(
mut quorum: usize,

View File

@@ -12,7 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
mod append;
mod error;
pub mod fileinfo;
mod filemeta;
@@ -23,7 +22,6 @@ mod replication;
pub mod test_data;
pub use append::*;
pub use error::*;
pub use fileinfo::*;
pub use filemeta::*;

View File

@@ -1,8 +1,36 @@
use bytes::Bytes;
use core::fmt;
use regex::Regex;
use rustfs_utils::http::RESERVED_METADATA_PREFIX_LOWER;
use serde::{Deserialize, Serialize};
use std::any::Any;
use std::collections::HashMap;
use std::time::Duration;
use time::OffsetDateTime;
use uuid::Uuid;
pub const REPLICATION_RESET: &str = "replication-reset";
pub const REPLICATION_STATUS: &str = "replication-status";
// ReplicateQueued - replication being queued trail
pub const REPLICATE_QUEUED: &str = "replicate:queue";
// ReplicateExisting - audit trail for existing objects replication
pub const REPLICATE_EXISTING: &str = "replicate:existing";
// ReplicateExistingDelete - audit trail for delete replication triggered for existing delete markers
pub const REPLICATE_EXISTING_DELETE: &str = "replicate:existing:delete";
// ReplicateMRF - audit trail for replication from Most Recent Failures (MRF) queue
pub const REPLICATE_MRF: &str = "replicate:mrf";
// ReplicateIncoming - audit trail of inline replication
pub const REPLICATE_INCOMING: &str = "replicate:incoming";
// ReplicateIncomingDelete - audit trail of inline replication of deletes.
pub const REPLICATE_INCOMING_DELETE: &str = "replicate:incoming:delete";
// ReplicateHeal - audit trail for healing of failed/pending replications
pub const REPLICATE_HEAL: &str = "replicate:heal";
// ReplicateHealDelete - audit trail of healing of failed/pending delete replications.
pub const REPLICATE_HEAL_DELETE: &str = "replicate:heal:delete";
/// StatusType of Replication for x-amz-replication-status header
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, Default, Hash)]
@@ -492,3 +520,371 @@ impl ReplicatedInfos {
ReplicationAction::None
}
}
#[derive(Serialize, Deserialize, Debug)]
pub struct MrfReplicateEntry {
#[serde(rename = "bucket")]
pub bucket: String,
#[serde(rename = "object")]
pub object: String,
#[serde(skip_serializing, skip_deserializing)]
pub version_id: Option<Uuid>,
#[serde(rename = "retryCount")]
pub retry_count: i32,
#[serde(skip_serializing, skip_deserializing)]
pub size: i64,
}
pub trait ReplicationWorkerOperation: Any + Send + Sync {
fn to_mrf_entry(&self) -> MrfReplicateEntry;
fn as_any(&self) -> &dyn Any;
fn get_bucket(&self) -> &str;
fn get_object(&self) -> &str;
fn get_size(&self) -> i64;
fn is_delete_marker(&self) -> bool;
fn get_op_type(&self) -> ReplicationType;
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct ReplicateTargetDecision {
pub replicate: bool,
pub synchronous: bool,
pub arn: String,
pub id: String,
}
impl ReplicateTargetDecision {
pub fn new(arn: String, replicate: bool, sync: bool) -> Self {
Self {
replicate,
synchronous: sync,
arn,
id: String::new(),
}
}
}
impl fmt::Display for ReplicateTargetDecision {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{};{};{};{}", self.replicate, self.synchronous, self.arn, self.id)
}
}
/// ReplicateDecision represents replication decision for each target
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ReplicateDecision {
pub targets_map: HashMap<String, ReplicateTargetDecision>,
}
impl ReplicateDecision {
pub fn new() -> Self {
Self {
targets_map: HashMap::new(),
}
}
/// Returns true if at least one target qualifies for replication
pub fn replicate_any(&self) -> bool {
self.targets_map.values().any(|t| t.replicate)
}
/// Returns true if at least one target qualifies for synchronous replication
pub fn is_synchronous(&self) -> bool {
self.targets_map.values().any(|t| t.synchronous)
}
/// Updates ReplicateDecision with target's replication decision
pub fn set(&mut self, target: ReplicateTargetDecision) {
self.targets_map.insert(target.arn.clone(), target);
}
/// Returns a stringified representation of internal replication status with all targets marked as `PENDING`
pub fn pending_status(&self) -> Option<String> {
let mut result = String::new();
for target in self.targets_map.values() {
if target.replicate {
result.push_str(&format!("{}={};", target.arn, ReplicationStatusType::Pending.as_str()));
}
}
if result.is_empty() { None } else { Some(result) }
}
}
impl fmt::Display for ReplicateDecision {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
let mut result = String::new();
for (key, value) in &self.targets_map {
result.push_str(&format!("{key}={value},"));
}
write!(f, "{}", result.trim_end_matches(','))
}
}
impl Default for ReplicateDecision {
fn default() -> Self {
Self::new()
}
}
// parse k-v pairs of target ARN to stringified ReplicateTargetDecision delimited by ',' into a
// ReplicateDecision struct
pub fn parse_replicate_decision(_bucket: &str, s: &str) -> std::io::Result<ReplicateDecision> {
let mut decision = ReplicateDecision::new();
if s.is_empty() {
return Ok(decision);
}
for p in s.split(',') {
if p.is_empty() {
continue;
}
let slc = p.split('=').collect::<Vec<&str>>();
if slc.len() != 2 {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidInput,
format!("invalid replicate decision format: {s}"),
));
}
let tgt_str = slc[1].trim_matches('"');
let tgt = tgt_str.split(';').collect::<Vec<&str>>();
if tgt.len() != 4 {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidInput,
format!("invalid replicate decision format: {s}"),
));
}
let tgt = ReplicateTargetDecision {
replicate: tgt[0] == "true",
synchronous: tgt[1] == "true",
arn: tgt[2].to_string(),
id: tgt[3].to_string(),
};
decision.targets_map.insert(slc[0].to_string(), tgt);
}
Ok(decision)
// r = ReplicateDecision{
// targetsMap: make(map[string]replicateTargetDecision),
// }
// if len(s) == 0 {
// return
// }
// for _, p := range strings.Split(s, ",") {
// if p == "" {
// continue
// }
// slc := strings.Split(p, "=")
// if len(slc) != 2 {
// return r, errInvalidReplicateDecisionFormat
// }
// tgtStr := strings.TrimSuffix(strings.TrimPrefix(slc[1], `"`), `"`)
// tgt := strings.Split(tgtStr, ";")
// if len(tgt) != 4 {
// return r, errInvalidReplicateDecisionFormat
// }
// r.targetsMap[slc[0]] = replicateTargetDecision{Replicate: tgt[0] == "true", Synchronous: tgt[1] == "true", Arn: tgt[2], ID: tgt[3]}
// }
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ReplicateObjectInfo {
pub name: String,
pub size: i64,
pub actual_size: i64,
pub bucket: String,
pub version_id: Option<Uuid>,
pub etag: Option<String>,
pub mod_time: Option<OffsetDateTime>,
pub replication_status: ReplicationStatusType,
pub replication_status_internal: Option<String>,
pub delete_marker: bool,
pub version_purge_status_internal: Option<String>,
pub version_purge_status: VersionPurgeStatusType,
pub replication_state: Option<ReplicationState>,
pub op_type: ReplicationType,
pub event_type: String,
pub dsc: ReplicateDecision,
pub existing_obj_resync: ResyncDecision,
pub target_statuses: HashMap<String, ReplicationStatusType>,
pub target_purge_statuses: HashMap<String, VersionPurgeStatusType>,
pub replication_timestamp: Option<OffsetDateTime>,
pub ssec: bool,
pub user_tags: String,
pub checksum: Option<Bytes>,
pub retry_count: u32,
}
impl ReplicationWorkerOperation for ReplicateObjectInfo {
fn as_any(&self) -> &dyn Any {
self
}
fn to_mrf_entry(&self) -> MrfReplicateEntry {
MrfReplicateEntry {
bucket: self.bucket.clone(),
object: self.name.clone(),
version_id: self.version_id,
retry_count: self.retry_count as i32,
size: self.size,
}
}
fn get_bucket(&self) -> &str {
&self.bucket
}
fn get_object(&self) -> &str {
&self.name
}
fn get_size(&self) -> i64 {
self.size
}
fn is_delete_marker(&self) -> bool {
self.delete_marker
}
fn get_op_type(&self) -> ReplicationType {
self.op_type
}
}
lazy_static::lazy_static! {
static ref REPL_STATUS_REGEX: Regex = Regex::new(r"([^=].*?)=([^,].*?);").unwrap();
}
impl ReplicateObjectInfo {
/// Returns replication status of a target
pub fn target_replication_status(&self, arn: &str) -> ReplicationStatusType {
let binding = self.replication_status_internal.clone().unwrap_or_default();
let captures = REPL_STATUS_REGEX.captures_iter(&binding);
for cap in captures {
if cap.len() == 3 && &cap[1] == arn {
return ReplicationStatusType::from(&cap[2]);
}
}
ReplicationStatusType::default()
}
/// Returns the relevant info needed by MRF
pub fn to_mrf_entry(&self) -> MrfReplicateEntry {
MrfReplicateEntry {
bucket: self.bucket.clone(),
object: self.name.clone(),
version_id: self.version_id,
retry_count: self.retry_count as i32,
size: self.size,
}
}
}
// constructs a replication status map from string representation
pub fn replication_statuses_map(s: &str) -> HashMap<String, ReplicationStatusType> {
let mut targets = HashMap::new();
let rep_stat_matches = REPL_STATUS_REGEX.captures_iter(s).map(|c| c.extract());
for (_, [arn, status]) in rep_stat_matches {
if arn.is_empty() {
continue;
}
let status = ReplicationStatusType::from(status);
targets.insert(arn.to_string(), status);
}
targets
}
// constructs a version purge status map from string representation
pub fn version_purge_statuses_map(s: &str) -> HashMap<String, VersionPurgeStatusType> {
let mut targets = HashMap::new();
let purge_status_matches = REPL_STATUS_REGEX.captures_iter(s).map(|c| c.extract());
for (_, [arn, status]) in purge_status_matches {
if arn.is_empty() {
continue;
}
let status = VersionPurgeStatusType::from(status);
targets.insert(arn.to_string(), status);
}
targets
}
pub fn get_replication_state(rinfos: &ReplicatedInfos, prev_state: &ReplicationState, _vid: Option<String>) -> ReplicationState {
let reset_status_map: Vec<(String, String)> = rinfos
.targets
.iter()
.filter(|v| !v.resync_timestamp.is_empty())
.map(|t| (target_reset_header(t.arn.as_str()), t.resync_timestamp.clone()))
.collect();
let repl_statuses = rinfos.replication_status_internal();
let vpurge_statuses = rinfos.version_purge_status_internal();
let mut reset_statuses_map = prev_state.reset_statuses_map.clone();
for (key, value) in reset_status_map {
reset_statuses_map.insert(key, value);
}
ReplicationState {
replicate_decision_str: prev_state.replicate_decision_str.clone(),
reset_statuses_map,
replica_timestamp: prev_state.replica_timestamp,
replica_status: prev_state.replica_status.clone(),
targets: replication_statuses_map(&repl_statuses.clone().unwrap_or_default()),
replication_status_internal: repl_statuses,
replication_timestamp: rinfos.replication_timestamp,
purge_targets: version_purge_statuses_map(&vpurge_statuses.clone().unwrap_or_default()),
version_purge_status_internal: vpurge_statuses,
..Default::default()
}
}
pub fn target_reset_header(arn: &str) -> String {
format!("{RESERVED_METADATA_PREFIX_LOWER}{REPLICATION_RESET}-{arn}")
}
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub struct ResyncTargetDecision {
pub replicate: bool,
pub reset_id: String,
pub reset_before_date: Option<OffsetDateTime>,
}
/// ResyncDecision is a struct representing a map with target's individual resync decisions
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ResyncDecision {
pub targets: HashMap<String, ResyncTargetDecision>,
}
impl ResyncDecision {
pub fn new() -> Self {
Self { targets: HashMap::new() }
}
/// Returns true if no targets with resync decision present
pub fn is_empty(&self) -> bool {
self.targets.is_empty()
}
pub fn must_resync(&self) -> bool {
self.targets.values().any(|v| v.replicate)
}
pub fn must_resync_target(&self, tgt_arn: &str) -> bool {
self.targets.get(tgt_arn).map(|v| v.replicate).unwrap_or(false)
}
}
impl Default for ResyncDecision {
fn default() -> Self {
Self::new()
}
}

View File

@@ -34,6 +34,7 @@ time = { workspace = true, features = ["serde-human-readable"] }
serde = { workspace = true, features = ["derive", "rc"] }
rustfs-ecstore = { workspace = true }
rustfs-policy.workspace = true
rustfs-config.workspace = true
serde_json.workspace = true
async-trait.workspace = true
thiserror.workspace = true

View File

@@ -33,7 +33,6 @@ use rustfs_policy::{
EMBEDDED_POLICY_TYPE, INHERITED_POLICY_TYPE, Policy, PolicyDoc, default::DEFAULT_POLICIES, iam_policy_claim_name_sa,
},
};
use rustfs_utils::crypto::base64_encode;
use rustfs_utils::path::path_join_buf;
use serde::{Deserialize, Serialize};
use serde_json::Value;
@@ -555,7 +554,10 @@ where
return Err(Error::PolicyTooLarge);
}
m.insert(SESSION_POLICY_NAME.to_owned(), Value::String(base64_encode(&policy_buf)));
m.insert(
SESSION_POLICY_NAME.to_owned(),
Value::String(base64_simd::URL_SAFE_NO_PAD.encode_to_string(&policy_buf)),
);
m.insert(iam_policy_claim_name_sa(), Value::String(EMBEDDED_POLICY_TYPE.to_owned()));
}
}

View File

@@ -35,14 +35,16 @@ use rustfs_policy::auth::{
is_access_key_valid, is_secret_key_valid,
};
use rustfs_policy::policy::Args;
use rustfs_policy::policy::opa;
use rustfs_policy::policy::{EMBEDDED_POLICY_TYPE, INHERITED_POLICY_TYPE, Policy, PolicyDoc, iam_policy_claim_name_sa};
use rustfs_utils::crypto::{base64_decode, base64_encode};
use serde_json::Value;
use serde_json::json;
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::OnceLock;
use time::OffsetDateTime;
use tracing::warn;
use tokio::sync::RwLock;
use tracing::{error, info, warn};
pub const MAX_SVCSESSION_POLICY_SIZE: usize = 4096;
@@ -53,6 +55,12 @@ pub const POLICYNAME: &str = "policy";
pub const SESSION_POLICY_NAME: &str = "sessionPolicy";
pub const SESSION_POLICY_NAME_EXTRACTED: &str = "sessionPolicy-extracted";
static POLICY_PLUGIN_CLIENT: OnceLock<Arc<RwLock<Option<rustfs_policy::policy::opa::AuthZPlugin>>>> = OnceLock::new();
fn get_policy_plugin_client() -> Arc<RwLock<Option<rustfs_policy::policy::opa::AuthZPlugin>>> {
POLICY_PLUGIN_CLIENT.get_or_init(|| Arc::new(RwLock::new(None))).clone()
}
pub struct IamSys<T> {
store: Arc<IamCache<T>>,
roles_map: HashMap<ARN, String>,
@@ -60,6 +68,20 @@ pub struct IamSys<T> {
impl<T: Store> IamSys<T> {
pub fn new(store: Arc<IamCache<T>>) -> Self {
tokio::spawn(async move {
match opa::lookup_config().await {
Ok(conf) => {
if conf.enable() {
Self::set_policy_plugin_client(opa::AuthZPlugin::new(conf)).await;
info!("OPA plugin enabled");
}
}
Err(e) => {
error!("Error loading OPA configuration err:{}", e);
}
};
});
Self {
store,
roles_map: HashMap::new(),
@@ -69,6 +91,18 @@ impl<T: Store> IamSys<T> {
self.store.api.has_watcher()
}
pub async fn set_policy_plugin_client(client: rustfs_policy::policy::opa::AuthZPlugin) {
let policy_plugin_client = get_policy_plugin_client();
let mut guard = policy_plugin_client.write().await;
*guard = Some(client);
}
pub async fn get_policy_plugin_client() -> Option<rustfs_policy::policy::opa::AuthZPlugin> {
let policy_plugin_client = get_policy_plugin_client();
let guard = policy_plugin_client.read().await;
guard.clone()
}
pub async fn load_group(&self, name: &str) -> Result<()> {
self.store.group_notification_handler(name).await
}
@@ -328,7 +362,10 @@ impl<T: Store> IamSys<T> {
m.insert("parent".to_owned(), Value::String(parent_user.to_owned()));
if !policy_buf.is_empty() {
m.insert(SESSION_POLICY_NAME.to_owned(), Value::String(base64_encode(&policy_buf)));
m.insert(
SESSION_POLICY_NAME.to_owned(),
Value::String(base64_simd::URL_SAFE_NO_PAD.encode_to_string(&policy_buf)),
);
m.insert(iam_policy_claim_name_sa(), Value::String(EMBEDDED_POLICY_TYPE.to_owned()));
} else {
m.insert(iam_policy_claim_name_sa(), Value::String(INHERITED_POLICY_TYPE.to_owned()));
@@ -421,7 +458,9 @@ impl<T: Store> IamSys<T> {
let op_sp = claims.get(SESSION_POLICY_NAME);
if let (Some(pt), Some(sp)) = (op_pt, op_sp) {
if pt == EMBEDDED_POLICY_TYPE {
let policy = serde_json::from_slice(&base64_decode(sp.as_str().unwrap_or_default().as_bytes())?)?;
let policy = serde_json::from_slice(
&base64_simd::URL_SAFE_NO_PAD.decode_to_vec(sp.as_str().unwrap_or_default().as_bytes())?,
)?;
return Ok((sa, Some(policy)));
}
}
@@ -480,7 +519,9 @@ impl<T: Store> IamSys<T> {
let op_sp = claims.get(SESSION_POLICY_NAME);
if let (Some(pt), Some(sp)) = (op_pt, op_sp) {
if pt == EMBEDDED_POLICY_TYPE {
let policy = serde_json::from_slice(&base64_decode(sp.as_str().unwrap_or_default().as_bytes())?)?;
let policy = serde_json::from_slice(
&base64_simd::URL_SAFE_NO_PAD.decode_to_vec(sp.as_str().unwrap_or_default().as_bytes())?,
)?;
return Ok((sa, Some(policy)));
}
}
@@ -493,7 +534,7 @@ impl<T: Store> IamSys<T> {
return Err(IamError::NoSuchServiceAccount(access_key.to_string()));
};
if u.credentials.is_service_account() {
if !u.credentials.is_service_account() {
return Err(IamError::NoSuchServiceAccount(access_key.to_string()));
}
@@ -766,6 +807,11 @@ impl<T: Store> IamSys<T> {
return true;
}
let opa_enable = Self::get_policy_plugin_client().await;
if let Some(opa_enable) = opa_enable {
return opa_enable.is_allowed(args).await;
}
let Ok((is_temp, parent_user)) = self.is_temp_user(args.account).await else { return false };
if is_temp {
@@ -866,7 +912,9 @@ pub fn get_claims_from_token_with_secret(token: &str, secret: &str) -> Result<Ha
if let Some(session_policy) = ms.claims.get(SESSION_POLICY_NAME) {
let policy_str = session_policy.as_str().unwrap_or_default();
let policy = base64_decode(policy_str.as_bytes()).map_err(|e| Error::other(format!("base64 decode err {e}")))?;
let policy = base64_simd::URL_SAFE_NO_PAD
.decode_to_vec(policy_str.as_bytes())
.map_err(|e| Error::other(format!("base64 decode err {e}")))?;
ms.claims.insert(
SESSION_POLICY_NAME_EXTRACTED.to_string(),
Value::String(String::from_utf8(policy).map_err(|e| Error::other(format!("utf8 decode err {e}")))?),

View File

@@ -635,7 +635,7 @@ impl KmsBackend for LocalKmsBackend {
}
async fn encrypt(&self, request: EncryptRequest) -> Result<EncryptResponse> {
let encrypt_request = crate::types::EncryptRequest {
let encrypt_request = EncryptRequest {
key_id: request.key_id.clone(),
plaintext: request.plaintext,
encryption_context: request.encryption_context,
@@ -719,14 +719,14 @@ impl KmsBackend for LocalKmsBackend {
.client
.load_master_key(key_id)
.await
.map_err(|_| crate::error::KmsError::key_not_found(format!("Key {key_id} not found")))?;
.map_err(|_| KmsError::key_not_found(format!("Key {key_id} not found")))?;
let (deletion_date_str, deletion_date_dt) = if request.force_immediate.unwrap_or(false) {
// For immediate deletion, actually delete the key from filesystem
let key_path = self.client.master_key_path(key_id);
tokio::fs::remove_file(&key_path)
.await
.map_err(|e| crate::error::KmsError::internal_error(format!("Failed to delete key file: {e}")))?;
.map_err(|e| KmsError::internal_error(format!("Failed to delete key file: {e}")))?;
// Remove from cache
let mut cache = self.client.key_cache.write().await;
@@ -756,9 +756,7 @@ impl KmsBackend for LocalKmsBackend {
// Schedule for deletion (default 30 days)
let days = request.pending_window_in_days.unwrap_or(30);
if !(7..=30).contains(&days) {
return Err(crate::error::KmsError::invalid_parameter(
"pending_window_in_days must be between 7 and 30".to_string(),
));
return Err(KmsError::invalid_parameter("pending_window_in_days must be between 7 and 30".to_string()));
}
let deletion_date = chrono::Utc::now() + chrono::Duration::days(days as i64);
@@ -772,16 +770,16 @@ impl KmsBackend for LocalKmsBackend {
let key_path = self.client.master_key_path(key_id);
let content = tokio::fs::read(&key_path)
.await
.map_err(|e| crate::error::KmsError::internal_error(format!("Failed to read key file: {e}")))?;
let stored_key: crate::backends::local::StoredMasterKey = serde_json::from_slice(&content)
.map_err(|e| crate::error::KmsError::internal_error(format!("Failed to parse stored key: {e}")))?;
.map_err(|e| KmsError::internal_error(format!("Failed to read key file: {e}")))?;
let stored_key: StoredMasterKey =
serde_json::from_slice(&content).map_err(|e| KmsError::internal_error(format!("Failed to parse stored key: {e}")))?;
// Decrypt the existing key material to preserve it
let existing_key_material = if let Some(ref cipher) = self.client.master_cipher {
let nonce = aes_gcm::Nonce::from_slice(&stored_key.nonce);
let nonce = Nonce::from_slice(&stored_key.nonce);
cipher
.decrypt(nonce, stored_key.encrypted_key_material.as_ref())
.map_err(|e| crate::error::KmsError::cryptographic_error("decrypt", e.to_string()))?
.map_err(|e| KmsError::cryptographic_error("decrypt", e.to_string()))?
} else {
stored_key.encrypted_key_material
};
@@ -820,10 +818,10 @@ impl KmsBackend for LocalKmsBackend {
.client
.load_master_key(key_id)
.await
.map_err(|_| crate::error::KmsError::key_not_found(format!("Key {key_id} not found")))?;
.map_err(|_| KmsError::key_not_found(format!("Key {key_id} not found")))?;
if master_key.status != KeyStatus::PendingDeletion {
return Err(crate::error::KmsError::invalid_key_state(format!("Key {key_id} is not pending deletion")));
return Err(KmsError::invalid_key_state(format!("Key {key_id} is not pending deletion")));
}
// Cancel the deletion by resetting the state

View File

@@ -68,7 +68,7 @@ async fn main() -> Result<(), NotificationError> {
key: WEBHOOK_QUEUE_DIR.to_string(),
value: current_root
.clone()
.join("../../deploy/logs/notify/webhook")
.join("../../deploy/logs/notify")
.to_str()
.unwrap()
.to_string(),
@@ -120,11 +120,7 @@ async fn main() -> Result<(), NotificationError> {
},
KV {
key: MQTT_QUEUE_DIR.to_string(),
value: current_root
.join("../../deploy/logs/notify/mqtt")
.to_str()
.unwrap()
.to_string(),
value: current_root.join("../../deploy/logs/notify").to_str().unwrap().to_string(),
hidden_if_empty: false,
},
KV {
@@ -137,7 +133,7 @@ async fn main() -> Result<(), NotificationError> {
let mqtt_kvs = KVS(mqtt_kvs_vec);
let mut mqtt_targets = std::collections::HashMap::new();
mqtt_targets.insert(DEFAULT_TARGET.to_string(), mqtt_kvs);
config.0.insert(NOTIFY_MQTT_SUB_SYS.to_string(), mqtt_targets);
// config.0.insert(NOTIFY_MQTT_SUB_SYS.to_string(), mqtt_targets);
// Load the configuration and initialize the system
*system.config.write().await = config;

View File

@@ -28,6 +28,7 @@ use rustfs_targets::EventName;
use rustfs_targets::arn::TargetID;
use std::sync::Arc;
use std::time::Duration;
use tokio::time::sleep;
use tracing::info;
#[tokio::main]
@@ -68,7 +69,7 @@ async fn main() -> Result<(), NotificationError> {
key: WEBHOOK_QUEUE_DIR.to_string(),
value: current_root
.clone()
.join("../../deploy/logs/notify/webhook")
.join("../../deploy/logs/notify")
.to_str()
.unwrap()
.to_string(),
@@ -91,7 +92,7 @@ async fn main() -> Result<(), NotificationError> {
system.init().await?;
info!("✅ System initialized with Webhook target.");
tokio::time::sleep(Duration::from_secs(1)).await;
sleep(Duration::from_secs(1)).await;
// --- Dynamically update system configuration: Add an MQTT Target ---
info!("\n---> Dynamically adding MQTT target...");
@@ -129,11 +130,7 @@ async fn main() -> Result<(), NotificationError> {
},
KV {
key: MQTT_QUEUE_DIR.to_string(),
value: current_root
.join("../../deploy/logs/notify/mqtt")
.to_str()
.unwrap()
.to_string(),
value: current_root.join("../../deploy/logs/notify").to_str().unwrap().to_string(),
hidden_if_empty: false,
},
KV {
@@ -152,7 +149,7 @@ async fn main() -> Result<(), NotificationError> {
.await?;
info!("✅ MQTT target added and system reloaded.");
tokio::time::sleep(Duration::from_secs(1)).await;
sleep(Duration::from_secs(1)).await;
// --- Loading and managing Bucket configurations ---
info!("\n---> Loading bucket notification config...");
@@ -176,7 +173,7 @@ async fn main() -> Result<(), NotificationError> {
system.send_event(event).await;
info!("✅ Event sent. Both Webhook and MQTT targets should receive it.");
tokio::time::sleep(Duration::from_secs(2)).await;
sleep(Duration::from_secs(2)).await;
// --- Dynamically remove configuration ---
info!("\n---> Dynamically removing Webhook target...");
@@ -188,5 +185,6 @@ async fn main() -> Result<(), NotificationError> {
info!("✅ Bucket 'my-bucket' config removed.");
info!("\nDemo completed successfully");
sleep(Duration::from_secs(1)).await;
Ok(())
}

View File

@@ -12,19 +12,20 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use axum::routing::get;
use axum::{
Router,
extract::Json,
extract::Query,
http::{HeaderMap, Response, StatusCode},
routing::post,
routing::{get, post},
};
use rustfs_utils::parse_and_resolve_address;
use serde::Deserialize;
use serde_json::Value;
use std::net::SocketAddr;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{SystemTime, UNIX_EPOCH};
use axum::extract::Query;
use serde::Deserialize;
use tokio::net::TcpListener;
#[derive(Deserialize)]
struct ResetParams {
@@ -32,9 +33,6 @@ struct ResetParams {
}
// Define a global variable and count the number of data received
use rustfs_utils::parse_and_resolve_address;
use std::sync::atomic::{AtomicU64, Ordering};
use tokio::net::TcpListener;
static WEBHOOK_COUNT: AtomicU64 = AtomicU64::new(0);

View File

@@ -296,8 +296,8 @@ impl NotificationSystem {
info!("Removing config for target {} of type {}", target_name, target_type);
self.update_config_and_reload(|config| {
let mut changed = false;
if let Some(targets) = config.0.get_mut(target_type) {
if targets.remove(target_name).is_some() {
if let Some(targets) = config.0.get_mut(&target_type.to_lowercase()) {
if targets.remove(&target_name.to_lowercase()).is_some() {
changed = true;
}
if targets.is_empty() {
@@ -307,6 +307,7 @@ impl NotificationSystem {
if !changed {
info!("Target {} of type {} not found, no changes made.", target_name, target_type);
}
debug!("Config after remove: {:?}", config);
changed
})
.await

View File

@@ -16,12 +16,9 @@ use crate::Event;
use crate::factory::{MQTTTargetFactory, TargetFactory, WebhookTargetFactory};
use futures::stream::{FuturesUnordered, StreamExt};
use hashbrown::{HashMap, HashSet};
use rustfs_config::notify::NOTIFY_ROUTE_PREFIX;
use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, ENV_PREFIX};
use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, ENV_PREFIX, notify::NOTIFY_ROUTE_PREFIX};
use rustfs_ecstore::config::{Config, KVS};
use rustfs_targets::Target;
use rustfs_targets::TargetError;
use rustfs_targets::target::ChannelTargetType;
use rustfs_targets::{Target, TargetError, target::ChannelTargetType};
use tracing::{debug, error, info, warn};
/// Registry for managing target factories
@@ -90,7 +87,9 @@ impl TargetRegistry {
let all_env: Vec<(String, String)> = std::env::vars().filter(|(key, _)| key.starts_with(ENV_PREFIX)).collect();
// A collection of asynchronous tasks for concurrently executing target creation
let mut tasks = FuturesUnordered::new();
let mut final_config = config.clone(); // Clone a configuration for aggregating the final result
// let final_config = config.clone(); // Clone a configuration for aggregating the final result
// Record the defaults for each segment so that the segment can eventually be rebuilt
let mut section_defaults: HashMap<String, KVS> = HashMap::new();
// 1. Traverse all registered plants and process them by target type
for (target_type, factory) in &self.factories {
tracing::Span::current().record("target_type", target_type.as_str());
@@ -98,12 +97,15 @@ impl TargetRegistry {
// 2. Prepare the configuration source
// 2.1. Get the configuration segment in the file, e.g. 'notify_webhook'
let section_name = format!("{NOTIFY_ROUTE_PREFIX}{target_type}");
let section_name = format!("{NOTIFY_ROUTE_PREFIX}{target_type}").to_lowercase();
let file_configs = config.0.get(&section_name).cloned().unwrap_or_default();
// 2.2. Get the default configuration for that type
let default_cfg = file_configs.get(DEFAULT_DELIMITER).cloned().unwrap_or_default();
debug!(?default_cfg, "Get the default configuration");
// Save defaults for eventual write back
section_defaults.insert(section_name.clone(), default_cfg.clone());
// *** Optimization point 1: Get all legitimate fields of the current target type ***
let valid_fields = factory.get_valid_fields();
debug!(?valid_fields, "Get the legitimate configuration fields");
@@ -111,7 +113,9 @@ impl TargetRegistry {
// 3. Resolve instance IDs and configuration overrides from environment variables
let mut instance_ids_from_env = HashSet::new();
// 3.1. Instance discovery: Based on the '..._ENABLE_INSTANCEID' format
let enable_prefix = format!("{ENV_PREFIX}{NOTIFY_ROUTE_PREFIX}{target_type}_{ENABLE_KEY}_").to_uppercase();
let enable_prefix =
format!("{ENV_PREFIX}{NOTIFY_ROUTE_PREFIX}{target_type}{DEFAULT_DELIMITER}{ENABLE_KEY}{DEFAULT_DELIMITER}")
.to_uppercase();
for (key, value) in &all_env {
if value.eq_ignore_ascii_case(rustfs_config::EnableState::One.as_str())
|| value.eq_ignore_ascii_case(rustfs_config::EnableState::On.as_str())
@@ -128,14 +132,14 @@ impl TargetRegistry {
// 3.2. Parse all relevant environment variable configurations
// 3.2.1. Build environment variable prefixes such as 'RUSTFS_NOTIFY_WEBHOOK_'
let env_prefix = format!("{ENV_PREFIX}{NOTIFY_ROUTE_PREFIX}{target_type}_").to_uppercase();
let env_prefix = format!("{ENV_PREFIX}{NOTIFY_ROUTE_PREFIX}{target_type}{DEFAULT_DELIMITER}").to_uppercase();
// 3.2.2. 'env_overrides' is used to store configurations parsed from environment variables in the format: {instance id -> {field -> value}}
let mut env_overrides: HashMap<String, HashMap<String, String>> = HashMap::new();
for (key, value) in &all_env {
if let Some(rest) = key.strip_prefix(&env_prefix) {
// Use rsplitn to split from the right side to properly extract the INSTANCE_ID at the end
// Format: <FIELD_NAME>_<INSTANCE_ID> or <FIELD_NAME>
let mut parts = rest.rsplitn(2, '_');
let mut parts = rest.rsplitn(2, DEFAULT_DELIMITER);
// The first part from the right is INSTANCE_ID
let instance_id_part = parts.next().unwrap_or(DEFAULT_DELIMITER);
@@ -224,7 +228,7 @@ impl TargetRegistry {
} else {
info!(instance_id = %id, "Skip the disabled target and will be removed from the final configuration");
// Remove disabled target from final configuration
final_config.0.entry(section_name.clone()).or_default().remove(&id);
// final_config.0.entry(section_name.clone()).or_default().remove(&id);
}
}
}
@@ -246,15 +250,50 @@ impl TargetRegistry {
}
// 7. Aggregate new configuration and write back to system configuration
if !successful_configs.is_empty() {
if !successful_configs.is_empty() || !section_defaults.is_empty() {
info!(
"Prepare to update {} successfully created target configurations to the system configuration...",
successful_configs.len()
);
let mut new_config = config.clone();
let mut successes_by_section: HashMap<String, HashMap<String, KVS>> = HashMap::new();
for (target_type, id, kvs) in successful_configs {
let section_name = format!("{NOTIFY_ROUTE_PREFIX}{target_type}").to_lowercase();
new_config.0.entry(section_name).or_default().insert(id, (*kvs).clone());
successes_by_section
.entry(section_name)
.or_default()
.insert(id.to_lowercase(), (*kvs).clone());
}
let mut new_config = config.clone();
// Collection of segments that need to be processed: Collect all segments where default items exist or where successful instances exist
let mut sections: HashSet<String> = HashSet::new();
sections.extend(section_defaults.keys().cloned());
sections.extend(successes_by_section.keys().cloned());
for section in sections {
let mut section_map: std::collections::HashMap<String, KVS> = std::collections::HashMap::new();
// Add default item
if let Some(default_kvs) = section_defaults.get(&section) {
if !default_kvs.is_empty() {
section_map.insert(DEFAULT_DELIMITER.to_string(), default_kvs.clone());
}
}
// Add successful instance item
if let Some(instances) = successes_by_section.get(&section) {
for (id, kvs) in instances {
section_map.insert(id.clone(), kvs.clone());
}
}
// Empty breaks are removed and non-empty breaks are replaced entirely.
if section_map.is_empty() {
new_config.0.remove(&section);
} else {
new_config.0.insert(section, section_map);
}
}
let Some(store) = rustfs_ecstore::global::new_object_layer_fn() else {

View File

@@ -29,17 +29,12 @@ documentation = "https://docs.rs/rustfs-obs/latest/rustfs_obs/"
workspace = true
[features]
default = ["file"]
file = []
default = []
gpu = ["dep:nvml-wrapper"]
webhook = ["dep:reqwest"]
kafka = ["dep:rdkafka"]
[dependencies]
rustfs-config = { workspace = true, features = ["constants", "observability"] }
rustfs-utils = { workspace = true, features = ["ip", "path"] }
async-trait = { workspace = true }
chrono = { workspace = true }
flexi_logger = { workspace = true }
nu-ansi-term = { workspace = true }
nvml-wrapper = { workspace = true, optional = true }
@@ -57,24 +52,9 @@ tracing-error = { workspace = true }
tracing-opentelemetry = { workspace = true }
tracing-subscriber = { workspace = true, features = ["registry", "std", "fmt", "env-filter", "tracing-log", "time", "local-time", "json"] }
tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
reqwest = { workspace = true, optional = true }
serde_json = { workspace = true }
sysinfo = { workspace = true }
thiserror = { workspace = true }
# Only enable kafka features and related dependencies on Linux
[target.'cfg(target_os = "linux")'.dependencies]
rdkafka = { workspace = true, features = ["tokio"], optional = true }
[dev-dependencies]
chrono = { workspace = true }
opentelemetry = { workspace = true }
opentelemetry_sdk = { workspace = true, features = ["rt-tokio"] }
opentelemetry-stdout = { workspace = true }
opentelemetry-otlp = { workspace = true, features = ["grpc-tonic"] }
opentelemetry-semantic-conventions = { workspace = true, features = ["semconv_experimental"] }
tokio = { workspace = true, features = ["full"] }
tracing = { workspace = true, features = ["std", "attributes"] }
tracing-subscriber = { workspace = true, features = ["registry", "std", "fmt"] }

View File

@@ -21,29 +21,4 @@ service_name = "rustfs"
service_version = "0.1.0"
environments = "develop"
logger_level = "debug"
local_logging_enabled = true # Default is false if not specified
#[[sinks]]
#type = "Kafka"
#bootstrap_servers = "localhost:9092"
#topic = "logs"
#batch_size = 100 # Default is 100 if not specified
#batch_timeout_ms = 100 # Default is 1000ms if not specified
#
#[[sinks]]
#type = "Webhook"
#endpoint = "http://localhost:8080/webhook"
#auth_token = ""
#batch_size = 100 # Default is 3 if not specified
#batch_timeout_ms = 100 # Default is 100ms if not specified
[[sinks]]
type = "File"
path = "deploy/logs/rustfs.log"
buffer_size = 102 # Default is 8192 bytes if not specified
flush_interval_ms = 1000
flush_threshold = 100
[logger]
queue_capacity = 10000
local_logging_enabled = true # Default is false if not specified

View File

@@ -13,33 +13,25 @@
// limitations under the License.
use opentelemetry::global;
use rustfs_obs::{BaseLogEntry, ServerLogEntry, SystemObserver, get_logger, init_obs, log_info};
use std::collections::HashMap;
use rustfs_obs::{SystemObserver, init_obs};
use std::time::{Duration, SystemTime};
use tracing::{error, info, instrument};
use tracing_core::Level;
use tracing::{Level, error, info, instrument};
#[tokio::main]
async fn main() {
let obs_conf = Some("crates/obs/examples/config.toml".to_string());
let (_logger, _guard) = init_obs(obs_conf).await;
let obs_conf = Some("http://localhost:4317".to_string());
let _guard = init_obs(obs_conf).await;
let span = tracing::span!(Level::INFO, "main");
let _enter = span.enter();
info!("Program starts");
// Simulate the operation
tokio::time::sleep(Duration::from_millis(100)).await;
run(
"service-demo".to_string(),
"object-demo".to_string(),
"user-demo".to_string(),
"service-demo".to_string(),
)
.await;
run("service-demo".to_string()).await;
info!("Program ends");
}
#[instrument(fields(bucket, object, user))]
async fn run(bucket: String, object: String, user: String, service_name: String) {
async fn run(service_name: String) {
let start_time = SystemTime::now();
info!("Log module initialization is completed service_name: {:?}", service_name);
@@ -56,21 +48,6 @@ async fn run(bucket: String, object: String, user: String, service_name: String)
Err(e) => error!("Failed to initialize process observer: {:?}", e),
}
let base_entry = BaseLogEntry::new()
.message(Some("run logger api_handler info".to_string()))
.request_id(Some("request_id".to_string()))
.timestamp(chrono::DateTime::from(start_time))
.tags(Some(HashMap::default()));
let server_entry = ServerLogEntry::new(Level::INFO, "api_handler".to_string())
.with_base(base_entry)
.user_id(Some(user.clone()))
.add_field("operation".to_string(), "login".to_string())
.add_field("bucket".to_string(), bucket.clone())
.add_field("object".to_string(), object.clone());
let result = get_logger().lock().await.log_server_entry(server_entry).await;
info!("Logging is completed {:?}", result);
put_object("bucket".to_string(), "object".to_string(), "user".to_string()).await;
info!("Logging is completed");
tokio::time::sleep(Duration::from_secs(2)).await;
@@ -97,8 +74,6 @@ async fn put_object(bucket: String, object: String, user: String) {
start_time.elapsed().unwrap().as_secs_f64()
);
let result = log_info("put_object logger info", "put_object").await;
info!("put_object is completed {:?}", result);
// Simulate the operation
tokio::time::sleep(Duration::from_millis(100)).await;

View File

@@ -13,16 +13,9 @@
// limitations under the License.
use rustfs_config::observability::{
DEFAULT_AUDIT_LOGGER_QUEUE_CAPACITY, DEFAULT_SINKS_FILE_BUFFER_SIZE, DEFAULT_SINKS_FILE_FLUSH_INTERVAL_MS,
DEFAULT_SINKS_FILE_FLUSH_THRESHOLD, DEFAULT_SINKS_KAFKA_BATCH_SIZE, DEFAULT_SINKS_KAFKA_BATCH_TIMEOUT_MS,
DEFAULT_SINKS_KAFKA_BROKERS, DEFAULT_SINKS_KAFKA_TOPIC, DEFAULT_SINKS_WEBHOOK_AUTH_TOKEN, DEFAULT_SINKS_WEBHOOK_ENDPOINT,
DEFAULT_SINKS_WEBHOOK_MAX_RETRIES, DEFAULT_SINKS_WEBHOOK_RETRY_DELAY_MS, ENV_AUDIT_LOGGER_QUEUE_CAPACITY, ENV_OBS_ENDPOINT,
ENV_OBS_ENVIRONMENT, ENV_OBS_LOCAL_LOGGING_ENABLED, ENV_OBS_LOG_DIRECTORY, ENV_OBS_LOG_FILENAME, ENV_OBS_LOG_KEEP_FILES,
ENV_OBS_LOG_ROTATION_SIZE_MB, ENV_OBS_LOG_ROTATION_TIME, ENV_OBS_LOGGER_LEVEL, ENV_OBS_METER_INTERVAL, ENV_OBS_SAMPLE_RATIO,
ENV_OBS_SERVICE_NAME, ENV_OBS_SERVICE_VERSION, ENV_OBS_USE_STDOUT, ENV_SINKS_FILE_BUFFER_SIZE,
ENV_SINKS_FILE_FLUSH_INTERVAL_MS, ENV_SINKS_FILE_FLUSH_THRESHOLD, ENV_SINKS_FILE_PATH, ENV_SINKS_KAFKA_BATCH_SIZE,
ENV_SINKS_KAFKA_BATCH_TIMEOUT_MS, ENV_SINKS_KAFKA_BROKERS, ENV_SINKS_KAFKA_TOPIC, ENV_SINKS_WEBHOOK_AUTH_TOKEN,
ENV_SINKS_WEBHOOK_ENDPOINT, ENV_SINKS_WEBHOOK_MAX_RETRIES, ENV_SINKS_WEBHOOK_RETRY_DELAY_MS,
ENV_OBS_ENDPOINT, ENV_OBS_ENVIRONMENT, ENV_OBS_LOCAL_LOGGING_ENABLED, ENV_OBS_LOG_DIRECTORY, ENV_OBS_LOG_FILENAME,
ENV_OBS_LOG_KEEP_FILES, ENV_OBS_LOG_ROTATION_SIZE_MB, ENV_OBS_LOG_ROTATION_TIME, ENV_OBS_LOGGER_LEVEL,
ENV_OBS_METER_INTERVAL, ENV_OBS_SAMPLE_RATIO, ENV_OBS_SERVICE_NAME, ENV_OBS_SERVICE_VERSION, ENV_OBS_USE_STDOUT,
};
use rustfs_config::{
APP_NAME, DEFAULT_LOG_KEEP_FILES, DEFAULT_LOG_LEVEL, DEFAULT_LOG_ROTATION_SIZE_MB, DEFAULT_LOG_ROTATION_TIME,
@@ -145,167 +138,10 @@ impl Default for OtelConfig {
}
}
/// Kafka Sink Configuration - Add batch parameters
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct KafkaSinkConfig {
pub brokers: String,
pub topic: String,
pub batch_size: Option<usize>, // Batch size, default 100
pub batch_timeout_ms: Option<u64>, // Batch timeout time, default 1000ms
}
impl KafkaSinkConfig {
pub fn new() -> Self {
Self::default()
}
}
impl Default for KafkaSinkConfig {
fn default() -> Self {
Self {
brokers: env::var(ENV_SINKS_KAFKA_BROKERS)
.ok()
.filter(|s| !s.trim().is_empty())
.unwrap_or_else(|| DEFAULT_SINKS_KAFKA_BROKERS.to_string()),
topic: env::var(ENV_SINKS_KAFKA_TOPIC)
.ok()
.filter(|s| !s.trim().is_empty())
.unwrap_or_else(|| DEFAULT_SINKS_KAFKA_TOPIC.to_string()),
batch_size: env::var(ENV_SINKS_KAFKA_BATCH_SIZE)
.ok()
.and_then(|v| v.parse().ok())
.or(Some(DEFAULT_SINKS_KAFKA_BATCH_SIZE)),
batch_timeout_ms: env::var(ENV_SINKS_KAFKA_BATCH_TIMEOUT_MS)
.ok()
.and_then(|v| v.parse().ok())
.or(Some(DEFAULT_SINKS_KAFKA_BATCH_TIMEOUT_MS)),
}
}
}
/// Webhook Sink Configuration - Add Retry Parameters
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct WebhookSinkConfig {
pub endpoint: String,
pub auth_token: String,
pub max_retries: Option<usize>, // Maximum number of retry times, default 3
pub retry_delay_ms: Option<u64>, // Retry the delay cardinality, default 100ms
}
impl WebhookSinkConfig {
pub fn new() -> Self {
Self::default()
}
}
impl Default for WebhookSinkConfig {
fn default() -> Self {
Self {
endpoint: env::var(ENV_SINKS_WEBHOOK_ENDPOINT)
.ok()
.filter(|s| !s.trim().is_empty())
.unwrap_or_else(|| DEFAULT_SINKS_WEBHOOK_ENDPOINT.to_string()),
auth_token: env::var(ENV_SINKS_WEBHOOK_AUTH_TOKEN)
.ok()
.filter(|s| !s.trim().is_empty())
.unwrap_or_else(|| DEFAULT_SINKS_WEBHOOK_AUTH_TOKEN.to_string()),
max_retries: env::var(ENV_SINKS_WEBHOOK_MAX_RETRIES)
.ok()
.and_then(|v| v.parse().ok())
.or(Some(DEFAULT_SINKS_WEBHOOK_MAX_RETRIES)),
retry_delay_ms: env::var(ENV_SINKS_WEBHOOK_RETRY_DELAY_MS)
.ok()
.and_then(|v| v.parse().ok())
.or(Some(DEFAULT_SINKS_WEBHOOK_RETRY_DELAY_MS)),
}
}
}
/// File Sink Configuration - Add buffering parameters
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct FileSinkConfig {
pub path: String,
pub buffer_size: Option<usize>, // Write buffer size, default 8192
pub flush_interval_ms: Option<u64>, // Refresh interval time, default 1000ms
pub flush_threshold: Option<usize>, // Refresh threshold, default 100 logs
}
impl FileSinkConfig {
pub fn new() -> Self {
Self::default()
}
}
impl Default for FileSinkConfig {
fn default() -> Self {
Self {
path: get_log_directory_to_string(ENV_SINKS_FILE_PATH),
buffer_size: env::var(ENV_SINKS_FILE_BUFFER_SIZE)
.ok()
.and_then(|v| v.parse().ok())
.or(Some(DEFAULT_SINKS_FILE_BUFFER_SIZE)),
flush_interval_ms: env::var(ENV_SINKS_FILE_FLUSH_INTERVAL_MS)
.ok()
.and_then(|v| v.parse().ok())
.or(Some(DEFAULT_SINKS_FILE_FLUSH_INTERVAL_MS)),
flush_threshold: env::var(ENV_SINKS_FILE_FLUSH_THRESHOLD)
.ok()
.and_then(|v| v.parse().ok())
.or(Some(DEFAULT_SINKS_FILE_FLUSH_THRESHOLD)),
}
}
}
/// Sink configuration collection
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type")]
pub enum SinkConfig {
File(FileSinkConfig),
Kafka(KafkaSinkConfig),
Webhook(WebhookSinkConfig),
}
impl SinkConfig {
pub fn new() -> Self {
Self::File(FileSinkConfig::new())
}
}
impl Default for SinkConfig {
fn default() -> Self {
Self::new()
}
}
///Logger Configuration
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct LoggerConfig {
pub queue_capacity: Option<usize>,
}
impl LoggerConfig {
pub fn new() -> Self {
Self {
queue_capacity: env::var(ENV_AUDIT_LOGGER_QUEUE_CAPACITY)
.ok()
.and_then(|v| v.parse().ok())
.or(Some(DEFAULT_AUDIT_LOGGER_QUEUE_CAPACITY)),
}
}
}
impl Default for LoggerConfig {
fn default() -> Self {
Self::new()
}
}
/// Overall application configuration
/// Add observability, sinks, and logger configuration
/// Add observability configuration
///
/// Observability: OpenTelemetry configuration
/// Sinks: Kafka, Webhook, File sink configuration
/// Logger: Logger configuration
///
/// # Example
/// ```
@@ -316,8 +152,6 @@ impl Default for LoggerConfig {
#[derive(Debug, Deserialize, Clone)]
pub struct AppConfig {
pub observability: OtelConfig,
pub sinks: Vec<SinkConfig>,
pub logger: Option<LoggerConfig>,
}
impl AppConfig {
@@ -328,16 +162,12 @@ impl AppConfig {
pub fn new() -> Self {
Self {
observability: OtelConfig::default(),
sinks: vec![SinkConfig::default()],
logger: Some(LoggerConfig::default()),
}
}
pub fn new_with_endpoint(endpoint: Option<String>) -> Self {
Self {
observability: OtelConfig::extract_otel_config_from_env(endpoint),
sinks: vec![SinkConfig::new()],
logger: Some(LoggerConfig::new()),
}
}
}

View File

@@ -1,88 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::entry::ObjectVersion;
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
/// Args - defines the arguments for API operations
/// Args is used to define the arguments for API operations.
///
/// # Example
/// ```
/// use rustfs_obs::Args;
/// use std::collections::HashMap;
///
/// let args = Args::new()
/// .set_bucket(Some("my-bucket".to_string()))
/// .set_object(Some("my-object".to_string()))
/// .set_version_id(Some("123".to_string()))
/// .set_metadata(Some(HashMap::new()));
/// ```
#[derive(Debug, Clone, Serialize, Deserialize, Default, Eq, PartialEq)]
pub struct Args {
#[serde(rename = "bucket", skip_serializing_if = "Option::is_none")]
pub bucket: Option<String>,
#[serde(rename = "object", skip_serializing_if = "Option::is_none")]
pub object: Option<String>,
#[serde(rename = "versionId", skip_serializing_if = "Option::is_none")]
pub version_id: Option<String>,
#[serde(rename = "objects", skip_serializing_if = "Option::is_none")]
pub objects: Option<Vec<ObjectVersion>>,
#[serde(rename = "metadata", skip_serializing_if = "Option::is_none")]
pub metadata: Option<HashMap<String, String>>,
}
impl Args {
/// Create a new Args object
pub fn new() -> Self {
Args {
bucket: None,
object: None,
version_id: None,
objects: None,
metadata: None,
}
}
/// Set the bucket
pub fn set_bucket(mut self, bucket: Option<String>) -> Self {
self.bucket = bucket;
self
}
/// Set the object
pub fn set_object(mut self, object: Option<String>) -> Self {
self.object = object;
self
}
/// Set the version ID
pub fn set_version_id(mut self, version_id: Option<String>) -> Self {
self.version_id = version_id;
self
}
/// Set the objects
pub fn set_objects(mut self, objects: Option<Vec<ObjectVersion>>) -> Self {
self.objects = objects;
self
}
/// Set the metadata
pub fn set_metadata(mut self, metadata: Option<HashMap<String, String>>) -> Self {
self.metadata = metadata;
self
}
}

View File

@@ -1,467 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{BaseLogEntry, LogRecord, ObjectVersion};
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use std::collections::HashMap;
/// API details structure
/// ApiDetails is used to define the details of an API operation
///
/// The `ApiDetails` structure contains the following fields:
/// - `name` - the name of the API operation
/// - `bucket` - the bucket name
/// - `object` - the object name
/// - `objects` - the list of objects
/// - `status` - the status of the API operation
/// - `status_code` - the status code of the API operation
/// - `input_bytes` - the input bytes
/// - `output_bytes` - the output bytes
/// - `header_bytes` - the header bytes
/// - `time_to_first_byte` - the time to first byte
/// - `time_to_first_byte_in_ns` - the time to first byte in nanoseconds
/// - `time_to_response` - the time to response
/// - `time_to_response_in_ns` - the time to response in nanoseconds
///
/// The `ApiDetails` structure contains the following methods:
/// - `new` - create a new `ApiDetails` with default values
/// - `set_name` - set the name
/// - `set_bucket` - set the bucket
/// - `set_object` - set the object
/// - `set_objects` - set the objects
/// - `set_status` - set the status
/// - `set_status_code` - set the status code
/// - `set_input_bytes` - set the input bytes
/// - `set_output_bytes` - set the output bytes
/// - `set_header_bytes` - set the header bytes
/// - `set_time_to_first_byte` - set the time to first byte
/// - `set_time_to_first_byte_in_ns` - set the time to first byte in nanoseconds
/// - `set_time_to_response` - set the time to response
/// - `set_time_to_response_in_ns` - set the time to response in nanoseconds
///
/// # Example
/// ```
/// use rustfs_obs::ApiDetails;
/// use rustfs_obs::ObjectVersion;
///
/// let api = ApiDetails::new()
/// .set_name(Some("GET".to_string()))
/// .set_bucket(Some("my-bucket".to_string()))
/// .set_object(Some("my-object".to_string()))
/// .set_objects(vec![ObjectVersion::new_with_object_name("my-object".to_string())])
/// .set_status(Some("OK".to_string()))
/// .set_status_code(Some(200))
/// .set_input_bytes(100)
/// .set_output_bytes(200)
/// .set_header_bytes(Some(50))
/// .set_time_to_first_byte(Some("100ms".to_string()))
/// .set_time_to_first_byte_in_ns(Some("100000000ns".to_string()))
/// .set_time_to_response(Some("200ms".to_string()))
/// .set_time_to_response_in_ns(Some("200000000ns".to_string()));
/// ```
#[derive(Debug, Serialize, Deserialize, Clone, Default, PartialEq, Eq)]
pub struct ApiDetails {
#[serde(rename = "name", skip_serializing_if = "Option::is_none")]
pub name: Option<String>,
#[serde(rename = "bucket", skip_serializing_if = "Option::is_none")]
pub bucket: Option<String>,
#[serde(rename = "object", skip_serializing_if = "Option::is_none")]
pub object: Option<String>,
#[serde(rename = "objects", skip_serializing_if = "Vec::is_empty", default)]
pub objects: Vec<ObjectVersion>,
#[serde(rename = "status", skip_serializing_if = "Option::is_none")]
pub status: Option<String>,
#[serde(rename = "statusCode", skip_serializing_if = "Option::is_none")]
pub status_code: Option<i32>,
#[serde(rename = "rx")]
pub input_bytes: i64,
#[serde(rename = "tx")]
pub output_bytes: i64,
#[serde(rename = "txHeaders", skip_serializing_if = "Option::is_none")]
pub header_bytes: Option<i64>,
#[serde(rename = "timeToFirstByte", skip_serializing_if = "Option::is_none")]
pub time_to_first_byte: Option<String>,
#[serde(rename = "timeToFirstByteInNS", skip_serializing_if = "Option::is_none")]
pub time_to_first_byte_in_ns: Option<String>,
#[serde(rename = "timeToResponse", skip_serializing_if = "Option::is_none")]
pub time_to_response: Option<String>,
#[serde(rename = "timeToResponseInNS", skip_serializing_if = "Option::is_none")]
pub time_to_response_in_ns: Option<String>,
}
impl ApiDetails {
/// Create a new `ApiDetails` with default values
pub fn new() -> Self {
ApiDetails {
name: None,
bucket: None,
object: None,
objects: Vec::new(),
status: None,
status_code: None,
input_bytes: 0,
output_bytes: 0,
header_bytes: None,
time_to_first_byte: None,
time_to_first_byte_in_ns: None,
time_to_response: None,
time_to_response_in_ns: None,
}
}
/// Set the name
pub fn set_name(mut self, name: Option<String>) -> Self {
self.name = name;
self
}
/// Set the bucket
pub fn set_bucket(mut self, bucket: Option<String>) -> Self {
self.bucket = bucket;
self
}
/// Set the object
pub fn set_object(mut self, object: Option<String>) -> Self {
self.object = object;
self
}
/// Set the objects
pub fn set_objects(mut self, objects: Vec<ObjectVersion>) -> Self {
self.objects = objects;
self
}
/// Set the status
pub fn set_status(mut self, status: Option<String>) -> Self {
self.status = status;
self
}
/// Set the status code
pub fn set_status_code(mut self, status_code: Option<i32>) -> Self {
self.status_code = status_code;
self
}
/// Set the input bytes
pub fn set_input_bytes(mut self, input_bytes: i64) -> Self {
self.input_bytes = input_bytes;
self
}
/// Set the output bytes
pub fn set_output_bytes(mut self, output_bytes: i64) -> Self {
self.output_bytes = output_bytes;
self
}
/// Set the header bytes
pub fn set_header_bytes(mut self, header_bytes: Option<i64>) -> Self {
self.header_bytes = header_bytes;
self
}
/// Set the time to first byte
pub fn set_time_to_first_byte(mut self, time_to_first_byte: Option<String>) -> Self {
self.time_to_first_byte = time_to_first_byte;
self
}
/// Set the time to first byte in nanoseconds
pub fn set_time_to_first_byte_in_ns(mut self, time_to_first_byte_in_ns: Option<String>) -> Self {
self.time_to_first_byte_in_ns = time_to_first_byte_in_ns;
self
}
/// Set the time to response
pub fn set_time_to_response(mut self, time_to_response: Option<String>) -> Self {
self.time_to_response = time_to_response;
self
}
/// Set the time to response in nanoseconds
pub fn set_time_to_response_in_ns(mut self, time_to_response_in_ns: Option<String>) -> Self {
self.time_to_response_in_ns = time_to_response_in_ns;
self
}
}
/// Entry - audit entry logs
/// AuditLogEntry is used to define the structure of an audit log entry
///
/// The `AuditLogEntry` structure contains the following fields:
/// - `base` - the base log entry
/// - `version` - the version of the audit log entry
/// - `deployment_id` - the deployment ID
/// - `event` - the event
/// - `entry_type` - the type of audit message
/// - `api` - the API details
/// - `remote_host` - the remote host
/// - `user_agent` - the user agent
/// - `req_path` - the request path
/// - `req_host` - the request host
/// - `req_claims` - the request claims
/// - `req_query` - the request query
/// - `req_header` - the request header
/// - `resp_header` - the response header
/// - `access_key` - the access key
/// - `parent_user` - the parent user
/// - `error` - the error
///
/// The `AuditLogEntry` structure contains the following methods:
/// - `new` - create a new `AuditEntry` with default values
/// - `new_with_values` - create a new `AuditEntry` with version, time, event and api details
/// - `with_base` - set the base log entry
/// - `set_version` - set the version
/// - `set_deployment_id` - set the deployment ID
/// - `set_event` - set the event
/// - `set_entry_type` - set the entry type
/// - `set_api` - set the API details
/// - `set_remote_host` - set the remote host
/// - `set_user_agent` - set the user agent
/// - `set_req_path` - set the request path
/// - `set_req_host` - set the request host
/// - `set_req_claims` - set the request claims
/// - `set_req_query` - set the request query
/// - `set_req_header` - set the request header
/// - `set_resp_header` - set the response header
/// - `set_access_key` - set the access key
/// - `set_parent_user` - set the parent user
/// - `set_error` - set the error
///
/// # Example
/// ```
/// use rustfs_obs::AuditLogEntry;
/// use rustfs_obs::ApiDetails;
/// use std::collections::HashMap;
///
/// let entry = AuditLogEntry::new()
/// .set_version("1.0".to_string())
/// .set_deployment_id(Some("123".to_string()))
/// .set_event("event".to_string())
/// .set_entry_type(Some("type".to_string()))
/// .set_api(ApiDetails::new())
/// .set_remote_host(Some("remote-host".to_string()))
/// .set_user_agent(Some("user-agent".to_string()))
/// .set_req_path(Some("req-path".to_string()))
/// .set_req_host(Some("req-host".to_string()))
/// .set_req_claims(Some(HashMap::new()))
/// .set_req_query(Some(HashMap::new()))
/// .set_req_header(Some(HashMap::new()))
/// .set_resp_header(Some(HashMap::new()))
/// .set_access_key(Some("access-key".to_string()))
/// .set_parent_user(Some("parent-user".to_string()))
/// .set_error(Some("error".to_string()));
#[derive(Debug, Serialize, Deserialize, Clone, Default)]
pub struct AuditLogEntry {
#[serde(flatten)]
pub base: BaseLogEntry,
pub version: String,
#[serde(rename = "deploymentid", skip_serializing_if = "Option::is_none")]
pub deployment_id: Option<String>,
pub event: String,
// Class of audit message - S3, admin ops, bucket management
#[serde(rename = "type", skip_serializing_if = "Option::is_none")]
pub entry_type: Option<String>,
pub api: ApiDetails,
#[serde(rename = "remotehost", skip_serializing_if = "Option::is_none")]
pub remote_host: Option<String>,
#[serde(rename = "userAgent", skip_serializing_if = "Option::is_none")]
pub user_agent: Option<String>,
#[serde(rename = "requestPath", skip_serializing_if = "Option::is_none")]
pub req_path: Option<String>,
#[serde(rename = "requestHost", skip_serializing_if = "Option::is_none")]
pub req_host: Option<String>,
#[serde(rename = "requestClaims", skip_serializing_if = "Option::is_none")]
pub req_claims: Option<HashMap<String, Value>>,
#[serde(rename = "requestQuery", skip_serializing_if = "Option::is_none")]
pub req_query: Option<HashMap<String, String>>,
#[serde(rename = "requestHeader", skip_serializing_if = "Option::is_none")]
pub req_header: Option<HashMap<String, String>>,
#[serde(rename = "responseHeader", skip_serializing_if = "Option::is_none")]
pub resp_header: Option<HashMap<String, String>>,
#[serde(rename = "accessKey", skip_serializing_if = "Option::is_none")]
pub access_key: Option<String>,
#[serde(rename = "parentUser", skip_serializing_if = "Option::is_none")]
pub parent_user: Option<String>,
#[serde(rename = "error", skip_serializing_if = "Option::is_none")]
pub error: Option<String>,
}
impl AuditLogEntry {
/// Create a new `AuditEntry` with default values
pub fn new() -> Self {
AuditLogEntry {
base: BaseLogEntry::new(),
version: String::new(),
deployment_id: None,
event: String::new(),
entry_type: None,
api: ApiDetails::new(),
remote_host: None,
user_agent: None,
req_path: None,
req_host: None,
req_claims: None,
req_query: None,
req_header: None,
resp_header: None,
access_key: None,
parent_user: None,
error: None,
}
}
/// Create a new `AuditEntry` with version, time, event and api details
pub fn new_with_values(version: String, time: DateTime<Utc>, event: String, api: ApiDetails) -> Self {
let mut base = BaseLogEntry::new();
base.timestamp = time;
AuditLogEntry {
base,
version,
deployment_id: None,
event,
entry_type: None,
api,
remote_host: None,
user_agent: None,
req_path: None,
req_host: None,
req_claims: None,
req_query: None,
req_header: None,
resp_header: None,
access_key: None,
parent_user: None,
error: None,
}
}
/// Set the base log entry
pub fn with_base(mut self, base: BaseLogEntry) -> Self {
self.base = base;
self
}
/// Set the version
pub fn set_version(mut self, version: String) -> Self {
self.version = version;
self
}
/// Set the deployment ID
pub fn set_deployment_id(mut self, deployment_id: Option<String>) -> Self {
self.deployment_id = deployment_id;
self
}
/// Set the event
pub fn set_event(mut self, event: String) -> Self {
self.event = event;
self
}
/// Set the entry type
pub fn set_entry_type(mut self, entry_type: Option<String>) -> Self {
self.entry_type = entry_type;
self
}
/// Set the API details
pub fn set_api(mut self, api: ApiDetails) -> Self {
self.api = api;
self
}
/// Set the remote host
pub fn set_remote_host(mut self, remote_host: Option<String>) -> Self {
self.remote_host = remote_host;
self
}
/// Set the user agent
pub fn set_user_agent(mut self, user_agent: Option<String>) -> Self {
self.user_agent = user_agent;
self
}
/// Set the request path
pub fn set_req_path(mut self, req_path: Option<String>) -> Self {
self.req_path = req_path;
self
}
/// Set the request host
pub fn set_req_host(mut self, req_host: Option<String>) -> Self {
self.req_host = req_host;
self
}
/// Set the request claims
pub fn set_req_claims(mut self, req_claims: Option<HashMap<String, Value>>) -> Self {
self.req_claims = req_claims;
self
}
/// Set the request query
pub fn set_req_query(mut self, req_query: Option<HashMap<String, String>>) -> Self {
self.req_query = req_query;
self
}
/// Set the request header
pub fn set_req_header(mut self, req_header: Option<HashMap<String, String>>) -> Self {
self.req_header = req_header;
self
}
/// Set the response header
pub fn set_resp_header(mut self, resp_header: Option<HashMap<String, String>>) -> Self {
self.resp_header = resp_header;
self
}
/// Set the access key
pub fn set_access_key(mut self, access_key: Option<String>) -> Self {
self.access_key = access_key;
self
}
/// Set the parent user
pub fn set_parent_user(mut self, parent_user: Option<String>) -> Self {
self.parent_user = parent_user;
self
}
/// Set the error
pub fn set_error(mut self, error: Option<String>) -> Self {
self.error = error;
self
}
}
impl LogRecord for AuditLogEntry {
fn to_json(&self) -> String {
serde_json::to_string(self).unwrap_or_else(|_| String::from("{}"))
}
fn get_timestamp(&self) -> DateTime<Utc> {
self.base.timestamp
}
}

View File

@@ -1,106 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use std::collections::HashMap;
/// Base log entry structure shared by all log types
/// This structure is used to serialize log entries to JSON
/// and send them to the log sinks
/// This structure is also used to deserialize log entries from JSON
/// This structure is also used to store log entries in the database
/// This structure is also used to query log entries from the database
///
/// The `BaseLogEntry` structure contains the following fields:
/// - `timestamp` - the timestamp of the log entry
/// - `request_id` - the request ID of the log entry
/// - `message` - the message of the log entry
/// - `tags` - the tags of the log entry
///
/// The `BaseLogEntry` structure contains the following methods:
/// - `new` - create a new `BaseLogEntry` with default values
/// - `message` - set the message
/// - `request_id` - set the request ID
/// - `tags` - set the tags
/// - `timestamp` - set the timestamp
///
/// # Example
/// ```
/// use rustfs_obs::BaseLogEntry;
/// use chrono::{DateTime, Utc};
/// use std::collections::HashMap;
///
/// let timestamp = Utc::now();
/// let request = Some("req-123".to_string());
/// let message = Some("This is a log message".to_string());
/// let tags = Some(HashMap::new());
///
/// let entry = BaseLogEntry::new()
/// .timestamp(timestamp)
/// .request_id(request)
/// .message(message)
/// .tags(tags);
/// ```
#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq, Default)]
pub struct BaseLogEntry {
#[serde(rename = "time")]
pub timestamp: DateTime<Utc>,
#[serde(rename = "requestID", skip_serializing_if = "Option::is_none")]
pub request_id: Option<String>,
#[serde(rename = "message", skip_serializing_if = "Option::is_none")]
pub message: Option<String>,
#[serde(rename = "tags", skip_serializing_if = "Option::is_none")]
pub tags: Option<HashMap<String, Value>>,
}
impl BaseLogEntry {
/// Create a new BaseLogEntry with default values
pub fn new() -> Self {
BaseLogEntry {
timestamp: Utc::now(),
request_id: None,
message: None,
tags: None,
}
}
/// Set the message
pub fn message(mut self, message: Option<String>) -> Self {
self.message = message;
self
}
/// Set the request ID
pub fn request_id(mut self, request_id: Option<String>) -> Self {
self.request_id = request_id;
self
}
/// Set the tags
pub fn tags(mut self, tags: Option<HashMap<String, Value>>) -> Self {
self.tags = tags;
self
}
/// Set the timestamp
pub fn timestamp(mut self, timestamp: DateTime<Utc>) -> Self {
self.timestamp = timestamp;
self
}
}

View File

@@ -1,158 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) mod args;
pub(crate) mod audit;
pub(crate) mod base;
pub(crate) mod unified;
use serde::de::Error;
use serde::{Deserialize, Deserializer, Serialize, Serializer};
use tracing_core::Level;
/// ObjectVersion is used across multiple modules
#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)]
pub struct ObjectVersion {
#[serde(rename = "name")]
pub object_name: String,
#[serde(rename = "versionId", skip_serializing_if = "Option::is_none")]
pub version_id: Option<String>,
}
impl ObjectVersion {
/// Create a new ObjectVersion object
pub fn new() -> Self {
ObjectVersion {
object_name: String::new(),
version_id: None,
}
}
/// Create a new ObjectVersion with object name
pub fn new_with_object_name(object_name: String) -> Self {
ObjectVersion {
object_name,
version_id: None,
}
}
/// Set the object name
pub fn set_object_name(mut self, object_name: String) -> Self {
self.object_name = object_name;
self
}
/// Set the version ID
pub fn set_version_id(mut self, version_id: Option<String>) -> Self {
self.version_id = version_id;
self
}
}
impl Default for ObjectVersion {
fn default() -> Self {
Self::new()
}
}
/// Log kind/level enum
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
pub enum LogKind {
#[serde(rename = "INFO")]
#[default]
Info,
#[serde(rename = "WARNING")]
Warning,
#[serde(rename = "ERROR")]
Error,
#[serde(rename = "FATAL")]
Fatal,
}
/// Trait for types that can be serialized to JSON and have a timestamp
/// This trait is used by `ServerLogEntry` to convert the log entry to JSON
/// and get the timestamp of the log entry
/// This trait is implemented by `ServerLogEntry`
///
/// # Example
/// ```
/// use rustfs_obs::LogRecord;
/// use chrono::{DateTime, Utc};
/// use rustfs_obs::ServerLogEntry;
/// use tracing_core::Level;
///
/// let log_entry = ServerLogEntry::new(Level::INFO, "api_handler".to_string());
/// let json = log_entry.to_json();
/// let timestamp = log_entry.get_timestamp();
/// ```
pub trait LogRecord {
fn to_json(&self) -> String;
fn get_timestamp(&self) -> chrono::DateTime<chrono::Utc>;
}
/// Wrapper for `tracing_core::Level` to implement `Serialize` and `Deserialize`
/// for `ServerLogEntry`
/// This is necessary because `tracing_core::Level` does not implement `Serialize`
/// and `Deserialize`
/// This is a workaround to allow `ServerLogEntry` to be serialized and deserialized
/// using `serde`
///
/// # Example
/// ```
/// use rustfs_obs::SerializableLevel;
/// use tracing_core::Level;
///
/// let level = Level::INFO;
/// let serializable_level = SerializableLevel::from(level);
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SerializableLevel(pub Level);
impl From<Level> for SerializableLevel {
fn from(level: Level) -> Self {
SerializableLevel(level)
}
}
impl From<SerializableLevel> for Level {
fn from(serializable_level: SerializableLevel) -> Self {
serializable_level.0
}
}
impl Serialize for SerializableLevel {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
serializer.serialize_str(self.0.as_str())
}
}
impl<'de> Deserialize<'de> for SerializableLevel {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
let s = String::deserialize(deserializer)?;
match s.as_str() {
"TRACE" => Ok(SerializableLevel(Level::TRACE)),
"DEBUG" => Ok(SerializableLevel(Level::DEBUG)),
"INFO" => Ok(SerializableLevel(Level::INFO)),
"WARN" => Ok(SerializableLevel(Level::WARN)),
"ERROR" => Ok(SerializableLevel(Level::ERROR)),
_ => Err(D::Error::custom("unknown log level")),
}
}
}

View File

@@ -1,301 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{AuditLogEntry, BaseLogEntry, LogKind, LogRecord, SerializableLevel};
use chrono::{DateTime, Utc};
use serde::{Deserialize, Serialize};
use tracing_core::Level;
/// Server log entry with structured fields
/// ServerLogEntry is used to log structured log entries from the server
///
/// The `ServerLogEntry` structure contains the following fields:
/// - `base` - the base log entry
/// - `level` - the log level
/// - `source` - the source of the log entry
/// - `user_id` - the user ID
/// - `fields` - the structured fields of the log entry
///
/// The `ServerLogEntry` structure contains the following methods:
/// - `new` - create a new `ServerLogEntry` with specified level and source
/// - `with_base` - set the base log entry
/// - `user_id` - set the user ID
/// - `fields` - set the fields
/// - `add_field` - add a field
///
/// # Example
/// ```
/// use rustfs_obs::ServerLogEntry;
/// use tracing_core::Level;
///
/// let entry = ServerLogEntry::new(Level::INFO, "test_module".to_string())
/// .user_id(Some("user-456".to_string()))
/// .add_field("operation".to_string(), "login".to_string());
/// ```
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct ServerLogEntry {
#[serde(flatten)]
pub base: BaseLogEntry,
pub level: SerializableLevel,
pub source: String,
#[serde(rename = "userId", skip_serializing_if = "Option::is_none")]
pub user_id: Option<String>,
#[serde(skip_serializing_if = "Vec::is_empty", default)]
pub fields: Vec<(String, String)>,
}
impl ServerLogEntry {
/// Create a new ServerLogEntry with specified level and source
pub fn new(level: Level, source: String) -> Self {
ServerLogEntry {
base: BaseLogEntry::new(),
level: SerializableLevel(level),
source,
user_id: None,
fields: Vec::new(),
}
}
/// Set the base log entry
pub fn with_base(mut self, base: BaseLogEntry) -> Self {
self.base = base;
self
}
/// Set the user ID
pub fn user_id(mut self, user_id: Option<String>) -> Self {
self.user_id = user_id;
self
}
/// Set fields
pub fn fields(mut self, fields: Vec<(String, String)>) -> Self {
self.fields = fields;
self
}
/// Add a field
pub fn add_field(mut self, key: String, value: String) -> Self {
self.fields.push((key, value));
self
}
}
impl LogRecord for ServerLogEntry {
fn to_json(&self) -> String {
serde_json::to_string(self).unwrap_or_else(|_| String::from("{}"))
}
fn get_timestamp(&self) -> DateTime<Utc> {
self.base.timestamp
}
}
/// Console log entry structure
/// ConsoleLogEntry is used to log console log entries
/// The `ConsoleLogEntry` structure contains the following fields:
/// - `base` - the base log entry
/// - `level` - the log level
/// - `console_msg` - the console message
/// - `node_name` - the node name
/// - `err` - the error message
///
/// The `ConsoleLogEntry` structure contains the following methods:
/// - `new` - create a new `ConsoleLogEntry`
/// - `new_with_console_msg` - create a new `ConsoleLogEntry` with console message and node name
/// - `with_base` - set the base log entry
/// - `set_level` - set the log level
/// - `set_node_name` - set the node name
/// - `set_console_msg` - set the console message
/// - `set_err` - set the error message
///
/// # Example
/// ```
/// use rustfs_obs::ConsoleLogEntry;
///
/// let entry = ConsoleLogEntry::new_with_console_msg("Test message".to_string(), "node-123".to_string());
/// ```
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ConsoleLogEntry {
#[serde(flatten)]
pub base: BaseLogEntry,
pub level: LogKind,
pub console_msg: String,
pub node_name: String,
#[serde(skip)]
pub err: Option<String>,
}
impl ConsoleLogEntry {
/// Create a new ConsoleLogEntry
pub fn new() -> Self {
ConsoleLogEntry {
base: BaseLogEntry::new(),
level: LogKind::Info,
console_msg: String::new(),
node_name: String::new(),
err: None,
}
}
/// Create a new ConsoleLogEntry with console message and node name
pub fn new_with_console_msg(console_msg: String, node_name: String) -> Self {
ConsoleLogEntry {
base: BaseLogEntry::new(),
level: LogKind::Info,
console_msg,
node_name,
err: None,
}
}
/// Set the base log entry
pub fn with_base(mut self, base: BaseLogEntry) -> Self {
self.base = base;
self
}
/// Set the log level
pub fn set_level(mut self, level: LogKind) -> Self {
self.level = level;
self
}
/// Set the node name
pub fn set_node_name(mut self, node_name: String) -> Self {
self.node_name = node_name;
self
}
/// Set the console message
pub fn set_console_msg(mut self, console_msg: String) -> Self {
self.console_msg = console_msg;
self
}
/// Set the error message
pub fn set_err(mut self, err: Option<String>) -> Self {
self.err = err;
self
}
}
impl Default for ConsoleLogEntry {
fn default() -> Self {
Self::new()
}
}
impl LogRecord for ConsoleLogEntry {
fn to_json(&self) -> String {
serde_json::to_string(self).unwrap_or_else(|_| String::from("{}"))
}
fn get_timestamp(&self) -> DateTime<Utc> {
self.base.timestamp
}
}
/// Unified log entry type
/// UnifiedLogEntry is used to log different types of log entries
///
/// The `UnifiedLogEntry` enum contains the following variants:
/// - `Server` - a server log entry
/// - `Audit` - an audit log entry
/// - `Console` - a console log entry
///
/// The `UnifiedLogEntry` enum contains the following methods:
/// - `to_json` - convert the log entry to JSON
/// - `get_timestamp` - get the timestamp of the log entry
///
/// # Example
/// ```
/// use rustfs_obs::{UnifiedLogEntry, ServerLogEntry};
/// use tracing_core::Level;
///
/// let server_entry = ServerLogEntry::new(Level::INFO, "test_module".to_string());
/// let unified = UnifiedLogEntry::Server(server_entry);
/// ```
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(tag = "type")]
pub enum UnifiedLogEntry {
#[serde(rename = "server")]
Server(ServerLogEntry),
#[serde(rename = "audit")]
Audit(Box<AuditLogEntry>),
#[serde(rename = "console")]
Console(ConsoleLogEntry),
}
impl LogRecord for UnifiedLogEntry {
fn to_json(&self) -> String {
match self {
UnifiedLogEntry::Server(entry) => entry.to_json(),
UnifiedLogEntry::Audit(entry) => entry.to_json(),
UnifiedLogEntry::Console(entry) => entry.to_json(),
}
}
fn get_timestamp(&self) -> DateTime<Utc> {
match self {
UnifiedLogEntry::Server(entry) => entry.get_timestamp(),
UnifiedLogEntry::Audit(entry) => entry.get_timestamp(),
UnifiedLogEntry::Console(entry) => entry.get_timestamp(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_base_log_entry() {
let base = BaseLogEntry::new()
.request_id(Some("req-123".to_string()))
.message(Some("Test message".to_string()));
assert_eq!(base.request_id, Some("req-123".to_string()));
assert_eq!(base.message, Some("Test message".to_string()));
}
#[test]
fn test_server_log_entry() {
let entry = ServerLogEntry::new(Level::INFO, "test_module".to_string())
.user_id(Some("user-456".to_string()))
.add_field("operation".to_string(), "login".to_string());
assert_eq!(entry.level.0, Level::INFO);
assert_eq!(entry.source, "test_module");
assert_eq!(entry.user_id, Some("user-456".to_string()));
assert_eq!(entry.fields.len(), 1);
assert_eq!(entry.fields[0], ("operation".to_string(), "login".to_string()));
}
#[test]
fn test_unified_log_entry_json() {
let server_entry = ServerLogEntry::new(Level::INFO, "test_source".to_string());
let unified = UnifiedLogEntry::Server(server_entry);
let json = unified.to_json();
assert!(json.contains("test_source"));
}
}

View File

@@ -12,9 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::logger::InitLogStatus;
use crate::AppConfig;
use crate::telemetry::{OtelGuard, init_telemetry};
use crate::{AppConfig, Logger, get_global_logger, init_global_logger};
use opentelemetry::metrics::Meter;
use rustfs_config::APP_NAME;
use std::sync::{Arc, Mutex};
use tokio::sync::{OnceCell, SetError};
use tracing::{error, info};
@@ -22,6 +23,23 @@ use tracing::{error, info};
/// Global guard for OpenTelemetry tracing
static GLOBAL_GUARD: OnceCell<Arc<Mutex<OtelGuard>>> = OnceCell::const_new();
/// Flag indicating if observability is enabled
pub(crate) static IS_OBSERVABILITY_ENABLED: OnceCell<bool> = OnceCell::const_new();
/// Name of the observability meter
pub(crate) static OBSERVABILITY_METER_NAME: OnceCell<String> = OnceCell::const_new();
/// Check whether Observability is enabled
pub fn is_observability_enabled() -> bool {
IS_OBSERVABILITY_ENABLED.get().copied().unwrap_or(false)
}
/// Get the global meter for observability
pub fn global_meter() -> Meter {
let meter_name = OBSERVABILITY_METER_NAME.get().map(|s| s.as_str()).unwrap_or(APP_NAME);
opentelemetry::global::meter(meter_name)
}
/// Error type for global guard operations
#[derive(Debug, thiserror::Error)]
pub enum GlobalError {
@@ -61,46 +79,14 @@ pub enum GlobalError {
///
/// # #[tokio::main]
/// # async fn main() {
/// let (logger, guard) = init_obs(None).await;
/// let guard = init_obs(None).await;
/// # }
/// ```
pub async fn init_obs(endpoint: Option<String>) -> (Arc<tokio::sync::Mutex<Logger>>, OtelGuard) {
pub async fn init_obs(endpoint: Option<String>) -> OtelGuard {
// Load the configuration file
let config = AppConfig::new_with_endpoint(endpoint);
let guard = init_telemetry(&config.observability);
let logger = init_global_logger(&config).await;
let obs_config = config.observability.clone();
tokio::spawn(async move {
let result = InitLogStatus::init_start_log(&obs_config).await;
match result {
Ok(_) => {
info!("Logger initialized successfully");
}
Err(e) => {
error!("Failed to initialize logger: {}", e);
}
}
});
(logger, guard)
}
/// Get the global logger instance
/// This function returns a reference to the global logger instance.
///
/// # Returns
/// A reference to the global logger instance
///
/// # Example
/// ```no_run
/// use rustfs_obs::get_logger;
///
/// let logger = get_logger();
/// ```
pub fn get_logger() -> &'static Arc<tokio::sync::Mutex<Logger>> {
get_global_logger()
init_telemetry(&config.observability)
}
/// Set the global guard for OpenTelemetry
@@ -117,7 +103,7 @@ pub fn get_logger() -> &'static Arc<tokio::sync::Mutex<Logger>> {
/// use rustfs_obs::{ init_obs, set_global_guard};
///
/// async fn init() -> Result<(), Box<dyn std::error::Error>> {
/// let (_, guard) = init_obs(None).await;
/// let guard = init_obs(None).await;
/// set_global_guard(guard)?;
/// Ok(())
/// }

View File

@@ -18,10 +18,7 @@
//!
//! ## feature mark
//!
//! - `file`: enable file logging enabled by default
//! - `gpu`: gpu monitoring function
//! - `kafka`: enable kafka metric output
//! - `webhook`: enable webhook notifications
//! - `full`: includes all functions
//!
//! to enable gpu monitoring add in cargo toml
@@ -41,27 +38,15 @@
///
/// # #[tokio::main]
/// # async fn main() {
/// let (logger, guard) = init_obs(None).await;
/// # let guard = init_obs(None).await;
/// # }
/// ```
mod config;
mod entry;
mod global;
mod logger;
mod metrics;
mod sinks;
mod system;
mod telemetry;
mod worker;
pub use config::{AppConfig, LoggerConfig, OtelConfig, SinkConfig};
pub use entry::args::Args;
pub use entry::audit::{ApiDetails, AuditLogEntry};
pub use entry::base::BaseLogEntry;
pub use entry::unified::{ConsoleLogEntry, ServerLogEntry, UnifiedLogEntry};
pub use entry::{LogKind, LogRecord, ObjectVersion, SerializableLevel};
pub use config::AppConfig;
pub use global::*;
pub use logger::Logger;
pub use logger::{get_global_logger, init_global_logger, start_logger};
pub use logger::{log_debug, log_error, log_info, log_trace, log_warn, log_with_context};
pub use system::SystemObserver;

View File

@@ -1,490 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::sinks::Sink;
use crate::{
AppConfig, AuditLogEntry, BaseLogEntry, ConsoleLogEntry, GlobalError, OtelConfig, ServerLogEntry, UnifiedLogEntry, sinks,
};
use rustfs_config::{APP_NAME, ENVIRONMENT, SERVICE_VERSION};
use std::sync::Arc;
use std::time::SystemTime;
use tokio::sync::mpsc::{self, Receiver, Sender};
use tokio::sync::{Mutex, OnceCell};
use tracing_core::Level;
// Add the global instance at the module level
static GLOBAL_LOGGER: OnceCell<Arc<Mutex<Logger>>> = OnceCell::const_new();
/// Server log processor
#[derive(Debug)]
pub struct Logger {
sender: Sender<UnifiedLogEntry>, // Log sending channel
queue_capacity: usize,
}
impl Logger {
/// Create a new Logger instance
/// Returns Logger and corresponding Receiver
pub fn new(config: &AppConfig) -> (Self, Receiver<UnifiedLogEntry>) {
// Get queue capacity from configuration, or use default values 10000
let queue_capacity = config.logger.as_ref().and_then(|l| l.queue_capacity).unwrap_or(10000);
let (sender, receiver) = mpsc::channel(queue_capacity);
(Logger { sender, queue_capacity }, receiver)
}
/// get the queue capacity
/// This function returns the queue capacity.
/// # Returns
/// The queue capacity
/// # Example
/// ```
/// use rustfs_obs::Logger;
/// async fn example(logger: &Logger) {
/// let _ = logger.get_queue_capacity();
/// }
/// ```
pub fn get_queue_capacity(&self) -> usize {
self.queue_capacity
}
/// Log a server entry
#[tracing::instrument(skip(self), fields(log_source = "logger_server"))]
pub async fn log_server_entry(&self, entry: ServerLogEntry) -> Result<(), GlobalError> {
self.log_entry(UnifiedLogEntry::Server(entry)).await
}
/// Log an audit entry
#[tracing::instrument(skip(self), fields(log_source = "logger_audit"))]
pub async fn log_audit_entry(&self, entry: AuditLogEntry) -> Result<(), GlobalError> {
self.log_entry(UnifiedLogEntry::Audit(Box::new(entry))).await
}
/// Log a console entry
#[tracing::instrument(skip(self), fields(log_source = "logger_console"))]
pub async fn log_console_entry(&self, entry: ConsoleLogEntry) -> Result<(), GlobalError> {
self.log_entry(UnifiedLogEntry::Console(entry)).await
}
/// Asynchronous logging of unified log entries
#[tracing::instrument(skip_all, fields(log_source = "logger"))]
pub async fn log_entry(&self, entry: UnifiedLogEntry) -> Result<(), GlobalError> {
// Extract information for tracing based on entry type
match &entry {
UnifiedLogEntry::Server(server) => {
tracing::Span::current()
.record("log_level", server.level.0.as_str())
.record("log_message", server.base.message.as_deref().unwrap_or("log message not set"))
.record("source", &server.source);
// Generate tracing event based on log level
match server.level.0 {
Level::ERROR => {
tracing::error!(target: "server_logs", message = %server.base.message.as_deref().unwrap_or(""));
}
Level::WARN => {
tracing::warn!(target: "server_logs", message = %server.base.message.as_deref().unwrap_or(""));
}
Level::INFO => {
tracing::info!(target: "server_logs", message = %server.base.message.as_deref().unwrap_or(""));
}
Level::DEBUG => {
tracing::debug!(target: "server_logs", message = %server.base.message.as_deref().unwrap_or(""));
}
Level::TRACE => {
tracing::trace!(target: "server_logs", message = %server.base.message.as_deref().unwrap_or(""));
}
}
}
UnifiedLogEntry::Audit(audit) => {
tracing::info!(
target: "audit_logs",
event = %audit.event,
api = %audit.api.name.as_deref().unwrap_or("unknown"),
message = %audit.base.message.as_deref().unwrap_or("")
);
}
UnifiedLogEntry::Console(console) => {
let level_str = match console.level {
crate::LogKind::Info => "INFO",
crate::LogKind::Warning => "WARN",
crate::LogKind::Error => "ERROR",
crate::LogKind::Fatal => "FATAL",
};
tracing::info!(
target: "console_logs",
level = %level_str,
node = %console.node_name,
message = %console.console_msg
);
}
}
// Send logs to async queue with improved error handling
match self.sender.try_send(entry) {
Ok(_) => Ok(()),
Err(mpsc::error::TrySendError::Full(entry)) => {
// Processing strategy when queue is full
tracing::warn!("Log queue full, applying backpressure");
match tokio::time::timeout(std::time::Duration::from_millis(500), self.sender.send(entry)).await {
Ok(Ok(_)) => Ok(()),
Ok(Err(_)) => Err(GlobalError::SendFailed("Channel closed")),
Err(_) => Err(GlobalError::Timeout("Queue backpressure timeout")),
}
}
Err(mpsc::error::TrySendError::Closed(_)) => Err(GlobalError::SendFailed("Logger channel closed")),
}
}
/// Write log with context information
/// This function writes log messages with context information.
///
/// # Parameters
/// - `message`: Message to be logged
/// - `source`: Source of the log
/// - `request_id`: Request ID
/// - `user_id`: User ID
/// - `fields`: Additional fields
///
/// # Returns
/// Result indicating whether the operation was successful
///
/// # Example
/// ```
/// use tracing_core::Level;
/// use rustfs_obs::Logger;
///
/// async fn example(logger: &Logger) {
/// let _ = logger.write_with_context("This is an information message", "example",Level::INFO, Some("req-12345".to_string()), Some("user-6789".to_string()), vec![("endpoint".to_string(), "/api/v1/data".to_string())]).await;
/// }
pub async fn write_with_context(
&self,
message: &str,
source: &str,
level: Level,
request_id: Option<String>,
user_id: Option<String>,
fields: Vec<(String, String)>,
) -> Result<(), GlobalError> {
let base = BaseLogEntry::new().message(Some(message.to_string())).request_id(request_id);
let server_entry = ServerLogEntry::new(level, source.to_string())
.user_id(user_id)
.fields(fields)
.with_base(base);
self.log_server_entry(server_entry).await
}
/// Write log
/// This function writes log messages.
/// # Parameters
/// - `message`: Message to be logged
/// - `source`: Source of the log
/// - `level`: Log level
///
/// # Returns
/// Result indicating whether the operation was successful
///
/// # Example
/// ```
/// use rustfs_obs::Logger;
/// use tracing_core::Level;
///
/// async fn example(logger: &Logger) {
/// let _ = logger.write("This is an information message", "example", Level::INFO).await;
/// }
/// ```
pub async fn write(&self, message: &str, source: &str, level: Level) -> Result<(), GlobalError> {
self.write_with_context(message, source, level, None, None, Vec::new()).await
}
/// Shutdown the logger
/// This function shuts down the logger.
///
/// # Returns
/// Result indicating whether the operation was successful
///
/// # Example
/// ```
/// use rustfs_obs::Logger;
///
/// async fn example(logger: Logger) {
/// let _ = logger.shutdown().await;
/// }
/// ```
pub async fn shutdown(self) -> Result<(), GlobalError> {
drop(self.sender); //Close the sending end so that the receiver knows that there is no new message
Ok(())
}
}
/// Start the log module
/// This function starts the log module.
/// It initializes the logger and starts the worker to process logs.
/// # Parameters
/// - `config`: Configuration information
/// - `sinks`: A vector of Sink instances
/// # Returns
/// The global logger instance
/// # Example
/// ```no_run
/// use rustfs_obs::{AppConfig, start_logger};
///
/// let config = AppConfig::default();
/// let sinks = vec![];
/// let logger = start_logger(&config, sinks);
/// ```
pub fn start_logger(config: &AppConfig, sinks: Vec<Arc<dyn Sink>>) -> Logger {
let (logger, receiver) = Logger::new(config);
tokio::spawn(crate::worker::start_worker(receiver, sinks));
logger
}
/// Initialize the global logger instance
/// This function initializes the global logger instance and returns a reference to it.
/// If the logger has been initialized before, it will return the existing logger instance.
///
/// # Parameters
/// - `config`: Configuration information
/// - `sinks`: A vector of Sink instances
///
/// # Returns
/// A reference to the global logger instance
///
/// # Example
/// ```
/// use rustfs_obs::{AppConfig,init_global_logger};
///
/// let config = AppConfig::default();
/// let logger = init_global_logger(&config);
/// ```
pub async fn init_global_logger(config: &AppConfig) -> Arc<Mutex<Logger>> {
let sinks = sinks::create_sinks(config).await;
let logger = Arc::new(Mutex::new(start_logger(config, sinks)));
GLOBAL_LOGGER.set(logger.clone()).expect("Logger already initialized");
logger
}
/// Get the global logger instance
///
/// This function returns a reference to the global logger instance.
///
/// # Returns
/// A reference to the global logger instance
///
/// # Example
/// ```no_run
/// use rustfs_obs::get_global_logger;
///
/// let logger = get_global_logger();
/// ```
pub fn get_global_logger() -> &'static Arc<Mutex<Logger>> {
GLOBAL_LOGGER.get().expect("Logger not initialized")
}
/// Log information
/// This function logs information messages.
///
/// # Parameters
/// - `message`: Message to be logged
/// - `source`: Source of the log
///
/// # Returns
/// Result indicating whether the operation was successful
///
/// # Example
/// ```no_run
/// use rustfs_obs::log_info;
///
/// async fn example() {
/// let _ = log_info("This is an information message", "example").await;
/// }
/// ```
pub async fn log_info(message: &str, source: &str) -> Result<(), GlobalError> {
get_global_logger().lock().await.write(message, source, Level::INFO).await
}
/// Log error
/// This function logs error messages.
/// # Parameters
/// - `message`: Message to be logged
/// - `source`: Source of the log
/// # Returns
/// Result indicating whether the operation was successful
/// # Example
/// ```no_run
/// use rustfs_obs::log_error;
///
/// async fn example() {
/// let _ = log_error("This is an error message", "example").await;
/// }
pub async fn log_error(message: &str, source: &str) -> Result<(), GlobalError> {
get_global_logger().lock().await.write(message, source, Level::ERROR).await
}
/// Log warning
/// This function logs warning messages.
/// # Parameters
/// - `message`: Message to be logged
/// - `source`: Source of the log
/// # Returns
/// Result indicating whether the operation was successful
///
/// # Example
/// ```no_run
/// use rustfs_obs::log_warn;
///
/// async fn example() {
/// let _ = log_warn("This is a warning message", "example").await;
/// }
/// ```
pub async fn log_warn(message: &str, source: &str) -> Result<(), GlobalError> {
get_global_logger().lock().await.write(message, source, Level::WARN).await
}
/// Log debug
/// This function logs debug messages.
/// # Parameters
/// - `message`: Message to be logged
/// - `source`: Source of the log
/// # Returns
/// Result indicating whether the operation was successful
///
/// # Example
/// ```no_run
/// use rustfs_obs::log_debug;
///
/// async fn example() {
/// let _ = log_debug("This is a debug message", "example").await;
/// }
/// ```
pub async fn log_debug(message: &str, source: &str) -> Result<(), GlobalError> {
get_global_logger().lock().await.write(message, source, Level::DEBUG).await
}
/// Log trace
/// This function logs trace messages.
/// # Parameters
/// - `message`: Message to be logged
/// - `source`: Source of the log
///
/// # Returns
/// Result indicating whether the operation was successful
///
/// # Example
/// ```no_run
/// use rustfs_obs::log_trace;
///
/// async fn example() {
/// let _ = log_trace("This is a trace message", "example").await;
/// }
/// ```
pub async fn log_trace(message: &str, source: &str) -> Result<(), GlobalError> {
get_global_logger().lock().await.write(message, source, Level::TRACE).await
}
/// Log with context information
/// This function logs messages with context information.
/// # Parameters
/// - `message`: Message to be logged
/// - `source`: Source of the log
/// - `level`: Log level
/// - `request_id`: Request ID
/// - `user_id`: User ID
/// - `fields`: Additional fields
/// # Returns
/// Result indicating whether the operation was successful
/// # Example
/// ```no_run
/// use tracing_core::Level;
/// use rustfs_obs::log_with_context;
///
/// async fn example() {
/// let _ = log_with_context("This is an information message", "example", Level::INFO, Some("req-12345".to_string()), Some("user-6789".to_string()), vec![("endpoint".to_string(), "/api/v1/data".to_string())]).await;
/// }
/// ```
pub async fn log_with_context(
message: &str,
source: &str,
level: Level,
request_id: Option<String>,
user_id: Option<String>,
fields: Vec<(String, String)>,
) -> Result<(), GlobalError> {
get_global_logger()
.lock()
.await
.write_with_context(message, source, level, request_id, user_id, fields)
.await
}
/// Log initialization status
#[derive(Debug)]
pub(crate) struct InitLogStatus {
pub timestamp: SystemTime,
pub service_name: String,
pub version: String,
pub environment: String,
}
impl Default for InitLogStatus {
fn default() -> Self {
Self {
timestamp: SystemTime::now(),
service_name: String::from(APP_NAME),
version: SERVICE_VERSION.to_string(),
environment: ENVIRONMENT.to_string(),
}
}
}
impl InitLogStatus {
pub fn new_config(config: &OtelConfig) -> Self {
let config = config.clone();
let environment = config.environment.unwrap_or(ENVIRONMENT.to_string());
let version = config.service_version.unwrap_or(SERVICE_VERSION.to_string());
Self {
timestamp: SystemTime::now(),
service_name: String::from(APP_NAME),
version,
environment,
}
}
pub async fn init_start_log(config: &OtelConfig) -> Result<(), GlobalError> {
let status = Self::new_config(config);
log_init_state(Some(status)).await
}
}
/// Log initialization details during system startup
async fn log_init_state(status: Option<InitLogStatus>) -> Result<(), GlobalError> {
let status = status.unwrap_or_default();
let base_entry = BaseLogEntry::new()
.timestamp(chrono::DateTime::from(status.timestamp))
.message(Some(format!(
"Service initialization started - {} v{} in {}",
status.service_name, status.version, status.environment
)))
.request_id(Some("system_init".to_string()));
let server_entry = ServerLogEntry::new(Level::INFO, "system_initialization".to_string())
.with_base(base_entry)
.user_id(Some("system".to_string()));
get_global_logger().lock().await.log_server_entry(server_entry).await?;
Ok(())
}

View File

@@ -1,178 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::sinks::Sink;
use crate::{LogRecord, UnifiedLogEntry};
use async_trait::async_trait;
use std::sync::Arc;
use tokio::fs::OpenOptions;
use tokio::io;
use tokio::io::AsyncWriteExt;
/// File Sink Implementation
pub struct FileSink {
path: String,
buffer_size: usize,
writer: Arc<tokio::sync::Mutex<io::BufWriter<tokio::fs::File>>>,
entry_count: std::sync::atomic::AtomicUsize,
last_flush: std::sync::atomic::AtomicU64,
flush_interval_ms: u64, // Time between flushes
flush_threshold: usize, // Number of entries before flush
}
impl FileSink {
/// Create a new FileSink instance
pub async fn new(
path: String,
buffer_size: usize,
flush_interval_ms: u64,
flush_threshold: usize,
) -> Result<Self, io::Error> {
// check if the file exists
let file_exists = tokio::fs::metadata(&path).await.is_ok();
// if the file not exists, create it
if !file_exists {
tokio::fs::create_dir_all(std::path::Path::new(&path).parent().unwrap()).await?;
tracing::debug!("File does not exist, creating it. Path: {:?}", path)
}
let file = if file_exists {
// If the file exists, open it in append mode
tracing::debug!("FileSink: File exists, opening in append mode. Path: {:?}", path);
OpenOptions::new().append(true).create(true).open(&path).await?
} else {
// If the file does not exist, create it
tracing::debug!("FileSink: File does not exist, creating a new file.");
// Create the file and write a header or initial content if needed
OpenOptions::new().create(true).truncate(true).write(true).open(&path).await?
};
let writer = io::BufWriter::with_capacity(buffer_size, file);
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64;
Ok(FileSink {
path,
buffer_size,
writer: Arc::new(tokio::sync::Mutex::new(writer)),
entry_count: std::sync::atomic::AtomicUsize::new(0),
last_flush: std::sync::atomic::AtomicU64::new(now),
flush_interval_ms,
flush_threshold,
})
}
#[allow(dead_code)]
async fn initialize_writer(&mut self) -> io::Result<()> {
let file = tokio::fs::File::create(&self.path).await?;
// Use buffer_size to create a buffer writer with a specified capacity
let buf_writer = io::BufWriter::with_capacity(self.buffer_size, file);
// Replace the original writer with the new Mutex
self.writer = Arc::new(tokio::sync::Mutex::new(buf_writer));
Ok(())
}
// Get the current buffer size
#[allow(dead_code)]
pub fn buffer_size(&self) -> usize {
self.buffer_size
}
// How to dynamically adjust the buffer size
#[allow(dead_code)]
pub async fn set_buffer_size(&mut self, new_size: usize) -> io::Result<()> {
if self.buffer_size != new_size {
self.buffer_size = new_size;
// Reinitialize the writer directly, without checking is_some()
self.initialize_writer().await?;
}
Ok(())
}
// Check if flushing is needed based on count or time
fn should_flush(&self) -> bool {
// Check entry count threshold
if self.entry_count.load(std::sync::atomic::Ordering::Relaxed) >= self.flush_threshold {
return true;
}
// Check time threshold
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64;
let last = self.last_flush.load(std::sync::atomic::Ordering::Relaxed);
now - last >= self.flush_interval_ms
}
}
#[async_trait]
impl Sink for FileSink {
async fn write(&self, entry: &UnifiedLogEntry) {
let line = format!("{entry:?}\n");
let mut writer = self.writer.lock().await;
if let Err(e) = writer.write_all(line.as_bytes()).await {
eprintln!(
"Failed to write log to file {}: {},entry timestamp:{:?}",
self.path,
e,
entry.get_timestamp()
);
return;
}
// Only flush periodically to improve performance
// Logic to determine when to flush could be added here
// Increment the entry count
self.entry_count.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
// Check if we should flush
if self.should_flush() {
if let Err(e) = writer.flush().await {
eprintln!("Failed to flush log file {}: {}", self.path, e);
return;
}
// Reset counters
self.entry_count.store(0, std::sync::atomic::Ordering::Relaxed);
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64;
self.last_flush.store(now, std::sync::atomic::Ordering::Relaxed);
}
}
}
impl Drop for FileSink {
fn drop(&mut self) {
let writer = self.writer.clone();
let path = self.path.clone();
tokio::task::spawn_blocking(move || {
let rt = tokio::runtime::Runtime::new().unwrap();
rt.block_on(async {
let mut writer = writer.lock().await;
if let Err(e) = writer.flush().await {
eprintln!("Failed to flush log file {path}: {e}");
}
});
});
}
}

View File

@@ -1,179 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::sinks::Sink;
use crate::{LogRecord, UnifiedLogEntry};
use async_trait::async_trait;
use std::sync::Arc;
/// Kafka Sink Implementation
pub struct KafkaSink {
producer: rdkafka::producer::FutureProducer,
topic: String,
batch_size: usize,
batch_timeout_ms: u64,
entries: Arc<tokio::sync::Mutex<Vec<UnifiedLogEntry>>>,
last_flush: Arc<std::sync::atomic::AtomicU64>,
}
impl KafkaSink {
/// Create a new KafkaSink instance
pub fn new(producer: rdkafka::producer::FutureProducer, topic: String, batch_size: usize, batch_timeout_ms: u64) -> Self {
// Create Arc-wrapped values first
let entries = Arc::new(tokio::sync::Mutex::new(Vec::with_capacity(batch_size)));
let last_flush = Arc::new(std::sync::atomic::AtomicU64::new(
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64,
));
let sink = KafkaSink {
producer: producer.clone(),
topic: topic.clone(),
batch_size,
batch_timeout_ms,
entries: entries.clone(),
last_flush: last_flush.clone(),
};
// Start background flusher
tokio::spawn(Self::periodic_flush(producer, topic, entries, last_flush, batch_timeout_ms));
sink
}
/// Add a getter method to read the batch_timeout_ms field
#[allow(dead_code)]
pub fn batch_timeout(&self) -> u64 {
self.batch_timeout_ms
}
/// Add a method to dynamically adjust the timeout if needed
#[allow(dead_code)]
pub fn set_batch_timeout(&mut self, new_timeout_ms: u64) {
self.batch_timeout_ms = new_timeout_ms;
}
async fn periodic_flush(
producer: rdkafka::producer::FutureProducer,
topic: String,
entries: Arc<tokio::sync::Mutex<Vec<UnifiedLogEntry>>>,
last_flush: Arc<std::sync::atomic::AtomicU64>,
timeout_ms: u64,
) {
loop {
tokio::time::sleep(tokio::time::Duration::from_millis(timeout_ms / 2)).await;
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64;
let last = last_flush.load(std::sync::atomic::Ordering::Relaxed);
if now - last >= timeout_ms {
let mut batch = entries.lock().await;
if !batch.is_empty() {
Self::send_batch(&producer, &topic, batch.drain(..).collect()).await;
last_flush.store(now, std::sync::atomic::Ordering::Relaxed);
}
}
}
}
async fn send_batch(producer: &rdkafka::producer::FutureProducer, topic: &str, entries: Vec<UnifiedLogEntry>) {
for entry in entries {
let payload = match serde_json::to_string(&entry) {
Ok(p) => p,
Err(e) => {
eprintln!("Failed to serialize log entry: {e}");
continue;
}
};
let span_id = entry.get_timestamp().to_rfc3339();
let _ = producer
.send(
rdkafka::producer::FutureRecord::to(topic).payload(&payload).key(&span_id),
std::time::Duration::from_secs(5),
)
.await;
}
}
}
#[async_trait]
impl Sink for KafkaSink {
async fn write(&self, entry: &UnifiedLogEntry) {
let mut batch = self.entries.lock().await;
batch.push(entry.clone());
let should_flush_by_size = batch.len() >= self.batch_size;
let should_flush_by_time = {
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64;
let last = self.last_flush.load(std::sync::atomic::Ordering::Relaxed);
now - last >= self.batch_timeout_ms
};
if should_flush_by_size || should_flush_by_time {
// Existing flush logic
let entries_to_send: Vec<UnifiedLogEntry> = batch.drain(..).collect();
let producer = self.producer.clone();
let topic = self.topic.clone();
self.last_flush.store(
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64,
std::sync::atomic::Ordering::Relaxed,
);
tokio::spawn(async move {
KafkaSink::send_batch(&producer, &topic, entries_to_send).await;
});
}
}
}
impl Drop for KafkaSink {
fn drop(&mut self) {
// Perform any necessary cleanup here
// For example, you might want to flush any remaining entries
let producer = self.producer.clone();
let topic = self.topic.clone();
let entries = self.entries.clone();
let last_flush = self.last_flush.clone();
tokio::spawn(async move {
let mut batch = entries.lock().await;
if !batch.is_empty() {
KafkaSink::send_batch(&producer, &topic, batch.drain(..).collect()).await;
last_flush.store(
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_millis() as u64,
std::sync::atomic::Ordering::Relaxed,
);
}
});
eprintln!("Dropping KafkaSink with topic: {0}", self.topic);
}
}

View File

@@ -1,123 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{AppConfig, SinkConfig, UnifiedLogEntry};
use async_trait::async_trait;
use std::sync::Arc;
#[cfg(feature = "file")]
mod file;
#[cfg(all(feature = "kafka", target_os = "linux"))]
mod kafka;
#[cfg(feature = "webhook")]
mod webhook;
/// Sink Trait definition, asynchronously write logs
#[async_trait]
pub trait Sink: Send + Sync {
async fn write(&self, entry: &UnifiedLogEntry);
}
/// Create a list of Sink instances
pub async fn create_sinks(config: &AppConfig) -> Vec<Arc<dyn Sink>> {
let mut sinks: Vec<Arc<dyn Sink>> = Vec::new();
for sink_config in &config.sinks {
match sink_config {
#[cfg(all(feature = "kafka", target_os = "linux"))]
SinkConfig::Kafka(kafka_config) => {
match rdkafka::config::ClientConfig::new()
.set("bootstrap.servers", &kafka_config.brokers)
.set("message.timeout.ms", "5000")
.create()
{
Ok(producer) => {
sinks.push(Arc::new(kafka::KafkaSink::new(
producer,
kafka_config.topic.clone(),
kafka_config
.batch_size
.unwrap_or(rustfs_config::observability::DEFAULT_SINKS_KAFKA_BATCH_SIZE),
kafka_config
.batch_timeout_ms
.unwrap_or(rustfs_config::observability::DEFAULT_SINKS_KAFKA_BATCH_TIMEOUT_MS),
)));
tracing::info!("Kafka sink created for topic: {}", kafka_config.topic);
}
Err(e) => {
tracing::error!("Failed to create Kafka producer: {}", e);
}
}
}
#[cfg(feature = "webhook")]
SinkConfig::Webhook(webhook_config) => {
sinks.push(Arc::new(webhook::WebhookSink::new(
webhook_config.endpoint.clone(),
webhook_config.auth_token.clone(),
webhook_config
.max_retries
.unwrap_or(rustfs_config::observability::DEFAULT_SINKS_WEBHOOK_MAX_RETRIES),
webhook_config
.retry_delay_ms
.unwrap_or(rustfs_config::observability::DEFAULT_SINKS_WEBHOOK_RETRY_DELAY_MS),
)));
tracing::info!("Webhook sink created for endpoint: {}", webhook_config.endpoint);
}
#[cfg(feature = "file")]
SinkConfig::File(file_config) => {
tracing::debug!("FileSink: Using path: {}", file_config.path);
match file::FileSink::new(
std::path::Path::new(&file_config.path)
.join(rustfs_config::DEFAULT_SINK_FILE_LOG_FILE)
.to_string_lossy()
.to_string(),
file_config
.buffer_size
.unwrap_or(rustfs_config::observability::DEFAULT_SINKS_FILE_BUFFER_SIZE),
file_config
.flush_interval_ms
.unwrap_or(rustfs_config::observability::DEFAULT_SINKS_FILE_FLUSH_INTERVAL_MS),
file_config
.flush_threshold
.unwrap_or(rustfs_config::observability::DEFAULT_SINKS_FILE_FLUSH_THRESHOLD),
)
.await
{
Ok(sink) => {
sinks.push(Arc::new(sink));
tracing::info!("File sink created for path: {}", file_config.path);
}
Err(e) => {
tracing::error!("Failed to create File sink: {}", e);
}
}
}
#[cfg(any(not(feature = "kafka"), not(target_os = "linux")))]
SinkConfig::Kafka(_) => {
tracing::warn!("Kafka sink is configured but the 'kafka' feature is not enabled");
}
#[cfg(not(feature = "webhook"))]
SinkConfig::Webhook(_) => {
tracing::warn!("Webhook sink is configured but the 'webhook' feature is not enabled");
}
#[cfg(not(feature = "file"))]
SinkConfig::File(_) => {
tracing::warn!("File sink is configured but the 'file' feature is not enabled");
}
}
}
sinks
}

View File

@@ -1,84 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::UnifiedLogEntry;
use crate::sinks::Sink;
use async_trait::async_trait;
/// Webhook Sink Implementation
pub struct WebhookSink {
endpoint: String,
auth_token: String,
client: reqwest::Client,
max_retries: usize,
retry_delay_ms: u64,
}
impl WebhookSink {
pub fn new(endpoint: String, auth_token: String, max_retries: usize, retry_delay_ms: u64) -> Self {
WebhookSink {
endpoint,
auth_token,
client: reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(10))
.build()
.unwrap_or_else(|_| reqwest::Client::new()),
max_retries,
retry_delay_ms,
}
}
}
#[async_trait]
impl Sink for WebhookSink {
async fn write(&self, entry: &UnifiedLogEntry) {
let mut retries = 0;
let url = self.endpoint.clone();
let entry_clone = entry.clone();
let auth_value = reqwest::header::HeaderValue::from_str(format!("Bearer {}", self.auth_token.clone()).as_str()).unwrap();
while retries < self.max_retries {
match self
.client
.post(&url)
.header(reqwest::header::AUTHORIZATION, auth_value.clone())
.json(&entry_clone)
.send()
.await
{
Ok(response) if response.status().is_success() => {
return;
}
_ => {
retries += 1;
if retries < self.max_retries {
tokio::time::sleep(tokio::time::Duration::from_millis(
self.retry_delay_ms * (1 << retries), // Exponential backoff
))
.await;
}
}
}
}
eprintln!("Failed to send log to webhook after {0} retries", self.max_retries);
}
}
impl Drop for WebhookSink {
fn drop(&mut self) {
// Perform any necessary cleanup here
// For example, you might want to log that the sink is being dropped
eprintln!("Dropping WebhookSink with URL: {0}", self.endpoint);
}
}

View File

@@ -12,7 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::OtelConfig;
use crate::config::OtelConfig;
use crate::global::{IS_OBSERVABILITY_ENABLED, OBSERVABILITY_METER_NAME};
use flexi_logger::{
Age, Cleanup, Criterion, DeferredNow, FileSpec, LogSpecification, Naming, Record, WriteMode,
WriteMode::{AsyncWith, BufferAndFlush},
@@ -63,7 +64,8 @@ use tracing_subscriber::{EnvFilter, Layer, layer::SubscriberExt, util::Subscribe
/// - The tracer provider (for distributed tracing)
/// - The meter provider (for metrics collection)
/// - The logger provider (for structured logging)
// Implement Debug trait correctly, rather than using derive, as some fields may not have implemented Debug
///
/// Implement Debug trait correctly, rather than using derive, as some fields may not have implemented Debug
pub struct OtelGuard {
tracer_provider: Option<SdkTracerProvider>,
meter_provider: Option<SdkMeterProvider>,
@@ -301,6 +303,8 @@ pub(crate) fn init_telemetry(config: &OtelConfig) -> OtelGuard {
logger_level,
env::var("RUST_LOG").unwrap_or_else(|_| "Not set".to_string())
);
IS_OBSERVABILITY_ENABLED.set(true).ok();
OBSERVABILITY_METER_NAME.set(service_name.to_string()).ok();
}
}

View File

@@ -29,7 +29,8 @@ documentation = "https://docs.rs/rustfs-policy/latest/rustfs_policy/"
workspace = true
[dependencies]
tokio.workspace = true
rustfs-config = { workspace = true, features = ["constants","opa"] }
tokio = { workspace = true, features = ["full"] }
time = { workspace = true, features = ["serde-human-readable"] }
serde = { workspace = true, features = ["derive", "rc"] }
serde_json.workspace = true
@@ -41,6 +42,10 @@ rand.workspace = true
base64-simd = { workspace = true }
jsonwebtoken = { workspace = true }
regex = { workspace = true }
reqwest.workspace = true
chrono.workspace = true
tracing.workspace = true
[dev-dependencies]
test-case.workspace = true
temp-env = { workspace = true }

View File

@@ -17,6 +17,7 @@ mod doc;
mod effect;
mod function;
mod id;
pub mod opa;
#[allow(clippy::module_inception)]
mod policy;
mod principal;

View File

@@ -0,0 +1,288 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::policy::Args as PArgs;
use rustfs_config::{ENV_PREFIX, opa::*};
use serde::Deserialize;
use serde_json::json;
use std::{collections::HashMap, env, time::Duration};
use tracing::{error, info};
#[derive(Debug, Clone, Default)]
pub struct Args {
pub url: String,
pub auth_token: String,
}
impl Args {
pub fn enable(&self) -> bool {
!self.url.is_empty()
}
}
#[derive(Debug, Clone)]
pub struct AuthZPlugin {
client: reqwest::Client,
args: Args,
}
fn check() -> Result<(), String> {
let env_list = env::vars();
let mut candidate = HashMap::new();
let prefix = format!("{ENV_PREFIX}{POLICY_PLUGIN_SUB_SYS}").to_uppercase();
for (key, value) in env_list {
if key.starts_with(&prefix) {
candidate.insert(key.to_string(), value);
}
}
//check required env vars
if candidate.remove(ENV_POLICY_PLUGIN_OPA_URL).is_none() {
return Err(format!("Missing required env var: {ENV_POLICY_PLUGIN_OPA_URL}"));
}
// check optional env vars
candidate.remove(ENV_POLICY_PLUGIN_AUTH_TOKEN);
if !candidate.is_empty() {
return Err(format!("Invalid env vars: {candidate:?}"));
}
Ok(())
}
async fn validate(config: &Args) -> Result<(), String> {
let client = reqwest::Client::new();
match client.post(&config.url).send().await {
Ok(resp) => {
match resp.status() {
reqwest::StatusCode::OK => {
info!("OPA is ready to accept requests.");
}
_ => {
return Err(format!("OPA returned an error: {}", resp.status()));
}
};
}
Err(err) => {
return Err(format!("Error connecting to OPA: {err}"));
}
};
Ok(())
}
pub async fn lookup_config() -> Result<Args, String> {
let args = Args::default();
let get_cfg =
|cfg: &str| -> Result<String, String> { env::var(cfg).map_err(|e| format!("Error getting env var {cfg}: {e:?}")) };
let url = get_cfg(ENV_POLICY_PLUGIN_OPA_URL);
if url.is_err() {
info!("OPA is not enabled.");
return Ok(args);
}
check()?;
let args = Args {
url: url.ok().unwrap(),
auth_token: get_cfg(ENV_POLICY_PLUGIN_AUTH_TOKEN).unwrap_or_default(),
};
validate(&args).await?;
Ok(args)
}
impl AuthZPlugin {
pub fn new(config: Args) -> Self {
let client = reqwest::Client::builder()
.timeout(Duration::from_secs(5))
.connect_timeout(Duration::from_secs(1))
.pool_max_idle_per_host(10)
.pool_idle_timeout(Some(Duration::from_secs(60)))
.tcp_keepalive(Some(Duration::from_secs(30)))
.tcp_nodelay(true)
.http2_keep_alive_interval(Some(Duration::from_secs(30)))
.http2_keep_alive_timeout(Duration::from_secs(15))
.build()
.unwrap();
Self { client, args: config }
}
pub async fn is_allowed(&self, args: &PArgs<'_>) -> bool {
let payload = self.build_opa_input(args);
let mut request = self.client.post(self.args.url.clone()).json(&payload);
if !self.args.auth_token.is_empty() {
request = request.header("Authorization", format!("Bearer {}", self.args.auth_token));
}
match request.send().await {
Ok(resp) => {
let status = resp.status();
if !status.is_success() {
error!("OPA returned non-success status: {}", status);
return false;
}
match resp.json::<OpaResponseEnum>().await {
Ok(response_enum) => match response_enum {
OpaResponseEnum::SimpleResult(result) => result.result,
OpaResponseEnum::AllowResult(response) => response.result.allow,
},
Err(err) => {
error!("Error parsing OPA response: {:?}", err);
false
}
}
}
Err(err) => {
error!("Error sending request to OPA: {:?}", err);
false
}
}
}
fn build_opa_input(&self, args: &PArgs<'_>) -> serde_json::Value {
let groups = match args.groups {
Some(g) => g.clone(),
None => vec![],
};
let action_str: &str = (&args.action).into();
json!({
// Core authorization parameters for OPA policy evaluation
"input":{
"identity": {
"account": args.account,
"groups": groups,
"is_owner": args.is_owner,
"claims": args.claims
},
"resource": {
"bucket": args.bucket,
"object": args.object,
"arn": if args.object.is_empty() {
format!("arn:aws:s3:::{}", args.bucket)
} else {
format!("arn:aws:s3:::{}/{}", args.bucket, args.object)
}
},
"action": action_str,
"context": {
"conditions": args.conditions,
"deny_only": args.deny_only,
"timestamp": chrono::Utc::now().to_rfc3339()
}
}
})
}
}
#[derive(Deserialize, Default)]
struct OpaResultAllow {
allow: bool,
}
#[derive(Deserialize, Default)]
struct OpaResult {
result: bool,
}
#[derive(Deserialize, Default)]
struct OpaResponse {
result: OpaResultAllow,
}
#[derive(Deserialize)]
#[serde(untagged)]
enum OpaResponseEnum {
SimpleResult(OpaResult),
AllowResult(OpaResponse),
}
#[cfg(test)]
mod tests {
use super::*;
use temp_env;
#[test]
fn test_check_valid_config() {
// Use temp_env to temporarily set environment variables
temp_env::with_vars(
[
("RUSTFS_POLICY_PLUGIN_URL", Some("http://localhost:8181/v1/data/rustfs/authz/allow")),
("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", Some("test-token")),
],
|| {
assert!(check().is_ok());
},
);
}
#[test]
fn test_check_missing_required_env() {
temp_env::with_var_unset("RUSTFS_POLICY_PLUGIN_URL", || {
temp_env::with_var("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", Some("test-token"), || {
let result = check();
assert!(result.is_err());
assert!(result.unwrap_err().contains("Missing required env var"));
});
});
}
#[test]
fn test_check_invalid_env_vars() {
temp_env::with_vars(
[
("RUSTFS_POLICY_PLUGIN_URL", Some("http://localhost:8181/v1/data/rustfs/authz/allow")),
("RUSTFS_POLICY_PLUGIN_INVALID", Some("invalid-value")),
],
|| {
let result = check();
assert!(result.is_err());
assert!(result.unwrap_err().contains("Invalid env vars"));
},
);
}
#[test]
fn test_lookup_config_not_enabled() {
temp_env::with_var_unset("RUSTFS_POLICY_PLUGIN_URL", || {
let rt = tokio::runtime::Runtime::new().unwrap();
let result = rt.block_on(async { lookup_config().await });
// Should return the default empty Args
assert!(result.is_ok());
let args = result.unwrap();
assert!(!args.enable());
assert_eq!(args.url, "");
assert_eq!(args.auth_token, "");
});
}
#[test]
fn test_args_enable() {
// Test Args enable method
let args_enabled = Args {
url: "http://localhost:8181".to_string(),
auth_token: "token".to_string(),
};
assert!(args_enabled.enable());
let args_disabled = Args {
url: "".to_string(),
auth_token: "".to_string(),
};
assert!(!args_disabled.enable());
}
}

View File

@@ -17,7 +17,6 @@ use crate::error::{Error, Result};
use serde::{Deserialize, Serialize};
use serde_json::Value;
use std::collections::{HashMap, HashSet};
/// DEFAULT_VERSION is the default version.
/// https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_version.html
pub const DEFAULT_VERSION: &str = "2012-10-17";
@@ -155,8 +154,8 @@ impl Validator for Policy {
type Error = Error;
fn is_valid(&self) -> Result<()> {
if !self.id.is_empty() && !self.id.eq(DEFAULT_VERSION) {
return Err(IamError::InvalidVersion(self.id.0.clone()).into());
if !self.version.is_empty() && !self.version.eq(DEFAULT_VERSION) {
return Err(IamError::InvalidVersion(self.version.clone()).into());
}
for statement in self.statements.iter() {
@@ -214,8 +213,8 @@ impl Validator for BucketPolicy {
type Error = Error;
fn is_valid(&self) -> Result<()> {
if !self.id.is_empty() && !self.id.eq(DEFAULT_VERSION) {
return Err(IamError::InvalidVersion(self.id.0.clone()).into());
if !self.version.is_empty() && !self.version.eq(DEFAULT_VERSION) {
return Err(IamError::InvalidVersion(self.version.clone()).into());
}
for statement in self.statements.iter() {

View File

@@ -44,6 +44,15 @@ rustfs-utils = { workspace = true, features = ["io", "hash", "compress"] }
serde_json.workspace = true
md-5 = { workspace = true }
tracing.workspace = true
thiserror.workspace = true
base64.workspace = true
sha1.workspace = true
sha2.workspace = true
base64-simd.workspace = true
crc64fast-nvme.workspace = true
s3s.workspace = true
hex-simd.workspace = true
crc32c.workspace = true
[dev-dependencies]
tokio-test = { workspace = true }
tokio-test = { workspace = true }

1092
crates/rio/src/checksum.rs Normal file

File diff suppressed because it is too large Load Diff

73
crates/rio/src/errors.rs Normal file
View File

@@ -0,0 +1,73 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use thiserror::Error;
/// SHA256 mismatch error - when content SHA256 does not match what was sent from client
#[derive(Error, Debug, Clone, PartialEq)]
#[error("Bad sha256: Expected {expected_sha256} does not match calculated {calculated_sha256}")]
pub struct Sha256Mismatch {
pub expected_sha256: String,
pub calculated_sha256: String,
}
/// Bad digest error - Content-MD5 you specified did not match what we received
#[derive(Error, Debug, Clone, PartialEq)]
#[error("Bad digest: Expected {expected_md5} does not match calculated {calculated_md5}")]
pub struct BadDigest {
pub expected_md5: String,
pub calculated_md5: String,
}
/// Size too small error - reader size too small
#[derive(Error, Debug, Clone, PartialEq)]
#[error("Size small: got {got}, want {want}")]
pub struct SizeTooSmall {
pub want: i64,
pub got: i64,
}
/// Size too large error - reader size too large
#[derive(Error, Debug, Clone, PartialEq)]
#[error("Size large: got {got}, want {want}")]
pub struct SizeTooLarge {
pub want: i64,
pub got: i64,
}
/// Size mismatch error
#[derive(Error, Debug, Clone, PartialEq)]
#[error("Size mismatch: got {got}, want {want}")]
pub struct SizeMismatch {
pub want: i64,
pub got: i64,
}
/// Checksum mismatch error - when content checksum does not match what was sent from client
#[derive(Error, Debug, Clone, PartialEq)]
#[error("Bad checksum: Want {want} does not match calculated {got}")]
pub struct ChecksumMismatch {
pub want: String,
pub got: String,
}
/// Invalid checksum error
#[derive(Error, Debug, Clone, PartialEq)]
#[error("invalid checksum")]
pub struct InvalidChecksum;
/// Check if an error is a checksum mismatch
pub fn is_checksum_mismatch(err: &(dyn std::error::Error + 'static)) -> bool {
err.downcast_ref::<ChecksumMismatch>().is_some()
}

View File

@@ -51,6 +51,7 @@ mod tests {
use crate::{CompressReader, EncryptReader, EtagReader, HashReader};
use crate::{WarpReader, resolve_etag_generic};
use md5::Md5;
use rustfs_utils::compress::CompressionAlgorithm;
use std::io::Cursor;
use tokio::io::BufReader;
@@ -72,7 +73,7 @@ mod tests {
let reader = BufReader::new(Cursor::new(&data[..]));
let reader = Box::new(WarpReader::new(reader));
let mut hash_reader =
HashReader::new(reader, data.len() as i64, data.len() as i64, Some("hash_etag".to_string()), false).unwrap();
HashReader::new(reader, data.len() as i64, data.len() as i64, Some("hash_etag".to_string()), None, false).unwrap();
// Test HashReader ETag resolution
assert_eq!(resolve_etag_generic(&mut hash_reader), Some("hash_etag".to_string()));
@@ -105,20 +106,30 @@ mod tests {
assert_eq!(resolve_etag_generic(&mut encrypt_reader), Some("encrypt_etag".to_string()));
}
#[test]
fn test_complex_nesting() {
#[tokio::test]
async fn test_complex_nesting() {
use md5::Digest;
use tokio::io::AsyncReadExt;
let data = b"test data for complex nesting";
let mut hasher = Md5::new();
hasher.update(data);
let etag = hasher.finalize();
let etag_hex = hex_simd::encode_to_string(etag, hex_simd::AsciiCase::Lower);
let reader = BufReader::new(Cursor::new(&data[..]));
let reader = Box::new(WarpReader::new(reader));
// Create a complex nested structure: CompressReader<EncryptReader<EtagReader<BufReader<Cursor>>>>
let etag_reader = EtagReader::new(reader, Some("nested_etag".to_string()));
let etag_reader = EtagReader::new(reader, Some(etag_hex.clone()));
let key = [0u8; 32];
let nonce = [0u8; 12];
let encrypt_reader = EncryptReader::new(etag_reader, key, nonce);
let mut compress_reader = CompressReader::new(encrypt_reader, CompressionAlgorithm::Gzip);
compress_reader.read_to_end(&mut Vec::new()).await.unwrap();
// Test that nested structure can resolve ETag
assert_eq!(resolve_etag_generic(&mut compress_reader), Some("nested_etag".to_string()));
assert_eq!(resolve_etag_generic(&mut compress_reader), Some(etag_hex));
}
#[test]
@@ -127,51 +138,80 @@ mod tests {
let reader = BufReader::new(Cursor::new(&data[..]));
let reader = Box::new(WarpReader::new(reader));
// Create nested structure: CompressReader<HashReader<BufReader<Cursor>>>
let hash_reader =
HashReader::new(reader, data.len() as i64, data.len() as i64, Some("hash_nested_etag".to_string()), false).unwrap();
let hash_reader = HashReader::new(
reader,
data.len() as i64,
data.len() as i64,
Some("hash_nested_etag".to_string()),
None,
false,
)
.unwrap();
let mut compress_reader = CompressReader::new(hash_reader, CompressionAlgorithm::Deflate);
// Test that nested HashReader can be resolved
assert_eq!(resolve_etag_generic(&mut compress_reader), Some("hash_nested_etag".to_string()));
}
#[test]
fn test_comprehensive_etag_extraction() {
#[tokio::test]
async fn test_comprehensive_etag_extraction() {
use md5::Digest;
use tokio::io::AsyncReadExt;
println!("🔍 Testing comprehensive ETag extraction with real reader types...");
// Test 1: Simple EtagReader
let data1 = b"simple test";
let mut hasher = Md5::new();
hasher.update(data1);
let etag = hasher.finalize();
let etag_hex = hex_simd::encode_to_string(etag, hex_simd::AsciiCase::Lower);
let reader1 = BufReader::new(Cursor::new(&data1[..]));
let reader1 = Box::new(WarpReader::new(reader1));
let mut etag_reader = EtagReader::new(reader1, Some("simple_etag".to_string()));
assert_eq!(resolve_etag_generic(&mut etag_reader), Some("simple_etag".to_string()));
let mut etag_reader = EtagReader::new(reader1, Some(etag_hex.clone()));
etag_reader.read_to_end(&mut Vec::new()).await.unwrap();
assert_eq!(resolve_etag_generic(&mut etag_reader), Some(etag_hex.clone()));
// Test 2: HashReader with ETag
let data2 = b"hash test";
let mut hasher = Md5::new();
hasher.update(data2);
let etag = hasher.finalize();
let etag_hex = hex_simd::encode_to_string(etag, hex_simd::AsciiCase::Lower);
let reader2 = BufReader::new(Cursor::new(&data2[..]));
let reader2 = Box::new(WarpReader::new(reader2));
let mut hash_reader =
HashReader::new(reader2, data2.len() as i64, data2.len() as i64, Some("hash_etag".to_string()), false).unwrap();
assert_eq!(resolve_etag_generic(&mut hash_reader), Some("hash_etag".to_string()));
HashReader::new(reader2, data2.len() as i64, data2.len() as i64, Some(etag_hex.clone()), None, false).unwrap();
hash_reader.read_to_end(&mut Vec::new()).await.unwrap();
assert_eq!(resolve_etag_generic(&mut hash_reader), Some(etag_hex.clone()));
// Test 3: Single wrapper - CompressReader<EtagReader>
let data3 = b"compress test";
let mut hasher = Md5::new();
hasher.update(data3);
let etag = hasher.finalize();
let etag_hex = hex_simd::encode_to_string(etag, hex_simd::AsciiCase::Lower);
let reader3 = BufReader::new(Cursor::new(&data3[..]));
let reader3 = Box::new(WarpReader::new(reader3));
let etag_reader3 = EtagReader::new(reader3, Some("compress_wrapped_etag".to_string()));
let etag_reader3 = EtagReader::new(reader3, Some(etag_hex.clone()));
let mut compress_reader = CompressReader::new(etag_reader3, CompressionAlgorithm::Zstd);
assert_eq!(resolve_etag_generic(&mut compress_reader), Some("compress_wrapped_etag".to_string()));
compress_reader.read_to_end(&mut Vec::new()).await.unwrap();
assert_eq!(resolve_etag_generic(&mut compress_reader), Some(etag_hex.clone()));
// Test 4: Double wrapper - CompressReader<EncryptReader<EtagReader>>
let data4 = b"double wrap test";
let mut hasher = Md5::new();
hasher.update(data4);
let etag = hasher.finalize();
let etag_hex = hex_simd::encode_to_string(etag, hex_simd::AsciiCase::Lower);
let reader4 = BufReader::new(Cursor::new(&data4[..]));
let reader4 = Box::new(WarpReader::new(reader4));
let etag_reader4 = EtagReader::new(reader4, Some("double_wrapped_etag".to_string()));
let etag_reader4 = EtagReader::new(reader4, Some(etag_hex.clone()));
let key = [1u8; 32];
let nonce = [1u8; 12];
let encrypt_reader4 = EncryptReader::new(etag_reader4, key, nonce);
let mut compress_reader4 = CompressReader::new(encrypt_reader4, CompressionAlgorithm::Gzip);
assert_eq!(resolve_etag_generic(&mut compress_reader4), Some("double_wrapped_etag".to_string()));
compress_reader4.read_to_end(&mut Vec::new()).await.unwrap();
assert_eq!(resolve_etag_generic(&mut compress_reader4), Some(etag_hex.clone()));
println!("✅ All ETag extraction methods work correctly!");
println!("✅ Trait-based approach handles recursive unwrapping!");
@@ -195,6 +235,7 @@ mod tests {
data.len() as i64,
data.len() as i64,
Some("real_world_etag".to_string()),
None,
false,
)
.unwrap();
@@ -239,7 +280,7 @@ mod tests {
let data = b"no etag test";
let reader = BufReader::new(Cursor::new(&data[..]));
let reader = Box::new(WarpReader::new(reader));
let mut hash_reader_no_etag = HashReader::new(reader, data.len() as i64, data.len() as i64, None, false).unwrap();
let mut hash_reader_no_etag = HashReader::new(reader, data.len() as i64, data.len() as i64, None, None, false).unwrap();
assert_eq!(resolve_etag_generic(&mut hash_reader_no_etag), None);
// Test with EtagReader that has None etag

View File

@@ -19,6 +19,7 @@ use pin_project_lite::pin_project;
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, ReadBuf};
use tracing::error;
pin_project! {
pub struct EtagReader {
@@ -43,7 +44,8 @@ impl EtagReader {
/// Get the final md5 value (etag) as a hex string, only compute once.
/// Can be called multiple times, always returns the same result after finished.
pub fn get_etag(&mut self) -> String {
format!("{:x}", self.md5.clone().finalize())
let etag = self.md5.clone().finalize().to_vec();
hex_simd::encode_to_string(etag, hex_simd::AsciiCase::Lower)
}
}
@@ -60,8 +62,10 @@ impl AsyncRead for EtagReader {
// EOF
*this.finished = true;
if let Some(checksum) = this.checksum {
let etag = format!("{:x}", this.md5.clone().finalize());
if *checksum != etag {
let etag = this.md5.clone().finalize().to_vec();
let etag_hex = hex_simd::encode_to_string(etag, hex_simd::AsciiCase::Lower);
if *checksum != etag_hex {
error!("Checksum mismatch, expected={:?}, actual={:?}", checksum, etag_hex);
return Poll::Ready(Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "Checksum mismatch")));
}
}
@@ -214,7 +218,7 @@ mod tests {
let data = b"checksum test data";
let mut hasher = Md5::new();
hasher.update(data);
let expected = format!("{:x}", hasher.finalize());
let expected = hex_simd::encode_to_string(hasher.finalize(), hex_simd::AsciiCase::Lower);
let reader = BufReader::new(&data[..]);
let reader = Box::new(WarpReader::new(reader));
let mut etag_reader = EtagReader::new(reader, Some(expected.clone()));
@@ -233,7 +237,7 @@ mod tests {
let wrong_checksum = "deadbeefdeadbeefdeadbeefdeadbeef".to_string();
let reader = BufReader::new(&data[..]);
let reader = Box::new(WarpReader::new(reader));
let mut etag_reader = EtagReader::new(reader, Some(wrong_checksum));
let mut etag_reader = EtagReader::new(reader, Some(wrong_checksum.clone()));
let mut buf = Vec::new();
// Verification failed, should return InvalidData error

Some files were not shown because too many files have changed in this diff Show More