From adc07e52094acd9670f0adcc42e2ed0248c98779 Mon Sep 17 00:00:00 2001 From: houseme Date: Thu, 21 Aug 2025 22:33:07 +0800 Subject: [PATCH] feat(targets): extract targets module into a standalone crate (#441) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * init audit logger module * add audit webhook default config kvs * feat: Add comprehensive tests for authentication module (#309) * feat: add comprehensive tests for authentication module - Add 33 unit tests covering all public functions in auth.rs - Test IAMAuth struct creation and secret key validation - Test check_claims_from_token with various credential types and scenarios - Test session token extraction from headers and query parameters - Test condition values generation for different user types - Test query parameter parsing with edge cases - Test Credentials helper methods (is_expired, is_temp, is_service_account) - Ensure tests handle global state dependencies gracefully - All tests pass successfully with 100% coverage of testable functions * style: fix code formatting issues * Add verification script for checking PR branch statuses and tests Co-authored-by: anzhengchao * fix: resolve clippy uninlined format args warning --------- Co-authored-by: Cursor Agent * feat: add basic tests for core storage module (#313) * feat: add basic tests for core storage module - Add 6 unit tests for FS struct and basic functionality - Test FS creation, Debug and Clone trait implementations - Test RUSTFS_OWNER constant definition and values - Test S3 error code creation and handling - Test compression format detection for common file types - Include comprehensive documentation about integration test needs Note: Full S3 API testing requires complex setup with storage backend, global configuration, and network infrastructure - better suited for integration tests rather than unit tests. * style: fix code formatting issues * fix: resolve clippy warnings in storage tests --------- Co-authored-by: Cursor Agent * feat: add tests for admin handlers module (#314) * feat: add tests for admin handlers module - Add 5 new unit tests for admin handler functionality - Test AccountInfo struct creation, serialization and default values - Test creation of all admin handler structs (13 handlers) - Test HealOpts JSON serialization and deserialization - Test HealOpts URL encoding/decoding with proper field types - Maintain existing test while adding comprehensive coverage - Include documentation about integration test requirements All tests pass successfully with proper error handling for complex dependencies. * style: fix code formatting issues * fix: resolve clippy warnings in admin handlers tests --------- Co-authored-by: Cursor Agent * build(deps): bump the dependencies group with 3 updates (#326) * perf: avoid transmitting parity shards when the object is good (#322) * upgrade version * Fix: fix data integrity check Signed-off-by: junxiang Mu <1948535941@qq.com> * Fix: Separate Clippy's fix and check commands into two commands. Signed-off-by: junxiang Mu <1948535941@qq.com> * fix: miss inline metadata (#345) * Update dependabot.yml * fix: Fixed an issue where the list_objects_v2 API did not return dire… (#352) * fix: Fixed an issue where the list_objects_v2 API did not return directory names when they conflicted with file names in the same bucket (e.g., test/ vs. test.txt, aaa/ vs. aaa.csv) (#335) * fix: adjusted the order of directory listings * init * fix * fix * feat: add docker usage for rustfs mcp (#365) * feat: enhance metadata extraction with object name for MIME type detection Signed-off-by: junxiang Mu <1948535941@qq.com> * Feature: lock support auto release Signed-off-by: junxiang Mu <1948535941@qq.com> * improve lock Signed-off-by: junxiang Mu <1948535941@qq.com> * Fix: fix scanner detect Signed-off-by: junxiang Mu <1948535941@qq.com> * Fix: clippy && fmt Signed-off-by: junxiang Mu <1948535941@qq.com> * refactor(ecstore): Optimize memory usage for object integrity verification Change the object integrity verification from reading all data to streaming processing to avoid memory overflow caused by large objects. Modify the TLS key log check to use environment variables directly instead of configuration constants. Add memory limits for object data reading in the AHM module. Signed-off-by: junxiang Mu <1948535941@qq.com> * Chore: reduce PR template checklist Signed-off-by: junxiang Mu <1948535941@qq.com> * Chore: remove comment code (#376) Signed-off-by: junxiang Mu <1948535941@qq.com> * chore: upgrade actions/checkout from v4 to v5 (#381) * chore: upgrade actions/checkout from v4 to v5 - Update GitHub Actions checkout action version - Ensure compatibility with latest workflow features - Maintain existing checkout behavior and configuration * upgrade version * fix * add and improve code for notify * feat: extend rustfs mcp with bucket creation and deletion (#416) * feat: extend rustfs mcp with bucket creation and deletion * update file to fix pipeline error * change variable name to fix pipeline error * fix(ecstore): add async-recursion to resolve nightly trait solver reg… (#415) * fix(ecstore): add async-recursion to resolve nightly trait solver regression The newest nightly compiler switched to the new trait solver, which currently rejects async recursive functions that were previously accepted. This causes the following compilation failures: - `LocalDisk::delete_file()` - `LocalDisk::scan_dir()` Add `async-recursion` as a workspace dependency and annotate both functions with `#[async_recursion]` so that the crate compiles cleanly with the latest nightly and will continue to build once the new solver lands in stable. Signed-off-by: reigadegr <2722688642@qq.com> * fix: resolve duplicate bound error in scan_dir function Replaced inline trait bounds with where clause to avoid duplication caused by macro expansion. Signed-off-by: reigadegr <2722688642@qq.com> --------- Signed-off-by: reigadegr <2722688642@qq.com> Co-authored-by: 安正超 * fix:make bucket exists (#428) * feat: include user-defined metadata in S3 response (#431) * fix: simplify Docker entrypoint following efficient user switching pattern (#421) * fix: simplify Docker entrypoint following efficient user switching pattern - Remove ALL file permission modifications (no chown at all) - Use chroot --userspec or gosu to switch user context - Extremely simple and fast implementation - Zero filesystem modifications for permissions Fixes #388 * Update entrypoint.sh Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update entrypoint.sh Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update entrypoint.sh Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * wip * wip * wip --------- Co-authored-by: Cursor Agent Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * docs: update doc/docker-data-dir README.md (#432) * add targets crates * feat(targets): extract targets module into a standalone crate - Move all target-related code (MQTT, Webhook, etc.) into a new `targets` crate - Update imports and dependencies to reference the new crate - Refactor interfaces to ensure compatibility with the new crate structure - Adjust Cargo.toml and workspace configuration accordingly * fix * fix --------- Signed-off-by: junxiang Mu <1948535941@qq.com> Signed-off-by: reigadegr <2722688642@qq.com> Co-authored-by: 安正超 Co-authored-by: Cursor Agent Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: zzhpro <56196563+zzhpro@users.noreply.github.com> Co-authored-by: junxiang Mu <1948535941@qq.com> Co-authored-by: weisd Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com> Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com> Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com> Co-authored-by: reigadegr <103645642+reigadegr@users.noreply.github.com> Co-authored-by: 0xdx2 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- Cargo.lock | 631 ++++++++++-------- Cargo.toml | 31 +- crates/audit-logger/Cargo.toml | 44 ++ crates/audit-logger/examples/config.json | 34 + crates/audit-logger/examples/main.rs | 17 + crates/audit-logger/src/entry/args.rs | 90 +++ crates/audit-logger/src/entry/audit.rs | 469 +++++++++++++ crates/audit-logger/src/entry/base.rs | 108 +++ crates/audit-logger/src/entry/mod.rs | 159 +++++ crates/audit-logger/src/entry/unified.rs | 266 ++++++++ crates/audit-logger/src/lib.rs | 8 + crates/audit-logger/src/logger/config.rs | 29 + crates/audit-logger/src/logger/dispatch.rs | 13 + crates/audit-logger/src/logger/entry.rs | 108 +++ crates/audit-logger/src/logger/factory.rs | 13 + crates/audit-logger/src/logger/mod.rs | 36 + crates/config/Cargo.toml | 7 +- crates/config/src/audit/mod.rs | 31 + crates/config/src/constants/env.rs | 7 + crates/config/src/lib.rs | 2 + crates/config/src/notify/mod.rs | 8 - crates/config/src/notify/mqtt.rs | 2 +- crates/config/src/notify/store.rs | 2 - crates/config/src/notify/webhook.rs | 2 +- crates/ecstore/Cargo.toml | 2 +- crates/ecstore/src/config/audit.rs | 84 +++ crates/ecstore/src/config/mod.rs | 4 +- crates/ecstore/src/config/notify.rs | 11 +- crates/notify/Cargo.toml | 8 +- crates/notify/examples/base.rs | 59 ++ crates/notify/examples/full_demo.rs | 23 +- crates/notify/examples/full_demo_one.rs | 20 +- crates/notify/src/error.rs | 106 +-- crates/notify/src/event.rs | 287 +------- crates/notify/src/factory.rs | 31 +- crates/notify/src/global.rs | 109 ++- crates/notify/src/integration.rs | 18 +- crates/notify/src/lib.rs | 54 +- crates/notify/src/notifier.rs | 30 +- crates/notify/src/registry.rs | 22 +- crates/notify/src/rules/config.rs | 4 +- crates/notify/src/rules/pattern_rules.rs | 2 +- crates/notify/src/rules/rules_map.rs | 4 +- crates/notify/src/rules/target_id_set.rs | 2 +- crates/notify/src/rules/xml_config.rs | 4 +- crates/notify/src/stream.rs | 31 +- crates/rio/Cargo.toml | 2 +- crates/targets/Cargo.toml | 31 + crates/{notify => targets}/src/arn.rs | 0 crates/targets/src/error.rs | 99 +++ crates/targets/src/event_name.rs | 290 ++++++++ crates/targets/src/lib.rs | 35 + crates/{notify => targets}/src/store.rs | 8 +- crates/{notify => targets}/src/target/mod.rs | 56 +- crates/{notify => targets}/src/target/mqtt.rs | 62 +- .../{notify => targets}/src/target/webhook.rs | 51 +- rustfs/Cargo.toml | 4 +- rustfs/src/admin/handlers/event.rs | 7 +- rustfs/src/admin/mod.rs | 23 +- rustfs/src/config/mod.rs | 2 +- rustfs/src/server/audit.rs | 13 + rustfs/src/server/mod.rs | 2 + rustfs/src/storage/ecfs.rs | 2 +- 63 files changed, 2837 insertions(+), 882 deletions(-) create mode 100644 crates/audit-logger/Cargo.toml create mode 100644 crates/audit-logger/examples/config.json create mode 100644 crates/audit-logger/examples/main.rs create mode 100644 crates/audit-logger/src/entry/args.rs create mode 100644 crates/audit-logger/src/entry/audit.rs create mode 100644 crates/audit-logger/src/entry/base.rs create mode 100644 crates/audit-logger/src/entry/mod.rs create mode 100644 crates/audit-logger/src/entry/unified.rs create mode 100644 crates/audit-logger/src/lib.rs create mode 100644 crates/audit-logger/src/logger/config.rs create mode 100644 crates/audit-logger/src/logger/dispatch.rs create mode 100644 crates/audit-logger/src/logger/entry.rs create mode 100644 crates/audit-logger/src/logger/factory.rs create mode 100644 crates/audit-logger/src/logger/mod.rs create mode 100644 crates/config/src/audit/mod.rs create mode 100644 crates/ecstore/src/config/audit.rs create mode 100644 crates/notify/examples/base.rs create mode 100644 crates/targets/Cargo.toml rename crates/{notify => targets}/src/arn.rs (100%) create mode 100644 crates/targets/src/error.rs create mode 100644 crates/targets/src/event_name.rs create mode 100644 crates/targets/src/lib.rs rename crates/{notify => targets}/src/store.rs (98%) rename crates/{notify => targets}/src/target/mod.rs (70%) rename crates/{notify => targets}/src/target/mqtt.rs (93%) rename crates/{notify => targets}/src/target/webhook.rs (90%) create mode 100644 rustfs/src/server/audit.rs diff --git a/Cargo.lock b/Cargo.lock index 1e169b51..615d2bfd 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -175,9 +175,9 @@ checksum = "b0674a1ddeecb70197781e945de4b3b8ffb61fa939a5597bcf48503737663100" [[package]] name = "arbitrary" -version = "1.4.1" +version = "1.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dde20b3d026af13f561bdd0f15edf01fc734f0dafcedbaf42bba506a9517f223" +checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1" dependencies = [ "derive_arbitrary", ] @@ -557,7 +557,7 @@ checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -597,7 +597,7 @@ checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -608,13 +608,13 @@ checksum = "8b75356056920673b02621b35afd0f7dda9306d03c79a30f5c56c44cf256e3de" [[package]] name = "async-trait" -version = "0.1.88" +version = "0.1.89" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e539d3fca749fcee5236ab05e93a52867dd549cc157c8cb7f99595f3cedffdb5" +checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -649,6 +649,15 @@ dependencies = [ "num-traits", ] +[[package]] +name = "atomic" +version = "0.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89cbf775b137e9b968e67227ef7f775587cde3fd31b0d8599dbd0f598a48340" +dependencies = [ + "bytemuck", +] + [[package]] name = "atomic-waker" version = "1.1.2" @@ -663,7 +672,7 @@ checksum = "99e1aca718ea7b89985790c94aad72d77533063fe00bc497bb79a7c2dae6a661" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -674,9 +683,9 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "aws-config" -version = "1.8.4" +version = "1.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "483020b893cdef3d89637e428d588650c71cfae7ea2e6ecbaee4de4ff99fb2dd" +checksum = "c478f5b10ce55c9a33f87ca3404ca92768b144fc1bfdede7c0121214a8283a25" dependencies = [ "aws-credential-types", "aws-runtime", @@ -764,9 +773,9 @@ dependencies = [ [[package]] name = "aws-sdk-s3" -version = "1.101.0" +version = "1.102.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b16efa59a199f5271bf21ab3e570c5297d819ce4f240e6cf0096d1dc0049c44" +checksum = "75ddb925e840f49446aa6338b67abdbec04b4ebf923b7da038ec4c35afb916cd" dependencies = [ "aws-credential-types", "aws-runtime", @@ -798,9 +807,9 @@ dependencies = [ [[package]] name = "aws-sdk-sso" -version = "1.79.0" +version = "1.81.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0a847168f15b46329fa32c7aca4e4f1a2e072f9b422f0adb19756f2e1457f111" +checksum = "79ede098271e3471036c46957cba2ba30888f53bda2515bf04b560614a30a36e" dependencies = [ "aws-credential-types", "aws-runtime", @@ -820,9 +829,9 @@ dependencies = [ [[package]] name = "aws-sdk-ssooidc" -version = "1.80.0" +version = "1.82.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b654dd24d65568738593e8239aef279a86a15374ec926ae8714e2d7245f34149" +checksum = "43326f724ba2cc957e6f3deac0ca1621a3e5d4146f5970c24c8a108dac33070f" dependencies = [ "aws-credential-types", "aws-runtime", @@ -842,9 +851,9 @@ dependencies = [ [[package]] name = "aws-sdk-sts" -version = "1.81.0" +version = "1.83.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c92ea8a7602321c83615c82b408820ad54280fb026e92de0eeea937342fafa24" +checksum = "a5468593c47efc31fdbe6c902d1a5fde8d9c82f78a3f8ccfe907b1e9434748cb" dependencies = [ "aws-credential-types", "aws-runtime", @@ -924,9 +933,9 @@ dependencies = [ [[package]] name = "aws-smithy-eventstream" -version = "0.60.10" +version = "0.60.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "604c7aec361252b8f1c871a7641d5e0ba3a7f5a586e51b66bc9510a5519594d9" +checksum = "182b03393e8c677347fb5705a04a9392695d47d20ef0a2f8cfe28c8e6b9b9778" dependencies = [ "aws-smithy-types", "bytes", @@ -956,9 +965,9 @@ dependencies = [ [[package]] name = "aws-smithy-http-client" -version = "1.0.6" +version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f108f1ca850f3feef3009bdcc977be201bca9a91058864d9de0684e64514bee0" +checksum = "4fdbad9bd9dbcc6c5e68c311a841b54b70def3ca3b674c42fbebb265980539f8" dependencies = [ "aws-smithy-async", "aws-smithy-runtime-api", @@ -969,7 +978,7 @@ dependencies = [ "http 1.3.1", "http-body 0.4.6", "hyper 0.14.32", - "hyper 1.6.0", + "hyper 1.7.0", "hyper-rustls 0.24.2", "hyper-rustls 0.27.7", "hyper-util", @@ -979,6 +988,7 @@ dependencies = [ "rustls-native-certs 0.8.1", "rustls-pki-types", "tokio", + "tokio-rustls 0.26.2", "tower", "tracing", ] @@ -1013,9 +1023,9 @@ dependencies = [ [[package]] name = "aws-smithy-runtime" -version = "1.8.6" +version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e107ce0783019dbff59b3a244aa0c114e4a8c9d93498af9162608cd5474e796" +checksum = "a3d57c8b53a72d15c8e190475743acf34e4996685e346a3448dd54ef696fc6e0" dependencies = [ "aws-smithy-async", "aws-smithy-http", @@ -1037,9 +1047,9 @@ dependencies = [ [[package]] name = "aws-smithy-runtime-api" -version = "1.8.7" +version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75d52251ed4b9776a3e8487b2a01ac915f73b2da3af8fc1e77e0fce697a550d4" +checksum = "07f5e0fc8a6b3f2303f331b94504bbf754d85488f402d6f1dd7a6080f99afe56" dependencies = [ "aws-smithy-async", "aws-smithy-types", @@ -1114,7 +1124,7 @@ dependencies = [ "http 1.3.1", "http-body 1.0.1", "http-body-util", - "hyper 1.6.0", + "hyper 1.7.0", "hyper-util", "itoa 1.0.15", "matchit", @@ -1232,7 +1242,7 @@ version = "0.69.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "271383c67ccabffb7381723dea0672a673f292304fcb45c01cc648c7a8d58088" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "cexpr", "clang-sys", "itertools 0.12.1", @@ -1245,7 +1255,7 @@ dependencies = [ "regex", "rustc-hash 1.1.0", "shlex", - "syn 2.0.104", + "syn 2.0.106", "which", ] @@ -1257,9 +1267,9 @@ checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" [[package]] name = "bitflags" -version = "2.9.1" +version = "2.9.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b8e56985ec62d17e9c1001dc89c88ecd7dc08e47eba5ec7c29c7b5eeecde967" +checksum = "6a65b545ab31d687cff52899d4890855fec459eb6afe0da6417b8a18da87aa29" dependencies = [ "serde", ] @@ -1325,7 +1335,7 @@ version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "340d2f0bdb2a43c1d3cd40513185b2bd7def0aa1052f956455114bc98f82dcf2" dependencies = [ - "objc2 0.6.1", + "objc2 0.6.2", ] [[package]] @@ -1354,9 +1364,9 @@ dependencies = [ [[package]] name = "brotli" -version = "8.0.1" +version = "8.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9991eea70ea4f293524138648e41ee89b0b2b12ddef3b255effa43c8056e0e0d" +checksum = "4bd8b9603c7aa97359dbd97ecf258968c95f3adddd6db2f7e7a5bef101c84560" dependencies = [ "alloc-no-stdlib", "alloc-stdlib", @@ -1399,6 +1409,12 @@ version = "3.19.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "46c5e41b57b8bba42a04676d81cb89e9ee8e859a1a66f80a5a72e1cb76b34d43" +[[package]] +name = "bytemuck" +version = "1.23.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3995eaeebcdf32f91f980d360f78732ddc061097ab4e39991ae7a6ace9194677" + [[package]] name = "byteorder" version = "1.5.0" @@ -1464,7 +1480,7 @@ version = "0.18.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ca26ef0159422fb77631dc9d17b102f253b876fe1586b03b803e63a309b4ee2" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "cairo-sys-rs", "glib", "libc", @@ -1511,7 +1527,7 @@ dependencies = [ "serde", "serde-untagged", "serde-value", - "thiserror 2.0.14", + "thiserror 2.0.15", "toml", "unicode-xid", "url", @@ -1529,7 +1545,7 @@ dependencies = [ "semver", "serde", "serde_json", - "thiserror 2.0.14", + "thiserror 2.0.15", ] [[package]] @@ -1540,9 +1556,9 @@ checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5" [[package]] name = "cc" -version = "1.2.32" +version = "1.2.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2352e5597e9c544d5e6d9c95190d5d27738ade584fa8db0a16e130e5c2b5296e" +checksum = "3ee0f8803222ba5a7e2777dd72ca451868909b1ac410621b676adf07280e9b5f" dependencies = [ "jobserver", "libc", @@ -1697,9 +1713,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.5.44" +version = "4.5.45" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c1f056bae57e3e54c3375c41ff79619ddd13460a17d7438712bd0d83fda4ff8" +checksum = "1fc0e74a703892159f5ae7d3aac52c8e6c392f5ae5f359c70b5881d60aaac318" dependencies = [ "clap_builder", "clap_derive", @@ -1719,14 +1735,14 @@ dependencies = [ [[package]] name = "clap_derive" -version = "4.5.41" +version = "4.5.45" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef4f52386a59ca4c860f7393bcf8abd8dfd91ecccc0f774635ff68e92eeef491" +checksum = "14cb31bb0a7d536caef2639baa7fad459e15c3144efefa6dbd1c84562c4739f6" dependencies = [ "heck 0.5.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -1766,7 +1782,7 @@ version = "0.26.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ad36507aeb7e16159dfe68db81ccc27571c3ccd4b76fb2fb72fc59e7a4b1b64c" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block", "cocoa-foundation 0.2.1", "core-foundation 0.10.1", @@ -1796,7 +1812,7 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "81411967c50ee9a1fc11365f8c585f863a22a9697c89239c452292c40ba79b0d" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block", "core-foundation 0.10.1", "core-graphics-types 0.2.0", @@ -1898,7 +1914,7 @@ checksum = "04382d0d9df7434af6b1b49ea1a026ef39df1b0738b1cc373368cf175354f6eb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -1918,7 +1934,7 @@ checksum = "95013972663dd72254b963e48857284080001ffee418731f065fcf5290a5530d" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -2007,7 +2023,7 @@ version = "0.24.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fa95a34622365fa5bbf40b20b75dba8dfa8c94c734aea8ac9a5ca38af14316f1" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "core-foundation 0.10.1", "core-graphics-types 0.2.0", "foreign-types", @@ -2031,7 +2047,7 @@ version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d44a101f213f6c4cdc1853d4b78aef6db6bdfa3468798cc1d9912f4735013eb" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "core-foundation 0.10.1", "libc", ] @@ -2249,7 +2265,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "13b588ba4ac1a99f7f2964d24b3d896ddc6bf847ee3855dbd4366f058cfcd331" dependencies = [ "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -2294,12 +2310,12 @@ dependencies = [ [[package]] name = "darling" -version = "0.21.1" +version = "0.21.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6b136475da5ef7b6ac596c0e956e37bad51b85b987ff3d5e230e964936736b2" +checksum = "08440b3dd222c3d0433e63e097463969485f112baff337dfdaca043a0d760570" dependencies = [ - "darling_core 0.21.1", - "darling_macro 0.21.1", + "darling_core 0.21.2", + "darling_macro 0.21.2", ] [[package]] @@ -2313,21 +2329,21 @@ dependencies = [ "proc-macro2", "quote", "strsim", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] name = "darling_core" -version = "0.21.1" +version = "0.21.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b44ad32f92b75fb438b04b68547e521a548be8acc339a6dacc4a7121488f53e6" +checksum = "d25b7912bc28a04ab1b7715a68ea03aaa15662b43a1a4b2c480531fd19f8bf7e" dependencies = [ "fnv", "ident_case", "proc-macro2", "quote", "strsim", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -2338,18 +2354,18 @@ checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" dependencies = [ "darling_core 0.20.11", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] name = "darling_macro" -version = "0.21.1" +version = "0.21.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b5be8a7a562d315a5b92a630c30cec6bcf663e6673f00fbb69cca66a6f521b9" +checksum = "ce154b9bea7fb0c8e8326e62d00354000c36e79770ff21b8c84e3aa267d9d531" dependencies = [ - "darling_core 0.21.1", + "darling_core 0.21.2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -2740,7 +2756,7 @@ checksum = "4800e1ff7ecf8f310887e9b54c9c444b8e215ccbc7b21c2f244cfae373b1ece7" dependencies = [ "datafusion-expr", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -2927,13 +2943,13 @@ dependencies = [ [[package]] name = "derive_arbitrary" -version = "1.4.1" +version = "1.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30542c1ad912e0e3d22a1935c290e12e8a29d704a420177a31faad4a601a0800" +checksum = "1e567bd82dcff979e4b03460c307b3cdc9e96fde3d73bed1496d2bc75d9dd62a" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -2954,7 +2970,7 @@ dependencies = [ "darling 0.20.11", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -2964,7 +2980,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ "derive_builder_core", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -2977,7 +2993,7 @@ dependencies = [ "proc-macro2", "quote", "rustc_version", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3086,7 +3102,7 @@ dependencies = [ "dioxus-rsx", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3280,7 +3296,7 @@ dependencies = [ "convert_case 0.6.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3373,7 +3389,7 @@ dependencies = [ "proc-macro2", "quote", "slab", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3385,7 +3401,7 @@ dependencies = [ "proc-macro2", "proc-macro2-diagnostics", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3446,7 +3462,7 @@ dependencies = [ "proc-macro2", "quote", "server_fn_macro", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3482,10 +3498,10 @@ version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "89a09f22a6c6069a18470eb92d2298acf25463f14256d24778e1230d789a2aec" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block2 0.6.1", "libc", - "objc2 0.6.1", + "objc2 0.6.2", ] [[package]] @@ -3496,7 +3512,7 @@ checksum = "97369cbbc041bc366949bc74d34658d6cda5621039731c6310521892a3a20ae0" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3528,7 +3544,7 @@ checksum = "788160fb30de9cdd857af31c6a2675904b16ece8fc2737b2c7127ba368c9d0f4" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3656,7 +3672,7 @@ dependencies = [ "heck 0.5.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3677,28 +3693,28 @@ checksum = "67c78a4d8fdf9953a5c9d458f9efe940fd97a0cab0941c075a813ac594733827" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] name = "enumset" -version = "1.1.7" +version = "1.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6ee17054f550fd7400e1906e2f9356c7672643ed34008a9e8abe147ccd2d821" +checksum = "f50acec76c668b4621fe3694e5ddee53c8fae2a03410026f50947d195eb44dc1" dependencies = [ "enumset_derive", ] [[package]] name = "enumset_derive" -version = "0.12.0" +version = "0.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76d07902c93376f1e96c34abc4d507c0911df3816cef50b01f5a2ff3ad8c370d" +checksum = "588eaef9dbc5d72c5fa4edf162527e67dab5d14ba61519ef7c8e233d5bdc092c" dependencies = [ - "darling 0.20.11", + "darling 0.21.2", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -3793,6 +3809,20 @@ dependencies = [ "rustc_version", ] +[[package]] +name = "figment" +version = "0.10.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8cb01cd46b0cf372153850f4c6c272d9cbea2da513e07538405148f95bd789f3" +dependencies = [ + "atomic", + "pear", + "serde", + "serde_json", + "uncased", + "version_check", +] + [[package]] name = "filetime" version = "0.2.25" @@ -3833,7 +3863,7 @@ version = "25.2.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1045398c1bfd89168b5fd3f1fc11f6e70b34f6f66300c87d44d3de849463abf1" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "rustc_version", ] @@ -3862,7 +3892,7 @@ dependencies = [ "regex", "serde", "serde_derive", - "thiserror 2.0.14", + "thiserror 2.0.15", "toml", "tracing", "tracing-subscriber", @@ -3909,7 +3939,7 @@ checksum = "1a5c6c585bc94aaf2c7b51dd4c2ba22680844aba4c687be581871a6f518c5742" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -4021,7 +4051,7 @@ checksum = "162ee34ebcb7c64a8abebc059ce0fee27c2262618d7b60ed8faf72fef13c3650" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -4274,7 +4304,7 @@ version = "0.18.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "233daaf6e83ae6a12a52055f568f9d7cf4671dabb78ff9560ab6da230ce00ee5" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "futures-channel", "futures-core", "futures-executor", @@ -4302,7 +4332,7 @@ dependencies = [ "proc-macro-error", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -4327,7 +4357,7 @@ version = "0.5.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b436093d1598b05e3b7fddc097b2bad32763f53a1beb25ab6f9718c6a60acd09" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "cocoa 0.25.0", "crossbeam-channel", "keyboard-types", @@ -4467,7 +4497,7 @@ dependencies = [ "proc-macro-error", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -4736,13 +4766,14 @@ dependencies = [ [[package]] name = "hyper" -version = "1.6.0" +version = "1.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cc2b571658e38e0c01b1fdca3bbbe93c00d3d71693ff2770043f8c29bc7d6f80" +checksum = "eb3aa54a13a0dfe7fbe3a59e0c76093041720fdc77b110cc0fc260fafb4dc51e" dependencies = [ + "atomic-waker", "bytes", "futures-channel", - "futures-util", + "futures-core", "h2 0.4.12", "http 1.3.1", "http-body 1.0.1", @@ -4750,6 +4781,7 @@ dependencies = [ "httpdate", "itoa 1.0.15", "pin-project-lite", + "pin-utils", "smallvec", "tokio", "want", @@ -4778,7 +4810,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3c93eb611681b207e1fe55d5a71ecf91572ec8a6705cdb6857f7d8d5242cf58" dependencies = [ "http 1.3.1", - "hyper 1.6.0", + "hyper 1.7.0", "hyper-util", "log", "rustls 0.23.31", @@ -4796,7 +4828,7 @@ version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0" dependencies = [ - "hyper 1.6.0", + "hyper 1.7.0", "hyper-util", "pin-project-lite", "tokio", @@ -4816,7 +4848,7 @@ dependencies = [ "futures-util", "http 1.3.1", "http-body 1.0.1", - "hyper 1.6.0", + "hyper 1.7.0", "ipnet", "libc", "percent-encoding", @@ -5011,13 +5043,19 @@ dependencies = [ "cfb", ] +[[package]] +name = "inlinable_string" +version = "0.1.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8fae54786f62fb2918dcfae3d568594e50eb9b5c25bf04371af6fe7516452fb" + [[package]] name = "inotify" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f37dccff2791ab604f9babef0ba14fbe0be30bd368dc541e2b08d07c8aa908f3" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "inotify-sys", "libc", ] @@ -5061,7 +5099,7 @@ version = "0.7.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d93587f37623a1a17d94ef2bc9ada592f5465fe7732084ab7beefabe5c77c0c4" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "cfg-if", "libc", ] @@ -5237,7 +5275,7 @@ version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b750dcadc39a09dbadd74e118f6dd6598df77fa01df0cfcdc52c28dece74528a" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "serde", "unicode-segmentation", ] @@ -5446,7 +5484,7 @@ version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "391290121bad3d37fbddad76d8f5d1c1c314cfc646d143d7e07a3086ddff0ce3" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "libc", "redox_syscall 0.5.17", ] @@ -5465,7 +5503,7 @@ dependencies = [ "once_cell", "serde", "sha2 0.10.9", - "thiserror 2.0.14", + "thiserror 2.0.15", "uuid", ] @@ -5526,7 +5564,7 @@ checksum = "656b3b27f8893f7bbf9485148ff9a65f019e3f33bd5cdc87c83cab16b3fd9ec8" dependencies = [ "libc", "neli", - "thiserror 2.0.14", + "thiserror 2.0.15", "windows-sys 0.59.0", ] @@ -5668,7 +5706,7 @@ dependencies = [ "manganis-core", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -5831,7 +5869,7 @@ version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c3f42e7bbe13d351b6bead8286a43aac9534b82bd3cc43e47037f012ebfd62d4" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "jni-sys", "log", "ndk-sys", @@ -5902,7 +5940,7 @@ version = "0.29.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "71e2746dc3a24dd78b3cfcb7be93368c6de9963d30f43a6a73998a9cf4b17b46" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "cfg-if", "cfg_aliases", "libc", @@ -5915,7 +5953,7 @@ version = "0.30.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "74523f3a35e05aba87a1d978330aef40f67b0304ac79c1c00b294c9830543db6" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "cfg-if", "cfg_aliases", "libc", @@ -5962,7 +6000,7 @@ version = "8.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4d3d07927151ff8575b7087f245456e549fea62edf0ec4e565a5ee50c8402bc3" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "fsevent-sys", "inotify", "kqueue", @@ -6155,7 +6193,7 @@ dependencies = [ "proc-macro-crate 3.3.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -6179,7 +6217,7 @@ version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0d5c6c0ef9702176a570f06ad94f3198bc29c524c8b498f1b9346e1b1bdcbb3a" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "libloading 0.8.8", "nvml-wrapper-sys", "static_assertions", @@ -6235,9 +6273,9 @@ dependencies = [ [[package]] name = "objc2" -version = "0.6.1" +version = "0.6.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88c6597e14493ab2e44ce58f2fdecf095a51f12ca57bec060a11c57332520551" +checksum = "561f357ba7f3a2a61563a186a163d0a3a5247e1089524a3981d49adb775078bc" dependencies = [ "objc2-encode", ] @@ -6248,7 +6286,7 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e4e89ad9e3d7d297152b17d39ed92cd50ca8063a89a9fa569046d41568891eff" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block2 0.5.1", "libc", "objc2 0.5.2", @@ -6264,9 +6302,9 @@ version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6f29f568bec459b0ddff777cec4fe3fd8666d82d5a40ebd0ff7e66134f89bcc" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block2 0.6.1", - "objc2 0.6.1", + "objc2 0.6.2", "objc2-core-foundation", "objc2-foundation 0.3.1", ] @@ -6277,7 +6315,7 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "617fbf49e071c178c0b24c080767db52958f716d9eabdf0890523aeae54773ef" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block2 0.5.1", "objc2 0.5.2", "objc2-foundation 0.2.2", @@ -6289,9 +6327,9 @@ version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1c10c2894a6fed806ade6027bcd50662746363a9589d3ec9d9bef30a4e4bc166" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "dispatch2", - "objc2 0.6.1", + "objc2 0.6.2", ] [[package]] @@ -6300,7 +6338,7 @@ version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "989c6c68c13021b5c2d6b71456ebb0f9dc78d752e86a98da7c716f4f9470f5a4" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "objc2-core-foundation", ] @@ -6328,7 +6366,7 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ee638a5da3799329310ad4cfa62fbf045d5f56e3ef5ba4149e7452dcf89d5a8" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block2 0.5.1", "libc", "objc2 0.5.2", @@ -6340,9 +6378,9 @@ version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "900831247d2fe1a09a683278e5384cfb8c80c79fe6b166f9d14bfdde0ea1b03c" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block2 0.6.1", - "objc2 0.6.1", + "objc2 0.6.2", "objc2-core-foundation", ] @@ -6362,7 +6400,7 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dd0cba1276f6023976a406a14ffa85e1fdd19df6b0f737b063b95f6c8c7aadd6" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block2 0.5.1", "objc2 0.5.2", "objc2-foundation 0.2.2", @@ -6374,7 +6412,7 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e42bee7bff906b14b167da2bac5efe6b6a07e6f7c0a21a7308d40c960242dc7a" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "block2 0.5.1", "objc2 0.5.2", "objc2-foundation 0.2.2", @@ -6469,7 +6507,7 @@ dependencies = [ "futures-sink", "js-sys", "pin-project-lite", - "thiserror 2.0.14", + "thiserror 2.0.15", "tracing", ] @@ -6498,7 +6536,7 @@ dependencies = [ "opentelemetry-proto", "opentelemetry_sdk", "prost 0.13.5", - "thiserror 2.0.14", + "thiserror 2.0.15", "tokio", "tonic 0.13.1", "tracing", @@ -6546,7 +6584,7 @@ dependencies = [ "percent-encoding", "rand 0.9.2", "serde_json", - "thiserror 2.0.14", + "thiserror 2.0.15", "tokio", "tokio-stream", ] @@ -6740,6 +6778,29 @@ dependencies = [ "hmac 0.12.1", ] +[[package]] +name = "pear" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bdeeaa00ce488657faba8ebf44ab9361f9365a97bd39ffb8a60663f57ff4b467" +dependencies = [ + "inlinable_string", + "pear_codegen", + "yansi", +] + +[[package]] +name = "pear_codegen" +version = "0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4bab5b985dc082b345f812b7df84e1bef27e7207b39e448439ba8bd69c93f147" +dependencies = [ + "proc-macro2", + "proc-macro2-diagnostics", + "quote", + "syn 2.0.106", +] + [[package]] name = "pem" version = "3.0.5" @@ -6921,7 +6982,7 @@ checksum = "6e918e4ff8c4549eb882f14b3a4bc8c8bc93de829416eacf579f1207a8fbf861" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -7116,12 +7177,12 @@ dependencies = [ [[package]] name = "prettyplease" -version = "0.2.36" +version = "0.2.37" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff24dfcda44452b9816fff4cd4227e1bb73ff5a2f1bc1105aa92fb8565ce44d2" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -7184,9 +7245,9 @@ checksum = "dc375e1527247fe1a97d8b7156678dfe7c1af2fc075c9a4db3690ecd2a148068" [[package]] name = "proc-macro2" -version = "1.0.97" +version = "1.0.101" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d61789d7719defeb74ea5fe81f2fdfdbd28a803847077cecce2ff14e1472f6f1" +checksum = "89ae43fd86e4158d6db51ad8e2b80f313af9cc74f5c0e03ccb87de09998732de" dependencies = [ "unicode-ident", ] @@ -7199,8 +7260,9 @@ checksum = "af066a9c399a26e020ada66a034357a868728e72cd426f3adcd35f80d88d88c8" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "version_check", + "yansi", ] [[package]] @@ -7241,7 +7303,7 @@ dependencies = [ "pulldown-cmark", "pulldown-cmark-to-cmark", "regex", - "syn 2.0.104", + "syn 2.0.106", "tempfile", ] @@ -7255,7 +7317,7 @@ dependencies = [ "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -7268,7 +7330,7 @@ dependencies = [ "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -7295,7 +7357,7 @@ version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e8bbe1a966bd2f362681a44f6edce3c2310ac21e4d5067a6e7ec396297a6ea0" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "memchr", "unicase", ] @@ -7344,7 +7406,7 @@ dependencies = [ "rustc-hash 2.1.1", "rustls 0.23.31", "socket2 0.5.10", - "thiserror 2.0.14", + "thiserror 2.0.15", "tokio", "tracing", "web-time", @@ -7365,7 +7427,7 @@ dependencies = [ "rustls 0.23.31", "rustls-pki-types", "slab", - "thiserror 2.0.14", + "thiserror 2.0.15", "tinyvec", "tracing", "web-time", @@ -7524,9 +7586,9 @@ checksum = "20675572f6f24e9e76ef639bc5552774ed45f1c30e2951e1e99c59888861c539" [[package]] name = "rayon" -version = "1.10.0" +version = "1.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b418a60154510ca1a002a752ca9714984e21e4241e804d32555251faf8b78ffa" +checksum = "368f01d005bf8fd9b1206fb6fa653e6c4a81ceb1466406b81792d87c5677a58f" dependencies = [ "either", "rayon-core", @@ -7534,9 +7596,9 @@ dependencies = [ [[package]] name = "rayon-core" -version = "1.12.1" +version = "1.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1465873a3dfdaa8ae7cb14b4383657caab0b3e8a0aa9ae8e04b044854c8dfce2" +checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91" dependencies = [ "crossbeam-deque", "crossbeam-utils", @@ -7595,7 +7657,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -7613,7 +7675,7 @@ version = "0.5.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5407465600fb0548f1442edf71dd20683c6ed326200ace4b1ef0763521bb3b77" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", ] [[package]] @@ -7624,7 +7686,7 @@ checksum = "a4e608c6638b9c18977b00b475ac1f28d14e84b27d8d42f70e0bf1e3dec127ac" dependencies = [ "getrandom 0.2.16", "libredox", - "thiserror 2.0.14", + "thiserror 2.0.15", ] [[package]] @@ -7654,7 +7716,7 @@ checksum = "1165225c21bff1f3bbce98f5a1f889949bc902d3575308cc7b0de30b4f6d27c7" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -7709,9 +7771,9 @@ checksum = "2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c" [[package]] name = "reqwest" -version = "0.12.22" +version = "0.12.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cbc931937e6ca3a06e3b6c0aa7841849b160a90351d6ab467a8b9b9959767531" +checksum = "d429f34c8092b2d42c7c93cec323bb4adeb7c67698f70839adec842ec10c7ceb" dependencies = [ "base64 0.22.1", "bytes", @@ -7723,7 +7785,7 @@ dependencies = [ "http 1.3.1", "http-body 1.0.1", "http-body-util", - "hyper 1.6.0", + "hyper 1.7.0", "hyper-rustls 0.27.7", "hyper-util", "js-sys", @@ -7798,7 +7860,7 @@ dependencies = [ "dispatch2", "js-sys", "log", - "objc2 0.6.1", + "objc2 0.6.2", "objc2-app-kit 0.3.1", "objc2-core-foundation", "objc2-foundation 0.3.1", @@ -7840,7 +7902,7 @@ dependencies = [ "schemars", "serde", "serde_json", - "thiserror 2.0.14", + "thiserror 2.0.15", "tokio", "tokio-util", "tracing", @@ -7852,11 +7914,11 @@ version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ad9720d9d2a943779f1dc3d47fa9072c7eeffaff4e1a82f67eb9f7ea52696091" dependencies = [ - "darling 0.21.1", + "darling 0.21.2", "proc-macro2", "quote", "serde_json", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -7940,7 +8002,7 @@ dependencies = [ "quote", "rust-embed-utils", "shellexpand", - "syn 2.0.104", + "syn 2.0.106", "walkdir", ] @@ -7982,7 +8044,7 @@ dependencies = [ "serde", "serde_json", "serde_yaml", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -8051,7 +8113,7 @@ dependencies = [ "futures", "http 1.3.1", "http-body 1.0.1", - "hyper 1.6.0", + "hyper 1.7.0", "hyper-util", "libsystemd", "matchit", @@ -8063,6 +8125,7 @@ dependencies = [ "rust-embed", "rustfs-ahm", "rustfs-appauth", + "rustfs-audit-logger", "rustfs-common", "rustfs-config", "rustfs-ecstore", @@ -8076,6 +8139,7 @@ dependencies = [ "rustfs-rio", "rustfs-s3select-api", "rustfs-s3select-query", + "rustfs-targets", "rustfs-utils", "rustfs-zip", "rustls 0.23.31", @@ -8086,7 +8150,7 @@ dependencies = [ "shadow-rs", "socket2 0.6.0", "sysctl", - "thiserror 2.0.14", + "thiserror 2.0.15", "tikv-jemallocator", "time", "tokio", @@ -8123,7 +8187,7 @@ dependencies = [ "serde_json", "serial_test", "tempfile", - "thiserror 2.0.14", + "thiserror 2.0.15", "tokio", "tokio-util", "tracing", @@ -8143,6 +8207,25 @@ dependencies = [ "serde_json", ] +[[package]] +name = "rustfs-audit-logger" +version = "0.0.5" +dependencies = [ + "async-trait", + "chrono", + "figment", + "reqwest", + "rustfs-targets", + "serde", + "serde_json", + "thiserror 2.0.15", + "tokio", + "tracing", + "tracing-core", + "url", + "uuid", +] + [[package]] name = "rustfs-checksums" version = "0.0.5" @@ -8196,7 +8279,7 @@ dependencies = [ "serde_json", "sha2 0.10.9", "test-case", - "thiserror 2.0.14", + "thiserror 2.0.15", "time", ] @@ -8222,7 +8305,7 @@ dependencies = [ "hex-simd", "hmac 0.12.1", "http 1.3.1", - "hyper 1.6.0", + "hyper 1.7.0", "hyper-rustls 0.27.7", "hyper-util", "lazy_static", @@ -8262,7 +8345,7 @@ dependencies = [ "smallvec", "temp-env", "tempfile", - "thiserror 2.0.14", + "thiserror 2.0.15", "time", "tokio", "tokio-stream", @@ -8290,7 +8373,7 @@ dependencies = [ "rustfs-utils", "s3s", "serde", - "thiserror 2.0.14", + "thiserror 2.0.15", "time", "tokio", "tracing", @@ -8335,7 +8418,7 @@ dependencies = [ "rustfs-utils", "serde", "serde_json", - "thiserror 2.0.14", + "thiserror 2.0.15", "time", "tokio", "tracing", @@ -8352,7 +8435,7 @@ dependencies = [ "rustfs-protos", "serde", "serde_json", - "thiserror 2.0.14", + "thiserror 2.0.15", "tokio", "tonic 0.14.1", "tracing", @@ -8366,7 +8449,7 @@ version = "0.0.5" dependencies = [ "chrono", "humantime", - "hyper 1.6.0", + "hyper 1.7.0", "serde", "serde_json", "time", @@ -8401,21 +8484,18 @@ dependencies = [ "futures", "once_cell", "quick-xml 0.38.1", - "reqwest", "rumqttc", "rustfs-config", "rustfs-ecstore", + "rustfs-targets", "rustfs-utils", "serde", "serde_json", - "snap", - "thiserror 2.0.14", + "thiserror 2.0.15", "tokio", "tracing", "tracing-subscriber", "url", - "urlencoding", - "uuid", "wildmatch", ] @@ -8442,7 +8522,7 @@ dependencies = [ "serde_json", "smallvec", "sysinfo", - "thiserror 2.0.14", + "thiserror 2.0.15", "tokio", "tracing", "tracing-core", @@ -8465,7 +8545,7 @@ dependencies = [ "serde_json", "strum", "test-case", - "thiserror 2.0.14", + "thiserror 2.0.15", "time", "tokio", ] @@ -8519,7 +8599,7 @@ dependencies = [ "futures-util", "hex", "hmac 0.12.1", - "hyper 1.6.0", + "hyper 1.7.0", "md5", "once_cell", "regex", @@ -8577,7 +8657,7 @@ version = "0.0.5" dependencies = [ "bytes", "http 1.3.1", - "hyper 1.6.0", + "hyper 1.7.0", "rustfs-utils", "s3s", "serde_urlencoded", @@ -8585,13 +8665,33 @@ dependencies = [ "tracing", ] +[[package]] +name = "rustfs-targets" +version = "0.0.5" +dependencies = [ + "async-trait", + "reqwest", + "rumqttc", + "rustfs-config", + "rustfs-utils", + "serde", + "serde_json", + "snap", + "thiserror 2.0.15", + "tokio", + "tracing", + "url", + "urlencoding", + "uuid", +] + [[package]] name = "rustfs-utils" version = "0.0.5" dependencies = [ "base64-simd", "blake3", - "brotli 8.0.1", + "brotli 8.0.2", "bytes", "crc32fast", "flate2", @@ -8599,7 +8699,7 @@ dependencies = [ "hex-simd", "highway", "hmac 0.12.1", - "hyper 1.6.0", + "hyper 1.7.0", "hyper-util", "local-ip-address", "lz4", @@ -8652,7 +8752,7 @@ version = "0.38.44" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fdb5bc1ae2baa591800df16c9ca78619bf65c0488b41b96ccec5d11220d8c154" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "errno", "libc", "linux-raw-sys 0.4.15", @@ -8665,7 +8765,7 @@ version = "1.0.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "11181fbabf243db407ef8df94a6ce0b2f9a733bd8be4ad02b4eda9602296cac8" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "errno", "libc", "linux-raw-sys 0.9.4", @@ -8848,7 +8948,7 @@ dependencies = [ "http-body 1.0.1", "http-body-util", "httparse", - "hyper 1.6.0", + "hyper 1.7.0", "itoa 1.0.15", "md-5", "memchr", @@ -8865,7 +8965,7 @@ dependencies = [ "smallvec", "std-next", "sync_wrapper", - "thiserror 2.0.14", + "thiserror 2.0.15", "time", "tokio", "tower", @@ -8886,9 +8986,9 @@ dependencies = [ [[package]] name = "scc" -version = "2.3.4" +version = "2.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22b2d775fb28f245817589471dd49c5edf64237f4a19d10ce9a92ff4651a27f4" +checksum = "46e6f046b7fef48e2660c57ed794263155d713de679057f2d0c169bfc6e756cc" dependencies = [ "sdd", ] @@ -8925,7 +9025,7 @@ dependencies = [ "proc-macro2", "quote", "serde_derive_internals", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -8976,7 +9076,7 @@ version = "2.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "897b2245f0b511c87893af39b033e5ca9cce68824c4d7e7630b5a1d339658d02" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "core-foundation 0.9.4", "core-foundation-sys", "libc", @@ -8989,7 +9089,7 @@ version = "3.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "80fb1d92c5028aa318b4b8bd7302a5bfcf48be96a37fc6fc790f806b0004ee0c" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "core-foundation 0.10.1", "core-foundation-sys", "libc", @@ -9061,9 +9161,9 @@ dependencies = [ [[package]] name = "serde-untagged" -version = "0.1.7" +version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "299d9c19d7d466db4ab10addd5703e4c615dec2a5a16dbbafe191045e87ee66e" +checksum = "34836a629bcbc6f1afdf0907a744870039b1e14c0561cb26094fa683b158eff3" dependencies = [ "erased-serde", "serde", @@ -9111,7 +9211,7 @@ checksum = "5b0276cf7f2c73365f7157c8123c21cd9a50fbbd844757af28ca1f5925fc2a00" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9122,7 +9222,7 @@ checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9136,9 +9236,9 @@ dependencies = [ [[package]] name = "serde_json" -version = "1.0.142" +version = "1.0.143" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "030fedb782600dcbd6f02d479bf0d817ac3bb40d644745b769d6a96bc3afc5a7" +checksum = "d401abef1d108fbd9cbaebc3e46611f4b1021f714a0597a71f41ee463f5f4a5a" dependencies = [ "itoa 1.0.15", "memchr", @@ -9175,7 +9275,7 @@ checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9234,7 +9334,7 @@ checksum = "5d69265a08751de7844521fd15003ae0a888e035773ba05695c5c759a6f89eef" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9276,7 +9376,7 @@ dependencies = [ "convert_case 0.6.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "xxhash-rust", ] @@ -9287,7 +9387,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f2aa8119b558a17992e0ac1fd07f080099564f24532858811ce04f742542440" dependencies = [ "server_fn_macro", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9441,7 +9541,7 @@ checksum = "297f631f50729c8c99b84667867963997ec0b50f32b2a7dbcab828ef0541e8bb" dependencies = [ "num-bigint", "num-traits", - "thiserror 2.0.14", + "thiserror 2.0.15", "time", ] @@ -9480,7 +9580,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f62f06db0370222f7f498ef478fce9f8df5828848d1d3517e3331936d7074f55" dependencies = [ "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9530,7 +9630,7 @@ dependencies = [ "heck 0.5.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9633,7 +9733,7 @@ checksum = "da5fc6819faabb412da764b99d3b713bb55083c11e7e0c00144d386cd6a1939c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9668,7 +9768,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "04082e93ed1a06debd9148c928234b46d2cf260bc65f44e1d1d3fa594c5beebc" dependencies = [ "simdutf8", - "thiserror 2.0.14", + "thiserror 2.0.15", ] [[package]] @@ -9720,7 +9820,7 @@ dependencies = [ "heck 0.5.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9820,9 +9920,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.104" +version = "2.0.106" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "17b6f705963418cdb9927482fa304bc562ece2fdd4f616084c50b7023b435a40" +checksum = "ede7c438028d4436d71104916910f5bb611972c5cfd7f89b8300a8186e6fada6" dependencies = [ "proc-macro2", "quote", @@ -9846,7 +9946,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -9855,7 +9955,7 @@ version = "0.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "01198a2debb237c62b6826ec7081082d951f46dbb64b0e8c7649a452230d1dfc" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "byteorder", "enum-as-inner", "libc", @@ -9883,7 +9983,7 @@ version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3c879d448e9d986b661742763247d3693ed13609438cf3d006f51f5368a5ba6b" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "core-foundation 0.9.4", "system-configuration-sys", ] @@ -9917,7 +10017,7 @@ version = "0.30.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6682a07cf5bab0b8a2bd20d0a542917ab928b5edb75ebd4eda6b05cbaab872da" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "cocoa 0.26.1", "core-foundation 0.10.1", "core-graphics 0.24.0", @@ -9959,7 +10059,7 @@ checksum = "f4e16beb8b2ac17db28eab8bca40e62dbfbb34c0fcdc6d9826b11b7b5d047dfd" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -10019,7 +10119,7 @@ dependencies = [ "cfg-if", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -10030,7 +10130,7 @@ checksum = "5c89e72a01ed4c579669add59014b9a524d609c0c88c6a585ce37485879f6ffb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "test-case-core", ] @@ -10051,11 +10151,11 @@ dependencies = [ [[package]] name = "thiserror" -version = "2.0.14" +version = "2.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b0949c3a6c842cbde3f1686d6eea5a010516deb7085f79db747562d4102f41e" +checksum = "80d76d3f064b981389ecb4b6b7f45a0bf9fdac1d5b9204c7bd6714fecc302850" dependencies = [ - "thiserror-impl 2.0.14", + "thiserror-impl 2.0.15", ] [[package]] @@ -10066,18 +10166,18 @@ checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] name = "thiserror-impl" -version = "2.0.14" +version = "2.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cc5b44b4ab9c2fdd0e0512e6bece8388e214c0749f5862b114cc5b7a25daf227" +checksum = "44d29feb33e986b6ea906bd9c3559a856983f92371b3eaa5e83782a351623de0" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -10184,9 +10284,9 @@ dependencies = [ [[package]] name = "tinyvec" -version = "1.9.0" +version = "1.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09b3661f17e86524eccd4371ab0429194e0d7c008abb45f7a7495b1719463c71" +checksum = "bfa5fdc3bce6191a1dbc8c02d5c8bffcf557bafa17c124c5264a458f1b0613fa" dependencies = [ "tinyvec_macros", ] @@ -10226,7 +10326,7 @@ checksum = "6e06d43f1345a3bcd39f6a56dbb7dcab2ba47e68e8ac134855e7e2bdbaf8cab8" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -10389,7 +10489,7 @@ dependencies = [ "http 1.3.1", "http-body 1.0.1", "http-body-util", - "hyper 1.6.0", + "hyper 1.7.0", "hyper-timeout", "hyper-util", "percent-encoding", @@ -10418,7 +10518,7 @@ dependencies = [ "http 1.3.1", "http-body 1.0.1", "http-body-util", - "hyper 1.6.0", + "hyper 1.7.0", "hyper-timeout", "hyper-util", "percent-encoding", @@ -10442,7 +10542,7 @@ dependencies = [ "prettyplease", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -10467,7 +10567,7 @@ dependencies = [ "prost-build", "prost-types", "quote", - "syn 2.0.104", + "syn 2.0.106", "tempfile", "tonic-build", ] @@ -10498,7 +10598,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "adc82fd73de2a9722ac5da747f12383d2bfdb93591ee6c58486e0097890f05f2" dependencies = [ "async-compression", - "bitflags 2.9.1", + "bitflags 2.9.2", "bytes", "futures-core", "futures-util", @@ -10559,7 +10659,7 @@ checksum = "81383ab64e72a7a8b8e13130c49e3dab29def6d0c7d76a03087b3cf71c5c6903" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -10673,14 +10773,14 @@ dependencies = [ "dirs", "libappindicator", "muda 0.15.3", - "objc2 0.6.1", + "objc2 0.6.2", "objc2-app-kit 0.3.1", "objc2-core-foundation", "objc2-core-graphics", "objc2-foundation 0.3.1", "once_cell", "png", - "thiserror 2.0.14", + "thiserror 2.0.15", "windows-sys 0.59.0", ] @@ -10784,6 +10884,15 @@ dependencies = [ "winapi", ] +[[package]] +name = "uncased" +version = "0.9.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e1b88fcfe09e89d3866a5c11019378088af2d24c3fbd4f0543f96b479ec90697" +dependencies = [ + "version_check", +] + [[package]] name = "unicase" version = "2.8.1" @@ -10894,7 +11003,7 @@ checksum = "22b7ad00068276db5fea436dba78daa7891b8d60db76e4f51cbdefbdecdab97e" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -11001,7 +11110,7 @@ checksum = "59195a1db0e95b920366d949ba5e0d3fc0e70b67c09be15ce5abb790106b0571" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -11047,7 +11156,7 @@ dependencies = [ "log", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "wasm-bindgen-shared", ] @@ -11082,7 +11191,7 @@ checksum = "8ae87ea40c9f689fc23f209965b6fb8a99ad69aeeb0231408be24920604395de" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "wasm-bindgen-backend", "wasm-bindgen-shared", ] @@ -11129,7 +11238,7 @@ version = "0.31.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c66a47e840dc20793f2264eb4b3e4ecb4b75d91c0dd4af04b456128e0bdd449d" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "rustix 1.0.8", "wayland-backend", "wayland-scanner", @@ -11141,7 +11250,7 @@ version = "0.32.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "efa790ed75fbfd71283bd2521a1cfdc022aabcc28bdcff00851f9e4ae88d9901" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", "wayland-backend", "wayland-client", "wayland-scanner", @@ -11281,7 +11390,7 @@ checksum = "1d228f15bba3b9d56dde8bddbee66fa24545bd17b48d5128ccf4a8742b18e431" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -11424,7 +11533,7 @@ checksum = "2bbd5b46c938e506ecbce286b6628a02171d56153ba733b6c741fc627ec9579b" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -11435,7 +11544,7 @@ checksum = "a47fddd13af08290e67f4acabf4b459f647552718f683a7b415d290ac744a836" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -11446,7 +11555,7 @@ checksum = "053c4c462dc91d3b1504c6fe5a726dd15e216ba718e84a0e46a88fbe5ded3515" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -11457,7 +11566,7 @@ checksum = "bd9211b69f8dcdfa817bfd14bf1c97c9188afa36f4750130fcdf3f400eca9fa8" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -11854,7 +11963,7 @@ version = "0.39.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6f42320e61fe2cfd34354ecb597f86f413484a798ba44a8ca1165c58d42da6c1" dependencies = [ - "bitflags 2.9.1", + "bitflags 2.9.2", ] [[package]] @@ -11866,7 +11975,7 @@ dependencies = [ "darling 0.20.11", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -12009,7 +12118,7 @@ checksum = "38da3c9736e16c5d3c8c597a9aaa5d1fa565d0532ae05e27c24aa62fb32c0ab6" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "synstructure", ] @@ -12083,7 +12192,7 @@ dependencies = [ "proc-macro-crate 3.3.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "zvariant_utils 2.1.0", ] @@ -12096,7 +12205,7 @@ dependencies = [ "proc-macro-crate 3.3.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "zbus_names 4.2.0", "zvariant 5.6.0", "zvariant_utils 3.2.0", @@ -12142,7 +12251,7 @@ checksum = "9ecf5b4cc5364572d7f4c329661bcc82724222973f2cab6f050a4e5c22f75181" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -12162,7 +12271,7 @@ checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "synstructure", ] @@ -12183,7 +12292,7 @@ checksum = "ce36e65b0d2999d2aafac989fb249189a141aee1f53c612c1f37d72631959f69" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -12216,7 +12325,7 @@ checksum = "5b96237efa0c878c64bd89c436f661be4e46b2f3eff1ebb976f7ef2321d2f58f" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -12241,7 +12350,7 @@ dependencies = [ "memchr", "pbkdf2", "sha1 0.10.6", - "thiserror 2.0.14", + "thiserror 2.0.15", "time", "xz2", "zeroize", @@ -12327,7 +12436,7 @@ dependencies = [ "proc-macro-crate 3.3.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "zvariant_utils 2.1.0", ] @@ -12340,7 +12449,7 @@ dependencies = [ "proc-macro-crate 3.3.0", "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", "zvariant_utils 3.2.0", ] @@ -12352,7 +12461,7 @@ checksum = "c51bcff7cc3dbb5055396bcf774748c3dab426b4b8659046963523cee4808340" dependencies = [ "proc-macro2", "quote", - "syn 2.0.104", + "syn 2.0.106", ] [[package]] @@ -12365,6 +12474,6 @@ dependencies = [ "quote", "serde", "static_assertions", - "syn 2.0.104", + "syn 2.0.106", "winnow 0.7.12", ] diff --git a/Cargo.toml b/Cargo.toml index e01b6731..63383960 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -17,6 +17,7 @@ members = [ "rustfs", # Core file system implementation "cli/rustfs-gui", # Graphical user interface client "crates/appauth", # Application authentication and authorization + "crates/audit-logger", # Audit logging system for file operations "crates/common", # Shared utilities and data structures "crates/config", # Configuration management "crates/crypto", # Cryptography and security features @@ -30,6 +31,7 @@ members = [ "crates/obs", # Observability utilities "crates/protos", # Protocol buffer definitions "crates/rio", # Rust I/O utilities and abstractions + "crates/targets", # Target-specific configurations and utilities "crates/s3select-api", # S3 Select API interface "crates/s3select-query", # S3 Select query engine "crates/signer", # client signer @@ -37,7 +39,7 @@ members = [ "crates/utils", # Utility functions and helpers "crates/workers", # Worker thread pools and task scheduling "crates/zip", # ZIP file handling and compression - "crates/ahm", + "crates/ahm", # Asynchronous Hash Map for concurrent data structures "crates/mcp", # MCP server for S3 operations ] resolver = "2" @@ -59,15 +61,11 @@ unsafe_code = "deny" [workspace.lints.clippy] all = "warn" -[patch.crates-io] -rustfs-utils = { path = "crates/utils" } -rustfs-filemeta = { path = "crates/filemeta" } -rustfs-rio = { path = "crates/rio" } - [workspace.dependencies] rustfs-ahm = { path = "crates/ahm", version = "0.0.5" } rustfs-s3select-api = { path = "crates/s3select-api", version = "0.0.5" } rustfs-appauth = { path = "crates/appauth", version = "0.0.5" } +rustfs-audit-logger = { path = "crates/audit-logger", version = "0.0.5" } rustfs-common = { path = "crates/common", version = "0.0.5" } rustfs-crypto = { path = "crates/crypto", version = "0.0.5" } rustfs-ecstore = { path = "crates/ecstore", version = "0.0.5" } @@ -89,6 +87,7 @@ rustfs-signer = { path = "crates/signer", version = "0.0.5" } rustfs-checksums = { path = "crates/checksums", version = "0.0.5" } rustfs-workers = { path = "crates/workers", version = "0.0.5" } rustfs-mcp = { path = "crates/mcp", version = "0.0.5" } +rustfs-targets = { path = "crates/targets", version = "0.0.5" } aes-gcm = { version = "0.10.3", features = ["std"] } anyhow = "1.0.99" arc-swap = "1.7.1" @@ -96,15 +95,15 @@ argon2 = { version = "0.5.3", features = ["std"] } atoi = "2.0.0" async-channel = "2.5.0" async-recursion = "1.1.1" -async-trait = "0.1.88" +async-trait = "0.1.89" async-compression = { version = "0.4.19" } atomic_enum = "0.3.0" -aws-config = { version = "1.8.4" } +aws-config = { version = "1.8.5" } aws-sdk-s3 = "1.101.0" axum = "0.8.4" base64-simd = "0.8.0" base64 = "0.22.1" -brotli = "8.0.1" +brotli = "8.0.2" bytes = { version = "1.10.1", features = ["serde"] } bytesize = "2.0.1" byteorder = "1.5.0" @@ -112,7 +111,7 @@ cfg-if = "1.0.1" crc-fast = "1.4.0" chacha20poly1305 = { version = "0.10.1" } chrono = { version = "0.4.41", features = ["serde"] } -clap = { version = "4.5.44", features = ["derive", "env"] } +clap = { version = "4.5.45", features = ["derive", "env"] } const-str = { version = "0.6.4", features = ["std", "proc"] } crc32fast = "1.5.0" criterion = { version = "0.7", features = ["html_reports"] } @@ -121,7 +120,7 @@ datafusion = "46.0.1" derive_builder = "0.20.2" dioxus = { version = "0.6.3", features = ["router"] } dirs = "6.0.0" -enumset = "1.1.7" +enumset = "1.1.9" flatbuffers = "25.2.10" flate2 = "1.1.2" flexi_logger = { version = "0.31.2", features = ["trc", "dont_minimize_extra_stacks"] } @@ -134,7 +133,7 @@ hex = "0.4.3" hex-simd = "0.8.0" highway = { version = "1.3.0" } hmac = "0.12.1" -hyper = "1.6.0" +hyper = "1.7.0" hyper-util = { version = "0.1.16", features = [ "tokio", "server-auto", @@ -193,7 +192,7 @@ rand = "0.9.2" rdkafka = { version = "0.38.0", features = ["tokio"] } reed-solomon-simd = { version = "3.0.1" } regex = { version = "1.11.1" } -reqwest = { version = "0.12.22", default-features = false, features = [ +reqwest = { version = "0.12.23", default-features = false, features = [ "rustls-tls", "charset", "http2", @@ -220,7 +219,7 @@ rustls-pemfile = "2.2.0" s3s = { version = "0.12.0-minio-preview.3" } schemars = "1.0.4" serde = { version = "1.0.219", features = ["derive"] } -serde_json = { version = "1.0.142", features = ["raw_value"] } +serde_json = { version = "1.0.143", features = ["raw_value"] } serde_urlencoded = "0.7.1" serial_test = "3.2.0" sha1 = "0.10.6" @@ -237,7 +236,7 @@ sysctl = "0.6.0" tempfile = "3.20.0" temp-env = "0.3.6" test-case = "3.3.1" -thiserror = "2.0.14" +thiserror = "2.0.15" time = { version = "0.3.41", features = [ "std", "parsing", @@ -278,7 +277,7 @@ zstd = "0.13.3" [workspace.metadata.cargo-shear] -ignored = ["rustfs", "rust-i18n", "rustfs-mcp"] +ignored = ["rustfs", "rust-i18n", "rustfs-mcp", "rustfs-audit-logger", "tokio-test"] [profile.wasm-dev] inherits = "dev" diff --git a/crates/audit-logger/Cargo.toml b/crates/audit-logger/Cargo.toml new file mode 100644 index 00000000..640ea3b3 --- /dev/null +++ b/crates/audit-logger/Cargo.toml @@ -0,0 +1,44 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "rustfs-audit-logger" +edition.workspace = true +license.workspace = true +repository.workspace = true +rust-version.workspace = true +version.workspace = true +homepage.workspace = true +description = "Audit logging system for RustFS, providing detailed logging of file operations and system events." +documentation = "https://docs.rs/audit-logger/latest/audit_logger/" +keywords = ["audit", "logging", "file-operations", "system-events", "RustFS"] +categories = ["web-programming", "development-tools::profiling", "asynchronous", "api-bindings", "development-tools::debugging"] + +[dependencies] +rustfs-targets = { workspace = true } +async-trait = { workspace = true } +chrono = { workspace = true } +reqwest = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +tracing = { workspace = true, features = ["std", "attributes"] } +tracing-core = { workspace = true } +tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] } +url = { workspace = true } +uuid = { workspace = true } +thiserror = { workspace = true } +figment = { version = "0.10", features = ["json", "env"] } + +[lints] +workspace = true diff --git a/crates/audit-logger/examples/config.json b/crates/audit-logger/examples/config.json new file mode 100644 index 00000000..329f26ad --- /dev/null +++ b/crates/audit-logger/examples/config.json @@ -0,0 +1,34 @@ +{ + "console": { + "enabled": true + }, + "logger_webhook": { + "default": { + "enabled": true, + "endpoint": "http://localhost:3000/logs", + "auth_token": "secret-token-for-logs", + "batch_size": 5, + "queue_size": 1000, + "max_retry": 3, + "retry_interval": "2s" + } + }, + "audit_webhook": { + "splunk": { + "enabled": true, + "endpoint": "http://localhost:3000/audit", + "auth_token": "secret-token-for-audit", + "batch_size": 10 + } + }, + "audit_kafka": { + "default": { + "enabled": false, + "brokers": [ + "kafka1:9092", + "kafka2:9092" + ], + "topic": "minio-audit-events" + } + } +} \ No newline at end of file diff --git a/crates/audit-logger/examples/main.rs b/crates/audit-logger/examples/main.rs new file mode 100644 index 00000000..7cc4a0b7 --- /dev/null +++ b/crates/audit-logger/examples/main.rs @@ -0,0 +1,17 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +fn main() { + println!("Audit Logger Example"); +} diff --git a/crates/audit-logger/src/entry/args.rs b/crates/audit-logger/src/entry/args.rs new file mode 100644 index 00000000..1ac65af1 --- /dev/null +++ b/crates/audit-logger/src/entry/args.rs @@ -0,0 +1,90 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![allow(dead_code)] + +use crate::entry::ObjectVersion; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; + +/// Args - defines the arguments for API operations +/// Args is used to define the arguments for API operations. +/// +/// # Example +/// ``` +/// use rustfs_audit_logger::Args; +/// use std::collections::HashMap; +/// +/// let args = Args::new() +/// .set_bucket(Some("my-bucket".to_string())) +/// .set_object(Some("my-object".to_string())) +/// .set_version_id(Some("123".to_string())) +/// .set_metadata(Some(HashMap::new())); +/// ``` +#[derive(Debug, Clone, Serialize, Deserialize, Default, Eq, PartialEq)] +pub struct Args { + #[serde(rename = "bucket", skip_serializing_if = "Option::is_none")] + pub bucket: Option, + #[serde(rename = "object", skip_serializing_if = "Option::is_none")] + pub object: Option, + #[serde(rename = "versionId", skip_serializing_if = "Option::is_none")] + pub version_id: Option, + #[serde(rename = "objects", skip_serializing_if = "Option::is_none")] + pub objects: Option>, + #[serde(rename = "metadata", skip_serializing_if = "Option::is_none")] + pub metadata: Option>, +} + +impl Args { + /// Create a new Args object + pub fn new() -> Self { + Args { + bucket: None, + object: None, + version_id: None, + objects: None, + metadata: None, + } + } + + /// Set the bucket + pub fn set_bucket(mut self, bucket: Option) -> Self { + self.bucket = bucket; + self + } + + /// Set the object + pub fn set_object(mut self, object: Option) -> Self { + self.object = object; + self + } + + /// Set the version ID + pub fn set_version_id(mut self, version_id: Option) -> Self { + self.version_id = version_id; + self + } + + /// Set the objects + pub fn set_objects(mut self, objects: Option>) -> Self { + self.objects = objects; + self + } + + /// Set the metadata + pub fn set_metadata(mut self, metadata: Option>) -> Self { + self.metadata = metadata; + self + } +} diff --git a/crates/audit-logger/src/entry/audit.rs b/crates/audit-logger/src/entry/audit.rs new file mode 100644 index 00000000..ad41b363 --- /dev/null +++ b/crates/audit-logger/src/entry/audit.rs @@ -0,0 +1,469 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![allow(dead_code)] + +use crate::{BaseLogEntry, LogRecord, ObjectVersion}; +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use std::collections::HashMap; + +/// API details structure +/// ApiDetails is used to define the details of an API operation +/// +/// The `ApiDetails` structure contains the following fields: +/// - `name` - the name of the API operation +/// - `bucket` - the bucket name +/// - `object` - the object name +/// - `objects` - the list of objects +/// - `status` - the status of the API operation +/// - `status_code` - the status code of the API operation +/// - `input_bytes` - the input bytes +/// - `output_bytes` - the output bytes +/// - `header_bytes` - the header bytes +/// - `time_to_first_byte` - the time to first byte +/// - `time_to_first_byte_in_ns` - the time to first byte in nanoseconds +/// - `time_to_response` - the time to response +/// - `time_to_response_in_ns` - the time to response in nanoseconds +/// +/// The `ApiDetails` structure contains the following methods: +/// - `new` - create a new `ApiDetails` with default values +/// - `set_name` - set the name +/// - `set_bucket` - set the bucket +/// - `set_object` - set the object +/// - `set_objects` - set the objects +/// - `set_status` - set the status +/// - `set_status_code` - set the status code +/// - `set_input_bytes` - set the input bytes +/// - `set_output_bytes` - set the output bytes +/// - `set_header_bytes` - set the header bytes +/// - `set_time_to_first_byte` - set the time to first byte +/// - `set_time_to_first_byte_in_ns` - set the time to first byte in nanoseconds +/// - `set_time_to_response` - set the time to response +/// - `set_time_to_response_in_ns` - set the time to response in nanoseconds +/// +/// # Example +/// ``` +/// use rustfs_audit_logger::ApiDetails; +/// use rustfs_audit_logger::ObjectVersion; +/// +/// let api = ApiDetails::new() +/// .set_name(Some("GET".to_string())) +/// .set_bucket(Some("my-bucket".to_string())) +/// .set_object(Some("my-object".to_string())) +/// .set_objects(vec![ObjectVersion::new_with_object_name("my-object".to_string())]) +/// .set_status(Some("OK".to_string())) +/// .set_status_code(Some(200)) +/// .set_input_bytes(100) +/// .set_output_bytes(200) +/// .set_header_bytes(Some(50)) +/// .set_time_to_first_byte(Some("100ms".to_string())) +/// .set_time_to_first_byte_in_ns(Some("100000000ns".to_string())) +/// .set_time_to_response(Some("200ms".to_string())) +/// .set_time_to_response_in_ns(Some("200000000ns".to_string())); +/// ``` +#[derive(Debug, Serialize, Deserialize, Clone, Default, PartialEq, Eq)] +pub struct ApiDetails { + #[serde(rename = "name", skip_serializing_if = "Option::is_none")] + pub name: Option, + #[serde(rename = "bucket", skip_serializing_if = "Option::is_none")] + pub bucket: Option, + #[serde(rename = "object", skip_serializing_if = "Option::is_none")] + pub object: Option, + #[serde(rename = "objects", skip_serializing_if = "Vec::is_empty", default)] + pub objects: Vec, + #[serde(rename = "status", skip_serializing_if = "Option::is_none")] + pub status: Option, + #[serde(rename = "statusCode", skip_serializing_if = "Option::is_none")] + pub status_code: Option, + #[serde(rename = "rx")] + pub input_bytes: i64, + #[serde(rename = "tx")] + pub output_bytes: i64, + #[serde(rename = "txHeaders", skip_serializing_if = "Option::is_none")] + pub header_bytes: Option, + #[serde(rename = "timeToFirstByte", skip_serializing_if = "Option::is_none")] + pub time_to_first_byte: Option, + #[serde(rename = "timeToFirstByteInNS", skip_serializing_if = "Option::is_none")] + pub time_to_first_byte_in_ns: Option, + #[serde(rename = "timeToResponse", skip_serializing_if = "Option::is_none")] + pub time_to_response: Option, + #[serde(rename = "timeToResponseInNS", skip_serializing_if = "Option::is_none")] + pub time_to_response_in_ns: Option, +} + +impl ApiDetails { + /// Create a new `ApiDetails` with default values + pub fn new() -> Self { + ApiDetails { + name: None, + bucket: None, + object: None, + objects: Vec::new(), + status: None, + status_code: None, + input_bytes: 0, + output_bytes: 0, + header_bytes: None, + time_to_first_byte: None, + time_to_first_byte_in_ns: None, + time_to_response: None, + time_to_response_in_ns: None, + } + } + + /// Set the name + pub fn set_name(mut self, name: Option) -> Self { + self.name = name; + self + } + + /// Set the bucket + pub fn set_bucket(mut self, bucket: Option) -> Self { + self.bucket = bucket; + self + } + + /// Set the object + pub fn set_object(mut self, object: Option) -> Self { + self.object = object; + self + } + + /// Set the objects + pub fn set_objects(mut self, objects: Vec) -> Self { + self.objects = objects; + self + } + + /// Set the status + pub fn set_status(mut self, status: Option) -> Self { + self.status = status; + self + } + + /// Set the status code + pub fn set_status_code(mut self, status_code: Option) -> Self { + self.status_code = status_code; + self + } + + /// Set the input bytes + pub fn set_input_bytes(mut self, input_bytes: i64) -> Self { + self.input_bytes = input_bytes; + self + } + + /// Set the output bytes + pub fn set_output_bytes(mut self, output_bytes: i64) -> Self { + self.output_bytes = output_bytes; + self + } + + /// Set the header bytes + pub fn set_header_bytes(mut self, header_bytes: Option) -> Self { + self.header_bytes = header_bytes; + self + } + + /// Set the time to first byte + pub fn set_time_to_first_byte(mut self, time_to_first_byte: Option) -> Self { + self.time_to_first_byte = time_to_first_byte; + self + } + + /// Set the time to first byte in nanoseconds + pub fn set_time_to_first_byte_in_ns(mut self, time_to_first_byte_in_ns: Option) -> Self { + self.time_to_first_byte_in_ns = time_to_first_byte_in_ns; + self + } + + /// Set the time to response + pub fn set_time_to_response(mut self, time_to_response: Option) -> Self { + self.time_to_response = time_to_response; + self + } + + /// Set the time to response in nanoseconds + pub fn set_time_to_response_in_ns(mut self, time_to_response_in_ns: Option) -> Self { + self.time_to_response_in_ns = time_to_response_in_ns; + self + } +} + +/// Entry - audit entry logs +/// AuditLogEntry is used to define the structure of an audit log entry +/// +/// The `AuditLogEntry` structure contains the following fields: +/// - `base` - the base log entry +/// - `version` - the version of the audit log entry +/// - `deployment_id` - the deployment ID +/// - `event` - the event +/// - `entry_type` - the type of audit message +/// - `api` - the API details +/// - `remote_host` - the remote host +/// - `user_agent` - the user agent +/// - `req_path` - the request path +/// - `req_host` - the request host +/// - `req_claims` - the request claims +/// - `req_query` - the request query +/// - `req_header` - the request header +/// - `resp_header` - the response header +/// - `access_key` - the access key +/// - `parent_user` - the parent user +/// - `error` - the error +/// +/// The `AuditLogEntry` structure contains the following methods: +/// - `new` - create a new `AuditEntry` with default values +/// - `new_with_values` - create a new `AuditEntry` with version, time, event and api details +/// - `with_base` - set the base log entry +/// - `set_version` - set the version +/// - `set_deployment_id` - set the deployment ID +/// - `set_event` - set the event +/// - `set_entry_type` - set the entry type +/// - `set_api` - set the API details +/// - `set_remote_host` - set the remote host +/// - `set_user_agent` - set the user agent +/// - `set_req_path` - set the request path +/// - `set_req_host` - set the request host +/// - `set_req_claims` - set the request claims +/// - `set_req_query` - set the request query +/// - `set_req_header` - set the request header +/// - `set_resp_header` - set the response header +/// - `set_access_key` - set the access key +/// - `set_parent_user` - set the parent user +/// - `set_error` - set the error +/// +/// # Example +/// ``` +/// use rustfs_audit_logger::AuditLogEntry; +/// use rustfs_audit_logger::ApiDetails; +/// use std::collections::HashMap; +/// +/// let entry = AuditLogEntry::new() +/// .set_version("1.0".to_string()) +/// .set_deployment_id(Some("123".to_string())) +/// .set_event("event".to_string()) +/// .set_entry_type(Some("type".to_string())) +/// .set_api(ApiDetails::new()) +/// .set_remote_host(Some("remote-host".to_string())) +/// .set_user_agent(Some("user-agent".to_string())) +/// .set_req_path(Some("req-path".to_string())) +/// .set_req_host(Some("req-host".to_string())) +/// .set_req_claims(Some(HashMap::new())) +/// .set_req_query(Some(HashMap::new())) +/// .set_req_header(Some(HashMap::new())) +/// .set_resp_header(Some(HashMap::new())) +/// .set_access_key(Some("access-key".to_string())) +/// .set_parent_user(Some("parent-user".to_string())) +/// .set_error(Some("error".to_string())); +#[derive(Debug, Serialize, Deserialize, Clone, Default)] +pub struct AuditLogEntry { + #[serde(flatten)] + pub base: BaseLogEntry, + pub version: String, + #[serde(rename = "deploymentid", skip_serializing_if = "Option::is_none")] + pub deployment_id: Option, + pub event: String, + // Class of audit message - S3, admin ops, bucket management + #[serde(rename = "type", skip_serializing_if = "Option::is_none")] + pub entry_type: Option, + pub api: ApiDetails, + #[serde(rename = "remotehost", skip_serializing_if = "Option::is_none")] + pub remote_host: Option, + #[serde(rename = "userAgent", skip_serializing_if = "Option::is_none")] + pub user_agent: Option, + #[serde(rename = "requestPath", skip_serializing_if = "Option::is_none")] + pub req_path: Option, + #[serde(rename = "requestHost", skip_serializing_if = "Option::is_none")] + pub req_host: Option, + #[serde(rename = "requestClaims", skip_serializing_if = "Option::is_none")] + pub req_claims: Option>, + #[serde(rename = "requestQuery", skip_serializing_if = "Option::is_none")] + pub req_query: Option>, + #[serde(rename = "requestHeader", skip_serializing_if = "Option::is_none")] + pub req_header: Option>, + #[serde(rename = "responseHeader", skip_serializing_if = "Option::is_none")] + pub resp_header: Option>, + #[serde(rename = "accessKey", skip_serializing_if = "Option::is_none")] + pub access_key: Option, + #[serde(rename = "parentUser", skip_serializing_if = "Option::is_none")] + pub parent_user: Option, + #[serde(rename = "error", skip_serializing_if = "Option::is_none")] + pub error: Option, +} + +impl AuditLogEntry { + /// Create a new `AuditEntry` with default values + pub fn new() -> Self { + AuditLogEntry { + base: BaseLogEntry::new(), + version: String::new(), + deployment_id: None, + event: String::new(), + entry_type: None, + api: ApiDetails::new(), + remote_host: None, + user_agent: None, + req_path: None, + req_host: None, + req_claims: None, + req_query: None, + req_header: None, + resp_header: None, + access_key: None, + parent_user: None, + error: None, + } + } + + /// Create a new `AuditEntry` with version, time, event and api details + pub fn new_with_values(version: String, time: DateTime, event: String, api: ApiDetails) -> Self { + let mut base = BaseLogEntry::new(); + base.timestamp = time; + + AuditLogEntry { + base, + version, + deployment_id: None, + event, + entry_type: None, + api, + remote_host: None, + user_agent: None, + req_path: None, + req_host: None, + req_claims: None, + req_query: None, + req_header: None, + resp_header: None, + access_key: None, + parent_user: None, + error: None, + } + } + + /// Set the base log entry + pub fn with_base(mut self, base: BaseLogEntry) -> Self { + self.base = base; + self + } + + /// Set the version + pub fn set_version(mut self, version: String) -> Self { + self.version = version; + self + } + + /// Set the deployment ID + pub fn set_deployment_id(mut self, deployment_id: Option) -> Self { + self.deployment_id = deployment_id; + self + } + + /// Set the event + pub fn set_event(mut self, event: String) -> Self { + self.event = event; + self + } + + /// Set the entry type + pub fn set_entry_type(mut self, entry_type: Option) -> Self { + self.entry_type = entry_type; + self + } + + /// Set the API details + pub fn set_api(mut self, api: ApiDetails) -> Self { + self.api = api; + self + } + + /// Set the remote host + pub fn set_remote_host(mut self, remote_host: Option) -> Self { + self.remote_host = remote_host; + self + } + + /// Set the user agent + pub fn set_user_agent(mut self, user_agent: Option) -> Self { + self.user_agent = user_agent; + self + } + + /// Set the request path + pub fn set_req_path(mut self, req_path: Option) -> Self { + self.req_path = req_path; + self + } + + /// Set the request host + pub fn set_req_host(mut self, req_host: Option) -> Self { + self.req_host = req_host; + self + } + + /// Set the request claims + pub fn set_req_claims(mut self, req_claims: Option>) -> Self { + self.req_claims = req_claims; + self + } + + /// Set the request query + pub fn set_req_query(mut self, req_query: Option>) -> Self { + self.req_query = req_query; + self + } + + /// Set the request header + pub fn set_req_header(mut self, req_header: Option>) -> Self { + self.req_header = req_header; + self + } + + /// Set the response header + pub fn set_resp_header(mut self, resp_header: Option>) -> Self { + self.resp_header = resp_header; + self + } + + /// Set the access key + pub fn set_access_key(mut self, access_key: Option) -> Self { + self.access_key = access_key; + self + } + + /// Set the parent user + pub fn set_parent_user(mut self, parent_user: Option) -> Self { + self.parent_user = parent_user; + self + } + + /// Set the error + pub fn set_error(mut self, error: Option) -> Self { + self.error = error; + self + } +} + +impl LogRecord for AuditLogEntry { + fn to_json(&self) -> String { + serde_json::to_string(self).unwrap_or_else(|_| String::from("{}")) + } + + fn get_timestamp(&self) -> DateTime { + self.base.timestamp + } +} diff --git a/crates/audit-logger/src/entry/base.rs b/crates/audit-logger/src/entry/base.rs new file mode 100644 index 00000000..164ef3ef --- /dev/null +++ b/crates/audit-logger/src/entry/base.rs @@ -0,0 +1,108 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![allow(dead_code)] + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use serde_json::Value; +use std::collections::HashMap; + +/// Base log entry structure shared by all log types +/// This structure is used to serialize log entries to JSON +/// and send them to the log sinks +/// This structure is also used to deserialize log entries from JSON +/// This structure is also used to store log entries in the database +/// This structure is also used to query log entries from the database +/// +/// The `BaseLogEntry` structure contains the following fields: +/// - `timestamp` - the timestamp of the log entry +/// - `request_id` - the request ID of the log entry +/// - `message` - the message of the log entry +/// - `tags` - the tags of the log entry +/// +/// The `BaseLogEntry` structure contains the following methods: +/// - `new` - create a new `BaseLogEntry` with default values +/// - `message` - set the message +/// - `request_id` - set the request ID +/// - `tags` - set the tags +/// - `timestamp` - set the timestamp +/// +/// # Example +/// ``` +/// use rustfs_audit_logger::BaseLogEntry; +/// use chrono::{DateTime, Utc}; +/// use std::collections::HashMap; +/// +/// let timestamp = Utc::now(); +/// let request = Some("req-123".to_string()); +/// let message = Some("This is a log message".to_string()); +/// let tags = Some(HashMap::new()); +/// +/// let entry = BaseLogEntry::new() +/// .timestamp(timestamp) +/// .request_id(request) +/// .message(message) +/// .tags(tags); +/// ``` +#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq, Default)] +pub struct BaseLogEntry { + #[serde(rename = "time")] + pub timestamp: DateTime, + + #[serde(rename = "requestID", skip_serializing_if = "Option::is_none")] + pub request_id: Option, + + #[serde(rename = "message", skip_serializing_if = "Option::is_none")] + pub message: Option, + + #[serde(rename = "tags", skip_serializing_if = "Option::is_none")] + pub tags: Option>, +} + +impl BaseLogEntry { + /// Create a new BaseLogEntry with default values + pub fn new() -> Self { + BaseLogEntry { + timestamp: Utc::now(), + request_id: None, + message: None, + tags: None, + } + } + + /// Set the message + pub fn message(mut self, message: Option) -> Self { + self.message = message; + self + } + + /// Set the request ID + pub fn request_id(mut self, request_id: Option) -> Self { + self.request_id = request_id; + self + } + + /// Set the tags + pub fn tags(mut self, tags: Option>) -> Self { + self.tags = tags; + self + } + + /// Set the timestamp + pub fn timestamp(mut self, timestamp: DateTime) -> Self { + self.timestamp = timestamp; + self + } +} diff --git a/crates/audit-logger/src/entry/mod.rs b/crates/audit-logger/src/entry/mod.rs new file mode 100644 index 00000000..4fb7769f --- /dev/null +++ b/crates/audit-logger/src/entry/mod.rs @@ -0,0 +1,159 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![allow(dead_code)] +pub(crate) mod args; +pub(crate) mod audit; +pub(crate) mod base; +pub(crate) mod unified; + +use serde::de::Error; +use serde::{Deserialize, Deserializer, Serialize, Serializer}; +use tracing_core::Level; + +/// ObjectVersion is used across multiple modules +#[derive(Debug, Clone, Serialize, Deserialize, Eq, PartialEq)] +pub struct ObjectVersion { + #[serde(rename = "name")] + pub object_name: String, + #[serde(rename = "versionId", skip_serializing_if = "Option::is_none")] + pub version_id: Option, +} + +impl ObjectVersion { + /// Create a new ObjectVersion object + pub fn new() -> Self { + ObjectVersion { + object_name: String::new(), + version_id: None, + } + } + + /// Create a new ObjectVersion with object name + pub fn new_with_object_name(object_name: String) -> Self { + ObjectVersion { + object_name, + version_id: None, + } + } + + /// Set the object name + pub fn set_object_name(mut self, object_name: String) -> Self { + self.object_name = object_name; + self + } + + /// Set the version ID + pub fn set_version_id(mut self, version_id: Option) -> Self { + self.version_id = version_id; + self + } +} + +impl Default for ObjectVersion { + fn default() -> Self { + Self::new() + } +} + +/// Log kind/level enum +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] +pub enum LogKind { + #[serde(rename = "INFO")] + #[default] + Info, + #[serde(rename = "WARNING")] + Warning, + #[serde(rename = "ERROR")] + Error, + #[serde(rename = "FATAL")] + Fatal, +} + +/// Trait for types that can be serialized to JSON and have a timestamp +/// This trait is used by `ServerLogEntry` to convert the log entry to JSON +/// and get the timestamp of the log entry +/// This trait is implemented by `ServerLogEntry` +/// +/// # Example +/// ``` +/// use rustfs_audit_logger::LogRecord; +/// use chrono::{DateTime, Utc}; +/// use rustfs_audit_logger::ServerLogEntry; +/// use tracing_core::Level; +/// +/// let log_entry = ServerLogEntry::new(Level::INFO, "api_handler".to_string()); +/// let json = log_entry.to_json(); +/// let timestamp = log_entry.get_timestamp(); +/// ``` +pub trait LogRecord { + fn to_json(&self) -> String; + fn get_timestamp(&self) -> chrono::DateTime; +} + +/// Wrapper for `tracing_core::Level` to implement `Serialize` and `Deserialize` +/// for `ServerLogEntry` +/// This is necessary because `tracing_core::Level` does not implement `Serialize` +/// and `Deserialize` +/// This is a workaround to allow `ServerLogEntry` to be serialized and deserialized +/// using `serde` +/// +/// # Example +/// ``` +/// use rustfs_audit_logger::SerializableLevel; +/// use tracing_core::Level; +/// +/// let level = Level::INFO; +/// let serializable_level = SerializableLevel::from(level); +/// ``` +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct SerializableLevel(pub Level); + +impl From for SerializableLevel { + fn from(level: Level) -> Self { + SerializableLevel(level) + } +} + +impl From for Level { + fn from(serializable_level: SerializableLevel) -> Self { + serializable_level.0 + } +} + +impl Serialize for SerializableLevel { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + serializer.serialize_str(self.0.as_str()) + } +} + +impl<'de> Deserialize<'de> for SerializableLevel { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + let s = String::deserialize(deserializer)?; + match s.as_str() { + "TRACE" => Ok(SerializableLevel(Level::TRACE)), + "DEBUG" => Ok(SerializableLevel(Level::DEBUG)), + "INFO" => Ok(SerializableLevel(Level::INFO)), + "WARN" => Ok(SerializableLevel(Level::WARN)), + "ERROR" => Ok(SerializableLevel(Level::ERROR)), + _ => Err(D::Error::custom("unknown log level")), + } + } +} diff --git a/crates/audit-logger/src/entry/unified.rs b/crates/audit-logger/src/entry/unified.rs new file mode 100644 index 00000000..e9ae3a94 --- /dev/null +++ b/crates/audit-logger/src/entry/unified.rs @@ -0,0 +1,266 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![allow(dead_code)] + +use crate::{AuditLogEntry, BaseLogEntry, LogKind, LogRecord, SerializableLevel}; +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use tracing_core::Level; + +/// Server log entry with structured fields +/// ServerLogEntry is used to log structured log entries from the server +/// +/// The `ServerLogEntry` structure contains the following fields: +/// - `base` - the base log entry +/// - `level` - the log level +/// - `source` - the source of the log entry +/// - `user_id` - the user ID +/// - `fields` - the structured fields of the log entry +/// +/// The `ServerLogEntry` structure contains the following methods: +/// - `new` - create a new `ServerLogEntry` with specified level and source +/// - `with_base` - set the base log entry +/// - `user_id` - set the user ID +/// - `fields` - set the fields +/// - `add_field` - add a field +/// +/// # Example +/// ``` +/// use rustfs_audit_logger::ServerLogEntry; +/// use tracing_core::Level; +/// +/// let entry = ServerLogEntry::new(Level::INFO, "test_module".to_string()) +/// .user_id(Some("user-456".to_string())) +/// .add_field("operation".to_string(), "login".to_string()); +/// ``` +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] +pub struct ServerLogEntry { + #[serde(flatten)] + pub base: BaseLogEntry, + + pub level: SerializableLevel, + pub source: String, + + #[serde(rename = "userId", skip_serializing_if = "Option::is_none")] + pub user_id: Option, + + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub fields: Vec<(String, String)>, +} + +impl ServerLogEntry { + /// Create a new ServerLogEntry with specified level and source + pub fn new(level: Level, source: String) -> Self { + ServerLogEntry { + base: BaseLogEntry::new(), + level: SerializableLevel(level), + source, + user_id: None, + fields: Vec::new(), + } + } + + /// Set the base log entry + pub fn with_base(mut self, base: BaseLogEntry) -> Self { + self.base = base; + self + } + + /// Set the user ID + pub fn user_id(mut self, user_id: Option) -> Self { + self.user_id = user_id; + self + } + + /// Set fields + pub fn fields(mut self, fields: Vec<(String, String)>) -> Self { + self.fields = fields; + self + } + + /// Add a field + pub fn add_field(mut self, key: String, value: String) -> Self { + self.fields.push((key, value)); + self + } +} + +impl LogRecord for ServerLogEntry { + fn to_json(&self) -> String { + serde_json::to_string(self).unwrap_or_else(|_| String::from("{}")) + } + + fn get_timestamp(&self) -> DateTime { + self.base.timestamp + } +} + +/// Console log entry structure +/// ConsoleLogEntry is used to log console log entries +/// The `ConsoleLogEntry` structure contains the following fields: +/// - `base` - the base log entry +/// - `level` - the log level +/// - `console_msg` - the console message +/// - `node_name` - the node name +/// - `err` - the error message +/// +/// The `ConsoleLogEntry` structure contains the following methods: +/// - `new` - create a new `ConsoleLogEntry` +/// - `new_with_console_msg` - create a new `ConsoleLogEntry` with console message and node name +/// - `with_base` - set the base log entry +/// - `set_level` - set the log level +/// - `set_node_name` - set the node name +/// - `set_console_msg` - set the console message +/// - `set_err` - set the error message +/// +/// # Example +/// ``` +/// use rustfs_audit_logger::ConsoleLogEntry; +/// +/// let entry = ConsoleLogEntry::new_with_console_msg("Test message".to_string(), "node-123".to_string()); +/// ``` +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct ConsoleLogEntry { + #[serde(flatten)] + pub base: BaseLogEntry, + + pub level: LogKind, + pub console_msg: String, + pub node_name: String, + + #[serde(skip)] + pub err: Option, +} + +impl ConsoleLogEntry { + /// Create a new ConsoleLogEntry + pub fn new() -> Self { + ConsoleLogEntry { + base: BaseLogEntry::new(), + level: LogKind::Info, + console_msg: String::new(), + node_name: String::new(), + err: None, + } + } + + /// Create a new ConsoleLogEntry with console message and node name + pub fn new_with_console_msg(console_msg: String, node_name: String) -> Self { + ConsoleLogEntry { + base: BaseLogEntry::new(), + level: LogKind::Info, + console_msg, + node_name, + err: None, + } + } + + /// Set the base log entry + pub fn with_base(mut self, base: BaseLogEntry) -> Self { + self.base = base; + self + } + + /// Set the log level + pub fn set_level(mut self, level: LogKind) -> Self { + self.level = level; + self + } + + /// Set the node name + pub fn set_node_name(mut self, node_name: String) -> Self { + self.node_name = node_name; + self + } + + /// Set the console message + pub fn set_console_msg(mut self, console_msg: String) -> Self { + self.console_msg = console_msg; + self + } + + /// Set the error message + pub fn set_err(mut self, err: Option) -> Self { + self.err = err; + self + } +} + +impl Default for ConsoleLogEntry { + fn default() -> Self { + Self::new() + } +} + +impl LogRecord for ConsoleLogEntry { + fn to_json(&self) -> String { + serde_json::to_string(self).unwrap_or_else(|_| String::from("{}")) + } + + fn get_timestamp(&self) -> DateTime { + self.base.timestamp + } +} + +/// Unified log entry type +/// UnifiedLogEntry is used to log different types of log entries +/// +/// The `UnifiedLogEntry` enum contains the following variants: +/// - `Server` - a server log entry +/// - `Audit` - an audit log entry +/// - `Console` - a console log entry +/// +/// The `UnifiedLogEntry` enum contains the following methods: +/// - `to_json` - convert the log entry to JSON +/// - `get_timestamp` - get the timestamp of the log entry +/// +/// # Example +/// ``` +/// use rustfs_audit_logger::{UnifiedLogEntry, ServerLogEntry}; +/// use tracing_core::Level; +/// +/// let server_entry = ServerLogEntry::new(Level::INFO, "test_module".to_string()); +/// let unified = UnifiedLogEntry::Server(server_entry); +/// ``` +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(tag = "type")] +pub enum UnifiedLogEntry { + #[serde(rename = "server")] + Server(ServerLogEntry), + + #[serde(rename = "audit")] + Audit(Box), + + #[serde(rename = "console")] + Console(ConsoleLogEntry), +} + +impl LogRecord for UnifiedLogEntry { + fn to_json(&self) -> String { + match self { + UnifiedLogEntry::Server(entry) => entry.to_json(), + UnifiedLogEntry::Audit(entry) => entry.to_json(), + UnifiedLogEntry::Console(entry) => entry.to_json(), + } + } + + fn get_timestamp(&self) -> DateTime { + match self { + UnifiedLogEntry::Server(entry) => entry.get_timestamp(), + UnifiedLogEntry::Audit(entry) => entry.get_timestamp(), + UnifiedLogEntry::Console(entry) => entry.get_timestamp(), + } + } +} diff --git a/crates/audit-logger/src/lib.rs b/crates/audit-logger/src/lib.rs new file mode 100644 index 00000000..afa0422b --- /dev/null +++ b/crates/audit-logger/src/lib.rs @@ -0,0 +1,8 @@ +mod entry; +mod logger; + +pub use entry::args::Args; +pub use entry::audit::{ApiDetails, AuditLogEntry}; +pub use entry::base::BaseLogEntry; +pub use entry::unified::{ConsoleLogEntry, ServerLogEntry, UnifiedLogEntry}; +pub use entry::{LogKind, LogRecord, ObjectVersion, SerializableLevel}; diff --git a/crates/audit-logger/src/logger/config.rs b/crates/audit-logger/src/logger/config.rs new file mode 100644 index 00000000..d6f8656f --- /dev/null +++ b/crates/audit-logger/src/logger/config.rs @@ -0,0 +1,29 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![allow(dead_code)] + +// Default value function +fn default_batch_size() -> usize { + 10 +} +fn default_queue_size() -> usize { + 10000 +} +fn default_max_retry() -> u32 { + 5 +} +fn default_retry_interval() -> std::time::Duration { + std::time::Duration::from_secs(3) +} diff --git a/crates/audit-logger/src/logger/dispatch.rs b/crates/audit-logger/src/logger/dispatch.rs new file mode 100644 index 00000000..439d7a7c --- /dev/null +++ b/crates/audit-logger/src/logger/dispatch.rs @@ -0,0 +1,13 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. diff --git a/crates/audit-logger/src/logger/entry.rs b/crates/audit-logger/src/logger/entry.rs new file mode 100644 index 00000000..4d2d25a3 --- /dev/null +++ b/crates/audit-logger/src/logger/entry.rs @@ -0,0 +1,108 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![allow(dead_code)] + +use chrono::{DateTime, Utc}; +use serde::Serialize; +use std::collections::HashMap; +use uuid::Uuid; + +///A Trait for a log entry that can be serialized and sent +pub trait Loggable: Serialize + Send + Sync + 'static { + fn to_json(&self) -> Result { + serde_json::to_string(self) + } +} + +/// Standard log entries +#[derive(Serialize, Debug)] +#[serde(rename_all = "camelCase")] +pub struct LogEntry { + pub deployment_id: String, + pub level: String, + pub message: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub trace: Option, + pub time: DateTime, + pub request_id: String, +} + +impl Loggable for LogEntry {} + +/// Audit log entry +#[derive(Serialize, Debug)] +#[serde(rename_all = "camelCase")] +pub struct AuditEntry { + pub version: String, + pub deployment_id: String, + pub time: DateTime, + pub trigger: String, + pub api: ApiDetails, + pub remote_host: String, + pub request_id: String, + pub user_agent: String, + pub access_key: String, + #[serde(skip_serializing_if = "HashMap::is_empty")] + pub tags: HashMap, +} + +impl Loggable for AuditEntry {} + +#[derive(Serialize, Debug)] +#[serde(rename_all = "camelCase")] +pub struct Trace { + pub message: String, + pub source: Vec, + #[serde(skip_serializing_if = "HashMap::is_empty")] + pub variables: HashMap, +} + +#[derive(Serialize, Debug)] +#[serde(rename_all = "camelCase")] +pub struct ApiDetails { + pub name: String, + pub bucket: String, + pub object: String, + pub status: String, + pub status_code: u16, + pub time_to_first_byte: String, + pub time_to_response: String, +} + +// Helper functions to create entries +impl AuditEntry { + pub fn new(api_name: &str, bucket: &str, object: &str) -> Self { + AuditEntry { + version: "1".to_string(), + deployment_id: "global-deployment-id".to_string(), + time: Utc::now(), + trigger: "incoming".to_string(), + api: ApiDetails { + name: api_name.to_string(), + bucket: bucket.to_string(), + object: object.to_string(), + status: "OK".to_string(), + status_code: 200, + time_to_first_byte: "10ms".to_string(), + time_to_response: "50ms".to_string(), + }, + remote_host: "127.0.0.1".to_string(), + request_id: Uuid::new_v4().to_string(), + user_agent: "Rust-Client/1.0".to_string(), + access_key: "minioadmin".to_string(), + tags: HashMap::new(), + } + } +} diff --git a/crates/audit-logger/src/logger/factory.rs b/crates/audit-logger/src/logger/factory.rs new file mode 100644 index 00000000..439d7a7c --- /dev/null +++ b/crates/audit-logger/src/logger/factory.rs @@ -0,0 +1,13 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. diff --git a/crates/audit-logger/src/logger/mod.rs b/crates/audit-logger/src/logger/mod.rs new file mode 100644 index 00000000..a192fbcd --- /dev/null +++ b/crates/audit-logger/src/logger/mod.rs @@ -0,0 +1,36 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +#![allow(dead_code)] + +pub mod config; +pub mod dispatch; +pub mod entry; +pub mod factory; + +use async_trait::async_trait; +use std::error::Error; + +/// General Log Target Trait +#[async_trait] +pub trait Target: Send + Sync { + /// Send a single logizable entry + async fn send(&self, entry: Box) -> Result<(), Box>; + + /// Returns the unique name of the target + fn name(&self) -> &str; + + /// Close target gracefully, ensuring all buffered logs are processed + async fn shutdown(&self); +} diff --git a/crates/config/Cargo.toml b/crates/config/Cargo.toml index c9095b31..7cf5ef86 100644 --- a/crates/config/Cargo.toml +++ b/crates/config/Cargo.toml @@ -31,8 +31,9 @@ const-str = { workspace = true, optional = true } workspace = true [features] -default = [] +default = ["constants"] +audit = ["dep:const-str", "constants"] constants = ["dep:const-str"] -notify = ["dep:const-str"] -observability = [] +notify = ["dep:const-str", "constants"] +observability = ["constants"] diff --git a/crates/config/src/audit/mod.rs b/crates/config/src/audit/mod.rs new file mode 100644 index 00000000..19ebbdde --- /dev/null +++ b/crates/config/src/audit/mod.rs @@ -0,0 +1,31 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Audit configuration module +//! //! This module defines the configuration for audit systems, including +//! webhook and other audit-related settings. +pub const AUDIT_WEBHOOK_SUB_SYS: &str = "audit_webhook"; + +pub const AUDIT_STORE_EXTENSION: &str = ".audit"; + +pub const WEBHOOK_ENDPOINT: &str = "endpoint"; +pub const WEBHOOK_AUTH_TOKEN: &str = "auth_token"; +pub const WEBHOOK_CLIENT_CERT: &str = "client_cert"; +pub const WEBHOOK_CLIENT_KEY: &str = "client_key"; +pub const WEBHOOK_BATCH_SIZE: &str = "batch_size"; +pub const WEBHOOK_QUEUE_SIZE: &str = "queue_size"; +pub const WEBHOOK_QUEUE_DIR: &str = "queue_dir"; +pub const WEBHOOK_MAX_RETRY: &str = "max_retry"; +pub const WEBHOOK_RETRY_INTERVAL: &str = "retry_interval"; +pub const WEBHOOK_HTTP_TIMEOUT: &str = "http_timeout"; diff --git a/crates/config/src/constants/env.rs b/crates/config/src/constants/env.rs index 22eae738..e78c2b90 100644 --- a/crates/config/src/constants/env.rs +++ b/crates/config/src/constants/env.rs @@ -16,6 +16,13 @@ pub const DEFAULT_DELIMITER: &str = "_"; pub const ENV_PREFIX: &str = "RUSTFS_"; pub const ENV_WORD_DELIMITER: &str = "_"; +pub const DEFAULT_DIR: &str = "/opt/rustfs/events"; // Default directory for event store +pub const DEFAULT_LIMIT: u64 = 100000; // Default store limit + +/// Standard config keys and values. +pub const ENABLE_KEY: &str = "enable"; +pub const COMMENT_KEY: &str = "comment"; + /// Medium-drawn lines separator /// This is used to separate words in environment variable names. pub const ENV_WORD_DELIMITER_DASH: &str = "-"; diff --git a/crates/config/src/lib.rs b/crates/config/src/lib.rs index 1c2afecd..98bd42c1 100644 --- a/crates/config/src/lib.rs +++ b/crates/config/src/lib.rs @@ -20,6 +20,8 @@ pub use constants::app::*; pub use constants::env::*; #[cfg(feature = "constants")] pub use constants::tls::*; +#[cfg(feature = "audit")] +pub mod audit; #[cfg(feature = "notify")] pub mod notify; #[cfg(feature = "observability")] diff --git a/crates/config/src/notify/mod.rs b/crates/config/src/notify/mod.rs index 09d8f6f6..ec86985d 100644 --- a/crates/config/src/notify/mod.rs +++ b/crates/config/src/notify/mod.rs @@ -29,14 +29,6 @@ pub const NOTIFY_PREFIX: &str = "notify"; pub const NOTIFY_ROUTE_PREFIX: &str = const_str::concat!(NOTIFY_PREFIX, "_"); -/// Standard config keys and values. -pub const ENABLE_KEY: &str = "enable"; -pub const COMMENT_KEY: &str = "comment"; - -/// Enable values -pub const ENABLE_ON: &str = "on"; -pub const ENABLE_OFF: &str = "off"; - #[allow(dead_code)] pub const NOTIFY_SUB_SYSTEMS: &[&str] = &[NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS]; diff --git a/crates/config/src/notify/mqtt.rs b/crates/config/src/notify/mqtt.rs index ba5ed6b1..0282d82e 100644 --- a/crates/config/src/notify/mqtt.rs +++ b/crates/config/src/notify/mqtt.rs @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::notify::{COMMENT_KEY, ENABLE_KEY}; +use crate::{COMMENT_KEY, ENABLE_KEY}; // MQTT Keys pub const MQTT_BROKER: &str = "broker"; diff --git a/crates/config/src/notify/store.rs b/crates/config/src/notify/store.rs index 5e3dd51c..ed838b05 100644 --- a/crates/config/src/notify/store.rs +++ b/crates/config/src/notify/store.rs @@ -12,8 +12,6 @@ // See the License for the specific language governing permissions and // limitations under the License. -pub const DEFAULT_DIR: &str = "/opt/rustfs/events"; // Default directory for event store -pub const DEFAULT_LIMIT: u64 = 100000; // Default store limit pub const DEFAULT_EXT: &str = ".unknown"; // Default file extension pub const COMPRESS_EXT: &str = ".snappy"; // Extension for compressed files diff --git a/crates/config/src/notify/webhook.rs b/crates/config/src/notify/webhook.rs index b4fefb5f..bbb78716 100644 --- a/crates/config/src/notify/webhook.rs +++ b/crates/config/src/notify/webhook.rs @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::notify::{COMMENT_KEY, ENABLE_KEY}; +use crate::{COMMENT_KEY, ENABLE_KEY}; // Webhook Keys pub const WEBHOOK_ENDPOINT: &str = "endpoint"; diff --git a/crates/ecstore/Cargo.toml b/crates/ecstore/Cargo.toml index 90ea21ab..37559a18 100644 --- a/crates/ecstore/Cargo.toml +++ b/crates/ecstore/Cargo.toml @@ -34,7 +34,7 @@ workspace = true default = [] [dependencies] -rustfs-config = { workspace = true, features = ["constants", "notify"] } +rustfs-config = { workspace = true, features = ["constants", "notify", "audit"] } async-trait.workspace = true bytes.workspace = true byteorder = { workspace = true } diff --git a/crates/ecstore/src/config/audit.rs b/crates/ecstore/src/config/audit.rs new file mode 100644 index 00000000..dde1cbaa --- /dev/null +++ b/crates/ecstore/src/config/audit.rs @@ -0,0 +1,84 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use crate::config::{KV, KVS}; +use rustfs_config::audit::{ + WEBHOOK_AUTH_TOKEN, WEBHOOK_BATCH_SIZE, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_HTTP_TIMEOUT, + WEBHOOK_MAX_RETRY, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_SIZE, WEBHOOK_RETRY_INTERVAL, +}; +use rustfs_config::{DEFAULT_DIR, DEFAULT_LIMIT, ENABLE_KEY, EnableState}; +use std::sync::LazyLock; + +#[allow(dead_code)] +#[allow(clippy::declare_interior_mutable_const)] +/// Default KVS for audit webhook settings. +pub const DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock = LazyLock::new(|| { + KVS(vec![ + KV { + key: ENABLE_KEY.to_owned(), + value: EnableState::Off.to_string(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_ENDPOINT.to_owned(), + value: "".to_owned(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_AUTH_TOKEN.to_owned(), + value: "".to_owned(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_CLIENT_CERT.to_owned(), + value: "".to_owned(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_CLIENT_KEY.to_owned(), + value: "".to_owned(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_BATCH_SIZE.to_owned(), + value: "1".to_owned(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_QUEUE_SIZE.to_owned(), + value: DEFAULT_LIMIT.to_string(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_QUEUE_DIR.to_owned(), + value: DEFAULT_DIR.to_owned(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_MAX_RETRY.to_owned(), + value: "0".to_owned(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_RETRY_INTERVAL.to_owned(), + value: "3s".to_owned(), + hidden_if_empty: false, + }, + KV { + key: WEBHOOK_HTTP_TIMEOUT.to_owned(), + value: "5s".to_owned(), + hidden_if_empty: false, + }, + ]) +}); diff --git a/crates/ecstore/src/config/mod.rs b/crates/ecstore/src/config/mod.rs index e957b7d8..f4562b38 100644 --- a/crates/ecstore/src/config/mod.rs +++ b/crates/ecstore/src/config/mod.rs @@ -12,6 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. +mod audit; pub mod com; #[allow(dead_code)] pub mod heal; @@ -21,8 +22,9 @@ pub mod storageclass; use crate::error::Result; use crate::store::ECStore; use com::{STORAGE_CLASS_SUB_SYS, lookup_configs, read_config_without_migrate}; +use rustfs_config::COMMENT_KEY; use rustfs_config::DEFAULT_DELIMITER; -use rustfs_config::notify::{COMMENT_KEY, NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS}; +use rustfs_config::notify::{NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS}; use serde::{Deserialize, Serialize}; use std::collections::HashMap; use std::sync::LazyLock; diff --git a/crates/ecstore/src/config/notify.rs b/crates/ecstore/src/config/notify.rs index 2e1dedc2..d9c7b3d8 100644 --- a/crates/ecstore/src/config/notify.rs +++ b/crates/ecstore/src/config/notify.rs @@ -14,10 +14,11 @@ use crate::config::{KV, KVS}; use rustfs_config::notify::{ - COMMENT_KEY, DEFAULT_DIR, DEFAULT_LIMIT, ENABLE_KEY, ENABLE_OFF, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, - MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TOPIC, MQTT_USERNAME, WEBHOOK_AUTH_TOKEN, - WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, + MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, + MQTT_TOPIC, MQTT_USERNAME, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, + WEBHOOK_QUEUE_LIMIT, }; +use rustfs_config::{COMMENT_KEY, DEFAULT_DIR, DEFAULT_LIMIT, ENABLE_KEY, EnableState}; use std::sync::LazyLock; /// The default configuration collection of webhooks， @@ -26,7 +27,7 @@ pub static DEFAULT_WEBHOOK_KVS: LazyLock = LazyLock::new(|| { KVS(vec![ KV { key: ENABLE_KEY.to_owned(), - value: ENABLE_OFF.to_owned(), + value: EnableState::Off.to_string(), hidden_if_empty: false, }, KV { @@ -73,7 +74,7 @@ pub static DEFAULT_MQTT_KVS: LazyLock = LazyLock::new(|| { KVS(vec![ KV { key: ENABLE_KEY.to_owned(), - value: ENABLE_OFF.to_owned(), + value: EnableState::Off.to_string(), hidden_if_empty: false, }, KV { diff --git a/crates/notify/Cargo.toml b/crates/notify/Cargo.toml index c98bd9ce..541ac861 100644 --- a/crates/notify/Cargo.toml +++ b/crates/notify/Cargo.toml @@ -29,6 +29,7 @@ documentation = "https://docs.rs/rustfs-notify/latest/rustfs_notify/" rustfs-config = { workspace = true, features = ["notify", "constants"] } rustfs-ecstore = { workspace = true } rustfs-utils = { workspace = true, features = ["path", "sys"] } +rustfs-targets = { workspace = true } async-trait = { workspace = true } chrono = { workspace = true, features = ["serde"] } dashmap = { workspace = true } @@ -36,23 +37,18 @@ futures = { workspace = true } form_urlencoded = { workspace = true } once_cell = { workspace = true } quick-xml = { workspace = true, features = ["serialize", "async-tokio"] } -reqwest = { workspace = true } rumqttc = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } -snap = { workspace = true } thiserror = { workspace = true } tokio = { workspace = true, features = ["rt-multi-thread", "sync", "time"] } tracing = { workspace = true } -tracing-subscriber = { workspace = true, features = ["env-filter"] } -uuid = { workspace = true, features = ["v4", "serde"] } url = { workspace = true } -urlencoding = { workspace = true } wildmatch = { workspace = true, features = ["serde"] } [dev-dependencies] tokio = { workspace = true, features = ["test-util"] } -reqwest = { workspace = true } +tracing-subscriber = { workspace = true, features = ["env-filter"] } axum = { workspace = true } [lints] diff --git a/crates/notify/examples/base.rs b/crates/notify/examples/base.rs new file mode 100644 index 00000000..3ec51aaf --- /dev/null +++ b/crates/notify/examples/base.rs @@ -0,0 +1,59 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::io::IsTerminal; +use tracing_subscriber::{EnvFilter, fmt, prelude::*, util::SubscriberInitExt}; + +#[allow(dead_code)] +fn main() { + init_logger(LogLevel::Info); + tracing::info!("Tracing logger initialized with Info level"); +} + +/// Initialize the tracing log system +pub fn init_logger(level: LogLevel) { + let filter = EnvFilter::default().add_directive(level.into()); + tracing_subscriber::registry() + .with(filter) + .with( + fmt::layer() + .with_target(true) + .with_target(true) + .with_ansi(std::io::stdout().is_terminal()) + .with_thread_names(true) + .with_thread_ids(true) + .with_file(true) + .with_line_number(true), + ) + .init(); +} + +/// Log level definition +pub enum LogLevel { + Debug, + Info, + Warn, + Error, +} + +impl From for tracing_subscriber::filter::Directive { + fn from(level: LogLevel) -> Self { + match level { + LogLevel::Debug => "debug".parse().unwrap(), + LogLevel::Info => "info".parse().unwrap(), + LogLevel::Warn => "warn".parse().unwrap(), + LogLevel::Error => "error".parse().unwrap(), + } + } +} diff --git a/crates/notify/examples/full_demo.rs b/crates/notify/examples/full_demo.rs index 4eb93df9..458e9ef4 100644 --- a/crates/notify/examples/full_demo.rs +++ b/crates/notify/examples/full_demo.rs @@ -12,15 +12,20 @@ // See the License for the specific language governing permissions and // limitations under the License. +mod base; + +use base::{LogLevel, init_logger}; +use rustfs_config::EnableState::On; use rustfs_config::notify::{ - DEFAULT_LIMIT, DEFAULT_TARGET, ENABLE_KEY, ENABLE_ON, MQTT_BROKER, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, - MQTT_TOPIC, MQTT_USERNAME, NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS, WEBHOOK_AUTH_TOKEN, WEBHOOK_ENDPOINT, - WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, + DEFAULT_TARGET, MQTT_BROKER, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_TOPIC, MQTT_USERNAME, + NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS, WEBHOOK_AUTH_TOKEN, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, }; +use rustfs_config::{DEFAULT_LIMIT, ENABLE_KEY}; use rustfs_ecstore::config::{Config, KV, KVS}; -use rustfs_notify::arn::TargetID; -use rustfs_notify::{BucketNotificationConfig, Event, EventName, LogLevel, NotificationError, init_logger}; +use rustfs_notify::{BucketNotificationConfig, Event, NotificationError}; use rustfs_notify::{initialize, notification_system}; +use rustfs_targets::EventName; +use rustfs_targets::arn::TargetID; use std::sync::Arc; use std::time::Duration; use tracing::info; @@ -46,7 +51,7 @@ async fn main() -> Result<(), NotificationError> { let webhook_kvs_vec = vec![ KV { key: ENABLE_KEY.to_string(), - value: ENABLE_ON.to_string(), + value: On.to_string(), hidden_if_empty: false, }, KV { @@ -85,7 +90,7 @@ async fn main() -> Result<(), NotificationError> { let mqtt_kvs_vec = vec![ KV { key: ENABLE_KEY.to_string(), - value: ENABLE_ON.to_string(), + value: On.to_string(), hidden_if_empty: false, }, KV { @@ -136,6 +141,10 @@ async fn main() -> Result<(), NotificationError> { // Load the configuration and initialize the system *system.config.write().await = config; + info!("---> Initializing notification system with Webhook and MQTT targets..."); + info!("Webhook Endpoint: {}", WEBHOOK_ENDPOINT); + info!("MQTT Broker: {}", MQTT_BROKER); + info!("system.init config: {:?}", system.config.read().await); system.init().await?; info!("✅ System initialized with Webhook and MQTT targets."); diff --git a/crates/notify/examples/full_demo_one.rs b/crates/notify/examples/full_demo_one.rs index 5d06eef9..20b77124 100644 --- a/crates/notify/examples/full_demo_one.rs +++ b/crates/notify/examples/full_demo_one.rs @@ -12,16 +12,20 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Using Global Accessories +mod base; + +use base::{LogLevel, init_logger}; +use rustfs_config::EnableState::On; use rustfs_config::notify::{ - DEFAULT_LIMIT, DEFAULT_TARGET, ENABLE_KEY, ENABLE_ON, MQTT_BROKER, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, - MQTT_TOPIC, MQTT_USERNAME, NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS, WEBHOOK_AUTH_TOKEN, WEBHOOK_ENDPOINT, - WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, + DEFAULT_TARGET, MQTT_BROKER, MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_TOPIC, MQTT_USERNAME, + NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS, WEBHOOK_AUTH_TOKEN, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, }; +use rustfs_config::{DEFAULT_LIMIT, ENABLE_KEY}; use rustfs_ecstore::config::{Config, KV, KVS}; -use rustfs_notify::arn::TargetID; -use rustfs_notify::{BucketNotificationConfig, Event, EventName, LogLevel, NotificationError, init_logger}; +use rustfs_notify::{BucketNotificationConfig, Event, NotificationError}; use rustfs_notify::{initialize, notification_system}; +use rustfs_targets::EventName; +use rustfs_targets::arn::TargetID; use std::sync::Arc; use std::time::Duration; use tracing::info; @@ -47,7 +51,7 @@ async fn main() -> Result<(), NotificationError> { let webhook_kvs_vec = vec![ KV { key: ENABLE_KEY.to_string(), - value: ENABLE_ON.to_string(), + value: On.to_string(), hidden_if_empty: false, }, KV { @@ -95,7 +99,7 @@ async fn main() -> Result<(), NotificationError> { let mqtt_kvs_vec = vec![ KV { key: ENABLE_KEY.to_string(), - value: ENABLE_ON.to_string(), + value: On.to_string(), hidden_if_empty: false, }, KV { diff --git a/crates/notify/src/error.rs b/crates/notify/src/error.rs index 8f5fcf0f..b06c6451 100644 --- a/crates/notify/src/error.rs +++ b/crates/notify/src/error.rs @@ -1,98 +1,22 @@ -// Copyright 2024 RustFS Team +// Copyright 2024 RustFS Team // -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at // -// http://www.apache.org/licenses/LICENSE-2.0 +// http://www.apache.org/licenses/LICENSE-2.0 // -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. -use crate::arn::TargetID; +use rustfs_targets::TargetError; +use rustfs_targets::arn::TargetID; use std::io; use thiserror::Error; -/// Error types for the store -#[derive(Debug, Error)] -pub enum StoreError { - #[error("I/O error: {0}")] - Io(#[from] io::Error), - - #[error("Serialization error: {0}")] - Serialization(String), - - #[error("Deserialization error: {0}")] - Deserialization(String), - - #[error("Compression error: {0}")] - Compression(String), - - #[error("Entry limit exceeded")] - LimitExceeded, - - #[error("Entry not found")] - NotFound, - - #[error("Invalid entry: {0}")] - Internal(String), // Added internal error type -} - -/// Error types for targets -#[derive(Debug, Error)] -pub enum TargetError { - #[error("Storage error: {0}")] - Storage(String), - - #[error("Network error: {0}")] - Network(String), - - #[error("Request error: {0}")] - Request(String), - - #[error("Timeout error: {0}")] - Timeout(String), - - #[error("Authentication error: {0}")] - Authentication(String), - - #[error("Configuration error: {0}")] - Configuration(String), - - #[error("Encoding error: {0}")] - Encoding(String), - - #[error("Serialization error: {0}")] - Serialization(String), - - #[error("Target not connected")] - NotConnected, - - #[error("Target initialization failed: {0}")] - Initialization(String), - - #[error("Invalid ARN: {0}")] - InvalidARN(String), - - #[error("Unknown error: {0}")] - Unknown(String), - - #[error("Target is disabled")] - Disabled, - - #[error("Configuration parsing error: {0}")] - ParseError(String), - - #[error("Failed to save configuration: {0}")] - SaveConfig(String), - - #[error("Server not initialized: {0}")] - ServerNotInitialized(String), -} - /// Error types for the notification system #[derive(Debug, Error)] pub enum NotificationError { @@ -135,9 +59,3 @@ pub enum NotificationError { #[error("Server not initialized")] ServerNotInitialized, } - -impl From for TargetError { - fn from(err: url::ParseError) -> Self { - TargetError::Configuration(format!("URL parse error: {err}")) - } -} diff --git a/crates/notify/src/event.rs b/crates/notify/src/event.rs index 308de9d9..6bbda17c 100644 --- a/crates/notify/src/event.rs +++ b/crates/notify/src/event.rs @@ -13,285 +13,11 @@ // limitations under the License. use chrono::{DateTime, Utc}; +use rustfs_targets::EventName; use serde::{Deserialize, Serialize}; use std::collections::HashMap; -use std::fmt; use url::form_urlencoded; -/// Error returned when parsing event name string fails。 -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct ParseEventNameError(String); - -impl fmt::Display for ParseEventNameError { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "Invalid event name:{}", self.0) - } -} - -impl std::error::Error for ParseEventNameError {} - -/// Represents the type of event that occurs on the object. -/// Based on AWS S3 event type and includes RustFS extension. -#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Hash)] -pub enum EventName { - // Single event type (values are 1-32 for compatible mask logic) - ObjectAccessedGet = 1, - ObjectAccessedGetRetention = 2, - ObjectAccessedGetLegalHold = 3, - ObjectAccessedHead = 4, - ObjectAccessedAttributes = 5, - ObjectCreatedCompleteMultipartUpload = 6, - ObjectCreatedCopy = 7, - ObjectCreatedPost = 8, - ObjectCreatedPut = 9, - ObjectCreatedPutRetention = 10, - ObjectCreatedPutLegalHold = 11, - ObjectCreatedPutTagging = 12, - ObjectCreatedDeleteTagging = 13, - ObjectRemovedDelete = 14, - ObjectRemovedDeleteMarkerCreated = 15, - ObjectRemovedDeleteAllVersions = 16, - ObjectRemovedNoOP = 17, - BucketCreated = 18, - BucketRemoved = 19, - ObjectReplicationFailed = 20, - ObjectReplicationComplete = 21, - ObjectReplicationMissedThreshold = 22, - ObjectReplicationReplicatedAfterThreshold = 23, - ObjectReplicationNotTracked = 24, - ObjectRestorePost = 25, - ObjectRestoreCompleted = 26, - ObjectTransitionFailed = 27, - ObjectTransitionComplete = 28, - ScannerManyVersions = 29, // ObjectManyVersions corresponding to Go - ScannerLargeVersions = 30, // ObjectLargeVersions corresponding to Go - ScannerBigPrefix = 31, // PrefixManyFolders corresponding to Go - LifecycleDelMarkerExpirationDelete = 32, // ILMDelMarkerExpirationDelete corresponding to Go - - // Compound "All" event type (no sequential value for mask) - ObjectAccessedAll, - ObjectCreatedAll, - ObjectRemovedAll, - ObjectReplicationAll, - ObjectRestoreAll, - ObjectTransitionAll, - ObjectScannerAll, // New, from Go - Everything, // New, from Go -} - -// Single event type sequential array for Everything.expand() -const SINGLE_EVENT_NAMES_IN_ORDER: [EventName; 32] = [ - EventName::ObjectAccessedGet, - EventName::ObjectAccessedGetRetention, - EventName::ObjectAccessedGetLegalHold, - EventName::ObjectAccessedHead, - EventName::ObjectAccessedAttributes, - EventName::ObjectCreatedCompleteMultipartUpload, - EventName::ObjectCreatedCopy, - EventName::ObjectCreatedPost, - EventName::ObjectCreatedPut, - EventName::ObjectCreatedPutRetention, - EventName::ObjectCreatedPutLegalHold, - EventName::ObjectCreatedPutTagging, - EventName::ObjectCreatedDeleteTagging, - EventName::ObjectRemovedDelete, - EventName::ObjectRemovedDeleteMarkerCreated, - EventName::ObjectRemovedDeleteAllVersions, - EventName::ObjectRemovedNoOP, - EventName::BucketCreated, - EventName::BucketRemoved, - EventName::ObjectReplicationFailed, - EventName::ObjectReplicationComplete, - EventName::ObjectReplicationMissedThreshold, - EventName::ObjectReplicationReplicatedAfterThreshold, - EventName::ObjectReplicationNotTracked, - EventName::ObjectRestorePost, - EventName::ObjectRestoreCompleted, - EventName::ObjectTransitionFailed, - EventName::ObjectTransitionComplete, - EventName::ScannerManyVersions, - EventName::ScannerLargeVersions, - EventName::ScannerBigPrefix, - EventName::LifecycleDelMarkerExpirationDelete, -]; - -const LAST_SINGLE_TYPE_VALUE: u32 = EventName::LifecycleDelMarkerExpirationDelete as u32; - -impl EventName { - /// The parsed string is EventName. - pub fn parse(s: &str) -> Result { - match s { - "s3:BucketCreated:*" => Ok(EventName::BucketCreated), - "s3:BucketRemoved:*" => Ok(EventName::BucketRemoved), - "s3:ObjectAccessed:*" => Ok(EventName::ObjectAccessedAll), - "s3:ObjectAccessed:Get" => Ok(EventName::ObjectAccessedGet), - "s3:ObjectAccessed:GetRetention" => Ok(EventName::ObjectAccessedGetRetention), - "s3:ObjectAccessed:GetLegalHold" => Ok(EventName::ObjectAccessedGetLegalHold), - "s3:ObjectAccessed:Head" => Ok(EventName::ObjectAccessedHead), - "s3:ObjectAccessed:Attributes" => Ok(EventName::ObjectAccessedAttributes), - "s3:ObjectCreated:*" => Ok(EventName::ObjectCreatedAll), - "s3:ObjectCreated:CompleteMultipartUpload" => Ok(EventName::ObjectCreatedCompleteMultipartUpload), - "s3:ObjectCreated:Copy" => Ok(EventName::ObjectCreatedCopy), - "s3:ObjectCreated:Post" => Ok(EventName::ObjectCreatedPost), - "s3:ObjectCreated:Put" => Ok(EventName::ObjectCreatedPut), - "s3:ObjectCreated:PutRetention" => Ok(EventName::ObjectCreatedPutRetention), - "s3:ObjectCreated:PutLegalHold" => Ok(EventName::ObjectCreatedPutLegalHold), - "s3:ObjectCreated:PutTagging" => Ok(EventName::ObjectCreatedPutTagging), - "s3:ObjectCreated:DeleteTagging" => Ok(EventName::ObjectCreatedDeleteTagging), - "s3:ObjectRemoved:*" => Ok(EventName::ObjectRemovedAll), - "s3:ObjectRemoved:Delete" => Ok(EventName::ObjectRemovedDelete), - "s3:ObjectRemoved:DeleteMarkerCreated" => Ok(EventName::ObjectRemovedDeleteMarkerCreated), - "s3:ObjectRemoved:NoOP" => Ok(EventName::ObjectRemovedNoOP), - "s3:ObjectRemoved:DeleteAllVersions" => Ok(EventName::ObjectRemovedDeleteAllVersions), - "s3:LifecycleDelMarkerExpiration:Delete" => Ok(EventName::LifecycleDelMarkerExpirationDelete), - "s3:Replication:*" => Ok(EventName::ObjectReplicationAll), - "s3:Replication:OperationFailedReplication" => Ok(EventName::ObjectReplicationFailed), - "s3:Replication:OperationCompletedReplication" => Ok(EventName::ObjectReplicationComplete), - "s3:Replication:OperationMissedThreshold" => Ok(EventName::ObjectReplicationMissedThreshold), - "s3:Replication:OperationReplicatedAfterThreshold" => Ok(EventName::ObjectReplicationReplicatedAfterThreshold), - "s3:Replication:OperationNotTracked" => Ok(EventName::ObjectReplicationNotTracked), - "s3:ObjectRestore:*" => Ok(EventName::ObjectRestoreAll), - "s3:ObjectRestore:Post" => Ok(EventName::ObjectRestorePost), - "s3:ObjectRestore:Completed" => Ok(EventName::ObjectRestoreCompleted), - "s3:ObjectTransition:Failed" => Ok(EventName::ObjectTransitionFailed), - "s3:ObjectTransition:Complete" => Ok(EventName::ObjectTransitionComplete), - "s3:ObjectTransition:*" => Ok(EventName::ObjectTransitionAll), - "s3:Scanner:ManyVersions" => Ok(EventName::ScannerManyVersions), - "s3:Scanner:LargeVersions" => Ok(EventName::ScannerLargeVersions), - "s3:Scanner:BigPrefix" => Ok(EventName::ScannerBigPrefix), - // ObjectScannerAll and Everything cannot be parsed from strings, because the Go version also does not define their string representation. - _ => Err(ParseEventNameError(s.to_string())), - } - } - - /// Returns a string representation of the event type. - pub fn as_str(&self) -> &'static str { - match self { - EventName::BucketCreated => "s3:BucketCreated:*", - EventName::BucketRemoved => "s3:BucketRemoved:*", - EventName::ObjectAccessedAll => "s3:ObjectAccessed:*", - EventName::ObjectAccessedGet => "s3:ObjectAccessed:Get", - EventName::ObjectAccessedGetRetention => "s3:ObjectAccessed:GetRetention", - EventName::ObjectAccessedGetLegalHold => "s3:ObjectAccessed:GetLegalHold", - EventName::ObjectAccessedHead => "s3:ObjectAccessed:Head", - EventName::ObjectAccessedAttributes => "s3:ObjectAccessed:Attributes", - EventName::ObjectCreatedAll => "s3:ObjectCreated:*", - EventName::ObjectCreatedCompleteMultipartUpload => "s3:ObjectCreated:CompleteMultipartUpload", - EventName::ObjectCreatedCopy => "s3:ObjectCreated:Copy", - EventName::ObjectCreatedPost => "s3:ObjectCreated:Post", - EventName::ObjectCreatedPut => "s3:ObjectCreated:Put", - EventName::ObjectCreatedPutTagging => "s3:ObjectCreated:PutTagging", - EventName::ObjectCreatedDeleteTagging => "s3:ObjectCreated:DeleteTagging", - EventName::ObjectCreatedPutRetention => "s3:ObjectCreated:PutRetention", - EventName::ObjectCreatedPutLegalHold => "s3:ObjectCreated:PutLegalHold", - EventName::ObjectRemovedAll => "s3:ObjectRemoved:*", - EventName::ObjectRemovedDelete => "s3:ObjectRemoved:Delete", - EventName::ObjectRemovedDeleteMarkerCreated => "s3:ObjectRemoved:DeleteMarkerCreated", - EventName::ObjectRemovedNoOP => "s3:ObjectRemoved:NoOP", - EventName::ObjectRemovedDeleteAllVersions => "s3:ObjectRemoved:DeleteAllVersions", - EventName::LifecycleDelMarkerExpirationDelete => "s3:LifecycleDelMarkerExpiration:Delete", - EventName::ObjectReplicationAll => "s3:Replication:*", - EventName::ObjectReplicationFailed => "s3:Replication:OperationFailedReplication", - EventName::ObjectReplicationComplete => "s3:Replication:OperationCompletedReplication", - EventName::ObjectReplicationNotTracked => "s3:Replication:OperationNotTracked", - EventName::ObjectReplicationMissedThreshold => "s3:Replication:OperationMissedThreshold", - EventName::ObjectReplicationReplicatedAfterThreshold => "s3:Replication:OperationReplicatedAfterThreshold", - EventName::ObjectRestoreAll => "s3:ObjectRestore:*", - EventName::ObjectRestorePost => "s3:ObjectRestore:Post", - EventName::ObjectRestoreCompleted => "s3:ObjectRestore:Completed", - EventName::ObjectTransitionAll => "s3:ObjectTransition:*", - EventName::ObjectTransitionFailed => "s3:ObjectTransition:Failed", - EventName::ObjectTransitionComplete => "s3:ObjectTransition:Complete", - EventName::ScannerManyVersions => "s3:Scanner:ManyVersions", - EventName::ScannerLargeVersions => "s3:Scanner:LargeVersions", - EventName::ScannerBigPrefix => "s3:Scanner:BigPrefix", - // Go's String() returns "" for ObjectScannerAll and Everything - EventName::ObjectScannerAll => "s3:Scanner:*", // Follow the pattern in Go Expand - EventName::Everything => "", // Go String() returns "" to unprocessed - } - } - - /// Returns the extended value of the abbreviation event type. - pub fn expand(&self) -> Vec { - match self { - EventName::ObjectAccessedAll => vec![ - EventName::ObjectAccessedGet, - EventName::ObjectAccessedHead, - EventName::ObjectAccessedGetRetention, - EventName::ObjectAccessedGetLegalHold, - EventName::ObjectAccessedAttributes, - ], - EventName::ObjectCreatedAll => vec![ - EventName::ObjectCreatedCompleteMultipartUpload, - EventName::ObjectCreatedCopy, - EventName::ObjectCreatedPost, - EventName::ObjectCreatedPut, - EventName::ObjectCreatedPutRetention, - EventName::ObjectCreatedPutLegalHold, - EventName::ObjectCreatedPutTagging, - EventName::ObjectCreatedDeleteTagging, - ], - EventName::ObjectRemovedAll => vec![ - EventName::ObjectRemovedDelete, - EventName::ObjectRemovedDeleteMarkerCreated, - EventName::ObjectRemovedNoOP, - EventName::ObjectRemovedDeleteAllVersions, - ], - EventName::ObjectReplicationAll => vec![ - EventName::ObjectReplicationFailed, - EventName::ObjectReplicationComplete, - EventName::ObjectReplicationNotTracked, - EventName::ObjectReplicationMissedThreshold, - EventName::ObjectReplicationReplicatedAfterThreshold, - ], - EventName::ObjectRestoreAll => vec![EventName::ObjectRestorePost, EventName::ObjectRestoreCompleted], - EventName::ObjectTransitionAll => vec![EventName::ObjectTransitionFailed, EventName::ObjectTransitionComplete], - EventName::ObjectScannerAll => vec![ - // New - EventName::ScannerManyVersions, - EventName::ScannerLargeVersions, - EventName::ScannerBigPrefix, - ], - EventName::Everything => { - // New - SINGLE_EVENT_NAMES_IN_ORDER.to_vec() - } - // A single type returns to itself directly - _ => vec![*self], - } - } - - /// Returns the mask of type. - /// The compound "All" type will be expanded. - pub fn mask(&self) -> u64 { - let value = *self as u32; - if value > 0 && value <= LAST_SINGLE_TYPE_VALUE { - // It's a single type - 1u64 << (value - 1) - } else { - // It's a compound type - let mut mask = 0u64; - for n in self.expand() { - mask |= n.mask(); // Recursively call mask - } - mask - } - } -} - -impl fmt::Display for EventName { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "{}", self.as_str()) - } -} - -/// Convert to `EventName` according to string -impl From<&str> for EventName { - fn from(event_str: &str) -> Self { - EventName::parse(event_str).unwrap_or_else(|e| panic!("{}", e)) - } -} - /// Represents the identity of the user who triggered the event #[derive(Debug, Clone, Serialize, Deserialize)] pub struct Identity { @@ -532,17 +258,6 @@ fn initialize_response_elements(elements: &mut HashMap, keys: &[ } } -/// Represents a log of events for sending to targets -#[derive(Debug, Clone, Serialize, Deserialize)] -pub struct EventLog { - /// The event name - pub event_name: EventName, - /// The object key - pub key: String, - /// The list of events - pub records: Vec, -} - #[derive(Debug, Clone)] pub struct EventArgs { pub event_name: EventName, diff --git a/crates/notify/src/factory.rs b/crates/notify/src/factory.rs index ada2de48..77945c37 100644 --- a/crates/notify/src/factory.rs +++ b/crates/notify/src/factory.rs @@ -12,19 +12,22 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::{ - error::TargetError, - target::{Target, mqtt::MQTTArgs, webhook::WebhookArgs}, -}; use async_trait::async_trait; use rumqttc::QoS; use rustfs_config::notify::{ - DEFAULT_DIR, DEFAULT_LIMIT, ENV_NOTIFY_MQTT_KEYS, ENV_NOTIFY_WEBHOOK_KEYS, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, - MQTT_PASSWORD, MQTT_QOS, MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TOPIC, MQTT_USERNAME, - NOTIFY_MQTT_KEYS, NOTIFY_WEBHOOK_KEYS, WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, - WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, + ENV_NOTIFY_MQTT_KEYS, ENV_NOTIFY_WEBHOOK_KEYS, MQTT_BROKER, MQTT_KEEP_ALIVE_INTERVAL, MQTT_PASSWORD, MQTT_QOS, + MQTT_QUEUE_DIR, MQTT_QUEUE_LIMIT, MQTT_RECONNECT_INTERVAL, MQTT_TOPIC, MQTT_USERNAME, NOTIFY_MQTT_KEYS, NOTIFY_WEBHOOK_KEYS, + WEBHOOK_AUTH_TOKEN, WEBHOOK_CLIENT_CERT, WEBHOOK_CLIENT_KEY, WEBHOOK_ENDPOINT, WEBHOOK_QUEUE_DIR, WEBHOOK_QUEUE_LIMIT, }; + +use crate::Event; +use rustfs_config::{DEFAULT_DIR, DEFAULT_LIMIT}; use rustfs_ecstore::config::KVS; +use rustfs_targets::{ + Target, + error::TargetError, + target::{mqtt::MQTTArgs, webhook::WebhookArgs}, +}; use std::collections::HashSet; use std::time::Duration; use tracing::{debug, warn}; @@ -34,7 +37,7 @@ use url::Url; #[async_trait] pub trait TargetFactory: Send + Sync { /// Creates a target from configuration - async fn create_target(&self, id: String, config: &KVS) -> Result, TargetError>; + async fn create_target(&self, id: String, config: &KVS) -> Result + Send + Sync>, TargetError>; /// Validates target configuration fn validate_config(&self, id: &str, config: &KVS) -> Result<(), TargetError>; @@ -53,7 +56,7 @@ pub struct WebhookTargetFactory; #[async_trait] impl TargetFactory for WebhookTargetFactory { - async fn create_target(&self, id: String, config: &KVS) -> Result, TargetError> { + async fn create_target(&self, id: String, config: &KVS) -> Result + Send + Sync>, TargetError> { // All config values are now read directly from the merged `config` KVS. let endpoint = config .lookup(WEBHOOK_ENDPOINT) @@ -72,9 +75,10 @@ impl TargetFactory for WebhookTargetFactory { .unwrap_or(DEFAULT_LIMIT), client_cert: config.lookup(WEBHOOK_CLIENT_CERT).unwrap_or_default(), client_key: config.lookup(WEBHOOK_CLIENT_KEY).unwrap_or_default(), + target_type: rustfs_targets::target::TargetType::NotifyEvent, }; - let target = crate::target::webhook::WebhookTarget::new(id, args)?; + let target = rustfs_targets::target::webhook::WebhookTarget::new(id, args)?; Ok(Box::new(target)) } @@ -119,7 +123,7 @@ pub struct MQTTTargetFactory; #[async_trait] impl TargetFactory for MQTTTargetFactory { - async fn create_target(&self, id: String, config: &KVS) -> Result, TargetError> { + async fn create_target(&self, id: String, config: &KVS) -> Result + Send + Sync>, TargetError> { let broker = config .lookup(MQTT_BROKER) .ok_or_else(|| TargetError::Configuration("Missing MQTT broker".to_string()))?; @@ -161,9 +165,10 @@ impl TargetFactory for MQTTTargetFactory { .lookup(MQTT_QUEUE_LIMIT) .and_then(|v| v.parse::().ok()) .unwrap_or(DEFAULT_LIMIT), + target_type: rustfs_targets::target::TargetType::NotifyEvent, }; - let target = crate::target::mqtt::MQTTTarget::new(id, args)?; + let target = rustfs_targets::target::mqtt::MQTTTarget::new(id, args)?; Ok(Box::new(target)) } diff --git a/crates/notify/src/global.rs b/crates/notify/src/global.rs index b55dd1f3..dd293d1c 100644 --- a/crates/notify/src/global.rs +++ b/crates/notify/src/global.rs @@ -12,11 +12,13 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::{Event, EventArgs, NotificationError, NotificationSystem}; +use crate::{BucketNotificationConfig, Event, EventArgs, NotificationError, NotificationSystem}; use once_cell::sync::Lazy; use rustfs_ecstore::config::Config; +use rustfs_targets::EventName; +use rustfs_targets::arn::TargetID; use std::sync::{Arc, OnceLock}; -use tracing::instrument; +use tracing::{error, instrument}; static NOTIFICATION_SYSTEM: OnceLock> = OnceLock::new(); // Create a globally unique Notifier instance @@ -57,6 +59,14 @@ pub struct Notifier {} impl Notifier { /// Notify an event asynchronously. /// This is the only entry point for all event notifications in the system. + /// # Parameter + /// - `args`: The event arguments containing details about the event to be notified. + /// + /// # Return value + /// Returns `()`, indicating that the notification has been sent. + /// + /// # Using + /// This function is used to notify events in the system, such as object creation, deletion, or updates. #[instrument(skip(self, args))] pub async fn notify(&self, args: EventArgs) { // Dependency injection or service positioning mode obtain NotificationSystem instance @@ -64,7 +74,7 @@ impl Notifier { // If the notification system itself cannot be retrieved, it will be returned directly Some(sys) => sys, None => { - tracing::error!("Notification system is not initialized."); + error!("Notification system is not initialized."); return; } }; @@ -76,6 +86,7 @@ impl Notifier { // Check if any subscribers are interested in the event if !notification_sys.has_subscriber(&args.bucket_name, &args.event_name).await { + error!("No subscribers for event: {} in bucket: {}", args.event_name, args.bucket_name); return; } @@ -83,4 +94,96 @@ impl Notifier { let event = Arc::new(Event::new(args)); notification_sys.send_event(event).await; } + + /// Add notification rules for the specified bucket and load configuration + /// # Parameter + /// - `bucket_name`: The name of the target bucket. + /// - `region`: The area where bucket is located. + /// - `event_names`: A list of event names that trigger notifications. + /// - `prefix`: The prefix of the object key that triggers notifications. + /// - `suffix`: The suffix of the object key that triggers notifications. + /// - `target_ids`: A list of target IDs that will receive notifications. + /// + /// # Return value + /// Returns `Result<(), NotificationError>`, Ok on success, and an error on failure + /// + /// # Using + /// This function allows you to dynamically add notification rules for a specific bucket. + pub async fn add_bucket_notification_rule( + &self, + bucket_name: &str, + region: &str, + event_names: &[EventName], + prefix: &str, + suffix: &str, + target_ids: &[TargetID], + ) -> Result<(), NotificationError> { + // Construct pattern, simple splicing of prefixes and suffixes + let mut pattern = String::new(); + if !prefix.is_empty() { + pattern.push_str(prefix); + } + pattern.push('*'); + if !suffix.is_empty() { + pattern.push_str(suffix); + } + + // Create BucketNotificationConfig + let mut bucket_config = BucketNotificationConfig::new(region); + for target_id in target_ids { + bucket_config.add_rule(event_names, pattern.clone(), target_id.clone()); + } + + // Get global NotificationSystem + let notification_sys = match notification_system() { + Some(sys) => sys, + None => return Err(NotificationError::ServerNotInitialized), + }; + + // Loading configuration + notification_sys + .load_bucket_notification_config(bucket_name, &bucket_config) + .await + } + + /// Dynamically add notification rules according to different event types. + /// + /// # Parameter + /// - `bucket_name`: The name of the target bucket. + /// - `region`: The area where bucket is located. + /// - `event_rules`: Each rule contains a list of event types, prefixes, suffixes, and target IDs. + /// + /// # Return value + /// Returns `Result<(), NotificationError>`, Ok on success, and an error on failure. + /// + /// # Using + /// Supports notification rules for adding multiple event types, prefixes, suffixes, and targets to the same bucket in batches. + pub async fn add_event_specific_rules( + &self, + bucket_name: &str, + region: &str, + event_rules: &[(Vec, &str, &str, Vec)], + ) -> Result<(), NotificationError> { + let mut bucket_config = BucketNotificationConfig::new(region); + + for (event_names, prefix, suffix, target_ids) in event_rules { + // Use `new_pattern` to construct a matching pattern + let pattern = crate::rules::pattern::new_pattern(Some(prefix), Some(suffix)); + + for target_id in target_ids { + bucket_config.add_rule(event_names, pattern.clone(), target_id.clone()); + } + } + + // Get global NotificationSystem instance + let notification_sys = match notification_system() { + Some(sys) => sys, + None => return Err(NotificationError::ServerNotInitialized), + }; + + // Loading configuration + notification_sys + .load_bucket_notification_config(bucket_name, &bucket_config) + .await + } } diff --git a/crates/notify/src/integration.rs b/crates/notify/src/integration.rs index 7f786285..d03454d2 100644 --- a/crates/notify/src/integration.rs +++ b/crates/notify/src/integration.rs @@ -12,13 +12,15 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::arn::TargetID; -use crate::store::{Key, Store}; use crate::{ - Event, EventName, StoreError, Target, error::NotificationError, notifier::EventNotifier, registry::TargetRegistry, - rules::BucketNotificationConfig, stream, + Event, error::NotificationError, notifier::EventNotifier, registry::TargetRegistry, rules::BucketNotificationConfig, stream, }; use rustfs_ecstore::config::{Config, KVS}; +use rustfs_targets::EventName; +use rustfs_targets::arn::TargetID; +use rustfs_targets::store::{Key, Store}; +use rustfs_targets::target::EntityTarget; +use rustfs_targets::{StoreError, Target}; use std::collections::HashMap; use std::sync::Arc; use std::sync::atomic::{AtomicUsize, Ordering}; @@ -127,7 +129,7 @@ impl NotificationSystem { let config = self.config.read().await; debug!("Initializing notification system with config: {:?}", *config); - let targets: Vec> = self.registry.create_targets_from_config(&config).await?; + let targets: Vec + Send + Sync>> = self.registry.create_targets_from_config(&config).await?; info!("{} notification targets were created", targets.len()); @@ -318,8 +320,8 @@ impl NotificationSystem { /// Enhanced event stream startup function, including monitoring and concurrency control fn enhanced_start_event_stream( &self, - store: Box + Send>, - target: Arc, + store: Box, Error = StoreError, Key = Key> + Send>, + target: Arc + Send + Sync>, metrics: Arc, semaphore: Arc, ) -> mpsc::Sender<()> { @@ -348,7 +350,7 @@ impl NotificationSystem { // Create a new target from configuration // This function will now be responsible for merging env, creating and persisting the final configuration. - let targets: Vec> = self + let targets: Vec + Send + Sync>> = self .registry .create_targets_from_config(&new_config) .await diff --git a/crates/notify/src/lib.rs b/crates/notify/src/lib.rs index 837020ca..5da8d58a 100644 --- a/crates/notify/src/lib.rs +++ b/crates/notify/src/lib.rs @@ -18,7 +18,6 @@ //! It supports sending events to various targets //! (like Webhook and MQTT) and includes features like event persistence and retry on failure. -pub mod arn; pub mod error; pub mod event; pub mod factory; @@ -27,59 +26,10 @@ pub mod integration; pub mod notifier; pub mod registry; pub mod rules; -pub mod store; pub mod stream; -pub mod target; - // Re-exports -pub use error::{NotificationError, StoreError, TargetError}; -pub use event::{Event, EventArgs, EventLog, EventName}; +pub use error::NotificationError; +pub use event::{Event, EventArgs}; pub use global::{initialize, is_notification_system_initialized, notification_system}; pub use integration::NotificationSystem; pub use rules::BucketNotificationConfig; -use std::io::IsTerminal; -pub use target::Target; - -use tracing_subscriber::{EnvFilter, fmt, prelude::*, util::SubscriberInitExt}; - -/// Initialize the tracing log system -/// -/// # Example -/// ``` -/// rustfs_notify::init_logger(rustfs_notify::LogLevel::Info); -/// ``` -pub fn init_logger(level: LogLevel) { - let filter = EnvFilter::default().add_directive(level.into()); - tracing_subscriber::registry() - .with(filter) - .with( - fmt::layer() - .with_target(true) - .with_target(true) - .with_ansi(std::io::stdout().is_terminal()) - .with_thread_names(true) - .with_thread_ids(true) - .with_file(true) - .with_line_number(true), - ) - .init(); -} - -/// Log level definition -pub enum LogLevel { - Debug, - Info, - Warn, - Error, -} - -impl From for tracing_subscriber::filter::Directive { - fn from(level: LogLevel) -> Self { - match level { - LogLevel::Debug => "debug".parse().unwrap(), - LogLevel::Info => "info".parse().unwrap(), - LogLevel::Warn => "warn".parse().unwrap(), - LogLevel::Error => "error".parse().unwrap(), - } - } -} diff --git a/crates/notify/src/notifier.rs b/crates/notify/src/notifier.rs index 24a765fd..c4586c69 100644 --- a/crates/notify/src/notifier.rs +++ b/crates/notify/src/notifier.rs @@ -12,9 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::arn::TargetID; -use crate::{EventName, error::NotificationError, event::Event, rules::RulesMap, target::Target}; +use crate::{error::NotificationError, event::Event, rules::RulesMap}; use dashmap::DashMap; +use rustfs_targets::EventName; +use rustfs_targets::Target; +use rustfs_targets::arn::TargetID; +use rustfs_targets::target::EntityTarget; use std::{collections::HashMap, sync::Arc}; use tokio::sync::RwLock; use tracing::{debug, error, info, instrument, warn}; @@ -121,7 +124,7 @@ impl EventNotifier { } /// Sends an event to the appropriate targets based on the bucket rules - #[instrument(skip(self, event))] + #[instrument(skip_all)] pub async fn send(&self, event: Arc) { let bucket_name = &event.s3.bucket.name; let object_key = &event.s3.object.key; @@ -149,8 +152,15 @@ impl EventNotifier { let target_name_for_task = cloned_target_for_task.name(); // Get the name before generating the task debug!("Preparing to send event to target: {}", target_name_for_task); // Use cloned data in closures to avoid borrowing conflicts + // Create an EntityTarget from the event + let entity_target: Arc> = Arc::new(EntityTarget { + object_name: object_key.to_string(), + bucket_name: bucket_name.to_string(), + event_name, + data: event_clone.clone().as_ref().clone(), + }); let handle = tokio::spawn(async move { - if let Err(e) = cloned_target_for_task.save(event_clone).await { + if let Err(e) = cloned_target_for_task.save(entity_target.clone()).await { error!("Failed to send event to target {}: {}", target_name_for_task, e); } else { debug!("Successfully saved event to target {}", target_name_for_task); @@ -180,7 +190,7 @@ impl EventNotifier { #[instrument(skip(self, targets_to_init))] pub async fn init_bucket_targets( &self, - targets_to_init: Vec>, + targets_to_init: Vec + Send + Sync>>, ) -> Result<(), NotificationError> { // Currently active, simpler logic let mut target_list_guard = self.target_list.write().await; //Gets a write lock for the TargetList @@ -189,7 +199,7 @@ impl EventNotifier { debug!("init bucket target: {}", target_boxed.name()); // TargetList::add method expectations Arc // Therefore, you need to convert Box to Arc - let target_arc: Arc = Arc::from(target_boxed); + let target_arc: Arc + Send + Sync> = Arc::from(target_boxed); target_list_guard.add(target_arc)?; // Add Arc to the list } info!( @@ -203,7 +213,7 @@ impl EventNotifier { /// A thread-safe list of targets pub struct TargetList { - targets: HashMap>, + targets: HashMap + Send + Sync>>, } impl Default for TargetList { @@ -219,7 +229,7 @@ impl TargetList { } /// Adds a target to the list - pub fn add(&mut self, target: Arc) -> Result<(), NotificationError> { + pub fn add(&mut self, target: Arc + Send + Sync>) -> Result<(), NotificationError> { let id = target.id(); if self.targets.contains_key(&id) { // Potentially update or log a warning/error if replacing an existing target. @@ -231,7 +241,7 @@ impl TargetList { /// Removes a target by ID. Note: This does not stop its associated event stream. /// Stream cancellation should be handled by EventNotifier. - pub async fn remove_target_only(&mut self, id: &TargetID) -> Option> { + pub async fn remove_target_only(&mut self, id: &TargetID) -> Option + Send + Sync>> { if let Some(target_arc) = self.targets.remove(id) { if let Err(e) = target_arc.close().await { // Target's own close logic @@ -258,7 +268,7 @@ impl TargetList { } /// Returns a target by ID - pub fn get(&self, id: &TargetID) -> Option> { + pub fn get(&self, id: &TargetID) -> Option + Send + Sync>> { self.targets.get(id).cloned() } diff --git a/crates/notify/src/registry.rs b/crates/notify/src/registry.rs index a65a9202..8bbaaa89 100644 --- a/crates/notify/src/registry.rs +++ b/crates/notify/src/registry.rs @@ -12,16 +12,15 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::target::ChannelTargetType; -use crate::{ - error::TargetError, - factory::{MQTTTargetFactory, TargetFactory, WebhookTargetFactory}, - target::Target, -}; +use crate::Event; +use crate::factory::{MQTTTargetFactory, TargetFactory, WebhookTargetFactory}; use futures::stream::{FuturesUnordered, StreamExt}; -use rustfs_config::notify::{ENABLE_KEY, NOTIFY_ROUTE_PREFIX}; -use rustfs_config::{DEFAULT_DELIMITER, ENV_PREFIX}; +use rustfs_config::notify::NOTIFY_ROUTE_PREFIX; +use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, ENV_PREFIX}; use rustfs_ecstore::config::{Config, KVS}; +use rustfs_targets::Target; +use rustfs_targets::TargetError; +use rustfs_targets::target::ChannelTargetType; use std::collections::{HashMap, HashSet}; use tracing::{debug, error, info, warn}; @@ -61,7 +60,7 @@ impl TargetRegistry { target_type: &str, id: String, config: &KVS, - ) -> Result, TargetError> { + ) -> Result + Send + Sync>, TargetError> { let factory = self .factories .get(target_type) @@ -83,7 +82,10 @@ impl TargetRegistry { /// 4. Combine the default configuration, file configuration, and environment variable configuration for each instance. /// 5. If the instance is enabled, create an asynchronous task for it to instantiate. /// 6. Concurrency executes all creation tasks and collects results. - pub async fn create_targets_from_config(&self, config: &Config) -> Result>, TargetError> { + pub async fn create_targets_from_config( + &self, + config: &Config, + ) -> Result + Send + Sync>>, TargetError> { // Collect only environment variables with the relevant prefix to reduce memory usage let all_env: Vec<(String, String)> = std::env::vars().filter(|(key, _)| key.starts_with(ENV_PREFIX)).collect(); // A collection of asynchronous tasks for concurrently executing target creation diff --git a/crates/notify/src/rules/config.rs b/crates/notify/src/rules/config.rs index c9e92953..81fbc4eb 100644 --- a/crates/notify/src/rules/config.rs +++ b/crates/notify/src/rules/config.rs @@ -14,11 +14,11 @@ use super::rules_map::RulesMap; use super::xml_config::ParseConfigError as BucketNotificationConfigError; -use crate::EventName; -use crate::arn::TargetID; use crate::rules::NotificationConfiguration; use crate::rules::pattern_rules; use crate::rules::target_id_set; +use rustfs_targets::EventName; +use rustfs_targets::arn::TargetID; use serde::{Deserialize, Serialize}; use std::collections::HashMap; use std::io::Read; diff --git a/crates/notify/src/rules/pattern_rules.rs b/crates/notify/src/rules/pattern_rules.rs index 01ad800c..ecc3dfaa 100644 --- a/crates/notify/src/rules/pattern_rules.rs +++ b/crates/notify/src/rules/pattern_rules.rs @@ -14,7 +14,7 @@ use super::pattern; use super::target_id_set::TargetIdSet; -use crate::arn::TargetID; +use rustfs_targets::arn::TargetID; use serde::{Deserialize, Serialize}; use std::collections::HashMap; diff --git a/crates/notify/src/rules/rules_map.rs b/crates/notify/src/rules/rules_map.rs index 9be13843..02aae743 100644 --- a/crates/notify/src/rules/rules_map.rs +++ b/crates/notify/src/rules/rules_map.rs @@ -14,8 +14,8 @@ use super::pattern_rules::PatternRules; use super::target_id_set::TargetIdSet; -use crate::arn::TargetID; -use crate::event::EventName; +use rustfs_targets::EventName; +use rustfs_targets::arn::TargetID; use serde::{Deserialize, Serialize}; use std::collections::HashMap; diff --git a/crates/notify/src/rules/target_id_set.rs b/crates/notify/src/rules/target_id_set.rs index 129d0dc3..45f790dc 100644 --- a/crates/notify/src/rules/target_id_set.rs +++ b/crates/notify/src/rules/target_id_set.rs @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::arn::TargetID; +use rustfs_targets::arn::TargetID; use std::collections::HashSet; /// TargetIDSet - A collection representation of TargetID. diff --git a/crates/notify/src/rules/xml_config.rs b/crates/notify/src/rules/xml_config.rs index 5925a66e..e8401dfd 100644 --- a/crates/notify/src/rules/xml_config.rs +++ b/crates/notify/src/rules/xml_config.rs @@ -13,8 +13,8 @@ // limitations under the License. use super::pattern; -use crate::arn::{ARN, ArnError, TargetIDError}; -use crate::event::EventName; +use rustfs_targets::EventName; +use rustfs_targets::arn::{ARN, ArnError, TargetIDError}; use serde::{Deserialize, Serialize}; use std::collections::HashSet; use std::io::Read; diff --git a/crates/notify/src/stream.rs b/crates/notify/src/stream.rs index 0d3d5255..9b37c13b 100644 --- a/crates/notify/src/stream.rs +++ b/crates/notify/src/stream.rs @@ -12,13 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::{ - Event, StoreError, - error::TargetError, - integration::NotificationMetrics, - store::{Key, Store}, - target::Target, -}; +use crate::{Event, integration::NotificationMetrics}; +use rustfs_targets::StoreError; +use rustfs_targets::Target; +use rustfs_targets::TargetError; +use rustfs_targets::store::{Key, Store}; +use rustfs_targets::target::EntityTarget; use std::sync::Arc; use std::time::{Duration, Instant}; use tokio::sync::{Semaphore, mpsc}; @@ -28,7 +27,7 @@ use tracing::{debug, error, info, warn}; /// Streams events from the store to the target pub async fn stream_events( store: &mut (dyn Store + Send), - target: &dyn Target, + target: &dyn Target, mut cancel_rx: mpsc::Receiver<()>, ) { info!("Starting event stream for target: {}", target.name()); @@ -107,7 +106,7 @@ pub async fn stream_events( /// Starts the event streaming process for a target pub fn start_event_stream( mut store: Box + Send>, - target: Arc, + target: Arc + Send + Sync>, ) -> mpsc::Sender<()> { let (cancel_tx, cancel_rx) = mpsc::channel(1); @@ -121,8 +120,8 @@ pub fn start_event_stream( /// Start event stream with batch processing pub fn start_event_stream_with_batching( - mut store: Box + Send>, - target: Arc, + mut store: Box, Error = StoreError, Key = Key> + Send>, + target: Arc + Send + Sync>, metrics: Arc, semaphore: Arc, ) -> mpsc::Sender<()> { @@ -138,8 +137,8 @@ pub fn start_event_stream_with_batching( /// Event stream processing with batch processing pub async fn stream_events_with_batching( - store: &mut (dyn Store + Send), - target: &dyn Target, + store: &mut (dyn Store, Error = StoreError, Key = Key> + Send), + target: &dyn Target, mut cancel_rx: mpsc::Receiver<()>, metrics: Arc, semaphore: Arc, @@ -156,7 +155,7 @@ pub async fn stream_events_with_batching( const MAX_RETRIES: usize = 5; const BASE_RETRY_DELAY: Duration = Duration::from_secs(2); - let mut batch = Vec::with_capacity(batch_size); + let mut batch: Vec> = Vec::with_capacity(batch_size); let mut batch_keys = Vec::with_capacity(batch_size); let mut last_flush = Instant::now(); @@ -234,9 +233,9 @@ pub async fn stream_events_with_batching( /// Processing event batches async fn process_batch( - batch: &mut Vec, + batch: &mut Vec>, batch_keys: &mut Vec, - target: &dyn Target, + target: &dyn Target, max_retries: usize, base_delay: Duration, metrics: &Arc, diff --git a/crates/rio/Cargo.toml b/crates/rio/Cargo.toml index 875b5fc2..835f23e2 100644 --- a/crates/rio/Cargo.toml +++ b/crates/rio/Cargo.toml @@ -45,4 +45,4 @@ serde_json.workspace = true md-5 = { workspace = true } [dev-dependencies] -tokio-test = { workspace = true } +tokio-test = { workspace = true } \ No newline at end of file diff --git a/crates/targets/Cargo.toml b/crates/targets/Cargo.toml new file mode 100644 index 00000000..f0ac418b --- /dev/null +++ b/crates/targets/Cargo.toml @@ -0,0 +1,31 @@ +[package] +name = "rustfs-targets" +edition.workspace = true +license.workspace = true +repository.workspace = true +rust-version.workspace = true +version.workspace = true +homepage.workspace = true +description = "Notification target abstraction and implementations for RustFS" +keywords = ["file-system", "notification", "target", "rustfs", "Minio"] +categories = ["web-programming", "development-tools", "filesystem"] +documentation = "https://docs.rs/rustfs-target/latest/rustfs_target/" + +[dependencies] +rustfs-config = { workspace = true, features = ["notify", "constants", "audit"] } +rustfs-utils = { workspace = true, features = ["sys"] } +async-trait = { workspace = true } +reqwest = { workspace = true } +rumqttc = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +snap = { workspace = true } +thiserror = { workspace = true } +tokio = { workspace = true, features = ["rt-multi-thread", "sync", "time"] } +tracing = { workspace = true } +url = { workspace = true } +urlencoding = { workspace = true } +uuid = { workspace = true, features = ["v4", "serde"] } + +[lints] +workspace = true diff --git a/crates/notify/src/arn.rs b/crates/targets/src/arn.rs similarity index 100% rename from crates/notify/src/arn.rs rename to crates/targets/src/arn.rs diff --git a/crates/targets/src/error.rs b/crates/targets/src/error.rs new file mode 100644 index 00000000..2e6d6091 --- /dev/null +++ b/crates/targets/src/error.rs @@ -0,0 +1,99 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::io; +use thiserror::Error; + +/// Error types for the store +#[derive(Debug, Error)] +pub enum StoreError { + #[error("I/O error: {0}")] + Io(#[from] io::Error), + + #[error("Serialization error: {0}")] + Serialization(String), + + #[error("Deserialization error: {0}")] + Deserialization(String), + + #[error("Compression error: {0}")] + Compression(String), + + #[error("Entry limit exceeded")] + LimitExceeded, + + #[error("Entry not found")] + NotFound, + + #[error("Invalid entry: {0}")] + Internal(String), // Added internal error type +} + +/// Error types for targets +#[derive(Debug, Error)] +pub enum TargetError { + #[error("Storage error: {0}")] + Storage(String), + + #[error("Network error: {0}")] + Network(String), + + #[error("Request error: {0}")] + Request(String), + + #[error("Timeout error: {0}")] + Timeout(String), + + #[error("Authentication error: {0}")] + Authentication(String), + + #[error("Configuration error: {0}")] + Configuration(String), + + #[error("Encoding error: {0}")] + Encoding(String), + + #[error("Serialization error: {0}")] + Serialization(String), + + #[error("Target not connected")] + NotConnected, + + #[error("Target initialization failed: {0}")] + Initialization(String), + + #[error("Invalid ARN: {0}")] + InvalidARN(String), + + #[error("Unknown error: {0}")] + Unknown(String), + + #[error("Target is disabled")] + Disabled, + + #[error("Configuration parsing error: {0}")] + ParseError(String), + + #[error("Failed to save configuration: {0}")] + SaveConfig(String), + + #[error("Server not initialized: {0}")] + ServerNotInitialized(String), +} + +impl From for TargetError { + fn from(err: url::ParseError) -> Self { + TargetError::Configuration(format!("URL parse error: {err}")) + } +} diff --git a/crates/targets/src/event_name.rs b/crates/targets/src/event_name.rs new file mode 100644 index 00000000..59c1a851 --- /dev/null +++ b/crates/targets/src/event_name.rs @@ -0,0 +1,290 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use serde::{Deserialize, Serialize}; +use std::fmt; + +/// Error returned when parsing event name string fails. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ParseEventNameError(String); + +impl fmt::Display for ParseEventNameError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "Invalid event name:{}", self.0) + } +} + +impl std::error::Error for ParseEventNameError {} + +/// Represents the type of event that occurs on the object. +/// Based on AWS S3 event type and includes RustFS extension. +#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Hash)] +pub enum EventName { + // Single event type (values are 1-32 for compatible mask logic) + ObjectAccessedGet = 1, + ObjectAccessedGetRetention = 2, + ObjectAccessedGetLegalHold = 3, + ObjectAccessedHead = 4, + ObjectAccessedAttributes = 5, + ObjectCreatedCompleteMultipartUpload = 6, + ObjectCreatedCopy = 7, + ObjectCreatedPost = 8, + ObjectCreatedPut = 9, + ObjectCreatedPutRetention = 10, + ObjectCreatedPutLegalHold = 11, + ObjectCreatedPutTagging = 12, + ObjectCreatedDeleteTagging = 13, + ObjectRemovedDelete = 14, + ObjectRemovedDeleteMarkerCreated = 15, + ObjectRemovedDeleteAllVersions = 16, + ObjectRemovedNoOP = 17, + BucketCreated = 18, + BucketRemoved = 19, + ObjectReplicationFailed = 20, + ObjectReplicationComplete = 21, + ObjectReplicationMissedThreshold = 22, + ObjectReplicationReplicatedAfterThreshold = 23, + ObjectReplicationNotTracked = 24, + ObjectRestorePost = 25, + ObjectRestoreCompleted = 26, + ObjectTransitionFailed = 27, + ObjectTransitionComplete = 28, + ScannerManyVersions = 29, // ObjectManyVersions corresponding to Go + ScannerLargeVersions = 30, // ObjectLargeVersions corresponding to Go + ScannerBigPrefix = 31, // PrefixManyFolders corresponding to Go + LifecycleDelMarkerExpirationDelete = 32, // ILMDelMarkerExpirationDelete corresponding to Go + + // Compound "All" event type (no sequential value for mask) + ObjectAccessedAll, + ObjectCreatedAll, + ObjectRemovedAll, + ObjectReplicationAll, + ObjectRestoreAll, + ObjectTransitionAll, + ObjectScannerAll, // New, from Go + Everything, // New, from Go +} + +// Single event type sequential array for Everything.expand() +const SINGLE_EVENT_NAMES_IN_ORDER: [EventName; 32] = [ + EventName::ObjectAccessedGet, + EventName::ObjectAccessedGetRetention, + EventName::ObjectAccessedGetLegalHold, + EventName::ObjectAccessedHead, + EventName::ObjectAccessedAttributes, + EventName::ObjectCreatedCompleteMultipartUpload, + EventName::ObjectCreatedCopy, + EventName::ObjectCreatedPost, + EventName::ObjectCreatedPut, + EventName::ObjectCreatedPutRetention, + EventName::ObjectCreatedPutLegalHold, + EventName::ObjectCreatedPutTagging, + EventName::ObjectCreatedDeleteTagging, + EventName::ObjectRemovedDelete, + EventName::ObjectRemovedDeleteMarkerCreated, + EventName::ObjectRemovedDeleteAllVersions, + EventName::ObjectRemovedNoOP, + EventName::BucketCreated, + EventName::BucketRemoved, + EventName::ObjectReplicationFailed, + EventName::ObjectReplicationComplete, + EventName::ObjectReplicationMissedThreshold, + EventName::ObjectReplicationReplicatedAfterThreshold, + EventName::ObjectReplicationNotTracked, + EventName::ObjectRestorePost, + EventName::ObjectRestoreCompleted, + EventName::ObjectTransitionFailed, + EventName::ObjectTransitionComplete, + EventName::ScannerManyVersions, + EventName::ScannerLargeVersions, + EventName::ScannerBigPrefix, + EventName::LifecycleDelMarkerExpirationDelete, +]; + +const LAST_SINGLE_TYPE_VALUE: u32 = EventName::LifecycleDelMarkerExpirationDelete as u32; + +impl EventName { + /// The parsed string is EventName. + pub fn parse(s: &str) -> Result { + match s { + "s3:BucketCreated:*" => Ok(EventName::BucketCreated), + "s3:BucketRemoved:*" => Ok(EventName::BucketRemoved), + "s3:ObjectAccessed:*" => Ok(EventName::ObjectAccessedAll), + "s3:ObjectAccessed:Get" => Ok(EventName::ObjectAccessedGet), + "s3:ObjectAccessed:GetRetention" => Ok(EventName::ObjectAccessedGetRetention), + "s3:ObjectAccessed:GetLegalHold" => Ok(EventName::ObjectAccessedGetLegalHold), + "s3:ObjectAccessed:Head" => Ok(EventName::ObjectAccessedHead), + "s3:ObjectAccessed:Attributes" => Ok(EventName::ObjectAccessedAttributes), + "s3:ObjectCreated:*" => Ok(EventName::ObjectCreatedAll), + "s3:ObjectCreated:CompleteMultipartUpload" => Ok(EventName::ObjectCreatedCompleteMultipartUpload), + "s3:ObjectCreated:Copy" => Ok(EventName::ObjectCreatedCopy), + "s3:ObjectCreated:Post" => Ok(EventName::ObjectCreatedPost), + "s3:ObjectCreated:Put" => Ok(EventName::ObjectCreatedPut), + "s3:ObjectCreated:PutRetention" => Ok(EventName::ObjectCreatedPutRetention), + "s3:ObjectCreated:PutLegalHold" => Ok(EventName::ObjectCreatedPutLegalHold), + "s3:ObjectCreated:PutTagging" => Ok(EventName::ObjectCreatedPutTagging), + "s3:ObjectCreated:DeleteTagging" => Ok(EventName::ObjectCreatedDeleteTagging), + "s3:ObjectRemoved:*" => Ok(EventName::ObjectRemovedAll), + "s3:ObjectRemoved:Delete" => Ok(EventName::ObjectRemovedDelete), + "s3:ObjectRemoved:DeleteMarkerCreated" => Ok(EventName::ObjectRemovedDeleteMarkerCreated), + "s3:ObjectRemoved:NoOP" => Ok(EventName::ObjectRemovedNoOP), + "s3:ObjectRemoved:DeleteAllVersions" => Ok(EventName::ObjectRemovedDeleteAllVersions), + "s3:LifecycleDelMarkerExpiration:Delete" => Ok(EventName::LifecycleDelMarkerExpirationDelete), + "s3:Replication:*" => Ok(EventName::ObjectReplicationAll), + "s3:Replication:OperationFailedReplication" => Ok(EventName::ObjectReplicationFailed), + "s3:Replication:OperationCompletedReplication" => Ok(EventName::ObjectReplicationComplete), + "s3:Replication:OperationMissedThreshold" => Ok(EventName::ObjectReplicationMissedThreshold), + "s3:Replication:OperationReplicatedAfterThreshold" => Ok(EventName::ObjectReplicationReplicatedAfterThreshold), + "s3:Replication:OperationNotTracked" => Ok(EventName::ObjectReplicationNotTracked), + "s3:ObjectRestore:*" => Ok(EventName::ObjectRestoreAll), + "s3:ObjectRestore:Post" => Ok(EventName::ObjectRestorePost), + "s3:ObjectRestore:Completed" => Ok(EventName::ObjectRestoreCompleted), + "s3:ObjectTransition:Failed" => Ok(EventName::ObjectTransitionFailed), + "s3:ObjectTransition:Complete" => Ok(EventName::ObjectTransitionComplete), + "s3:ObjectTransition:*" => Ok(EventName::ObjectTransitionAll), + "s3:Scanner:ManyVersions" => Ok(EventName::ScannerManyVersions), + "s3:Scanner:LargeVersions" => Ok(EventName::ScannerLargeVersions), + "s3:Scanner:BigPrefix" => Ok(EventName::ScannerBigPrefix), + // ObjectScannerAll and Everything cannot be parsed from strings, because the Go version also does not define their string representation. + _ => Err(ParseEventNameError(s.to_string())), + } + } + + /// Returns a string representation of the event type. + pub fn as_str(&self) -> &'static str { + match self { + EventName::BucketCreated => "s3:BucketCreated:*", + EventName::BucketRemoved => "s3:BucketRemoved:*", + EventName::ObjectAccessedAll => "s3:ObjectAccessed:*", + EventName::ObjectAccessedGet => "s3:ObjectAccessed:Get", + EventName::ObjectAccessedGetRetention => "s3:ObjectAccessed:GetRetention", + EventName::ObjectAccessedGetLegalHold => "s3:ObjectAccessed:GetLegalHold", + EventName::ObjectAccessedHead => "s3:ObjectAccessed:Head", + EventName::ObjectAccessedAttributes => "s3:ObjectAccessed:Attributes", + EventName::ObjectCreatedAll => "s3:ObjectCreated:*", + EventName::ObjectCreatedCompleteMultipartUpload => "s3:ObjectCreated:CompleteMultipartUpload", + EventName::ObjectCreatedCopy => "s3:ObjectCreated:Copy", + EventName::ObjectCreatedPost => "s3:ObjectCreated:Post", + EventName::ObjectCreatedPut => "s3:ObjectCreated:Put", + EventName::ObjectCreatedPutTagging => "s3:ObjectCreated:PutTagging", + EventName::ObjectCreatedDeleteTagging => "s3:ObjectCreated:DeleteTagging", + EventName::ObjectCreatedPutRetention => "s3:ObjectCreated:PutRetention", + EventName::ObjectCreatedPutLegalHold => "s3:ObjectCreated:PutLegalHold", + EventName::ObjectRemovedAll => "s3:ObjectRemoved:*", + EventName::ObjectRemovedDelete => "s3:ObjectRemoved:Delete", + EventName::ObjectRemovedDeleteMarkerCreated => "s3:ObjectRemoved:DeleteMarkerCreated", + EventName::ObjectRemovedNoOP => "s3:ObjectRemoved:NoOP", + EventName::ObjectRemovedDeleteAllVersions => "s3:ObjectRemoved:DeleteAllVersions", + EventName::LifecycleDelMarkerExpirationDelete => "s3:LifecycleDelMarkerExpiration:Delete", + EventName::ObjectReplicationAll => "s3:Replication:*", + EventName::ObjectReplicationFailed => "s3:Replication:OperationFailedReplication", + EventName::ObjectReplicationComplete => "s3:Replication:OperationCompletedReplication", + EventName::ObjectReplicationNotTracked => "s3:Replication:OperationNotTracked", + EventName::ObjectReplicationMissedThreshold => "s3:Replication:OperationMissedThreshold", + EventName::ObjectReplicationReplicatedAfterThreshold => "s3:Replication:OperationReplicatedAfterThreshold", + EventName::ObjectRestoreAll => "s3:ObjectRestore:*", + EventName::ObjectRestorePost => "s3:ObjectRestore:Post", + EventName::ObjectRestoreCompleted => "s3:ObjectRestore:Completed", + EventName::ObjectTransitionAll => "s3:ObjectTransition:*", + EventName::ObjectTransitionFailed => "s3:ObjectTransition:Failed", + EventName::ObjectTransitionComplete => "s3:ObjectTransition:Complete", + EventName::ScannerManyVersions => "s3:Scanner:ManyVersions", + EventName::ScannerLargeVersions => "s3:Scanner:LargeVersions", + EventName::ScannerBigPrefix => "s3:Scanner:BigPrefix", + // Go's String() returns "" for ObjectScannerAll and Everything + EventName::ObjectScannerAll => "s3:Scanner:*", // Follow the pattern in Go Expand + EventName::Everything => "", // Go String() returns "" to unprocessed + } + } + + /// Returns the extended value of the abbreviation event type. + pub fn expand(&self) -> Vec { + match self { + EventName::ObjectAccessedAll => vec![ + EventName::ObjectAccessedGet, + EventName::ObjectAccessedHead, + EventName::ObjectAccessedGetRetention, + EventName::ObjectAccessedGetLegalHold, + EventName::ObjectAccessedAttributes, + ], + EventName::ObjectCreatedAll => vec![ + EventName::ObjectCreatedCompleteMultipartUpload, + EventName::ObjectCreatedCopy, + EventName::ObjectCreatedPost, + EventName::ObjectCreatedPut, + EventName::ObjectCreatedPutRetention, + EventName::ObjectCreatedPutLegalHold, + EventName::ObjectCreatedPutTagging, + EventName::ObjectCreatedDeleteTagging, + ], + EventName::ObjectRemovedAll => vec![ + EventName::ObjectRemovedDelete, + EventName::ObjectRemovedDeleteMarkerCreated, + EventName::ObjectRemovedNoOP, + EventName::ObjectRemovedDeleteAllVersions, + ], + EventName::ObjectReplicationAll => vec![ + EventName::ObjectReplicationFailed, + EventName::ObjectReplicationComplete, + EventName::ObjectReplicationNotTracked, + EventName::ObjectReplicationMissedThreshold, + EventName::ObjectReplicationReplicatedAfterThreshold, + ], + EventName::ObjectRestoreAll => vec![EventName::ObjectRestorePost, EventName::ObjectRestoreCompleted], + EventName::ObjectTransitionAll => vec![EventName::ObjectTransitionFailed, EventName::ObjectTransitionComplete], + EventName::ObjectScannerAll => vec![ + // New + EventName::ScannerManyVersions, + EventName::ScannerLargeVersions, + EventName::ScannerBigPrefix, + ], + EventName::Everything => { + // New + SINGLE_EVENT_NAMES_IN_ORDER.to_vec() + } + // A single type returns to itself directly + _ => vec![*self], + } + } + + /// Returns the mask of type. + /// The compound "All" type will be expanded. + pub fn mask(&self) -> u64 { + let value = *self as u32; + if value > 0 && value <= LAST_SINGLE_TYPE_VALUE { + // It's a single type + 1u64 << (value - 1) + } else { + // It's a compound type + let mut mask = 0u64; + for n in self.expand() { + mask |= n.mask(); // Recursively call mask + } + mask + } + } +} + +impl fmt::Display for EventName { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.as_str()) + } +} + +/// Convert to `EventName` according to string +impl From<&str> for EventName { + fn from(event_str: &str) -> Self { + EventName::parse(event_str).unwrap_or_else(|e| panic!("{}", e)) + } +} diff --git a/crates/targets/src/lib.rs b/crates/targets/src/lib.rs new file mode 100644 index 00000000..a3e35160 --- /dev/null +++ b/crates/targets/src/lib.rs @@ -0,0 +1,35 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +pub mod arn; +pub mod error; +mod event_name; +pub mod store; +pub mod target; + +pub use error::{StoreError, TargetError}; +pub use event_name::EventName; +use serde::{Deserialize, Serialize}; +pub use target::Target; + +/// Represents a log of events for sending to targets +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct TargetLog { + /// The event name + pub event_name: EventName, + /// The object key + pub key: String, + /// The list of events + pub records: Vec, +} diff --git a/crates/notify/src/store.rs b/crates/targets/src/store.rs similarity index 98% rename from crates/notify/src/store.rs rename to crates/targets/src/store.rs index bf54e1e2..139e32be 100644 --- a/crates/notify/src/store.rs +++ b/crates/targets/src/store.rs @@ -13,7 +13,8 @@ // limitations under the License. use crate::error::StoreError; -use rustfs_config::notify::{COMPRESS_EXT, DEFAULT_EXT, DEFAULT_LIMIT}; +use rustfs_config::DEFAULT_LIMIT; +use rustfs_config::notify::{COMPRESS_EXT, DEFAULT_EXT}; use serde::{Serialize, de::DeserializeOwned}; use snap::raw::{Decoder, Encoder}; use std::sync::{Arc, RwLock}; @@ -123,7 +124,10 @@ pub fn parse_key(s: &str) -> Key { } /// Trait for a store that can store and retrieve items of type T -pub trait Store: Send + Sync { +pub trait Store: Send + Sync +where + T: Send + Sync + 'static + Clone + Serialize, +{ /// The error type for the store type Error; /// The key type for the store diff --git a/crates/notify/src/target/mod.rs b/crates/targets/src/target/mod.rs similarity index 70% rename from crates/notify/src/target/mod.rs rename to crates/targets/src/target/mod.rs index a7913f3a..627fa8d0 100644 --- a/crates/notify/src/target/mod.rs +++ b/crates/targets/src/target/mod.rs @@ -14,8 +14,11 @@ use crate::arn::TargetID; use crate::store::{Key, Store}; -use crate::{Event, StoreError, TargetError}; +use crate::{EventName, StoreError, TargetError}; use async_trait::async_trait; +use serde::de::DeserializeOwned; +use serde::{Deserialize, Serialize}; +use std::fmt::Formatter; use std::sync::Arc; pub mod mqtt; @@ -23,7 +26,10 @@ pub mod webhook; /// Trait for notification targets #[async_trait] -pub trait Target: Send + Sync + 'static { +pub trait Target: Send + Sync + 'static +where + E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned, +{ /// Returns the ID of the target fn id(&self) -> TargetID; @@ -36,7 +42,7 @@ pub trait Target: Send + Sync + 'static { async fn is_active(&self) -> Result; /// Saves an event (either sends it immediately or stores it for later) - async fn save(&self, event: Arc) -> Result<(), TargetError>; + async fn save(&self, event: Arc>) -> Result<(), TargetError>; /// Sends an event from the store async fn send_from_store(&self, key: Key) -> Result<(), TargetError>; @@ -45,10 +51,10 @@ pub trait Target: Send + Sync + 'static { async fn close(&self) -> Result<(), TargetError>; /// Returns the store associated with the target (if any) - fn store(&self) -> Option<&(dyn Store + Send + Sync)>; + fn store(&self) -> Option<&(dyn Store, Error = StoreError, Key = Key> + Send + Sync)>; /// Returns the type of the target - fn clone_dyn(&self) -> Box; + fn clone_dyn(&self) -> Box + Send + Sync>; /// Initialize the target, such as establishing a connection, etc. async fn init(&self) -> Result<(), TargetError> { @@ -60,6 +66,17 @@ pub trait Target: Send + Sync + 'static { fn is_enabled(&self) -> bool; } +#[derive(Debug, Serialize, Clone, Deserialize)] +pub struct EntityTarget +where + E: Send + Sync + 'static + Clone + Serialize, +{ + pub object_name: String, + pub bucket_name: String, + pub event_name: EventName, + pub data: E, +} + /// The `ChannelTargetType` enum represents the different types of channel Target /// used in the notification system. /// @@ -75,7 +92,7 @@ pub trait Target: Send + Sync + 'static { /// /// example usage: /// ```rust -/// use rustfs_notify::target::ChannelTargetType; +/// use rustfs_targets::target::ChannelTargetType; /// /// let target_type = ChannelTargetType::Webhook; /// assert_eq!(target_type.as_str(), "webhook"); @@ -101,7 +118,7 @@ impl ChannelTargetType { } impl std::fmt::Display for ChannelTargetType { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { match self { ChannelTargetType::Webhook => write!(f, "webhook"), ChannelTargetType::Kafka => write!(f, "kafka"), @@ -117,3 +134,28 @@ pub fn parse_bool(value: &str) -> Result { _ => Err(TargetError::ParseError(format!("Unable to parse boolean: {value}"))), } } + +/// `TargetType` enum represents the type of target in the notification system. +#[derive(Debug, Clone)] +pub enum TargetType { + AuditLog, + NotifyEvent, +} + +impl TargetType { + pub fn as_str(&self) -> &'static str { + match self { + TargetType::AuditLog => "audit_log", + TargetType::NotifyEvent => "notify_event", + } + } +} + +impl std::fmt::Display for TargetType { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + match self { + TargetType::AuditLog => write!(f, "audit_log"), + TargetType::NotifyEvent => write!(f, "notify_event"), + } + } +} diff --git a/crates/notify/src/target/mqtt.rs b/crates/targets/src/target/mqtt.rs similarity index 93% rename from crates/notify/src/target/mqtt.rs rename to crates/targets/src/target/mqtt.rs index 9beda90b..9dbf6411 100644 --- a/crates/notify/src/target/mqtt.rs +++ b/crates/targets/src/target/mqtt.rs @@ -13,18 +13,13 @@ // limitations under the License. use crate::store::Key; -use crate::target::ChannelTargetType; -use crate::{ - StoreError, Target, - arn::TargetID, - error::TargetError, - event::{Event, EventLog}, - store::Store, -}; +use crate::target::{ChannelTargetType, EntityTarget, TargetType}; +use crate::{StoreError, Target, TargetLog, arn::TargetID, error::TargetError, store::Store}; use async_trait::async_trait; use rumqttc::{AsyncClient, EventLoop, MqttOptions, Outgoing, Packet, QoS}; use rumqttc::{ConnectionError, mqttbytes::Error as MqttBytesError}; -use rustfs_config::notify::STORE_EXTENSION; +use serde::Serialize; +use serde::de::DeserializeOwned; use std::sync::Arc; use std::{ path::PathBuf, @@ -62,6 +57,8 @@ pub struct MQTTArgs { pub queue_dir: String, /// The maximum number of events to store pub queue_limit: u64, + /// the target type + pub target_type: TargetType, } impl MQTTArgs { @@ -77,6 +74,10 @@ impl MQTTArgs { } } + if self.topic.is_empty() { + return Err(TargetError::Configuration("MQTT topic cannot be empty".to_string())); + } + if !self.queue_dir.is_empty() { let path = std::path::Path::new(&self.queue_dir); if !path.is_absolute() { @@ -100,16 +101,22 @@ struct BgTaskManager { } /// A target that sends events to an MQTT broker -pub struct MQTTTarget { +pub struct MQTTTarget +where + E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned, +{ id: TargetID, args: MQTTArgs, client: Arc>>, - store: Option + Send + Sync>>, + store: Option, Error = StoreError, Key = Key> + Send + Sync>>, connected: Arc, bg_task_manager: Arc, } -impl MQTTTarget { +impl MQTTTarget +where + E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned, +{ /// Creates a new MQTTTarget #[instrument(skip(args), fields(target_id_as_string = %id))] pub fn new(id: String, args: MQTTArgs) -> Result { @@ -121,7 +128,12 @@ impl MQTTTarget { // Ensure the directory name is valid for filesystem let specific_queue_path = base_path.join(unique_dir_name); debug!(target_id = %target_id, path = %specific_queue_path.display(), "Initializing queue store for MQTT target"); - let store = crate::store::QueueStore::::new(specific_queue_path, args.queue_limit, STORE_EXTENSION); + let extension = match args.target_type { + TargetType::AuditLog => rustfs_config::audit::AUDIT_STORE_EXTENSION, + TargetType::NotifyEvent => rustfs_config::notify::STORE_EXTENSION, + }; + + let store = crate::store::QueueStore::>::new(specific_queue_path, args.queue_limit, extension); if let Err(e) = store.open() { error!( target_id = %target_id, @@ -130,7 +142,7 @@ impl MQTTTarget { ); return Err(TargetError::Storage(format!("{e}"))); } - Some(Box::new(store) as Box + Send + Sync>) + Some(Box::new(store) as Box, Error = StoreError, Key = Key> + Send + Sync>) } else { None }; @@ -237,18 +249,18 @@ impl MQTTTarget { } #[instrument(skip(self, event), fields(target_id = %self.id))] - async fn send(&self, event: &Event) -> Result<(), TargetError> { + async fn send(&self, event: &EntityTarget) -> Result<(), TargetError> { let client_guard = self.client.lock().await; let client = client_guard .as_ref() .ok_or_else(|| TargetError::Configuration("MQTT client not initialized".to_string()))?; - let object_name = urlencoding::decode(&event.s3.object.key) + let object_name = urlencoding::decode(&event.object_name) .map_err(|e| TargetError::Encoding(format!("Failed to decode object key: {e}")))?; - let key = format!("{}/{}", event.s3.bucket.name, object_name); + let key = format!("{}/{}", event.bucket_name, object_name); - let log = EventLog { + let log = TargetLog { event_name: event.event_name, key, records: vec![event.clone()], @@ -256,7 +268,6 @@ impl MQTTTarget { let data = serde_json::to_vec(&log).map_err(|e| TargetError::Serialization(format!("Failed to serialize event: {e}")))?; - // Vec Convert to String, only for printing logs let data_string = String::from_utf8(data.clone()) .map_err(|e| TargetError::Encoding(format!("Failed to convert event data to UTF-8: {e}")))?; debug!("Sending event to mqtt target: {}, event log: {}", self.id, data_string); @@ -278,7 +289,7 @@ impl MQTTTarget { Ok(()) } - pub fn clone_target(&self) -> Box { + pub fn clone_target(&self) -> Box + Send + Sync> { Box::new(MQTTTarget { id: self.id.clone(), args: self.args.clone(), @@ -444,7 +455,10 @@ fn is_fatal_mqtt_error(err: &ConnectionError) -> bool { } #[async_trait] -impl Target for MQTTTarget { +impl Target for MQTTTarget +where + E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned, +{ fn id(&self) -> TargetID { self.id.clone() } @@ -476,7 +490,7 @@ impl Target for MQTTTarget { } #[instrument(skip(self, event), fields(target_id = %self.id))] - async fn save(&self, event: Arc) -> Result<(), TargetError> { + async fn save(&self, event: Arc>) -> Result<(), TargetError> { if let Some(store) = &self.store { debug!(target_id = %self.id, "Event saved to store start"); // If store is configured, ONLY put the event into the store. @@ -620,11 +634,11 @@ impl Target for MQTTTarget { Ok(()) } - fn store(&self) -> Option<&(dyn Store + Send + Sync)> { + fn store(&self) -> Option<&(dyn Store, Error = StoreError, Key = Key> + Send + Sync)> { self.store.as_deref() } - fn clone_dyn(&self) -> Box { + fn clone_dyn(&self) -> Box + Send + Sync> { self.clone_target() } diff --git a/crates/notify/src/target/webhook.rs b/crates/targets/src/target/webhook.rs similarity index 90% rename from crates/notify/src/target/webhook.rs rename to crates/targets/src/target/webhook.rs index 0b3f93de..8ae03888 100644 --- a/crates/notify/src/target/webhook.rs +++ b/crates/targets/src/target/webhook.rs @@ -12,17 +12,18 @@ // See the License for the specific language governing permissions and // limitations under the License. -use crate::target::ChannelTargetType; +use crate::target::{ChannelTargetType, EntityTarget, TargetType}; use crate::{ - StoreError, Target, + StoreError, Target, TargetLog, arn::TargetID, error::TargetError, - event::{Event, EventLog}, store::{Key, Store}, }; use async_trait::async_trait; use reqwest::{Client, StatusCode, Url}; use rustfs_config::notify::STORE_EXTENSION; +use serde::Serialize; +use serde::de::DeserializeOwned; use std::{ path::PathBuf, sync::{ @@ -53,6 +54,8 @@ pub struct WebhookArgs { pub client_cert: String, /// The client key for TLS (PEM format) pub client_key: String, + /// the target type + pub target_type: TargetType, } impl WebhookArgs { @@ -84,20 +87,26 @@ impl WebhookArgs { } /// A target that sends events to a webhook -pub struct WebhookTarget { +pub struct WebhookTarget +where + E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned, +{ id: TargetID, args: WebhookArgs, http_client: Arc, // Add Send + Sync constraints to ensure thread safety - store: Option + Send + Sync>>, + store: Option, Error = StoreError, Key = Key> + Send + Sync>>, initialized: AtomicBool, addr: String, cancel_sender: mpsc::Sender<()>, } -impl WebhookTarget { +impl WebhookTarget +where + E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned, +{ /// Clones the WebhookTarget, creating a new instance with the same configuration - pub fn clone_box(&self) -> Box { + pub fn clone_box(&self) -> Box + Send + Sync> { Box::new(WebhookTarget { id: self.id.clone(), args: self.args.clone(), @@ -144,7 +153,7 @@ impl WebhookTarget { let queue_store = if !args.queue_dir.is_empty() { let queue_dir = PathBuf::from(&args.queue_dir).join(format!("rustfs-{}-{}", ChannelTargetType::Webhook.as_str(), target_id.id)); - let store = crate::store::QueueStore::::new(queue_dir, args.queue_limit, STORE_EXTENSION); + let store = crate::store::QueueStore::>::new(queue_dir, args.queue_limit, STORE_EXTENSION); if let Err(e) = store.open() { error!("Failed to open store for Webhook target {}: {}", target_id.id, e); @@ -152,7 +161,7 @@ impl WebhookTarget { } // Make sure that the Store trait implemented by QueueStore matches the expected error type - Some(Box::new(store) as Box + Send + Sync>) + Some(Box::new(store) as Box, Error = StoreError, Key = Key> + Send + Sync>) } else { None }; @@ -203,17 +212,17 @@ impl WebhookTarget { Ok(()) } - async fn send(&self, event: &Event) -> Result<(), TargetError> { + async fn send(&self, event: &EntityTarget) -> Result<(), TargetError> { info!("Webhook Sending event to webhook target: {}", self.id); - let object_name = urlencoding::decode(&event.s3.object.key) + let object_name = urlencoding::decode(&event.object_name) .map_err(|e| TargetError::Encoding(format!("Failed to decode object key: {e}")))?; - let key = format!("{}/{}", event.s3.bucket.name, object_name); + let key = format!("{}/{}", event.bucket_name, object_name); - let log = EventLog { + let log = TargetLog { event_name: event.event_name, key, - records: vec![event.clone()], + records: vec![event.data.clone()], }; let data = serde_json::to_vec(&log).map_err(|e| TargetError::Serialization(format!("Failed to serialize event: {e}")))?; @@ -275,12 +284,14 @@ impl WebhookTarget { } #[async_trait] -impl Target for WebhookTarget { +impl Target for WebhookTarget +where + E: Send + Sync + 'static + Clone + Serialize + DeserializeOwned, +{ fn id(&self) -> TargetID { self.id.clone() } - // Make sure Future is Send async fn is_active(&self) -> Result { let socket_addr = lookup_host(&self.addr) .await @@ -305,7 +316,7 @@ impl Target for WebhookTarget { } } - async fn save(&self, event: Arc) -> Result<(), TargetError> { + async fn save(&self, event: Arc>) -> Result<(), TargetError> { if let Some(store) = &self.store { // Call the store method directly, no longer need to acquire the lock store @@ -379,17 +390,15 @@ impl Target for WebhookTarget { Ok(()) } - fn store(&self) -> Option<&(dyn Store + Send + Sync)> { + fn store(&self) -> Option<&(dyn Store, Error = StoreError, Key = Key> + Send + Sync)> { // Returns the reference to the internal store self.store.as_deref() } - fn clone_dyn(&self) -> Box { + fn clone_dyn(&self) -> Box + Send + Sync> { self.clone_box() } - // The existing init method can meet the needs well, but we need to make sure it complies with the Target trait - // We can use the existing init method, but adjust the return value to match the trait requirement async fn init(&self) -> Result<(), TargetError> { // If the target is disabled, return to success directly if !self.is_enabled() { diff --git a/rustfs/Cargo.toml b/rustfs/Cargo.toml index 6d753eff..a72746f2 100644 --- a/rustfs/Cargo.toml +++ b/rustfs/Cargo.toml @@ -49,8 +49,10 @@ rustfs-config = { workspace = true, features = ["constants", "notify"] } rustfs-notify = { workspace = true } rustfs-obs = { workspace = true } rustfs-utils = { workspace = true, features = ["full"] } -rustfs-protos.workspace = true +rustfs-protos = { workspace = true } rustfs-s3select-query = { workspace = true } +rustfs-audit-logger = { workspace = true } +rustfs-targets = { workspace = true } atoi = { workspace = true } atomic_enum = { workspace = true } axum.workspace = true diff --git a/rustfs/src/admin/handlers/event.rs b/rustfs/src/admin/handlers/event.rs index 99959744..f4ee2d50 100644 --- a/rustfs/src/admin/handlers/event.rs +++ b/rustfs/src/admin/handlers/event.rs @@ -18,9 +18,10 @@ use crate::admin::router::Operation; use crate::auth::{check_key_valid, get_session_token}; use http::{HeaderMap, StatusCode}; use matchit::Params; -use rustfs_config::notify::{ENABLE_KEY, ENABLE_ON, NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS}; -use rustfs_notify::EventName; +use rustfs_config::notify::{NOTIFY_MQTT_SUB_SYS, NOTIFY_WEBHOOK_SUB_SYS}; +use rustfs_config::{ENABLE_KEY, EnableState}; use rustfs_notify::rules::{BucketNotificationConfig, PatternRules}; +use rustfs_targets::EventName; use s3s::header::CONTENT_LENGTH; use s3s::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, header::CONTENT_TYPE, s3_error}; use serde::{Deserialize, Serialize}; @@ -78,7 +79,7 @@ impl Operation for SetNotificationTarget { .map_err(|e| s3_error!(InvalidArgument, "invalid json body for target config: {}", e))?; // If there is an enable key, add an enable key value to "on" if !kvs_map.contains_key(ENABLE_KEY) { - kvs_map.insert(ENABLE_KEY.to_string(), ENABLE_ON.to_string()); + kvs_map.insert(ENABLE_KEY.to_string(), EnableState::On.to_string()); } let kvs = rustfs_ecstore::config::KVS( diff --git a/rustfs/src/admin/mod.rs b/rustfs/src/admin/mod.rs index 0449b290..2fde47f9 100644 --- a/rustfs/src/admin/mod.rs +++ b/rustfs/src/admin/mod.rs @@ -25,7 +25,10 @@ use handlers::{ sts, tier, user, }; -use crate::admin::handlers::event::{ListNotificationTargets, RemoveNotificationTarget, SetNotificationTarget}; +use crate::admin::handlers::event::{ + GetBucketNotification, ListNotificationTargets, RemoveBucketNotification, RemoveNotificationTarget, SetBucketNotification, + SetNotificationTarget, +}; use handlers::{GetReplicationMetricsHandler, ListRemoteTargetHandler, RemoveRemoteTargetHandler, SetRemoteTargetHandler}; use hyper::Method; use router::{AdminOperation, S3Router}; @@ -389,5 +392,23 @@ fn register_user_route(r: &mut S3Router) -> std::io::Result<()> AdminOperation(&RemoveNotificationTarget {}), )?; + r.insert( + Method::POST, + format!("{}{}", ADMIN_PREFIX, "/v3/target-set-bucket").as_str(), + AdminOperation(&SetBucketNotification {}), + )?; + + r.insert( + Method::POST, + format!("{}{}", ADMIN_PREFIX, "/v3/target-get-bucket").as_str(), + AdminOperation(&GetBucketNotification {}), + )?; + + r.insert( + Method::POST, + format!("{}{}", ADMIN_PREFIX, "/v3/target-remove-bucket").as_str(), + AdminOperation(&RemoveBucketNotification {}), + )?; + Ok(()) } diff --git a/rustfs/src/config/mod.rs b/rustfs/src/config/mod.rs index 038f5f76..ddc248ce 100644 --- a/rustfs/src/config/mod.rs +++ b/rustfs/src/config/mod.rs @@ -72,7 +72,7 @@ pub struct Opt { #[arg(long, default_value_t = rustfs_config::DEFAULT_OBS_ENDPOINT.to_string(), env = "RUSTFS_OBS_ENDPOINT")] pub obs_endpoint: String, - /// tls path for rustfs api and console. + /// tls path for rustfs API and console. #[arg(long, env = "RUSTFS_TLS_PATH")] pub tls_path: Option, diff --git a/rustfs/src/server/audit.rs b/rustfs/src/server/audit.rs new file mode 100644 index 00000000..439d7a7c --- /dev/null +++ b/rustfs/src/server/audit.rs @@ -0,0 +1,13 @@ +// Copyright 2024 RustFS Team +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. diff --git a/rustfs/src/server/mod.rs b/rustfs/src/server/mod.rs index 5efd8617..3b86e513 100644 --- a/rustfs/src/server/mod.rs +++ b/rustfs/src/server/mod.rs @@ -12,10 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. +mod audit; mod http; mod hybrid; mod layer; mod service_state; + pub(crate) use http::start_http_server; pub(crate) use service_state::SHUTDOWN_TIMEOUT; pub(crate) use service_state::ServiceState; diff --git a/rustfs/src/storage/ecfs.rs b/rustfs/src/storage/ecfs.rs index e34570dc..6c8f7d01 100644 --- a/rustfs/src/storage/ecfs.rs +++ b/rustfs/src/storage/ecfs.rs @@ -76,7 +76,6 @@ use rustfs_ecstore::store_api::PutObjReader; use rustfs_ecstore::store_api::StorageAPI; use rustfs_filemeta::headers::RESERVED_METADATA_PREFIX_LOWER; use rustfs_filemeta::headers::{AMZ_DECODED_CONTENT_LENGTH, AMZ_OBJECT_TAGGING}; -use rustfs_notify::EventName; use rustfs_policy::auth; use rustfs_policy::policy::action::Action; use rustfs_policy::policy::action::S3Action; @@ -86,6 +85,7 @@ use rustfs_rio::EtagReader; use rustfs_rio::HashReader; use rustfs_rio::Reader; use rustfs_rio::WarpReader; +use rustfs_targets::EventName; use rustfs_utils::CompressionAlgorithm; use rustfs_utils::path::path_join_buf; use rustfs_zip::CompressionFormat;