sdgoij/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-01-17 01:30:33 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,973 Commits 19 Branches 80 Tags
18cd9a8b4618b9337e5d3bbbf1edf2d8167ead3e
Commit Graph

317 Commits

Author SHA1 Message Date
houseme
18cd9a8b46 build(deps): bump the dependencies group with 5 updates (#896) 2025-11-20 13:04:24 +08:00
Copilot
601f3456bc Fix large file upload freeze with adaptive buffer sizing (#869) 
* Initial plan

* Fix large file upload freeze by increasing StreamReader buffer size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add comprehensive documentation for large file upload freeze fix

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade s3s version

* Fix compilation error: use BufReader instead of non-existent StreamReader::with_capacity

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Update documentation with correct BufReader implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add tokio feature `io-util`

* Implement adaptive buffer sizing based on file size

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Constants are managed uniformly and fmt code

* fix

* Fix: Trigger self-heal on read when shards missing from rejoined nodes (#871)

* Initial plan

* Fix: Trigger self-heal when missing shards detected during read

- Added proactive heal detection in get_object_with_fileinfo
- When reading an object, now checks if any shards are missing even if read succeeds
- Sends low-priority heal request to reconstruct missing shards on rejoined nodes
- This fixes the issue where data written during node outage is not healed when node rejoins

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* Unify CRC implementations to crc-fast (#873)

* Initial plan

* Replace CRC libraries with unified crc-fast implementation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fix

* fix: replace low to Normal

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-17 23:15:20 +08:00
Copilot
b7964081ce Fix KMS configuration synchronization across cluster nodes (#855) 
* Initial plan

* Add KMS configuration persistence to cluster storage

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Apply code formatting to KMS configuration changes

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add comment

* fix fmt

* fix

* Fix overlapping dependabot cargo configurations

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* improve code for comment and replace  `Once_Cell` to `std::sync::OnceLock`

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-11-16 00:05:03 +08:00
houseme
c242957c6f build(deps): bump the dependencies group with 8 updates (#857) 2025-11-15 19:51:07 +08:00
houseme
55e3a1f7e0 fix(audit): prevent state transition when no targets exist (#854) 
Avoid setting AuditSystemState::Starting when target list is empty.
Now checks target availability before state transition, keeping the
system in Stopped state if no enabled targets are found.

- Check targets.is_empty() before setting Starting state
- Return early with Ok(()) when no targets exist
- Maintain consistent state machine behavior
- Prevent transient "Starting" state with no actual targets

Resolves issue where audit system would incorrectly enter Starting
state even when configuration contained no enabled targets.
 2025-11-14 20:23:21 +08:00
houseme
51584986e1 feat(obs): unify metrics initialization and fix exporter move error (#851) 
* feat(obs): unify metrics initialization and fix exporter move error

- Fix Rust E0382 (use after move) by removing duplicate MetricExporter consumption.
- Consolidate MeterProvider construction into single Recorder builder path.
- Remove redundant Recorder::builder(...).install_global() call.
- Ensure PeriodicReader setup is performed only once (HTTP + optional stdout).
- Set global meter provider and metrics recorder exactly once.
- Preserve existing behavior for stdout/file vs HTTP modes.
- Minor cleanup: consistent resource reuse and interval handling.

* update telemetry.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix

* fix

* fix

* fix: modify logger level from error to event

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-11-14 00:50:07 +08:00
houseme
d4817a4bea fix: modify logger from warn to info (#842) 
* fix: modify logger from warn to info

* upgrade version
 2025-11-12 11:13:38 +08:00
houseme
7e1a9e2ede 🔒 Upgrade Cryptography Libraries to Latest RC Versions (#837) 
* fix

* chore: upgrade cryptography libraries to RC versions

- Upgrade aes-gcm to 0.11.0-rc.2 with rand_core support
- Upgrade chacha20poly1305 to 0.11.0-rc.2
- Upgrade argon2 to 0.6.0-rc.2 with std features
- Upgrade hmac to 0.13.0-rc.3
- Upgrade pbkdf2 to 0.13.0-rc.2
- Upgrade rsa to 0.10.0-rc.10
- Upgrade sha1 and sha2 to 0.11.0-rc.3
- Upgrade md-5 to 0.11.0-rc.3

These upgrades provide enhanced security features and performance
improvements while maintaining backward compatibility with existing
encryption workflows.

* add

* improve code

* fix
 2025-11-11 21:10:03 +08:00
houseme
98be7df0f5 feat(storage): refactor audit and notification with OperationHelper (#825) 
* improve code for audit

* improve code ecfs.rs

* improve code

* improve code for ecfs.rs

* feat(storage): refactor audit and notification with OperationHelper

This commit introduces a significant refactoring of the audit logging and event notification mechanisms within `ecfs.rs`.

The core of this change is the new `OperationHelper` struct, which encapsulates and simplifies the logic for both concerns. It replaces the previous `AuditHelper` and manual event dispatching.

Key improvements include:

- **Unified Handling**: `OperationHelper` manages both audit and notification builders, providing a single, consistent entry point for S3 operations.
- **RAII for Automation**: By leveraging the `Drop` trait, the helper automatically dispatches logs and notifications when it goes out of scope. This simplifies S3 method implementations and ensures cleanup even on early returns.
- **Fluent API**: A builder-like pattern with methods such as `.object()`, `.version_id()`, and `.suppress_event()` makes the code more readable and expressive.
- **Context-Aware Logic**: The helper's `.complete()` method intelligently populates log details based on the operation's `S3Result` and only triggers notifications on success.
- **Modular Design**: All helper logic is now isolated in `rustfs/src/storage/helper.rs`, improving separation of concerns and making `ecfs.rs` cleaner.

This refactoring significantly enhances code clarity, reduces boilerplate, and improves the robustness of logging and notification handling across the storage layer.

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* improve code for audit and notify

* fix

* fix

* fix
 2025-11-10 17:30:50 +08:00
houseme
b26aad4129 improve code for logger (#822) 
* improve code for logger

* fix
 2025-11-08 22:36:24 +08:00
houseme
29056a767a Refactor Telemetry Initialization and Environment Utilities (#811) 
* improve code for metrics

* improve code for metrics

* fix

* fix

* Refactor telemetry initialization and environment functions ordering

- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.

* fix

* up

* fix

* fix

* improve code for obs

* fix

* fix
 2025-11-07 20:01:54 +08:00
houseme
38f26b7c94 improve import，crate version，and copyright (#790) 2025-11-05 09:10:06 +08:00
houseme
d934e3905b Refactor telemetry initialization for non-production environments (#789) 
* add dep `scopeguard`

* improve for tracing

* fix

* fix

* improve code for import

* add logger trace id

* fix

* fix

* fix

* fix

* fix
 2025-11-05 00:55:08 +08:00
dependabot[bot]
a9d5fbac54 build(deps): bump the dependencies group with 6 updates (#777) 
* build(deps): bump the dependencies group with 6 updates

Bumps the dependencies group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [axum-extra](https://github.com/tokio-rs/axum) | `0.10.3` | `0.12.0` |
| [aws-config](https://github.com/smithy-lang/smithy-rs) | `1.8.8` | `1.8.10` |
| [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust) | `1.109.0` | `1.110.0` |
| [aws-smithy-types](https://github.com/smithy-lang/smithy-rs) | `1.3.3` | `1.3.4` |
| [clap](https://github.com/clap-rs/clap) | `4.5.50` | `4.5.51` |
| [matchit](https://github.com/ibraheemdev/matchit) | `0.8.4` | `0.9.0` |


Updates `axum-extra` from 0.10.3 to 0.12.0
- [Release notes](https://github.com/tokio-rs/axum/releases)
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/axum/compare/axum-extra-v0.10.3...axum-extra-v0.12.0)

Updates `aws-config` from 1.8.8 to 1.8.10
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `aws-sdk-s3` from 1.109.0 to 1.110.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `aws-smithy-types` from 1.3.3 to 1.3.4
- [Release notes](https://github.com/smithy-lang/smithy-rs/releases)
- [Changelog](https://github.com/smithy-lang/smithy-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/smithy-lang/smithy-rs/commits)

Updates `clap` from 4.5.50 to 4.5.51
- [Release notes](https://github.com/clap-rs/clap/releases)
- [Changelog](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md)
- [Commits](https://github.com/clap-rs/clap/compare/clap_complete-v4.5.50...clap_complete-v4.5.51)

Updates `matchit` from 0.8.4 to 0.9.0
- [Release notes](https://github.com/ibraheemdev/matchit/releases)
- [Commits](https://github.com/ibraheemdev/matchit/compare/v0.8.4...v0.9.0)

---
updated-dependencies:
- dependency-name: axum-extra
  dependency-version: 0.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: aws-config
  dependency-version: 1.8.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: aws-sdk-s3
  dependency-version: 1.110.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: aws-smithy-types
  dependency-version: 1.3.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: clap
  dependency-version: 4.5.51
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: matchit
  dependency-version: 0.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* upgrade crates version

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-11-03 00:34:54 +08:00
houseme
d30c42f85a feat(admin): Add admin v3 API routes and profiling endpoints for RustFS (#774) 
* add Jemalloc

* feat: optimize AI rules with unified .rules.md  (#401)

* feat: optimize AI rules with unified .rules.md and entry points

- Create .rules.md as the central AI coding rules file
- Add .copilot-rules.md as GitHub Copilot entry point
- Add CLAUDE.md as Claude AI entry point
- Incorporate principles from rustfs.com project
- Add three critical rules:
  1. Use English for all code comments and documentation
  2. Clean up temporary scripts after use
  3. Only make confident modifications

* Update CLAUDE.md

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* feat: translate chinese to english (#402)

* Checkpoint before follow-up message

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Translate project documentation and comments from Chinese to English

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Fix typo: "unparseable" to "unparsable" in version test comment

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Refactor compression test code with minor syntax improvements

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* fix: the automatic logout issue and user list display failure on Windows systems (#353) (#343) (#403)

Co-authored-by: 安正超 <anzhengchao@gmail.com>

* upgrade version

* improve code for profiling

* fix

* Initial plan

* feat: Implement layered DNS resolver with caching and validation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Integrate DNS resolver into main application and fix formatting

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement enhanced DNS resolver with Moka cache and layered fallback

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement hickory-resolver with TLS support for enhanced DNS resolution

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade

* add .gitignore config

* fix

* add

* add

* up

* improve linux profiling

* fix

* fix

* fix

* feat(admin): Refactor profiling endpoints

Replaces the existing pprof profiling endpoints with new trigger-based APIs for CPU and memory profiling. This change simplifies the handler logic by moving the profiling implementation to a dedicated module.

A new handler file `admin/handlers/profile.rs` is created to contain the logic for these new endpoints. The core profiling functions are now expected to be in the `profiling` module, which the new handlers call to generate and save profile data.

* cargo shear --fix

* fix

* fix

* fix

---------

Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2025-11-01 03:16:37 +08:00
houseme
0714c7a9ca modify logger level from info to error (#744) 
* modify logger level from `info` to `error`

* fix test

* improve tokio runtime config

* add rustfs helm chart files (#747)

* add rustfs helm chart files

* update readme file with helm chart

* delete helm chart license file

* fix typo in readme file

* fix: restore localized samples in tests (#749)

* fix: restore required localized examples

* style: fix formatting issues

* improve code for Observability

* upgrade crates version

* fix

* up

* fix

---------

Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
 2025-10-29 19:20:53 +08:00
likewu
2aca1f77af Fix/ilm (#721) 
* fix tip remote tier error
* fix transitioned_object
* fix filemeta
* add GCS R2
* add aliyun tencent huaweicloud azure gcs r2 backend tier
* fix signer
* change azure to s3
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-10-27 20:23:50 +08:00
houseme
c5264f9703 improve code for metrics and switch tokio-tar to astral-tokio-tar (#705) 
* improve code for metrics and switch tokio-tar to astral-tokio-tar

* remove log

* fix
 2025-10-24 13:07:56 +08:00
houseme
e22b24684f chore: bump dependencies, add metrics support, remove DNS resolver (#699) 
* upgrade version

* add metrics

* remove dns resolver

* add metrics counter for create bucket

* fix

* fix

* fix
 2025-10-24 00:16:17 +08:00
weisd
a65856bdf4 Fix CRC32C Checksum Implementation and Enhance Authentication System (#678) 
* fix: get_condition_values

* fix checksum crc32c

* fix clippy
 2025-10-21 21:28:00 +08:00
weisd
cd1e244c68 Refactor: Introduce content checksums and improve multipart/object metadata handling (#671) 
* feat:  adapt to s3s typed etag support

* refactor: move replication struct to rustfs_filemeta, fix filemeta transition bug

* add head_object checksum, filter object metadata output

* fix multipart checksum

* fix multipart checksum

* add content md5,sha256 check

* fix test

* fix cargo

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
 2025-10-20 23:46:13 +08:00
Nugine
7f24dbda19 build(deps): upgrade s3s (#667) 
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-10-19 18:32:01 +08:00
houseme
42d3645d6f fix(targets): make target removal and reload transactional; prevent reappearing entries (#662) 
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <1948535941@qq.com>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <1948535941@qq.com>

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix: validate mqtt broker

* improve code for `import`

* fix

* improve

* remove logger from `rustfs-obs` crate

* remove code for config Observability

* fix

* improve code

* fix comment

* up

* up

* upgrade version

* fix

* fmt

* upgrade tokio version to 1.48.0

* upgrade `datafusion` and `reed-solomon-simd` version

* fix

* fmt

* improve code for notify webhook example

* improve code

* fix

* fix

* fmt

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: Nugine <nugine@foxmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-10-17 15:34:53 +08:00
gatewayJ
aae768f446 feat: Simple OPA support (#644) 
* opa-feature

* Update crates/policy/src/policy/opa.rs

* add the content related to 'Copyright'

---------

Co-authored-by: root <root@debian.localdomain>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-10-16 10:35:26 +08:00
安正超
d447b3e426 feat: adapt to s3s typed etag support (#653) 
* feat:  adapt to s3s typed etag support

* refactor: streamline etag handling
 2025-10-15 21:27:20 +08:00
安正超
639bf0c233 Revert "feat(append): implement object append operations with state tracking (#599)" (#646) 
This reverts commit 4f73760a45.
 2025-10-12 23:47:51 +08:00
Copilot
ad99019749 Add complete MNMD Docker deployment example with startup coordination and VolumeNotFound fix (#642) 
* Initial plan

* Add MNMD Docker deployment example with 4 nodes x 4 drives

- Create docs/examples/mnmd/ directory structure
- Add docker-compose.yml with proper disk indexing (1..4)
- Add wait-and-start.sh for startup coordination
- Add README.md with usage instructions and alternatives
- Add CHECKLIST.md with step-by-step verification
- Fixes VolumeNotFound issue by using correct volume paths
- Implements health checks and startup ordering
- Uses service names for stable inter-node addressing

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add docs/examples README as index for deployment examples

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Add automated test script for MNMD deployment

- Add test-deployment.sh with comprehensive validation
- Test container status, health, endpoints, connectivity
- Update README to reference test script
- Make script executable

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* improve code

* improve code

* improve dep crates `cargo shear --fix`

* upgrade aws-sdk-s3

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-10-12 13:15:14 +08:00
houseme
aac9b1edb7 chore: improve event and docker-compose ,Improve the permissions of the endpoint health interface, upgrade otel from 0.30.0 to 0.31.0 (#620) 
* feat: improve code for notify

* upgrade starshard version

* upgrade version

* Fix ETag format to comply with HTTP standards by wrapping with quotes (#592)

* Initial plan

* Fix ETag format to comply with HTTP standards by wrapping with quotes

Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>

* bufigx

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

* Improve lock (#596)

* improve lock

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(tests): add wait_for_object_absence helper and improve lifecycle test reliability

Signed-off-by: Mu junxiang <1948535941@qq.com>

* chore: remove dirty docs

Signed-off-by: Mu junxiang <1948535941@qq.com>

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>

* feat(append): implement object append operations with state tracking (#599)

* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>

* build(deps): upgrade s3s (#595)

Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix: validate mqtt broker

* improve code for `import`

* upgrade otel relation crates version

* fix：dep("jsonwebtoken") feature = 'rust_crypto'

* fix

* fix

* fix

* upgrade version

* improve code for ecfs

* chore: improve event and docker-compose ,Improve the permissions of the `endpoint` health interface

* fix

* fix

* fix

* fix

* improve code

* fix

---------

Signed-off-by: Mu junxiang <1948535941@qq.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: overtrue <1472352+overtrue@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: Nugine <nugine@foxmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-10-11 09:08:25 +08:00
Nugine
626c7ed34a fix: CompleteMultipartUpload encryption (#626) 2025-10-08 20:27:40 +08:00
houseme
0e680eae31 fix typos and bump the dependencies group with 9 updates (#614) 
* fix typos

* build(deps): bump the dependencies group with 9 updates (#613)

Bumps the dependencies group with 9 updates:

| Package | From | To |
| --- | --- | --- |
| [axum](https://github.com/tokio-rs/axum) | `0.8.4` | `0.8.6` |
| [axum-extra](https://github.com/tokio-rs/axum) | `0.10.1` | `0.10.3` |
| [regex](https://github.com/rust-lang/regex) | `1.11.2` | `1.11.3` |
| [serde](https://github.com/serde-rs/serde) | `1.0.226` | `1.0.228` |
| [shadow-rs](https://github.com/baoyachi/shadow-rs) | `1.3.0` | `1.4.0` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.37.0` | `0.37.1` |
| [thiserror](https://github.com/dtolnay/thiserror) | `2.0.16` | `2.0.17` |
| [tokio-rustls](https://github.com/rustls/tokio-rustls) | `0.26.3` | `0.26.4` |
| [zeroize](https://github.com/RustCrypto/utils) | `1.8.1` | `1.8.2` |


Updates `axum` from 0.8.4 to 0.8.6
- [Release notes](https://github.com/tokio-rs/axum/releases)
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/axum/compare/axum-v0.8.4...axum-v0.8.6)

Updates `axum-extra` from 0.10.1 to 0.10.3
- [Release notes](https://github.com/tokio-rs/axum/releases)
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/axum/compare/axum-extra-v0.10.1...axum-extra-v0.10.3)

Updates `regex` from 1.11.2 to 1.11.3
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.11.2...1.11.3)

Updates `serde` from 1.0.226 to 1.0.228
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.226...v1.0.228)

Updates `shadow-rs` from 1.3.0 to 1.4.0
- [Release notes](https://github.com/baoyachi/shadow-rs/releases)
- [Commits](https://github.com/baoyachi/shadow-rs/compare/1.3.0...v1.4.0)

Updates `sysinfo` from 0.37.0 to 0.37.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.37.0...v0.37.1)

Updates `thiserror` from 2.0.16 to 2.0.17
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/2.0.16...2.0.17)

Updates `tokio-rustls` from 0.26.3 to 0.26.4
- [Release notes](https://github.com/rustls/tokio-rustls/releases)
- [Commits](https://github.com/rustls/tokio-rustls/compare/v/0.26.3...v/0.26.4)

Updates `zeroize` from 1.8.1 to 1.8.2
- [Commits](https://github.com/RustCrypto/utils/compare/zeroize-v1.8.1...zeroize-v1.8.2)

---
updated-dependencies:
- dependency-name: axum
  dependency-version: 0.8.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: axum-extra
  dependency-version: 0.10.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: regex
  dependency-version: 1.11.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serde
  dependency-version: 1.0.228
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: shadow-rs
  dependency-version: 1.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: sysinfo
  dependency-version: 0.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: thiserror
  dependency-version: 2.0.17
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tokio-rustls
  dependency-version: 0.26.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: zeroize
  dependency-version: 1.8.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-10-02 23:29:18 +08:00
weisd
7622b37f7b add iam notification (#604) 
move tonic service to rustfs
 2025-09-30 17:32:23 +08:00
Nugine
f1dd3a982e build(deps): upgrade s3s (#595) 
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-09-28 21:10:42 +08:00
guojidan
4f73760a45 feat(append): implement object append operations with state tracking (#599) 
* feat(append): implement object append operations with state tracking

Signed-off-by: junxiang Mu <1948535941@qq.com>

* chore: rebase

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
 2025-09-27 20:06:26 -07:00
weisd
90f21a9102 refactor: Reimplement bucket replication system with enhanced architecture (#590) 
* feat:refactor replication

* use aws sdk for replication client

* refactor/replication

* merge main

* fix lifecycle test
 2025-09-26 14:27:53 +08:00
houseme
9b7f4d477a Fix Tokio Runtime Initialization: Remove Private API Usage and Ensure IO Enabled (#587) 
* fix: remove code

* improve code for tokio runtime config

* improve code for main

* fix: add tokio enable_all

* upgrade version

* improve for Cargo.toml
 2025-09-24 22:23:31 +08:00
Copilot
29b0935be7 RustFS rustfs-audit Complete Implementation with Enterprise Observability (#557) 
* Initial plan

* Implement core audit system with multi-target fan-out and configuration management

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Changes before error encountered

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* Complete audit system with comprehensive observability and test coverage

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* improve code

* fix

* improve code

* fix test

* fix test

* fix

* add `rustfs-audit` to `rustfs`

* upgrade crate version

* fmt

* fmt

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-09-24 08:23:46 +08:00
guojidan
9ddf6a011d feature: support kms && encryt (#573) 
* feat(kms): implement key management service with local and vault backends

Signed-off-by: junxiang Mu <1948535941@qq.com>

* feat(kms): enhance security with zeroize for sensitive data and improve key management

Signed-off-by: junxiang Mu <1948535941@qq.com>

* remove Hashi word

Signed-off-by: junxiang Mu <1948535941@qq.com>

* refactor: remove unused request structs from kms handlers

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
 2025-09-22 17:53:05 +08:00
houseme
f7e188eee7 feat: upgrade datafusion to v50.0.0 and update related dependencies f… (#563) 
* feat: upgrade datafusion to v50.0.0 and update related dependencies for compatibility

* fix

* fmt
 2025-09-18 23:30:25 +08:00
houseme
4b9cb512f2 remove crate rustfs-audit-logger (#562) 2025-09-18 17:46:46 +08:00
houseme
a12a3bedc3 feat(obs): optimize WriteMode selection logic in init_telemetry (#546) 
- Refactor WriteMode selection to ensure all variables moved into thread closures are owned types, preventing lifetime issues.
- Simplify and clarify WriteMode assignment for production and non-production environments.
- Improve code readability and maintainability for logger initialization.
 2025-09-16 08:25:37 +08:00
Copilot
cafec06b7e [Optimization] Enhance obs module telemetry.rs with environment-aware logging and production security (#539) 
* Initial plan

* Implement environment-aware logging with production stdout auto-disable

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add mimalloc crate

* fix

* improve code

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
 2025-09-15 14:52:20 +08:00
houseme
0c3079ae5e remove deps (#529) 2025-09-13 21:06:40 +08:00
Copilot
29c004d935 feat: enhance console separation with enterprise-grade security, monitoring, and advanced tower-http integration (#513) 
* Initial plan

* feat: implement console service separation from endpoint

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: add console separation documentation and tests

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: enhance console separation with configurable CORS and improved Docker support

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: implement enhanced console separation with security hardening and monitoring

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* refactor: implement console TLS following endpoint logic and improve configuration

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* add tower-http feature "timeout｜limit"

* add dependencies crates `axum-server`

* refactor: reconstruct console server with enhanced tower-http features and environment variables

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade dep

* improve code for dns and console port `:9001`

* improve code

* fix

* docs: comprehensive improvement of console separation documentation and Docker deployment standards

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* fmt

* add logs

* improve code for Config handler

* remove logs

* fix

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-09-13 14:48:14 +08:00
guojidan
124c31a68b refactor(profiling): Remove performance profiling support for Windows and optimize dependency management (#518) 
Remove the pprof performance profiling functionality on the Windows platform, as this platform does not support the relevant features
Move the pprof dependency to the platform-specific configuration for non-Windows systems
Update the performance profiling endpoint handling logic to distinguish between platform support statuses
Add the CLAUDE.md document to explain project build and architecture information

Signed-off-by: RustFS Developer <dandan@rustfs.com>
Co-authored-by: RustFS Developer <dandan@rustfs.com>
 2025-09-12 09:11:44 +08:00
guojidan
62a01f3801 Performance: improve (#514) 
* Performance: improve

Signed-off-by: junxiang Mu <1948535941@qq.com>

* remove dirty

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix some err

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
 2025-09-11 19:48:28 +08:00
guojidan
d4beb1cc0b Fix lock (#510) 
* Refactor: reimplement lock

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Fix: fix test case failed

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Improve: lock pref

Signed-off-by: junxiang Mu <1948535941@qq.com>

* fix(lock): Fix resource cleanup issue when batch lock acquisition fails
Ensure that the locks already acquired are properly released when batch lock acquisition fails to avoid memory leaks
Improve the lock protection mechanism to prevent double release issues
Add complete Apache license declarations to all files

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: junxiang Mu <1948535941@qq.com>
 2025-09-11 12:10:35 +08:00
Copilot
ca9a2b6ab9 feat: Implement enhanced DNS resolver with hickory-resolver, TLS support, and layered fallback for Kubernetes environments (#505) 
* Initial plan

* feat: Implement layered DNS resolver with caching and validation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Integrate DNS resolver into main application and fix formatting

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement enhanced DNS resolver with Moka cache and layered fallback

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement hickory-resolver with TLS support for enhanced DNS resolution

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-09-10 21:16:33 +08:00
guojidan
9d5ed1acac Feature/scanner performance optimization (#498) 
* Refactor: reimplement scanner

Signed-off-by: RustFS Developer <dandan@rustfs.com>

* comment lock

Signed-off-by: junxiang Mu <1948535941@qq.com>

* remove dirty file

Signed-off-by: junxiang Mu <1948535941@qq.com>

* Fix: fix rebase

* fix(scanner): Improve error handling and logging

Signed-off-by: junxiang Mu <1948535941@qq.com>

---------

Signed-off-by: RustFS Developer <dandan@rustfs.com>
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: RustFS Developer <dandan@rustfs.com>
 2025-09-08 18:35:45 +08:00
dependabot[bot]
cf4d63795f build(deps): bump crc-fast from 1.4.0 to 1.5.0 in the dependencies group (#481) 
Bumps the dependencies group with 1 update: [crc-fast](https://github.com/awesomized/crc-fast-rust).


Updates `crc-fast` from 1.4.0 to 1.5.0
- [Release notes](https://github.com/awesomized/crc-fast-rust/releases)
- [Changelog](https://github.com/awesomized/crc-fast-rust/blob/main/CHANGELOG.md)
- [Commits](https://github.com/awesomized/crc-fast-rust/compare/1.4.0...1.5.0)

---
updated-dependencies:
- dependency-name: crc-fast
  dependency-version: 1.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: weisd <im@weisd.in>
 2025-09-03 17:30:08 +08:00
houseme
04bf4b0f98 feat: add S3 object legal hold and retention management APIs (#476) 
* add bucket rule

* translation

* improve code for event notice add rule
 2025-09-02 00:14:10 +08:00