* fix getobject content length resp
* Fix regression in exception handling for non-existent key with enhanced compression predicate and metadata improvements (#915)
* Initial plan
* Fix GetObject regression by excluding error responses from compression
The issue was that CompressionLayer was attempting to compress error responses,
which could cause Content-Length header mismatches. By excluding 4xx and 5xx
responses from compression, we ensure error responses (like NoSuchKey) are sent
correctly without body truncation.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* Enhance NoSuchKey fix with improved compression predicate and comprehensive tests
- Enhanced ShouldCompress predicate with size-based exclusion (< 256 bytes)
- Added detailed documentation explaining the compression logic
- Added debug logging for better observability
- Created comprehensive test suite with 4 test cases:
- test_get_deleted_object_returns_nosuchkey
- test_head_deleted_object_returns_nosuchkey
- test_get_nonexistent_object_returns_nosuchkey
- test_multiple_gets_deleted_object
- Added extensive inline documentation and comments
- Created docs/fix-nosuchkey-regression.md with full analysis
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* Add compression best practices documentation
Added comprehensive guide covering:
- Best practices for HTTP response compression
- Common pitfalls and solutions
- Performance considerations and trade-offs
- Testing guidelines and examples
- Monitoring and alerting recommendations
- Migration guide for existing services
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fmt
* fmt
* Fix/objectdelete (#917)
* fix getobject content length resp
* fix delete object
---------
Co-authored-by: houseme <housemecn@gmail.com>
* Add comprehensive analysis of NoSuchKey fix and related improvements
Created detailed documentation analyzing:
- HTTP compression layer fix (primary issue)
- Content-length calculation fix from PR #917
- Delete object metadata fixes from PR #917
- How all components work together
- Complete scenario walkthrough
- Performance impact analysis
- Testing strategy and deployment checklist
This ties together all the changes in the PR branch including the merged
improvements from PR #917.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* replace `once_cell` to `std`
* fmt
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: weisd <im@weisd.in>
* fmt
---------
Co-authored-by: weisd <weishidavip@163.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: weisd <im@weisd.in>
* Initial plan
* Implement priority-based heal queue with deduplication
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* Apply cargo fmt formatting fixes
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* Add queue monitoring, better error handling, and adaptive processing
- Add priority-based queue statistics tracking
- Implement queue capacity warnings (>80% full)
- Process multiple tasks per cycle when capacity allows
- Add proper error logging for failed heal request submissions
- Add Hash trait to HealPriority for HashMap support
- Improve observability with detailed queue status logs
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* Add comprehensive diagnostic logging to heal flow
- Add detailed logging for shard availability checks during reads
- Log successful heal request submissions
- Add detailed logging in heal_object for disk checking
- Log each disk's heal decision with reason and endpoint
- Add logging for heal writing and renaming phases
- Improve variable naming (nil_count -> available_shards)
- Add success confirmation messages for renamed healed data
These logs will help diagnose where the heal flow is failing.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* feat(heal): enhance robustness, safety, and observability of heal process
- **Logging & Observability**:
- Upgrade critical heal logs from [info](/crates/e2e_test/src/reliant/node_interact_test.rs:196:0-213:1) to `warn` for better visibility.
- Implement structured logging with `tracing` fields for machine readability.
- Add `#[tracing::instrument]` to [HealTask](c/crates/ahm/src/heal/task.rs:182:0-205:1) and [SetDisks](/crates/ecstore/src/set_disk.rs:120:0-131:1) methods for automatic context propagation.
- **Robustness**:
- Add exponential backoff retry (3 attempts) for acquiring write locks in [heal_object](/crates/ahm/src/heal/storage.rs:438:4-460:5) to handle contention.
- Handle [rename_data](/crates/ecstore/src/set_disk.rs:392:4-516:5) failures gracefully by preserving temporary files instead of forcing deletion, preventing potential data loss.
- **Data Safety**:
- Fix [object_exists](/crates/ahm/src/heal/storage.rs:395:4-412:5) to propagate IO errors instead of treating them as "object not found".
- Update [ErasureSetHealer](/crates/ahm/src/heal/erasure_healer.rs:28:0-33:1) to mark objects as failed rather than skipped when existence checks error, ensuring they are tracked for retry.
* fix
* fmt
* improve code for heal_object
* fix
* fix
* fix
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Avoid setting AuditSystemState::Starting when target list is empty.
Now checks target availability before state transition, keeping the
system in Stopped state if no enabled targets are found.
- Check targets.is_empty() before setting Starting state
- Return early with Ok(()) when no targets exist
- Maintain consistent state machine behavior
- Prevent transient "Starting" state with no actual targets
Resolves issue where audit system would incorrectly enter Starting
state even when configuration contained no enabled targets.
* feat(obs): unify metrics initialization and fix exporter move error
- Fix Rust E0382 (use after move) by removing duplicate MetricExporter consumption.
- Consolidate MeterProvider construction into single Recorder builder path.
- Remove redundant Recorder::builder(...).install_global() call.
- Ensure PeriodicReader setup is performed only once (HTTP + optional stdout).
- Set global meter provider and metrics recorder exactly once.
- Preserve existing behavior for stdout/file vs HTTP modes.
- Minor cleanup: consistent resource reuse and interval handling.
* update telemetry.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix
* fix
* fix
* fix: modify logger level from error to event
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix
* chore: upgrade cryptography libraries to RC versions
- Upgrade aes-gcm to 0.11.0-rc.2 with rand_core support
- Upgrade chacha20poly1305 to 0.11.0-rc.2
- Upgrade argon2 to 0.6.0-rc.2 with std features
- Upgrade hmac to 0.13.0-rc.3
- Upgrade pbkdf2 to 0.13.0-rc.2
- Upgrade rsa to 0.10.0-rc.10
- Upgrade sha1 and sha2 to 0.11.0-rc.3
- Upgrade md-5 to 0.11.0-rc.3
These upgrades provide enhanced security features and performance
improvements while maintaining backward compatibility with existing
encryption workflows.
* add
* improve code
* fix
* improve code for audit
* improve code ecfs.rs
* improve code
* improve code for ecfs.rs
* feat(storage): refactor audit and notification with OperationHelper
This commit introduces a significant refactoring of the audit logging and event notification mechanisms within `ecfs.rs`.
The core of this change is the new `OperationHelper` struct, which encapsulates and simplifies the logic for both concerns. It replaces the previous `AuditHelper` and manual event dispatching.
Key improvements include:
- **Unified Handling**: `OperationHelper` manages both audit and notification builders, providing a single, consistent entry point for S3 operations.
- **RAII for Automation**: By leveraging the `Drop` trait, the helper automatically dispatches logs and notifications when it goes out of scope. This simplifies S3 method implementations and ensures cleanup even on early returns.
- **Fluent API**: A builder-like pattern with methods such as `.object()`, `.version_id()`, and `.suppress_event()` makes the code more readable and expressive.
- **Context-Aware Logic**: The helper's `.complete()` method intelligently populates log details based on the operation's `S3Result` and only triggers notifications on success.
- **Modular Design**: All helper logic is now isolated in `rustfs/src/storage/helper.rs`, improving separation of concerns and making `ecfs.rs` cleaner.
This refactoring significantly enhances code clarity, reduces boilerplate, and improves the robustness of logging and notification handling across the storage layer.
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* improve code for audit and notify
* fix
* fix
* fix
* improve code for metrics
* improve code for metrics
* fix
* fix
* Refactor telemetry initialization and environment functions ordering
- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.
* fix
* up
* fix
* fix
* improve code for obs
* fix
* fix
* fix: resolve logic errors in ahm heal module
- Fix response publishing logic in HealChannelProcessor to properly handle errors
- Fix negative index handling in DiskStatusChange event to fail fast instead of silently converting to 0
- Enhance timeout control in heal_erasure_set Step 3 loop to immediately respond to cancellation/timeout
- Add proper error propagation for task cancellation and timeout in bucket healing loop
* fix: stabilize performance impact measurement test
- Increase measurement count from 3 to 5 runs for better stability
- Increase workload from 5000 to 10000 operations for more accurate timing
- Use median of 5 measurements instead of single measurement
- Ensure with_scanner duration is at least baseline to avoid negative overhead
- Increase wait time for scanner state stabilization
* wip
* Update crates/ahm/src/heal/channel.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* refactor: remove redundant ok_or_else + expect in event.rs
Replace redundant ok_or_else() + expect() pattern with
unwrap_or_else() + panic!() to avoid creating unnecessary Error
type when the value will panic anyway. This also defers error
message formatting until the error actually occurs.
* Update crates/ahm/src/heal/task.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix(ahm): fix logic errors and add unit tests
- Fix panic in HealEvent::to_heal_request for invalid indices
- Replace unwrap() calls with proper error handling in resume.rs
- Fix race conditions and timeout calculation in task.rs
- Fix semaphore acquisition error handling in erasure_healer.rs
- Improve error message for large objects in storage.rs
- Add comprehensive unit tests for progress, event, and channel modules
- Fix clippy warning: move test module to end of file in heal_channel.rs
* style: apply cargo fmt formatting
* refactor(ahm): address copilot review suggestions
- Add comment to check_control_flags explaining why return value is discarded
- Fix hardcoded median index in performance test using constant and dynamic calculation
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* add Jemalloc
* feat: optimize AI rules with unified .rules.md (#401)
* feat: optimize AI rules with unified .rules.md and entry points
- Create .rules.md as the central AI coding rules file
- Add .copilot-rules.md as GitHub Copilot entry point
- Add CLAUDE.md as Claude AI entry point
- Incorporate principles from rustfs.com project
- Add three critical rules:
1. Use English for all code comments and documentation
2. Clean up temporary scripts after use
3. Only make confident modifications
* Update CLAUDE.md
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: translate chinese to english (#402)
* Checkpoint before follow-up message
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Translate project documentation and comments from Chinese to English
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Fix typo: "unparseable" to "unparsable" in version test comment
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Refactor compression test code with minor syntax improvements
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix: the automatic logout issue and user list display failure on Windows systems (#353) (#343) (#403)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
* upgrade version
* improve code for profiling
* fix
* Initial plan
* feat: Implement layered DNS resolver with caching and validation
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* feat: Integrate DNS resolver into main application and fix formatting
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* feat: Implement enhanced DNS resolver with Moka cache and layered fallback
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* feat: Implement hickory-resolver with TLS support for enhanced DNS resolution
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* upgrade
* add .gitignore config
* fix
* add
* add
* up
* improve linux profiling
* fix
* fix
* fix
* feat(admin): Refactor profiling endpoints
Replaces the existing pprof profiling endpoints with new trigger-based APIs for CPU and memory profiling. This change simplifies the handler logic by moving the profiling implementation to a dedicated module.
A new handler file `admin/handlers/profile.rs` is created to contain the logic for these new endpoints. The core profiling functions are now expected to be in the `profiling` module, which the new handlers call to generate and save profile data.
* cargo shear --fix
* fix
* fix
* fix
---------
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* Add InvalidRangeSpec error
* Add EntityTooSmall to from_u32
* Add InvalidRangeSpec to from_u32
* Map InvalidRangeSpec to correct S3ErrorCode
* Return Error::InvalidRangeSpec
* Use auto implementation
* Add default storage class to ListObjectsV2
Resolves#764
* Add storage_class to response
* Make storage class optional so default won't be an empty string
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix: when the error returned by make_bucket is BucketExists, replace BucketAlreadyExists with BucketAlreadyOwnedByYou (#719)
* test: In the test_api_error_from_storage_error_mappings test method, modify the corresponding mapping relationships
---------
Co-authored-by: weisd <im@weisd.in>
* improve code for dns log
* fix
* Improve comments, remove unused parameters in config.rs (opt), add observability enable flag, and enhance error logging in run function execution.
- Reduce metrics push frequency from default to 3s for better performance
- Optimize resource utilization during metrics collection
- Improve real-time monitoring responsiveness
Related to admin metrics optimization on fix/admin-metrics branch
* chore(docs): move root `examples` to `docs/examples/docker` and update README
- Move root `examples/` contents into `docs/examples/docker/`.
- Update `docs/examples/README.md` to add migration note, new `docker/` entry and usage examples.
- Replace references from `examples/` to `docs/examples/docker/` where applicable.
- Reminder: verify CI and external links still point to the correct paths.
* fix
- Normalize ETags by removing quotes before comparison in complete_multipart_upload
- Fix ETag comparison in replication logic to handle quoted ETags from API responses
- Fix ETag comparison in transition object logic
- Add unit tests for trim_etag function
This fixes the ETag mismatch error when uploading large files (5GB+) via multipart upload,
which was caused by PR #592 adding quotes to ETag responses while internal storage remains unquoted.
Fixes#625
* feat(append): implement object append operations with state tracking
Signed-off-by: junxiang Mu <1948535941@qq.com>
* chore: rebase
Signed-off-by: junxiang Mu <1948535941@qq.com>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Add lock timeout support and track acquisition time in lock state
- Improve lock conflict handling with detailed error messages
- Optimize lock reuse when already held by same owner
- Refactor lock state to store owner info and timeout duration
- Update all lock operations to handle new state structure
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix: remove code
* improve code for tokio runtime config
* improve code for main
* fix: add tokio enable_all
* upgrade version
* improve for Cargo.toml
* fix: fix datausageinfo
Signed-off-by: junxiang Mu <1948535941@qq.com>
* feat(data-usage): implement local disk snapshot aggregation for data usage statistics
Signed-off-by: junxiang Mu <1948535941@qq.com>
* feat(scanner): improve data usage collection with local scan aggregation
Signed-off-by: junxiang Mu <1948535941@qq.com>
* refactor: improve object existence check and code style
Signed-off-by: junxiang Mu <1948535941@qq.com>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
* feat(kms): implement key management service with local and vault backends
Signed-off-by: junxiang Mu <1948535941@qq.com>
* feat(kms): enhance security with zeroize for sensitive data and improve key management
Signed-off-by: junxiang Mu <1948535941@qq.com>
* remove Hashi word
Signed-off-by: junxiang Mu <1948535941@qq.com>
* refactor: remove unused request structs from kms handlers
Signed-off-by: junxiang Mu <1948535941@qq.com>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Refactor WriteMode selection to ensure all variables moved into thread closures are owned types, preventing lifetime issues.
- Simplify and clarify WriteMode assignment for production and non-production environments.
- Improve code readability and maintainability for logger initialization.
* fix(main): update dns_init startup logic and remove unused code in http.rs
- Refactored the DNS resolver initialization logic in `main.rs` for improved startup reliability and error handling.
- Removed unused code from `http.rs` to keep the codebase clean and maintainable.
* feat(console): support accessing console endpoint via port 9000
- Added compatibility to allow console access through port 9000.
- Improved endpoint detection and routing for console service on standard and custom ports.
- Enhanced user experience for environments using port 9000 as the default access point.
- Refactored the DNS resolver initialization logic in `main.rs` for improved startup reliability and error handling.
- Removed unused code from `http.rs` to keep the codebase clean and maintainable.
Increase the size of the notification pool to reduce the thundering herd effect under high concurrency
Implement an adaptive timeout mechanism that dynamically adjusts based on system load and priority
Add a lock protection mechanism to prevent premature cleanup of active locks
Add lock acquisition methods for high-priority and critical-priority locks
Improve the cleanup strategy to be more conservative under high load
Add detailed debug logs to assist in diagnosing lock issues
Signed-off-by: junxiang Mu <1948535941@qq.com>
Remove the pprof performance profiling functionality on the Windows platform, as this platform does not support the relevant features
Move the pprof dependency to the platform-specific configuration for non-Windows systems
Update the performance profiling endpoint handling logic to distinguish between platform support statuses
Add the CLAUDE.md document to explain project build and architecture information
Signed-off-by: RustFS Developer <dandan@rustfs.com>
Co-authored-by: RustFS Developer <dandan@rustfs.com>
* feat(lock): Add support for disabling lock manager
Implement control of lock system activation and deactivation via environment variables
Add DisabledLockManager for lock-free operation scenarios
Introduce LockManager trait to uniformly manage different lock managers
Signed-off-by: junxiang Mu <1948535941@qq.com>
* refactor(lock): Optimize implementation of global lock manager and parsing of boolean environment variables
Refactor the implementation of the global lock manager: wrap FastObjectLockManager with Arc and add the as_fast_lock_manager method
Extract the boolean environment variable parsing logic into an independent function parse_bool_env_var
Signed-off-by: junxiang Mu <1948535941@qq.com>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Refactor: reimplement lock
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Fix: fix test case failed
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Improve: lock pref
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix(lock): Fix resource cleanup issue when batch lock acquisition fails
Ensure that the locks already acquired are properly released when batch lock acquisition fails to avoid memory leaks
Improve the lock protection mechanism to prevent double release issues
Add complete Apache license declarations to all files
Signed-off-by: junxiang Mu <1948535941@qq.com>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Merge all AI rules from .rules.md, .cursorrules, and CLAUDE.md into AGENTS.md
- Add competitor keyword prohibition rules (minio, ceph, swift, etc.)
- Simplify rules by removing overly detailed code examples
- Integrate new development principles as highest priority
- Remove old tool-specific rule files
- Fix clippy warnings for format string improvements
* Update mod.rs
The following code uses a separator that is not compatible with Windows:
format!("{}/{}", file_config.path.clone(), rustfs_config::DEFAULT_SINK_FILE_LOG_FILE)
Change it to the following code:
std::path::Path::new(&file_config.path)
.join(rustfs_config::DEFAULT_SINK_FILE_LOG_FILE)
.to_string_lossy()
.to_string()
* Replaced format! macro with PathBuf::join to fix path separator issue on Windows.Tested on Windows 10 with Rust 1.85.0, paths now correctly use \ separator.
* feat: support conditional writes
* refactor: avoid using unwrap
* fix: obtain lock before check in CompleteMultiPartUpload
* refactor: do not obtain a lock when getting object meta
* fix: avoid using unwrap and modifying incoming arguments
* test: add e2e tests for conditional writes
---------
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
* feat: add support for range requests in upload_part_copy and implement parse_copy_source_range function
* style: format debug and error logging for improved readability
* feat: implement parse_copy_source_range function and improve error handling in range requests
* Update rustfs/src/storage/ecfs.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: correct return type in parse_copy_source_range function
* fix: remove unnecessary unwrap in parse_copy_source_range tests
* fix: simplify etag comparison in copy condition validation
---------
Co-authored-by: DamonXue <damonxue2@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
This commit completely removes the rustfs-gui module from the project. The deletion includes:
- All source code files (*.rs) and associated resources
- GUI-specific dependencies from Cargo.toml
- Build scripts and configuration files specific to the GUI module
- Documentation and assets related to the graphical interface
The removal is performed because:
- The GUI component is no longer maintained
- Focus is shifting to core functionality and CLI interface
- Limited resources available for GUI development and maintenance
The core filesystem functionality remains available through the rustfs library and CLI interface.
* init audit logger module
* add audit webhook default config kvs
* feat: Add comprehensive tests for authentication module (#309)
* feat: add comprehensive tests for authentication module
- Add 33 unit tests covering all public functions in auth.rs
- Test IAMAuth struct creation and secret key validation
- Test check_claims_from_token with various credential types and scenarios
- Test session token extraction from headers and query parameters
- Test condition values generation for different user types
- Test query parameter parsing with edge cases
- Test Credentials helper methods (is_expired, is_temp, is_service_account)
- Ensure tests handle global state dependencies gracefully
- All tests pass successfully with 100% coverage of testable functions
* style: fix code formatting issues
* Add verification script for checking PR branch statuses and tests
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* fix: resolve clippy uninlined format args warning
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: add basic tests for core storage module (#313)
* feat: add basic tests for core storage module
- Add 6 unit tests for FS struct and basic functionality
- Test FS creation, Debug and Clone trait implementations
- Test RUSTFS_OWNER constant definition and values
- Test S3 error code creation and handling
- Test compression format detection for common file types
- Include comprehensive documentation about integration test needs
Note: Full S3 API testing requires complex setup with storage backend,
global configuration, and network infrastructure - better suited for
integration tests rather than unit tests.
* style: fix code formatting issues
* fix: resolve clippy warnings in storage tests
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: add tests for admin handlers module (#314)
* feat: add tests for admin handlers module
- Add 5 new unit tests for admin handler functionality
- Test AccountInfo struct creation, serialization and default values
- Test creation of all admin handler structs (13 handlers)
- Test HealOpts JSON serialization and deserialization
- Test HealOpts URL encoding/decoding with proper field types
- Maintain existing test while adding comprehensive coverage
- Include documentation about integration test requirements
All tests pass successfully with proper error handling for complex dependencies.
* style: fix code formatting issues
* fix: resolve clippy warnings in admin handlers tests
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* build(deps): bump the dependencies group with 3 updates (#326)
* perf: avoid transmitting parity shards when the object is good (#322)
* upgrade version
* Fix: fix data integrity check
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Fix: Separate Clippy's fix and check commands into two commands.
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix: miss inline metadata (#345)
* Update dependabot.yml
* fix: Fixed an issue where the list_objects_v2 API did not return dire… (#352)
* fix: Fixed an issue where the list_objects_v2 API did not return directory names when they conflicted with file names in the same bucket (e.g., test/ vs. test.txt, aaa/ vs. aaa.csv) (#335)
* fix: adjusted the order of directory listings
* init
* fix
* fix
* feat: add docker usage for rustfs mcp (#365)
* feat: enhance metadata extraction with object name for MIME type detection
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Feature: lock support auto release
Signed-off-by: junxiang Mu <1948535941@qq.com>
* improve lock
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Fix: fix scanner detect
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Fix: clippy && fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
* refactor(ecstore): Optimize memory usage for object integrity verification
Change the object integrity verification from reading all data to streaming processing to avoid memory overflow caused by large objects.
Modify the TLS key log check to use environment variables directly instead of configuration constants.
Add memory limits for object data reading in the AHM module.
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Chore: reduce PR template checklist
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Chore: remove comment code (#376)
Signed-off-by: junxiang Mu <1948535941@qq.com>
* chore: upgrade actions/checkout from v4 to v5 (#381)
* chore: upgrade actions/checkout from v4 to v5
- Update GitHub Actions checkout action version
- Ensure compatibility with latest workflow features
- Maintain existing checkout behavior and configuration
* upgrade version
* fix
* add and improve code for notify
* feat: extend rustfs mcp with bucket creation and deletion (#416)
* feat: extend rustfs mcp with bucket creation and deletion
* update file to fix pipeline error
* change variable name to fix pipeline error
* fix(ecstore): add async-recursion to resolve nightly trait solver reg… (#415)
* fix(ecstore): add async-recursion to resolve nightly trait solver regression
The newest nightly compiler switched to the new trait solver, which
currently rejects async recursive functions that were previously accepted.
This causes the following compilation failures:
- `LocalDisk::delete_file()`
- `LocalDisk::scan_dir()`
Add `async-recursion` as a workspace dependency and annotate both functions with `#[async_recursion]` so that the crate compiles cleanly with the latest nightly and will continue to build once the new solver lands in stable.
Signed-off-by: reigadegr <2722688642@qq.com>
* fix: resolve duplicate bound error in scan_dir function
Replaced inline trait bounds with where clause to avoid duplication caused by macro expansion.
Signed-off-by: reigadegr <2722688642@qq.com>
---------
Signed-off-by: reigadegr <2722688642@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
* fix:make bucket exists (#428)
* feat: include user-defined metadata in S3 response (#431)
* fix: simplify Docker entrypoint following efficient user switching pattern (#421)
* fix: simplify Docker entrypoint following efficient user switching pattern
- Remove ALL file permission modifications (no chown at all)
- Use chroot --userspec or gosu to switch user context
- Extremely simple and fast implementation
- Zero filesystem modifications for permissions
Fixes#388
* Update entrypoint.sh
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update entrypoint.sh
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update entrypoint.sh
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* wip
* wip
* wip
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* docs: update doc/docker-data-dir README.md (#432)
* add targets crates
* feat(targets): extract targets module into a standalone crate
- Move all target-related code (MQTT, Webhook, etc.) into a new `targets` crate
- Update imports and dependencies to reference the new crate
- Refactor interfaces to ensure compatibility with the new crate structure
- Adjust Cargo.toml and workspace configuration accordingly
* fix
* fix
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
Signed-off-by: reigadegr <2722688642@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: zzhpro <56196563+zzhpro@users.noreply.github.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: reigadegr <103645642+reigadegr@users.noreply.github.com>
Co-authored-by: 0xdx2 <xuedamon2@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* refactor: replace print statements with proper logging and fix grammar
- Fix English grammar errors in existing log messages
- Add tracing imports where needed
- Improve log message clarity and consistency
- Follow project logging best practices using tracing crate
* fix: resolve clippy warnings and format code
- Fix unused import warnings by making test imports conditional with #[cfg(test)]
- Fix unused variable warning by prefixing with underscore
- Run cargo fmt to fix formatting issues
- Ensure all code passes clippy checks with -D warnings flag
* refactor: move tracing::debug import into test module
Move the tracing::debug import from file-level #[cfg(test)] into the test module itself for better code organization and consistency with other test modules
* Checkpoint before follow-up message
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* refactor: move tracing::debug import into test module in user_agent.rs
Complete the refactoring by moving the tracing::debug import from file-level #[cfg(test)] into the test module for consistency across all test files
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(ecstore): add async-recursion to resolve nightly trait solver regression
The newest nightly compiler switched to the new trait solver, which
currently rejects async recursive functions that were previously accepted.
This causes the following compilation failures:
- `LocalDisk::delete_file()`
- `LocalDisk::scan_dir()`
Add `async-recursion` as a workspace dependency and annotate both functions with `#[async_recursion]` so that the crate compiles cleanly with the latest nightly and will continue to build once the new solver lands in stable.
Signed-off-by: reigadegr <2722688642@qq.com>
* fix: resolve duplicate bound error in scan_dir function
Replaced inline trait bounds with where clause to avoid duplication caused by macro expansion.
Signed-off-by: reigadegr <2722688642@qq.com>
---------
Signed-off-by: reigadegr <2722688642@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
* Checkpoint before follow-up message
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Translate project documentation and comments from Chinese to English
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Fix typo: "unparseable" to "unparsable" in version test comment
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Refactor compression test code with minor syntax improvements
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: optimize AI rules with unified .rules.md and entry points
- Create .rules.md as the central AI coding rules file
- Add .copilot-rules.md as GitHub Copilot entry point
- Add CLAUDE.md as Claude AI entry point
- Incorporate principles from rustfs.com project
- Add three critical rules:
1. Use English for all code comments and documentation
2. Clean up temporary scripts after use
3. Only make confident modifications
* Update CLAUDE.md
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
- RELEASE: GitHub release tag without 'v' prefix (e.g., 1.0.0-alpha.42)
- VERSION: filename version with 'v' prefix (e.g., v1.0.0-alpha.42)
- Download URL uses RELEASE for path, VERSION for filename
- Fixes incorrect URL generation that was adding extra 'v' prefix
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
- Modified Dockerfile to download pre-built binaries from GitHub releases
- For latest releases, use GitHub API to find the correct download URL
- For specific versions, construct the GitHub release URL directly
- Updated docker-buildx.sh script messages to reflect new download source
- This change addresses security concerns about potential tampering with binaries from dl.rustfs.com
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* chore: upgrade actions/checkout from v4 to v5
- Update GitHub Actions checkout action version
- Ensure compatibility with latest workflow features
- Maintain existing checkout behavior and configuration
* upgrade version
Change the object integrity verification from reading all data to streaming processing to avoid memory overflow caused by large objects.
Modify the TLS key log check to use environment variables directly instead of configuration constants.
Add memory limits for object data reading in the AHM module.
Signed-off-by: junxiang Mu <1948535941@qq.com>
* feat(lifecycle): Implement object lifecycle management functionality
Add a lifecycle module to automatically handle object expiration and transition during scanning
Modify the file metadata cache module to be publicly visible to support lifecycle operations
Adjust the scanning interval to a shorter time for testing lifecycle rules
Implement the parsing and execution logic for S3 lifecycle configurations
Add integration tests to verify the lifecycle expiration functionality
Update dependencies to support the new lifecycle features
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix cargo dependencies
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix fmt
Signed-off-by: junxiang Mu <1948535941@qq.com>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix: add tokio-test
* fix: "called `unwrap` on `v` after checking its variant with `is_some`"
= help: try using `if let` or `match`
= help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_unwrap
= note: `-D clippy::unnecessary-unwrap` implied by `-D warnings`
= help: to override `-D warnings` add `#[allow(clippy::unnecessary_unwrap)]`
* fmt
* set toolchain 1.88.0
* fmt
* fix: cliip
* fix: Fixed an issue where the list_objects_v2 API did not return directory names when they conflicted with file names in the same bucket (e.g., test/ vs. test.txt, aaa/ vs. aaa.csv) (#335)
* fix: adjusted the order of directory listings
* feat: add tests for admin handlers module
- Add 5 new unit tests for admin handler functionality
- Test AccountInfo struct creation, serialization and default values
- Test creation of all admin handler structs (13 handlers)
- Test HealOpts JSON serialization and deserialization
- Test HealOpts URL encoding/decoding with proper field types
- Maintain existing test while adding comprehensive coverage
- Include documentation about integration test requirements
All tests pass successfully with proper error handling for complex dependencies.
* style: fix code formatting issues
* fix: resolve clippy warnings in admin handlers tests
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: add basic tests for core storage module
- Add 6 unit tests for FS struct and basic functionality
- Test FS creation, Debug and Clone trait implementations
- Test RUSTFS_OWNER constant definition and values
- Test S3 error code creation and handling
- Test compression format detection for common file types
- Include comprehensive documentation about integration test needs
Note: Full S3 API testing requires complex setup with storage backend,
global configuration, and network infrastructure - better suited for
integration tests rather than unit tests.
* style: fix code formatting issues
* fix: resolve clippy warnings in storage tests
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: add comprehensive tests for authentication module
- Add 33 unit tests covering all public functions in auth.rs
- Test IAMAuth struct creation and secret key validation
- Test check_claims_from_token with various credential types and scenarios
- Test session token extraction from headers and query parameters
- Test condition values generation for different user types
- Test query parameter parsing with edge cases
- Test Credentials helper methods (is_expired, is_temp, is_service_account)
- Ensure tests handle global state dependencies gracefully
- All tests pass successfully with 100% coverage of testable functions
* style: fix code formatting issues
* Add verification script for checking PR branch statuses and tests
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* fix: resolve clippy uninlined format args warning
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
- Add global metrics system to common crate for cross-module usage
- Integrate global metrics collection into AHM scanner operations
- Update ECStore to use common metrics system instead of local implementation
- Add chrono dependency to AHM crate for timestamp handling
- Re-export IlmAction from common metrics in ECStore lifecycle module
- Update scanner methods to use global metrics for cycle, disk, and volume scans
- Maintain backward compatibility with local metrics collector
- Fix clippy warnings and ensure proper code formatting
This change enables unified metrics collection across the entire RustFS system,
allowing better monitoring and observability of scanner operations.
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Remove HealType::Disk and related disk-specific healing methods
- Integrate disk format healing into heal_erasure_set with include_format_heal option
- Update auto disk scanner to use ErasureSet heal type instead of Disk heal
- Fix disk status change event handling to use ErasureSet heal requests
- Add proper bucket list retrieval for auto healing scenarios
- Update data scanner to submit ErasureSet heal tasks for offline disks
- Remove duplicate healing logic between Disk and ErasureSet types
- Ensure all healing operations go through unified ErasureSet healing path
- Fix multi-threaded test conflicts in AHM heal integration tests
- Remove global environment sharing to prevent test state pollution
- Fix test_all_disk_method by clearing global disk map before test
- Improve data scanner and cache value implementations
- Update dependencies and resolve clippy warnings
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Add global unbounded channel in common crate for heal requests
- Implement channel processor in ahm to handle heal commands
- Add Start/Query/Cancel commands support via channel
- Integrate heal manager initialization in main.rs
- Replace direct MRF calls with channel-based heal requests in ecstore
- Support advanced heal options including pool_index and set_index
- Enable admin handlers to send heal requests via channel
- Add heal module with core types (HealType, HealRequest, HealTask)
- Implement HealManager for task scheduling and execution
- Add HealStorageAPI trait and ECStoreHealStorage implementation
- Integrate heal capabilities into scanner for automatic repair
- Support multiple heal types: object, bucket, disk, metadata, MRF, EC decode
- Add progress tracking and event system for heal operations
- Merge heal and scanner error types for unified error handling
- Include comprehensive logging and metrics for heal operations
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Move scanner code to scanner/ subdirectory for better organization
- Add data usage statistics collection and persistence
- Implement histogram support for size and version distribution
- Add global cancel token management for scanner operations
- Integrate scanner with ECStore for comprehensive data analysis
- Update error handling and improve test isolation
- Add data usage API endpoints and backend integration
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Refactor NamespaceLock to use a unified client vector and quorum mechanism,
removing legacy local/distributed lock split and related code.
- Update LockRequest to split timeout into acquire_timeout and ttl, and add
builder methods for both.
- Adjust all batch lock APIs to accept ttl and use new LockRequest fields.
- Update all affected tests and documentation for the new API.
Signed-off-by: junxiang Mu <1948535941@qq.com>
- Remove unused core/rwlock.rs and manager/ modules (ManagerFactory, LifecycleManager, NamespaceManager)
- Move all lock-related code into crates/lock/src with clear submodules: client, core, utils, etc.
- Ensure only necessary files and APIs are exposed, improve maintainability
- No functional logic change, pure structure and cleanup refactor
Signed-off-by: dandan <dandan@dandandeMac-Studio.local>
- Fix Windows zip command issue by using PowerShell Compress-Archive
- Add Windows support for OSS upload with ossutil
- Replace Chinese comments with English in build.yml
- Fix bash syntax error in package_zip function
- Improve code formatting and consistency
- Update various configuration files for better cross-platform support
Resolves Windows build failures in GitHub Actions.
Resolves#277
- Add --fix flag to automatically fix clippy warnings
- Add --allow-dirty flag to run on dirty Git trees
- Improves code quality in pre-commit workflow
* fix unzip error
* fix url change error
fix url change error
* Simplify user experience and integrate console and endpoint
Simplify user experience and integrate console and endpoint
* Add gnu to build.yml
* feat: enhance entrypoint and Dockerfiles for flexible volume and permission management\n\n- Support batch mount and permission fix in entrypoint.sh\n- Add coreutils/shadow (alpine) and coreutils/passwd (ubuntu) for UID/GID/ownership\n- Use ENTRYPOINT for unified startup\n- Make local dev and prod Dockerfile behavior consistent\n- Improve security and user experience\n\nBREAKING CHANGE: entrypoint.sh and Dockerfile now require additional packages for permission management, and support batch volume mount via RUSTFS_VOLUMES.
* chore: update Dockerfile comments to English only
* fix(entrypoint): improve local/remote volume detection and permission logic in entrypoint.sh
* Update entrypoint.sh
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update entrypoint.sh
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update Dockerfile
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The Docker workflow was not triggering for tag-based releases because it had
'branches: [main]' restriction in the workflow_run configuration. When pushing
tags, the triggering workflow runs on the tag, not on main branch.
Changes:
- Remove 'branches: [main]' from workflow_run trigger
- Simplify tag detection using github.event.workflow_run context instead of API calls
- Use official workflow_run event properties (head_branch, event) for reliable detection
- Support both 'refs/tags/VERSION' and direct 'VERSION' formats
- Add better logging for debugging workflow trigger issues
This fixes the issue where Docker images were not built for tagged releases.
- Replace git commands with GitHub API calls for tag detection
- Add proper commit checkout for workflow_run events
- Use gh CLI and curl fallback for better reliability
- Add debug output to help troubleshoot tag detection issues
This should fix the issue where Docker builds were not triggered for tagged releases
due to missing tag information in the workflow_run environment.
BREAKING CHANGE: Fixed Docker workflow that was incorrectly skipping builds for tagged releases
- Fix logic to detect tag pushes using git refs instead of branch names
- Properly identify tag pushes vs branch pushes using git show-ref
- Support both v-prefixed and bare version formats
- Ensure Docker images are built for all tagged releases including prereleases
- Add new Makefile Commands section with simplified docker-dev* commands
- Update Development Workflow to use new dev-env-* commands
- Update directory structure (remove deleted alpine/ directory)
- Reorganize build instructions to prioritize Makefile over direct scripts
- Add Common Development Tasks section with make help commands
- Remove obsolete .docker/alpine/Dockerfile.protoc (superseded by Dockerfile.source)
- Simplify Makefile commands by removing backward compatibility aliases
* Replace docker-buildx-source* with shorter docker-dev* commands
* Replace start/stop with explicit dev-env-start/dev-env-stop commands
- Fix Docker workflow version matching logic to correctly distinguish:
* 1.0.0 vs 1.0.0-alpha.11 (prerelease detection)
* Support both v1.0.0 and 1.0.0 formats (with/without v prefix)
* Reorder case patterns to match prereleases before releases
BREAKING CHANGE: Removed legacy command aliases
- Use 'make docker-dev-local' instead of 'make docker-buildx-source-local'
- Use 'make dev-env-start' instead of 'make start'
- Merge release logic into build.yml to avoid cross-workflow artifact access issues
- Add release jobs (create-release, upload-release-assets, update-latest-version, publish-release) that run only for tag pushes
- Use standard actions/download-artifact@v4 within the same workflow (no cross-workflow limitations)
- Deprecate standalone release.yml workflow with warning job and confirmation requirement
- Remove references to deleted release-notes-template.md file from both workflows
- Update build summary messages to reflect integrated release process
This resolves the 'Prepare release assets' failure by eliminating the need for cross-workflow artifact access.
- Replace actions/download-artifact@v4 with GitHub API calls to access artifacts from triggering workflow
- Add proper permissions (contents: read, actions: read) to prepare-assets job
- Handle both workflow_run and workflow_dispatch trigger scenarios
- Fix the root cause: workflow_run events cannot access artifacts from triggering workflows using standard download-artifact action
Fixes the 'Prepare release assets' step failure by implementing cross-workflow artifact access through GitHub API.
* fix: use correct tag reference in release workflow wait-for-artifacts step
- Change ref from github.ref to needs.release-check.outputs.tag
- Fix issue where wait-on-check-action receives full git reference (refs/tags/1.0.0-alpha.21)
instead of clean tag name (1.0.0-alpha.21)
- This resolves timeout errors when waiting for build artifacts during release process
Fixes the release workflow failure for tag 1.0.0-alpha.21
* refactor: use workflow_run trigger for release workflow instead of push
- Replace push trigger with workflow_run to eliminate timing issues
- Release workflow now triggers only after Build workflow completes successfully
- Remove wait-for-artifacts step completely (no longer needed)
- Add should_release condition to control release execution
- Support both tag pushes and manual releases via workflow_dispatch
- Align with docker.yml pattern for better reliability
This completely resolves the release workflow timeout issues by ensuring
build artifacts are always available before the release process starts.
Fixes the fundamental timing issue where release.yml and build.yml
were racing against each other when triggered by the same tag push.
- Change ref from github.ref to needs.release-check.outputs.tag
- Fix issue where wait-on-check-action receives full git reference (refs/tags/1.0.0-alpha.21)
instead of clean tag name (1.0.0-alpha.21)
- This resolves timeout errors when waiting for build artifacts during release process
Fixes the release workflow failure for tag 1.0.0-alpha.21
* feat: disable Docker builds for development versions
- Remove dev-latest, main-latest, and dev-* version options from manual triggers
- Skip Docker builds for development versions in workflow_run events
- Only build Docker images for releases (v1.0.0) and prereleases (v1.0.0-alpha1)
- Simplify tags generation logic by removing development branch handling
- Update workflow documentation to reflect release-only Docker strategy
BREAKING CHANGE: Development Docker images are no longer built automatically
* feat: remove dev channel support from Dockerfile
- Remove CHANNEL build argument (no longer needed)
- Simplify download logic to only support release channel
- Remove dev-specific package download paths
- Update BASE_URL to point directly to release directory
- Remove channel label from Docker image metadata
- Streamline version handling (latest vs specific release)
This aligns with the workflow changes that disabled dev Docker builds.
- Replace manual cut -c1-7 with git rev-parse --short in docker.yml
- Ensures consistent short SHA length between build.yml and docker.yml
- Git automatically adjusts length for uniqueness, preventing conflicts
- Remove linux_builds_success output and related variables
- Simplify build-docker condition to only check should_build
- The should_build check already includes workflow success verification
- Reduce code complexity while maintaining the same functionality
* fix: correct YAML indentation error in docker workflow
- Fix incorrect indentation at line 237 in .github/workflows/docker.yml
- Step 'Extract metadata and generate tags' had 12 spaces instead of 6
- This was causing YAML syntax validation to fail
* fix: restore unified build-rustfs task with correct YAML syntax
- Revert complex job separation back to single build-rustfs task
- Maintain Linux and macOS builds in unified matrix
- Fix YAML indentation and syntax issues
- Docker builds will use only Linux binaries as designed in Dockerfile
* feat: implement precise Docker build triggering using workflow_run
- Use workflow_run event to trigger Docker builds independently
- Add precise Linux build status checking via GitHub API
- Only trigger Docker builds when both Linux architectures succeed
- Remove coupling between build.yml and docker.yml workflows
- Improve TARGETPLATFORM consistency in Dockerfile
This resolves the issue where Docker builds would trigger even if
Linux ARM64 builds failed, causing missing binary artifacts during
multi-architecture Docker image creation.
- Fix incorrect indentation at line 237 in .github/workflows/docker.yml
- Step 'Extract metadata and generate tags' had 12 spaces instead of 6
- This was causing YAML syntax validation to fail
- Change docker.yml to use workflow_call triggered by build.yml
- Remove redundant force_build parameter from build.yml
- Simplify build_docker parameter (build implies push in CI/CD)
- Add proper dependency chain: build.yml -> docker.yml -> registry
- Update documentation to reflect new architecture
- Mark Dockerfile.source as local development only
- Remove cargo.config.toml file that was causing build issues
- Restore .github/workflows/build.yml to working state from commit 2e9792577f
- These changes ensure the build system works correctly again
* fix: Resolve zstd-sys Zig compilation issues
- Remove specific Zig version constraint in action.yml to use default version
- Clean up duplicate environment variable settings in build-rustfs.sh
- Add CARGO_TARGET_*_LINKER environment variables for better cross-compilation support
- Optimize build configuration for consistent cross-platform compilation
Fixes compilation issues with zstd-sys when using Zig cross-compilation.
Aligns with previously working configuration that uses default Zig version.
* fix: Restore working build configuration from 4fb4b353
- Restore matrix.cross parameter to differentiate cross-compilation
- Use simple cargo zigbuild instead of complex build-rustfs.sh script
- Remove unnecessary zstd dependencies from action.yml
- Restore console asset download step
- Use correct target directory path for packaging
- Align with known working configuration from commit 4fb4b353
This reverts to the proven working build approach that successfully
performed cross-platform compilation.
* fix: Align build-rustfs.sh with working version logic
- Simplify build logic to match working version 4fb4b353
- Use exact same build commands as the working build.yml:
* cargo build for native compilation
* cargo zigbuild for Linux ARM64 cross-compilation
* cross build for Windows ARM64 cross-compilation
- Remove complex environment variable setup that caused conflicts
- Add touch rustfs/build.rs to match working version
- Use -p rustfs --bins flag consistent with working version
This ensures build-rustfs.sh (if used) follows the proven working approach.
- Update to mlugg/setup-zig@v2 for better stability and features
- Use Zig 0.13.0 for improved musl target support
- Add system zstd libraries (libzstd-dev, zstd) to Ubuntu dependencies
- Configure environment variables for zstd-sys to use pkg-config
- Enable pkg-config feature for zstd dependency to prefer system library
- Add proper C/C++ compiler configuration for musl targets
Fixes the 'error: unable to parse target query x86_64-unknown-linux-musl: UnknownOperatingSystem'
compilation error in zstd-sys during cross-compilation.
- Add generate_sha256() function to handle cross-platform SHA256 generation
- Use shasum -a 256 on macOS instead of sha256sum
- Use sha256sum on Linux with shasum as fallback
- Replace direct sha256sum usage in build script with new function
- Fixes 'sha256sum: command not found' error on macOS builds
* feat: implement Docker improvements and binary build scripts
This commit transforms the RustFS Docker build system to follow MinIO's best practices:
## 🏗️ Binary Build Script (build-rustfs.sh)
- Create independent binary compilation script for multi-platform builds
- Support x86_64 and aarch64 Linux musl targets
- Include checksum generation and optional binary signing
- Support cross-compilation and upload functionality
- Automated target installation and environment setup
## 🐳 Docker Improvements
- Rewrite Dockerfiles to download precompiled binaries instead of building from source
- Follow MinIO's approach for security and binary verification
- Add comprehensive LABEL metadata (version, build-date, vcs-ref)
- Implement proper environment variable management
- Add signature verification with minisign (commented for future use)
- Include static curl download for minimal runtime dependencies
## 🚀 Enhanced Build Script (docker-buildx.sh)
- Inspired by MinIO's docker-buildx.sh for consistency and reliability
- Support multiple platforms with proper build arguments
- Auto-detect git versions and pass metadata to containers
- Improved error messages with helpful troubleshooting hints
- Cleanup and cache management between builds
## 🛠️ Supporting Scripts
- scripts/download-static-curl.sh: Download statically compiled curl
- scripts/setup-test-binaries.sh: Create test binaries for local development
## 📋 Key Benefits
- Faster Docker builds (download vs compile)
- Better security with signature verification
- Consistent with industry standards (MinIO approach)
- Proper multi-platform support
- Enhanced metadata and traceability
- Independent binary distribution capability
* feat: update Docker files to use Aliyun OSS for binary downloads
* feat: merge stash with OSS binary download improvements
- Remove old build_rustfs.sh script
- Keep Aliyun OSS download URLs for binary retrieval
- Maintain Docker build improvements from stash
- Resolve merge conflicts between stash and OSS updates
* feat: improve build-rustfs.sh with auto platform detection
- Auto-detect current platform using uname (like old build_rustfs.sh)
- Default to building for current platform only
- Add --all-platforms flag for cross-compilation to Linux musl targets
- Support macOS (darwin) and Linux platforms
- Auto-enable cross compilation when needed
- Provide better usage examples and platform detection info
This makes the script much more user-friendly by default while
maintaining flexibility for cross-compilation scenarios.
* refactor: simplify build-rustfs.sh for CI/CD pipeline usage
- Remove cross-compilation complexity (each CI runner builds natively)
- Focus on single platform builds per runner
- Remove --all-platforms and --cross options
- Simplify to match CI/CD workflow where:
* Linux x86_64 runner builds Linux x86_64 binary
* Linux ARM64 runner builds Linux ARM64 binary
* macOS x86_64 runner builds macOS x86_64 binary
* macOS ARM64 runner builds macOS ARM64 binary
- Keep signing and upload functionality for release CI
- Make the script's purpose and usage clearer
This aligns with the user's understanding that build scripts should
focus on native compilation for the current platform only.
* feat: update download server domain to dl.rustfs.com
- Update Dockerfile to use dl.rustfs.com/dev/ for development binaries
- Update Dockerfile.release to use dl.rustfs.com/release/ for release binaries
- Update docker-buildx.sh error messages with new URLs
- Update build-rustfs.sh upload target to dl.rustfs.com
- Update test scripts to reference new domain
- Clean up remaining git conflict markers
This centralizes all binary downloads through the official
dl.rustfs.com domain instead of direct OSS access.
* fix: correct dl.rustfs.com path structure to include /artifacts/rustfs/
- Update all download URLs to use correct path structure:
* Dev: https://dl.rustfs.com/artifacts/rustfs/dev/
* Release: https://dl.rustfs.com/artifacts/rustfs/release/
- Test confirmed both paths return HTTP 200 with application/zip content-type
- Update Dockerfile, Dockerfile.release, docker-buildx.sh, and build-rustfs.sh
- Update test scripts with correct base path
The dl.rustfs.com domain requires the /artifacts/rustfs/ prefix
to access the binary files correctly.
* feat: refactor Dockerfile to download binaries from GitHub Releases
- Changed binary download source from dl.rustfs.com to GitHub Releases
- Added support for latest release auto-detection via GitHub API
- Enhanced error handling with detailed messages and helpful links
- Added optional checksum verification using SHA256SUMS
- Improved architecture support for amd64 and arm64
- Removed unnecessary minisign installation
- Added jq dependency for JSON parsing
* feat: consolidate Docker build to use single Dockerfile
- Removed Dockerfile.release and use unified Dockerfile instead
- Updated docker-buildx.sh to use single Dockerfile with build args
- Both latest and release variants now use GitHub Releases
- Simplified build process and reduced maintenance overhead
- Updated error messages to point to GitHub releases
* chore: remove unused Dockerfile.obs
- Removed Dockerfile.obs as it's no longer needed
- Simplified Docker build configuration
* feat: unify Docker prebuild variants to use GitHub Releases
- Updated .docker/alpine/Dockerfile.prebuild to download from GitHub Releases
- Updated .docker/ubuntu/Dockerfile.prebuild to download from GitHub Releases
- All prebuild variants now consistently use GitHub Releases as binary source
- Added checksum verification for all prebuild variants
- Updated .docker/README.md to reflect unified GitHub Releases approach
- Improved error handling and user guidance in all prebuild Dockerfiles
* feat: major Docker structure simplification and consolidation
## 🎯 Simplified Docker Structure
Moved from complex multi-directory structure to clean root-level organization:
### Before:
- Dockerfile (production)
- .docker/alpine/Dockerfile.prebuild (duplicate)
- .docker/alpine/Dockerfile.source
- .docker/ubuntu/Dockerfile.prebuild (duplicate)
- .docker/ubuntu/Dockerfile.source
- .docker/ubuntu/Dockerfile.dev
### After:
- Dockerfile (production - Alpine + GitHub Releases)
- Dockerfile.source (source build - Ubuntu + cross-compilation)
- Dockerfile.dev (development - Ubuntu + full toolchain)
## 🔧 Key Changes
- **Eliminated Duplicates**: Removed redundant prebuild variants
- **Moved Core Files**: Dockerfile.{source,dev} now in root directory
- **Unified Configuration**: cargo.config.toml moved to root
- **Updated References**: Fixed all GitHub Actions and docker-compose paths
- **Simplified CI Matrix**: Reduced from 5 to 3 Docker variants
## 📦 Preserved Valuable Diversity
- **Production**: Alpine-based for minimal size
- **Source**: Ubuntu-based with cross-compilation support
- **Development**: Ubuntu-based with full development tools
## 🚀 Benefits
- ✅ Cleaner project structure
- ✅ Easier maintenance and navigation
- ✅ Reduced CI/CD complexity
- ✅ Faster build matrix execution
- ✅ Maintained functionality and flexibility
* chore: remove duplicate cargo.config.toml from .docker directory
The file is now in the root directory and no longer needed in .docker/
* fix: update all references to removed Dockerfile files
- Updated .docker/compose/README.md to reference Dockerfile.source instead of Dockerfile.obs
- Updated docker-compose.yml to use Dockerfile.source instead of Dockerfile.dev
- Updated scripts/build-docker-multiarch.sh to use Dockerfile.source for devenv builds
- Updated .github/workflows/docker.yml to use Dockerfile.source for dev builds
- Updated Makefile to use Dockerfile.source for init-devenv target
- Updated .docker/README.md to remove references to non-existent Dockerfile.dev
- Ensured all Docker configurations consistently use the unified Dockerfile structure
* chore: remove unnecessary console static assets download
- Remove obsolete download steps from build.yml and performance.yml
- Console static assets are already embedded via rust-embed in rustfs/static/
- The download from dl.rustfs.com is no longer needed as project contains complete console assets
- This improves build reliability and reduces external dependencies
- Replaced with verification steps that confirm embedded assets are present
* feat: update Makefile and README.md for new Docker build system
- Updated Makefile to use unified Docker build system:
- Replace references to non-existent Dockerfile.ubuntu22.04 and Dockerfile.rockylinux9.3
- Add new docker-buildx targets using docker-buildx.sh script
- Deprecate old docker-build-multiarch targets with warnings
- Add docker-build-production and docker-build-source targets
- Update help-docker with new command structure
- Updated README.md with docker-buildx.sh usage:
- Add comprehensive Docker build from source section
- Document multi-architecture build capabilities
- Include both script and Make target examples
- Show registry flexibility and build optimization features
- Update step numbers in quickstart guide
- Improve developer experience with clear documentation and updated tooling
- Maintain backward compatibility with deprecation warnings
* feat: integrate console assets download into build-rustfs.sh
- Added console download functionality to build-rustfs.sh:
- New flags: --download-console, --force-console-update, --console-version
- Intelligent detection of existing console assets
- Retry logic with fallback error handling
- Consistent with Docker build asset management
- Updated scripts to use unified build process:
- scripts/static.sh: Now uses build-rustfs.sh for console downloads
- scripts/run.sh: Uses build-rustfs.sh instead of direct curl
- scripts/run.ps1: Updated with guidance for Windows users
- Benefits:
- Unified asset management across all build processes
- Consistent version handling and retry logic
- Eliminates duplicate download logic
- Better error handling and user feedback
- Preparation for CI/CD integration
- Removed unused download-static-curl.sh script
This change centralizes console asset management and prepares for
streamlined CI/CD processes where build-rustfs.sh becomes the
single point of truth for binary and asset builds.
* fix: update PowerShell script to use unified console asset management
- Updated scripts/run.ps1 to use build-rustfs.sh for console asset downloads
- Added guidance for Windows users to use the unified build script
- Maintains consistency across all platform-specific scripts
* feat: add binary verification to build script
- Add verify_binary function to test built binaries
- Test --help and --version commands
- Verify binary structure with readelf/otool
- Add --skip-verification option for cross-compilation
- Include verification status in build output
- Automatic error handling if verification fails
* feat: add platform selection support to build script
- Add --platform parameter to build-rustfs.sh for target platform selection
- Implement cross-compilation support with automatic 'cross' tool detection
- Auto-enable --skip-verification for cross-compilation scenarios
- Update all Makefile build targets to use unified build-rustfs.sh script
- Add helpful error messages and suggestions for cross-compilation failures
- Update help documentation with platform selection examples
- Improve build consistency across different architectures
* feat: modernize CI/CD build process with build-rustfs.sh
- Replace manual cargo build commands with unified build-rustfs.sh script
- Simplify matrix configuration by removing cross-compilation flags
- Ensure consistency between local and CI/CD builds
- Automatic cross-compilation tool detection and installation
- Built-in binary verification for quality assurance
- Unified console asset management
- Better error handling and suggestions
Benefits:
- Consistent build process across all environments
- Automatic detection and handling of cross-compilation scenarios
- Built-in quality checks with binary verification
- Reduced CI/CD configuration complexity
- Better maintainability with single source of truth for build logic
* feat: optimize CI/CD workspace path management
- Add WORKSPACE_DIR environment variable to cache github.workspace
- Set default working-directory at job level for consistency
- Use explicit workspace paths in critical operations
- Improve reliability and maintainability of CI/CD paths
- Ensure consistent behavior across different GitHub Actions environments
Benefits:
- More explicit and reliable path handling
- Better maintainability with centralized workspace reference
- Reduced risk of path-related issues in CI/CD
- Consistent working directory across all job steps
* refactor: simplify CI/CD path management - remove redundant workspace references
- Remove unnecessary WORKSPACE_DIR environment variable
- Remove redundant defaults.run.working-directory setting
- Use relative paths since GITHUB_WORKSPACE is the default working directory
- Follow GitHub Actions best practices by leveraging default behavior
As per GitHub Actions documentation, GITHUB_WORKSPACE is already the default
working directory, so explicit specification is unnecessary in most cases.
* docs: update Docker README to reflect current project state
- Fix directory structure: remove non-existent nginx/ directory
- Correct base OS: Dockerfile.source uses Debian Bookworm, not Ubuntu 22.04
- Add docker-buildx.sh script documentation
- Update Docker tag examples to match actual CI/CD workflows
- Add CI/CD integration section explaining automated builds
- Document build variants and manual build options
- Reflect current project architecture and tooling
These updates ensure the documentation accurately represents the current
Docker build system and CI/CD workflows.
* fix: update Docker command in rustfs README
- Replace quay.io registry with Docker Hub (rustfs/rustfs:latest)
- Remove separate console port 9001, console now runs on main port 9000
- Add both Docker and Podman examples for user choice
- Fix console access URL to use unified port
This aligns with the recent console port consolidation changes
and the project's move to Docker Hub as the primary registry.
* wip
* fix: remove unnecessary entrypoint.sh and fix Docker paths
* Update Dockerfile
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* cleanup: remove unused DOCKERFILE_PATH variable from Makefile
* feat: update Docker build to use dl.rustfs.com for binary downloads
- Replace GitHub releases download with dl.rustfs.com
- Add CHANNEL parameter support (release/dev)
- Update docker-buildx.sh to support channel-specific builds
- Improve error messages with new download URLs
- Support both latest and specific version downloads
- Add channel validation in build script
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit introduces two key enhancements: the integration of Grafana Tempo for distributed tracing and the implementation of a dual-stack TCP listener for improved network compatibility.
- **Observability**:
- Adds the `tempo` service to the `docker-compose.yml` observability stack.
- Tempo is configured to collect and store traces, integrating with the existing OpenTelemetry setup.
- A custom `tempo-entrypoint.sh` script is included to manage volume permissions on startup.
- **Networking**:
- Modifies `http.rs` to support dual-stack (IPv4/IPv6) connections on a single socket.
- By setting the `IPV6_V6ONLY` socket option to `false`, the server can now accept both IPv6 and IPv4-mapped IPv6 traffic, enhancing cross-platform support.
* feat: enhance Docker build system with advanced version selection
## New Features
- Add force_rebuild parameter for Docker workflow manual triggers
- Improve version pattern matching with better regex validation
- Add comprehensive Docker Build Guide documentation
- Enhanced logging and error reporting for build process
- Support for prerelease version detection (alpha, beta, rc)
## Improvements
- Better version pattern validation for releases and dev builds
- More detailed build logs with context and warnings
- Clear documentation for all Docker image variants and use cases
- Updated README with Docker version examples and guide reference
## Documentation
- New comprehensive Docker Build Guide (docs/DOCKER_BUILD_GUIDE.md)
- Updated README with version-specific Docker examples
- Workflow dependency diagram and troubleshooting guide
- Complete reference for all supported version patterns
This enhancement provides a robust, well-documented Docker build system
that supports flexible version selection while maintaining deterministic
build behavior without fallback mechanisms.
* fix: simplify dev version regex pattern in docker workflow
* fix: simplify version number regex pattern in docker workflow
* feat: remove docs directory
* fix: correct SHA length matching in main-latest filename generation
* refactor: use bash string operations instead of sed for main-latest filename generation
* refactor: simplify filename generation by removing redundant intermediate variables
* feat: add dev-latest version generation for all development builds
* feat: add dev-latest support to Docker workflow
* feat: enhance Docker build system with advanced version selection
## New Features
- Add force_rebuild parameter for Docker workflow manual triggers
- Improve version pattern matching with better regex validation
- Add comprehensive Docker Build Guide documentation
- Enhanced logging and error reporting for build process
- Support for prerelease version detection (alpha, beta, rc)
## Improvements
- Better version pattern validation for releases and dev builds
- More detailed build logs with context and warnings
- Clear documentation for all Docker image variants and use cases
- Updated README with Docker version examples and guide reference
## Documentation
- New comprehensive Docker Build Guide (docs/DOCKER_BUILD_GUIDE.md)
- Updated README with version-specific Docker examples
- Workflow dependency diagram and troubleshooting guide
- Complete reference for all supported version patterns
This enhancement provides a robust, well-documented Docker build system
that supports flexible version selection while maintaining deterministic
build behavior without fallback mechanisms.
* fix: simplify dev version regex pattern in docker workflow
* fix: simplify version number regex pattern in docker workflow
* feat: remove docs directory
* fix: correct SHA length matching in main-latest filename generation
* feat: enhance Docker build system with advanced version selection
## New Features
- Add force_rebuild parameter for Docker workflow manual triggers
- Improve version pattern matching with better regex validation
- Add comprehensive Docker Build Guide documentation
- Enhanced logging and error reporting for build process
- Support for prerelease version detection (alpha, beta, rc)
## Improvements
- Better version pattern validation for releases and dev builds
- More detailed build logs with context and warnings
- Clear documentation for all Docker image variants and use cases
- Updated README with Docker version examples and guide reference
## Documentation
- New comprehensive Docker Build Guide (docs/DOCKER_BUILD_GUIDE.md)
- Updated README with version-specific Docker examples
- Workflow dependency diagram and troubleshooting guide
- Complete reference for all supported version patterns
This enhancement provides a robust, well-documented Docker build system
that supports flexible version selection while maintaining deterministic
build behavior without fallback mechanisms.
* fix: simplify dev version regex pattern in docker workflow
* fix: simplify version number regex pattern in docker workflow
* feat: remove docs directory
- Add version input parameter to docker.yml workflow_dispatch
- Support main-latest, latest, dev-xxx, and specific version patterns
- Remove complex fallback mechanism from all Dockerfile variants
- Add clear error handling with helpful user guidance
- Create main-latest versions for development builds
- Ensure Docker builds require explicit VERSION parameter
- Update all Docker variants (production, alpine, ubuntu) consistently
This change solves the build dependency issue where Docker builds
could fail when expected binary artifacts don't exist, by providing
a clean version selection mechanism without unpredictable fallbacks.
- Add automatic creation of latest version files for release and prerelease builds
- Simplify installation script by providing direct latest URLs
- Support rustfs-linux-{arch}-latest.zip naming convention
- Improve build artifact management and user experience
* feat: implement multi-channel release system with artifact naming
- Add dedicated release.yml workflow for handling GitHub releases
- Refactor build.yml to support dev/release/prerelease artifact naming
- Update docker.yml to support version-specific image tagging
- Implement artifact naming rules:
- Dev: rustfs-{platform}-{arch}-dev-{sha}.zip
- Release: rustfs-{platform}-{arch}-v{version}.zip
- Prerelease: rustfs-{platform}-{arch}-v{version}.zip
- Add OSS upload directory separation (dev/ vs release/)
- Only stable releases update latest.json and create latest tags
- Separate GitHub Release creation from build workflow
- Add comprehensive build summaries and status reporting
This enables proper multi-channel distribution with clear artifact
identification and prevents confusion between dev and stable releases.
* fix: support version tags without v prefix (1.0.0 instead of v1.0.0)
- Update trigger patterns from 'v*.*.*' to '*.*.*' in all workflows
- Fix version extraction logic to handle tags without v prefix
- Maintain backward compatibility with existing logic
Note: Artifact naming still includes 'v' prefix for clarity
(e.g., tag '1.0.0' creates 'rustfs-linux-x86_64-v1.0.0.zip')
* feat: update Dockerfile to support multi-channel release system
- Add build arguments for VERSION, BUILD_TYPE, and TARGETARCH
- Support dynamic artifact download based on build type:
- Development: downloads from artifacts/rustfs/dev/
- Release: downloads from artifacts/rustfs/release/
- Auto-generate correct filenames based on new naming convention:
- Dev: rustfs-linux-{arch}-dev-{sha}.zip
- Release: rustfs-linux-{arch}-v{version}.zip
- Add architecture mapping for multi-platform builds
- Pass BUILD_TYPE parameter from docker.yml workflow
- Improve error handling with helpful download path suggestions
This ensures Docker images use the correct pre-built binaries
from the new multi-channel release system.
* feat: optimize and consolidate Dockerfile structure
## Major Improvements:
### ✅ Created Missing Files
- Add .docker/Dockerfile.alpine for lightweight Alpine-based builds
- Support both pre-built binary download and source compilation
### 🔧 Fixed Critical Issues
- Fix Dockerfile.obs: ubuntu:latest → ubuntu:22.04 (stable version)
- Add proper security practices (non-root user, health checks)
- Add proper error handling and environment variables
### 🗑️ Eliminated Redundancy
- Remove .docker/Dockerfile.ubuntu22.04 (duplicate of devenv)
- Update docker.yml workflow to use devenv for ubuntu variant
- Consolidate similar functionality into fewer, better files
### 🚀 Enhanced Functionality
- Make devenv Dockerfile dual-purpose (dev environment + runtime)
- Add VERSION/BUILD_TYPE support for dynamic binary downloads
- Improve security with proper user management
- Add comprehensive health checks and error handling
### 📊 Final Dockerfile Structure:
1. Dockerfile (production, Alpine-based, pre-built binaries)
2. Dockerfile.multi-stage (full source builds, Ubuntu-based)
3. Dockerfile.obs (observability builds, Ubuntu-based)
4. .docker/Dockerfile.alpine (lightweight Alpine variant)
5. .docker/Dockerfile.devenv (development + ubuntu variant)
6. .docker/Dockerfile.rockylinux9.3 (RockyLinux variant)
This reduces redundancy while maintaining all necessary build variants
and improving maintainability across the entire container ecosystem.
* refactor: streamline Dockerfile structure and remove unused files
## 🎯 Major Cleanup:
### 🗑️ Removed Unused Files (2 files)
- Delete Dockerfile.obs (not referenced anywhere)
- Delete .docker/Dockerfile.rockylinux9.3 (not referenced anywhere)
### 📁 Reorganized File Layout
- Move Dockerfile.multi-stage → .docker/Dockerfile.multi-stage
- Update docker-compose.yml to use new path
- Keep main Dockerfile in root (production use)
- Consolidate variants in .docker/ directory
### ✅ Final Clean Structure:
### 📊 Before vs After:
- **Before**: 7 files (1 missing, 2 unused, scattered layout)
- **After**: 4 files (all used, organized layout)
- **Reduction**: 43% fewer files, 100% utilization
This eliminates confusion and reduces maintenance overhead while
keeping all actually needed functionality intact.
* refactor: implement comprehensive Docker tag strategy with production variant
- Restore production variant as default with explicit naming
- Add support for prerelease channels (alpha, beta, rc)
- Implement rolling development tags (dev, dev-variant)
- Support semantic versioning with variant combinations
- Update documentation with complete tag strategy examples
- Align with GPT-suggested comprehensive tagging approach
Tag examples:
- rustfs/rustfs:1.2.3 (main production)
- rustfs/rustfs:1.2.3-production (explicit production)
- rustfs/rustfs:1.2.3-alpine (Alpine variant)
- rustfs/rustfs:alpha (latest alpha)
- rustfs/rustfs:dev (latest development)
- rustfs/rustfs:dev-13e4a0b (specific commit)
* perf: optimize Docker build speed with comprehensive caching and compilation improvements
- Add dual caching strategy: GitHub Actions + Registry cache
- Implement sccache for Rust compilation caching across builds
- Configure parallel compilation with all available CPU cores
- Add optimized cargo configuration for faster builds
- Enable sparse registry protocol for dependency resolution
- Configure LLD linker for faster linking
- Add BuildKit optimizations with inline cache
- Disable provenance/SBOM generation for faster builds
- Document build performance improvements and timings
Performance improvements:
- Source builds: ~40-50% faster with cache hits
- Pre-built binaries: ~30-40% faster
- Parallel matrix builds reduce total CI time significantly
- Registry cache provides persistent cross-run benefits
* refactor: consolidate Docker variants and eliminate duplication
- Replace root Dockerfile with enhanced Alpine prebuild version
- Remove redundant alpine variant from build matrix
- Root Dockerfile now includes:
- Non-root user security
- Health checks
- Better error handling
- protoc/flatc tool support
- Update documentation to reflect simplified 4-variant strategy
- Remove duplicate .docker/alpine/Dockerfile.prebuild
Build matrix now:
- production (root Dockerfile - Alpine prebuild)
- alpine-source (Alpine source build)
- ubuntu (Ubuntu prebuild)
- ubuntu-source (Ubuntu source build)
Benefits:
- Eliminates functional duplication
- Improves security with non-root execution
- Maintains same image variants with better quality
- Simplifies maintenance
* fix: restore alpine variant for better user choice
- Restore alpine variant (rustfs/rustfs:1.2.3-alpine)
- Re-add .docker/alpine/Dockerfile.prebuild
- Update build matrix to include 5 variants again:
- production (default)
- alpine (explicit Alpine choice)
- alpine-source (Alpine source build)
- ubuntu (Ubuntu pre-built)
- ubuntu-source (Ubuntu source build)
- Update documentation to reflect restored alpine tags
- Fix build performance table to include all variants
User feedback: Alpine variant provides explicit choice even if
similar to production variant. Better UX with clear options.
* fix: remove redundant rustup target add commands in Alpine Dockerfiles
- Remove 'rustup target add x86_64-unknown-linux-musl' from Alpine source build
- Remove redundant target add from Alpine prebuild fallback path
- Remove redundant target add from root Dockerfile fallback path
Reason: rust:alpine base image already has x86_64-unknown-linux-musl
as the default target since Alpine uses musl libc by default.
Thanks to @houseme for spotting this redundancy in code review.
* fix: add missing RUSTFS_VOLUMES environment variable in Dockerfiles
- Add RUSTFS_VOLUMES=/data to all Dockerfile variants
- This fixes the issue where CMD ['/app/rustfs'] was used without providing the required volumes parameter
- The volumes parameter is required by the application and can be provided via command line or RUSTFS_VOLUMES environment variable
* fix: update docker-compose configurations to ensure all environments work correctly
- Added missing access key and secret key environment variables to docker-compose.yaml
- This ensures the distributed test environment has proper authentication credentials
- Complementary fix to the previous Dockerfile updates for consistent configuration
* fix: recreate missing Dockerfile.obs with complete content
- The file was accidentally left empty after initial creation
- Now contains proper Ubuntu-based configuration for observability environment
- Includes all necessary environment variables including RUSTFS_VOLUMES
- Supports docker-compose-obs.yaml configuration
* refactor: organize Docker Compose configurations and eliminate duplication
- Move specialized configurations to .docker/compose/ directory
- Rename docker-compose.yaml → docker-compose.cluster.yaml (distributed testing)
- Rename docker-compose-obs.yaml → docker-compose.observability.yaml (observability testing)
- Keep docker-compose.yml as the main production configuration
- Add comprehensive README explaining different configuration purposes
- Eliminates confusion between similar filenames
- Provides clear guidance on when to use each configuration
* fix: correct relative paths in moved Docker Compose configurations
- Fix binary volume mount paths in docker-compose.cluster.yaml (./target → ../../target)
- Fix Dockerfile.obs context path in docker-compose.observability.yaml (. → ../..)
- Fix observability config file paths (./.docker → ../../.docker)
- Update README.md with correct usage instructions for new locations
- All configurations now correctly reference files relative to their new positions
* refactor: move Dockerfile.obs to .docker/compose/ directory for better organization
- Move Dockerfile.obs from root to .docker/compose/ directory
- Update all dockerfile references in docker-compose.observability.yaml
- Keep related files (Dockerfile.obs + docker-compose.observability.yaml) together
- Clean up root directory by removing specialized-purpose Dockerfile
- Update README.md to document new file organization
- Improves project structure and file discoverability
* refactor: improve Docker build configuration for better clarity
- Move Dockerfile.obs back to project root for simpler build context
- Update docker-compose.observability.yaml to use cleaner dockerfile reference
- Change from '.docker/compose/Dockerfile.obs' to simply 'Dockerfile.obs'
- Maintain context as '../..' for access to project files
- Remove redundant Dockerfile.obs documentation from compose README
- This follows Docker best practices: simple context + Dockerfile at context root
* wip
* refactor(config): Unify S3 API and Console ports
This commit streamlines the server configuration by unifying the S3 API and the WebUI (Console) to serve on a single port.
Previously, the console was managed by separate configuration options (`RUSTFS_CONSOLE_ENABLE` and `RUSTFS_CONSOLE_ADDRESS`), requiring a distinct port. This added complexity to deployment and configuration.
With this change:
- The `RUSTFS_CONSOLE_ADDRESS` and `RUSTFS_CONSOLE_FS_ENDPOINT` environment variables are removed.
- The WebUI is now always available and served directly from the main application port defined by `RUSTFS_ADDRESS`.
- This simplifies setup, reduces the number of exposed ports, and makes the application easier to manage and deploy, especially in containerized environments.
Users should update their startup scripts and remove the deprecated `RUSTFS_CONSOLE_*` variables.
* improve docker comprose config file and remove docs dir
This commit streamlines the server configuration by unifying the S3 API and the WebUI (Console) to serve on a single port.
Previously, the console was managed by separate configuration options (`RUSTFS_CONSOLE_ENABLE` and `RUSTFS_CONSOLE_ADDRESS`), requiring a distinct port. This added complexity to deployment and configuration.
With this change:
- The `RUSTFS_CONSOLE_ADDRESS` and `RUSTFS_CONSOLE_FS_ENDPOINT` environment variables are removed.
- The WebUI is now always available and served directly from the main application port defined by `RUSTFS_ADDRESS`.
- This simplifies setup, reduces the number of exposed ports, and makes the application easier to manage and deploy, especially in containerized environments.
Users should update their startup scripts and remove the deprecated `RUSTFS_CONSOLE_*` variables.
* feat: add browser redirect layer to route GET requests to console
* refactor: move RedirectLayer to separate layer.rs file
* feat: restrict redirect layer to only handle root path and index.html
* feat: restrict redirect layer to only handle root path /rustfs and index.html
* fix unzip error
* fix url change error
fix url change error
* Simplify user experience and integrate console and endpoint
Simplify user experience and integrate console and endpoint
* merge console router
* make code happy
* Scanner (#156)
* feat: integrate CancellationToken for unified background services management
- Consolidate data scanner and auto heal cancellation tokens into single unified token
- Move GLOBAL_BACKGROUND_SERVICES_CANCEL_TOKEN to global.rs for centralized management
- Add graceful shutdown support to MRF heal routine with MinIO-compatible logic
- Implement heal_routine_with_cancel method preserving original healing logic
- Update main.rs to use unified background services shutdown mechanism
- Enhance error handling with proper ecstore Result types
- Fix clippy warnings for needless return statements
- Maintain backward compatibility while adding modern cancellation support
This change provides a cleaner architecture for background service lifecycle management
and ensures all healing services can be gracefully shut down through a single token.
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix: Refact heal and scanner design
Signed-off-by: junxiang Mu <1948535941@qq.com>
* refact: step 2
Signed-off-by: junxiang Mu <1948535941@qq.com>
* feat: refactor scanner module and add data usage statistics
- Move scanner code to scanner/ subdirectory for better organization
- Add data usage statistics collection and persistence
- Implement histogram support for size and version distribution
- Add global cancel token management for scanner operations
- Integrate scanner with ECStore for comprehensive data analysis
- Update error handling and improve test isolation
- Add data usage API endpoints and backend integration
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Chore: fix ref and fix comment
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix: fix clippy
Signed-off-by: junxiang Mu <1948535941@qq.com>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: dandan <dandan@dandandeMac-Studio.local>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: guojidan <63799833+guojidan@users.noreply.github.com>
Co-authored-by: dandan <dandan@dandandeMac-Studio.local>
* feat: integrate CancellationToken for unified background services management
- Consolidate data scanner and auto heal cancellation tokens into single unified token
- Move GLOBAL_BACKGROUND_SERVICES_CANCEL_TOKEN to global.rs for centralized management
- Add graceful shutdown support to MRF heal routine with MinIO-compatible logic
- Implement heal_routine_with_cancel method preserving original healing logic
- Update main.rs to use unified background services shutdown mechanism
- Enhance error handling with proper ecstore Result types
- Fix clippy warnings for needless return statements
- Maintain backward compatibility while adding modern cancellation support
This change provides a cleaner architecture for background service lifecycle management
and ensures all healing services can be gracefully shut down through a single token.
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix: Refact heal and scanner design
Signed-off-by: junxiang Mu <1948535941@qq.com>
* refact: step 2
Signed-off-by: junxiang Mu <1948535941@qq.com>
* feat: refactor scanner module and add data usage statistics
- Move scanner code to scanner/ subdirectory for better organization
- Add data usage statistics collection and persistence
- Implement histogram support for size and version distribution
- Add global cancel token management for scanner operations
- Integrate scanner with ECStore for comprehensive data analysis
- Update error handling and improve test isolation
- Add data usage API endpoints and backend integration
Signed-off-by: junxiang Mu <1948535941@qq.com>
* Chore: fix ref and fix comment
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix: fix clippy
Signed-off-by: junxiang Mu <1948535941@qq.com>
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: dandan <dandan@dandandeMac-Studio.local>
Added language selection links to the README for easier access to translated versions: German, Spanish, French, Japanese, Korean, Portuguese, and Russian.
- Replace paths-ignore with paths for more precise control
- Only trigger on Rust source files, Cargo files, and workflow itself
- Improve efficiency by avoiding unnecessary performance tests
- Follow best practices for targeted workflow execution
* feat: ensure workflows never skip execution during version releases
- Modified skip-duplicate-actions to never skip when pushing tags
- Updated all workflow jobs to force execution for tag pushes (version releases)
- Ensures complete CI/CD pipeline execution for releases including:
- All tests and lint checks
- Multi-platform builds
- GUI builds
- Release asset creation
- OSS uploads
This guarantees that version releases always undergo full validation
and build processes, maintaining release quality and consistency.
* refactor: simplify workflow skip logic using do_not_skip parameter
- Replace complex conditional expressions with do_not_skip: ['release', 'push']
- Add skip-duplicate-actions to docker.yml workflow
- Ensure all workflows use consistent skip mechanism
- Maintain release and tag push execution guarantee
- Simplify job conditions by removing redundant tag checks
This change makes workflows more maintainable and follows
official skip-duplicate-actions best practices.
* feat: enhance build and release workflow with multi-platform support
- Add Windows support (x86_64 and ARM64) to build matrix
- Add macOS Intel x86_64 support alongside Apple Silicon
- Improve cross-platform builds with proper toolchain selection
- Use GitHub CLI (gh) for release management instead of GitHub Actions
- Add automatic checksum generation (SHA256/SHA512) for all binaries
- Support different archive formats per platform (zip for Windows, tar.gz for Unix)
- Add comprehensive release notes with installation guides
- Enhanced error handling for console assets download
- Platform-specific build information in packages
- Support both binary and GUI application releases
- Update OSS upload to handle multiple file formats
This brings RustFS builds up to enterprise-grade standards with:
- 6 binary targets (Linux x86_64/ARM64, macOS x86_64/ARM64, Windows x86_64/ARM64)
- Professional release management with checksums
- User-friendly installation instructions
- Multi-platform GUI applications
* feat: add core development principles to cursor rules
- Add precision-first development principle: 每次改动都要精准,没把握就别改
- Add GitHub CLI priority rule: GitHub PR 创建优先使用 gh 命令
- Emphasize careful analysis before making changes
- Promote use of gh commands for better automation and integration
* refactor: translate cursor rules to English
- Translate core development principles from Chinese to English
- Maintain consistency with project's English-first policy
- Update 'Every change must be precise' principle
- Update 'GitHub PR creation prioritizes gh command usage' rule
- Ensure all cursor rules are in English for better accessibility
* fix: prevent workflow changes from triggering CI/CD pipelines
- Add .github/** to paths-ignore in build.yml workflow
- Add .github/** to paths-ignore in docker.yml workflow
- Update skip-duplicate paths_ignore to include .github files
- Workflow changes should not trigger performance, build, or docker workflows
- Saves unnecessary CI/CD resource usage when updating workflow configurations
- Consistent with performance.yml which already ignores .github/**
* feat: enhance build and release workflow with multi-platform support
- Add Windows support (x86_64 and ARM64) to build matrix
- Add macOS Intel x86_64 support alongside Apple Silicon
- Improve cross-platform builds with proper toolchain selection
- Use GitHub CLI (gh) for release management instead of GitHub Actions
- Add automatic checksum generation (SHA256/SHA512) for all binaries
- Support different archive formats per platform (zip for Windows, tar.gz for Unix)
- Add comprehensive release notes with installation guides
- Enhanced error handling for console assets download
- Platform-specific build information in packages
- Support both binary and GUI application releases
- Update OSS upload to handle multiple file formats
This brings RustFS builds up to enterprise-grade standards with:
- 6 binary targets (Linux x86_64/ARM64, macOS x86_64/ARM64, Windows x86_64/ARM64)
- Professional release management with checksums
- User-friendly installation instructions
- Multi-platform GUI applications
* feat: add core development principles to cursor rules
- Add precision-first development principle: 每次改动都要精准,没把握就别改
- Add GitHub CLI priority rule: GitHub PR 创建优先使用 gh 命令
- Emphasize careful analysis before making changes
- Promote use of gh commands for better automation and integration
* refactor: translate cursor rules to English
- Translate core development principles from Chinese to English
- Maintain consistency with project's English-first policy
- Update 'Every change must be precise' principle
- Update 'GitHub PR creation prioritizes gh command usage' rule
- Ensure all cursor rules are in English for better accessibility
- Fix performance.yml: now ignores *.md, README*, and docs/**
- Fix build.yml: now ignores documentation files and images
- Fix docker.yml: prevent Docker builds on README changes
- Replace 'paths:' with 'paths-ignore:' to properly exclude docs
- Reduces unnecessary CI runs for documentation-only PRs
This resolves the issue where README changes triggered expensive
CI pipelines including Performance Testing and Docker builds.
* docs: simplify all crates README files
- Remove extensive code examples and detailed documentation
- Convert to minimal module introductions with core feature lists
- Direct users to main RustFS repository for comprehensive docs
- Updated 20 crate README files for consistency and brevity
Files updated:
- crates/rio/README.md (415→15 lines)
- crates/s3select-api/README.md (592→15 lines)
- crates/s3select-query/README.md (658→15 lines)
- crates/signer/README.md (407→15 lines)
- crates/utils/README.md (395→15 lines)
- crates/workers/README.md (463→15 lines)
- crates/zip/README.md (408→15 lines)
* docs: restore original headers in crates README files
- Add back RustFS logo image and CI badges
- Restore formatted headers and structured layout
- Keep simplified content with module introductions
- Maintain consistent documentation structure across all crates
All 20 crate README files now have proper headers while keeping
the simplified content that directs users to the main repository.
* rules: enforce PR-only workflow for main branch
- Strengthen rule that ALL changes must go through pull requests
- Explicitly forbid direct commits to main branch under any circumstances
- Add comprehensive PR requirements and enforcement guidelines
- Clarify that PRs are the ONLY way to merge to main branch
- Add requirement for PR approval before merging
- Include enforcement mechanisms for branch protection
- Remove invalid github-token parameter from arduino/setup-protoc action
- Fix cross-compilation RUSTFLAGS issue by conditionally setting target-cpu=native
- Update workflow tag triggers from v* to * for non-v prefixed tags
- Optimize Zig and cargo-zigbuild installation using official actions
This resolves build failures in aarch64-unknown-linux-musl target where
zig was receiving invalid x86_64 CPU flags during cross-compilation.
- Fix random number generation API usage
- Adjust sleep calculation to follow MinIO pattern
- Ensure proper random range for scanner cycles
Signed-off-by: junxiang Mu <1948535941@qq.com>
* fix: restore Zig and cargo-zigbuild caching in GitHub Actions setup action
Use mlugg/setup-zig and taiki-e/cache-cargo-install-action to speed up cross-compilation tool installation and avoid repeated downloads. All comments and code are in English.
* fix: use correct taiki-e/install-action for cargo-zigbuild
Use taiki-e/install-action@cargo-zigbuild instead of taiki-e/cache-cargo-install-action@v2 to match the original implementation from PR #77.
* refactor: remove explicit Zig version to use latest stable
Move the construction of the hybrid service stack, including all middleware and the RPC service, from the main `run` function into the `process_connection` function.
This change ensures that each incoming connection gets its own isolated service instance. This improves modularity by making the connection handling logic more self-contained and simplifies the main server loop.
Key changes:
- The `hybrid_service` and `rpc_service` are now created inside `process_connection`.
- The `run` function's responsibility is reduced to accepting TCP connections and spawning tasks for `process_connection`.
- Only execute builds on tag push, scheduled runs, or commit message contains --build
- Add latest.json version tracking to rustfs-version OSS bucket
- Create GitHub Release with all build artifacts automatically
- Update comments to English for consistency
- Reduce unnecessary CI resource usage while maintaining automation
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.
Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
* chore: Add copyright and license headers
This commit adds the Apache 2.0 license and a copyright notice to the header of all source files. This ensures that the licensing and copyright information is clearly stated within the codebase.
* cargo fmt
* fix
* fmt
* fix clippy
- Add #[allow(clippy::too_many_arguments)] for functions with 8 parameters
- Replace vec initialization with array where appropriate
- Fix test code warnings by adding #[allow] attributes
- Improve code quality while maintaining functionality
- Change console_enable default value from false to true
- This makes the console server available by default for better user experience
- Users can still disable it by setting RUSTFS_CONSOLE_ENABLE=false
- Refactor `unmarshal_msg` to use `rmp_serde::from_slice` instead of manual parsing
- Add serde field renaming attributes to `FileMetaVersion` struct
- Remove 428 lines of manual MessagePack parsing code
- Improve string comparison using `!object.is_empty()` instead of `object != ""`
- Update volume directory numbering in run script from test{0..4} to test{1..4}
- Clean up unused imports and code
Fixes#517
- Refactor find_version_index to accept Uuid directly instead of string
- Split from_meta_cache_entries_sorted into separate methods for versions and infos
- Add support for after_version_id filtering in version listing
- Fix S3 error code for replication configuration not found
- Improve error handling and logging levels
- Clean up import statements and remove debug code
- Remove specific bucket metadata error variants (BucketPolicyNotFound, etc.)
- Unify all configuration-not-found errors to use StorageError::ConfigNotFound
- Simplify error handling in bucket metadata system by removing redundant error conversions
- Clean up error matching patterns in metadata getter methods
- Remove debug print statement from get_bucket_targets_config
- Update error handling in S3 API layer to use unified ConfigNotFound error
This change improves code maintainability by reducing error type complexity
while preserving the same error semantics for API consumers.
- Remove reed-solomon-erasure dependency and all related code
- Simplify ReedSolomonEncoder from enum to struct with SIMD-only implementation
- Eliminate all conditional compilation (#[cfg(feature = ...)])
- Add instance caching with RwLock-based encoder/decoder reuse
- Implement reset mechanism to avoid unnecessary allocations
- Ensure thread safety with proper cache management
- Update documentation and benchmark scripts for SIMD-only approach
- Apply code formatting across all files
Breaking Changes:
- Removes support for reed-solomon-erasure feature flag
- API remains compatible but implementation is now SIMD-only
Performance Impact:
- Improved encoding/decoding performance through SIMD optimization
- Reduced memory allocations via instance caching
- Enhanced thread safety and concurrency support
* chore(ci): upgrade protoc from 30.2 to 31.1
- Update protoc version in GitHub Actions setup workflow
- Use arduino/setup-protoc@v3 to install the latest protoc version
- Ensure compatibility with current project requirements
- Improve proto file compilation performance and stability
This upgrade aligns our development environment with the latest protobuf standards.
* modify package version
* refactor(deps): centralize crate versions in root Cargo.toml
- Move all dependency versions to workspace.dependencies section
- Standardize AWS SDK and related crates versions
- Update tokio, bytes, and futures crates to latest stable versions
- Ensure consistent version use across all workspace members
- Implement workspace inheritance for common dependencies
This change simplifies dependency management and ensures version consistency across the project.
* fix
* modify
* chore(ci): upgrade protoc from 30.2 to 31.1
- Update protoc version in GitHub Actions setup workflow
- Use arduino/setup-protoc@v3 to install the latest protoc version
- Ensure compatibility with current project requirements
- Improve proto file compilation performance and stability
This upgrade aligns our development environment with the latest protobuf standards.
* modify package version
* refactor(deps): centralize crate versions in root Cargo.toml
- Move all dependency versions to workspace.dependencies section
- Standardize AWS SDK and related crates versions
- Update tokio, bytes, and futures crates to latest stable versions
- Ensure consistent version use across all workspace members
- Implement workspace inheritance for common dependencies
This change simplifies dependency management and ensures version consistency across the project.
* fix
* modify
- Update opentelemetry from 0.29.1 to 0.30.0
- Update related opentelemetry dependencies for compatibility
- Ensure compatibility with existing observability implementation
- Improve tracing and metrics collection capabilities
This upgrade provides better performance and stability for our observability stack.
* chore(ci): upgrade protoc from 30.2 to 31.1
- Update protoc version in GitHub Actions setup workflow
- Use arduino/setup-protoc@v3 to install the latest protoc version
- Ensure compatibility with current project requirements
- Improve proto file compilation performance and stability
This upgrade aligns our development environment with the latest protobuf standards.
* modify package version
* refactor(deps): centralize crate versions in root Cargo.toml
- Move all dependency versions to workspace.dependencies section
- Standardize AWS SDK and related crates versions
- Update tokio, bytes, and futures crates to latest stable versions
- Ensure consistent version use across all workspace members
- Implement workspace inheritance for common dependencies
This change simplifies dependency management and ensures version consistency across the project.
* fix
* modify
- Update opentelemetry from 0.29.1 to 0.30.0
- Update related opentelemetry dependencies for compatibility
- Ensure compatibility with existing observability implementation
- Improve tracing and metrics collection capabilities
This upgrade provides better performance and stability for our observability stack.
- Replace fixed `flexi_logger::Duplicate::Info` with dynamic level from config
- Convert logger_level string to corresponding LevelFilter enum
- Ensure terminal output respects the same log level as file logs
- Fix documentation to reflect the dynamic level behavior
- Create English and Chinese README files for the openobserve-otel directory
- Document configuration details for both OpenObserve and OTel Collector
- Include setup instructions and application integration examples
- Add badges for both OpenObserve and OpenTelemetry projects
- Add 2-minute timeout to samply record command with proper error handling
- Improve test volume directory creation
- Add workflow_dispatch for manual triggering
- Add job timeout of 10 minutes
- Set environment variables to match run.sh configuration
- Add run number to artifact name for better identification
- Add proper error checking and output when profiling fails
- Set artifact retention period to 7 days
This change simplifies the crate name to better reflect its core functionality
as the event handling system for RustFS. The renamed package maintains all
existing functionality while improving naming consistency across the project.
- Updated all imports and references to use the new crate name
- Maintained API compatibility with existing implementations
- Updated tests to reflect the name change
* init rustfs config
* improve code for rustfs-config crate
* add
* improve code for comment
* fix: modify rustfs-config crate name
* add default fn
* improve error logger
* fix: modify docker config yaml
* improve code for config
* feat: restrict kafka feature to Linux only
- Add target-specific feature configuration in Cargo.toml for obs and event-notifier crates
- Implement conditional compilation for kafka feature only on Linux systems
- Add appropriate error handling for non-Linux platforms
- Ensure backward compatibility with existing code
* refactor(ci): optimize build workflow for better efficiency
- Integrate GUI build steps into main build-rustfs job
- Add conditional GUI build execution based on tag releases
- Simplify workflow by removing redundant build-rustfs-gui job
- Copy binary directly to embedded-rustfs directory without downloading artifacts
- Update merge job dependency to only rely on build-rustfs
- Improve cross-platform compatibility for Windows binary naming (.exe)
- Streamline artifact uploading and OSS publishing process
- Maintain consistent conditional logic for release operations
* refactor(ci): optimize build workflow for better efficiency
- Integrate GUI build steps into main build-rustfs job
- Add conditional GUI build execution based on tag releases
- Simplify workflow by removing redundant build-rustfs-gui job
- Copy binary directly to embedded-rustfs directory without downloading artifacts
- Update merge job dependency to only rely on build-rustfs
- Improve cross-platform compatibility for Windows binary naming (.exe)
- Streamline artifact uploading and OSS publishing process
- Maintain consistent conditional logic for release operations
* fix(ci): add repo-token to setup-protoc action for authentication
- Add GITHUB_TOKEN parameter to arduino/setup-protoc@v3 action
- Ensure proper authentication for Protoc installation in CI workflow
- Maintain consistent setup across different CI environments
* modify config
* improve readme.md
* remove env config relation
* add allow(dead_code)
- Add GITHUB_TOKEN parameter to arduino/setup-protoc@v3 action
- Ensure proper authentication for Protoc installation in CI workflow
- Maintain consistent setup across different CI environments
- Integrate GUI build steps into main build-rustfs job
- Add conditional GUI build execution based on tag releases
- Simplify workflow by removing redundant build-rustfs-gui job
- Copy binary directly to embedded-rustfs directory without downloading artifacts
- Update merge job dependency to only rely on build-rustfs
- Improve cross-platform compatibility for Windows binary naming (.exe)
- Streamline artifact uploading and OSS publishing process
- Maintain consistent conditional logic for release operations
- Integrate GUI build steps into main build-rustfs job
- Add conditional GUI build execution based on tag releases
- Simplify workflow by removing redundant build-rustfs-gui job
- Copy binary directly to embedded-rustfs directory without downloading artifacts
- Update merge job dependency to only rely on build-rustfs
- Improve cross-platform compatibility for Windows binary naming (.exe)
- Streamline artifact uploading and OSS publishing process
- Maintain consistent conditional logic for release operations
Added support for both ARM64 Linux variants to the CI/CD build pipeline:
1. Enabled the previously commented `aarch64-unknown-linux-gnu` target build
2. Re-enabled the `aarch64-unknown-linux-musl` target build
3. Updated the build matrix to ensure proper runner selection:
- Ubuntu runners build all Linux targets
- macOS runners build only Apple Silicon targets
4. Maintained compatibility with the existing build scripts and packaging process
This expansion gives users more options for deploying on ARM64 Linux platforms, supporting both glibc and musl libc environments for maximum compatibility and performance.
Added ARM64 macOS (Apple Silicon) build target support to the CI/CD pipeline by:
1. Including `aarch64-apple-darwin` as a new build variant in the build matrix
2. Adding proper exclusion rules to ensure the target only runs on macOS runners
3. Ensuring compatibility with the existing build scripts and packaging process
This change enables native builds for Apple Silicon Macs, improving performance for users with M1/M2/M3/M4 processors while maintaining the same artifact organization and deployment process.
* add tracing instrument
* fix rebalance/decom
* modify Telemetry filter order
* feat: improve address binding and port handling mechanism (#366)
* feat: improve address binding and port handling mechanism
1. Add support for ":port" format to enable dual-stack binding (IPv4/IPv6)
2. Implement automatic port allocation when port 0 is specified
3. Optimize server startup process with unified address resolution
4. Enhance error handling and logging for address resolution
5. Improve graceful shutdown with signal listening
6. Clean up commented code in console.rs
Files:
- ecstore/src/utils/net.rs
- rustfs/src/console.rs
- rustfs/src/main.rs
Branch: feature/server-and-console-port
* improve code for console
* improve code
* improve code for console and net.rs
* Update rustfs/src/main.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update rustfs/src/utils/mod.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* upgrade config file
* modify
* fix readme
Signed-off-by: junxiang Mu <1948535941@qq.com>
* improve readme.md
* improve code for readme.md
add chinese readme.md
* Implement Storage Service Event Notification System
Added event notification capability to the storage module, enabling the storage service to publish object operation events. Key changes include:
1. Created `event_notifier` module providing core functionality:
- `create_metadata()` - Creates event metadata objects with default configuration ID
- `send_event()` - Asynchronously sends event notifications with error handling
2. Integrated the `rustfs_event_notifier` library:
- Supports object creation, deletion, and access events
- Provides event metadata building and management
- Includes proper error propagation
These changes enable the system to trigger notifications when storage operations occur, facilitating auditing, monitoring, and integration with other systems.
* fix
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: weisd <im@weisd.in>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
* feat: improve address binding and port handling mechanism
1. Add support for ":port" format to enable dual-stack binding (IPv4/IPv6)
2. Implement automatic port allocation when port 0 is specified
3. Optimize server startup process with unified address resolution
4. Enhance error handling and logging for address resolution
5. Improve graceful shutdown with signal listening
6. Clean up commented code in console.rs
Files:
- ecstore/src/utils/net.rs
- rustfs/src/console.rs
- rustfs/src/main.rs
Branch: feature/server-and-console-port
* improve code for console
* improve code
* improve code for console and net.rs
* Update rustfs/src/main.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update rustfs/src/utils/mod.rs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* improve code for opentelemetry and add system metrics
* feat: add TraceLayer for HTTP service and improve metrics
- Add TraceLayer to HTTP server for request tracing
- Implement system metrics for process monitoring
- Optimize init_telemetry method for better resource management
- Add graceful shutdown handling for telemetry components
- Fix GracefulShutdown ownership issues with Arc wrapper
* improve code for init_process_observer
* remove tomlfmt.toml
* Translation comment
* improve code for console CompressionLayer params
- Add TraceLayer to HTTP server for request tracing
- Implement system metrics for process monitoring
- Optimize init_telemetry method for better resource management
- Add graceful shutdown handling for telemetry components
- Fix GracefulShutdown ownership issues with Arc wrapper
* init event notifer
* feat: implement event notification system
- Add core event notification interfaces
- Support multiple notification backends:
- Webhook (default)
- Kafka
- MQTT
- HTTP Producer
- Implement configurable event filtering
- Add async event dispatching with backpressure handling
- Provide serialization/deserialization for event payloads
This module enables system events to be published to various endpoints
with consistent delivery guarantees and failure handling.
* feat(event-notifier): improve notification system initialization safety
- Add READY atomic flag to track full initialization status
- Implement initialize_safe and start_safe methods with mutex protection
- Add wait_until_ready function with configurable timeout
- Create initialize_and_start_with_ready_check helper method
- Replace sleep-based waiting with proper readiness checks
- Add safety checks before sending events
- Replace chrono with std::time for time handling
- Update error handling to provide clear initialization status
This change reduces race conditions in multi-threaded environments
and ensures events are only processed when the system is fully ready.
* fix sql
Signed-off-by: junxiang Mu <1948535941@qq.com>
* move protobuf generate into bin crate
Signed-off-by: junxiang Mu <1948535941@qq.com>
* improve Cargo toml
* improve code
* improve code for global
* improve code
* feat(event-notifier): improve environment variable handling
- Fix deserialization error when parsing config from environment variables
- Add proper array format support for adapters configuration
- Update environment variable examples with correct format
- Improve documentation for configuration loading
- Implement helper functions for environment variable validation
This change fixes the "invalid type: map, expected a sequence" error
by ensuring proper formatting of array-type fields in environment variables.
* feat: integrate event-notifier system with rustfs
- Rename package from rustfs-event-notifier to event-notifier for consistency
- Add shutdown hooks for event notification system in main process
- Handle graceful termination of notification services on server shutdown
- Implement initialization and configuration loading for event notification
- Fix environment variable configuration to properly parse adapter arrays
- Update example code to demonstrate proper configuration usage
This change ensures proper integration between rustfs and the event
notification system, with clean startup and shutdown sequences to prevent
resource leaks during application lifecycle.
* feat: improve webhook server and run script integration
- Enhance webhook example with proper shutdown handling using tokio::select!
- Update run.sh to automatically start webhook server alongside main service
- Add event notification configuration to run.sh using environment variables
- Set proper port bindings to ensure webhook server starts on port 3000
- Improve console output for better debugging experience
- Fix race condition during service startup and shutdown
This change ensures proper integration between the webhook server and
the main rustfs service, providing a seamless development experience
with automatic service discovery and clean termination.
* improve for logger
* improve code for global.rs
* fix: modify webhook port
* fix
---------
Signed-off-by: junxiang Mu <1948535941@qq.com>
Co-authored-by: junxiang Mu <1948535941@qq.com>
- Enhance webhook example with proper shutdown handling using tokio::select!
- Update run.sh to automatically start webhook server alongside main service
- Add event notification configuration to run.sh using environment variables
- Set proper port bindings to ensure webhook server starts on port 3000
- Improve console output for better debugging experience
- Fix race condition during service startup and shutdown
This change ensures proper integration between the webhook server and
the main rustfs service, providing a seamless development experience
with automatic service discovery and clean termination.
- Rename package from rustfs-event-notifier to event-notifier for consistency
- Add shutdown hooks for event notification system in main process
- Handle graceful termination of notification services on server shutdown
- Implement initialization and configuration loading for event notification
- Fix environment variable configuration to properly parse adapter arrays
- Update example code to demonstrate proper configuration usage
This change ensures proper integration between rustfs and the event
notification system, with clean startup and shutdown sequences to prevent
resource leaks during application lifecycle.
- Fix deserialization error when parsing config from environment variables
- Add proper array format support for adapters configuration
- Update environment variable examples with correct format
- Improve documentation for configuration loading
- Implement helper functions for environment variable validation
This change fixes the "invalid type: map, expected a sequence" error
by ensuring proper formatting of array-type fields in environment variables.
- Add READY atomic flag to track full initialization status
- Implement initialize_safe and start_safe methods with mutex protection
- Add wait_until_ready function with configurable timeout
- Create initialize_and_start_with_ready_check helper method
- Replace sleep-based waiting with proper readiness checks
- Add safety checks before sending events
- Replace chrono with std::time for time handling
- Update error handling to provide clear initialization status
This change reduces race conditions in multi-threaded environments
and ensures events are only processed when the system is fully ready.
- Add core event notification interfaces
- Support multiple notification backends:
- Webhook (default)
- Kafka
- MQTT
- HTTP Producer
- Implement configurable event filtering
- Add async event dispatching with backpressure handling
- Provide serialization/deserialization for event payloads
This module enables system events to be published to various endpoints
with consistent delivery guarantees and failure handling.
- Change build target from `x86_64-unknown-linux-gnu` to `x86_64-unknown-linux-musl`
- Default install `x86_64-unknown-linux-musl` toolchain in setup action
- Add `musl-tools` and `build-essential` to system dependencies
* feature/observability: (27 commits)
modify default value
TryInto cover
upgrade reqwest version from 0.12.12 to 0.12.15
improve code for FileSink
improve code for config and FileSink
webhook add auth_token
upgrade docker images
feat(obs): enhance OpenTelemetry configuration and logging
merge main
feat: add metrics_handler
add prometheus
upgrade opentelemetry create from 0.28.0 to 0.29.0
update .gitignore
replace log to tracing
improve code for observability
improve code for main
feat(observability): add obs_config option and document stdout export
improve logger entry for Observability
improve log struct
improve code
...
Improve observability setup with the following changes:
- Replace static OnceCell with tokio::sync::OnceCell for guard management
- Add logger_level to OtelConfig for configurable tracing verbosity
- Improve telemetry initialization with better error handling
- Enhance logging filters and span configuration
Breaking Changes:
- Configuration now requires logger_level field
- Global guard management uses async-safe primitives
Example config:
observability:
endpoint: "http://localhost:4317"
logger_level: "debug" # New required field
- Add obs_config parameter to config struct with default path
- Document how to modify use_stdout value in README.md
- Support configuring observability via file or environment variables
This change helps users configure telemetry output destination for better
observability options in different deployment scenarios.
1. Added a new build process `build-rustfs-gui` in `build.yaml` to streamline the build operations for the RustFS GUI.
2. Optimized `cli/rustfs-gui/utils/helper.rs` by using `rust-embed` to embed the `rustfs` resources directly into the binary.
Ensure the static directory is copied as a sibling to the rustfs binary
in the artifact package, maintaining the proper directory structure
for the application to locate its resources.
- Fix artifact pattern matching in merge step to use consistent name pattern
- Standardize artifact naming convention with clear format
- Improve static assets handling with simpler download and extraction
- Add 7-day retention policy for artifacts to manage storage
- Use output variables to ensure consistency between artifact names and paths
- Optimize package creation process to eliminate redundant nesting
- Enhance error handling and script readability
- Implemented `KafkaAuditTarget` to send audit entries to a Kafka topic.
- Implemented `WebhookAuditTarget` to send audit entries to a specified webhook URL.
- Updated `AuditLogger` to support multiple audit targets including Kafka and Webhook.
- Added examples and documentation for the new audit targets.
- Add initialization logic for the `rustfs-logging` crate.
- Provide examples for logging utilities.
- Include tests for logging and telemetry functionalities.
- Ensure proper configuration of dependencies in `Cargo.toml`.
* Improve init_logger function and format code with cargo fmt
- Enhance `init_logger` function for better logging configuration.
- Apply `cargo fmt` to format the codebase.
* Refactor main.rs for clarity and modularity in rustfs-gui
- Move `init_logger` function to `utils/logger.rs` for better separation of concerns.
- Create a dedicated `router` module in `router/router.rs`.
- Update `main.rs` to use the new `logger` and `router` modules.
- Apply `cargo fmt` to format the codebase.
- Create `SettingViews` component in `views/setting.rs`.
- Update `views/mod.rs` to include the new `SettingViews` component.
- Update routing in `main.rs` to include the new route for `SettingViews`.
- Create `Home` component in `components/Home.rs`.
- Create `Navbar` component in `components/navbar.rs`.
- Create `HomeView` component in `views/home.rs`.
- Update `views/mod.rs` to include new views.
- Update routing in `main.rs` to include new routes for `HomeView`.
* Add new CSS, JS, and icon assets to rustfs-gui
- Add `navbar.css` for styling the navigation bar.
- Add `sts.js` for handling tab switching and password toggling.
- Include new icon assets in `assets/icons` directory.
- Update `Dioxus.toml` to reference new assets.
* Remove unused icons from dioxus.toml
- Remove `assets/icons/icon_*.png` from the icon list.
* Initialize rustfs-gui crate
- Set up the `rustfs-gui` package in the workspace.
- Add `dioxus` dependency with `router` feature.
- Configure workspace settings for edition, license, repository, and rust-version.
- Include lints configuration for the workspace.
* Add utility functions and configurations for rustfs-gui
- Implement `RustFSConfig` struct with default values and methods for loading, saving, and clearing configurations.
- Add `ServiceCommand` enum for managing service commands.
- Implement `ServiceManager` struct for starting, stopping, and restarting the service.
- Include helper functions for checking service status, preparing the service, downloading files, and unzipping files.
- Add logging and error handling for service operations.
- Set up the `rustfs-gui` package in the workspace.
- Add `dioxus` dependency with `router` feature.
- Configure workspace settings for edition, license, repository, and rust-version.
- Include lints configuration for the workspace.
<!-- Briefly describe the main changes and motivation for this PR -->
## Checklist
- [ ] I have read and followed the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines
- [ ] Passed `make pre-commit`
- [ ] Added/updated necessary tests
- [ ] Documentation updated (if needed)
- [ ] CI/CD passed (if applicable)
## Impact
- [ ] Breaking change (compatibility)
- [ ] Requires doc/config/deployment update
- [ ] Other impact:
## Additional Notes
<!-- Any extra information for reviewers -->
---
Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)) and sign the CLA if this is your first contribution.
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name:"issue-translator"
on:
issue_comment:
types:[created ]
issues:
types:[opened ]
permissions:
contents:read
issues:write
jobs:
build:
runs-on:ubuntu-latest
steps:
- uses:usthe/issues-translate-action@v2.7
with:
IS_MODIFY_TITLE:false
# not require, default false, . Decide whether to modify the issue title
# if true, the robot account @Issues-translate-bot must have modification permissions, invite @Issues-translate-bot to your project or use your custom bot.
CUSTOM_BOT_NOTE:Bot detected the issue body's language is not English, translate it automatically.
# not require. Customize the translation robot prefix message.
- **⚡ Efficient Development**: Avoid over-design, only do necessary work
- **🛡️ Safe and Reliable**: Always follow development processes, ensure code quality and system stability
- **🔒 Cautious Modification**: Only modify when clearly knowing what needs to be changed and having confidence
### Additional AI Behavior Rules
1. **Use English for all code comments and documentation** - All comments, variable names, function names, documentation, and user-facing text in code should be in English
2. **Clean up temporary scripts after use** - Any temporary scripts, test files, or helper files created during AI work should be removed after task completion
3. **Only make confident modifications** - Do not make speculative changes or "convenient" modifications outside the task scope. If uncertain about a change, ask for clarification rather than guessing
## Project Overview
RustFS is a high-performance distributed object storage system written in Rust, compatible with S3 API. The project adopts a modular architecture, supporting erasure coding storage, multi-tenant management, observability, and other enterprise-level features.
- All storage operations must support erasure coding
- Implement read/write quorum mechanisms
- Support data integrity verification
### 2. Network Communication
- Use gRPC for internal service communication
- HTTP/HTTPS support for S3-compatible API
- Implement connection pooling and retry mechanisms
### 3. Metadata Management
- Use FlatBuffers for serialization
- Support version control and migration
- Implement metadata caching
These rules should serve as guiding principles when developing the RustFS project, ensuring code quality, performance, and maintainability.
## Branch Management and Development Workflow
### 4. Code Operations
### Branch Management
#### Branch Management
- **🚨 CRITICAL: NEVER modify code directly on main or master branch - THIS IS ABSOLUTELY FORBIDDEN 🚨**
- **⚠️ ANY DIRECT COMMITS TO MASTER/MAIN WILL BE REJECTED AND MUST BE REVERTED IMMEDIATELY ⚠️**
- **Always work on feature branches - NO EXCEPTIONS**
- Always check the .cursorrules file before starting to ensure you understand the project guidelines
- **MANDATORY workflow for ALL changes:**
- **🚨 CRITICAL: NEVER modify code directly on main or master branch - THIS IS ABSOLUTELY FORBIDDEN 🚨**
- **⚠️ ANY DIRECT COMMITS TO MASTER/MAIN WILL BE REJECTED AND MUST BE REVERTED IMMEDIATELY ⚠️**
- **🔒 ALL CHANGES MUST GO THROUGH PULL REQUESTS - NO DIRECT COMMITS TO MAIN UNDER ANY CIRCUMSTANCES 🔒**
- **Always work on feature branches - NO EXCEPTIONS**
- Always check the .rules.md file before starting to ensure you understand the project guidelines
- **MANDATORY workflow for ALL changes:**
1. `git checkout main` (switch to main branch)
2. `git pull` (get latest changes)
3. `git checkout -b feat/your-feature-name` (create and switch to feature branch)
4. Make your changes ONLY on the feature branch
5. Test thoroughly before committing
6. Commit and push to the feature branch
7. Create a pull request for code review
- Use descriptive branch names following the pattern: `feat/feature-name`, `fix/issue-name`, `refactor/component-name`, etc.
- **Double-check current branch before ANY commit: `git branch` to ensure you're NOT on main/master**
- Ensure all changes are made on feature branches and merged through pull requests
7. **Create a pull request for code review - THIS IS THE ONLY WAY TO MERGE TO MAIN**
8. **Wait for PR approval before merging - NEVER merge your own PRs without review**
- Use descriptive branch names following the pattern: `feat/feature-name`, `fix/issue-name`, `refactor/component-name`, etc.
- **Double-check current branch before ANY commit: `git branch` to ensure you're NOT on main/master**
- **Pull Request Requirements:**
- All changes must be submitted via PR regardless of size or urgency
- PRs must include comprehensive description and testing information
- PRs must pass all CI/CD checks before merging
- PRs require at least one approval from code reviewers
- Even hotfixes and emergency changes must go through PR process
- **Enforcement:**
- Main branch should be protected with branch protection rules
- Direct pushes to main should be blocked by repository settings
- Any accidental direct commits to main must be immediately reverted via PR
#### Development Workflow
- Use English for all code comments, documentation, and variable names
- Write meaningful and descriptive names for variables, functions, and methods
- Avoid meaningless test content like "debug 111" or placeholder values
- Before each change, carefully read the existing code to ensure you understand the code structure and implementation, do not break existing logic implementation, do not introduce new issues
- Ensure each change provides sufficient test cases to guarantee code correctness
- Do not arbitrarily modify numbers and constants in test cases, carefully analyze their meaning to ensure test case correctness
- When writing or modifying tests, check existing test cases to ensure they have scientific naming and rigorous logic testing, if not compliant, modify test cases to ensure scientific and rigorous testing
- **Before committing any changes, run `cargo clippy --all-targets --all-features -- -D warnings` to ensure all code passes Clippy checks**
- After each development completion, first git add . then git commit -m "feat: feature description" or "fix: issue description", ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- **Keep commit messages concise and under 72 characters** for the title line, use body for detailed explanations if needed
- After each development completion, first git push to remote repository
- After each change completion, summarize the changes, do not create summary files, provide a brief change description, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- Provide change descriptions needed for PR in the conversation, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- **Always provide PR descriptions in English** after completing any changes, including:
- Clear and concise title following Conventional Commits format
- Detailed description of what was changed and why
- List of key changes and improvements
- Any breaking changes or migration notes if applicable
- Testing information and verification steps
- **Provide PR descriptions in copyable markdown format** enclosed in code blocks for easy one-click copying
### Development Workflow
## 🎯 **Core Development Principles**
- **🔴 Every change must be precise - don't modify unless you're confident**
- Carefully analyze code logic and ensure complete understanding before making changes
- When uncertain, prefer asking users or consulting documentation over blind modifications
- Use small iterative steps, modify only necessary parts at a time
- Evaluate impact scope before changes to ensure no new issues are introduced
- Prefer using `gh pr create` command to create Pull Requests
- Avoid having users manually create PRs through web interface
- Provide clear and professional PR titles and descriptions
- Using `gh` commands ensures better integration and automation
## 📝 **Code Quality Requirements**
- Use English for all code comments, documentation, and variable names
- Write meaningful and descriptive names for variables, functions, and methods
- Avoid meaningless test content like "debug 111" or placeholder values
- Before each change, carefully read the existing code to ensure you understand the code structure and implementation, do not break existing logic implementation, do not introduce new issues
- Ensure each change provides sufficient test cases to guarantee code correctness
- Do not arbitrarily modify numbers and constants in test cases, carefully analyze their meaning to ensure test case correctness
- When writing or modifying tests, check existing test cases to ensure they have scientific naming and rigorous logic testing, if not compliant, modify test cases to ensure scientific and rigorous testing
- **Before committing any changes, run `cargo clippy --all-targets --all-features -- -D warnings` to ensure all code passes Clippy checks**
- After each development completion, first git add . then git commit -m "feat: feature description" or "fix: issue description", ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- **Keep commit messages concise and under 72 characters** for the title line, use body for detailed explanations if needed
- After each development completion, first git push to remote repository
- After each change completion, summarize the changes, do not create summary files, provide a brief change description, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- Provide change descriptions needed for PR in the conversation, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- **Always provide PR descriptions in English** after completing any changes, including:
- Clear and concise title following Conventional Commits format
- Detailed description of what was changed and why
- List of key changes and improvements
- Any breaking changes or migration notes if applicable
- Testing information and verification steps
- **Provide PR descriptions in copyable markdown format** enclosed in code blocks for easy one-click copying
## 🚫 AI Documentation Generation Restrictions
### Forbidden Summary Documents
- **Strictly forbidden to create any form of AI-generated summary documents**
- **Do not create documents containing large amounts of emoji, detailed formatting tables and typical AI style**
- **Do not generate the following types of documents in the project:**
- Respond to the user in Chinese; use English in all other contexts.
## Project Structure & Module Organization
The workspace root hosts shared dependencies in `Cargo.toml`. The service binary lives under `rustfs/src/main.rs`, while reusable crates sit in `crates/` (`crypto`, `iam`, `kms`, and `e2e_test`). Local fixtures for standalone flows reside in `test_standalone/`, deployment manifests are under `deploy/`, Docker assets sit at the root, and automation lives in `scripts/`. Skim each crate’s README or module docs before contributing changes.
## Build, Test, and Development Commands
Run `cargo check --all-targets` for fast validation. Build release binaries via `cargo build --release` or the pipeline-aligned `make build`. Use `./build-rustfs.sh --dev` for iterative development and `./build-rustfs.sh --platform <target>` for cross-compiles. Prefer `make pre-commit` before pushing to cover formatting, clippy, checks, and tests.
Always ensure `cargo fmt --all --check`, `cargo test --workspace --exclude e2e_test`, and `cargo clippy --all-targets --all-features -- -D warnings` complete successfully after each code change to keep the tree healthy and warning-free.
## Coding Style & Naming Conventions
Formatting follows the repo `rustfmt.toml` (130-column width). Use `snake_case` for items, `PascalCase` for types, and `SCREAMING_SNAKE_CASE` for constants. Avoid `unwrap()` or `expect()` outside tests; bubble errors with `Result` and crate-specific `thiserror` types. Keep async code non-blocking and offload CPU-heavy work with `tokio::task::spawn_blocking` when necessary.
## Testing Guidelines
Co-locate unit tests with their modules and give behavior-led names such as `handles_expired_token`. Integration suites belong in each crate’s `tests/` directory, while exhaustive end-to-end scenarios live in `crates/e2e_test/`. Run `cargo test --workspace --exclude e2e_test` during iteration, `cargo nextest run --all --exclude e2e_test` when available, and finish with `cargo test --all` before requesting review. Use `NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY=` for KMS e2e tests.
When fixing bugs or adding features, include regression tests that capture the new behavior so future changes cannot silently break it.
## Commit & Pull Request Guidelines
Work on feature branches (e.g., `feat/...`) after syncing `main`. Follow Conventional Commits under 72 characters (e.g., `feat: add kms key rotation`). Each commit must compile, format cleanly, and pass `make pre-commit`. Open PRs with a concise summary, note verification commands, link relevant issues, and wait for reviewer approval.
## Security & Configuration Tips
Do not commit secrets or cloud credentials; prefer environment variables or vault tooling. Review IAM- and KMS-related changes with a second maintainer. Confirm proxy settings before running sensitive tests to avoid leaking traffic outside localhost.
| Developed based on Rust language, memory is safer | Developed in Go or C, with potential issues like memory GC/leaks |
| No telemetry. Guards against unauthorized cross-border data egress, ensuring full compliance with global regulations including GDPR (EU/UK), CCPA (US), APPI (Japan) |Potential legal exposure and data telemetry risks |
| Permissive Apache 2.0 License | AGPL V3 License and other License, polluted open source and License traps, infringement of intellectual property rights |
| 100% S3 compatible—works with any cloud provider, anywhere | Full support for S3, but no local cloud vendor support |
| Rust-based development, strong support for secure and innovative devices | Poor support for edge gateways and secure innovative devices |
| Stable commercial prices, free community support | High pricing, with costs up to $250,000 for 1PiB |
| No risk | Intellectual property risks and risks of prohibited uses |
RustFS container run as non-root user `rustfs` with id `1000`, if you run docker with `-v` to mount host directory into docker container, please make sure the owner of host directory has been changed to `1000`, otherwise you will encounter permission denied error.
- **Automatic version detection**: Uses git tags or commit hashes
- **Registry flexibility**: Supports Docker Hub, GitHub Container Registry, etc.
- **Build optimization**: Includes caching and parallel builds
You can also use Make targets for convenience:
```bash
docker compose -f docker-compose.yml up -d
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=v1.0.0 # Build specific version
make help-docker # Show all Docker-related commands
```
#### Access Monitoring Dashboards
> **Heads-up (macOS cross-compilation)**: macOS keeps the default `ulimit -n` at 256, so `cargo zigbuild` or `./build-rustfs.sh --platform ...` may fail with `ProcessFdQuotaExceeded` when targeting Linux. The build script now tries to raise the limit automatically, but if you still see the warning, run `ulimit -n 4096` (or higher) in your shell before building.
[](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
Prevent `sub` and `sup` elements from affecting the line height in all browsers.
*/
sub,
sup{
font-size:75%;
line-height:0;
position:relative;
vertical-align:baseline;
}
sub{
bottom:-0.25em;
}
sup{
top:-0.5em;
}
/*
1. Remove text indentation from table contents in Chrome and Safari. (https://bugs.chromium.org/p/chromium/issues/detail?id=999088, https://bugs.webkit.org/show_bug.cgi?id=201297)
2. Correct table border color inheritance in all Chrome and Safari. (https://bugs.chromium.org/p/chromium/issues/detail?id=935729, https://bugs.webkit.org/show_bug.cgi?id=195016)
3. Remove gaps between table borders by default.
*/
table{
text-indent:0;
/* 1 */
border-color:inherit;
/* 2 */
border-collapse:collapse;
/* 3 */
}
/*
1. Change the font styles in all browsers.
2. Remove the margin in Firefox and Safari.
3. Remove default padding in all browsers.
*/
button,
input,
optgroup,
select,
textarea{
font-family:inherit;
/* 1 */
font-feature-settings:inherit;
/* 1 */
font-variation-settings:inherit;
/* 1 */
font-size:100%;
/* 1 */
font-weight:inherit;
/* 1 */
line-height:inherit;
/* 1 */
letter-spacing:inherit;
/* 1 */
color:inherit;
/* 1 */
margin:0;
/* 2 */
padding:0;
/* 3 */
}
/*
Remove the inheritance of text transform in Edge and Firefox.
*/
button,
select{
text-transform:none;
}
/*
1. Correct the inability to style clickable types in iOS and Safari.
2. Remove default button styles.
*/
button,
input:where([type='button']),
input:where([type='reset']),
input:where([type='submit']){
-webkit-appearance:button;
/* 1 */
background-color:transparent;
/* 2 */
background-image:none;
/* 2 */
}
/*
Use the modern Firefox focus style for all focusable elements.
*/
:-moz-focusring{
outline:auto;
}
/*
Remove the additional `:invalid` styles in Firefox. (https://github.com/mozilla/gecko-dev/blob/2f9eacd9d3d995c937b4251a5557d95d494c9be1/layout/style/res/forms.css#L728-L737)
*/
:-moz-ui-invalid{
box-shadow:none;
}
/*
Add the correct vertical alignment in Chrome and Firefox.
*/
progress{
vertical-align:baseline;
}
/*
Correct the cursor style of increment and decrement buttons in Safari.
*/
::-webkit-inner-spin-button,
::-webkit-outer-spin-button{
height:auto;
}
/*
1. Correct the odd appearance in Chrome and Safari.
2. Correct the outline style in Safari.
*/
[type='search']{
-webkit-appearance:textfield;
/* 1 */
outline-offset:-2px;
/* 2 */
}
/*
Remove the inner padding in Chrome and Safari on macOS.
*/
::-webkit-search-decoration{
-webkit-appearance:none;
}
/*
1. Correct the inability to style clickable types in iOS and Safari.
2. Change font properties to `inherit` in Safari.
*/
::-webkit-file-upload-button{
-webkit-appearance:button;
/* 1 */
font:inherit;
/* 2 */
}
/*
Add the correct display in Chrome and Safari.
*/
summary{
display:list-item;
}
/*
Removes the default spacing and border for appropriate elements.
*/
blockquote,
dl,
dd,
h1,
h2,
h3,
h4,
h5,
h6,
hr,
figure,
p,
pre{
margin:0;
}
fieldset{
margin:0;
padding:0;
}
legend{
padding:0;
}
ol,
ul,
menu{
list-style:none;
margin:0;
padding:0;
}
/*
Reset default styling for dialogs.
*/
dialog{
padding:0;
}
/*
Prevent resizing textareas horizontally by default.
*/
textarea{
resize:vertical;
}
/*
1. Reset the default placeholder opacity in Firefox. (https://github.com/tailwindlabs/tailwindcss/issues/3300)
2. Set the default placeholder color to the user's configured gray 400 color.
Make sure disabled buttons don't get the pointer cursor.
*/
:disabled{
cursor:default;
}
/*
1. Make replaced elements `display: block` by default. (https://github.com/mozdevs/cssremedy/issues/14)
2. Add `vertical-align: middle` to align replaced elements more sensibly by default. (https://github.com/jensimmons/cssremedy/issues/14#issuecomment-634934210)
This can trigger a poorly considered lint error in some tools but is included by design.
*/
img,
svg,
video,
canvas,
audio,
iframe,
embed,
object{
display:block;
/* 1 */
vertical-align:middle;
/* 2 */
}
/*
Constrain images and videos to the parent width and preserve their intrinsic aspect ratio. (https://github.com/mozdevs/cssremedy/issues/14)
*/
img,
video{
max-width:100%;
height:auto;
}
/* Make elements with the HTML hidden attribute stay hidden by default */
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.