* Initial plan
* feat: add concurrency-aware buffer sizing and hot object caching for GetObject
- Implement adaptive buffer sizing based on concurrent request load
- Add per-request tracking with automatic cleanup using RAII guards
- Implement hot object cache (LRU) for frequently accessed small files (<= 10MB)
- Add disk I/O semaphore to prevent saturation under extreme load
- Integrate concurrency module into GetObject implementation
- Buffer sizes now adapt: low concurrency uses large buffers for throughput,
high concurrency uses smaller buffers for fairness and memory efficiency
- Add comprehensive metrics collection for monitoring performance
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add comprehensive documentation and tests for concurrent GetObject optimization
- Add detailed technical documentation explaining the solution
- Document root cause analysis and solution architecture
- Include performance expectations and testing recommendations
- Add integration tests for concurrency tracking and buffer sizing
- Add cache behavior tests
- Include benchmark tests for concurrent request handling
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix: address code review issues in concurrency module
- Fix race condition in cache size tracking by using consistent atomic operations within lock
- Correct buffer sizing logic: 1-2 requests use 100%, 3-4 use 75%, 5-8 use 50%, >8 use 40%
- Improve error message for semaphore acquire failure
- Document limitation of streaming cache implementation (not yet implemented)
- Add TODO for proper streaming cache with suggested approaches
- Update tests to match corrected buffer sizing thresholds
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add comprehensive implementation summary for concurrent GetObject optimization
- Executive summary of problem and solution
- Detailed architecture documentation for each optimization
- Integration points and code examples
- Complete testing strategy and performance validation steps
- Monitoring and observability guidelines with Prometheus queries
- Deployment considerations and rollout strategy
- Future enhancement roadmap
- Success criteria and quantitative metrics
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* refactor: optimize cache with lru 0.16.2 read-first pattern and add advanced features
- Implement optimized read-first cache access using peek() to reduce write lock contention
- Add batch cache operations: get_cached_batch() for efficient multi-object retrieval
- Add cache utility methods: is_cached(), remove_cached(), get_hot_keys()
- Implement warm_cache() for pre-populating cache on startup
- Add get_advanced_buffer_size() with file size and sequential read optimization
- Enhance test suite with 8 new comprehensive tests covering:
- Batch operations and cache warming
- Hot keys tracking and analysis
- Cache removal and LRU behavior verification
- Concurrent cache access performance
- Advanced buffer sizing strategies
- Improve documentation and code comments in English throughout
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add final comprehensive optimization summary
- Complete technical documentation of all optimizations
- Detailed performance analysis and metrics
- Production deployment guide with examples
- Comprehensive API reference and usage patterns
- Migration guide and future enhancement roadmap
- All documentation in professional English
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* add moka crate for cache
* feat: replace LRU with Moka cache and add comprehensive metrics
- Replace lru crate with moka 0.12.11 for superior concurrent performance
- Implement lock-free cache with automatic TTL/TTI expiration
- Add size-based eviction using Moka's weigher function
- Integrate comprehensive metrics collection throughout GetObject flow:
* Cache hit/miss tracking with per-key access counts
* Request concurrency gauges
* Disk permit wait time histograms
* Total request duration tracking
* Response size and buffer size histograms
- Deep integration with ecfs.rs GetObject operation
- Add hit rate calculation method
- Enhanced CacheStats with hit/miss counters
- Lock-free concurrent reads for better scalability
Moka advantages over LRU:
- True lock-free concurrent access
- Built-in TTL and TTI support
- Automatic size-based eviction
- Better performance under high concurrency
- Native async support
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add comprehensive Moka cache migration and metrics documentation
- Complete technical documentation of LRU to Moka migration
- Detailed performance comparison and benchmarks
- Comprehensive metrics catalog with 15+ Prometheus metrics
- Prometheus query examples for monitoring
- Dashboard and alerting recommendations
- Migration guide with code examples
- Troubleshooting guide for common issues
- Future enhancement roadmap
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* refactor: update tests for Moka cache implementation
- Completely refactor test suite to align with Moka-based concurrency.rs
- Add Clone derive to ConcurrencyManager for test convenience
- Update all tests to handle Moka's async behavior with proper delays
- Add new tests:
* test_cache_hit_rate - validate hit rate calculation
* test_ttl_expiration - verify TTL configuration
* test_is_cached_no_side_effects - ensure contains doesn't affect LRU
* bench_concurrent_cache_performance - benchmark concurrent access
- Updated existing tests:
* test_moka_cache_operations - renamed and updated for Moka API
* test_moka_cache_eviction - validate automatic eviction
* test_hot_keys_tracking - improved assertions for sorted results
* test_concurrent_cache_access - validate lock-free performance
- All tests now include appropriate sleep delays for Moka's async processing
- Enhanced documentation and assertions for better test clarity
- Total: 18 comprehensive integration tests
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* docs: add comprehensive Moka test suite documentation
- Complete test suite documentation for all 18 tests
- Detailed test patterns and best practices for Moka
- Debugging guide for common test failures
- Performance expectations and CI integration
- 100% API coverage validation
- Future enhancement roadmap
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fmt
* docs: enhance code comments and test documentation throughout
- Significantly improve module-level documentation in concurrency.rs
- Add detailed explanations of performance characteristics and expected improvements
- Document the solution to issue #911 with before/after metrics
- Enhance constant documentation with purpose and impact explanations
- Add comprehensive RAII guard documentation with thread safety notes
- Improve test documentation with detailed test strategy and rationale
- Add Moka-specific test patterns and timing guidance
- Include performance expectations for each test category
- Document why each test matters for solving the original issue
- All documentation written in professional English
- Follow Rust documentation best practices with examples
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* remove lru crate
* upgrade version
* fix: resolve test errors by correcting module structure and test assertions
- Fix test import paths to use crate:: instead of rustfs:: (binary-only crate)
- Keep test file in src/storage/ instead of tests/ (no lib.rs exists)
- Add #[cfg(test)] guard to mod declaration in storage/mod.rs
- Fix Arc type annotations for Moka's ConcurrencyManager in concurrent tests
- Correct test_buffer_size_bounds assertions to match actual implementation:
* Minimum buffer is 32KB for files <100KB, 64KB otherwise
* Maximum buffer respects base_buffer_size when concurrency is low
* Buffer sizing doesn't cap at file size, only at min/max constraints
- All 17 integration tests now pass successfully
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix: modify `TimeoutLayer::new` to `TimeoutLayer::with_status_code` and improve docker health check
* fix
* feat: implement cache writeback for small objects in GetObject
- Add cache writeback logic for objects meeting caching criteria:
* No range/part request (full object retrieval)
* Object size known and <= 10MB (max_object_size threshold)
* Not encrypted (SSE-C or managed encryption)
- Read eligible objects into memory and cache via background task
- Serve response from in-memory data for immediate client response
- Add metrics counter for cache writeback operations
- Add 3 new tests for cache writeback functionality:
* test_cache_writeback_flow - validates round-trip caching
* test_cache_writeback_size_limit - ensures large objects aren't cached
* test_cache_writeback_concurrent - validates thread-safe concurrent writes
- Update test suite documentation (now 20 comprehensive tests)
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* improve code for const
* cargo clippy
* feat: add cache enable/disable configuration via environment variable
- Add is_cache_enabled() method to ConcurrencyManager
- Read RUSTFS_OBJECT_CACHE_ENABLE env var (default: false) at startup
- Update ecfs.rs to check is_cache_enabled() before cache lookup and writeback
- Cache lookup and writeback now respect the enable flag
- Add test_cache_enable_configuration test
- Constants already exist in rustfs_config:
* ENV_OBJECT_CACHE_ENABLE = "RUSTFS_OBJECT_CACHE_ENABLE"
* DEFAULT_OBJECT_CACHE_ENABLE = false
- Total: 21 comprehensive tests passing
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fmt
* fix
* fix
* feat: implement comprehensive CachedGetObject response cache with metadata
- Add CachedGetObject struct with full response metadata fields:
* body, content_length, content_type, e_tag, last_modified
* expires, cache_control, content_disposition, content_encoding
* storage_class, version_id, delete_marker, tag_count, etc.
- Add dual cache architecture in HotObjectCache:
* Legacy simple byte cache for backward compatibility
* New response cache for complete GetObject responses
- Add ConcurrencyManager methods for response caching:
* get_cached_object() - retrieve cached response with metadata
* put_cached_object() - store complete response
* invalidate_cache() - invalidate on write operations
* invalidate_cache_versioned() - invalidate both version and latest
* make_cache_key() - generate cache keys with version support
* max_object_size() - get cache threshold
- Add builder pattern for CachedGetObject construction
- Add 6 new tests for response cache functionality (27 total):
* test_cached_get_object_basic - basic operations
* test_cached_get_object_versioned - version key handling
* test_cache_invalidation - write operation invalidation
* test_cache_invalidation_versioned - versioned invalidation
* test_cached_get_object_size_limit - size enforcement
* test_max_object_size - threshold accessor
All 27 tests pass successfully.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* feat: integrate CachedGetObject cache in ecfs.rs with full metadata and cache invalidation
Integration of CachedGetObject response cache in ecfs.rs:
1. get_object: Cache lookup uses get_cached_object() with full metadata
- Returns complete response with e_tag, last_modified, content_type, etc.
- Parses last_modified from RFC3339 string
- Supports versioned cache keys via make_cache_key()
2. get_object: Cache writeback uses put_cached_object() with metadata
- Stores content_type, e_tag, last_modified in CachedGetObject
- Background writeback via tokio::spawn()
3. Cache invalidation added to write operations:
- put_object: invalidate_cache_versioned() after store.put_object()
- put_object_extract: invalidate_cache_versioned() after each file extraction
- copy_object: invalidate_cache_versioned() after store.copy_object()
- delete_object: invalidate_cache_versioned() after store.delete_object()
- delete_objects: invalidate_cache_versioned() for each deleted object
- complete_multipart_upload: invalidate_cache_versioned() after completion
4. Fixed test_adaptive_buffer_sizing to be more robust with parallel tests
All 27 tests pass.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix: add error logging for time parsing in cache lookup and writeback
- Add warning log when RFC3339 parsing fails in cache lookup
- Add warning log when time formatting fails in cache writeback
- Improves debugging for cache-related issues
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* upgrade version
* fmt
* add http: 4138 and add max_object_size key
* fix
* fix
* fix
* modify metrics key
* add
* upgrade crates version and improve docker observability
* feat: implement adaptive I/O strategy based on disk permit wait time
Implements adaptive I/O strategy for GetObject operations based on disk permit
wait times, as requested in issue #911.
Key changes:
1. New types in concurrency.rs:
- IoLoadLevel enum: Low (<10ms), Medium (10-50ms), High (50-200ms), Critical (>200ms)
- IoStrategy struct: contains optimized I/O parameters based on load level
- IoLoadMetrics: rolling window metrics for load tracking
2. New ConcurrencyManager methods:
- calculate_io_strategy(): calculates adaptive strategy from permit wait duration
- record_permit_wait(): records observation for load tracking
- smoothed_load_level(): returns averaged load level for stability
- io_load_stats(): returns (avg_wait, p95_wait, max_wait, count) for monitoring
- adaptive_buffer_size(): convenience method for buffer sizing
3. Integration in ecfs.rs get_object:
- Calculate IoStrategy after acquiring disk permit
- Use strategy buffer_size instead of static sizing
- Consider strategy.cache_writeback_enabled in cache decision
- Record new metrics: io.load.level gauge, io.buffer.multiplier gauge,
io.strategy.selected counter by load level
4. New tests (4 added, 31 total):
- test_io_load_level_classification: validates load level thresholds
- test_io_strategy_buffer_sizing: validates buffer multipliers
- test_calculate_io_strategy: validates manager strategy calculation
- test_io_load_stats: validates stats tracking
All 31 tests pass. Clippy clean. Formatted.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* fix
* docs: add comprehensive architecture design and fix cache hit event notification
1. Added CONCURRENCY_ARCHITECTURE.md with complete design documentation:
- Executive summary and problem statement
- Architecture overview with request flow diagram
- Detailed module analysis for concurrency.rs and ecfs.rs
- Critical analysis of helper.complete() for cache hits
- Adaptive I/O strategy design with algorithm
- Cache architecture with CachedGetObject structure
- Metrics and monitoring with Prometheus queries
- Performance characteristics and future enhancements
2. Fixed critical issue: Cache hit path now calls helper.complete()
- S3 bucket notifications (s3:GetObject events) now trigger for cache hits
- Event-driven workflows (Lambda, SNS) work correctly for all object access
- Maintains audit trail for both cache hits and misses
All 31 tests pass.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix: set object info and version_id on helper before complete() for cache hits
When serving from cache, properly configure the OperationHelper before
calling complete() to ensure S3 bucket notifications include complete
object metadata:
1. Build ObjectInfo from cached metadata:
- bucket, name, size, actual_size
- etag, mod_time, version_id, delete_marker
- storage_class, content_type, content_encoding
- user_metadata (user_defined)
2. Set helper.object(event_info).version_id(version_id_str) before complete()
3. Updated CONCURRENCY_ARCHITECTURE.md with:
- Complete code example for cache hit event notification
- Explanation of why ObjectInfo is required
- Documentation of version_id handling
This ensures:
- Lambda triggers receive proper object metadata for cache hits
- SNS/SQS notifications include complete information
- Audit logs contain accurate object details
- Version-specific event routing works correctly
All 31 tests pass.
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* fix
* improve code
* fmt
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
* improve code for metrics
* improve code for metrics
* fix
* fix
* Refactor telemetry initialization and environment functions ordering
- Reorder functions in envs.rs by type size (8-bit to 64-bit, signed before unsigned) and add missing variants like get_env_opt_u16.
- Optimize init_telemetry to support three modes: stdout logging (default error level with span tracing), file rolling logs (size-based with retention), and HTTP-based observability with sub-endpoints (trace, metric, log) falling back to unified endpoint.
- Fix stdout logging issue by retaining WorkerGuard in OtelGuard to prevent premature release of async writer threads.
- Enhance observability mode with HTTP protocol, compression, and proper resource management.
- Update OtelGuard to include tracing_guard for stdout and flexi_logger_handles for file logging.
- Improve error handling and configuration extraction in OtelConfig.
* fix
* up
* fix
* fix
* improve code for obs
* fix
* fix
* add Jemalloc
* feat: optimize AI rules with unified .rules.md (#401)
* feat: optimize AI rules with unified .rules.md and entry points
- Create .rules.md as the central AI coding rules file
- Add .copilot-rules.md as GitHub Copilot entry point
- Add CLAUDE.md as Claude AI entry point
- Incorporate principles from rustfs.com project
- Add three critical rules:
1. Use English for all code comments and documentation
2. Clean up temporary scripts after use
3. Only make confident modifications
* Update CLAUDE.md
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat: translate chinese to english (#402)
* Checkpoint before follow-up message
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Translate project documentation and comments from Chinese to English
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Fix typo: "unparseable" to "unparsable" in version test comment
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Refactor compression test code with minor syntax improvements
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix: the automatic logout issue and user list display failure on Windows systems (#353) (#343) (#403)
Co-authored-by: 安正超 <anzhengchao@gmail.com>
* upgrade version
* improve code for profiling
* fix
* Initial plan
* feat: Implement layered DNS resolver with caching and validation
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* feat: Integrate DNS resolver into main application and fix formatting
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* feat: Implement enhanced DNS resolver with Moka cache and layered fallback
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* feat: Implement hickory-resolver with TLS support for enhanced DNS resolution
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* upgrade
* add .gitignore config
* fix
* add
* add
* up
* improve linux profiling
* fix
* fix
* fix
* feat(admin): Refactor profiling endpoints
Replaces the existing pprof profiling endpoints with new trigger-based APIs for CPU and memory profiling. This change simplifies the handler logic by moving the profiling implementation to a dedicated module.
A new handler file `admin/handlers/profile.rs` is created to contain the logic for these new endpoints. The core profiling functions are now expected to be in the `profiling` module, which the new handlers call to generate and save profile data.
* cargo shear --fix
* fix
* fix
* fix
---------
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
* improve code for dns log
* fix
* Improve comments, remove unused parameters in config.rs (opt), add observability enable flag, and enhance error logging in run function execution.
* fix: remove code
* improve code for tokio runtime config
* improve code for main
* fix: add tokio enable_all
* upgrade version
* improve for Cargo.toml
- Refactor WriteMode selection to ensure all variables moved into thread closures are owned types, preventing lifetime issues.
- Simplify and clarify WriteMode assignment for production and non-production environments.
- Improve code readability and maintainability for logger initialization.
* Checkpoint before follow-up message
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Translate project documentation and comments from Chinese to English
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Fix typo: "unparseable" to "unparsable" in version test comment
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
* Refactor compression test code with minor syntax improvements
Co-authored-by: anzhengchao <anzhengchao@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
- Fix Windows zip command issue by using PowerShell Compress-Archive
- Add Windows support for OSS upload with ossutil
- Replace Chinese comments with English in build.yml
- Fix bash syntax error in package_zip function
- Improve code formatting and consistency
- Update various configuration files for better cross-platform support
Resolves Windows build failures in GitHub Actions.
* feat: implement Docker improvements and binary build scripts
This commit transforms the RustFS Docker build system to follow MinIO's best practices:
## 🏗️ Binary Build Script (build-rustfs.sh)
- Create independent binary compilation script for multi-platform builds
- Support x86_64 and aarch64 Linux musl targets
- Include checksum generation and optional binary signing
- Support cross-compilation and upload functionality
- Automated target installation and environment setup
## 🐳 Docker Improvements
- Rewrite Dockerfiles to download precompiled binaries instead of building from source
- Follow MinIO's approach for security and binary verification
- Add comprehensive LABEL metadata (version, build-date, vcs-ref)
- Implement proper environment variable management
- Add signature verification with minisign (commented for future use)
- Include static curl download for minimal runtime dependencies
## 🚀 Enhanced Build Script (docker-buildx.sh)
- Inspired by MinIO's docker-buildx.sh for consistency and reliability
- Support multiple platforms with proper build arguments
- Auto-detect git versions and pass metadata to containers
- Improved error messages with helpful troubleshooting hints
- Cleanup and cache management between builds
## 🛠️ Supporting Scripts
- scripts/download-static-curl.sh: Download statically compiled curl
- scripts/setup-test-binaries.sh: Create test binaries for local development
## 📋 Key Benefits
- Faster Docker builds (download vs compile)
- Better security with signature verification
- Consistent with industry standards (MinIO approach)
- Proper multi-platform support
- Enhanced metadata and traceability
- Independent binary distribution capability
* feat: update Docker files to use Aliyun OSS for binary downloads
* feat: merge stash with OSS binary download improvements
- Remove old build_rustfs.sh script
- Keep Aliyun OSS download URLs for binary retrieval
- Maintain Docker build improvements from stash
- Resolve merge conflicts between stash and OSS updates
* feat: improve build-rustfs.sh with auto platform detection
- Auto-detect current platform using uname (like old build_rustfs.sh)
- Default to building for current platform only
- Add --all-platforms flag for cross-compilation to Linux musl targets
- Support macOS (darwin) and Linux platforms
- Auto-enable cross compilation when needed
- Provide better usage examples and platform detection info
This makes the script much more user-friendly by default while
maintaining flexibility for cross-compilation scenarios.
* refactor: simplify build-rustfs.sh for CI/CD pipeline usage
- Remove cross-compilation complexity (each CI runner builds natively)
- Focus on single platform builds per runner
- Remove --all-platforms and --cross options
- Simplify to match CI/CD workflow where:
* Linux x86_64 runner builds Linux x86_64 binary
* Linux ARM64 runner builds Linux ARM64 binary
* macOS x86_64 runner builds macOS x86_64 binary
* macOS ARM64 runner builds macOS ARM64 binary
- Keep signing and upload functionality for release CI
- Make the script's purpose and usage clearer
This aligns with the user's understanding that build scripts should
focus on native compilation for the current platform only.
* feat: update download server domain to dl.rustfs.com
- Update Dockerfile to use dl.rustfs.com/dev/ for development binaries
- Update Dockerfile.release to use dl.rustfs.com/release/ for release binaries
- Update docker-buildx.sh error messages with new URLs
- Update build-rustfs.sh upload target to dl.rustfs.com
- Update test scripts to reference new domain
- Clean up remaining git conflict markers
This centralizes all binary downloads through the official
dl.rustfs.com domain instead of direct OSS access.
* fix: correct dl.rustfs.com path structure to include /artifacts/rustfs/
- Update all download URLs to use correct path structure:
* Dev: https://dl.rustfs.com/artifacts/rustfs/dev/
* Release: https://dl.rustfs.com/artifacts/rustfs/release/
- Test confirmed both paths return HTTP 200 with application/zip content-type
- Update Dockerfile, Dockerfile.release, docker-buildx.sh, and build-rustfs.sh
- Update test scripts with correct base path
The dl.rustfs.com domain requires the /artifacts/rustfs/ prefix
to access the binary files correctly.
* feat: refactor Dockerfile to download binaries from GitHub Releases
- Changed binary download source from dl.rustfs.com to GitHub Releases
- Added support for latest release auto-detection via GitHub API
- Enhanced error handling with detailed messages and helpful links
- Added optional checksum verification using SHA256SUMS
- Improved architecture support for amd64 and arm64
- Removed unnecessary minisign installation
- Added jq dependency for JSON parsing
* feat: consolidate Docker build to use single Dockerfile
- Removed Dockerfile.release and use unified Dockerfile instead
- Updated docker-buildx.sh to use single Dockerfile with build args
- Both latest and release variants now use GitHub Releases
- Simplified build process and reduced maintenance overhead
- Updated error messages to point to GitHub releases
* chore: remove unused Dockerfile.obs
- Removed Dockerfile.obs as it's no longer needed
- Simplified Docker build configuration
* feat: unify Docker prebuild variants to use GitHub Releases
- Updated .docker/alpine/Dockerfile.prebuild to download from GitHub Releases
- Updated .docker/ubuntu/Dockerfile.prebuild to download from GitHub Releases
- All prebuild variants now consistently use GitHub Releases as binary source
- Added checksum verification for all prebuild variants
- Updated .docker/README.md to reflect unified GitHub Releases approach
- Improved error handling and user guidance in all prebuild Dockerfiles
* feat: major Docker structure simplification and consolidation
## 🎯 Simplified Docker Structure
Moved from complex multi-directory structure to clean root-level organization:
### Before:
- Dockerfile (production)
- .docker/alpine/Dockerfile.prebuild (duplicate)
- .docker/alpine/Dockerfile.source
- .docker/ubuntu/Dockerfile.prebuild (duplicate)
- .docker/ubuntu/Dockerfile.source
- .docker/ubuntu/Dockerfile.dev
### After:
- Dockerfile (production - Alpine + GitHub Releases)
- Dockerfile.source (source build - Ubuntu + cross-compilation)
- Dockerfile.dev (development - Ubuntu + full toolchain)
## 🔧 Key Changes
- **Eliminated Duplicates**: Removed redundant prebuild variants
- **Moved Core Files**: Dockerfile.{source,dev} now in root directory
- **Unified Configuration**: cargo.config.toml moved to root
- **Updated References**: Fixed all GitHub Actions and docker-compose paths
- **Simplified CI Matrix**: Reduced from 5 to 3 Docker variants
## 📦 Preserved Valuable Diversity
- **Production**: Alpine-based for minimal size
- **Source**: Ubuntu-based with cross-compilation support
- **Development**: Ubuntu-based with full development tools
## 🚀 Benefits
- ✅ Cleaner project structure
- ✅ Easier maintenance and navigation
- ✅ Reduced CI/CD complexity
- ✅ Faster build matrix execution
- ✅ Maintained functionality and flexibility
* chore: remove duplicate cargo.config.toml from .docker directory
The file is now in the root directory and no longer needed in .docker/
* fix: update all references to removed Dockerfile files
- Updated .docker/compose/README.md to reference Dockerfile.source instead of Dockerfile.obs
- Updated docker-compose.yml to use Dockerfile.source instead of Dockerfile.dev
- Updated scripts/build-docker-multiarch.sh to use Dockerfile.source for devenv builds
- Updated .github/workflows/docker.yml to use Dockerfile.source for dev builds
- Updated Makefile to use Dockerfile.source for init-devenv target
- Updated .docker/README.md to remove references to non-existent Dockerfile.dev
- Ensured all Docker configurations consistently use the unified Dockerfile structure
* chore: remove unnecessary console static assets download
- Remove obsolete download steps from build.yml and performance.yml
- Console static assets are already embedded via rust-embed in rustfs/static/
- The download from dl.rustfs.com is no longer needed as project contains complete console assets
- This improves build reliability and reduces external dependencies
- Replaced with verification steps that confirm embedded assets are present
* feat: update Makefile and README.md for new Docker build system
- Updated Makefile to use unified Docker build system:
- Replace references to non-existent Dockerfile.ubuntu22.04 and Dockerfile.rockylinux9.3
- Add new docker-buildx targets using docker-buildx.sh script
- Deprecate old docker-build-multiarch targets with warnings
- Add docker-build-production and docker-build-source targets
- Update help-docker with new command structure
- Updated README.md with docker-buildx.sh usage:
- Add comprehensive Docker build from source section
- Document multi-architecture build capabilities
- Include both script and Make target examples
- Show registry flexibility and build optimization features
- Update step numbers in quickstart guide
- Improve developer experience with clear documentation and updated tooling
- Maintain backward compatibility with deprecation warnings
* feat: integrate console assets download into build-rustfs.sh
- Added console download functionality to build-rustfs.sh:
- New flags: --download-console, --force-console-update, --console-version
- Intelligent detection of existing console assets
- Retry logic with fallback error handling
- Consistent with Docker build asset management
- Updated scripts to use unified build process:
- scripts/static.sh: Now uses build-rustfs.sh for console downloads
- scripts/run.sh: Uses build-rustfs.sh instead of direct curl
- scripts/run.ps1: Updated with guidance for Windows users
- Benefits:
- Unified asset management across all build processes
- Consistent version handling and retry logic
- Eliminates duplicate download logic
- Better error handling and user feedback
- Preparation for CI/CD integration
- Removed unused download-static-curl.sh script
This change centralizes console asset management and prepares for
streamlined CI/CD processes where build-rustfs.sh becomes the
single point of truth for binary and asset builds.
* fix: update PowerShell script to use unified console asset management
- Updated scripts/run.ps1 to use build-rustfs.sh for console asset downloads
- Added guidance for Windows users to use the unified build script
- Maintains consistency across all platform-specific scripts
* feat: add binary verification to build script
- Add verify_binary function to test built binaries
- Test --help and --version commands
- Verify binary structure with readelf/otool
- Add --skip-verification option for cross-compilation
- Include verification status in build output
- Automatic error handling if verification fails
* feat: add platform selection support to build script
- Add --platform parameter to build-rustfs.sh for target platform selection
- Implement cross-compilation support with automatic 'cross' tool detection
- Auto-enable --skip-verification for cross-compilation scenarios
- Update all Makefile build targets to use unified build-rustfs.sh script
- Add helpful error messages and suggestions for cross-compilation failures
- Update help documentation with platform selection examples
- Improve build consistency across different architectures
* feat: modernize CI/CD build process with build-rustfs.sh
- Replace manual cargo build commands with unified build-rustfs.sh script
- Simplify matrix configuration by removing cross-compilation flags
- Ensure consistency between local and CI/CD builds
- Automatic cross-compilation tool detection and installation
- Built-in binary verification for quality assurance
- Unified console asset management
- Better error handling and suggestions
Benefits:
- Consistent build process across all environments
- Automatic detection and handling of cross-compilation scenarios
- Built-in quality checks with binary verification
- Reduced CI/CD configuration complexity
- Better maintainability with single source of truth for build logic
* feat: optimize CI/CD workspace path management
- Add WORKSPACE_DIR environment variable to cache github.workspace
- Set default working-directory at job level for consistency
- Use explicit workspace paths in critical operations
- Improve reliability and maintainability of CI/CD paths
- Ensure consistent behavior across different GitHub Actions environments
Benefits:
- More explicit and reliable path handling
- Better maintainability with centralized workspace reference
- Reduced risk of path-related issues in CI/CD
- Consistent working directory across all job steps
* refactor: simplify CI/CD path management - remove redundant workspace references
- Remove unnecessary WORKSPACE_DIR environment variable
- Remove redundant defaults.run.working-directory setting
- Use relative paths since GITHUB_WORKSPACE is the default working directory
- Follow GitHub Actions best practices by leveraging default behavior
As per GitHub Actions documentation, GITHUB_WORKSPACE is already the default
working directory, so explicit specification is unnecessary in most cases.
* docs: update Docker README to reflect current project state
- Fix directory structure: remove non-existent nginx/ directory
- Correct base OS: Dockerfile.source uses Debian Bookworm, not Ubuntu 22.04
- Add docker-buildx.sh script documentation
- Update Docker tag examples to match actual CI/CD workflows
- Add CI/CD integration section explaining automated builds
- Document build variants and manual build options
- Reflect current project architecture and tooling
These updates ensure the documentation accurately represents the current
Docker build system and CI/CD workflows.
* fix: update Docker command in rustfs README
- Replace quay.io registry with Docker Hub (rustfs/rustfs:latest)
- Remove separate console port 9001, console now runs on main port 9000
- Add both Docker and Podman examples for user choice
- Fix console access URL to use unified port
This aligns with the recent console port consolidation changes
and the project's move to Docker Hub as the primary registry.
* wip
* fix: remove unnecessary entrypoint.sh and fix Docker paths
* Update Dockerfile
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* cleanup: remove unused DOCKERFILE_PATH variable from Makefile
* feat: update Docker build to use dl.rustfs.com for binary downloads
- Replace GitHub releases download with dl.rustfs.com
- Add CHANNEL parameter support (release/dev)
- Update docker-buildx.sh to support channel-specific builds
- Improve error messages with new download URLs
- Support both latest and specific version downloads
- Add channel validation in build script
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit streamlines the server configuration by unifying the S3 API and the WebUI (Console) to serve on a single port.
Previously, the console was managed by separate configuration options (`RUSTFS_CONSOLE_ENABLE` and `RUSTFS_CONSOLE_ADDRESS`), requiring a distinct port. This added complexity to deployment and configuration.
With this change:
- The `RUSTFS_CONSOLE_ADDRESS` and `RUSTFS_CONSOLE_FS_ENDPOINT` environment variables are removed.
- The WebUI is now always available and served directly from the main application port defined by `RUSTFS_ADDRESS`.
- This simplifies setup, reduces the number of exposed ports, and makes the application easier to manage and deploy, especially in containerized environments.
Users should update their startup scripts and remove the deprecated `RUSTFS_CONSOLE_*` variables.
* chore: Add copyright and license headers
This commit adds the Apache 2.0 license and a copyright notice to the header of all source files. This ensures that the licensing and copyright information is clearly stated within the codebase.
* cargo fmt
* fix
* fmt
* fix clippy
- Refactor `unmarshal_msg` to use `rmp_serde::from_slice` instead of manual parsing
- Add serde field renaming attributes to `FileMetaVersion` struct
- Remove 428 lines of manual MessagePack parsing code
- Improve string comparison using `!object.is_empty()` instead of `object != ""`
- Update volume directory numbering in run script from test{0..4} to test{1..4}
- Clean up unused imports and code
Fixes#517
