sdgoij/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-01-17 01:30:33 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(admin): Add admin v3 API routes and profiling endpoints for RustFS (#774)

Browse Source 
* add Jemalloc

* feat: optimize AI rules with unified .rules.md  (#401)

* feat: optimize AI rules with unified .rules.md and entry points

- Create .rules.md as the central AI coding rules file
- Add .copilot-rules.md as GitHub Copilot entry point
- Add CLAUDE.md as Claude AI entry point
- Incorporate principles from rustfs.com project
- Add three critical rules:
  1. Use English for all code comments and documentation
  2. Clean up temporary scripts after use
  3. Only make confident modifications

* Update CLAUDE.md

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* feat: translate chinese to english (#402)

* Checkpoint before follow-up message

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Translate project documentation and comments from Chinese to English

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Fix typo: "unparseable" to "unparsable" in version test comment

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Refactor compression test code with minor syntax improvements

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* fix: the automatic logout issue and user list display failure on Windows systems (#353) (#343) (#403)

Co-authored-by: 安正超 <anzhengchao@gmail.com>

* upgrade version

* improve code for profiling

* fix

* Initial plan

* feat: Implement layered DNS resolver with caching and validation

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Integrate DNS resolver into main application and fix formatting

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement enhanced DNS resolver with Moka cache and layered fallback

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* feat: Implement hickory-resolver with TLS support for enhanced DNS resolution

Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>

* upgrade

* add .gitignore config

* fix

* add

* add

* up

* improve linux profiling

* fix

* fix

* fix

* feat(admin): Refactor profiling endpoints

Replaces the existing pprof profiling endpoints with new trigger-based APIs for CPU and memory profiling. This change simplifies the handler logic by moving the profiling implementation to a dedicated module.

A new handler file `admin/handlers/profile.rs` is created to contain the logic for these new endpoints. The core profiling functions are now expected to be in the `profiling` module, which the new handlers call to generate and save profile data.

* cargo shear --fix

* fix

* fix

* fix

---------

Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: shiro.lee <69624924+shiroleeee@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
This commit is contained in:
houseme
2025-11-01 03:16:37 +08:00
committed by GitHub
parent 79012be2c8
commit d30c42f85a
17 changed files with 1442 additions and 167 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
.copilot-rules.md Normal file
View File

@@ -0,0 +1,58 @@
# GitHub Copilot Rules for RustFS Project
## Core Rules Reference
This project follows the comprehensive AI coding rules defined in `.rules.md`. Please refer to that file for the complete set of development guidelines, coding standards, and best practices.
## Copilot-Specific Configuration
When using GitHub Copilot for this project, ensure you:
1. **Review the unified rules**: Always check `.rules.md` for the latest project guidelines
2. **Follow branch protection**: Never attempt to commit directly to main/master branch
3. **Use English**: All code comments, documentation, and variable names must be in English
4. **Clean code practices**: Only make modifications you're confident about
5. **Test thoroughly**: Ensure all changes pass formatting, linting, and testing requirements
## Quick Reference
### Critical Rules
- 🚫 **NEVER commit directly to main/master branch**
- ✅ **ALWAYS work on feature branches**
- 📝 **ALWAYS use English for code and documentation**
- 🧹 **ALWAYS clean up temporary files after use**
- 🎯 **ONLY make confident, necessary modifications**
### Pre-commit Checklist
```bash
# Before committing, always run:
cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings
cargo check --all-targets
cargo test
```
### Branch Workflow
```bash
git checkout main
git pull origin main
git checkout -b feat/your-feature-name
# Make your changes
git add .
git commit -m "feat: your feature description"
git push origin feat/your-feature-name
gh pr create
```
## Important Notes
- This file serves as an entry point for GitHub Copilot
- All detailed rules and guidelines are maintained in `.rules.md`
- Updates to coding standards should be made in `.rules.md` to ensure consistency across all AI tools
- When in doubt, always refer to `.rules.md` for authoritative guidance
## See Also
- [.rules.md](./.rules.md) - Complete AI coding rules and guidelines
- [CONTRIBUTING.md](./CONTRIBUTING.md) - Contribution guidelines
- [README.md](./README.md) - Project overview and setup instructions

702
.rules.md Normal file
View File

@@ -0,0 +1,702 @@
# RustFS Project AI Coding Rules
## 🚨🚨🚨 CRITICAL DEVELOPMENT RULES - ZERO TOLERANCE 🚨🚨🚨
### ⛔️ ABSOLUTE PROHIBITION: NEVER COMMIT DIRECTLY TO MASTER/MAIN BRANCH ⛔️
**🔥 THIS IS THE MOST CRITICAL RULE - VIOLATION WILL RESULT IN IMMEDIATE REVERSAL 🔥**
- **🚫 ZERO DIRECT COMMITS TO MAIN/MASTER BRANCH - ABSOLUTELY FORBIDDEN**
- **🚫 ANY DIRECT COMMIT TO MAIN BRANCH MUST BE IMMEDIATELY REVERTED**
- **🚫 NO EXCEPTIONS FOR HOTFIXES, EMERGENCIES, OR URGENT CHANGES**
- **🚫 NO EXCEPTIONS FOR SMALL CHANGES, TYPOS, OR DOCUMENTATION UPDATES**
- **🚫 NO EXCEPTIONS FOR ANYONE - MAINTAINERS, CONTRIBUTORS, OR ADMINS**
### 📋 MANDATORY WORKFLOW - STRICTLY ENFORCED
**EVERY SINGLE CHANGE MUST FOLLOW THIS WORKFLOW:**
1. **Check current branch**: `git branch` (MUST NOT be on main/master)
2. **Switch to main**: `git checkout main`
3. **Pull latest**: `git pull origin main`
4. **Create feature branch**: `git checkout -b feat/your-feature-name`
5. **Make changes ONLY on feature branch**
6. **Test thoroughly before committing**
7. **Commit and push to feature branch**: `git push origin feat/your-feature-name`
8. **Create Pull Request**: Use `gh pr create` (MANDATORY)
9. **Wait for PR approval**: NO self-merging allowed
10. **Merge through GitHub interface**: ONLY after approval
### 🔒 ENFORCEMENT MECHANISMS
- **Branch protection rules**: Main branch is protected
- **Pre-commit hooks**: Will block direct commits to main
- **CI/CD checks**: All PRs must pass before merging
- **Code review requirement**: At least one approval needed
- **Automated reversal**: Direct commits to main will be automatically reverted
## 🎯 Core AI Development Principles
### Five Execution Steps
#### 1. Task Analysis and Planning
- **Clear Objectives**: Deeply understand task requirements and expected results before starting coding
- **Plan Development**: List specific files, components, and functions that need modification, explaining the reasons for changes
- **Risk Assessment**: Evaluate the impact of changes on existing functionality, develop rollback plans
#### 2. Precise Code Location
- **File Identification**: Determine specific files and line numbers that need modification
- **Impact Analysis**: Avoid modifying irrelevant files, clearly state the reason for each file modification
- **Minimization Principle**: Unless explicitly required by the task, do not create new abstraction layers or refactor existing code
#### 3. Minimal Code Changes
- **Focus on Core**: Only write code directly required by the task
- **Avoid Redundancy**: Do not add unnecessary logs, comments, tests, or error handling
- **Isolation**: Ensure new code does not interfere with existing functionality, maintain code independence
#### 4. Strict Code Review
- **Correctness Check**: Verify the correctness and completeness of code logic
- **Style Consistency**: Ensure code conforms to established project coding style
- **Side Effect Assessment**: Evaluate the impact of changes on downstream systems
#### 5. Clear Delivery Documentation
- **Change Summary**: Detailed explanation of all modifications and reasons
- **File List**: List all modified files and their specific changes
- **Risk Statement**: Mark any assumptions or potential risk points
### Core Principles
- **🎯 Precise Execution**: Strictly follow task requirements, no arbitrary innovation
- **⚡ Efficient Development**: Avoid over-design, only do necessary work
- **🛡️ Safe and Reliable**: Always follow development processes, ensure code quality and system stability
- **🔒 Cautious Modification**: Only modify when clearly knowing what needs to be changed and having confidence
### Additional AI Behavior Rules
1. **Use English for all code comments and documentation** - All comments, variable names, function names, documentation, and user-facing text in code should be in English
2. **Clean up temporary scripts after use** - Any temporary scripts, test files, or helper files created during AI work should be removed after task completion
3. **Only make confident modifications** - Do not make speculative changes or "convenient" modifications outside the task scope. If uncertain about a change, ask for clarification rather than guessing
## Project Overview
RustFS is a high-performance distributed object storage system written in Rust, compatible with S3 API. The project adopts a modular architecture, supporting erasure coding storage, multi-tenant management, observability, and other enterprise-level features.
## Core Architecture Principles
### 1. Modular Design
- Project uses Cargo workspace structure, containing multiple independent crates
- Core modules: `rustfs` (main service), `ecstore` (erasure coding storage), `common` (shared components)
- Functional modules: `iam` (identity management), `madmin` (management interface), `crypto` (encryption), etc.
- Tool modules: `cli` (command line tool), `crates/*` (utility libraries)
### 2. Asynchronous Programming Pattern
- Comprehensive use of `tokio` async runtime
- Prioritize `async/await` syntax
- Use `async-trait` for async methods in traits
- Avoid blocking operations, use `spawn_blocking` when necessary
### 3. Error Handling Strategy
- **Use modular, type-safe error handling with `thiserror`**
- Each module should define its own error type using `thiserror::Error` derive macro
- Support error chains and context information through `#[from]` and `#[source]` attributes
- Use `Result<T>` type aliases for consistency within each module
- Error conversion between modules should use explicit `From` implementations
- Follow the pattern: `pub type Result<T> = core::result::Result<T, Error>`
- Use `#[error("description")]` attributes for clear error messages
- Support error downcasting when needed through `other()` helper methods
- Implement `Clone` for errors when required by the domain logic
## Code Style Guidelines
### 1. Formatting Configuration
```toml
max_width = 130
fn_call_width = 90
single_line_let_else_max_width = 100
```
### 2. **🔧 MANDATORY Code Formatting Rules**
**CRITICAL**: All code must be properly formatted before committing. This project enforces strict formatting standards to maintain code consistency and readability.
#### Pre-commit Requirements (MANDATORY)
Before every commit, you **MUST**:
1. **Format your code**:
```bash
cargo fmt --all
```
2. **Verify formatting**:
```bash
cargo fmt --all --check
```
3. **Pass clippy checks**:
```bash
cargo clippy --all-targets --all-features -- -D warnings
```
4. **Ensure compilation**:
```bash
cargo check --all-targets
```
#### Quick Commands
Use these convenient Makefile targets for common tasks:
```bash
# Format all code
make fmt
# Check if code is properly formatted
make fmt-check
# Run clippy checks
make clippy
# Run compilation check
make check
# Run tests
make test
# Run all pre-commit checks (format + clippy + check + test)
make pre-commit
# Setup git hooks (one-time setup)
make setup-hooks
```
### 3. Naming Conventions
- Use `snake_case` for functions, variables, modules
- Use `PascalCase` for types, traits, enums
- Constants use `SCREAMING_SNAKE_CASE`
- Global variables prefix `GLOBAL_`, e.g., `GLOBAL_Endpoints`
- Use meaningful and descriptive names for variables, functions, and methods
- Avoid meaningless names like `temp`, `data`, `foo`, `bar`, `test123`
- Choose names that clearly express the purpose and intent
### 4. Type Declaration Guidelines
- **Prefer type inference over explicit type declarations** when the type is obvious from context
- Let the Rust compiler infer types whenever possible to reduce verbosity and improve maintainability
- Only specify types explicitly when:
- The type cannot be inferred by the compiler
- Explicit typing improves code clarity and readability
- Required for API boundaries (function signatures, public struct fields)
- Needed to resolve ambiguity between multiple possible types
### 5. Documentation Comments
- Public APIs must have documentation comments
- Use `///` for documentation comments
- Complex functions add `# Examples` and `# Parameters` descriptions
- Error cases use `# Errors` descriptions
- Always use English for all comments and documentation
- Avoid meaningless comments like "debug 111" or placeholder text
### 6. Import Guidelines
- Standard library imports first
- Third-party crate imports in the middle
- Project internal imports last
- Group `use` statements with blank lines between groups
## Asynchronous Programming Guidelines
### 1. Trait Definition
```rust
#[async_trait::async_trait]
pub trait StorageAPI: Send + Sync {
async fn get_object(&self, bucket: &str, object: &str) -> Result<ObjectInfo>;
}
```
### 2. Error Handling
```rust
// Use ? operator to propagate errors
async fn example_function() -> Result<()> {
let data = read_file("path").await?;
process_data(data).await?;
Ok(())
}
```
### 3. Concurrency Control
- Use `Arc` and `Mutex`/`RwLock` for shared state management
- Prioritize async locks from `tokio::sync`
- Avoid holding locks for long periods
## Logging and Tracing Guidelines
### 1. Tracing Usage
```rust
#[tracing::instrument(skip(self, data))]
async fn process_data(&self, data: &[u8]) -> Result<()> {
info!("Processing {} bytes", data.len());
// Implementation logic
}
```
### 2. Log Levels
- `error!`: System errors requiring immediate attention
- `warn!`: Warning information that may affect functionality
- `info!`: Important business information
- `debug!`: Debug information for development use
- `trace!`: Detailed execution paths
### 3. Structured Logging
```rust
info!(
counter.rustfs_api_requests_total = 1_u64,
key_request_method = %request.method(),
key_request_uri_path = %request.uri().path(),
"API request processed"
);
```
## Error Handling Guidelines
### 1. Error Type Definition
```rust
// Use thiserror for module-specific error types
#[derive(thiserror::Error, Debug)]
pub enum MyError {
#[error("IO error: {0}")]
Io(#[from] std::io::Error),
#[error("Storage error: {0}")]
Storage(#[from] ecstore::error::StorageError),
#[error("Custom error: {message}")]
Custom { message: String },
#[error("File not found: {path}")]
FileNotFound { path: String },
#[error("Invalid configuration: {0}")]
InvalidConfig(String),
}
// Provide Result type alias for the module
pub type Result<T> = core::result::Result<T, MyError>;
```
### 2. Error Helper Methods
```rust
impl MyError {
/// Create error from any compatible error type
pub fn other<E>(error: E) -> Self
where
E: Into<Box<dyn std::error::Error + Send + Sync>>,
{
MyError::Io(std::io::Error::other(error))
}
}
```
### 3. Error Context and Propagation
```rust
// Use ? operator for clean error propagation
async fn example_function() -> Result<()> {
let data = read_file("path").await?;
process_data(data).await?;
Ok(())
}
// Add context to errors
fn process_with_context(path: &str) -> Result<()> {
std::fs::read(path)
.map_err(|e| MyError::Custom {
message: format!("Failed to read {}: {}", path, e)
})?;
Ok(())
}
```
## Performance Optimization Guidelines
### 1. Memory Management
- Use `Bytes` instead of `Vec<u8>` for zero-copy operations
- Avoid unnecessary cloning, use reference passing
- Use `Arc` for sharing large objects
### 2. Concurrency Optimization
```rust
// Use join_all for concurrent operations
let futures = disks.iter().map(|disk| disk.operation());
let results = join_all(futures).await;
```
### 3. Caching Strategy
- Use `LazyLock` for global caching
- Implement LRU cache to avoid memory leaks
## Testing Guidelines
### 1. Unit Tests
```rust
#[cfg(test)]
mod tests {
use super::*;
use test_case::test_case;
#[tokio::test]
async fn test_async_function() {
let result = async_function().await;
assert!(result.is_ok());
}
#[test_case("input1", "expected1")]
#[test_case("input2", "expected2")]
fn test_with_cases(input: &str, expected: &str) {
assert_eq!(function(input), expected);
}
}
```
### 2. Integration Tests
- Use `e2e_test` module for end-to-end testing
- Simulate real storage environments
### 3. Test Quality Standards
- Write meaningful test cases that verify actual functionality
- Avoid placeholder or debug content like "debug 111", "test test", etc.
- Use descriptive test names that clearly indicate what is being tested
- Each test should have a clear purpose and verify specific behavior
- Test data should be realistic and representative of actual use cases
## Cross-Platform Compatibility Guidelines
### 1. CPU Architecture Compatibility
- **Always consider multi-platform and different CPU architecture compatibility** when writing code
- Support major architectures: x86_64, aarch64 (ARM64), and other target platforms
- Use conditional compilation for architecture-specific code:
```rust
#[cfg(target_arch = "x86_64")]
fn optimized_x86_64_function() { /* x86_64 specific implementation */ }
#[cfg(target_arch = "aarch64")]
fn optimized_aarch64_function() { /* ARM64 specific implementation */ }
#[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))]
fn generic_function() { /* Generic fallback implementation */ }
```
### 2. Platform-Specific Dependencies
- Use feature flags for platform-specific dependencies
- Provide fallback implementations for unsupported platforms
- Test on multiple architectures in CI/CD pipeline
### 3. Endianness Considerations
- Use explicit byte order conversion when dealing with binary data
- Prefer `to_le_bytes()`, `from_le_bytes()` for consistent little-endian format
- Use `byteorder` crate for complex binary format handling
### 4. SIMD and Performance Optimizations
- Use portable SIMD libraries like `wide` or `packed_simd`
- Provide fallback implementations for non-SIMD architectures
- Use runtime feature detection when appropriate
## Security Guidelines
### 1. Memory Safety
- Disable `unsafe` code (workspace.lints.rust.unsafe_code = "deny")
- Use `rustls` instead of `openssl`
### 2. Authentication and Authorization
```rust
// Use IAM system for permission checks
let identity = iam.authenticate(&access_key, &secret_key).await?;
iam.authorize(&identity, &action, &resource).await?;
```
## Configuration Management Guidelines
### 1. Environment Variables
- Use `RUSTFS_` prefix
- Support both configuration files and environment variables
- Provide reasonable default values
### 2. Configuration Structure
```rust
#[derive(Debug, Deserialize, Clone)]
pub struct Config {
pub address: String,
pub volumes: String,
#[serde(default)]
pub console_enable: bool,
}
```
## Dependency Management Guidelines
### 1. Workspace Dependencies
- Manage versions uniformly at workspace level
- Use `workspace = true` to inherit configuration
### 2. Feature Flags
```rust
[features]
default = ["file"]
gpu = ["dep:nvml-wrapper"]
kafka = ["dep:rdkafka"]
```
## Deployment and Operations Guidelines
### 1. Containerization
- Provide Dockerfile and docker-compose configuration
- Support multi-stage builds to optimize image size
### 2. Observability
- Integrate OpenTelemetry for distributed tracing
- Support Prometheus metrics collection
- Provide Grafana dashboards
### 3. Health Checks
```rust
// Implement health check endpoint
async fn health_check() -> Result<HealthStatus> {
// Check component status
}
```
## Code Review Checklist
### 1. **Code Formatting and Quality (MANDATORY)**
- [ ] **Code is properly formatted** (`cargo fmt --all --check` passes)
- [ ] **All clippy warnings are resolved** (`cargo clippy --all-targets --all-features -- -D warnings` passes)
- [ ] **Code compiles successfully** (`cargo check --all-targets` passes)
- [ ] **Pre-commit hooks are working** and all checks pass
- [ ] **No formatting-related changes** mixed with functional changes (separate commits)
### 2. Functionality
- [ ] Are all error cases properly handled?
- [ ] Is there appropriate logging?
- [ ] Is there necessary test coverage?
### 3. Performance
- [ ] Are unnecessary memory allocations avoided?
- [ ] Are async operations used correctly?
- [ ] Are there potential deadlock risks?
### 4. Security
- [ ] Are input parameters properly validated?
- [ ] Are there appropriate permission checks?
- [ ] Is information leakage avoided?
### 5. Cross-Platform Compatibility
- [ ] Does the code work on different CPU architectures (x86_64, aarch64)?
- [ ] Are platform-specific features properly gated with conditional compilation?
- [ ] Is byte order handling correct for binary data?
- [ ] Are there appropriate fallback implementations for unsupported platforms?
### 6. Code Commits and Documentation
- [ ] Does it comply with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)?
- [ ] Are commit messages concise and under 72 characters for the title line?
- [ ] Commit titles should be concise and in English, avoid Chinese
- [ ] Is PR description provided in copyable markdown format for easy copying?
## Common Patterns and Best Practices
### 1. Resource Management
```rust
// Use RAII pattern for resource management
pub struct ResourceGuard {
resource: Resource,
}
impl Drop for ResourceGuard {
fn drop(&mut self) {
// Clean up resources
}
}
```
### 2. Dependency Injection
```rust
// Use dependency injection pattern
pub struct Service {
config: Arc<Config>,
storage: Arc<dyn StorageAPI>,
}
```
### 3. Graceful Shutdown
```rust
// Implement graceful shutdown
async fn shutdown_gracefully(shutdown_rx: &mut Receiver<()>) {
tokio::select! {
_ = shutdown_rx.recv() => {
info!("Received shutdown signal");
// Perform cleanup operations
}
_ = tokio::time::sleep(SHUTDOWN_TIMEOUT) => {
warn!("Shutdown timeout reached");
}
}
}
```
## Domain-Specific Guidelines
### 1. Storage Operations
- All storage operations must support erasure coding
- Implement read/write quorum mechanisms
- Support data integrity verification
### 2. Network Communication
- Use gRPC for internal service communication
- HTTP/HTTPS support for S3-compatible API
- Implement connection pooling and retry mechanisms
### 3. Metadata Management
- Use FlatBuffers for serialization
- Support version control and migration
- Implement metadata caching
## Branch Management and Development Workflow
### Branch Management
- **🚨 CRITICAL: NEVER modify code directly on main or master branch - THIS IS ABSOLUTELY FORBIDDEN 🚨**
- **⚠️ ANY DIRECT COMMITS TO MASTER/MAIN WILL BE REJECTED AND MUST BE REVERTED IMMEDIATELY ⚠️**
- **🔒 ALL CHANGES MUST GO THROUGH PULL REQUESTS - NO DIRECT COMMITS TO MAIN UNDER ANY CIRCUMSTANCES 🔒**
- **Always work on feature branches - NO EXCEPTIONS**
- Always check the .rules.md file before starting to ensure you understand the project guidelines
- **MANDATORY workflow for ALL changes:**
1. `git checkout main` (switch to main branch)
2. `git pull` (get latest changes)
3. `git checkout -b feat/your-feature-name` (create and switch to feature branch)
4. Make your changes ONLY on the feature branch
5. Test thoroughly before committing
6. Commit and push to the feature branch
7. **Create a pull request for code review - THIS IS THE ONLY WAY TO MERGE TO MAIN**
8. **Wait for PR approval before merging - NEVER merge your own PRs without review**
- Use descriptive branch names following the pattern: `feat/feature-name`, `fix/issue-name`, `refactor/component-name`, etc.
- **Double-check current branch before ANY commit: `git branch` to ensure you're NOT on main/master**
- **Pull Request Requirements:**
- All changes must be submitted via PR regardless of size or urgency
- PRs must include comprehensive description and testing information
- PRs must pass all CI/CD checks before merging
- PRs require at least one approval from code reviewers
- Even hotfixes and emergency changes must go through PR process
- **Enforcement:**
- Main branch should be protected with branch protection rules
- Direct pushes to main should be blocked by repository settings
- Any accidental direct commits to main must be immediately reverted via PR
### Development Workflow
## 🎯 **Core Development Principles**
- **🔴 Every change must be precise - don't modify unless you're confident**
- Carefully analyze code logic and ensure complete understanding before making changes
- When uncertain, prefer asking users or consulting documentation over blind modifications
- Use small iterative steps, modify only necessary parts at a time
- Evaluate impact scope before changes to ensure no new issues are introduced
- **🚀 GitHub PR creation prioritizes gh command usage**
- Prefer using `gh pr create` command to create Pull Requests
- Avoid having users manually create PRs through web interface
- Provide clear and professional PR titles and descriptions
- Using `gh` commands ensures better integration and automation
## 📝 **Code Quality Requirements**
- Use English for all code comments, documentation, and variable names
- Write meaningful and descriptive names for variables, functions, and methods
- Avoid meaningless test content like "debug 111" or placeholder values
- Before each change, carefully read the existing code to ensure you understand the code structure and implementation, do not break existing logic implementation, do not introduce new issues
- Ensure each change provides sufficient test cases to guarantee code correctness
- Do not arbitrarily modify numbers and constants in test cases, carefully analyze their meaning to ensure test case correctness
- When writing or modifying tests, check existing test cases to ensure they have scientific naming and rigorous logic testing, if not compliant, modify test cases to ensure scientific and rigorous testing
- **Before committing any changes, run `cargo clippy --all-targets --all-features -- -D warnings` to ensure all code passes Clippy checks**
- After each development completion, first git add . then git commit -m "feat: feature description" or "fix: issue description", ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- **Keep commit messages concise and under 72 characters** for the title line, use body for detailed explanations if needed
- After each development completion, first git push to remote repository
- After each change completion, summarize the changes, do not create summary files, provide a brief change description, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- Provide change descriptions needed for PR in the conversation, ensure compliance with [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- **Always provide PR descriptions in English** after completing any changes, including:
- Clear and concise title following Conventional Commits format
- Detailed description of what was changed and why
- List of key changes and improvements
- Any breaking changes or migration notes if applicable
- Testing information and verification steps
- **Provide PR descriptions in copyable markdown format** enclosed in code blocks for easy one-click copying
## 🚫 AI Documentation Generation Restrictions
### Forbidden Summary Documents
- **Strictly forbidden to create any form of AI-generated summary documents**
- **Do not create documents containing large amounts of emoji, detailed formatting tables and typical AI style**
- **Do not generate the following types of documents in the project:**
- Benchmark summary documents (BENCHMARK*.md)
- Implementation comparison analysis documents (IMPLEMENTATION_COMPARISON*.md)
- Performance analysis report documents
- Architecture summary documents
- Feature comparison documents
- Any documents with large amounts of emoji and formatted content
- **If documentation is needed, only create when explicitly requested by the user, and maintain a concise and practical style**
- **Documentation should focus on actually needed information, avoiding excessive formatting and decorative content**
- **Any discovered AI-generated summary documents should be immediately deleted**
### Allowed Documentation Types
- README.md (project introduction, keep concise)
- Technical documentation (only create when explicitly needed)
- User manual (only create when explicitly needed)
- API documentation (generated from code)
- Changelog (CHANGELOG.md)
These rules should serve as guiding principles when developing the RustFS project, ensuring code quality, performance, and maintainability.

84
CLAUDE.md
View File

@@ -4,23 +4,28 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Project Overview
RustFS is a high-performance distributed object storage software built with Rust, providing S3-compatible APIs and advanced features like data lakes, AI, and big data support. It's designed as an alternative to MinIO with better performance and a more business-friendly Apache 2.0 license.
RustFS is a high-performance distributed object storage software built with Rust, providing S3-compatible APIs and
advanced features like data lakes, AI, and big data support. It's designed as an alternative to MinIO with better
performance and a more business-friendly Apache 2.0 license.
## Build Commands
### Primary Build Commands
- `cargo build --release` - Build the main RustFS binary
- `./build-rustfs.sh` - Recommended build script that handles console resources and cross-platform compilation
- `./build-rustfs.sh --dev` - Development build with debug symbols
- `make build` or `just build` - Use Make/Just for standardized builds
### Platform-Specific Builds
- `./build-rustfs.sh --platform x86_64-unknown-linux-musl` - Build for musl target
- `./build-rustfs.sh --platform aarch64-unknown-linux-gnu` - Build for ARM64
- `make build-musl` or `just build-musl` - Build musl variant
- `make build-cross-all` - Build all supported architectures
### Testing Commands
- `cargo test --workspace --exclude e2e_test` - Run unit tests (excluding e2e tests)
- `cargo nextest run --all --exclude e2e_test` - Use nextest if available (faster)
- `cargo test --all --doc` - Run documentation tests
@@ -28,22 +33,30 @@ RustFS is a high-performance distributed object storage software built with Rust
- `make pre-commit` - Run all quality checks (fmt, clippy, check, test)
### End-to-End Testing
- `cargo test --package e2e_test` - Run all e2e tests
- `./scripts/run_e2e_tests.sh` - Run e2e tests via script
- `./scripts/run_scanner_benchmarks.sh` - Run scanner performance benchmarks
### KMS-Specific Testing (with proxy bypass)
- `NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1` - Run complete KMS end-to-end test
- `NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test kms:: -- --nocapture --test-threads=1` - Run all KMS tests
-
`NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1` -
Run complete KMS end-to-end test
-
`NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test kms:: -- --nocapture --test-threads=1` -
Run all KMS tests
- `cargo test --package e2e_test test_local_kms_key_isolation -- --nocapture --test-threads=1` - Test KMS key isolation
- `cargo test --package e2e_test test_local_kms_large_file -- --nocapture --test-threads=1` - Test KMS with large files
### Code Quality
- `cargo fmt --all` - Format code
- `cargo clippy --all-targets --all-features -- -D warnings` - Lint code
- `make pre-commit` or `just pre-commit` - Run all quality checks (fmt, clippy, check, test)
### Quick Development Commands
- `make help` or `just help` - Show all available commands with descriptions
- `make help-build` - Show detailed build options and cross-compilation help
- `make help-docker` - Show comprehensive Docker build and deployment options
@@ -52,6 +65,7 @@ RustFS is a high-performance distributed object storage software built with Rust
- `./scripts/probe.sh` - Health check and connectivity testing
### Docker Build Commands
- `make docker-buildx` - Build multi-architecture production images
- `make docker-dev-local` - Build development image for local use
- `./docker-buildx.sh --push` - Build and push production images
@@ -61,6 +75,7 @@ RustFS is a high-performance distributed object storage software built with Rust
### Core Components
**Main Binary (`rustfs/`):**
- Entry point at `rustfs/src/main.rs`
- Core modules: admin, auth, config, server, storage, license management, profiling
- HTTP server with S3-compatible APIs
@@ -68,10 +83,11 @@ RustFS is a high-performance distributed object storage software built with Rust
- Parallel service initialization with DNS resolver, bucket metadata, and IAM
**Key Crates (`crates/`):**
- `ecstore` - Erasure coding storage implementation (core storage layer)
- `iam` - Identity and Access Management
- `kms` - Key Management Service for encryption and key handling
- `madmin` - Management dashboard and admin API interface
- `madmin` - Management dashboard and admin API interface
- `s3select-api` & `s3select-query` - S3 Select API and query engine
- `config` - Configuration management with notify features
- `crypto` - Cryptography and security features
@@ -94,6 +110,7 @@ RustFS is a high-performance distributed object storage software built with Rust
- `targets` - Target-specific configurations and utilities
### Build System
- Cargo workspace with 25+ crates (including new KMS functionality)
- Custom `build-rustfs.sh` script for advanced build options
- Multi-architecture Docker builds via `docker-buildx.sh`
@@ -103,10 +120,11 @@ RustFS is a high-performance distributed object storage software built with Rust
- Performance benchmarking and audit workflows
### Key Dependencies
- `axum` - HTTP framework for S3 API server
- `tokio` - Async runtime
- `s3s` - S3 protocol implementation library
- `datafusion` - For S3 Select query processing
- `datafusion` - For S3 Select query processing
- `hyper`/`hyper-util` - HTTP client/server utilities
- `rustls` - TLS implementation
- `serde`/`serde_json` - Serialization
@@ -115,6 +133,7 @@ RustFS is a high-performance distributed object storage software built with Rust
- `tikv-jemallocator` - Memory allocator for Linux GNU builds
### Development Workflow
- Console resources are embedded during build via `rust-embed`
- Protocol buffers generated via custom `gproto` binary
- E2E tests in separate crate (`e2e_test`) with comprehensive KMS testing
@@ -124,14 +143,16 @@ RustFS is a high-performance distributed object storage software built with Rust
- Git hooks setup available via `make setup-hooks` or `just setup-hooks`
### Performance & Observability
- Performance profiling available with `pprof` integration (disabled on Windows)
- Profiling enabled via environment variables in production
- Built-in observability with OpenTelemetry integration
- Background services (scanner, heal) can be controlled via environment variables:
- `RUSTFS_ENABLE_SCANNER` (default: true)
- `RUSTFS_ENABLE_HEAL` (default: true)
- `RUSTFS_ENABLE_SCANNER` (default: true)
- `RUSTFS_ENABLE_HEAL` (default: true)
### Service Architecture
- Service state management with graceful shutdown handling
- Parallel initialization of core systems (DNS, bucket metadata, IAM)
- Event notification system with MQTT and webhook support
@@ -139,6 +160,7 @@ RustFS is a high-performance distributed object storage software built with Rust
- Jemalloc allocator for Linux GNU targets for better performance
## Environment Variables
- `RUSTFS_ENABLE_SCANNER` - Enable/disable background data scanner (default: true)
- `RUSTFS_ENABLE_HEAL` - Enable/disable auto-heal functionality (default: true)
- Various profiling and observability controls
@@ -146,12 +168,14 @@ RustFS is a high-performance distributed object storage software built with Rust
- Test environment configurations in `scripts/dev_rustfs.env`
### KMS Environment Variables
- `NO_PROXY=127.0.0.1,localhost` - Required for KMS E2E tests to bypass proxy
- `HTTP_PROXY=` `HTTPS_PROXY=` `http_proxy=` `https_proxy=` - Clear proxy settings for local KMS testing
## KMS (Key Management Service) Architecture
### KMS Implementation Status
- **Full KMS Integration:** Complete implementation with Local and Vault backends
- **Automatic Configuration:** KMS auto-configures on startup with `--kms-enable` flag
- **Encryption Support:** Full S3-compatible server-side encryption (SSE-S3, SSE-KMS, SSE-C)
@@ -159,18 +183,21 @@ RustFS is a high-performance distributed object storage software built with Rust
- **Production Ready:** Comprehensive testing including large files and key isolation
### KMS Configuration
- **Local Backend:** `--kms-backend local --kms-key-dir <path> --kms-default-key-id <id>`
- **Vault Backend:** `--kms-backend vault --kms-vault-endpoint <url> --kms-vault-key-name <name>`
- **Auto-startup:** KMS automatically initializes when `--kms-enable` is provided
- **Manual Configuration:** Also supports dynamic configuration via admin API
### S3 Encryption Support
- **SSE-S3:** Server-side encryption with S3-managed keys (`ServerSideEncryption: AES256`)
- **SSE-KMS:** Server-side encryption with KMS-managed keys (`ServerSideEncryption: aws:kms`)
- **SSE-C:** Server-side encryption with customer-provided keys
- **Response Headers:** All encryption types return correct `server_side_encryption` headers in PUT/GET responses
### KMS Testing Architecture
- **Comprehensive E2E Tests:** Located in `crates/e2e_test/src/kms/`
- **Test Environments:** Automated test environment setup with temporary directories
- **Encryption Coverage:** Tests all three encryption types (SSE-S3, SSE-KMS, SSE-C)
@@ -178,6 +205,7 @@ RustFS is a high-performance distributed object storage software built with Rust
- **Edge Cases:** Key isolation, large file handling, error scenarios
### Key Files for KMS
- `crates/kms/` - Core KMS implementation with Local/Vault backends
- `rustfs/src/main.rs` - KMS auto-initialization in `init_kms_system()`
- `rustfs/src/storage/ecfs.rs` - SSE encryption/decryption in PUT/GET operations
@@ -186,54 +214,62 @@ RustFS is a high-performance distributed object storage software built with Rust
- `crates/rio/src/encrypt_reader.rs` - Streaming encryption for large files
## Code Style and Safety Requirements
- **Language Requirements:**
- Communicate with me in Chinese, but **only English can be used in code files**
- Code comments, function names, variable names, and all text in source files must be in English only
- No Chinese characters, emojis, or non-ASCII characters are allowed in any source code files
- This includes comments, strings, documentation, and any other text within code files
- Communicate with me in Chinese, but **only English can be used in code files**
- Code comments, function names, variable names, and all text in source files must be in English only
- No Chinese characters, emojis, or non-ASCII characters are allowed in any source code files
- This includes comments, strings, documentation, and any other text within code files
- **Safety-Critical Rules:**
- `unsafe_code = "deny"` enforced at workspace level
- Never use `unwrap()`, `expect()`, or panic-inducing code except in tests
- Avoid blocking I/O operations in async contexts
- Use proper error handling with `Result<T, E>` and `Option<T>`
- Follow Rust's ownership and borrowing rules strictly
- `unsafe_code = "deny"` enforced at workspace level
- Never use `unwrap()`, `expect()`, or panic-inducing code except in tests
- Avoid blocking I/O operations in async contexts
- Use proper error handling with `Result<T, E>` and `Option<T>`
- Follow Rust's ownership and borrowing rules strictly
- **Performance Guidelines:**
- Use `cargo clippy --all-targets --all-features -- -D warnings` to catch issues
- Prefer `anyhow` for error handling in applications, `thiserror` for libraries
- Use appropriate async runtimes and avoid blocking calls
- Use `cargo clippy --all-targets --all-features -- -D warnings` to catch issues
- Prefer `anyhow` for error handling in applications, `thiserror` for libraries
- Use appropriate async runtimes and avoid blocking calls
- **Testing Standards:**
- All new features must include comprehensive tests
- Use `#[cfg(test)]` for test-only code that may use panic macros
- E2E tests should cover KMS integration scenarios
- All new features must include comprehensive tests
- Use `#[cfg(test)]` for test-only code that may use panic macros
- E2E tests should cover KMS integration scenarios
## Common Development Tasks
### Running KMS Tests Locally
1. **Clear proxy settings:** KMS tests require direct localhost connections
2. **Use serial execution:** `--test-threads=1` prevents port conflicts
3. **Enable output:** `--nocapture` shows detailed test logs
4. **Full command:** `NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1`
4. **Full command:**
`NO_PROXY=127.0.0.1,localhost HTTP_PROXY= HTTPS_PROXY= http_proxy= https_proxy= cargo test --package e2e_test test_local_kms_end_to_end -- --nocapture --test-threads=1`
### KMS Development Workflow
1. **Code changes:** Modify KMS-related code in `crates/kms/` or `rustfs/src/`
2. **Compile:** Always run `cargo build` after changes
3. **Test specific functionality:** Use targeted test commands for faster iteration
4. **Full validation:** Run complete end-to-end tests before commits
### Debugging KMS Issues
- **Server startup:** Check that KMS auto-initializes with debug logs
- **Encryption failures:** Verify SSE headers are correctly set in both PUT and GET responses
- **Test failures:** Use `--nocapture` to see detailed error messages
- **Key management:** Test admin API endpoints with proper authentication
## Important Reminders
- **Always compile after code changes:** Use `cargo build` to catch errors early
- **Don't bypass tests:** All functionality must be properly tested, not worked around
- **Use proper error handling:** Never use `unwrap()` or `expect()` in production code (except tests)
- **Follow S3 compatibility:** Ensure all encryption types return correct HTTP response headers
# important-instruction-reminders
Do what has been asked; nothing more, nothing less.
NEVER create files unless they're absolutely necessary for achieving your goal.
ALWAYS prefer editing an existing file to creating a new one.
NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User.
NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly
requested by the User.

143
Cargo.lock generated
View File

@@ -3161,6 +3161,25 @@ dependencies = [
"syn 2.0.108",
]
[[package]]
name = "env_filter"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1bf3c259d255ca70051b30e2e95b5446cdb8949ac4cd22c0d7fd634d89f568e2"
dependencies = [
"log",
]
[[package]]
name = "env_logger"
version = "0.11.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13c863f0904021b108aa8b2f55046443e6b1ebde8fd4a15c399893aae4fa069f"
dependencies = [
"env_filter",
"log",
]
[[package]]
name = "equator"
version = "0.4.2"
@@ -3609,7 +3628,7 @@ dependencies = [
"http-body-util",
"opentelemetry-semantic-conventions",
"percent-encoding",
"prost",
"prost 0.14.1",
"prost-types",
"reqwest",
"rustc_version",
@@ -3717,7 +3736,7 @@ dependencies = [
"mime",
"percent-encoding",
"pin-project",
"prost",
"prost 0.14.1",
"prost-types",
"reqwest",
"serde",
@@ -4357,6 +4376,28 @@ dependencies = [
"str_stack",
]
[[package]]
name = "inferno"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e96d2465363ed2d81857759fc864cf6bb7997f79327aec028d65bd7989393685"
dependencies = [
"ahash",
"clap",
"crossbeam-channel",
"crossbeam-utils",
"dashmap",
"env_logger",
"indexmap 2.12.0",
"itoa",
"log",
"num-format",
"once_cell",
"quick-xml 0.37.5",
"rgb",
"str_stack",
]
[[package]]
name = "inotify"
version = "0.11.0"
@@ -4464,6 +4505,23 @@ version = "1.0.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4a5f13b858c8d314ee3e8f639011f7ccefe71f97f96e50151fb991f267928e2c"
[[package]]
name = "jemalloc_pprof"
version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "74ff642505c7ce8d31c0d43ec0e235c6fd4585d9b8172d8f9dd04d36590200b5"
dependencies = [
"anyhow",
"libc",
"mappings",
"once_cell",
"pprof_util",
"tempfile",
"tikv-jemalloc-ctl",
"tokio",
"tracing",
]
[[package]]
name = "jobserver"
version = "0.1.34"
@@ -4793,6 +4851,19 @@ dependencies = [
"pkg-config",
]
[[package]]
name = "mappings"
version = "0.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "db4d277bb50d4508057e7bddd7fcd19ef4a4cc38051b6a5a36868d75ae2cbeb9"
dependencies = [
"anyhow",
"libc",
"once_cell",
"pprof_util",
"tracing",
]
[[package]]
name = "matchers"
version = "0.2.0"
@@ -5405,7 +5476,7 @@ dependencies = [
"opentelemetry-http",
"opentelemetry-proto",
"opentelemetry_sdk 0.31.0",
"prost",
"prost 0.14.1",
"thiserror 2.0.17",
"tokio",
"tonic",
@@ -5420,7 +5491,7 @@ checksum = "a7175df06de5eaee9909d4805a3d07e28bb752c34cab57fa9cff549da596b30f"
dependencies = [
"opentelemetry 0.31.0",
"opentelemetry_sdk 0.31.0",
"prost",
"prost 0.14.1",
"tonic",
"tonic-prost",
]
@@ -5919,7 +5990,7 @@ dependencies = [
"backtrace",
"cfg-if",
"findshlibs",
"inferno",
"inferno 0.11.21",
"libc",
"log",
"nix 0.26.4",
@@ -5933,6 +6004,21 @@ dependencies = [
"thiserror 2.0.17",
]
[[package]]
name = "pprof_util"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f9aba4251d95ac86f14c33e688d57a9344bfcff29e9b0c5a063fc66b5facc8a1"
dependencies = [
"anyhow",
"backtrace",
"flate2",
"inferno 0.12.3",
"num",
"paste",
"prost 0.13.5",
]
[[package]]
name = "ppv-lite86"
version = "0.2.21"
@@ -5980,6 +6066,16 @@ dependencies = [
"unicode-ident",
]
[[package]]
name = "prost"
version = "0.13.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2796faa41db3ec313a31f7624d9286acf277b52de526150b7e69f3debf891ee5"
dependencies = [
"bytes",
"prost-derive 0.13.5",
]
[[package]]
name = "prost"
version = "0.14.1"
@@ -5987,7 +6083,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7231bd9b3d3d33c86b58adbac74b5ec0ad9f496b19d22801d773636feaa95f3d"
dependencies = [
"bytes",
"prost-derive",
"prost-derive 0.14.1",
]
[[package]]
@@ -6003,7 +6099,7 @@ dependencies = [
"once_cell",
"petgraph 0.7.1",
"prettyplease",
"prost",
"prost 0.14.1",
"prost-types",
"pulldown-cmark",
"pulldown-cmark-to-cmark",
@@ -6012,6 +6108,19 @@ dependencies = [
"tempfile",
]
[[package]]
name = "prost-derive"
version = "0.13.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [
"anyhow",
"itertools 0.14.0",
"proc-macro2",
"quote",
"syn 2.0.108",
]
[[package]]
name = "prost-derive"
version = "0.14.1"
@@ -6031,7 +6140,7 @@ version = "0.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b9b4db3d6da204ed77bb26ba83b6122a73aeb2e87e25fbf7ad2e84c4ccbf8f72"
dependencies = [
"prost",
"prost 0.14.1",
]
[[package]]
@@ -6676,13 +6785,13 @@ dependencies = [
"http-body 1.0.1",
"hyper 1.7.0",
"hyper-util",
"jemalloc_pprof",
"libsystemd",
"matchit",
"md5",
"metrics",
"mimalloc",
"mime_guess",
"opentelemetry 0.31.0",
"pin-project-lite",
"pprof",
"reqwest",
@@ -6719,6 +6828,7 @@ dependencies = [
"sysctl",
"sysinfo",
"thiserror 2.0.17",
"tikv-jemalloc-ctl",
"tikv-jemallocator",
"time",
"tokio",
@@ -7146,7 +7256,7 @@ name = "rustfs-protos"
version = "0.0.5"
dependencies = [
"flatbuffers",
"prost",
"prost 0.14.1",
"rustfs-common",
"tonic",
"tonic-prost",
@@ -8623,6 +8733,17 @@ dependencies = [
"ordered-float",
]
[[package]]
name = "tikv-jemalloc-ctl"
version = "0.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "661f1f6a57b3a36dc9174a2c10f19513b4866816e13425d3e418b11cc37bc24c"
dependencies = [
"libc",
"paste",
"tikv-jemalloc-sys",
]
[[package]]
name = "tikv-jemalloc-sys"
version = "0.6.1+5.3.0-1-ge13ca993e8ccb9ba9847cc330696e02839f328f7"
@@ -8898,7 +9019,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "66bd50ad6ce1252d87ef024b3d64fe4c3cf54a86fb9ef4c631fdd0ded7aeaa67"
dependencies = [
"bytes",
"prost",
"prost 0.14.1",
"tonic",
]

10
Cargo.toml
View File

@@ -264,6 +264,16 @@ opentelemetry_sdk = { version = "0.31.0" }
opentelemetry-semantic-conventions = { version = "0.31.0", features = ["semconv_experimental"] }
opentelemetry-stdout = { version = "0.31.0" }
# Performance Analysis and Memory Profiling
# Use tikv-jemallocator as memory allocator and enable performance analysis
tikv-jemallocator = { version = "0.6", features = ["profiling", "stats", "unprefixed_malloc_on_supported_platforms", "background_threads"] }
# Used to control and obtain statistics for jemalloc at runtime
tikv-jemalloc-ctl = { version = "0.6", features = ["use_std", "stats", "profiling"] }
# Used to generate pprof-compatible memory profiling data and support symbolization and flame graphs
jemalloc_pprof = { version = "0.8.1", features = ["symbolize", "flamegraph"] }
# Used to generate CPU performance analysis data and flame diagrams
pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
mimalloc = "0.1"
[workspace.metadata.cargo-shear]

2
crates/config/src/constants/app.rs
View File

@@ -138,7 +138,7 @@ pub const DEFAULT_LOG_ROTATION_SIZE_MB: u64 = 100;
/// It is used to rotate the logs of the application.
/// Default value: hour, eg: day,hour,minute,second
/// Environment variable: RUSTFS_OBS_LOG_ROTATION_TIME
pub const DEFAULT_LOG_ROTATION_TIME: &str = "day";
pub const DEFAULT_LOG_ROTATION_TIME: &str = "hour";
/// Default log keep files for rustfs
/// This is the default log keep files for rustfs.

1
crates/config/src/constants/mod.rs
View File

@@ -15,6 +15,7 @@
pub(crate) mod app;
pub(crate) mod console;
pub(crate) mod env;
pub(crate) mod profiler;
pub(crate) mod runtime;
pub(crate) mod targets;
pub(crate) mod tls;

41
crates/config/src/constants/profiler.rs Normal file
View File

@@ -0,0 +1,41 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub const ENV_ENABLE_PROFILING: &str = "RUSTFS_ENABLE_PROFILING";
// CPU profiling
pub const ENV_CPU_MODE: &str = "RUSTFS_PROF_CPU_MODE"; // off|continuous|periodic
pub const ENV_CPU_FREQ: &str = "RUSTFS_PROF_CPU_FREQ";
pub const ENV_CPU_INTERVAL_SECS: &str = "RUSTFS_PROF_CPU_INTERVAL_SECS";
pub const ENV_CPU_DURATION_SECS: &str = "RUSTFS_PROF_CPU_DURATION_SECS";
// Memory profiling (jemalloc)
pub const ENV_MEM_PERIODIC: &str = "RUSTFS_PROF_MEM_PERIODIC";
pub const ENV_MEM_INTERVAL_SECS: &str = "RUSTFS_PROF_MEM_INTERVAL_SECS";
// Output directory
pub const ENV_OUTPUT_DIR: &str = "RUSTFS_PROF_OUTPUT_DIR";
// Defaults
pub const DEFAULT_ENABLE_PROFILING: bool = false;
pub const DEFAULT_CPU_MODE: &str = "off";
pub const DEFAULT_CPU_FREQ: usize = 100;
pub const DEFAULT_CPU_INTERVAL_SECS: u64 = 300;
pub const DEFAULT_CPU_DURATION_SECS: u64 = 60;
pub const DEFAULT_MEM_PERIODIC: bool = false;
pub const DEFAULT_MEM_INTERVAL_SECS: u64 = 300;
pub const DEFAULT_OUTPUT_DIR: &str = ".";

2
crates/config/src/lib.rs
View File

@@ -21,6 +21,8 @@ pub use constants::console::*;
#[cfg(feature = "constants")]
pub use constants::env::*;
#[cfg(feature = "constants")]
pub use constants::profiler::*;
#[cfg(feature = "constants")]
pub use constants::runtime::*;
#[cfg(feature = "constants")]
pub use constants::targets::*;

16
rustfs/Cargo.toml
View File

@@ -123,8 +123,6 @@ zip = { workspace = true }
# Observability and Metrics
metrics = { workspace = true }
opentelemetry = { workspace = true }
[target.'cfg(any(target_os = "macos", target_os = "freebsd", target_os = "netbsd", target_os = "openbsd"))'.dependencies]
sysctl = { workspace = true }
@@ -132,14 +130,14 @@ sysctl = { workspace = true }
[target.'cfg(target_os = "linux")'.dependencies]
libsystemd.workspace = true
[target.'cfg(all(target_os = "linux", target_env = "gnu"))'.dependencies]
tikv-jemallocator = "0.6.1"
[target.'cfg(all(target_os = "linux", target_env = "musl"))'.dependencies]
mimalloc = "0.1"
[target.'cfg(not(target_os = "windows"))'.dependencies]
pprof = { version = "0.15.0", features = ["flamegraph", "protobuf-codec"] }
mimalloc = { workspace = true }
[target.'cfg(all(target_os = "linux", target_env = "gnu"))'.dependencies]
tikv-jemallocator = { workspace = true }
[target.'cfg(all(not(target_env = "msvc"), not(target_os = "windows")))'.dependencies]
tikv-jemalloc-ctl = { workspace = true }
jemalloc_pprof = { workspace = true }
pprof = { workspace = true }
[dev-dependencies]
uuid = { workspace = true, features = ["v4"] }

116
rustfs/src/admin/handlers.rs
View File

@@ -81,14 +81,13 @@ pub mod kms_dynamic;
pub mod kms_keys;
pub mod policies;
pub mod pools;
pub mod profile;
pub mod rebalance;
pub mod service_account;
pub mod sts;
pub mod tier;
pub mod trace;
pub mod user;
#[cfg(not(target_os = "windows"))]
use pprof::protos::Message;
#[allow(dead_code)]
#[derive(Debug, Serialize, Default)]
@@ -1264,14 +1263,8 @@ impl Operation for ProfileHandler {
#[cfg(not(target_os = "windows"))]
{
use crate::profiling;
if !profiling::is_profiler_enabled() {
return Ok(S3Response::new((
StatusCode::SERVICE_UNAVAILABLE,
Body::from("Profiler not enabled. Set RUSTFS_ENABLE_PROFILING=true to enable profiling".to_string()),
)));
}
use rustfs_config::{DEFAULT_CPU_FREQ, ENV_CPU_FREQ};
use rustfs_utils::get_env_usize;
let queries = extract_query_params(&req.uri);
let seconds = queries.get("seconds").and_then(|s| s.parse::<u64>().ok()).unwrap_or(30);
@@ -1284,73 +1277,56 @@ impl Operation for ProfileHandler {
)));
}
let guard = match profiling::get_profiler_guard() {
Some(guard) => guard,
None => {
return Ok(S3Response::new((
StatusCode::SERVICE_UNAVAILABLE,
Body::from("Profiler not initialized".to_string()),
)));
}
};
info!("Starting CPU profile collection for {} seconds", seconds);
tokio::time::sleep(std::time::Duration::from_secs(seconds)).await;
let guard_lock = match guard.lock() {
Ok(guard) => guard,
Err(_) => {
error!("Failed to acquire profiler guard lock");
return Ok(S3Response::new((
StatusCode::INTERNAL_SERVER_ERROR,
Body::from("Failed to acquire profiler lock".to_string()),
)));
}
};
let report = match guard_lock.report().build() {
Ok(report) => report,
Err(e) => {
error!("Failed to build profiler report: {}", e);
return Ok(S3Response::new((
StatusCode::INTERNAL_SERVER_ERROR,
Body::from(format!("Failed to build profile report: {e}")),
)));
}
};
info!("CPU profile collection completed");
match format.as_str() {
"protobuf" | "pb" => {
let profile = report.pprof().unwrap();
let mut body = Vec::new();
if let Err(e) = profile.write_to_vec(&mut body) {
error!("Failed to serialize protobuf profile: {}", e);
return Ok(S3Response::new((
"protobuf" | "pb" => match crate::profiling::dump_cpu_pprof_for(std::time::Duration::from_secs(seconds)).await {
Ok(path) => match tokio::fs::read(&path).await {
Ok(bytes) => {
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, "application/octet-stream".parse().unwrap());
Ok(S3Response::with_headers((StatusCode::OK, Body::from(bytes)), headers))
}
Err(e) => Ok(S3Response::new((
StatusCode::INTERNAL_SERVER_ERROR,
Body::from("Failed to serialize profile".to_string()),
)));
}
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, "application/octet-stream".parse().unwrap());
Ok(S3Response::with_headers((StatusCode::OK, Body::from(body)), headers))
}
Body::from(format!("Failed to read profile file: {e}")),
))),
},
Err(e) => Ok(S3Response::new((
StatusCode::INTERNAL_SERVER_ERROR,
Body::from(format!("Failed to collect CPU profile: {e}")),
))),
},
"flamegraph" | "svg" => {
let mut flamegraph_buf = Vec::new();
match report.flamegraph(&mut flamegraph_buf) {
Ok(()) => (),
let freq = get_env_usize(ENV_CPU_FREQ, DEFAULT_CPU_FREQ) as i32;
let guard = match pprof::ProfilerGuard::new(freq) {
Ok(g) => g,
Err(e) => {
error!("Failed to generate flamegraph: {}", e);
return Ok(S3Response::new((
StatusCode::INTERNAL_SERVER_ERROR,
Body::from(format!("Failed to generate flamegraph: {e}")),
Body::from(format!("Failed to create profiler: {e}")),
)));
}
};
tokio::time::sleep(std::time::Duration::from_secs(seconds)).await;
let report = match guard.report().build() {
Ok(r) => r,
Err(e) => {
return Ok(S3Response::new((
StatusCode::INTERNAL_SERVER_ERROR,
Body::from(format!("Failed to build profile report: {e}")),
)));
}
};
let mut flamegraph_buf = Vec::new();
if let Err(e) = report.flamegraph(&mut flamegraph_buf) {
return Ok(S3Response::new((
StatusCode::INTERNAL_SERVER_ERROR,
Body::from(format!("Failed to generate flamegraph: {e}")),
)));
}
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, "image/svg+xml".parse().unwrap());
Ok(S3Response::with_headers((StatusCode::OK, Body::from(flamegraph_buf)), headers))
@@ -1380,9 +1356,11 @@ impl Operation for ProfileStatusHandler {
#[cfg(not(target_os = "windows"))]
let status = {
use crate::profiling;
use rustfs_config::{DEFAULT_ENABLE_PROFILING, ENV_ENABLE_PROFILING};
use rustfs_utils::get_env_bool;
if profiling::is_profiler_enabled() {
let enabled = get_env_bool(ENV_ENABLE_PROFILING, DEFAULT_ENABLE_PROFILING);
if enabled {
HashMap::from([
("enabled", "true"),
("status", "running"),

79
rustfs/src/admin/handlers/profile.rs Normal file
View File

@@ -0,0 +1,79 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::admin::router::Operation;
use http::header::CONTENT_TYPE;
use http::{HeaderMap, StatusCode};
use matchit::Params;
use s3s::{Body, S3Request, S3Response, S3Result, s3_error};
use tracing::info;
pub struct TriggerProfileCPU {}
#[async_trait::async_trait]
impl Operation for TriggerProfileCPU {
async fn call(&self, _req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
info!("Triggering CPU profile dump via S3 request...");
#[cfg(target_os = "windows")]
{
let mut header = HeaderMap::new();
header.insert(CONTENT_TYPE, "text/plain".parse().unwrap());
return Ok(S3Response::with_headers(
(StatusCode::NOT_IMPLEMENTED, Body::from("CPU profiling is not supported on Windows")),
header,
));
}
#[cfg(not(target_os = "windows"))]
{
let dur = std::time::Duration::from_secs(60);
match crate::profiling::dump_cpu_pprof_for(dur).await {
Ok(path) => {
let mut header = HeaderMap::new();
header.insert(CONTENT_TYPE, "text/html".parse().unwrap());
Ok(S3Response::with_headers((StatusCode::OK, Body::from(path.display().to_string())), header))
}
Err(e) => Err(s3_error!(InternalError, "{}", format!("Failed to dump CPU profile: {e}"))),
}
}
}
}
pub struct TriggerProfileMemory {}
#[async_trait::async_trait]
impl Operation for TriggerProfileMemory {
async fn call(&self, _req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
info!("Triggering Memory profile dump via S3 request...");
#[cfg(target_os = "windows")]
{
let mut header = HeaderMap::new();
header.insert(CONTENT_TYPE, "text/plain".parse().unwrap());
return Ok(S3Response::with_headers(
(StatusCode::NOT_IMPLEMENTED, Body::from("Memory profiling is not supported on Windows")),
header,
));
}
#[cfg(not(target_os = "windows"))]
{
match crate::profiling::dump_memory_pprof_now().await {
Ok(path) => {
let mut header = HeaderMap::new();
header.insert(CONTENT_TYPE, "text/html".parse().unwrap());
Ok(S3Response::with_headers((StatusCode::OK, Body::from(path.display().to_string())), header))
}
Err(e) => Err(s3_error!(InternalError, "{}", format!("Failed to dump Memory profile: {e}"))),
}
}
}
}

10
rustfs/src/admin/mod.rs
View File

@@ -22,12 +22,13 @@ pub mod utils;
#[cfg(test)]
mod console_test;
// use ecstore::global::{is_dist_erasure, is_erasure};
use handlers::{
GetReplicationMetricsHandler, HealthCheckHandler, ListRemoteTargetHandler, RemoveRemoteTargetHandler, SetRemoteTargetHandler,
bucket_meta,
event::{ListNotificationTargets, ListTargetsArns, NotificationTarget, RemoveNotificationTarget},
group, kms, kms_dynamic, kms_keys, policies, pools, rebalance,
group, kms, kms_dynamic, kms_keys, policies, pools,
profile::{TriggerProfileCPU, TriggerProfileMemory},
rebalance,
service_account::{AddServiceAccount, DeleteServiceAccount, InfoServiceAccount, ListServiceAccount, UpdateServiceAccount},
sts, tier, user,
};
@@ -44,6 +45,8 @@ pub fn make_admin_route(console_enabled: bool) -> std::io::Result<impl S3Route>
// Health check endpoint for monitoring and orchestration
r.insert(Method::GET, "/health", AdminOperation(&HealthCheckHandler {}))?;
r.insert(Method::GET, "/profile/cpu", AdminOperation(&TriggerProfileCPU {}))?;
r.insert(Method::GET, "/profile/memory", AdminOperation(&TriggerProfileMemory {}))?;
// 1
r.insert(Method::POST, "/", AdminOperation(&sts::AssumeRoleHandle {}))?;
@@ -515,7 +518,8 @@ fn register_user_route(r: &mut S3Router<AdminOperation>) -> std::io::Result<()>
format!("{}{}", ADMIN_PREFIX, "/v3/target/{target_type}/{target_name}/reset").as_str(),
AdminOperation(&RemoveNotificationTarget {}),
)?;
// arns
// arns list
r.insert(
Method::GET,
format!("{}{}", ADMIN_PREFIX, "/v3/target/arns").as_str(),

21
rustfs/src/admin/router.rs
View File

@@ -12,6 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::admin::ADMIN_PREFIX;
use crate::admin::console::is_console_path;
use crate::admin::console::make_console_server;
use crate::admin::rpc::RPC_PREFIX;
use hyper::HeaderMap;
use hyper::Method;
use hyper::StatusCode;
@@ -30,11 +34,6 @@ use s3s::s3_error;
use tower::Service;
use tracing::error;
use crate::admin::ADMIN_PREFIX;
use crate::admin::console::is_console_path;
use crate::admin::console::make_console_server;
use crate::admin::rpc::RPC_PREFIX;
pub struct S3Router<T> {
router: Router<T>,
console_enabled: bool,
@@ -85,12 +84,13 @@ where
T: Operation,
{
fn is_match(&self, method: &Method, uri: &Uri, headers: &HeaderMap, _: &mut Extensions) -> bool {
if method == Method::GET && uri.path() == "/health" {
let path = uri.path();
if method == Method::GET && (path == "/health" || path == "/profile/cpu" || path == "/profile/memory") {
return true;
}
// AssumeRole
if method == Method::POST && uri.path() == "/" {
if method == Method::POST && path == "/" {
if let Some(val) = headers.get(header::CONTENT_TYPE) {
if val.as_bytes() == b"application/x-www-form-urlencoded" {
return true;
@@ -98,17 +98,18 @@ where
}
}
uri.path().starts_with(ADMIN_PREFIX) || uri.path().starts_with(RPC_PREFIX) || is_console_path(uri.path())
path.starts_with(ADMIN_PREFIX) || path.starts_with(RPC_PREFIX) || is_console_path(path)
}
// check_access before call
async fn check_access(&self, req: &mut S3Request<Body>) -> S3Result<()> {
// Allow unauthenticated access to health check
if req.method == Method::GET && req.uri.path() == "/health" {
let path = req.uri.path();
if req.method == Method::GET && (path == "/health" || path == "/profile/cpu" || path == "/profile/memory") {
return Ok(());
}
// Allow unauthenticated access to console static files if console is enabled
if self.console_enabled && is_console_path(req.uri.path()) {
if self.console_enabled && is_console_path(path) {
return Ok(());
}

8
rustfs/src/main.rs
View File

@@ -25,6 +25,7 @@ mod storage;
mod update;
mod version;
// Ensure the correct path for parse_license is imported
use crate::server::{
SHUTDOWN_TIMEOUT, ServiceState, ServiceStateManager, ShutdownSignal, init_event_notifier, shutdown_event_notifier,
start_audit_system, start_http_server, stop_audit_system, wait_for_shutdown,
@@ -62,6 +63,7 @@ use rustfs_obs::{init_obs, set_global_guard};
use rustfs_targets::arn::TargetID;
use rustfs_utils::net::parse_and_resolve_address;
use s3s::s3_error;
use std::env;
use std::io::{Error, Result};
use std::str::FromStr;
use std::sync::Arc;
@@ -126,7 +128,7 @@ async fn async_main() -> Result<()> {
// Initialize performance profiling if enabled
#[cfg(not(target_os = "windows"))]
profiling::start_profiling_if_enabled();
profiling::init_from_env().await;
// Run parameters
match run(opt).await {
@@ -349,7 +351,7 @@ async fn run(opt: config::Opt) -> Result<()> {
/// Returns true if the environment variable is not set or set to true/1/yes/on/enabled,
/// false if set to false/0/no/off/disabled
fn parse_bool_env_var(var_name: &str, default: bool) -> bool {
std::env::var(var_name)
env::var(var_name)
.unwrap_or_else(|_| default.to_string())
.parse::<bool>()
.unwrap_or(default)
@@ -436,7 +438,7 @@ async fn handle_shutdown(
}
fn init_update_check() {
let update_check_enable = std::env::var(ENV_UPDATE_CHECK)
let update_check_enable = env::var(ENV_UPDATE_CHECK)
.unwrap_or_else(|_| DEFAULT_UPDATE_CHECK.to_string())
.parse::<bool>()
.unwrap_or(DEFAULT_UPDATE_CHECK);

292
rustfs/src/profiling.rs
View File

@@ -12,52 +12,272 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use pprof::ProfilerGuard;
use std::sync::{Arc, Mutex, OnceLock};
use tracing::info;
use chrono::Utc;
use jemalloc_pprof::PROF_CTL;
use pprof::protos::Message;
use rustfs_config::{
DEFAULT_CPU_DURATION_SECS, DEFAULT_CPU_FREQ, DEFAULT_CPU_INTERVAL_SECS, DEFAULT_CPU_MODE, DEFAULT_ENABLE_PROFILING,
DEFAULT_MEM_INTERVAL_SECS, DEFAULT_MEM_PERIODIC, DEFAULT_OUTPUT_DIR, ENV_CPU_DURATION_SECS, ENV_CPU_FREQ,
ENV_CPU_INTERVAL_SECS, ENV_CPU_MODE, ENV_ENABLE_PROFILING, ENV_MEM_INTERVAL_SECS, ENV_MEM_PERIODIC, ENV_OUTPUT_DIR,
};
use rustfs_utils::{get_env_bool, get_env_str, get_env_u64, get_env_usize};
use std::fs::{File, create_dir_all};
use std::io::Write;
use std::path::{Path, PathBuf};
use std::sync::{Arc, OnceLock};
use std::time::Duration;
use tokio::sync::Mutex;
use tokio::time::sleep;
use tracing::{debug, error, info, warn};
static PROFILER_GUARD: OnceLock<Arc<Mutex<ProfilerGuard<'static>>>> = OnceLock::new();
static CPU_CONT_GUARD: OnceLock<Arc<Mutex<Option<pprof::ProfilerGuard<'static>>>>> = OnceLock::new();
pub fn init_profiler() -> Result<(), Box<dyn std::error::Error>> {
let guard = pprof::ProfilerGuardBuilder::default()
.frequency(1000)
.blocklist(&["libc", "libgcc", "pthread", "vdso"])
.build()
.map_err(|e| format!("Failed to build profiler guard: {e}"))?;
/// CPU profiling mode
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum CpuMode {
Off,
Continuous,
Periodic,
}
PROFILER_GUARD
.set(Arc::new(Mutex::new(guard)))
.map_err(|_| "Failed to set profiler guard (already initialized)")?;
/// Get or create output directory
fn output_dir() -> PathBuf {
let dir = get_env_str(ENV_OUTPUT_DIR, DEFAULT_OUTPUT_DIR);
let p = PathBuf::from(dir);
if let Err(e) = create_dir_all(&p) {
warn!("profiling: create output dir {} failed: {}, fallback to current dir", p.display(), e);
return PathBuf::from(".");
}
p
}
info!("Performance profiler initialized");
/// Read CPU profiling mode from env
fn read_cpu_mode() -> CpuMode {
match get_env_str(ENV_CPU_MODE, DEFAULT_CPU_MODE).to_lowercase().as_str() {
"continuous" => CpuMode::Continuous,
"periodic" => CpuMode::Periodic,
_ => CpuMode::Off,
}
}
/// Generate timestamp string for filenames
fn ts() -> String {
Utc::now().format("%Y%m%dT%H%M%S").to_string()
}
/// Write pprof report to file in protobuf format
fn write_pprof_report_pb(report: &pprof::Report, path: &Path) -> Result<(), String> {
let profile = report.pprof().map_err(|e| format!("pprof() failed: {e}"))?;
let mut buf = Vec::with_capacity(512 * 1024);
profile.write_to_vec(&mut buf).map_err(|e| format!("encode failed: {e}"))?;
let mut f = File::create(path).map_err(|e| format!("create file failed: {e}"))?;
f.write_all(&buf).map_err(|e| format!("write file failed: {e}"))?;
Ok(())
}
pub fn is_profiler_enabled() -> bool {
PROFILER_GUARD.get().is_some()
/// Internal: dump CPU pprof from existing guard
async fn dump_cpu_with_guard(guard: &pprof::ProfilerGuard<'_>) -> Result<PathBuf, String> {
let report = guard.report().build().map_err(|e| format!("build report failed: {e}"))?;
let out = output_dir().join(format!("cpu_profile_{}.pb", ts()));
write_pprof_report_pb(&report, &out)?;
info!("CPU profile exported: {}", out.display());
Ok(out)
}
pub fn get_profiler_guard() -> Option<Arc<Mutex<ProfilerGuard<'static>>>> {
PROFILER_GUARD.get().cloned()
}
pub fn start_profiling_if_enabled() {
let enable_profiling = std::env::var("RUSTFS_ENABLE_PROFILING")
.unwrap_or_else(|_| "false".to_string())
.parse::<bool>()
.unwrap_or(false);
if enable_profiling {
match init_profiler() {
Ok(()) => {
info!("Performance profiling enabled via RUSTFS_ENABLE_PROFILING environment variable");
}
Err(e) => {
tracing::error!("Failed to initialize profiler: {}", e);
info!("Performance profiling disabled due to initialization error");
}
// Public API: dump CPU for a duration; if continuous guard exists, snapshot immediately.
pub async fn dump_cpu_pprof_for(duration: Duration) -> Result<PathBuf, String> {
if let Some(cell) = CPU_CONT_GUARD.get() {
let guard_slot = cell.lock().await;
if let Some(ref guard) = *guard_slot {
debug!("profiling: using continuous profiler guard for CPU dump");
return dump_cpu_with_guard(guard).await;
}
}
let freq = get_env_usize(ENV_CPU_FREQ, DEFAULT_CPU_FREQ) as i32;
let guard = pprof::ProfilerGuard::new(freq).map_err(|e| format!("create profiler failed: {e}"))?;
sleep(duration).await;
dump_cpu_with_guard(&guard).await
}
// Public API: dump memory pprof now (jemalloc)
pub async fn dump_memory_pprof_now() -> Result<PathBuf, String> {
let out = output_dir().join(format!("mem_profile_{}.pb", ts()));
let mut f = File::create(&out).map_err(|e| format!("create file failed: {e}"))?;
let prof_ctl_cell = PROF_CTL
.as_ref()
.ok_or_else(|| "jemalloc profiling control not available".to_string())?;
let mut prof_ctl = prof_ctl_cell.lock().await;
if !prof_ctl.activated() {
return Err("jemalloc profiling is not active".to_string());
}
let bytes = prof_ctl.dump_pprof().map_err(|e| format!("dump pprof failed: {e}"))?;
f.write_all(&bytes).map_err(|e| format!("write file failed: {e}"))?;
info!("Memory profile exported: {}", out.display());
Ok(out)
}
// Jemalloc status check (No forced placement, only status observation)
pub async fn check_jemalloc_profiling() {
use tikv_jemalloc_ctl::{config, epoch, stats};
if let Err(e) = epoch::advance() {
warn!("jemalloc epoch advance failed: {e}");
}
match config::malloc_conf::read() {
Ok(conf) => debug!("jemalloc malloc_conf: {}", conf),
Err(e) => debug!("jemalloc read malloc_conf failed: {e}"),
}
match std::env::var("MALLOC_CONF") {
Ok(v) => debug!("MALLOC_CONF={}", v),
Err(_) => debug!("MALLOC_CONF is not set"),
}
if let Some(lock) = PROF_CTL.as_ref() {
let ctl = lock.lock().await;
info!(activated = ctl.activated(), "jemalloc profiling status");
} else {
info!("Performance profiling disabled. Set RUSTFS_ENABLE_PROFILING=true to enable");
info!("jemalloc profiling controller is NOT available");
}
let _ = epoch::advance();
macro_rules! show {
($name:literal, $reader:expr) => {
match $reader {
Ok(v) => debug!(concat!($name, "={}"), v),
Err(e) => debug!(concat!($name, " read failed: {}"), e),
}
};
}
show!("allocated", stats::allocated::read());
show!("resident", stats::resident::read());
show!("mapped", stats::mapped::read());
show!("metadata", stats::metadata::read());
show!("active", stats::active::read());
}
// Internal: start continuous CPU profiling
async fn start_cpu_continuous(freq_hz: i32) {
let cell = CPU_CONT_GUARD.get_or_init(|| Arc::new(Mutex::new(None))).clone();
let mut slot = cell.lock().await;
if slot.is_some() {
warn!("profiling: continuous CPU guard already running");
return;
}
match pprof::ProfilerGuardBuilder::default()
.frequency(freq_hz)
.blocklist(&["libc", "libgcc", "pthread", "vdso"])
.build()
{
Ok(guard) => {
*slot = Some(guard);
info!(freq = freq_hz, "start continuous CPU profiling");
}
Err(e) => warn!("start continuous CPU profiling failed: {e}"),
}
}
// Internal: start periodic CPU sampling loop
async fn start_cpu_periodic(freq_hz: i32, interval: Duration, duration: Duration) {
info!(freq = freq_hz, ?interval, ?duration, "start periodic CPU profiling");
tokio::spawn(async move {
loop {
sleep(interval).await;
let guard = match pprof::ProfilerGuard::new(freq_hz) {
Ok(g) => g,
Err(e) => {
warn!("periodic CPU profiler create failed: {e}");
continue;
}
};
sleep(duration).await;
match guard.report().build() {
Ok(report) => {
let out = output_dir().join(format!("cpu_profile_{}.pb", ts()));
if let Err(e) = write_pprof_report_pb(&report, &out) {
warn!("write periodic CPU pprof failed: {e}");
} else {
info!("periodic CPU profile exported: {}", out.display());
}
}
Err(e) => warn!("periodic CPU report build failed: {e}"),
}
}
});
}
// Internal: start periodic memory dump when jemalloc profiling is active
async fn start_memory_periodic(interval: Duration) {
info!(?interval, "start periodic memory pprof dump");
tokio::spawn(async move {
loop {
sleep(interval).await;
let Some(lock) = PROF_CTL.as_ref() else {
debug!("skip memory dump: PROF_CTL not available");
continue;
};
let mut ctl = lock.lock().await;
if !ctl.activated() {
debug!("skip memory dump: jemalloc profiling not active");
continue;
}
let out = output_dir().join(format!("mem_profile_periodic_{}.pb", ts()));
match File::create(&out) {
Err(e) => {
error!("periodic mem dump create file failed: {}", e);
continue;
}
Ok(mut f) => match ctl.dump_pprof() {
Ok(bytes) => {
if let Err(e) = f.write_all(&bytes) {
error!("periodic mem dump write failed: {}", e);
} else {
info!("periodic memory profile dumped to {}", out.display());
}
}
Err(e) => error!("periodic mem dump failed: {}", e),
},
}
}
});
}
// Public: unified init entry, avoid duplication/conflict
pub async fn init_from_env() {
let enabled = get_env_bool(ENV_ENABLE_PROFILING, DEFAULT_ENABLE_PROFILING);
if !enabled {
debug!("profiling: disabled by env");
return;
}
// Jemalloc state check once (no dump)
check_jemalloc_profiling().await;
// CPU
let cpu_mode = read_cpu_mode();
let cpu_freq = get_env_usize(ENV_CPU_FREQ, DEFAULT_CPU_FREQ) as i32;
let cpu_interval = Duration::from_secs(get_env_u64(ENV_CPU_INTERVAL_SECS, DEFAULT_CPU_INTERVAL_SECS));
let cpu_duration = Duration::from_secs(get_env_u64(ENV_CPU_DURATION_SECS, DEFAULT_CPU_DURATION_SECS));
match cpu_mode {
CpuMode::Off => debug!("profiling: CPU mode off"),
CpuMode::Continuous => start_cpu_continuous(cpu_freq).await,
CpuMode::Periodic => start_cpu_periodic(cpu_freq, cpu_interval, cpu_duration).await,
}
// Memory
let mem_periodic = get_env_bool(ENV_MEM_PERIODIC, DEFAULT_MEM_PERIODIC);
let mem_interval = Duration::from_secs(get_env_u64(ENV_MEM_INTERVAL_SECS, DEFAULT_MEM_INTERVAL_SECS));
if mem_periodic {
start_memory_periodic(mem_interval).await;
}
}

24
scripts/run.sh
View File

@@ -116,7 +116,29 @@ if [ -n "$1" ]; then
export RUSTFS_VOLUMES="$1"
fi
# Enable jemalloc for memory profiling
# MALLOC_CONF parameters:
# prof:true - Enable heap profiling
# prof_active:true - Start profiling immediately
# lg_prof_sample:16 - Average number of bytes between samples (2^16 = 65536 bytes)
# log:true - Enable logging
# narenas:2 - Number of arenas (controls concurrency and memory fragmentation)
# lg_chunk:21 - Chunk size (2^21 = 2MB)
# background_thread:true - Enable background threads for purging
# dirty_decay_ms:1000 - Time (ms) before dirty pages are purged
# muzzy_decay_ms:1000 - Time (ms) before muzzy pages are purged
# You can override these defaults by setting the MALLOC_CONF environment variable before running this script.
if [ -z "$MALLOC_CONF" ]; then
export MALLOC_CONF="prof:true,prof_active:true,lg_prof_sample:16,log:true,narenas:2,lg_chunk:21,background_thread:true,dirty_decay_ms:1000,muzzy_decay_ms:1000"
fi
# Start webhook server
#cargo run --example webhook -p rustfs-notify &
# Start main service
cargo run --bin rustfs
# To run with profiling enabled, uncomment the following line and comment the next line
#cargo run --profile profiling --bin rustfs
# To run in release mode, use the following line
cargo run --profile release --bin rustfs
# To run in debug mode, use the following line
#cargo run --bin rustfs