diff --git a/.cursorrules b/.cursorrules
index 83a208f5..74c37ad3 100644
--- a/.cursorrules
+++ b/.cursorrules
@@ -5,15 +5,18 @@
### π¨ NEVER COMMIT DIRECTLY TO MASTER/MAIN BRANCH π¨
- **This is the most important rule - NEVER modify code directly on main or master branch**
+- **ALL CHANGES MUST GO THROUGH PULL REQUESTS - NO EXCEPTIONS**
- **Always work on feature branches and use pull requests for all changes**
- **Any direct commits to master/main branch are strictly forbidden**
+- **Pull requests are the ONLY way to merge code to main branch**
- Before starting any development, always:
1. `git checkout main` (switch to main branch)
2. `git pull` (get latest changes)
3. `git checkout -b feat/your-feature-name` (create and switch to feature branch)
4. Make your changes on the feature branch
5. Commit and push to the feature branch
- 6. Create a pull request for review
+ 6. **Create a pull request for review - THIS IS MANDATORY**
+ 7. **Wait for PR approval and merge through GitHub interface only**
## Project Overview
@@ -817,6 +820,7 @@ These rules should serve as guiding principles when developing the RustFS projec
- **π¨ CRITICAL: NEVER modify code directly on main or master branch - THIS IS ABSOLUTELY FORBIDDEN π¨**
- **β οΈ ANY DIRECT COMMITS TO MASTER/MAIN WILL BE REJECTED AND MUST BE REVERTED IMMEDIATELY β οΈ**
+- **π ALL CHANGES MUST GO THROUGH PULL REQUESTS - NO DIRECT COMMITS TO MAIN UNDER ANY CIRCUMSTANCES π**
- **Always work on feature branches - NO EXCEPTIONS**
- Always check the .cursorrules file before starting to ensure you understand the project guidelines
- **MANDATORY workflow for ALL changes:**
@@ -826,10 +830,20 @@ These rules should serve as guiding principles when developing the RustFS projec
4. Make your changes ONLY on the feature branch
5. Test thoroughly before committing
6. Commit and push to the feature branch
- 7. Create a pull request for code review
+ 7. **Create a pull request for code review - THIS IS THE ONLY WAY TO MERGE TO MAIN**
+ 8. **Wait for PR approval before merging - NEVER merge your own PRs without review**
- Use descriptive branch names following the pattern: `feat/feature-name`, `fix/issue-name`, `refactor/component-name`, etc.
- **Double-check current branch before ANY commit: `git branch` to ensure you're NOT on main/master**
-- Ensure all changes are made on feature branches and merged through pull requests
+- **Pull Request Requirements:**
+ - All changes must be submitted via PR regardless of size or urgency
+ - PRs must include comprehensive description and testing information
+ - PRs must pass all CI/CD checks before merging
+ - PRs require at least one approval from code reviewers
+ - Even hotfixes and emergency changes must go through PR process
+- **Enforcement:**
+ - Main branch should be protected with branch protection rules
+ - Direct pushes to main should be blocked by repository settings
+ - Any accidental direct commits to main must be immediately reverted via PR
#### Development Workflow
diff --git a/crates/appauth/README.md b/crates/appauth/README.md
index 17cb3eaa..364a91e2 100644
--- a/crates/appauth/README.md
+++ b/crates/appauth/README.md
@@ -3,7 +3,7 @@
# RustFS AppAuth - Application Authentication
- Secure application authentication and authorization for RustFS object storage
+ Application-level authentication and authorization module for RustFS distributed object storage
@@ -17,461 +17,21 @@
## π Overview
-**RustFS AppAuth** provides secure application authentication and authorization mechanisms for the [RustFS](https://rustfs.com) distributed object storage system. It implements modern cryptographic standards including RSA-based authentication, JWT tokens, and secure session management for application-level access control.
-
-> **Note:** This is a security-critical submodule of RustFS that provides essential application authentication capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS AppAuth** provides application-level authentication and authorization capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π Authentication Methods
-
-- **RSA Authentication**: Public-key cryptography for secure authentication
-- **JWT Tokens**: JSON Web Token support for stateless authentication
-- **API Keys**: Simple API key-based authentication
-- **Session Management**: Secure session handling and lifecycle management
-
-### π‘οΈ Security Features
-
-- **Cryptographic Signing**: RSA digital signatures for request validation
-- **Token Encryption**: Encrypted token storage and transmission
-- **Key Rotation**: Automatic key rotation and management
-- **Audit Logging**: Comprehensive authentication event logging
-
-### π Performance Features
-
-- **Base64 Optimization**: High-performance base64 encoding/decoding
-- **Token Caching**: Efficient token validation caching
-- **Parallel Verification**: Concurrent authentication processing
-- **Hardware Acceleration**: Leverage CPU crypto extensions
-
-### π§ Integration Features
-
-- **S3 Compatibility**: AWS S3-compatible authentication
-- **Multi-Tenant**: Support for multiple application tenants
-- **Permission Mapping**: Fine-grained permission assignment
-- **External Integration**: LDAP, OAuth, and custom authentication providers
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-appauth = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Authentication Setup
-
-```rust
-use rustfs_appauth::{AppAuthenticator, AuthConfig, AuthMethod};
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Configure authentication
- let config = AuthConfig {
- auth_method: AuthMethod::RSA,
- key_size: 2048,
- token_expiry: Duration::from_hours(24),
- enable_caching: true,
- audit_logging: true,
- };
-
- // Initialize authenticator
- let authenticator = AppAuthenticator::new(config).await?;
-
- // Generate application credentials
- let app_credentials = authenticator.generate_app_credentials("my-app").await?;
-
- println!("App ID: {}", app_credentials.app_id);
- println!("Public Key: {}", app_credentials.public_key);
-
- Ok(())
-}
-```
-
-### RSA-Based Authentication
-
-```rust
-use rustfs_appauth::{RSAAuthenticator, AuthRequest, AuthResponse};
-
-async fn rsa_authentication_example() -> Result<(), Box> {
- // Create RSA authenticator
- let rsa_auth = RSAAuthenticator::new(2048).await?;
-
- // Generate key pair for application
- let (private_key, public_key) = rsa_auth.generate_keypair().await?;
-
- // Register application
- let app_id = rsa_auth.register_application("my-storage-app", &public_key).await?;
- println!("Application registered with ID: {}", app_id);
-
- // Create authentication request
- let auth_request = AuthRequest {
- app_id: app_id.clone(),
- timestamp: chrono::Utc::now(),
- request_data: b"GET /bucket/object".to_vec(),
- };
-
- // Sign request with private key
- let signed_request = rsa_auth.sign_request(&auth_request, &private_key).await?;
-
- // Verify authentication
- let auth_response = rsa_auth.authenticate(&signed_request).await?;
-
- match auth_response {
- AuthResponse::Success { session_token, permissions } => {
- println!("Authentication successful!");
- println!("Session token: {}", session_token);
- println!("Permissions: {:?}", permissions);
- }
- AuthResponse::Failed { reason } => {
- println!("Authentication failed: {}", reason);
- }
- }
-
- Ok(())
-}
-```
-
-### JWT Token Management
-
-```rust
-use rustfs_appauth::{JWTManager, TokenClaims, TokenRequest};
-
-async fn jwt_management_example() -> Result<(), Box> {
- // Create JWT manager
- let jwt_manager = JWTManager::new("your-secret-key").await?;
-
- // Create token claims
- let claims = TokenClaims {
- app_id: "my-app".to_string(),
- user_id: Some("user123".to_string()),
- permissions: vec![
- "read:bucket".to_string(),
- "write:bucket".to_string(),
- ],
- expires_at: chrono::Utc::now() + chrono::Duration::hours(24),
- issued_at: chrono::Utc::now(),
- };
-
- // Generate JWT token
- let token = jwt_manager.generate_token(&claims).await?;
- println!("Generated token: {}", token);
-
- // Validate token
- let validation_result = jwt_manager.validate_token(&token).await?;
-
- match validation_result {
- Ok(validated_claims) => {
- println!("Token valid for app: {}", validated_claims.app_id);
- println!("Permissions: {:?}", validated_claims.permissions);
- }
- Err(e) => {
- println!("Token validation failed: {}", e);
- }
- }
-
- // Refresh token
- let refreshed_token = jwt_manager.refresh_token(&token).await?;
- println!("Refreshed token: {}", refreshed_token);
-
- Ok(())
-}
-```
-
-### API Key Authentication
-
-```rust
-use rustfs_appauth::{APIKeyManager, APIKeyConfig, KeyPermissions};
-
-async fn api_key_authentication() -> Result<(), Box> {
- let api_key_manager = APIKeyManager::new().await?;
-
- // Create API key configuration
- let key_config = APIKeyConfig {
- app_name: "storage-client".to_string(),
- permissions: KeyPermissions {
- read_buckets: vec!["public-*".to_string()],
- write_buckets: vec!["uploads".to_string()],
- admin_access: false,
- },
- expires_at: Some(chrono::Utc::now() + chrono::Duration::days(90)),
- rate_limit: Some(1000), // requests per hour
- };
-
- // Generate API key
- let api_key = api_key_manager.generate_key(&key_config).await?;
- println!("Generated API key: {}", api_key.key);
- println!("Key ID: {}", api_key.key_id);
-
- // Authenticate with API key
- let auth_result = api_key_manager.authenticate(&api_key.key).await?;
-
- if auth_result.is_valid {
- println!("API key authentication successful");
- println!("Rate limit remaining: {}", auth_result.rate_limit_remaining);
- }
-
- // List API keys for application
- let keys = api_key_manager.list_keys("storage-client").await?;
- for key in keys {
- println!("Key: {} - Status: {} - Expires: {:?}",
- key.key_id, key.status, key.expires_at);
- }
-
- // Revoke API key
- api_key_manager.revoke_key(&api_key.key_id).await?;
- println!("API key revoked successfully");
-
- Ok(())
-}
-```
-
-### Session Management
-
-```rust
-use rustfs_appauth::{SessionManager, SessionConfig, SessionInfo};
-
-async fn session_management_example() -> Result<(), Box> {
- // Configure session management
- let session_config = SessionConfig {
- session_timeout: Duration::from_hours(8),
- max_sessions_per_app: 10,
- require_refresh: true,
- secure_cookies: true,
- };
-
- let session_manager = SessionManager::new(session_config).await?;
-
- // Create new session
- let session_info = SessionInfo {
- app_id: "web-app".to_string(),
- user_id: Some("user456".to_string()),
- ip_address: "192.168.1.100".to_string(),
- user_agent: "RustFS-Client/1.0".to_string(),
- };
-
- let session = session_manager.create_session(&session_info).await?;
- println!("Session created: {}", session.session_id);
-
- // Validate session
- let validation = session_manager.validate_session(&session.session_id).await?;
-
- if validation.is_valid {
- println!("Session is valid, expires at: {}", validation.expires_at);
- }
-
- // Refresh session
- session_manager.refresh_session(&session.session_id).await?;
- println!("Session refreshed");
-
- // Get active sessions
- let active_sessions = session_manager.get_active_sessions("web-app").await?;
- println!("Active sessions: {}", active_sessions.len());
-
- // Terminate session
- session_manager.terminate_session(&session.session_id).await?;
- println!("Session terminated");
-
- Ok(())
-}
-```
-
-### Multi-Tenant Authentication
-
-```rust
-use rustfs_appauth::{MultiTenantAuth, TenantConfig, TenantPermissions};
-
-async fn multi_tenant_auth_example() -> Result<(), Box> {
- let multi_tenant_auth = MultiTenantAuth::new().await?;
-
- // Create tenant configurations
- let tenant1_config = TenantConfig {
- tenant_id: "company-a".to_string(),
- name: "Company A".to_string(),
- permissions: TenantPermissions {
- max_buckets: 100,
- max_storage_gb: 1000,
- allowed_regions: vec!["us-east-1".to_string(), "us-west-2".to_string()],
- },
- auth_methods: vec![AuthMethod::RSA, AuthMethod::JWT],
- };
-
- let tenant2_config = TenantConfig {
- tenant_id: "company-b".to_string(),
- name: "Company B".to_string(),
- permissions: TenantPermissions {
- max_buckets: 50,
- max_storage_gb: 500,
- allowed_regions: vec!["eu-west-1".to_string()],
- },
- auth_methods: vec![AuthMethod::APIKey],
- };
-
- // Register tenants
- multi_tenant_auth.register_tenant(&tenant1_config).await?;
- multi_tenant_auth.register_tenant(&tenant2_config).await?;
-
- // Authenticate application for specific tenant
- let auth_request = TenantAuthRequest {
- tenant_id: "company-a".to_string(),
- app_id: "app-1".to_string(),
- credentials: AuthCredentials::RSA {
- signature: "signed-data".to_string(),
- public_key: "public-key-data".to_string(),
- },
- };
-
- let auth_result = multi_tenant_auth.authenticate(&auth_request).await?;
-
- if auth_result.is_authenticated {
- println!("Multi-tenant authentication successful");
- println!("Tenant: {}", auth_result.tenant_id);
- println!("Permissions: {:?}", auth_result.permissions);
- }
-
- Ok(())
-}
-```
-
-### Authentication Middleware
-
-```rust
-use rustfs_appauth::{AuthMiddleware, AuthContext, MiddlewareConfig};
-use axum::{Router, middleware, Extension};
-
-async fn setup_auth_middleware() -> Result> {
- // Configure authentication middleware
- let middleware_config = MiddlewareConfig {
- skip_paths: vec!["/health".to_string(), "/metrics".to_string()],
- require_auth: true,
- audit_requests: true,
- };
-
- let auth_middleware = AuthMiddleware::new(middleware_config).await?;
-
- // Create router with authentication middleware
- let app = Router::new()
- .route("/api/buckets", axum::routing::get(list_buckets))
- .route("/api/objects", axum::routing::post(upload_object))
- .layer(middleware::from_fn(auth_middleware.authenticate))
- .layer(Extension(auth_middleware));
-
- Ok(app)
-}
-
-async fn list_buckets(
- Extension(auth_context): Extension,
-) -> Result> {
- // Use authentication context
- println!("Authenticated app: {}", auth_context.app_id);
- println!("Permissions: {:?}", auth_context.permissions);
-
- // Your bucket listing logic here
- Ok("Bucket list".to_string())
-}
-```
-
-## ποΈ Architecture
-
-### AppAuth Architecture
-
-```
-AppAuth Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Authentication API β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β RSA Auth β JWT Tokens β API Keys β Sessions β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Cryptographic Operations β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Signing/ β Token β Key β Session β
-β Verification β Management β Management β Storage β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Security Infrastructure β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Authentication Methods
-
-| Method | Security Level | Use Case | Performance |
-|--------|----------------|----------|-------------|
-| RSA | High | Enterprise applications | Medium |
-| JWT | Medium-High | Web applications | High |
-| API Key | Medium | Service-to-service | Very High |
-| Session | Medium | Interactive applications | High |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test RSA authentication
-cargo test rsa_auth
-
-# Test JWT tokens
-cargo test jwt_tokens
-
-# Test API key management
-cargo test api_keys
-
-# Test session management
-cargo test sessions
-
-# Integration tests
-cargo test --test integration
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Dependencies**: RSA cryptographic libraries
-- **Security**: Secure key storage recommended
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS IAM](../iam) - Identity and access management
-- [RustFS Signer](../signer) - Request signing
-- [RustFS Crypto](../crypto) - Cryptographic operations
+- JWT-based authentication with secure token management
+- RBAC (Role-Based Access Control) for fine-grained permissions
+- Multi-tenant application isolation and management
+- OAuth 2.0 and OpenID Connect integration
+- API key management and rotation
+- Session management with configurable expiration
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [AppAuth API Reference](https://docs.rustfs.com/appauth/)
-- [Security Guide](https://docs.rustfs.com/security/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/common/README.md b/crates/common/README.md
index 51477195..2a15767b 100644
--- a/crates/common/README.md
+++ b/crates/common/README.md
@@ -3,7 +3,7 @@
# RustFS Common - Shared Components
- Common types, utilities, and shared components for RustFS distributed object storage
+ Shared components and common utilities module for RustFS distributed object storage
@@ -17,279 +17,21 @@
## π Overview
-**RustFS Common** provides shared components, types, and utilities used across all RustFS modules. This foundational library ensures consistency, reduces code duplication, and provides essential building blocks for the [RustFS](https://rustfs.com) distributed object storage system.
-
-> **Note:** This is a foundational submodule of RustFS that provides essential shared components for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Common** provides shared components and common utilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π§ Core Types
-
-- **Common Data Structures**: Shared types and enums
-- **Error Handling**: Unified error types and utilities
-- **Result Types**: Consistent result handling patterns
-- **Constants**: System-wide constants and defaults
-
-### π οΈ Utilities
-
-- **Async Helpers**: Common async patterns and utilities
-- **Serialization**: Shared serialization utilities
-- **Logging**: Common logging and tracing setup
-- **Metrics**: Shared metrics and observability
-
-### π Network Components
-
-- **gRPC Common**: Shared gRPC types and utilities
-- **Protocol Helpers**: Common protocol implementations
-- **Connection Management**: Shared connection utilities
-- **Request/Response Types**: Common API types
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-common = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Common Types
-
-```rust
-use rustfs_common::{Result, Error, ObjectInfo, BucketInfo};
-
-fn main() -> Result<()> {
- // Use common result type
- let result = some_operation()?;
-
- // Use common object info
- let object = ObjectInfo {
- name: "example.txt".to_string(),
- size: 1024,
- etag: "d41d8cd98f00b204e9800998ecf8427e".to_string(),
- last_modified: chrono::Utc::now(),
- content_type: "text/plain".to_string(),
- };
-
- println!("Object: {} ({} bytes)", object.name, object.size);
- Ok(())
-}
-```
-
-### Error Handling
-
-```rust
-use rustfs_common::{Error, ErrorKind, Result};
-
-fn example_operation() -> Result {
- // Different error types
- match some_condition {
- true => Ok("Success".to_string()),
- false => Err(Error::new(
- ErrorKind::InvalidInput,
- "Invalid operation parameters"
- )),
- }
-}
-
-fn handle_errors() {
- match example_operation() {
- Ok(value) => println!("Success: {}", value),
- Err(e) => {
- match e.kind() {
- ErrorKind::InvalidInput => println!("Input error: {}", e),
- ErrorKind::NotFound => println!("Not found: {}", e),
- ErrorKind::PermissionDenied => println!("Access denied: {}", e),
- _ => println!("Other error: {}", e),
- }
- }
- }
-}
-```
-
-### Async Utilities
-
-```rust
-use rustfs_common::async_utils::{timeout_with_default, retry_with_backoff, spawn_task};
-use std::time::Duration;
-
-async fn async_operations() -> Result<()> {
- // Timeout with default value
- let result = timeout_with_default(
- Duration::from_secs(5),
- expensive_operation(),
- "default_value".to_string()
- ).await;
-
- // Retry with exponential backoff
- let result = retry_with_backoff(
- 3, // max attempts
- Duration::from_millis(100), // initial delay
- || async { fallible_operation().await }
- ).await?;
-
- // Spawn background task
- spawn_task("background-worker", async {
- background_work().await;
- });
-
- Ok(())
-}
-```
-
-### Metrics and Observability
-
-```rust
-use rustfs_common::metrics::{Counter, Histogram, Gauge, MetricsRegistry};
-
-fn setup_metrics() -> Result<()> {
- let registry = MetricsRegistry::new();
-
- // Create metrics
- let requests_total = Counter::new("requests_total", "Total number of requests")?;
- let request_duration = Histogram::new(
- "request_duration_seconds",
- "Request duration in seconds"
- )?;
- let active_connections = Gauge::new(
- "active_connections",
- "Number of active connections"
- )?;
-
- // Register metrics
- registry.register(Box::new(requests_total))?;
- registry.register(Box::new(request_duration))?;
- registry.register(Box::new(active_connections))?;
-
- Ok(())
-}
-```
-
-### gRPC Common Types
-
-```rust
-use rustfs_common::grpc::{GrpcResult, GrpcError, TonicStatus};
-use tonic::{Request, Response, Status};
-
-async fn grpc_service_example(
- request: Request
-) -> GrpcResult {
- let req = request.into_inner();
-
- // Validate request
- if req.name.is_empty() {
- return Err(GrpcError::invalid_argument("Name cannot be empty"));
- }
-
- // Process request
- let response = MyResponse {
- result: format!("Processed: {}", req.name),
- status: "success".to_string(),
- };
-
- Ok(Response::new(response))
-}
-
-// Error conversion
-impl From for Status {
- fn from(err: Error) -> Self {
- match err.kind() {
- ErrorKind::NotFound => Status::not_found(err.to_string()),
- ErrorKind::PermissionDenied => Status::permission_denied(err.to_string()),
- ErrorKind::InvalidInput => Status::invalid_argument(err.to_string()),
- _ => Status::internal(err.to_string()),
- }
- }
-}
-```
-
-## ποΈ Architecture
-
-### Common Module Structure
-
-```
-Common Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Public API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Core Types β Error Types β Result Types β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Async Utils β Metrics β gRPC Common β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Constants β Serialization β Logging β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Foundation Types β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Core Components
-
-| Component | Purpose | Usage |
-|-----------|---------|-------|
-| Types | Common data structures | Shared across all modules |
-| Errors | Unified error handling | Consistent error reporting |
-| Async Utils | Async patterns | Common async operations |
-| Metrics | Observability | Performance monitoring |
-| gRPC | Protocol support | Service communication |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test specific components
-cargo test types
-cargo test errors
-cargo test async_utils
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Dependencies**: Minimal, focused on essential functionality
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS Utils](../utils) - Utility functions
-- [RustFS Config](../config) - Configuration management
+- Shared data structures and type definitions
+- Common error handling and result types
+- Utility functions used across modules
+- Configuration structures and validation
+- Logging and tracing infrastructure
+- Cross-platform compatibility helpers
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Common API Reference](https://docs.rustfs.com/common/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π§ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/config/README.md b/crates/config/README.md
index 65a6843f..998d1371 100644
--- a/crates/config/README.md
+++ b/crates/config/README.md
@@ -3,7 +3,7 @@
# RustFS Config - Configuration Management
- Centralized configuration management for RustFS distributed object storage
+ Configuration management and validation module for RustFS distributed object storage
@@ -17,388 +17,21 @@
## π Overview
-**RustFS Config** is the configuration management module for the [RustFS](https://rustfs.com) distributed object storage system. It provides centralized configuration handling, environment-based configuration loading, validation, and runtime configuration updates for all RustFS components.
-
-> **Note:** This is a foundational submodule of RustFS that provides essential configuration management capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Config** provides configuration management and validation capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### βοΈ Configuration Management
-
-- **Multi-Format Support**: JSON, YAML, TOML configuration formats
-- **Environment Variables**: Automatic environment variable override
-- **Default Values**: Comprehensive default configuration
-- **Validation**: Configuration validation and error reporting
-
-### π§ Advanced Features
-
-- **Hot Reload**: Runtime configuration updates without restart
-- **Profile Support**: Environment-specific configuration profiles
-- **Secret Management**: Secure handling of sensitive configuration
-- **Configuration Merging**: Hierarchical configuration composition
-
-### π οΈ Developer Features
-
-- **Type Safety**: Strongly typed configuration structures
-- **Documentation**: Auto-generated configuration documentation
-- **CLI Integration**: Command-line configuration override
-- **Testing Support**: Configuration mocking for tests
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-config = "0.1.0"
-
-# With specific features
-rustfs-config = { version = "0.1.0", features = ["constants", "notify"] }
-```
-
-### Feature Flags
-
-Available features:
-
-- `constants` - Configuration constants and compile-time values
-- `notify` - Configuration change notification support
-- `observability` - Observability and metrics configuration
-- `default` - Core configuration functionality
-
-## π§ Usage
-
-### Basic Configuration Loading
-
-```rust
-use rustfs_config::{Config, ConfigBuilder, ConfigFormat};
-
-fn main() -> Result<(), Box> {
- // Load configuration from file
- let config = Config::from_file("config.yaml")?;
-
- // Load with environment overrides
- let config = ConfigBuilder::new()
- .add_file("config.yaml")
- .add_env_prefix("RUSTFS")
- .build()?;
-
- // Access configuration values
- println!("Server address: {}", config.server.address);
- println!("Storage path: {}", config.storage.path);
-
- Ok(())
-}
-```
-
-### Environment-Based Configuration
-
-```rust
-use rustfs_config::{Config, Environment};
-
-async fn load_environment_config() -> Result<(), Box> {
- // Load configuration based on environment
- let env = Environment::detect()?;
- let config = Config::for_environment(env).await?;
-
- match env {
- Environment::Development => {
- println!("Using development configuration");
- println!("Debug mode: {}", config.debug.enabled);
- }
- Environment::Production => {
- println!("Using production configuration");
- println!("Log level: {}", config.logging.level);
- }
- Environment::Testing => {
- println!("Using test configuration");
- println!("Test database: {}", config.database.test_url);
- }
- }
-
- Ok(())
-}
-```
-
-### Configuration Structure
-
-```rust
-use rustfs_config::{Config, ServerConfig, StorageConfig, SecurityConfig};
-use serde::{Deserialize, Serialize};
-
-#[derive(Debug, Deserialize, Serialize)]
-pub struct ApplicationConfig {
- pub server: ServerConfig,
- pub storage: StorageConfig,
- pub security: SecurityConfig,
- pub logging: LoggingConfig,
- pub monitoring: MonitoringConfig,
-}
-
-#[derive(Debug, Deserialize, Serialize)]
-pub struct ServerConfig {
- pub address: String,
- pub port: u16,
- pub workers: usize,
- pub timeout: std::time::Duration,
-}
-
-#[derive(Debug, Deserialize, Serialize)]
-pub struct StorageConfig {
- pub path: String,
- pub max_size: u64,
- pub compression: bool,
- pub erasure_coding: ErasureCodingConfig,
-}
-
-fn load_typed_config() -> Result> {
- let config: ApplicationConfig = Config::builder()
- .add_file("config.yaml")
- .add_env_prefix("RUSTFS")
- .set_default("server.port", 9000)?
- .set_default("server.workers", 4)?
- .build_typed()?;
-
- Ok(config)
-}
-```
-
-### Configuration Validation
-
-```rust
-use rustfs_config::{Config, ValidationError, Validator};
-
-#[derive(Debug)]
-pub struct ConfigValidator;
-
-impl Validator for ConfigValidator {
- fn validate(&self, config: &ApplicationConfig) -> Result<(), ValidationError> {
- // Validate server configuration
- if config.server.port < 1024 {
- return Err(ValidationError::new("server.port", "Port must be >= 1024"));
- }
-
- if config.server.workers == 0 {
- return Err(ValidationError::new("server.workers", "Workers must be > 0"));
- }
-
- // Validate storage configuration
- if !std::path::Path::new(&config.storage.path).exists() {
- return Err(ValidationError::new("storage.path", "Storage path does not exist"));
- }
-
- // Validate erasure coding parameters
- if config.storage.erasure_coding.data_drives + config.storage.erasure_coding.parity_drives > 16 {
- return Err(ValidationError::new("storage.erasure_coding", "Total drives cannot exceed 16"));
- }
-
- Ok(())
- }
-}
-
-fn validate_configuration() -> Result<(), Box> {
- let config: ApplicationConfig = Config::load_with_validation(
- "config.yaml",
- ConfigValidator,
- )?;
-
- println!("Configuration is valid!");
- Ok(())
-}
-```
-
-### Hot Configuration Reload
-
-```rust
-use rustfs_config::{ConfigWatcher, ConfigEvent};
-use tokio::sync::mpsc;
-
-async fn watch_configuration_changes() -> Result<(), Box> {
- let (tx, mut rx) = mpsc::channel::(100);
-
- // Start configuration watcher
- let watcher = ConfigWatcher::new("config.yaml", tx)?;
- watcher.start().await?;
-
- // Handle configuration changes
- while let Some(event) = rx.recv().await {
- match event {
- ConfigEvent::Changed(new_config) => {
- println!("Configuration changed, reloading...");
- // Apply new configuration
- apply_configuration(new_config).await?;
- }
- ConfigEvent::Error(err) => {
- eprintln!("Configuration error: {}", err);
- }
- }
- }
-
- Ok(())
-}
-
-async fn apply_configuration(config: ApplicationConfig) -> Result<(), Box> {
- // Update server configuration
- // Update storage configuration
- // Update security settings
- // etc.
- Ok(())
-}
-```
-
-### Configuration Profiles
-
-```rust
-use rustfs_config::{Config, Profile, ProfileManager};
-
-fn load_profile_based_config() -> Result<(), Box> {
- let profile_manager = ProfileManager::new("configs/")?;
-
- // Load specific profile
- let config = profile_manager.load_profile("production")?;
-
- // Load with fallback
- let config = profile_manager
- .load_profile("staging")
- .or_else(|_| profile_manager.load_profile("default"))?;
-
- // Merge multiple profiles
- let config = profile_manager
- .merge_profiles(&["base", "production", "regional"])?;
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Configuration Architecture
-
-```
-Config Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Configuration API β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β File Loader β Env Loader β CLI Parser β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Configuration Merger β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Validation β Watching β Hot Reload β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Type System Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Configuration Sources
-
-| Source | Priority | Format | Example |
-|--------|----------|---------|---------|
-| Command Line | 1 (Highest) | Key-Value | `--server.port=8080` |
-| Environment Variables | 2 | Key-Value | `RUSTFS_SERVER_PORT=8080` |
-| Configuration File | 3 | JSON/YAML/TOML | `config.yaml` |
-| Default Values | 4 (Lowest) | Code | Compile-time defaults |
-
-## π Configuration Reference
-
-### Server Configuration
-
-```yaml
-server:
- address: "0.0.0.0"
- port: 9000
- workers: 4
- timeout: "30s"
- tls:
- enabled: true
- cert_file: "/etc/ssl/server.crt"
- key_file: "/etc/ssl/server.key"
-```
-
-### Storage Configuration
-
-```yaml
-storage:
- path: "/var/lib/rustfs"
- max_size: "1TB"
- compression: true
- erasure_coding:
- data_drives: 8
- parity_drives: 4
- stripe_size: "1MB"
-```
-
-### Security Configuration
-
-```yaml
-security:
- auth:
- enabled: true
- method: "jwt"
- secret_key: "${JWT_SECRET}"
- encryption:
- algorithm: "AES-256-GCM"
- key_rotation_interval: "24h"
-```
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test configuration loading
-cargo test config_loading
-
-# Test validation
-cargo test validation
-
-# Test hot reload
-cargo test hot_reload
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Dependencies**: Minimal external dependencies
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS Utils](../utils) - Utility functions
-- [RustFS Common](../common) - Common types and utilities
+- Multi-format configuration support (TOML, YAML, JSON, ENV)
+- Environment variable integration and override
+- Configuration validation and type safety
+- Hot-reload capabilities for dynamic updates
+- Default value management and fallbacks
+- Secure credential handling and encryption
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Config API Reference](https://docs.rustfs.com/config/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with βοΈ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/crypto/README.md b/crates/crypto/README.md
index 08c7b68f..eddf88ee 100644
--- a/crates/crypto/README.md
+++ b/crates/crypto/README.md
@@ -1,9 +1,9 @@
[](https://rustfs.com)
-# RustFS Crypto Module
+# RustFS Crypto - Cryptographic Operations
- High-performance cryptographic module for RustFS distributed object storage
+ High-performance cryptographic operations module for RustFS distributed object storage
@@ -17,313 +17,21 @@
## π Overview
-The **RustFS Crypto Module** is a core cryptographic component of the [RustFS](https://rustfs.com) distributed object storage system. This module provides secure, high-performance encryption and decryption capabilities, JWT token management, and cross-platform cryptographic operations designed specifically for enterprise-grade storage systems.
-
-> **Note:** This is a submodule of RustFS and is designed to work seamlessly within the RustFS ecosystem. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Crypto** provides high-performance cryptographic operations for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π Encryption & Decryption
-
-- **Multiple Algorithms**: Support for AES-GCM, ChaCha20Poly1305, and PBKDF2
-- **Key Derivation**: Argon2id and PBKDF2 for secure key generation
-- **Memory Safety**: Built with Rust's memory safety guarantees
-- **Cross-Platform**: Optimized for x86_64, aarch64, s390x, and other architectures
-
-### π« JWT Management
-
-- **Token Generation**: Secure JWT token creation with HS512 algorithm
-- **Token Validation**: Robust JWT token verification and decoding
-- **Claims Management**: Flexible claims handling with JSON support
-
-### π‘οΈ Security Features
-
-- **FIPS Compliance**: Optional FIPS 140-2 compatible mode
-- **Hardware Acceleration**: Automatic detection and utilization of CPU crypto extensions
-- **Secure Random**: Cryptographically secure random number generation
-- **Side-Channel Protection**: Resistant to timing attacks
-
-### π Performance
-
-- **Zero-Copy Operations**: Efficient memory usage with `Bytes` support
-- **Async/Await**: Full async support for non-blocking operations
-- **Hardware Optimization**: CPU-specific optimizations for better performance
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-crypto = "0.1.0"
-```
-
-### Feature Flags
-
-```toml
-[dependencies]
-rustfs-crypto = { version = "0.1.0", features = ["crypto", "fips"] }
-```
-
-Available features:
-
-- `crypto` (default): Enable all cryptographic functions
-- `fips`: Enable FIPS 140-2 compliance mode
-- `default`: Includes both `crypto` and `fips`
-
-## π§ Usage
-
-### Basic Encryption/Decryption
-
-```rust
-use rustfs_crypto::{encrypt_data, decrypt_data};
-
-fn main() -> Result<(), Box> {
- let password = b"my_secure_password";
- let data = b"sensitive information";
-
- // Encrypt data
- let encrypted = encrypt_data(password, data)?;
- println!("Encrypted {} bytes", encrypted.len());
-
- // Decrypt data
- let decrypted = decrypt_data(password, &encrypted)?;
- assert_eq!(data, decrypted.as_slice());
- println!("Successfully decrypted data");
-
- Ok(())
-}
-```
-
-### JWT Token Management
-
-```rust
-use rustfs_crypto::{jwt_encode, jwt_decode};
-use serde_json::json;
-
-fn main() -> Result<(), Box> {
- let secret = b"jwt_secret_key";
- let claims = json!({
- "sub": "user123",
- "exp": 1234567890,
- "iat": 1234567890
- });
-
- // Create JWT token
- let token = jwt_encode(secret, &claims)?;
- println!("Generated token: {}", token);
-
- // Verify and decode token
- let decoded = jwt_decode(&token, secret)?;
- println!("Decoded claims: {:?}", decoded.claims);
-
- Ok(())
-}
-```
-
-### Advanced Usage with Custom Configuration
-
-```rust
-use rustfs_crypto::{encrypt_data, decrypt_data, Error};
-
-#[cfg(feature = "crypto")]
-fn secure_storage_example() -> Result<(), Error> {
- // Large data encryption
- let large_data = vec![0u8; 1024 * 1024]; // 1MB
- let password = b"complex_password_123!@#";
-
- // Encrypt with automatic algorithm selection
- let encrypted = encrypt_data(password, &large_data)?;
-
- // Decrypt and verify
- let decrypted = decrypt_data(password, &encrypted)?;
- assert_eq!(large_data.len(), decrypted.len());
-
- println!("Successfully processed {} bytes", large_data.len());
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Supported Encryption Algorithms
-
-| Algorithm | Key Derivation | Use Case | FIPS Compliant |
-|-----------|---------------|----------|----------------|
-| AES-GCM | Argon2id | General purpose, hardware accelerated | β
|
-| ChaCha20Poly1305 | Argon2id | Software-only environments | β |
-| AES-GCM | PBKDF2 | FIPS compliance required | β
|
-
-### Cross-Platform Support
-
-The module automatically detects and optimizes for:
-
-- **x86/x86_64**: AES-NI and PCLMULQDQ instructions
-- **aarch64**: ARM Crypto Extensions
-- **s390x**: IBM Z Crypto Extensions
-- **Other architectures**: Fallback to software implementations
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Run tests with all features
-cargo test --all-features
-
-# Run benchmarks
-cargo bench
-
-# Test cross-platform compatibility
-cargo test --target x86_64-unknown-linux-gnu
-cargo test --target aarch64-unknown-linux-gnu
-```
-
-## π Performance
-
-The crypto module is designed for high-performance scenarios:
-
-- **Encryption Speed**: Up to 2GB/s on modern hardware
-- **Memory Usage**: Minimal heap allocation with zero-copy operations
-- **CPU Utilization**: Automatic hardware acceleration detection
-- **Scalability**: Thread-safe operations for concurrent access
-
-## π€ Integration with RustFS
-
-This module is specifically designed to integrate with other RustFS components:
-
-- **Storage Layer**: Provides encryption for object storage
-- **Authentication**: JWT tokens for API authentication
-- **Configuration**: Secure configuration data encryption
-- **Metadata**: Encrypted metadata storage
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Architectures**: x86_64, aarch64, s390x, and more
-
-## π Security Considerations
-
-- All cryptographic operations use industry-standard algorithms
-- Key derivation follows best practices (Argon2id, PBKDF2)
-- Memory is securely cleared after use
-- Timing attack resistance is built-in
-- Hardware security modules (HSM) support planned
-
-## π Known Issues
-
-- Hardware acceleration detection may not work on all virtualized environments
-- FIPS mode requires additional system-level configuration
-- Some older CPU architectures may have reduced performance
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS ECStore](../ecstore) - Erasure coding storage engine
-- [RustFS IAM](../iam) - Identity and access management
-- [RustFS Policy](../policy) - Policy engine
+- AES-GCM encryption with hardware acceleration
+- RSA and ECDSA digital signature support
+- Secure hash functions (SHA-256, BLAKE3)
+- Key derivation and management utilities
+- Stream ciphers for large data encryption
+- Hardware security module integration
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [API Reference](https://docs.rustfs.com/crypto/)
-- [Security Guide](https://docs.rustfs.com/security/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details on:
-
-- Code style and formatting requirements
-- Testing procedures and coverage
-- Security considerations for cryptographic code
-- Pull request process and review guidelines
-
-### Development Setup
-
-```bash
-# Clone the repository
-git clone https://github.com/rustfs/rustfs.git
-cd rustfs
-
-# Navigate to crypto module
-cd crates/crypto
-
-# Install dependencies
-cargo build
-
-# Run tests
-cargo test
-
-# Format code
-cargo fmt
-
-# Run linter
-cargo clippy
-```
-
-## π¬ Getting Help
-
-- **Documentation**: [docs.rustfs.com](https://docs.rustfs.com)
-- **Issues**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
-- **Security**: Report security issues to
-
-## π Contact
-
-- **Bugs**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **Business**:
-- **Jobs**:
-- **General Discussion**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
-
-## π₯ Contributors
-
-This module is maintained by the RustFS team and community contributors. Special thanks to all who have contributed to making RustFS cryptography secure and efficient.
-
-
- 
-
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
-```
-Copyright 2024 RustFS Team
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-```
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with β€οΈ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/ecstore/README.md b/crates/ecstore/README.md
index bf11fdef..5283939d 100644
--- a/crates/ecstore/README.md
+++ b/crates/ecstore/README.md
@@ -1,6 +1,6 @@
[](https://rustfs.com)
-# RustFS ECStore - Erasure Coding Storage Engine
+# RustFS ECStore - Erasure Coding Storage
High-performance erasure coding storage engine for RustFS distributed object storage
@@ -17,425 +17,24 @@
## π Overview
-**RustFS ECStore** is the core storage engine of the [RustFS](https://rustfs.com) distributed object storage system. It provides enterprise-grade erasure coding capabilities, data integrity protection, and high-performance object storage operations. This module serves as the foundation for RustFS's distributed storage architecture.
-
-> **Note:** This is a core submodule of RustFS and provides the primary storage capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS ECStore** provides erasure coding storage capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π§ Erasure Coding Storage
-
-- **Reed-Solomon Erasure Coding**: Advanced error correction with configurable redundancy
-- **Data Durability**: Protection against disk failures and bit rot
-- **Automatic Repair**: Self-healing capabilities for corrupted or missing data
-- **Configurable Parity**: Flexible parity configurations (4+2, 8+4, 16+4, etc.)
-
-### πΎ Storage Management
-
-- **Multi-Disk Support**: Intelligent disk management and load balancing
-- **Storage Classes**: Support for different storage tiers and policies
-- **Bucket Management**: Advanced bucket operations and lifecycle management
-- **Object Versioning**: Complete versioning support with metadata tracking
-
-### π Performance & Scalability
-
-- **High Throughput**: Optimized for large-scale data operations
-- **Parallel Processing**: Concurrent read/write operations across multiple disks
-- **Memory Efficient**: Smart caching and memory management
-- **SIMD Optimization**: Hardware-accelerated erasure coding operations
-
-### π‘οΈ Data Integrity
-
-- **Bitrot Detection**: Real-time data corruption detection
-- **Checksum Verification**: Multiple checksum algorithms (MD5, SHA256, XXHash)
-- **Healing System**: Automatic background healing and repair
-- **Data Scrubbing**: Proactive data integrity scanning
-
-### π Advanced Features
-
-- **Compression**: Built-in compression support for space optimization
-- **Replication**: Cross-region replication capabilities
-- **Notification System**: Real-time event notifications
-- **Metrics & Monitoring**: Comprehensive performance metrics
-
-## ποΈ Architecture
-
-### Storage Layout
-
-```
-ECStore Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Storage API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Bucket Management β Object Operations β Metadata Mgmt β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Erasure Coding Engine β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Disk Management β Healing System β Cache β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Physical Storage Devices β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Erasure Coding Schemes
-
-| Configuration | Data Drives | Parity Drives | Fault Tolerance | Storage Efficiency |
-|---------------|-------------|---------------|-----------------|-------------------|
-| 4+2 | 4 | 2 | 2 disk failures | 66.7% |
-| 8+4 | 8 | 4 | 4 disk failures | 66.7% |
-| 16+4 | 16 | 4 | 4 disk failures | 80% |
-| Custom | N | K | K disk failures | N/(N+K) |
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-ecstore = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Storage Operations
-
-```rust
-use rustfs_ecstore::{StorageAPI, new_object_layer_fn};
-use std::sync::Arc;
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Initialize storage layer
- let storage = new_object_layer_fn("/path/to/storage").await?;
-
- // Create a bucket
- storage.make_bucket("my-bucket", None).await?;
-
- // Put an object
- let data = b"Hello, RustFS!";
- storage.put_object("my-bucket", "hello.txt", data.to_vec()).await?;
-
- // Get an object
- let retrieved = storage.get_object("my-bucket", "hello.txt", None).await?;
- println!("Retrieved: {}", String::from_utf8_lossy(&retrieved.data));
-
- Ok(())
-}
-```
-
-### Advanced Configuration
-
-```rust
-use rustfs_ecstore::{StorageAPI, config::Config};
-
-async fn setup_storage_with_config() -> Result<(), Box> {
- let config = Config {
- erasure_sets: vec![
- // 8+4 configuration for high durability
- ErasureSet::new(8, 4, vec![
- "/disk1", "/disk2", "/disk3", "/disk4",
- "/disk5", "/disk6", "/disk7", "/disk8",
- "/disk9", "/disk10", "/disk11", "/disk12"
- ])
- ],
- healing_enabled: true,
- compression_enabled: true,
- ..Default::default()
- };
-
- let storage = new_object_layer_fn("/path/to/storage")
- .with_config(config)
- .await?;
-
- Ok(())
-}
-```
-
-### Bucket Management
-
-```rust
-use rustfs_ecstore::{StorageAPI, bucket::BucketInfo};
-
-async fn bucket_operations(storage: Arc) -> Result<(), Box> {
- // Create bucket with specific configuration
- let bucket_info = BucketInfo {
- name: "enterprise-bucket".to_string(),
- versioning_enabled: true,
- lifecycle_config: Some(lifecycle_config()),
- ..Default::default()
- };
-
- storage.make_bucket_with_config(bucket_info).await?;
-
- // List buckets
- let buckets = storage.list_buckets().await?;
- for bucket in buckets {
- println!("Bucket: {}, Created: {}", bucket.name, bucket.created);
- }
-
- // Set bucket policy
- storage.set_bucket_policy("enterprise-bucket", policy_json).await?;
-
- Ok(())
-}
-```
-
-### Healing and Maintenance
-
-```rust
-use rustfs_ecstore::{heal::HealingManager, StorageAPI};
-
-async fn healing_operations(storage: Arc) -> Result<(), Box> {
- // Check storage health
- let health = storage.storage_info().await?;
- println!("Storage Health: {:?}", health);
-
- // Trigger healing for specific bucket
- let healing_result = storage.heal_bucket("my-bucket").await?;
- println!("Healing completed: {:?}", healing_result);
-
- // Background healing status
- let healing_status = storage.healing_status().await?;
- println!("Background healing: {:?}", healing_status);
-
- Ok(())
-}
-```
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Run tests with specific features
-cargo test --features "compression,healing"
-
-# Run benchmarks
-cargo bench
-
-# Run erasure coding benchmarks
-cargo bench --bench erasure_benchmark
-
-# Run comparison benchmarks
-cargo bench --bench comparison_benchmark
-```
-
-## π Performance Benchmarks
-
-ECStore is designed for high-performance storage operations:
-
-### Throughput Performance
-
-- **Sequential Write**: Up to 10GB/s on NVMe storage
-- **Sequential Read**: Up to 12GB/s with parallel reads
-- **Random I/O**: 100K+ IOPS for small objects
-- **Erasure Coding**: 5GB/s encoding/decoding throughput
-
-### Scalability Metrics
-
-- **Storage Capacity**: Exabyte-scale deployments
-- **Concurrent Operations**: 10,000+ concurrent requests
-- **Disk Scaling**: Support for 1000+ disks per node
-- **Fault Tolerance**: Up to 50% disk failure resilience
-
-## π§ Configuration
-
-### Storage Configuration
-
-```toml
-[storage]
-# Erasure coding configuration
-erasure_set_size = 12 # Total disks per set
-data_drives = 8 # Data drives per set
-parity_drives = 4 # Parity drives per set
-
-# Performance tuning
-read_quorum = 6 # Minimum disks for read
-write_quorum = 7 # Minimum disks for write
-parallel_reads = true # Enable parallel reads
-compression = true # Enable compression
-
-# Healing configuration
-healing_enabled = true
-healing_interval = "24h"
-bitrot_check_interval = "168h" # Weekly bitrot check
-```
-
-### Advanced Features
-
-```rust
-use rustfs_ecstore::config::StorageConfig;
-
-let config = StorageConfig {
- // Enable advanced features
- bitrot_protection: true,
- automatic_healing: true,
- compression_level: 6,
- checksum_algorithm: ChecksumAlgorithm::XXHash64,
-
- // Performance tuning
- read_buffer_size: 1024 * 1024, // 1MB read buffer
- write_buffer_size: 4 * 1024 * 1024, // 4MB write buffer
- concurrent_operations: 1000,
-
- // Storage optimization
- small_object_threshold: 128 * 1024, // 128KB
- large_object_threshold: 64 * 1024 * 1024, // 64MB
-
- ..Default::default()
-};
-```
-
-## π€ Integration with RustFS
-
-ECStore integrates seamlessly with other RustFS components:
-
-- **API Server**: Provides S3-compatible storage operations
-- **IAM Module**: Handles authentication and authorization
-- **Policy Engine**: Implements bucket policies and access controls
-- **Notification System**: Publishes storage events
-- **Monitoring**: Provides detailed metrics and health status
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Storage**: Local disks, network storage, cloud storage
-- **Memory**: Minimum 4GB RAM (8GB+ recommended)
-- **Network**: High-speed network for distributed deployments
-
-## π Performance Tuning
-
-### Optimization Tips
-
-1. **Disk Configuration**:
- - Use dedicated disks for each erasure set
- - Prefer NVMe over SATA for better performance
- - Ensure consistent disk sizes within erasure sets
-
-2. **Memory Settings**:
- - Allocate sufficient memory for caching
- - Tune read/write buffer sizes based on workload
- - Enable memory-mapped files for large objects
-
-3. **Network Optimization**:
- - Use high-speed network connections
- - Configure proper MTU sizes
- - Enable network compression for WAN scenarios
-
-4. **CPU Optimization**:
- - Utilize SIMD instructions for erasure coding
- - Balance CPU cores across erasure sets
- - Enable hardware-accelerated checksums
-
-## π Troubleshooting
-
-### Common Issues
-
-1. **Disk Failures**:
- - Check disk health using `storage_info()`
- - Trigger healing with `heal_bucket()`
- - Replace failed disks and re-add to cluster
-
-2. **Performance Issues**:
- - Monitor disk I/O utilization
- - Check network bandwidth usage
- - Verify erasure coding configuration
-
-3. **Data Integrity**:
- - Run bitrot detection scans
- - Verify checksums for critical data
- - Check healing system status
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS Crypto](../crypto) - Cryptographic operations
-- [RustFS IAM](../iam) - Identity and access management
-- [RustFS Policy](../policy) - Policy engine
-- [RustFS FileMeta](../filemeta) - File metadata management
+- Reed-Solomon erasure coding implementation
+- Configurable redundancy levels (N+K schemes)
+- Automatic data healing and reconstruction
+- Multi-drive support with intelligent placement
+- Parallel encoding/decoding for performance
+- Efficient disk space utilization
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Storage API Reference](https://docs.rustfs.com/ecstore/)
-- [Erasure Coding Guide](https://docs.rustfs.com/erasure-coding/)
-- [Performance Tuning](https://docs.rustfs.com/performance/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details on:
-
-- Storage engine architecture and design patterns
-- Erasure coding implementation guidelines
-- Performance optimization techniques
-- Testing procedures for storage operations
-- Documentation standards for storage APIs
-
-### Development Setup
-
-```bash
-# Clone the repository
-git clone https://github.com/rustfs/rustfs.git
-cd rustfs
-
-# Navigate to ECStore module
-cd crates/ecstore
-
-# Install dependencies
-cargo build
-
-# Run tests
-cargo test
-
-# Run benchmarks
-cargo bench
-
-# Format code
-cargo fmt
-
-# Run linter
-cargo clippy
-```
-
-## π¬ Getting Help
-
-- **Documentation**: [docs.rustfs.com](https://docs.rustfs.com)
-- **Issues**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
-- **Storage Support**:
-
-## π Contact
-
-- **Bugs**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **Business**:
-- **Jobs**:
-- **General Discussion**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
-
-## π₯ Contributors
-
-This module is maintained by the RustFS storage team and community contributors. Special thanks to all who have contributed to making RustFS storage reliable and efficient.
-
-
-
-
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
```
Copyright 2024 RustFS Team
diff --git a/crates/ecstore/README_cn.md b/crates/ecstore/README_cn.md
index d3ff83df..5bd11b23 100644
--- a/crates/ecstore/README_cn.md
+++ b/crates/ecstore/README_cn.md
@@ -1,103 +1,19 @@
# ECStore - Erasure Coding Storage
-ECStore provides erasure coding functionality for the RustFS project, using high-performance Reed-Solomon SIMD
-implementation for optimal performance.
+ECStore provides erasure coding functionality for the RustFS project, using high-performance Reed-Solomon SIMD implementation for optimal performance.
-## Reed-Solomon Implementation
+## Features
-### SIMD Backend (Only)
+- **Reed-Solomon Implementation**: High-performance SIMD-optimized erasure coding
+- **Cross-Platform Compatibility**: Support for x86_64, aarch64, and other architectures
+- **Performance Optimized**: SIMD instructions for maximum throughput
+- **Thread Safety**: Safe concurrent access with caching optimizations
+- **Scalable**: Excellent performance for high-throughput scenarios
-- **Performance**: Uses SIMD optimization for high-performance encoding/decoding
-- **Compatibility**: Works with any shard size through SIMD implementation
-- **Reliability**: High-performance SIMD implementation for large data processing
-- **Use case**: Optimized for maximum performance in large data processing scenarios
+## Documentation
-### Usage Example
+For complete documentation, examples, and usage information, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
-```rust
-use rustfs_ecstore::erasure_coding::Erasure;
+## License
-// Create erasure coding instance
-// 4 data shards, 2 parity shards, 1KB block size
-let erasure = Erasure::new(4, 2, 1024);
-
-// Encode data
-let data = b"hello world from rustfs erasure coding";
-let shards = erasure.encode_data(data) ?;
-
-// Simulate loss of one shard
-let mut shards_opt: Vec > > = shards
-.iter()
-.map( | b| Some(b.to_vec()))
-.collect();
-shards_opt[2] = None; // Lose shard 2
-
-// Reconstruct missing data
-erasure.decode_data( & mut shards_opt) ?;
-
-// Recover original data
-let mut recovered = Vec::new();
-for shard in shards_opt.iter().take(4) { // Only data shards
-recovered.extend_from_slice(shard.as_ref().unwrap());
-}
-recovered.truncate(data.len());
-assert_eq!(&recovered, data);
-```
-
-## Performance Considerations
-
-### SIMD Implementation Benefits
-
-- **High Throughput**: Optimized for large block sizes (>= 1KB recommended)
-- **CPU Optimization**: Leverages modern CPU SIMD instructions
-- **Scalability**: Excellent performance for high-throughput scenarios
-
-### Implementation Details
-
-#### `reed-solomon-simd`
-
-- **Instance Caching**: Encoder/decoder instances are cached and reused for optimal performance
-- **Thread Safety**: Thread-safe with RwLock-based caching
-- **SIMD Optimization**: Leverages CPU SIMD instructions for maximum performance
-- **Reset Capability**: Cached instances are reset for different parameters, avoiding unnecessary allocations
-
-### Performance Tips
-
-1. **Batch Operations**: When possible, batch multiple small operations into larger blocks
-2. **Block Size Optimization**: Use block sizes that are multiples of 64 bytes for optimal SIMD performance
-3. **Memory Allocation**: Pre-allocate buffers when processing multiple blocks
-4. **Cache Warming**: Initial operations may be slower due to cache setup, subsequent operations benefit from caching
-
-## Cross-Platform Compatibility
-
-The SIMD implementation supports:
-
-- x86_64 with advanced SIMD instructions (AVX2, SSE)
-- aarch64 (ARM64) with NEON SIMD optimizations
-- Other architectures with fallback implementations
-
-The implementation automatically selects the best available SIMD instructions for the target platform, providing optimal
-performance across different architectures.
-
-## Testing and Benchmarking
-
-Run performance benchmarks:
-
-```bash
-# Run erasure coding benchmarks
-cargo bench --bench erasure_benchmark
-
-# Run comparison benchmarks
-cargo bench --bench comparison_benchmark
-
-# Generate benchmark reports
-./run_benchmarks.sh
-```
-
-## Error Handling
-
-All operations return `Result` types with comprehensive error information:
-
-- Encoding errors: Invalid parameters, insufficient memory
-- Decoding errors: Too many missing shards, corrupted data
-- Configuration errors: Invalid shard counts, unsupported parameters
\ No newline at end of file
+This project is licensed under the Apache License, Version 2.0.
diff --git a/crates/filemeta/README.md b/crates/filemeta/README.md
index 07a1acc5..4bc6b066 100644
--- a/crates/filemeta/README.md
+++ b/crates/filemeta/README.md
@@ -3,7 +3,7 @@
# RustFS FileMeta - File Metadata Management
- High-performance file metadata management for RustFS distributed object storage
+ Advanced file metadata management and indexing module for RustFS distributed object storage
@@ -17,246 +17,21 @@
## π Overview
-**RustFS FileMeta** is the metadata management module for the [RustFS](https://rustfs.com) distributed object storage system. It provides efficient storage, retrieval, and management of file metadata, supporting features like versioning, tagging, and extended attributes with high performance and reliability.
-
-> **Note:** This is a core submodule of RustFS that provides essential metadata management capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS FileMeta** provides advanced file metadata management and indexing capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π Metadata Management
-- **File Information**: Complete file metadata including size, timestamps, and checksums
-- **Object Versioning**: Version-aware metadata management
-- **Extended Attributes**: Custom metadata and tagging support
-- **Inline Metadata**: Optimized storage for small metadata
-
-### π Performance Features
-- **FlatBuffers Serialization**: Zero-copy metadata serialization
-- **Efficient Storage**: Optimized metadata storage layout
-- **Fast Lookups**: High-performance metadata queries
-- **Batch Operations**: Bulk metadata operations
-
-### π§ Advanced Capabilities
-- **Schema Evolution**: Forward and backward compatible metadata schemas
-- **Compression**: Metadata compression for space efficiency
-- **Validation**: Metadata integrity verification
-- **Migration**: Seamless metadata format migration
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-filemeta = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Metadata Operations
-
-```rust
-use rustfs_filemeta::{FileInfo, XLMeta};
-use std::collections::HashMap;
-
-fn main() -> Result<(), Box> {
- // Create file metadata
- let mut file_info = FileInfo::new();
- file_info.name = "example.txt".to_string();
- file_info.size = 1024;
- file_info.mod_time = chrono::Utc::now();
-
- // Add custom metadata
- let mut user_defined = HashMap::new();
- user_defined.insert("author".to_string(), "john@example.com".to_string());
- user_defined.insert("department".to_string(), "engineering".to_string());
- file_info.user_defined = user_defined;
-
- // Create XL metadata
- let xl_meta = XLMeta::new(file_info);
-
- // Serialize metadata
- let serialized = xl_meta.serialize()?;
-
- // Deserialize metadata
- let deserialized = XLMeta::deserialize(&serialized)?;
-
- println!("File: {}, Size: {}", deserialized.file_info.name, deserialized.file_info.size);
-
- Ok(())
-}
-```
-
-### Advanced Metadata Management
-
-```rust
-use rustfs_filemeta::{XLMeta, FileInfo, VersionInfo};
-
-async fn advanced_metadata_example() -> Result<(), Box> {
- // Create versioned metadata
- let mut xl_meta = XLMeta::new(FileInfo::default());
-
- // Set version information
- xl_meta.set_version_info(VersionInfo {
- version_id: "v1.0.0".to_string(),
- is_latest: true,
- delete_marker: false,
- restore_ongoing: false,
- });
-
- // Add checksums
- xl_meta.add_checksum("md5", "d41d8cd98f00b204e9800998ecf8427e");
- xl_meta.add_checksum("sha256", "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855");
-
- // Set object tags
- xl_meta.set_tags(vec![
- ("Environment".to_string(), "Production".to_string()),
- ("Owner".to_string(), "DataTeam".to_string()),
- ]);
-
- // Set retention information
- xl_meta.set_retention_info(
- chrono::Utc::now() + chrono::Duration::days(365),
- "GOVERNANCE".to_string(),
- );
-
- // Validate metadata
- xl_meta.validate()?;
-
- Ok(())
-}
-```
-
-### Inline Metadata Operations
-
-```rust
-use rustfs_filemeta::{InlineMetadata, MetadataSize};
-
-fn inline_metadata_example() -> Result<(), Box> {
- // Create inline metadata for small files
- let mut inline_meta = InlineMetadata::new();
-
- // Set basic properties
- inline_meta.set_content_type("text/plain");
- inline_meta.set_content_encoding("gzip");
- inline_meta.set_cache_control("max-age=3600");
-
- // Add custom headers
- inline_meta.add_header("x-custom-field", "custom-value");
- inline_meta.add_header("x-app-version", "1.2.3");
-
- // Check if metadata fits inline storage
- if inline_meta.size() <= MetadataSize::INLINE_THRESHOLD {
- println!("Metadata can be stored inline");
- } else {
- println!("Metadata requires separate storage");
- }
-
- // Serialize for storage
- let bytes = inline_meta.to_bytes()?;
-
- // Deserialize from storage
- let restored = InlineMetadata::from_bytes(&bytes)?;
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Metadata Storage Layout
-
-```
-FileMeta Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Metadata API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β XL Metadata β Inline Metadata β Version Info β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β FlatBuffers Serialization β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Compression β Validation β Migration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Storage Backend Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Metadata Types
-
-| Type | Use Case | Storage | Performance |
-|------|----------|---------|-------------|
-| XLMeta | Large objects with rich metadata | Separate file | High durability |
-| InlineMeta | Small objects with minimal metadata | Embedded | Fastest access |
-| VersionMeta | Object versioning information | Version-specific | Version-aware |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Run serialization benchmarks
-cargo bench
-
-# Test metadata validation
-cargo test validation
-
-# Test schema migration
-cargo test migration
-```
-
-## π Performance
-
-FileMeta is optimized for high-performance metadata operations:
-
-- **Serialization**: Zero-copy FlatBuffers serialization
-- **Storage**: Compact binary format reduces I/O
-- **Caching**: Intelligent metadata caching
-- **Batch Operations**: Efficient bulk metadata processing
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Memory**: Minimal memory footprint
-- **Storage**: Compatible with RustFS storage backend
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS ECStore](../ecstore) - Erasure coding storage engine
-- [RustFS Utils](../utils) - Utility functions
-- [RustFS Proto](../protos) - Protocol definitions
+- High-performance metadata storage and retrieval
+- Advanced indexing with full-text search capabilities
+- File attribute management and custom metadata
+- Version tracking and history management
+- Distributed metadata replication
+- Real-time metadata synchronization
## π Documentation
-For comprehensive documentation, visit:
-- [RustFS Documentation](https://docs.rustfs.com)
-- [FileMeta API Reference](https://docs.rustfs.com/filemeta/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with β€οΈ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/iam/README.md b/crates/iam/README.md
index a0b25296..721704c8 100644
--- a/crates/iam/README.md
+++ b/crates/iam/README.md
@@ -1,9 +1,9 @@
[](https://rustfs.com)
-# RustFS IAM - Identity and Access Management
+# RustFS IAM - Identity & Access Management
- Enterprise-grade identity and access management for RustFS distributed object storage
+ Identity and access management system for RustFS distributed object storage
@@ -17,592 +17,21 @@
## π Overview
-**RustFS IAM** is the identity and access management module for the [RustFS](https://rustfs.com) distributed object storage system. It provides comprehensive authentication, authorization, and access control capabilities, ensuring secure and compliant access to storage resources.
-
-> **Note:** This is a core submodule of RustFS and provides essential security and access control features for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS IAM** provides identity and access management capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π Authentication & Authorization
-
-- **Multi-Factor Authentication**: Support for various authentication methods
-- **Access Key Management**: Secure generation and management of access keys
-- **JWT Token Support**: Stateless authentication with JWT tokens
-- **Session Management**: Secure session handling and token refresh
-
-### π₯ User Management
-
-- **User Accounts**: Complete user lifecycle management
-- **Service Accounts**: Automated service authentication
-- **Temporary Accounts**: Time-limited access credentials
-- **Group Management**: Organize users into groups for easier management
-
-### π‘οΈ Access Control
-
-- **Role-Based Access Control (RBAC)**: Flexible role and permission system
-- **Policy-Based Access Control**: Fine-grained access policies
-- **Resource-Level Permissions**: Granular control over storage resources
-- **API-Level Authorization**: Secure API access control
-
-### π Credential Management
-
-- **Secure Key Generation**: Cryptographically secure key generation
-- **Key Rotation**: Automatic and manual key rotation capabilities
-- **Credential Validation**: Real-time credential verification
-- **Secret Management**: Secure storage and retrieval of secrets
-
-### π’ Enterprise Features
-
-- **LDAP Integration**: Enterprise directory service integration
-- **SSO Support**: Single Sign-On capabilities
-- **Audit Logging**: Comprehensive access audit trails
-- **Compliance Features**: Meet regulatory compliance requirements
-
-## ποΈ Architecture
-
-### IAM System Architecture
-
-```
-IAM Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β IAM API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Authentication β Authorization β User Management β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Policy Engine Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Credential Store β Cache Layer β Token Manager β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Storage Backend Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Security Model
-
-| Component | Description | Security Level |
-|-----------|-------------|----------------|
-| Access Keys | API authentication credentials | High |
-| JWT Tokens | Stateless authentication tokens | High |
-| Session Management | User session handling | Medium |
-| Policy Enforcement | Access control policies | Critical |
-| Audit Logging | Security event tracking | High |
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-iam = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic IAM Setup
-
-```rust
-use rustfs_iam::{init_iam_sys, get};
-use rustfs_ecstore::ECStore;
-use std::sync::Arc;
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Initialize with ECStore backend
- let ecstore = Arc::new(ECStore::new("/path/to/storage").await?);
-
- // Initialize IAM system
- init_iam_sys(ecstore).await?;
-
- // Get IAM system instance
- let iam = get()?;
-
- println!("IAM system initialized successfully");
- Ok(())
-}
-```
-
-### User Management
-
-```rust
-use rustfs_iam::{get, manager::UserInfo};
-
-async fn user_management_example() -> Result<(), Box> {
- let iam = get()?;
-
- // Create a new user
- let user_info = UserInfo {
- access_key: "AKIAIOSFODNN7EXAMPLE".to_string(),
- secret_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY".to_string(),
- status: "enabled".to_string(),
- ..Default::default()
- };
-
- iam.create_user("john-doe", user_info).await?;
-
- // List users
- let users = iam.list_users().await?;
- for user in users {
- println!("User: {}, Status: {}", user.name, user.status);
- }
-
- // Update user status
- iam.set_user_status("john-doe", "disabled").await?;
-
- // Delete user
- iam.delete_user("john-doe").await?;
-
- Ok(())
-}
-```
-
-### Group Management
-
-```rust
-use rustfs_iam::{get, manager::GroupInfo};
-
-async fn group_management_example() -> Result<(), Box> {
- let iam = get()?;
-
- // Create a group
- let group_info = GroupInfo {
- name: "developers".to_string(),
- members: vec!["john-doe".to_string(), "jane-smith".to_string()],
- policies: vec!["read-only-policy".to_string()],
- ..Default::default()
- };
-
- iam.create_group(group_info).await?;
-
- // Add user to group
- iam.add_user_to_group("alice", "developers").await?;
-
- // Remove user from group
- iam.remove_user_from_group("alice", "developers").await?;
-
- // List groups
- let groups = iam.list_groups().await?;
- for group in groups {
- println!("Group: {}, Members: {}", group.name, group.members.len());
- }
-
- Ok(())
-}
-```
-
-### Policy Management
-
-```rust
-use rustfs_iam::{get, manager::PolicyDocument};
-
-async fn policy_management_example() -> Result<(), Box> {
- let iam = get()?;
-
- // Create a policy
- let policy_doc = PolicyDocument {
- version: "2012-10-17".to_string(),
- statement: vec![
- Statement {
- effect: "Allow".to_string(),
- action: vec!["s3:GetObject".to_string()],
- resource: vec!["arn:aws:s3:::my-bucket/*".to_string()],
- ..Default::default()
- }
- ],
- ..Default::default()
- };
-
- iam.create_policy("read-only-policy", policy_doc).await?;
-
- // Attach policy to user
- iam.attach_user_policy("john-doe", "read-only-policy").await?;
-
- // Detach policy from user
- iam.detach_user_policy("john-doe", "read-only-policy").await?;
-
- // List policies
- let policies = iam.list_policies().await?;
- for policy in policies {
- println!("Policy: {}", policy.name);
- }
-
- Ok(())
-}
-```
-
-### Service Account Management
-
-```rust
-use rustfs_iam::{get, manager::ServiceAccountInfo};
-
-async fn service_account_example() -> Result<(), Box> {
- let iam = get()?;
-
- // Create service account
- let service_account = ServiceAccountInfo {
- name: "backup-service".to_string(),
- description: "Automated backup service".to_string(),
- policies: vec!["backup-policy".to_string()],
- ..Default::default()
- };
-
- iam.create_service_account(service_account).await?;
-
- // Generate credentials for service account
- let credentials = iam.generate_service_account_credentials("backup-service").await?;
- println!("Service Account Credentials: {:?}", credentials);
-
- // Rotate service account credentials
- iam.rotate_service_account_credentials("backup-service").await?;
-
- Ok(())
-}
-```
-
-### Authentication and Authorization
-
-```rust
-use rustfs_iam::{get, auth::Credentials};
-
-async fn auth_example() -> Result<(), Box> {
- let iam = get()?;
-
- // Authenticate user
- let credentials = Credentials {
- access_key: "AKIAIOSFODNN7EXAMPLE".to_string(),
- secret_key: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY".to_string(),
- session_token: None,
- };
-
- let auth_result = iam.authenticate(&credentials).await?;
- println!("Authentication successful: {}", auth_result.user_name);
-
- // Check authorization
- let authorized = iam.is_authorized(
- &auth_result.user_name,
- "s3:GetObject",
- "arn:aws:s3:::my-bucket/file.txt"
- ).await?;
-
- if authorized {
- println!("User is authorized to access the resource");
- } else {
- println!("User is not authorized to access the resource");
- }
-
- Ok(())
-}
-```
-
-### Temporary Credentials
-
-```rust
-use rustfs_iam::{get, manager::TemporaryCredentials};
-use std::time::Duration;
-
-async fn temp_credentials_example() -> Result<(), Box> {
- let iam = get()?;
-
- // Create temporary credentials
- let temp_creds = iam.create_temporary_credentials(
- "john-doe",
- Duration::from_secs(3600), // 1 hour
- Some("read-only-policy".to_string())
- ).await?;
-
- println!("Temporary Access Key: {}", temp_creds.access_key);
- println!("Expires at: {}", temp_creds.expiration);
-
- // Validate temporary credentials
- let is_valid = iam.validate_temporary_credentials(&temp_creds.access_key).await?;
- println!("Temporary credentials valid: {}", is_valid);
-
- Ok(())
-}
-```
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Run tests with specific features
-cargo test --features "ldap,sso"
-
-# Run integration tests
-cargo test --test integration
-
-# Run authentication tests
-cargo test auth
-
-# Run authorization tests
-cargo test authz
-```
-
-## π Security Best Practices
-
-### Key Management
-
-- Rotate access keys regularly
-- Use strong, randomly generated keys
-- Store keys securely using environment variables or secret management systems
-- Implement key rotation policies
-
-### Access Control
-
-- Follow the principle of least privilege
-- Use groups for easier permission management
-- Regularly audit user permissions
-- Implement resource-based policies
-
-### Monitoring and Auditing
-
-- Enable comprehensive audit logging
-- Monitor failed authentication attempts
-- Set up alerts for suspicious activities
-- Regular security reviews
-
-## π Performance Considerations
-
-### Caching Strategy
-
-- **User Cache**: Cache user information for faster lookups
-- **Policy Cache**: Cache policy documents to reduce latency
-- **Token Cache**: Cache JWT tokens for stateless authentication
-- **Permission Cache**: Cache authorization decisions
-
-### Scalability
-
-- **Distributed Cache**: Use distributed caching for multi-node deployments
-- **Database Optimization**: Optimize database queries for user/group lookups
-- **Connection Pooling**: Use connection pooling for database connections
-- **Async Operations**: Leverage async programming for better throughput
-
-## π§ Configuration
-
-### Basic Configuration
-
-```toml
-[iam]
-# Authentication settings
-jwt_secret = "your-jwt-secret-key"
-jwt_expiration = "24h"
-session_timeout = "30m"
-
-# Password policy
-min_password_length = 8
-require_special_chars = true
-require_numbers = true
-require_uppercase = true
-
-# Account lockout
-max_login_attempts = 5
-lockout_duration = "15m"
-
-# Audit settings
-audit_enabled = true
-audit_log_path = "/var/log/rustfs/iam-audit.log"
-```
-
-### Advanced Configuration
-
-```rust
-use rustfs_iam::config::IamConfig;
-
-let config = IamConfig {
- // Authentication settings
- jwt_secret: "your-secure-jwt-secret".to_string(),
- jwt_expiration_hours: 24,
- session_timeout_minutes: 30,
-
- // Security settings
- password_policy: PasswordPolicy {
- min_length: 8,
- require_special_chars: true,
- require_numbers: true,
- require_uppercase: true,
- max_age_days: 90,
- },
-
- // Rate limiting
- rate_limit: RateLimit {
- max_requests_per_minute: 100,
- burst_size: 10,
- },
-
- // Audit settings
- audit_enabled: true,
- audit_log_level: "info".to_string(),
-
- ..Default::default()
-};
-```
-
-## π€ Integration with RustFS
-
-IAM integrates seamlessly with other RustFS components:
-
-- **ECStore**: Provides user and policy storage backend
-- **Policy Engine**: Implements fine-grained access control
-- **Crypto Module**: Handles secure key generation and JWT operations
-- **API Server**: Provides authentication and authorization for S3 API
-- **Admin Interface**: Manages users, groups, and policies
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Database**: Compatible with RustFS storage backend
-- **Memory**: Minimum 2GB RAM for caching
-- **Network**: Secure connections for authentication
-
-## π Troubleshooting
-
-### Common Issues
-
-1. **Authentication Failures**:
- - Check access key and secret key validity
- - Verify user account status (enabled/disabled)
- - Check for account lockout due to failed attempts
-
-2. **Authorization Errors**:
- - Verify user has required permissions
- - Check policy attachments (user/group policies)
- - Validate resource ARN format
-
-3. **Performance Issues**:
- - Monitor cache hit rates
- - Check database connection pool utilization
- - Verify JWT token size and complexity
-
-### Debug Commands
-
-```bash
-# Check IAM system status
-rustfs-cli iam status
-
-# List all users
-rustfs-cli iam list-users
-
-# Validate user credentials
-rustfs-cli iam validate-credentials --access-key
-
-# Test policy evaluation
-rustfs-cli iam test-policy --user --action --resource
-```
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS ECStore](../ecstore) - Erasure coding storage engine
-- [RustFS Policy](../policy) - Policy engine for access control
-- [RustFS Crypto](../crypto) - Cryptographic operations
-- [RustFS MadAdmin](../madmin) - Administrative interface
+- User and group management with RBAC
+- Service account and API key authentication
+- Policy engine with fine-grained permissions
+- LDAP/Active Directory integration
+- Multi-factor authentication support
+- Session management and token validation
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [IAM API Reference](https://docs.rustfs.com/iam/)
-- [Security Guide](https://docs.rustfs.com/security/)
-- [Authentication Guide](https://docs.rustfs.com/auth/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details on:
-
-- Security-first development practices
-- IAM system architecture guidelines
-- Authentication and authorization patterns
-- Testing procedures for security features
-- Documentation standards for security APIs
-
-### Development Setup
-
-```bash
-# Clone the repository
-git clone https://github.com/rustfs/rustfs.git
-cd rustfs
-
-# Navigate to IAM module
-cd crates/iam
-
-# Install dependencies
-cargo build
-
-# Run tests
-cargo test
-
-# Run security tests
-cargo test security
-
-# Format code
-cargo fmt
-
-# Run linter
-cargo clippy
-```
-
-## π¬ Getting Help
-
-- **Documentation**: [docs.rustfs.com](https://docs.rustfs.com)
-- **Issues**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
-- **Security**: Report security issues to
-
-## π Contact
-
-- **Bugs**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **Business**:
-- **Jobs**:
-- **General Discussion**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
-
-## π₯ Contributors
-
-This module is maintained by the RustFS security team and community contributors. Special thanks to all who have contributed to making RustFS secure and compliant.
-
-
-
-
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
-```
-Copyright 2024 RustFS Team
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-```
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π by the RustFS Security Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/lock/README.md b/crates/lock/README.md
index c9b2d70e..a9a0874a 100644
--- a/crates/lock/README.md
+++ b/crates/lock/README.md
@@ -3,7 +3,7 @@
# RustFS Lock - Distributed Locking
- Distributed locking and synchronization for RustFS object storage
+ High-performance distributed locking system for RustFS object storage
@@ -17,376 +17,21 @@
## π Overview
-**RustFS Lock** provides distributed locking and synchronization primitives for the [RustFS](https://rustfs.com) distributed object storage system. It ensures data consistency and prevents race conditions in multi-node environments through various locking mechanisms and coordination protocols.
-
-> **Note:** This is a core submodule of RustFS that provides essential distributed locking capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Lock** provides distributed locking capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π Distributed Locking
-
-- **Exclusive Locks**: Mutual exclusion across cluster nodes
-- **Shared Locks**: Reader-writer lock semantics
-- **Timeout Support**: Configurable lock timeouts and expiration
-- **Deadlock Prevention**: Automatic deadlock detection and resolution
-
-### π Synchronization Primitives
-
-- **Distributed Mutex**: Cross-node mutual exclusion
-- **Distributed Semaphore**: Resource counting across nodes
-- **Distributed Barrier**: Coordination point for multiple nodes
-- **Distributed Condition Variables**: Wait/notify across nodes
-
-### π‘οΈ Consistency Guarantees
-
-- **Linearizable Operations**: Strong consistency guarantees
-- **Fault Tolerance**: Automatic recovery from node failures
-- **Network Partition Handling**: CAP theorem aware implementations
-- **Consensus Integration**: Raft-based consensus for critical locks
-
-### π Performance Features
-
-- **Lock Coalescing**: Efficient batching of lock operations
-- **Adaptive Timeouts**: Dynamic timeout adjustment
-- **Lock Hierarchy**: Hierarchical locking for better scalability
-- **Optimistic Locking**: Reduced contention through optimistic approaches
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-lock = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Distributed Lock
-
-```rust
-use rustfs_lock::{DistributedLock, LockManager, LockOptions};
-use std::time::Duration;
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Create lock manager
- let lock_manager = LockManager::new("cluster-endpoint").await?;
-
- // Acquire distributed lock
- let lock_options = LockOptions {
- timeout: Duration::from_secs(30),
- auto_renew: true,
- ..Default::default()
- };
-
- let lock = lock_manager.acquire_lock("resource-key", lock_options).await?;
-
- // Critical section
- {
- println!("Lock acquired, performing critical operations...");
- // Your critical code here
- tokio::time::sleep(Duration::from_secs(2)).await;
- }
-
- // Release lock
- lock.release().await?;
- println!("Lock released");
-
- Ok(())
-}
-```
-
-### Distributed Mutex
-
-```rust
-use rustfs_lock::{DistributedMutex, LockManager};
-use std::sync::Arc;
-
-async fn distributed_mutex_example() -> Result<(), Box> {
- let lock_manager = Arc::new(LockManager::new("cluster-endpoint").await?);
-
- // Create distributed mutex
- let mutex = DistributedMutex::new(lock_manager.clone(), "shared-resource");
-
- // Spawn multiple tasks
- let mut handles = vec![];
-
- for i in 0..5 {
- let mutex = mutex.clone();
- let handle = tokio::spawn(async move {
- let _guard = mutex.lock().await.unwrap();
- println!("Task {} acquired mutex", i);
-
- // Simulate work
- tokio::time::sleep(Duration::from_secs(1)).await;
-
- println!("Task {} releasing mutex", i);
- // Guard is automatically released when dropped
- });
-
- handles.push(handle);
- }
-
- // Wait for all tasks to complete
- for handle in handles {
- handle.await?;
- }
-
- Ok(())
-}
-```
-
-### Distributed Semaphore
-
-```rust
-use rustfs_lock::{DistributedSemaphore, LockManager};
-use std::sync::Arc;
-
-async fn distributed_semaphore_example() -> Result<(), Box> {
- let lock_manager = Arc::new(LockManager::new("cluster-endpoint").await?);
-
- // Create distributed semaphore with 3 permits
- let semaphore = DistributedSemaphore::new(
- lock_manager.clone(),
- "resource-pool",
- 3
- );
-
- // Spawn multiple tasks
- let mut handles = vec![];
-
- for i in 0..10 {
- let semaphore = semaphore.clone();
- let handle = tokio::spawn(async move {
- let _permit = semaphore.acquire().await.unwrap();
- println!("Task {} acquired permit", i);
-
- // Simulate work
- tokio::time::sleep(Duration::from_secs(2)).await;
-
- println!("Task {} releasing permit", i);
- // Permit is automatically released when dropped
- });
-
- handles.push(handle);
- }
-
- // Wait for all tasks to complete
- for handle in handles {
- handle.await?;
- }
-
- Ok(())
-}
-```
-
-### Distributed Barrier
-
-```rust
-use rustfs_lock::{DistributedBarrier, LockManager};
-use std::sync::Arc;
-
-async fn distributed_barrier_example() -> Result<(), Box> {
- let lock_manager = Arc::new(LockManager::new("cluster-endpoint").await?);
-
- // Create distributed barrier for 3 participants
- let barrier = DistributedBarrier::new(
- lock_manager.clone(),
- "sync-point",
- 3
- );
-
- // Spawn multiple tasks
- let mut handles = vec![];
-
- for i in 0..3 {
- let barrier = barrier.clone();
- let handle = tokio::spawn(async move {
- println!("Task {} doing work...", i);
-
- // Simulate different work durations
- tokio::time::sleep(Duration::from_secs(i + 1)).await;
-
- println!("Task {} waiting at barrier", i);
- barrier.wait().await.unwrap();
-
- println!("Task {} passed barrier", i);
- });
-
- handles.push(handle);
- }
-
- // Wait for all tasks to complete
- for handle in handles {
- handle.await?;
- }
-
- Ok(())
-}
-```
-
-### Lock with Automatic Renewal
-
-```rust
-use rustfs_lock::{DistributedLock, LockManager, LockOptions};
-use std::time::Duration;
-
-async fn auto_renewal_example() -> Result<(), Box> {
- let lock_manager = LockManager::new("cluster-endpoint").await?;
-
- let lock_options = LockOptions {
- timeout: Duration::from_secs(10),
- auto_renew: true,
- renew_interval: Duration::from_secs(3),
- max_renewals: 5,
- ..Default::default()
- };
-
- let lock = lock_manager.acquire_lock("long-running-task", lock_options).await?;
-
- // Long-running operation
- for i in 0..20 {
- println!("Working on step {}", i);
- tokio::time::sleep(Duration::from_secs(2)).await;
-
- // Check if lock is still valid
- if !lock.is_valid().await? {
- println!("Lock lost, aborting operation");
- break;
- }
- }
-
- lock.release().await?;
- Ok(())
-}
-```
-
-### Hierarchical Locking
-
-```rust
-use rustfs_lock::{LockManager, LockHierarchy, LockOptions};
-
-async fn hierarchical_locking_example() -> Result<(), Box> {
- let lock_manager = LockManager::new("cluster-endpoint").await?;
-
- // Create lock hierarchy
- let hierarchy = LockHierarchy::new(vec![
- "global-lock".to_string(),
- "bucket-lock".to_string(),
- "object-lock".to_string(),
- ]);
-
- // Acquire locks in hierarchy order
- let locks = lock_manager.acquire_hierarchical_locks(
- hierarchy,
- LockOptions::default()
- ).await?;
-
- // Critical section with hierarchical locks
- {
- println!("All hierarchical locks acquired");
- // Perform operations that require the full lock hierarchy
- tokio::time::sleep(Duration::from_secs(1)).await;
- }
-
- // Locks are automatically released in reverse order
- locks.release_all().await?;
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Lock Architecture
-
-```
-Lock Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Lock API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Mutex β Semaphore β Barrier β Condition β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Lock Manager β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Consensus β Heartbeat β Timeout β Recovery β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Distributed Coordination β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Lock Types
-
-| Type | Use Case | Guarantees |
-|------|----------|------------|
-| Exclusive | Critical sections | Mutual exclusion |
-| Shared | Reader-writer | Multiple readers |
-| Semaphore | Resource pooling | Counting semaphore |
-| Barrier | Synchronization | Coordination point |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test distributed locking
-cargo test distributed_lock
-
-# Test synchronization primitives
-cargo test sync_primitives
-
-# Test fault tolerance
-cargo test fault_tolerance
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Network**: Cluster connectivity required
-- **Consensus**: Raft consensus for critical operations
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS Common](../common) - Common types and utilities
-- [RustFS Protos](../protos) - Protocol buffer definitions
+- Distributed lock management across cluster nodes
+- Read-write lock support with concurrent readers
+- Lock timeout and automatic lease renewal
+- Deadlock detection and prevention
+- High-availability with leader election
+- Performance-optimized locking algorithms
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Lock API Reference](https://docs.rustfs.com/lock/)
-- [Distributed Systems Guide](https://docs.rustfs.com/distributed/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/madmin/README.md b/crates/madmin/README.md
index b01634e0..7cc890cc 100644
--- a/crates/madmin/README.md
+++ b/crates/madmin/README.md
@@ -3,7 +3,7 @@
# RustFS MadAdmin - Administrative Interface
- Administrative interface and management APIs for RustFS distributed object storage
+ Advanced administrative interface and management tools for RustFS distributed object storage
@@ -17,335 +17,21 @@
## π Overview
-**RustFS MadAdmin** provides comprehensive administrative interfaces and management APIs for the [RustFS](https://rustfs.com) distributed object storage system. It enables cluster management, monitoring, configuration, and administrative operations through both programmatic APIs and interactive interfaces.
-
-> **Note:** This is a core submodule of RustFS that provides essential administrative capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS MadAdmin** provides advanced administrative interface and management tools for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### ποΈ Cluster Management
-
-- **Node Management**: Add, remove, and monitor cluster nodes
-- **Service Discovery**: Automatic service discovery and registration
-- **Load Balancing**: Distribute load across cluster nodes
-- **Health Monitoring**: Real-time cluster health monitoring
-
-### π System Monitoring
-
-- **Performance Metrics**: CPU, memory, disk, and network metrics
-- **Storage Analytics**: Capacity planning and usage analytics
-- **Alert Management**: Configurable alerts and notifications
-- **Dashboard Interface**: Web-based monitoring dashboard
-
-### βοΈ Configuration Management
-
-- **Dynamic Configuration**: Runtime configuration updates
-- **Policy Management**: Access control and bucket policies
-- **User Management**: User and group administration
-- **Backup Configuration**: Backup and restore settings
-
-### π§ Administrative Operations
-
-- **Data Migration**: Cross-cluster data migration
-- **Healing Operations**: Data integrity repair and healing
-- **Rebalancing**: Storage rebalancing operations
-- **Maintenance Mode**: Graceful maintenance operations
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-madmin = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Admin Client
-
-```rust
-use rustfs_madmin::{AdminClient, AdminConfig, ServerInfo};
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Create admin client
- let config = AdminConfig {
- endpoint: "https://admin.rustfs.local:9001".to_string(),
- access_key: "admin".to_string(),
- secret_key: "password".to_string(),
- region: "us-east-1".to_string(),
- };
-
- let client = AdminClient::new(config).await?;
-
- // Get server information
- let server_info = client.server_info().await?;
- println!("Server Version: {}", server_info.version);
- println!("Uptime: {}", server_info.uptime);
-
- Ok(())
-}
-```
-
-### Cluster Management
-
-```rust
-use rustfs_madmin::{AdminClient, AddServerRequest, RemoveServerRequest};
-
-async fn cluster_management(client: &AdminClient) -> Result<(), Box> {
- // List cluster nodes
- let nodes = client.list_servers().await?;
- for node in nodes {
- println!("Node: {} - Status: {}", node.endpoint, node.state);
- }
-
- // Add new server to cluster
- let add_request = AddServerRequest {
- endpoint: "https://new-node.rustfs.local:9000".to_string(),
- access_key: "node-key".to_string(),
- secret_key: "node-secret".to_string(),
- };
-
- client.add_server(add_request).await?;
- println!("New server added successfully");
-
- // Remove server from cluster
- let remove_request = RemoveServerRequest {
- endpoint: "https://old-node.rustfs.local:9000".to_string(),
- };
-
- client.remove_server(remove_request).await?;
- println!("Server removed successfully");
-
- Ok(())
-}
-```
-
-### System Monitoring
-
-```rust
-use rustfs_madmin::{AdminClient, MetricsRequest, AlertConfig};
-
-async fn monitoring_operations(client: &AdminClient) -> Result<(), Box> {
- // Get system metrics
- let metrics = client.get_metrics(MetricsRequest::default()).await?;
-
- println!("CPU Usage: {:.2}%", metrics.cpu_usage);
- println!("Memory Usage: {:.2}%", metrics.memory_usage);
- println!("Disk Usage: {:.2}%", metrics.disk_usage);
-
- // Get storage information
- let storage_info = client.storage_info().await?;
- println!("Total Capacity: {} GB", storage_info.total_capacity / 1024 / 1024 / 1024);
- println!("Used Capacity: {} GB", storage_info.used_capacity / 1024 / 1024 / 1024);
-
- // Configure alerts
- let alert_config = AlertConfig {
- name: "high-cpu-usage".to_string(),
- condition: "cpu_usage > 80".to_string(),
- notification_endpoint: "https://webhook.example.com/alerts".to_string(),
- enabled: true,
- };
-
- client.set_alert_config(alert_config).await?;
-
- Ok(())
-}
-```
-
-### User and Policy Management
-
-```rust
-use rustfs_madmin::{AdminClient, UserInfo, PolicyDocument};
-
-async fn user_management(client: &AdminClient) -> Result<(), Box> {
- // Create user
- let user_info = UserInfo {
- access_key: "user123".to_string(),
- secret_key: "user-secret".to_string(),
- status: "enabled".to_string(),
- policy: Some("readwrite-policy".to_string()),
- };
-
- client.add_user("new-user", user_info).await?;
-
- // List users
- let users = client.list_users().await?;
- for (username, info) in users {
- println!("User: {} - Status: {}", username, info.status);
- }
-
- // Set user policy
- let policy_doc = PolicyDocument {
- version: "2012-10-17".to_string(),
- statement: vec![/* policy statements */],
- };
-
- client.set_user_policy("new-user", policy_doc).await?;
-
- Ok(())
-}
-```
-
-### Data Operations
-
-```rust
-use rustfs_madmin::{AdminClient, HealRequest, RebalanceRequest};
-
-async fn data_operations(client: &AdminClient) -> Result<(), Box> {
- // Start healing operation
- let heal_request = HealRequest {
- bucket: Some("important-bucket".to_string()),
- prefix: Some("documents/".to_string()),
- recursive: true,
- dry_run: false,
- };
-
- let heal_result = client.heal(heal_request).await?;
- println!("Healing started: {}", heal_result.heal_sequence);
-
- // Check healing status
- let heal_status = client.heal_status(&heal_result.heal_sequence).await?;
- println!("Healing progress: {:.2}%", heal_status.progress);
-
- // Start rebalancing
- let rebalance_request = RebalanceRequest {
- servers: vec![], // Empty means all servers
- };
-
- client.start_rebalance(rebalance_request).await?;
- println!("Rebalancing started");
-
- Ok(())
-}
-```
-
-### Configuration Management
-
-```rust
-use rustfs_madmin::{AdminClient, ConfigUpdate, NotificationTarget};
-
-async fn configuration_management(client: &AdminClient) -> Result<(), Box> {
- // Get current configuration
- let config = client.get_config().await?;
- println!("Current config version: {}", config.version);
-
- // Update configuration
- let config_update = ConfigUpdate {
- region: Some("us-west-2".to_string()),
- browser: Some(true),
- compression: Some(true),
- // ... other config fields
- };
-
- client.set_config(config_update).await?;
-
- // Configure notification targets
- let notification_target = NotificationTarget {
- arn: "arn:aws:sns:us-east-1:123456789012:my-topic".to_string(),
- target_type: "webhook".to_string(),
- endpoint: "https://webhook.example.com/notifications".to_string(),
- };
-
- client.set_notification_target("bucket1", notification_target).await?;
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### MadAdmin Architecture
-
-```
-MadAdmin Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Admin API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Cluster Mgmt β Monitoring β User Management β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Data Ops β Config Mgmt β Notification β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β HTTP/gRPC Client Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Storage System Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Administrative Operations
-
-| Category | Operations | Description |
-|----------|------------|-------------|
-| Cluster | Add/Remove nodes, Health checks | Cluster management |
-| Monitoring | Metrics, Alerts, Dashboard | System monitoring |
-| Data | Healing, Rebalancing, Migration | Data operations |
-| Config | Settings, Policies, Notifications | Configuration |
-| Users | Authentication, Authorization | User management |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test admin operations
-cargo test admin_ops
-
-# Test cluster management
-cargo test cluster
-
-# Test monitoring
-cargo test monitoring
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Network**: Administrative access to RustFS cluster
-- **Permissions**: Administrative credentials required
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS IAM](../iam) - Identity and access management
-- [RustFS Policy](../policy) - Policy engine
-- [RustFS Common](../common) - Common types and utilities
+- Comprehensive cluster management and monitoring
+- Real-time performance metrics and analytics
+- Automated backup and disaster recovery tools
+- User and permission management interface
+- System health monitoring and alerting
+- Configuration management and deployment tools
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [MadAdmin API Reference](https://docs.rustfs.com/madmin/)
-- [Administrative Guide](https://docs.rustfs.com/admin/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with ποΈ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/notify/README.md b/crates/notify/README.md
index 2e9c8a95..8b83fcaa 100644
--- a/crates/notify/README.md
+++ b/crates/notify/README.md
@@ -3,7 +3,7 @@
# RustFS Notify - Event Notification System
- Real-time event notification system for RustFS distributed object storage
+ Real-time event notification and messaging system for RustFS distributed object storage
@@ -17,399 +17,21 @@
## π Overview
-**RustFS Notify** is the event notification system for the [RustFS](https://rustfs.com) distributed object storage platform. It provides real-time event publishing and delivery to various targets including webhooks, MQTT brokers, and message queues, enabling seamless integration with external systems and workflows.
-
-> **Note:** This is a core submodule of RustFS that provides essential event notification capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Notify** provides real-time event notification and messaging capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π‘ Event Publishing
-
-- **Real-time Events**: Instant notification of storage events
-- **Event Filtering**: Advanced filtering based on object patterns and event types
-- **Reliable Delivery**: Guaranteed delivery with retry mechanisms
-- **Batch Processing**: Efficient batch event delivery
-
-### π― Multiple Targets
-
-- **Webhooks**: HTTP/HTTPS webhook notifications
-- **MQTT**: MQTT broker integration for IoT scenarios
-- **Message Queues**: Integration with popular message queue systems
-- **Custom Targets**: Extensible target system for custom integrations
-
-### π§ Advanced Features
-
-- **Event Transformation**: Custom event payload transformation
-- **Pattern Matching**: Flexible pattern-based event filtering
-- **Rate Limiting**: Configurable rate limiting for targets
-- **Dead Letter Queue**: Failed event handling and recovery
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-notify = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Event Notification
-
-```rust
-use rustfs_notify::{Event, EventType, NotificationTarget, NotifySystem};
-use rustfs_notify::target::WebhookTarget;
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Create notification system
- let notify_system = NotifySystem::new().await?;
-
- // Create webhook target
- let webhook = WebhookTarget::new(
- "webhook-1",
- "https://api.example.com/webhook",
- vec![EventType::ObjectCreated, EventType::ObjectRemoved],
- );
-
- // Add target to notification system
- notify_system.add_target(Box::new(webhook)).await?;
-
- // Create and send event
- let event = Event::new(
- EventType::ObjectCreated,
- "my-bucket",
- "path/to/object.txt",
- "user123",
- );
-
- notify_system.publish_event(event).await?;
-
- Ok(())
-}
-```
-
-### Advanced Event Configuration
-
-```rust
-use rustfs_notify::{Event, EventType, NotificationConfig};
-use rustfs_notify::target::{WebhookTarget, MqttTarget};
-use rustfs_notify::filter::{EventFilter, PatternRule};
-
-async fn setup_advanced_notifications() -> Result<(), Box> {
- let notify_system = NotifySystem::new().await?;
-
- // Create webhook with custom configuration
- let webhook_config = NotificationConfig {
- retry_attempts: 3,
- retry_delay: std::time::Duration::from_secs(5),
- timeout: std::time::Duration::from_secs(30),
- rate_limit: Some(100), // 100 events per minute
- ..Default::default()
- };
-
- let webhook = WebhookTarget::builder()
- .id("production-webhook")
- .url("https://api.example.com/events")
- .events(vec![EventType::ObjectCreated, EventType::ObjectRemoved])
- .config(webhook_config)
- .headers(vec![
- ("Authorization".to_string(), "Bearer token123".to_string()),
- ("Content-Type".to_string(), "application/json".to_string()),
- ])
- .build()?;
-
- // Create MQTT target
- let mqtt_target = MqttTarget::builder()
- .id("iot-mqtt")
- .broker_url("mqtt://broker.example.com:1883")
- .topic("storage/events")
- .qos(1)
- .events(vec![EventType::ObjectCreated])
- .build()?;
-
- // Add targets
- notify_system.add_target(Box::new(webhook)).await?;
- notify_system.add_target(Box::new(mqtt_target)).await?;
-
- Ok(())
-}
-```
-
-### Event Filtering and Pattern Matching
-
-```rust
-use rustfs_notify::filter::{EventFilter, PatternRule, ConditionRule};
-
-fn setup_event_filters() -> Result> {
- let filter = EventFilter::builder()
- // Only images and documents
- .pattern_rule(PatternRule::new(
- "suffix",
- vec!["*.jpg", "*.png", "*.pdf", "*.doc"]
- ))
- // Exclude temporary files
- .pattern_rule(PatternRule::new(
- "exclude",
- vec!["*/tmp/*", "*.tmp"]
- ))
- // Only files larger than 1MB
- .condition_rule(ConditionRule::new(
- "object_size",
- ">",
- 1024 * 1024
- ))
- // Only from specific buckets
- .bucket_filter(vec!["important-bucket", "backup-bucket"])
- .build()?;
-
- Ok(filter)
-}
-```
-
-### Custom Target Implementation
-
-```rust
-use rustfs_notify::{Event, NotificationTarget, TargetResult};
-use async_trait::async_trait;
-
-pub struct SlackTarget {
- id: String,
- webhook_url: String,
- channel: String,
-}
-
-#[async_trait]
-impl NotificationTarget for SlackTarget {
- fn id(&self) -> &str {
- &self.id
- }
-
- async fn deliver_event(&self, event: &Event) -> TargetResult<()> {
- let message = format!(
- "π Storage Event: {} in bucket `{}` - object `{}`",
- event.event_type,
- event.bucket_name,
- event.object_name
- );
-
- let payload = serde_json::json!({
- "text": message,
- "channel": self.channel,
- });
-
- let client = reqwest::Client::new();
- let response = client
- .post(&self.webhook_url)
- .json(&payload)
- .send()
- .await?;
-
- if response.status().is_success() {
- Ok(())
- } else {
- Err(format!("Slack delivery failed: {}", response.status()).into())
- }
- }
-
- fn supports_event_type(&self, event_type: &EventType) -> bool {
- // Support all event types
- true
- }
-}
-```
-
-### Event Transformation
-
-```rust
-use rustfs_notify::{Event, EventTransformer};
-use serde_json::{json, Value};
-
-pub struct CustomEventTransformer;
-
-impl EventTransformer for CustomEventTransformer {
- fn transform(&self, event: &Event) -> Value {
- json!({
- "eventVersion": "2.1",
- "eventSource": "rustfs:s3",
- "eventTime": event.timestamp.to_rfc3339(),
- "eventName": event.event_type.to_string(),
- "s3": {
- "bucket": {
- "name": event.bucket_name,
- "arn": format!("arn:aws:s3:::{}", event.bucket_name)
- },
- "object": {
- "key": event.object_name,
- "size": event.object_size.unwrap_or(0),
- "eTag": event.etag.as_ref().unwrap_or(&"".to_string()),
- }
- },
- "userIdentity": {
- "principalId": event.user_identity
- }
- })
- }
-}
-```
-
-## ποΈ Architecture
-
-### Notification System Architecture
-
-```
-Notify Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Event Publisher β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Event Filter β Event Queue β Event Transformer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Target Manager β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Webhook Target β MQTT Target β Custom Targets β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Delivery Engine β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Supported Event Types
-
-| Event Type | Description | Triggers |
-|-----------|-------------|----------|
-| `ObjectCreated` | Object creation events | PUT, POST, COPY |
-| `ObjectRemoved` | Object deletion events | DELETE |
-| `ObjectAccessed` | Object access events | GET, HEAD |
-| `ObjectRestore` | Object restoration events | Restore operations |
-| `BucketCreated` | Bucket creation events | CreateBucket |
-| `BucketRemoved` | Bucket deletion events | DeleteBucket |
-
-### Target Types
-
-| Target | Protocol | Use Case | Reliability |
-|--------|----------|----------|------------|
-| Webhook | HTTP/HTTPS | Web applications, APIs | High |
-| MQTT | MQTT | IoT devices, real-time systems | Medium |
-| Message Queue | AMQP, Redis | Microservices, async processing | High |
-| Custom | Any | Specialized integrations | Configurable |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Run integration tests
-cargo test --test integration
-
-# Test webhook delivery
-cargo test webhook
-
-# Test MQTT integration
-cargo test mqtt
-
-# Run with coverage
-cargo test --features test-coverage
-```
-
-## βοΈ Configuration
-
-### Basic Configuration
-
-```toml
-[notify]
-# Global settings
-enabled = true
-max_concurrent_deliveries = 100
-default_retry_attempts = 3
-default_timeout = "30s"
-
-# Queue settings
-event_queue_size = 10000
-batch_size = 100
-batch_timeout = "5s"
-
-# Dead letter queue
-dlq_enabled = true
-dlq_max_size = 1000
-```
-
-### Target Configuration
-
-```toml
-[[notify.targets]]
-type = "webhook"
-id = "primary-webhook"
-url = "https://api.example.com/webhook"
-events = ["ObjectCreated", "ObjectRemoved"]
-retry_attempts = 5
-timeout = "30s"
-
-[[notify.targets]]
-type = "mqtt"
-id = "iot-broker"
-broker_url = "mqtt://broker.example.com:1883"
-topic = "storage/events"
-qos = 1
-events = ["ObjectCreated"]
-```
-
-## π Performance
-
-The notification system is designed for high-throughput scenarios:
-
-- **Async Processing**: Non-blocking event delivery
-- **Batch Delivery**: Efficient batch processing for high-volume events
-- **Connection Pooling**: Reused connections for better performance
-- **Rate Limiting**: Configurable rate limiting to prevent overwhelming targets
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Network**: Outbound connectivity for target delivery
-- **Memory**: Scales with event queue size
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS ECStore](../ecstore) - Erasure coding storage engine
-- [RustFS Config](../config) - Configuration management
-- [RustFS Utils](../utils) - Utility functions
+- Real-time event streaming and notifications
+- Multiple notification targets (HTTP, Kafka, Redis, Email)
+- Event filtering and routing based on criteria
+- Message queuing with guaranteed delivery
+- Event replay and auditing capabilities
+- High-throughput messaging with batching support
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Notify API Reference](https://docs.rustfs.com/notify/)
-- [Event Configuration Guide](https://docs.rustfs.com/events/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π‘ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/obs/README.md b/crates/obs/README.md
index de9a62f1..be3d6df6 100644
--- a/crates/obs/README.md
+++ b/crates/obs/README.md
@@ -3,7 +3,7 @@
# RustFS Obs - Observability & Monitoring
- Comprehensive observability and monitoring solution for RustFS distributed object storage
+ Comprehensive observability and monitoring system for RustFS distributed object storage
@@ -17,457 +17,21 @@
## π Overview
-**RustFS Obs** provides comprehensive observability and monitoring capabilities for the [RustFS](https://rustfs.com) distributed object storage system. It includes metrics collection, distributed tracing, logging, alerting, and performance monitoring to ensure optimal system operation and troubleshooting.
-
-> **Note:** This is a critical operational submodule of RustFS that provides essential observability capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Obs** provides comprehensive observability and monitoring capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π Metrics Collection
-
-- **Prometheus Integration**: Native Prometheus metrics export
-- **Custom Metrics**: Application-specific performance metrics
-- **System Metrics**: CPU, memory, disk, and network monitoring
-- **Business Metrics**: Storage usage, request rates, and error tracking
-
-### π Distributed Tracing
-
-- **OpenTelemetry Support**: Standard distributed tracing
-- **Request Tracking**: End-to-end request lifecycle tracking
-- **Performance Analysis**: Latency and bottleneck identification
-- **Cross-Service Correlation**: Trace requests across microservices
-
-### π Structured Logging
-
-- **JSON Logging**: Machine-readable structured logs
-- **Log Levels**: Configurable log levels and filtering
-- **Context Propagation**: Request context in all logs
-- **Log Aggregation**: Centralized log collection support
-
-### π¨ Alerting & Notifications
-
-- **Rule-Based Alerts**: Configurable alerting rules
-- **Multiple Channels**: Email, Slack, webhook notifications
-- **Alert Escalation**: Tiered alerting and escalation policies
-- **Alert Correlation**: Group related alerts together
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-obs = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Observability Setup
-
-```rust
-use rustfs_obs::{ObservabilityConfig, MetricsCollector, TracingProvider};
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Configure observability
- let config = ObservabilityConfig {
- service_name: "rustfs-storage".to_string(),
- metrics_endpoint: "http://prometheus:9090".to_string(),
- tracing_endpoint: "http://jaeger:14268/api/traces".to_string(),
- log_level: "info".to_string(),
- enable_metrics: true,
- enable_tracing: true,
- };
-
- // Initialize observability
- let obs = rustfs_obs::init(config).await?;
-
- // Your application code here
- run_application().await?;
-
- // Shutdown observability
- obs.shutdown().await?;
-
- Ok(())
-}
-```
-
-### Metrics Collection
-
-```rust
-use rustfs_obs::metrics::{Counter, Histogram, Gauge, register_counter};
-
-// Define metrics
-lazy_static! {
- static ref REQUESTS_TOTAL: Counter = register_counter!(
- "rustfs_requests_total",
- "Total number of requests",
- &["method", "status"]
- ).unwrap();
-
- static ref REQUEST_DURATION: Histogram = register_histogram!(
- "rustfs_request_duration_seconds",
- "Request duration in seconds",
- &["method"]
- ).unwrap();
-
- static ref ACTIVE_CONNECTIONS: Gauge = register_gauge!(
- "rustfs_active_connections",
- "Number of active connections"
- ).unwrap();
-}
-
-async fn handle_request(method: &str) -> Result<(), Box> {
- let _timer = REQUEST_DURATION.with_label_values(&[method]).start_timer();
-
- // Increment active connections
- ACTIVE_CONNECTIONS.inc();
-
- // Simulate request processing
- tokio::time::sleep(Duration::from_millis(100)).await;
-
- // Record request completion
- REQUESTS_TOTAL.with_label_values(&[method, "success"]).inc();
-
- // Decrement active connections
- ACTIVE_CONNECTIONS.dec();
-
- Ok(())
-}
-```
-
-### Distributed Tracing
-
-```rust
-use rustfs_obs::tracing::{trace_fn, Span, SpanContext};
-use tracing::{info, instrument};
-
-#[instrument(skip(data))]
-async fn process_upload(bucket: &str, key: &str, data: &[u8]) -> Result> {
- let span = Span::current();
- span.set_attribute("bucket", bucket);
- span.set_attribute("key", key);
- span.set_attribute("size", data.len() as i64);
-
- info!("Starting upload process");
-
- // Validate data
- let validation_result = validate_data(data).await?;
- span.add_event("data_validated", &[("result", &validation_result)]);
-
- // Store data
- let storage_result = store_data(bucket, key, data).await?;
- span.add_event("data_stored", &[("etag", &storage_result.etag)]);
-
- // Update metadata
- update_metadata(bucket, key, &storage_result).await?;
- span.add_event("metadata_updated", &[]);
-
- info!("Upload completed successfully");
- Ok(storage_result.etag)
-}
-
-#[instrument]
-async fn validate_data(data: &[u8]) -> Result> {
- // Validation logic
- tokio::time::sleep(Duration::from_millis(50)).await;
- Ok("valid".to_string())
-}
-
-#[instrument]
-async fn store_data(bucket: &str, key: &str, data: &[u8]) -> Result> {
- // Storage logic
- tokio::time::sleep(Duration::from_millis(200)).await;
- Ok(StorageResult {
- etag: "d41d8cd98f00b204e9800998ecf8427e".to_string(),
- })
-}
-```
-
-### Structured Logging
-
-```rust
-use rustfs_obs::logging::{LogEvent, LogLevel, StructuredLogger};
-use serde_json::json;
-
-async fn logging_example() -> Result<(), Box> {
- let logger = StructuredLogger::new();
-
- // Basic logging
- logger.info("Application started").await;
-
- // Structured logging with context
- logger.log(LogEvent {
- level: LogLevel::Info,
- message: "Processing upload request".to_string(),
- context: json!({
- "bucket": "example-bucket",
- "key": "example-object",
- "size": 1024,
- "user_id": "user123",
- "request_id": "req-456"
- }),
- timestamp: chrono::Utc::now(),
- }).await;
-
- // Error logging with details
- logger.error_with_context(
- "Failed to process upload",
- json!({
- "error_code": "STORAGE_FULL",
- "bucket": "example-bucket",
- "available_space": 0,
- "required_space": 1024
- })
- ).await;
-
- Ok(())
-}
-```
-
-### Alerting Configuration
-
-```rust
-use rustfs_obs::alerting::{AlertManager, AlertRule, NotificationChannel};
-
-async fn setup_alerting() -> Result<(), Box> {
- let alert_manager = AlertManager::new().await?;
-
- // Configure notification channels
- let slack_channel = NotificationChannel::Slack {
- webhook_url: "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK".to_string(),
- channel: "#rustfs-alerts".to_string(),
- };
-
- let email_channel = NotificationChannel::Email {
- smtp_server: "smtp.example.com".to_string(),
- recipients: vec!["admin@example.com".to_string()],
- };
-
- alert_manager.add_notification_channel("slack", slack_channel).await?;
- alert_manager.add_notification_channel("email", email_channel).await?;
-
- // Define alert rules
- let high_error_rate = AlertRule {
- name: "high_error_rate".to_string(),
- description: "High error rate detected".to_string(),
- condition: "rate(rustfs_requests_total{status!=\"success\"}[5m]) > 0.1".to_string(),
- severity: "critical".to_string(),
- notifications: vec!["slack".to_string(), "email".to_string()],
- cooldown: Duration::from_minutes(15),
- };
-
- let low_disk_space = AlertRule {
- name: "low_disk_space".to_string(),
- description: "Disk space running low".to_string(),
- condition: "rustfs_disk_usage_percent > 85".to_string(),
- severity: "warning".to_string(),
- notifications: vec!["slack".to_string()],
- cooldown: Duration::from_minutes(30),
- };
-
- alert_manager.add_rule(high_error_rate).await?;
- alert_manager.add_rule(low_disk_space).await?;
-
- // Start alert monitoring
- alert_manager.start().await?;
-
- Ok(())
-}
-```
-
-### Performance Monitoring
-
-```rust
-use rustfs_obs::monitoring::{PerformanceMonitor, SystemMetrics, ApplicationMetrics};
-
-async fn performance_monitoring() -> Result<(), Box> {
- let monitor = PerformanceMonitor::new().await?;
-
- // Start system monitoring
- monitor.start_system_monitoring(Duration::from_secs(10)).await?;
-
- // Custom application metrics
- let app_metrics = ApplicationMetrics::new();
-
- // Monitor specific operations
- let upload_metrics = app_metrics.create_operation_monitor("upload");
- let download_metrics = app_metrics.create_operation_monitor("download");
-
- // Simulate operations with monitoring
- tokio::spawn(async move {
- loop {
- // Monitor upload operation
- let upload_timer = upload_metrics.start_timer();
- simulate_upload().await;
- upload_timer.record_success();
-
- // Monitor download operation
- let download_timer = download_metrics.start_timer();
- match simulate_download().await {
- Ok(_) => download_timer.record_success(),
- Err(_) => download_timer.record_error(),
- }
-
- tokio::time::sleep(Duration::from_secs(1)).await;
- }
- });
-
- // Periodic metrics reporting
- tokio::spawn(async move {
- let mut interval = tokio::time::interval(Duration::from_secs(60));
-
- loop {
- interval.tick().await;
-
- let system_metrics = monitor.get_system_metrics().await;
- let app_metrics = monitor.get_application_metrics().await;
-
- println!("=== System Metrics ===");
- println!("CPU Usage: {:.2}%", system_metrics.cpu_usage);
- println!("Memory Usage: {:.2}%", system_metrics.memory_usage);
- println!("Disk Usage: {:.2}%", system_metrics.disk_usage);
-
- println!("=== Application Metrics ===");
- println!("Upload Throughput: {:.2} ops/sec", app_metrics.upload_throughput);
- println!("Download Throughput: {:.2} ops/sec", app_metrics.download_throughput);
- println!("Error Rate: {:.2}%", app_metrics.error_rate);
- }
- });
-
- Ok(())
-}
-```
-
-### Health Checks
-
-```rust
-use rustfs_obs::health::{HealthChecker, HealthStatus, HealthCheck};
-
-async fn setup_health_checks() -> Result<(), Box> {
- let health_checker = HealthChecker::new();
-
- // Add component health checks
- health_checker.add_check("database", Box::new(DatabaseHealthCheck)).await;
- health_checker.add_check("storage", Box::new(StorageHealthCheck)).await;
- health_checker.add_check("cache", Box::new(CacheHealthCheck)).await;
-
- // Start health monitoring
- health_checker.start_monitoring(Duration::from_secs(30)).await?;
-
- // Expose health endpoint
- health_checker.expose_http_endpoint("0.0.0.0:8080").await?;
-
- Ok(())
-}
-
-struct DatabaseHealthCheck;
-
-#[async_trait::async_trait]
-impl HealthCheck for DatabaseHealthCheck {
- async fn check(&self) -> HealthStatus {
- // Perform database health check
- match check_database_connection().await {
- Ok(_) => HealthStatus::Healthy,
- Err(e) => HealthStatus::Unhealthy(e.to_string()),
- }
- }
-}
-```
-
-## ποΈ Architecture
-
-### Observability Architecture
-
-```
-Observability Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Observability API β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Metrics β Tracing β Logging β Alerting β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Data Collection & Processing β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Prometheus β OpenTelemetry β Structured β Alert Mgr β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β External Integrations β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Monitoring Stack
-
-| Component | Purpose | Integration |
-|-----------|---------|-------------|
-| Prometheus | Metrics storage | Pull-based metrics collection |
-| Jaeger | Distributed tracing | OpenTelemetry traces |
-| Grafana | Visualization | Dashboards and alerts |
-| ELK Stack | Log aggregation | Structured log processing |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test metrics collection
-cargo test metrics
-
-# Test tracing functionality
-cargo test tracing
-
-# Test alerting
-cargo test alerting
-
-# Integration tests
-cargo test --test integration
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **External Services**: Prometheus, Jaeger (optional)
-- **Network**: HTTP endpoint exposure capability
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS Common](../common) - Common types and utilities
-- [RustFS Config](../config) - Configuration management
+- OpenTelemetry integration for distributed tracing
+- Prometheus metrics collection and exposition
+- Structured logging with configurable levels
+- Performance profiling and analytics
+- Real-time health checks and status monitoring
+- Custom dashboards and alerting integration
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Obs API Reference](https://docs.rustfs.com/obs/)
-- [Monitoring Guide](https://docs.rustfs.com/monitoring/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/policy/README.md b/crates/policy/README.md
index ebed0332..0da2bbb6 100644
--- a/crates/policy/README.md
+++ b/crates/policy/README.md
@@ -1,9 +1,9 @@
[](https://rustfs.com)
-# RustFS Policy Engine
+# RustFS Policy - Policy Engine
- Advanced policy-based access control engine for RustFS distributed object storage
+ Advanced policy engine and access control system for RustFS distributed object storage
@@ -17,574 +17,21 @@
## π Overview
-**RustFS Policy Engine** is a sophisticated access control system for the [RustFS](https://rustfs.com) distributed object storage platform. It provides fine-grained, attribute-based access control (ABAC) with support for complex policy expressions, dynamic evaluation, and AWS IAM-compatible policy syntax.
-
-> **Note:** This is a core submodule of RustFS that provides essential access control and authorization capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Policy** provides advanced policy engine and access control capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π Access Control
-
-- **AWS IAM Compatible**: Full support for AWS IAM policy syntax
-- **Fine-Grained Permissions**: Resource-level and action-level access control
-- **Dynamic Policy Evaluation**: Real-time policy evaluation with context
-- **Conditional Access**: Support for complex conditional expressions
-
-### π Policy Management
-
-- **Policy Documents**: Structured policy definition and management
-- **Policy Versioning**: Version control for policy documents
-- **Policy Validation**: Syntax and semantic validation
-- **Policy Templates**: Pre-built policy templates for common use cases
-
-### π― Advanced Features
-
-- **Attribute-Based Access Control (ABAC)**: Context-aware access decisions
-- **Function-Based Conditions**: Rich set of condition functions
-- **Principal-Based Policies**: User, group, and service account policies
-- **Resource-Based Policies**: Bucket and object-level policies
-
-### π οΈ Integration Features
-
-- **ARN Support**: AWS-style Amazon Resource Names
-- **Multi-Tenant Support**: Isolated policy evaluation per tenant
-- **Real-Time Evaluation**: High-performance policy evaluation engine
-- **Audit Trail**: Comprehensive policy evaluation logging
-
-## ποΈ Architecture
-
-### Policy Engine Architecture
-
-```
-Policy Engine Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Policy API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Policy Parser β Policy Validator β Policy Store β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Policy Evaluation Engine β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Condition Functions β Principal Resolver β Resource Mgr β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Authentication Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Policy Decision Flow
-
-```
-Policy Decision Flow:
-βββββββββββββββ βββββββββββββββ βββββββββββββββ
-β Request βββββΆβ Policy βββββΆβ Decision β
-β (Subject, β β Evaluation β β (Allow/ β
-β Action, β β Engine β β Deny/ β
-β Resource) β β β β Not Found) β
-βββββββββββββββ βββββββββββββββ βββββββββββββββ
- β β β
- βΌ βΌ βΌ
-βββββββββββββββ βββββββββββββββ βββββββββββββββ
-β Context β β Condition β β Audit β
-β Information β β Functions β β Log β
-βββββββββββββββ βββββββββββββββ βββββββββββββββ
-```
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-policy = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Policy Creation
-
-```rust
-use rustfs_policy::policy::{Policy, Statement, Effect, Action, Resource};
-use rustfs_policy::arn::ARN;
-use serde_json::json;
-
-fn main() -> Result<(), Box> {
- // Create a simple policy
- let policy = Policy::new(
- "2012-10-17".to_string(),
- vec![
- Statement::new(
- Effect::Allow,
- vec![Action::from_str("s3:GetObject")?],
- vec![Resource::from_str("arn:aws:s3:::my-bucket/*")?],
- None, // No conditions
- ),
- ],
- );
-
- // Serialize to JSON
- let policy_json = serde_json::to_string_pretty(&policy)?;
- println!("Policy JSON:\n{}", policy_json);
-
- Ok(())
-}
-```
-
-### Advanced Policy with Conditions
-
-```rust
-use rustfs_policy::policy::{Policy, Statement, Effect, Action, Resource};
-use rustfs_policy::policy::function::{Function, FunctionName};
-use serde_json::json;
-
-fn create_conditional_policy() -> Result> {
- // Create a policy with IP address restrictions
- let policy = Policy::new(
- "2012-10-17".to_string(),
- vec![
- Statement::builder()
- .effect(Effect::Allow)
- .action(Action::from_str("s3:GetObject")?)
- .resource(Resource::from_str("arn:aws:s3:::secure-bucket/*")?)
- .condition(
- "IpAddress",
- "aws:SourceIp",
- json!(["192.168.1.0/24", "10.0.0.0/8"])
- )
- .build(),
- Statement::builder()
- .effect(Effect::Deny)
- .action(Action::from_str("s3:*")?)
- .resource(Resource::from_str("arn:aws:s3:::secure-bucket/*")?)
- .condition(
- "DateGreaterThan",
- "aws:CurrentTime",
- json!("2024-12-31T23:59:59Z")
- )
- .build(),
- ],
- );
-
- Ok(policy)
-}
-```
-
-### Policy Evaluation
-
-```rust
-use rustfs_policy::policy::{Policy, PolicyDoc};
-use rustfs_policy::auth::{Identity, Request};
-use rustfs_policy::arn::ARN;
-
-async fn evaluate_policy_example() -> Result<(), Box> {
- // Load policy from storage
- let policy_doc = PolicyDoc::try_from(policy_bytes)?;
-
- // Create evaluation context
- let identity = Identity::new(
- "user123".to_string(),
- vec!["group1".to_string(), "group2".to_string()],
- );
-
- let request = Request::new(
- "s3:GetObject".to_string(),
- ARN::from_str("arn:aws:s3:::my-bucket/file.txt")?,
- Some(json!({
- "aws:SourceIp": "192.168.1.100",
- "aws:CurrentTime": "2024-01-15T10:30:00Z"
- })),
- );
-
- // Evaluate policy
- let result = policy_doc.policy.evaluate(&identity, &request).await?;
-
- match result {
- Effect::Allow => println!("Access allowed"),
- Effect::Deny => println!("Access denied"),
- Effect::NotEvaluated => println!("No applicable policy found"),
- }
-
- Ok(())
-}
-```
-
-### Policy Templates
-
-```rust
-use rustfs_policy::policy::{Policy, Statement, Effect};
-use rustfs_policy::templates::PolicyTemplate;
-
-fn create_common_policies() -> Result<(), Box> {
- // Read-only policy template
- let read_only_policy = PolicyTemplate::read_only_bucket("my-bucket")?;
-
- // Full access policy template
- let full_access_policy = PolicyTemplate::full_access_bucket("my-bucket")?;
-
- // Admin policy template
- let admin_policy = PolicyTemplate::admin_all_resources()?;
-
- // Custom policy with multiple permissions
- let custom_policy = Policy::builder()
- .version("2012-10-17")
- .statement(
- Statement::builder()
- .effect(Effect::Allow)
- .action("s3:GetObject")
- .action("s3:PutObject")
- .resource("arn:aws:s3:::uploads/*")
- .condition("StringEquals", "s3:x-amz-acl", "bucket-owner-full-control")
- .build()
- )
- .statement(
- Statement::builder()
- .effect(Effect::Allow)
- .action("s3:ListBucket")
- .resource("arn:aws:s3:::uploads")
- .condition("StringLike", "s3:prefix", "user/${aws:username}/*")
- .build()
- )
- .build();
-
- Ok(())
-}
-```
-
-### Resource-Based Policies
-
-```rust
-use rustfs_policy::policy::{Policy, Statement, Effect, Principal};
-use rustfs_policy::arn::ARN;
-
-fn create_resource_policy() -> Result> {
- // Create a bucket policy allowing cross-account access
- let bucket_policy = Policy::builder()
- .version("2012-10-17")
- .statement(
- Statement::builder()
- .effect(Effect::Allow)
- .principal(Principal::AWS("arn:aws:iam::123456789012:root".to_string()))
- .action("s3:GetObject")
- .resource("arn:aws:s3:::shared-bucket/*")
- .condition("StringEquals", "s3:ExistingObjectTag/Department", "Finance")
- .build()
- )
- .statement(
- Statement::builder()
- .effect(Effect::Allow)
- .principal(Principal::AWS("arn:aws:iam::123456789012:user/john".to_string()))
- .action("s3:PutObject")
- .resource("arn:aws:s3:::shared-bucket/uploads/*")
- .condition("StringEquals", "s3:x-amz-acl", "bucket-owner-full-control")
- .build()
- )
- .build();
-
- Ok(bucket_policy)
-}
-```
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Run policy evaluation tests
-cargo test policy_evaluation
-
-# Run condition function tests
-cargo test condition_functions
-
-# Run ARN parsing tests
-cargo test arn_parsing
-
-# Run policy validation tests
-cargo test policy_validation
-
-# Run with test coverage
-cargo test --features test-coverage
-```
-
-## π Policy Syntax
-
-### Basic Policy Structure
-
-```json
-{
- "Version": "2012-10-17",
- "Statement": [
- {
- "Effect": "Allow",
- "Action": [
- "s3:GetObject",
- "s3:PutObject"
- ],
- "Resource": [
- "arn:aws:s3:::my-bucket/*"
- ],
- "Condition": {
- "StringEquals": {
- "s3:x-amz-acl": "bucket-owner-full-control"
- }
- }
- }
- ]
-}
-```
-
-### Supported Actions
-
-| Action Category | Actions | Description |
-|----------------|---------|-------------|
-| Object Operations | `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject` | Object-level operations |
-| Bucket Operations | `s3:ListBucket`, `s3:CreateBucket`, `s3:DeleteBucket` | Bucket-level operations |
-| Access Control | `s3:GetBucketAcl`, `s3:PutBucketAcl` | Access control operations |
-| Versioning | `s3:GetObjectVersion`, `s3:DeleteObjectVersion` | Object versioning operations |
-| Multipart Upload | `s3:ListMultipartUploads`, `s3:AbortMultipartUpload` | Multipart upload operations |
-
-### Condition Functions
-
-| Function | Description | Example |
-|----------|-------------|---------|
-| `StringEquals` | Exact string matching | `"StringEquals": {"s3:x-amz-acl": "private"}` |
-| `StringLike` | Wildcard string matching | `"StringLike": {"s3:prefix": "photos/*"}` |
-| `IpAddress` | IP address/CIDR matching | `"IpAddress": {"aws:SourceIp": "192.168.1.0/24"}` |
-| `DateGreaterThan` | Date comparison | `"DateGreaterThan": {"aws:CurrentTime": "2024-01-01T00:00:00Z"}` |
-| `NumericEquals` | Numeric comparison | `"NumericEquals": {"s3:max-keys": "100"}` |
-| `Bool` | Boolean comparison | `"Bool": {"aws:SecureTransport": "true"}` |
-
-## π§ Configuration
-
-### Policy Engine Configuration
-
-```toml
-[policy]
-# Policy evaluation settings
-max_policy_size = 2048 # Maximum policy size in KB
-max_conditions_per_statement = 10
-max_statements_per_policy = 100
-
-# Performance settings
-cache_policy_documents = true
-cache_ttl_seconds = 300
-max_cached_policies = 1000
-
-# Security settings
-require_secure_transport = true
-allow_anonymous_access = false
-default_effect = "deny"
-
-# Audit settings
-audit_policy_evaluation = true
-audit_log_path = "/var/log/rustfs/policy-audit.log"
-```
-
-### Advanced Configuration
-
-```rust
-use rustfs_policy::config::PolicyConfig;
-
-let config = PolicyConfig {
- // Policy parsing settings
- max_policy_size_kb: 2048,
- max_conditions_per_statement: 10,
- max_statements_per_policy: 100,
-
- // Evaluation settings
- default_effect: Effect::Deny,
- require_explicit_deny: false,
- cache_policy_documents: true,
- cache_ttl_seconds: 300,
-
- // Security settings
- require_secure_transport: true,
- allow_anonymous_access: false,
- validate_resource_arns: true,
-
- // Performance settings
- max_cached_policies: 1000,
- evaluation_timeout_ms: 100,
-
- ..Default::default()
-};
-```
-
-## π Performance Optimization
-
-### Caching Strategy
-
-- **Policy Document Cache**: Cache parsed policy documents
-- **Evaluation Result Cache**: Cache evaluation results for identical requests
-- **Condition Cache**: Cache condition function results
-- **Principal Cache**: Cache principal resolution results
-
-### Best Practices
-
-1. **Minimize Policy Size**: Keep policies as small as possible
-2. **Use Specific Actions**: Avoid overly broad action patterns
-3. **Optimize Conditions**: Use efficient condition functions
-4. **Cache Frequently Used Policies**: Enable policy caching for better performance
-
-## π€ Integration with RustFS
-
-The Policy Engine integrates seamlessly with other RustFS components:
-
-- **IAM Module**: Provides policy storage and user/group management
-- **ECStore**: Implements resource-based access control
-- **API Server**: Enforces policies on S3 API operations
-- **Audit System**: Logs policy evaluation decisions
-- **Admin Interface**: Manages policy documents and templates
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Memory**: Minimum 1GB RAM for policy caching
-- **Storage**: Compatible with RustFS storage backend
-
-## π Troubleshooting
-
-### Common Issues
-
-1. **Policy Parse Errors**:
- - Check JSON syntax validity
- - Verify action and resource ARN formats
- - Validate condition function syntax
-
-2. **Policy Evaluation Failures**:
- - Check principal resolution
- - Verify resource ARN matching
- - Debug condition function evaluation
-
-3. **Performance Issues**:
- - Monitor policy cache hit rates
- - Check policy document sizes
- - Optimize condition functions
-
-### Debug Commands
-
-```bash
-# Validate policy syntax
-rustfs-cli policy validate --file policy.json
-
-# Test policy evaluation
-rustfs-cli policy test --policy policy.json --user john --action s3:GetObject --resource arn:aws:s3:::bucket/key
-
-# Show policy evaluation trace
-rustfs-cli policy trace --policy policy.json --user john --action s3:GetObject --resource arn:aws:s3:::bucket/key
-```
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS IAM](../iam) - Identity and access management
-- [RustFS ECStore](../ecstore) - Erasure coding storage engine
-- [RustFS Crypto](../crypto) - Cryptographic operations
-- [RustFS Utils](../utils) - Utility functions
+- AWS-compatible bucket policy engine
+- Fine-grained resource-based access control
+- Condition-based policy evaluation
+- Policy validation and syntax checking
+- Role-based access control integration
+- Dynamic policy evaluation with context
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Policy Engine API Reference](https://docs.rustfs.com/policy/)
-- [Policy Language Guide](https://docs.rustfs.com/policy-language/)
-- [Access Control Guide](https://docs.rustfs.com/access-control/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details on:
-
-- Policy engine architecture and design patterns
-- Policy language syntax and semantics
-- Condition function implementation
-- Performance optimization techniques
-- Security considerations for access control
-
-### Development Setup
-
-```bash
-# Clone the repository
-git clone https://github.com/rustfs/rustfs.git
-cd rustfs
-
-# Navigate to Policy module
-cd crates/policy
-
-# Install dependencies
-cargo build
-
-# Run tests
-cargo test
-
-# Run policy validation tests
-cargo test policy_validation
-
-# Format code
-cargo fmt
-
-# Run linter
-cargo clippy
-```
-
-## π¬ Getting Help
-
-- **Documentation**: [docs.rustfs.com](https://docs.rustfs.com)
-- **Issues**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **Discussions**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
-- **Security**: Report security issues to
-
-## π Contact
-
-- **Bugs**: [GitHub Issues](https://github.com/rustfs/rustfs/issues)
-- **Business**:
-- **Jobs**:
-- **General Discussion**: [GitHub Discussions](https://github.com/rustfs/rustfs/discussions)
-
-## π₯ Contributors
-
-This module is maintained by the RustFS security team and community contributors. Special thanks to all who have contributed to making RustFS access control robust and flexible.
-
-
-
-
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
-```
-Copyright 2024 RustFS Team
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-```
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π‘οΈ by the RustFS Security Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/protos/README.md b/crates/protos/README.md
index 495900e7..73059598 100644
--- a/crates/protos/README.md
+++ b/crates/protos/README.md
@@ -3,7 +3,7 @@
# RustFS Protos - Protocol Buffer Definitions
- Protocol buffer definitions and gRPC interfaces for RustFS distributed object storage
+ Protocol buffer definitions and gRPC services for RustFS distributed object storage
@@ -17,410 +17,21 @@
## π Overview
-**RustFS Protos** provides protocol buffer definitions and gRPC service interfaces for the [RustFS](https://rustfs.com) distributed object storage system. It defines the communication protocols, message formats, and service contracts used across all RustFS components.
-
-> **Note:** This is a foundational submodule of RustFS that provides essential communication protocols for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Protos** provides protocol buffer definitions and gRPC services for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π‘ gRPC Services
-
-- **Storage Service**: Core storage operations (get, put, delete)
-- **Admin Service**: Administrative and management operations
-- **Metadata Service**: Metadata management and queries
-- **Lock Service**: Distributed locking and coordination
-
-### π¦ Message Types
-
-- **Storage Messages**: Object and bucket operation messages
-- **Administrative Messages**: Cluster management messages
-- **Metadata Messages**: File and object metadata structures
-- **Error Messages**: Standardized error reporting
-
-### π§ Protocol Features
-
-- **Versioning**: Protocol version compatibility management
-- **Extensions**: Custom field extensions for future expansion
-- **Streaming**: Support for streaming large data transfers
-- **Compression**: Built-in message compression support
-
-### π οΈ Code Generation
-
-- **Rust Bindings**: Automatic Rust code generation
-- **Type Safety**: Strong typing for all protocol messages
-- **Documentation**: Generated API documentation
-- **Validation**: Message validation and constraints
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-protos = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic gRPC Client
-
-```rust
-use rustfs_protos::storage::{StorageServiceClient, GetObjectRequest, PutObjectRequest};
-use tonic::transport::Channel;
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Connect to storage service
- let channel = Channel::from_static("http://storage.rustfs.local:9000")
- .connect()
- .await?;
-
- let mut client = StorageServiceClient::new(channel);
-
- // Get object
- let request = GetObjectRequest {
- bucket: "example-bucket".to_string(),
- key: "example-object".to_string(),
- version_id: None,
- range: None,
- };
-
- let response = client.get_object(request).await?;
- let object_data = response.into_inner();
-
- println!("Retrieved object: {} bytes", object_data.content.len());
- println!("Content type: {}", object_data.content_type);
- println!("ETag: {}", object_data.etag);
-
- Ok(())
-}
-```
-
-### Storage Operations
-
-```rust
-use rustfs_protos::storage::{
- StorageServiceClient, PutObjectRequest, DeleteObjectRequest,
- ListObjectsRequest, CreateBucketRequest
-};
-
-async fn storage_operations_example() -> Result<(), Box> {
- let mut client = StorageServiceClient::connect("http://storage.rustfs.local:9000").await?;
-
- // Create bucket
- let create_bucket_request = CreateBucketRequest {
- bucket: "new-bucket".to_string(),
- region: "us-east-1".to_string(),
- acl: None,
- storage_class: "STANDARD".to_string(),
- };
-
- client.create_bucket(create_bucket_request).await?;
- println!("Bucket created successfully");
-
- // Put object
- let put_request = PutObjectRequest {
- bucket: "new-bucket".to_string(),
- key: "test-file.txt".to_string(),
- content: b"Hello, RustFS!".to_vec(),
- content_type: "text/plain".to_string(),
- metadata: std::collections::HashMap::new(),
- storage_class: None,
- };
-
- let put_response = client.put_object(put_request).await?;
- println!("Object uploaded, ETag: {}", put_response.into_inner().etag);
-
- // List objects
- let list_request = ListObjectsRequest {
- bucket: "new-bucket".to_string(),
- prefix: None,
- marker: None,
- max_keys: Some(100),
- delimiter: None,
- };
-
- let list_response = client.list_objects(list_request).await?;
- let objects = list_response.into_inner();
-
- for object in objects.contents {
- println!("Object: {} (size: {} bytes)", object.key, object.size);
- }
-
- Ok(())
-}
-```
-
-### Administrative Operations
-
-```rust
-use rustfs_protos::admin::{
- AdminServiceClient, GetServerInfoRequest, AddServerRequest,
- ListServersRequest, ConfigUpdateRequest
-};
-
-async fn admin_operations_example() -> Result<(), Box> {
- let mut client = AdminServiceClient::connect("http://admin.rustfs.local:9001").await?;
-
- // Get server information
- let info_request = GetServerInfoRequest {};
- let info_response = client.get_server_info(info_request).await?;
- let server_info = info_response.into_inner();
-
- println!("Server version: {}", server_info.version);
- println!("Uptime: {} seconds", server_info.uptime_seconds);
- println!("Memory usage: {} MB", server_info.memory_usage_mb);
-
- // List cluster servers
- let list_request = ListServersRequest {};
- let list_response = client.list_servers(list_request).await?;
- let servers = list_response.into_inner();
-
- for server in servers.servers {
- println!("Server: {} - Status: {}", server.endpoint, server.status);
- }
-
- // Add new server
- let add_request = AddServerRequest {
- endpoint: "https://new-node.rustfs.local:9000".to_string(),
- access_key: "node-access-key".to_string(),
- secret_key: "node-secret-key".to_string(),
- };
-
- client.add_server(add_request).await?;
- println!("New server added to cluster");
-
- Ok(())
-}
-```
-
-### Metadata Operations
-
-```rust
-use rustfs_protos::metadata::{
- MetadataServiceClient, SearchObjectsRequest, GetObjectMetadataRequest,
- UpdateObjectMetadataRequest
-};
-
-async fn metadata_operations_example() -> Result<(), Box> {
- let mut client = MetadataServiceClient::connect("http://metadata.rustfs.local:9002").await?;
-
- // Search objects
- let search_request = SearchObjectsRequest {
- query: "content_type:image/*".to_string(),
- bucket: Some("photos-bucket".to_string()),
- limit: Some(50),
- offset: Some(0),
- };
-
- let search_response = client.search_objects(search_request).await?;
- let results = search_response.into_inner();
-
- for object in results.objects {
- println!("Found: {} ({})", object.key, object.content_type);
- }
-
- // Get object metadata
- let metadata_request = GetObjectMetadataRequest {
- bucket: "photos-bucket".to_string(),
- key: "vacation-photo.jpg".to_string(),
- };
-
- let metadata_response = client.get_object_metadata(metadata_request).await?;
- let metadata = metadata_response.into_inner();
-
- println!("Object metadata:");
- for (key, value) in metadata.metadata {
- println!(" {}: {}", key, value);
- }
-
- Ok(())
-}
-```
-
-### Lock Service Operations
-
-```rust
-use rustfs_protos::lock::{
- LockServiceClient, AcquireLockRequest, ReleaseLockRequest,
- LockType, LockMode
-};
-use std::time::Duration;
-
-async fn lock_operations_example() -> Result<(), Box> {
- let mut client = LockServiceClient::connect("http://lock.rustfs.local:9003").await?;
-
- // Acquire distributed lock
- let acquire_request = AcquireLockRequest {
- resource_id: "bucket/important-data".to_string(),
- lock_type: LockType::Exclusive as i32,
- timeout_seconds: 30,
- auto_renew: true,
- };
-
- let acquire_response = client.acquire_lock(acquire_request).await?;
- let lock_token = acquire_response.into_inner().lock_token;
-
- println!("Lock acquired: {}", lock_token);
-
- // Perform critical operations
- tokio::time::sleep(Duration::from_secs(5)).await;
-
- // Release lock
- let release_request = ReleaseLockRequest {
- lock_token: lock_token.clone(),
- };
-
- client.release_lock(release_request).await?;
- println!("Lock released: {}", lock_token);
-
- Ok(())
-}
-```
-
-### Streaming Operations
-
-```rust
-use rustfs_protos::storage::{StorageServiceClient, StreamUploadRequest, StreamDownloadRequest};
-use tokio_stream::wrappers::ReceiverStream;
-
-async fn streaming_operations_example() -> Result<(), Box> {
- let mut client = StorageServiceClient::connect("http://storage.rustfs.local:9000").await?;
-
- // Streaming upload
- let (tx, rx) = tokio::sync::mpsc::channel(100);
- let request_stream = ReceiverStream::new(rx);
-
- // Send upload metadata
- tx.send(StreamUploadRequest {
- bucket: "large-files".to_string(),
- key: "big-video.mp4".to_string(),
- content_type: "video/mp4".to_string(),
- chunk: vec![], // Empty chunk for metadata
- }).await?;
-
- // Send file chunks
- let mut file = tokio::fs::File::open("big-video.mp4").await?;
- let mut buffer = vec![0u8; 64 * 1024]; // 64KB chunks
-
- tokio::spawn(async move {
- loop {
- match file.read(&mut buffer).await {
- Ok(0) => break, // EOF
- Ok(n) => {
- let chunk_request = StreamUploadRequest {
- bucket: String::new(),
- key: String::new(),
- content_type: String::new(),
- chunk: buffer[..n].to_vec(),
- };
-
- if tx.send(chunk_request).await.is_err() {
- break;
- }
- }
- Err(_) => break,
- }
- }
- });
-
- let upload_response = client.stream_upload(request_stream).await?;
- println!("Upload completed: {}", upload_response.into_inner().etag);
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Protocol Architecture
-
-```
-Protocol Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β gRPC Services β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Storage API β Admin API β Metadata API β Lock API β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Protocol Buffer Messages β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Requests β Responses β Streaming β Errors β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Transport Layer (HTTP/2) β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Service Definitions
-
-| Service | Purpose | Key Operations |
-|---------|---------|----------------|
-| Storage | Object operations | Get, Put, Delete, List |
-| Admin | Cluster management | Add/Remove nodes, Config |
-| Metadata | Metadata queries | Search, Index, Update |
-| Lock | Distributed locking | Acquire, Release, Renew |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test protocol buffer compilation
-cargo test proto_compilation
-
-# Test service interfaces
-cargo test service_interfaces
-
-# Test message serialization
-cargo test serialization
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Dependencies**: Protocol Buffers compiler (protoc)
-- **Network**: gRPC transport support
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS Common](../common) - Common types and utilities
-- [RustFS Lock](../lock) - Distributed locking
+- Comprehensive gRPC service definitions
+- Cross-language compatibility with Protocol Buffers
+- Efficient binary serialization for network communication
+- Versioned API schemas with backward compatibility
+- Type-safe message definitions
+- Code generation for multiple programming languages
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Protos API Reference](https://docs.rustfs.com/protos/)
-- [gRPC Guide](https://docs.rustfs.com/grpc/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π‘ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/rio/README.md b/crates/rio/README.md
index 06cf1ca5..0f1e1cde 100644
--- a/crates/rio/README.md
+++ b/crates/rio/README.md
@@ -17,398 +17,21 @@
## π Overview
-**RustFS Rio** provides high-performance asynchronous I/O operations for the [RustFS](https://rustfs.com) distributed object storage system. It implements efficient data streaming, encryption, compression, and integrity checking with zero-copy operations and optimized buffering strategies.
-
-> **Note:** This is a performance-critical submodule of RustFS that provides essential I/O capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Rio** provides high-performance asynchronous I/O operations for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π High-Performance I/O
-
-- **Zero-Copy Operations**: Efficient data movement without unnecessary copying
-- **Async Streaming**: Non-blocking streaming I/O with backpressure handling
-- **Vectored I/O**: Scatter-gather operations for improved throughput
-- **Buffer Management**: Intelligent buffer pooling and reuse
-
-### π Cryptographic Operations
-
-- **AES-GCM Encryption**: Hardware-accelerated encryption/decryption
-- **Streaming Encryption**: Encrypt data on-the-fly without buffering
-- **Key Management**: Secure key derivation and rotation
-- **Digital Signatures**: Data integrity verification
-
-### π¦ Compression Support
-
-- **Multi-Algorithm**: Support for various compression algorithms
-- **Streaming Compression**: Real-time compression during transfer
-- **Adaptive Compression**: Dynamic algorithm selection based on data
-- **Compression Levels**: Configurable compression vs. speed tradeoffs
-
-### π§ Data Integrity
-
-- **CRC32 Checksums**: Fast integrity checking
-- **MD5 Hashing**: Legacy compatibility and verification
-- **Merkle Trees**: Hierarchical integrity verification
-- **Error Correction**: Automatic error detection and correction
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-rio = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Streaming I/O
-
-```rust
-use rustfs_rio::{StreamReader, StreamWriter, BufferPool};
-use tokio::io::{AsyncReadExt, AsyncWriteExt};
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Create buffer pool for efficient memory management
- let buffer_pool = BufferPool::new(64 * 1024, 100); // 64KB buffers, 100 in pool
-
- // Create streaming reader
- let mut reader = StreamReader::new(input_source, buffer_pool.clone());
-
- // Create streaming writer
- let mut writer = StreamWriter::new(output_destination, buffer_pool.clone());
-
- // High-performance streaming copy
- let mut buffer = vec![0u8; 8192];
- loop {
- let n = reader.read(&mut buffer).await?;
- if n == 0 {
- break;
- }
- writer.write_all(&buffer[..n]).await?;
- }
-
- writer.flush().await?;
- Ok(())
-}
-```
-
-### Encrypted Streaming
-
-```rust
-use rustfs_rio::{EncryptedWriter, EncryptedReader, EncryptionKey};
-use aes_gcm::{Aes256Gcm, Key, Nonce};
-
-async fn encrypted_streaming_example() -> Result<(), Box> {
- // Generate encryption key
- let key = EncryptionKey::generate()?;
-
- // Create encrypted writer
- let mut encrypted_writer = EncryptedWriter::new(
- output_stream,
- key.clone(),
- Aes256Gcm::new(&key.into())
- )?;
-
- // Write encrypted data
- encrypted_writer.write_all(b"Hello, encrypted world!").await?;
- encrypted_writer.finalize().await?;
-
- // Create encrypted reader
- let mut encrypted_reader = EncryptedReader::new(
- input_stream,
- key.clone(),
- Aes256Gcm::new(&key.into())
- )?;
-
- // Read decrypted data
- let mut decrypted_data = Vec::new();
- encrypted_reader.read_to_end(&mut decrypted_data).await?;
-
- println!("Decrypted: {}", String::from_utf8(decrypted_data)?);
- Ok(())
-}
-```
-
-### Compressed Streaming
-
-```rust
-use rustfs_rio::{CompressedWriter, CompressedReader, CompressionAlgorithm};
-
-async fn compressed_streaming_example() -> Result<(), Box> {
- // Create compressed writer
- let mut compressed_writer = CompressedWriter::new(
- output_stream,
- CompressionAlgorithm::Zstd,
- 6 // compression level
- )?;
-
- // Write compressed data
- compressed_writer.write_all(b"This data will be compressed").await?;
- compressed_writer.write_all(b"and streamed efficiently").await?;
- compressed_writer.finish().await?;
-
- // Create compressed reader
- let mut compressed_reader = CompressedReader::new(
- input_stream,
- CompressionAlgorithm::Zstd
- )?;
-
- // Read decompressed data
- let mut decompressed_data = Vec::new();
- compressed_reader.read_to_end(&mut decompressed_data).await?;
-
- println!("Decompressed: {}", String::from_utf8(decompressed_data)?);
- Ok(())
-}
-```
-
-### Integrity Checking
-
-```rust
-use rustfs_rio::{ChecksumWriter, ChecksumReader, ChecksumAlgorithm};
-
-async fn integrity_checking_example() -> Result<(), Box> {
- // Create checksum writer
- let mut checksum_writer = ChecksumWriter::new(
- output_stream,
- ChecksumAlgorithm::Crc32
- );
-
- // Write data with checksum calculation
- checksum_writer.write_all(b"Data with integrity checking").await?;
- let write_checksum = checksum_writer.finalize().await?;
-
- println!("Write checksum: {:08x}", write_checksum);
-
- // Create checksum reader
- let mut checksum_reader = ChecksumReader::new(
- input_stream,
- ChecksumAlgorithm::Crc32
- );
-
- // Read data with checksum verification
- let mut data = Vec::new();
- checksum_reader.read_to_end(&mut data).await?;
- let read_checksum = checksum_reader.checksum();
-
- println!("Read checksum: {:08x}", read_checksum);
-
- // Verify integrity
- if write_checksum == read_checksum {
- println!("Data integrity verified!");
- } else {
- println!("Data corruption detected!");
- }
-
- Ok(())
-}
-```
-
-### Multi-Layer Streaming
-
-```rust
-use rustfs_rio::{MultiLayerWriter, MultiLayerReader, Layer};
-
-async fn multi_layer_streaming_example() -> Result<(), Box> {
- // Create multi-layer writer (compression + encryption + checksum)
- let mut writer = MultiLayerWriter::new(output_stream)
- .add_layer(Layer::Compression(CompressionAlgorithm::Zstd, 6))
- .add_layer(Layer::Encryption(encryption_key.clone()))
- .add_layer(Layer::Checksum(ChecksumAlgorithm::Crc32))
- .build()?;
-
- // Write data through all layers
- writer.write_all(b"This data will be compressed, encrypted, and checksummed").await?;
- let final_checksum = writer.finalize().await?;
-
- // Create multi-layer reader (reverse order)
- let mut reader = MultiLayerReader::new(input_stream)
- .add_layer(Layer::Checksum(ChecksumAlgorithm::Crc32))
- .add_layer(Layer::Decryption(encryption_key.clone()))
- .add_layer(Layer::Decompression(CompressionAlgorithm::Zstd))
- .build()?;
-
- // Read data through all layers
- let mut data = Vec::new();
- reader.read_to_end(&mut data).await?;
-
- // Verify final checksum
- if reader.verify_checksum(final_checksum)? {
- println!("All layers verified successfully!");
- }
-
- Ok(())
-}
-```
-
-### Vectored I/O Operations
-
-```rust
-use rustfs_rio::{VectoredWriter, VectoredReader, IoVec};
-
-async fn vectored_io_example() -> Result<(), Box> {
- // Create vectored writer
- let mut vectored_writer = VectoredWriter::new(output_stream);
-
- // Prepare multiple buffers
- let header = b"HEADER";
- let data = b"Important data content";
- let footer = b"FOOTER";
-
- // Write multiple buffers in one operation
- let io_vecs = vec![
- IoVec::new(header),
- IoVec::new(data),
- IoVec::new(footer),
- ];
-
- let bytes_written = vectored_writer.write_vectored(&io_vecs).await?;
- println!("Wrote {} bytes in vectored operation", bytes_written);
-
- // Create vectored reader
- let mut vectored_reader = VectoredReader::new(input_stream);
-
- // Read into multiple buffers
- let mut header_buf = vec![0u8; 6];
- let mut data_buf = vec![0u8; 22];
- let mut footer_buf = vec![0u8; 6];
-
- let mut read_vecs = vec![
- IoVec::new_mut(&mut header_buf),
- IoVec::new_mut(&mut data_buf),
- IoVec::new_mut(&mut footer_buf),
- ];
-
- let bytes_read = vectored_reader.read_vectored(&mut read_vecs).await?;
- println!("Read {} bytes in vectored operation", bytes_read);
-
- Ok(())
-}
-```
-
-### Async Stream Processing
-
-```rust
-use rustfs_rio::{AsyncStreamProcessor, ProcessorChain};
-use futures::StreamExt;
-
-async fn stream_processing_example() -> Result<(), Box> {
- // Create processor chain
- let processor = ProcessorChain::new()
- .add_processor(Box::new(CompressionProcessor::new(CompressionAlgorithm::Zstd)))
- .add_processor(Box::new(EncryptionProcessor::new(encryption_key)))
- .add_processor(Box::new(ChecksumProcessor::new(ChecksumAlgorithm::Crc32)));
-
- // Create async stream processor
- let mut stream_processor = AsyncStreamProcessor::new(input_stream, processor);
-
- // Process stream chunks
- while let Some(chunk) = stream_processor.next().await {
- let processed_chunk = chunk?;
-
- // Handle processed chunk
- output_stream.write_all(&processed_chunk).await?;
- }
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Rio Architecture
-
-```
-Rio I/O Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Stream API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Encryption β Compression β Checksum β Vectored I/O β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Buffer Management β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Zero-Copy β Async I/O β Backpressure Control β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Tokio Runtime Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Performance Features
-
-| Feature | Benefit | Implementation |
-|---------|---------|----------------|
-| Zero-Copy | Reduced memory usage | Direct buffer operations |
-| Async I/O | High concurrency | Tokio-based operations |
-| Vectored I/O | Reduced syscalls | Scatter-gather operations |
-| Buffer Pooling | Memory efficiency | Reusable buffer management |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test streaming operations
-cargo test streaming
-
-# Test encryption
-cargo test encryption
-
-# Test compression
-cargo test compression
-
-# Run benchmarks
-cargo bench
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Dependencies**: Tokio async runtime
-- **Hardware**: AES-NI support recommended for encryption
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS Utils](../utils) - Utility functions
-- [RustFS Crypto](../crypto) - Cryptographic operations
+- Zero-copy streaming I/O operations
+- Hardware-accelerated encryption/decryption
+- Multi-algorithm compression support
+- Efficient buffer management and pooling
+- Vectored I/O for improved throughput
+- Real-time data integrity verification
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Rio API Reference](https://docs.rustfs.com/rio/)
-- [Performance Guide](https://docs.rustfs.com/performance/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/s3select-api/README.md b/crates/s3select-api/README.md
index 03afd58d..61d345d9 100644
--- a/crates/s3select-api/README.md
+++ b/crates/s3select-api/README.md
@@ -17,575 +17,21 @@
## π Overview
-**RustFS S3Select API** provides AWS S3 Select compatible SQL query capabilities for the [RustFS](https://rustfs.com) distributed object storage system. It enables clients to retrieve subsets of data from objects using SQL expressions, reducing data transfer and improving query performance through server-side filtering.
-
-> **Note:** This is a high-performance submodule of RustFS that provides essential SQL query capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS S3Select API** provides AWS S3 Select compatible SQL query capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π SQL Query Support
-
-- **Standard SQL**: Support for SELECT, WHERE, GROUP BY, ORDER BY clauses
-- **Data Types**: Support for strings, numbers, booleans, timestamps
-- **Functions**: Built-in SQL functions (aggregation, string, date functions)
-- **Complex Expressions**: Nested queries and complex conditional logic
-
-### π Format Support
-
-- **CSV Files**: Comma-separated values with customizable delimiters
-- **JSON Documents**: JSON objects and arrays with path expressions
-- **Parquet Files**: Columnar format with schema evolution
-- **Apache Arrow**: High-performance columnar data format
-
-### π Performance Features
-
-- **Streaming Processing**: Process large files without loading into memory
-- **Parallel Execution**: Multi-threaded query execution
-- **Predicate Pushdown**: Push filters down to storage layer
-- **Columnar Processing**: Efficient columnar data processing with Apache DataFusion
-
-### π§ S3 Compatibility
-
-- **S3 Select API**: Full compatibility with AWS S3 Select API
-- **Request Formats**: Support for JSON and XML request formats
-- **Response Streaming**: Streaming query results back to clients
-- **Error Handling**: AWS-compatible error responses
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-s3select-api = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic S3 Select Query
-
-```rust
-use rustfs_s3select_api::{S3SelectService, SelectRequest, InputSerialization, OutputSerialization};
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Create S3 Select service
- let s3select = S3SelectService::new().await?;
-
- // Configure input format (CSV)
- let input_serialization = InputSerialization::CSV {
- file_header_info: "USE".to_string(),
- record_delimiter: "\n".to_string(),
- field_delimiter: ",".to_string(),
- quote_character: "\"".to_string(),
- quote_escape_character: "\"".to_string(),
- comments: "#".to_string(),
- };
-
- // Configure output format
- let output_serialization = OutputSerialization::JSON {
- record_delimiter: "\n".to_string(),
- };
-
- // Create select request
- let select_request = SelectRequest {
- bucket: "sales-data".to_string(),
- key: "2024/sales.csv".to_string(),
- expression: "SELECT name, revenue FROM S3Object WHERE revenue > 10000".to_string(),
- expression_type: "SQL".to_string(),
- input_serialization,
- output_serialization,
- request_progress: false,
- };
-
- // Execute query
- let mut result_stream = s3select.select_object_content(select_request).await?;
-
- // Process streaming results
- while let Some(event) = result_stream.next().await {
- match event? {
- SelectEvent::Records(data) => {
- println!("Query result: {}", String::from_utf8(data)?);
- }
- SelectEvent::Stats(stats) => {
- println!("Bytes scanned: {}", stats.bytes_scanned);
- println!("Bytes processed: {}", stats.bytes_processed);
- println!("Bytes returned: {}", stats.bytes_returned);
- }
- SelectEvent::Progress(progress) => {
- println!("Progress: {}%", progress.details.bytes_processed_percent);
- }
- SelectEvent::End => {
- println!("Query completed");
- break;
- }
- }
- }
-
- Ok(())
-}
-```
-
-### CSV Data Processing
-
-```rust
-use rustfs_s3select_api::{S3SelectService, CSVInputSerialization};
-
-async fn csv_processing_example() -> Result<(), Box> {
- let s3select = S3SelectService::new().await?;
-
- // Configure CSV input with custom settings
- let csv_input = CSVInputSerialization {
- file_header_info: "USE".to_string(),
- record_delimiter: "\r\n".to_string(),
- field_delimiter: "|".to_string(),
- quote_character: "'".to_string(),
- quote_escape_character: "\\".to_string(),
- comments: "//".to_string(),
- allow_quoted_record_delimiter: false,
- };
-
- // Query with aggregation
- let select_request = SelectRequest {
- bucket: "analytics".to_string(),
- key: "user-events.csv".to_string(),
- expression: r#"
- SELECT
- event_type,
- COUNT(*) as event_count,
- AVG(CAST(duration as DECIMAL)) as avg_duration
- FROM S3Object
- WHERE timestamp >= '2024-01-01'
- GROUP BY event_type
- ORDER BY event_count DESC
- "#.to_string(),
- expression_type: "SQL".to_string(),
- input_serialization: InputSerialization::CSV(csv_input),
- output_serialization: OutputSerialization::JSON {
- record_delimiter: "\n".to_string(),
- },
- request_progress: true,
- };
-
- let mut result_stream = s3select.select_object_content(select_request).await?;
-
- let mut total_events = 0;
- while let Some(event) = result_stream.next().await {
- match event? {
- SelectEvent::Records(data) => {
- let result: serde_json::Value = serde_json::from_slice(&data)?;
- println!("Event type: {}, Count: {}, Avg duration: {}",
- result["event_type"], result["event_count"], result["avg_duration"]);
- total_events += result["event_count"].as_u64().unwrap_or(0);
- }
- SelectEvent::Progress(progress) => {
- println!("Processing: {}%", progress.details.bytes_processed_percent);
- }
- _ => {}
- }
- }
-
- println!("Total events processed: {}", total_events);
- Ok(())
-}
-```
-
-### JSON Data Querying
-
-```rust
-use rustfs_s3select_api::{JSONInputSerialization, JSONType};
-
-async fn json_querying_example() -> Result<(), Box> {
- let s3select = S3SelectService::new().await?;
-
- // Configure JSON input
- let json_input = JSONInputSerialization {
- json_type: JSONType::Lines, // JSON Lines format
- };
-
- // Query nested JSON data
- let select_request = SelectRequest {
- bucket: "logs".to_string(),
- key: "application.jsonl".to_string(),
- expression: r#"
- SELECT
- s.timestamp,
- s.level,
- s.message,
- s.metadata.user_id,
- s.metadata.request_id
- FROM S3Object[*] s
- WHERE s.level = 'ERROR'
- AND s.metadata.user_id IS NOT NULL
- ORDER BY s.timestamp DESC
- "#.to_string(),
- expression_type: "SQL".to_string(),
- input_serialization: InputSerialization::JSON(json_input),
- output_serialization: OutputSerialization::JSON {
- record_delimiter: "\n".to_string(),
- },
- request_progress: false,
- };
-
- let mut result_stream = s3select.select_object_content(select_request).await?;
-
- while let Some(event) = result_stream.next().await {
- if let SelectEvent::Records(data) = event? {
- let log_entry: serde_json::Value = serde_json::from_slice(&data)?;
- println!("Error at {}: {} (User: {}, Request: {})",
- log_entry["timestamp"],
- log_entry["message"],
- log_entry["user_id"],
- log_entry["request_id"]
- );
- }
- }
-
- Ok(())
-}
-```
-
-### Parquet File Analysis
-
-```rust
-use rustfs_s3select_api::{ParquetInputSerialization};
-
-async fn parquet_analysis_example() -> Result<(), Box> {
- let s3select = S3SelectService::new().await?;
-
- // Parquet files don't need serialization configuration
- let parquet_input = ParquetInputSerialization {};
-
- // Complex analytical query
- let select_request = SelectRequest {
- bucket: "data-warehouse".to_string(),
- key: "sales/2024/q1/sales_data.parquet".to_string(),
- expression: r#"
- SELECT
- region,
- product_category,
- SUM(amount) as total_sales,
- COUNT(*) as transaction_count,
- AVG(amount) as avg_transaction,
- MIN(amount) as min_sale,
- MAX(amount) as max_sale
- FROM S3Object
- WHERE sale_date >= '2024-01-01'
- AND sale_date < '2024-04-01'
- AND amount > 0
- GROUP BY region, product_category
- HAVING SUM(amount) > 50000
- ORDER BY total_sales DESC
- LIMIT 20
- "#.to_string(),
- expression_type: "SQL".to_string(),
- input_serialization: InputSerialization::Parquet(parquet_input),
- output_serialization: OutputSerialization::JSON {
- record_delimiter: "\n".to_string(),
- },
- request_progress: true,
- };
-
- let mut result_stream = s3select.select_object_content(select_request).await?;
-
- while let Some(event) = result_stream.next().await {
- match event? {
- SelectEvent::Records(data) => {
- let sales_data: serde_json::Value = serde_json::from_slice(&data)?;
- println!("Region: {}, Category: {}, Total Sales: ${:.2}",
- sales_data["region"],
- sales_data["product_category"],
- sales_data["total_sales"]
- );
- }
- SelectEvent::Stats(stats) => {
- println!("Query statistics:");
- println!(" Bytes scanned: {}", stats.bytes_scanned);
- println!(" Bytes processed: {}", stats.bytes_processed);
- println!(" Bytes returned: {}", stats.bytes_returned);
- }
- _ => {}
- }
- }
-
- Ok(())
-}
-```
-
-### Advanced SQL Functions
-
-```rust
-async fn advanced_sql_functions_example() -> Result<(), Box> {
- let s3select = S3SelectService::new().await?;
-
- // Query with various SQL functions
- let select_request = SelectRequest {
- bucket: "analytics".to_string(),
- key: "user_data.csv".to_string(),
- expression: r#"
- SELECT
- -- String functions
- UPPER(name) as name_upper,
- SUBSTRING(email, 1, POSITION('@' IN email) - 1) as username,
- LENGTH(description) as desc_length,
-
- -- Date functions
- EXTRACT(YEAR FROM registration_date) as reg_year,
- DATE_DIFF('day', registration_date, last_login) as days_since_reg,
-
- -- Numeric functions
- ROUND(score, 2) as rounded_score,
- CASE
- WHEN score >= 90 THEN 'Excellent'
- WHEN score >= 70 THEN 'Good'
- WHEN score >= 50 THEN 'Average'
- ELSE 'Poor'
- END as score_category,
-
- -- Conditional logic
- COALESCE(nickname, SUBSTRING(name, 1, POSITION(' ' IN name) - 1)) as display_name
-
- FROM S3Object
- WHERE registration_date IS NOT NULL
- AND score IS NOT NULL
- ORDER BY score DESC
- "#.to_string(),
- expression_type: "SQL".to_string(),
- input_serialization: InputSerialization::CSV {
- file_header_info: "USE".to_string(),
- record_delimiter: "\n".to_string(),
- field_delimiter: ",".to_string(),
- quote_character: "\"".to_string(),
- quote_escape_character: "\"".to_string(),
- comments: "#".to_string(),
- },
- output_serialization: OutputSerialization::JSON {
- record_delimiter: "\n".to_string(),
- },
- request_progress: false,
- };
-
- let mut result_stream = s3select.select_object_content(select_request).await?;
-
- while let Some(event) = result_stream.next().await {
- if let SelectEvent::Records(data) = event? {
- let user: serde_json::Value = serde_json::from_slice(&data)?;
- println!("User: {} ({}) - Score: {} ({})",
- user["display_name"],
- user["username"],
- user["rounded_score"],
- user["score_category"]
- );
- }
- }
-
- Ok(())
-}
-```
-
-### Streaming Large Datasets
-
-```rust
-use rustfs_s3select_api::{SelectObjectContentStream, ProgressDetails};
-
-async fn streaming_large_datasets() -> Result<(), Box> {
- let s3select = S3SelectService::new().await?;
-
- let select_request = SelectRequest {
- bucket: "big-data".to_string(),
- key: "large_dataset.csv".to_string(),
- expression: "SELECT * FROM S3Object WHERE status = 'active'".to_string(),
- expression_type: "SQL".to_string(),
- input_serialization: InputSerialization::CSV {
- file_header_info: "USE".to_string(),
- record_delimiter: "\n".to_string(),
- field_delimiter: ",".to_string(),
- quote_character: "\"".to_string(),
- quote_escape_character: "\"".to_string(),
- comments: "".to_string(),
- },
- output_serialization: OutputSerialization::JSON {
- record_delimiter: "\n".to_string(),
- },
- request_progress: true,
- };
-
- let mut result_stream = s3select.select_object_content(select_request).await?;
-
- let mut processed_count = 0;
- let mut output_file = tokio::fs::File::create("filtered_results.jsonl").await?;
-
- while let Some(event) = result_stream.next().await {
- match event? {
- SelectEvent::Records(data) => {
- // Write results to file
- output_file.write_all(&data).await?;
- processed_count += 1;
-
- if processed_count % 1000 == 0 {
- println!("Processed {} records", processed_count);
- }
- }
- SelectEvent::Progress(progress) => {
- println!("Progress: {:.1}% ({} bytes processed)",
- progress.details.bytes_processed_percent,
- progress.details.bytes_processed
- );
- }
- SelectEvent::Stats(stats) => {
- println!("Final statistics:");
- println!(" Total bytes scanned: {}", stats.bytes_scanned);
- println!(" Total bytes processed: {}", stats.bytes_processed);
- println!(" Total bytes returned: {}", stats.bytes_returned);
- println!(" Processing efficiency: {:.2}%",
- (stats.bytes_returned as f64 / stats.bytes_scanned as f64) * 100.0
- );
- }
- SelectEvent::End => {
- println!("Streaming completed. Total records: {}", processed_count);
- break;
- }
- }
- }
-
- output_file.flush().await?;
- Ok(())
-}
-```
-
-### HTTP API Integration
-
-```rust
-use rustfs_s3select_api::{S3SelectHandler, SelectRequestXML};
-use axum::{Router, Json, extract::{Path, Query}};
-
-async fn setup_s3select_http_api() -> Router {
- let s3select_handler = S3SelectHandler::new().await.unwrap();
-
- Router::new()
- .route("/buckets/:bucket/objects/:key/select",
- axum::routing::post(handle_select_object_content))
- .layer(Extension(s3select_handler))
-}
-
-async fn handle_select_object_content(
- Path((bucket, key)): Path<(String, String)>,
- Extension(handler): Extension,
- body: String,
-) -> Result> {
- // Parse S3 Select request (XML or JSON)
- let select_request = handler.parse_request(&body, &bucket, &key).await?;
-
- // Execute query
- let result_stream = handler.execute_select(select_request).await?;
-
- // Return streaming response
- let response = axum::response::Response::builder()
- .header("content-type", "application/xml")
- .header("x-amz-request-id", "12345")
- .body(axum::body::Body::from_stream(result_stream))?;
-
- Ok(response)
-}
-```
-
-## ποΈ Architecture
-
-### S3Select API Architecture
-
-```
-S3Select API Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β S3 Select HTTP API β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Request β Response β Streaming β Error β
-β Parsing β Formatting β Results β Handling β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Query Engine (DataFusion) β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β SQL Parser β Optimizer β Execution β Streamingβ
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Storage Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Supported Data Formats
-
-| Format | Features | Use Cases |
-|--------|----------|-----------|
-| CSV | Custom delimiters, headers, quotes | Log files, exports |
-| JSON | Objects, arrays, nested data | APIs, documents |
-| JSON Lines | Streaming JSON records | Event logs, analytics |
-| Parquet | Columnar, schema evolution | Data warehousing |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test SQL parsing
-cargo test sql_parsing
-
-# Test format support
-cargo test format_support
-
-# Test streaming
-cargo test streaming
-
-# Integration tests
-cargo test --test integration
-
-# Performance tests
-cargo test --test performance --release
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Dependencies**: Apache DataFusion, Arrow
-- **Memory**: Sufficient RAM for query processing
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS S3Select Query](../s3select-query) - Query engine implementation
-- [RustFS ECStore](../ecstore) - Storage backend
+- Standard SQL query support (SELECT, WHERE, GROUP BY, ORDER BY)
+- Multiple data format support (CSV, JSON, Parquet, Arrow)
+- Streaming processing for large files
+- AWS S3 Select API compatibility
+- Parallel query execution
+- Predicate pushdown optimization
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [S3Select API Reference](https://docs.rustfs.com/s3select-api/)
-- [SQL Reference](https://docs.rustfs.com/sql/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/s3select-query/README.md b/crates/s3select-query/README.md
index e5789f4b..a97e0f2a 100644
--- a/crates/s3select-query/README.md
+++ b/crates/s3select-query/README.md
@@ -1,6 +1,6 @@
[](https://rustfs.com)
-# RustFS S3Select Query - High-Performance Query Engine
+# RustFS S3Select Query - SQL Query Engine
Apache DataFusion-powered SQL query engine for RustFS S3 Select implementation
@@ -17,641 +17,21 @@
## π Overview
-**RustFS S3Select Query** is the high-performance query engine that powers SQL processing for the [RustFS](https://rustfs.com) S3 Select API. Built on Apache DataFusion, it provides blazing-fast SQL execution with advanced optimization techniques, streaming processing, and support for multiple data formats.
-
-> **Note:** This is a core performance-critical submodule of RustFS that provides the SQL query execution engine for the S3 Select API. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS S3Select Query** provides Apache DataFusion-powered SQL query engine capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π High-Performance Query Engine
-
-- **Apache DataFusion**: Built on the fastest SQL engine in Rust
-- **Vectorized Processing**: SIMD-accelerated columnar processing
-- **Parallel Execution**: Multi-threaded query execution
-- **Memory Efficient**: Streaming processing with minimal memory footprint
-
-### π Advanced SQL Support
-
-- **Standard SQL**: Full support for SQL:2016 standard
-- **Complex Queries**: Joins, subqueries, window functions, CTEs
-- **Aggregations**: Group by, having, order by with optimizations
-- **Built-in Functions**: 200+ SQL functions including UDFs
-
-### π§ Query Optimization
-
-- **Cost-Based Optimizer**: Intelligent query planning
-- **Predicate Pushdown**: Push filters to data sources
-- **Projection Pushdown**: Only read required columns
-- **Join Optimization**: Hash joins, sort-merge joins
-
-### π Data Format Support
-
-- **Parquet**: Native columnar format with predicate pushdown
-- **CSV**: Efficient CSV parsing with schema inference
-- **JSON**: Nested JSON processing with path expressions
-- **Arrow**: Zero-copy Arrow format processing
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-s3select-query = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Query Engine Setup
-
-```rust
-use rustfs_s3select_query::{QueryEngine, DataSource, QueryResult};
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Create query engine
- let query_engine = QueryEngine::new().await?;
-
- // Register data source
- let data_source = DataSource::from_csv("s3://bucket/data.csv").await?;
- query_engine.register_table("sales", data_source).await?;
-
- // Execute SQL query
- let sql = "SELECT region, SUM(amount) as total FROM sales GROUP BY region";
- let result = query_engine.execute_query(sql).await?;
-
- // Process results
- while let Some(batch) = result.next().await {
- let batch = batch?;
- println!("Batch with {} rows", batch.num_rows());
-
- // Convert to JSON for display
- let json_rows = batch.to_json()?;
- for row in json_rows {
- println!("{}", row);
- }
- }
-
- Ok(())
-}
-```
-
-### Advanced Query Execution
-
-```rust
-use rustfs_s3select_query::{
- QueryEngine, QueryPlan, ExecutionConfig,
- DataSource, SchemaRef, RecordBatch
-};
-
-async fn advanced_query_example() -> Result<(), Box> {
- // Configure execution settings
- let config = ExecutionConfig::new()
- .with_target_partitions(8)
- .with_batch_size(8192)
- .with_max_memory(1024 * 1024 * 1024); // 1GB memory limit
-
- let query_engine = QueryEngine::with_config(config).await?;
-
- // Register multiple data sources
- let customers = DataSource::from_parquet("s3://warehouse/customers.parquet").await?;
- let orders = DataSource::from_csv("s3://logs/orders.csv").await?;
- let products = DataSource::from_json("s3://catalog/products.json").await?;
-
- query_engine.register_table("customers", customers).await?;
- query_engine.register_table("orders", orders).await?;
- query_engine.register_table("products", products).await?;
-
- // Complex analytical query
- let sql = r#"
- SELECT
- c.customer_segment,
- p.category,
- COUNT(*) as order_count,
- SUM(o.amount) as total_revenue,
- AVG(o.amount) as avg_order_value,
- STDDEV(o.amount) as revenue_stddev
- FROM customers c
- JOIN orders o ON c.customer_id = o.customer_id
- JOIN products p ON o.product_id = p.product_id
- WHERE o.order_date >= '2024-01-01'
- AND o.status = 'completed'
- GROUP BY c.customer_segment, p.category
- HAVING SUM(o.amount) > 10000
- ORDER BY total_revenue DESC
- LIMIT 50
- "#;
-
- // Get query plan for optimization analysis
- let plan = query_engine.create_logical_plan(sql).await?;
- println!("Query plan:\n{}", plan.display_indent());
-
- // Execute with streaming results
- let mut result_stream = query_engine.execute_stream(sql).await?;
-
- let mut total_rows = 0;
- while let Some(batch) = result_stream.next().await {
- let batch = batch?;
- total_rows += batch.num_rows();
-
- // Process batch
- for row_idx in 0..batch.num_rows() {
- let segment = batch.column_by_name("customer_segment")?
- .as_any().downcast_ref::()
- .unwrap().value(row_idx);
- let category = batch.column_by_name("category")?
- .as_any().downcast_ref::()
- .unwrap().value(row_idx);
- let revenue = batch.column_by_name("total_revenue")?
- .as_any().downcast_ref::()
- .unwrap().value(row_idx);
-
- println!("Segment: {}, Category: {}, Revenue: ${:.2}",
- segment, category, revenue);
- }
- }
-
- println!("Total rows processed: {}", total_rows);
- Ok(())
-}
-```
-
-### Custom Data Sources
-
-```rust
-use rustfs_s3select_query::{DataSource, TableProvider, SchemaRef};
-use datafusion::arrow::datatypes::{Schema, Field, DataType};
-use datafusion::arrow::record_batch::RecordBatch;
-
-struct CustomS3DataSource {
- bucket: String,
- key: String,
- schema: SchemaRef,
-}
-
-impl CustomS3DataSource {
- async fn new(bucket: &str, key: &str) -> Result> {
- // Infer schema from S3 object
- let schema = Self::infer_schema(bucket, key).await?;
-
- Ok(Self {
- bucket: bucket.to_string(),
- key: key.to_string(),
- schema: Arc::new(schema),
- })
- }
-
- async fn infer_schema(bucket: &str, key: &str) -> Result> {
- // Read sample data to infer schema
- let sample_data = read_s3_sample(bucket, key).await?;
-
- // Create schema based on data format
- let schema = Schema::new(vec![
- Field::new("id", DataType::Int64, false),
- Field::new("name", DataType::Utf8, false),
- Field::new("value", DataType::Float64, true),
- Field::new("timestamp", DataType::Timestamp(TimeUnit::Millisecond, None), false),
- ]);
-
- Ok(schema)
- }
-}
-
-#[async_trait::async_trait]
-impl TableProvider for CustomS3DataSource {
- fn as_any(&self) -> &dyn std::any::Any {
- self
- }
-
- fn schema(&self) -> SchemaRef {
- self.schema.clone()
- }
-
- async fn scan(
- &self,
- projection: Option<&Vec>,
- filters: &[Expr],
- limit: Option,
- ) -> Result, DataFusionError> {
- // Create execution plan for scanning S3 data
- let scan_plan = S3ScanExec::new(
- self.bucket.clone(),
- self.key.clone(),
- self.schema.clone(),
- projection.cloned(),
- filters.to_vec(),
- limit,
- );
-
- Ok(Arc::new(scan_plan))
- }
-}
-
-async fn custom_data_source_example() -> Result<(), Box> {
- let query_engine = QueryEngine::new().await?;
-
- // Register custom data source
- let custom_source = CustomS3DataSource::new("analytics", "events.parquet").await?;
- query_engine.register_table("events", Arc::new(custom_source)).await?;
-
- // Query custom data source
- let sql = "SELECT * FROM events WHERE timestamp > NOW() - INTERVAL '1 day'";
- let result = query_engine.execute_query(sql).await?;
-
- // Process results
- while let Some(batch) = result.next().await {
- let batch = batch?;
- println!("Custom source batch: {} rows", batch.num_rows());
- }
-
- Ok(())
-}
-```
-
-### Query Optimization and Analysis
-
-```rust
-use rustfs_s3select_query::{QueryEngine, QueryOptimizer, QueryMetrics};
-
-async fn query_optimization_example() -> Result<(), Box> {
- let query_engine = QueryEngine::new().await?;
-
- // Register data source
- let data_source = DataSource::from_parquet("s3://warehouse/sales.parquet").await?;
- query_engine.register_table("sales", data_source).await?;
-
- let sql = r#"
- SELECT
- region,
- product_category,
- SUM(amount) as total_sales,
- COUNT(*) as transaction_count
- FROM sales
- WHERE sale_date >= '2024-01-01'
- AND amount > 100
- GROUP BY region, product_category
- ORDER BY total_sales DESC
- "#;
-
- // Analyze query plan
- let logical_plan = query_engine.create_logical_plan(sql).await?;
- println!("Logical Plan:\n{}", logical_plan.display_indent());
-
- let physical_plan = query_engine.create_physical_plan(&logical_plan).await?;
- println!("Physical Plan:\n{}", physical_plan.display_indent());
-
- // Execute with metrics
- let start_time = std::time::Instant::now();
- let mut result_stream = query_engine.execute_stream(sql).await?;
-
- let mut total_rows = 0;
- let mut total_batches = 0;
-
- while let Some(batch) = result_stream.next().await {
- let batch = batch?;
- total_rows += batch.num_rows();
- total_batches += 1;
- }
-
- let execution_time = start_time.elapsed();
-
- // Get execution metrics
- let metrics = query_engine.get_execution_metrics().await?;
-
- println!("Query Performance:");
- println!(" Execution time: {:?}", execution_time);
- println!(" Total rows: {}", total_rows);
- println!(" Total batches: {}", total_batches);
- println!(" Rows per second: {:.2}", total_rows as f64 / execution_time.as_secs_f64());
- println!(" Memory used: {} bytes", metrics.memory_used);
- println!(" Bytes scanned: {}", metrics.bytes_scanned);
-
- Ok(())
-}
-```
-
-### Streaming Query Processing
-
-```rust
-use rustfs_s3select_query::{StreamingQueryEngine, StreamingResult};
-use futures::StreamExt;
-
-async fn streaming_processing_example() -> Result<(), Box> {
- let streaming_engine = StreamingQueryEngine::new().await?;
-
- // Register streaming data source
- let stream_source = DataSource::from_streaming_csv("s3://logs/stream.csv").await?;
- streaming_engine.register_table("log_stream", stream_source).await?;
-
- // Continuous query with windowing
- let sql = r#"
- SELECT
- TUMBLE_START(timestamp, INTERVAL '5' MINUTE) as window_start,
- COUNT(*) as event_count,
- AVG(response_time) as avg_response_time,
- MAX(response_time) as max_response_time
- FROM log_stream
- WHERE status_code >= 400
- GROUP BY TUMBLE(timestamp, INTERVAL '5' MINUTE)
- "#;
-
- let mut result_stream = streaming_engine.execute_streaming_query(sql).await?;
-
- // Process streaming results
- while let Some(window_result) = result_stream.next().await {
- let batch = window_result?;
-
- for row_idx in 0..batch.num_rows() {
- let window_start = batch.column_by_name("window_start")?
- .as_any().downcast_ref::()
- .unwrap().value(row_idx);
- let event_count = batch.column_by_name("event_count")?
- .as_any().downcast_ref::()
- .unwrap().value(row_idx);
- let avg_response = batch.column_by_name("avg_response_time")?
- .as_any().downcast_ref::()
- .unwrap().value(row_idx);
-
- println!("Window {}: {} errors, avg response time: {:.2}ms",
- window_start, event_count, avg_response);
- }
- }
-
- Ok(())
-}
-```
-
-### User-Defined Functions (UDFs)
-
-```rust
-use rustfs_s3select_query::{QueryEngine, ScalarUDF, Volatility};
-use datafusion::arrow::datatypes::{DataType, Field};
-
-async fn custom_functions_example() -> Result<(), Box> {
- let query_engine = QueryEngine::new().await?;
-
- // Register custom scalar function
- let extract_domain_udf = ScalarUDF::new(
- "extract_domain",
- vec![DataType::Utf8],
- DataType::Utf8,
- Volatility::Immutable,
- Arc::new(|args: &[ArrayRef]| {
- let emails = args[0].as_any().downcast_ref::().unwrap();
- let mut domains = Vec::new();
-
- for i in 0..emails.len() {
- if let Some(email) = emails.value_opt(i) {
- if let Some(domain) = email.split('@').nth(1) {
- domains.push(Some(domain.to_string()));
- } else {
- domains.push(None);
- }
- } else {
- domains.push(None);
- }
- }
-
- Ok(Arc::new(StringArray::from(domains)))
- }),
- );
-
- query_engine.register_udf(extract_domain_udf).await?;
-
- // Register aggregate function
- let percentile_udf = AggregateUDF::new(
- "percentile_90",
- vec![DataType::Float64],
- DataType::Float64,
- Volatility::Immutable,
- Arc::new(|| Box::new(PercentileAccumulator::new(0.9))),
- );
-
- query_engine.register_udaf(percentile_udf).await?;
-
- // Use custom functions in query
- let data_source = DataSource::from_csv("s3://users/profiles.csv").await?;
- query_engine.register_table("users", data_source).await?;
-
- let sql = r#"
- SELECT
- extract_domain(email) as domain,
- COUNT(*) as user_count,
- percentile_90(score) as p90_score
- FROM users
- GROUP BY extract_domain(email)
- ORDER BY user_count DESC
- "#;
-
- let result = query_engine.execute_query(sql).await?;
-
- while let Some(batch) = result.next().await {
- let batch = batch?;
-
- for row_idx in 0..batch.num_rows() {
- let domain = batch.column_by_name("domain")?
- .as_any().downcast_ref::()
- .unwrap().value(row_idx);
- let user_count = batch.column_by_name("user_count")?
- .as_any().downcast_ref::()
- .unwrap().value(row_idx);
- let p90_score = batch.column_by_name("p90_score")?
- .as_any().downcast_ref::()
- .unwrap().value(row_idx);
-
- println!("Domain: {}, Users: {}, P90 Score: {:.2}",
- domain, user_count, p90_score);
- }
- }
-
- Ok(())
-}
-```
-
-### Query Caching and Materialization
-
-```rust
-use rustfs_s3select_query::{QueryEngine, QueryCache, MaterializedView};
-
-async fn query_caching_example() -> Result<(), Box> {
- let mut query_engine = QueryEngine::new().await?;
-
- // Enable query result caching
- let cache_config = QueryCache::new()
- .with_max_size(1024 * 1024 * 1024) // 1GB cache
- .with_ttl(Duration::from_secs(300)); // 5 minutes TTL
-
- query_engine.enable_caching(cache_config).await?;
-
- // Register data source
- let data_source = DataSource::from_parquet("s3://warehouse/transactions.parquet").await?;
- query_engine.register_table("transactions", data_source).await?;
-
- // Create materialized view for common queries
- let materialized_view = MaterializedView::new(
- "daily_sales",
- r#"
- SELECT
- DATE(transaction_date) as date,
- SUM(amount) as total_sales,
- COUNT(*) as transaction_count
- FROM transactions
- GROUP BY DATE(transaction_date)
- "#.to_string(),
- Duration::from_secs(3600), // Refresh every hour
- );
-
- query_engine.register_materialized_view(materialized_view).await?;
-
- // Query using materialized view
- let sql = r#"
- SELECT
- date,
- total_sales,
- LAG(total_sales, 1) OVER (ORDER BY date) as prev_day_sales,
- (total_sales - LAG(total_sales, 1) OVER (ORDER BY date)) /
- LAG(total_sales, 1) OVER (ORDER BY date) * 100 as growth_rate
- FROM daily_sales
- WHERE date >= CURRENT_DATE - INTERVAL '30' DAY
- ORDER BY date DESC
- "#;
-
- // First execution - cache miss
- let start_time = std::time::Instant::now();
- let result1 = query_engine.execute_query(sql).await?;
- let mut rows1 = 0;
- while let Some(batch) = result1.next().await {
- rows1 += batch?.num_rows();
- }
- let first_execution_time = start_time.elapsed();
-
- // Second execution - cache hit
- let start_time = std::time::Instant::now();
- let result2 = query_engine.execute_query(sql).await?;
- let mut rows2 = 0;
- while let Some(batch) = result2.next().await {
- rows2 += batch?.num_rows();
- }
- let second_execution_time = start_time.elapsed();
-
- println!("First execution: {:?} ({} rows)", first_execution_time, rows1);
- println!("Second execution: {:?} ({} rows)", second_execution_time, rows2);
- println!("Cache speedup: {:.2}x",
- first_execution_time.as_secs_f64() / second_execution_time.as_secs_f64());
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Query Engine Architecture
-
-```
-Query Engine Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β SQL Query Interface β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Parser β Planner β Optimizer β Executor β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Apache DataFusion Core β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Vectorized β Parallel β Streaming β Memory β
-β Processing β Execution β Engine β Managementβ
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Data Source Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Parquet β CSV β JSON β Arrow β
-β Reader β Parser β Parser β Format β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Execution Flow
-
-1. **SQL Parsing**: Convert SQL string to logical plan
-2. **Logical Optimization**: Apply rule-based optimizations
-3. **Physical Planning**: Create physical execution plan
-4. **Execution**: Execute plan with streaming results
-5. **Result Streaming**: Return results as Arrow batches
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test query execution
-cargo test query_execution
-
-# Test optimization
-cargo test optimization
-
-# Test data formats
-cargo test data_formats
-
-# Benchmark tests
-cargo test --test benchmarks --release
-
-# Integration tests
-cargo test --test integration
-```
-
-## π Performance Benchmarks
-
-| Operation | Throughput | Latency | Memory |
-|-----------|------------|---------|---------|
-| CSV Scan | 2.5 GB/s | 10ms | 50MB |
-| Parquet Scan | 5.0 GB/s | 5ms | 30MB |
-| JSON Parse | 1.2 GB/s | 15ms | 80MB |
-| Aggregation | 1.8 GB/s | 20ms | 100MB |
-| Join | 800 MB/s | 50ms | 200MB |
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **CPU**: Multi-core recommended for parallel processing
-- **Memory**: Variable based on query complexity
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS S3Select API](../s3select-api) - S3 Select API implementation
-- [RustFS ECStore](../ecstore) - Storage backend
+- Apache DataFusion integration for high-performance queries
+- Vectorized processing with SIMD acceleration
+- Parallel query execution across multiple threads
+- Cost-based query optimization
+- Support for complex SQL operations (joins, subqueries, window functions)
+- Multiple data format support (Parquet, CSV, JSON, Arrow)
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [S3Select Query Reference](https://docs.rustfs.com/s3select-query/)
-- [DataFusion Integration Guide](https://docs.rustfs.com/datafusion/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with β‘ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/signer/README.md b/crates/signer/README.md
index 6ae86384..d602ea66 100644
--- a/crates/signer/README.md
+++ b/crates/signer/README.md
@@ -17,390 +17,21 @@
## π Overview
-**RustFS Signer** provides AWS-compatible request signing and authentication for the [RustFS](https://rustfs.com) distributed object storage system. It implements AWS Signature Version 4 (SigV4) signing algorithm, pre-signed URLs, and various authentication methods to ensure secure API access.
-
-> **Note:** This is a security-critical submodule of RustFS that provides essential authentication capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Signer** provides AWS-compatible request signing and authentication capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π AWS-Compatible Signing
-
-- **SigV4 Implementation**: Full AWS Signature Version 4 support
-- **Pre-signed URLs**: Temporary access URLs with expiration
-- **Chunked Upload**: Streaming upload with signature validation
-- **Multi-part Upload**: Signature validation for large files
-
-### π‘οΈ Authentication Methods
-
-- **Access Key/Secret**: Traditional AWS-style authentication
-- **STS Token**: Temporary security token support
-- **IAM Role**: Role-based authentication
-- **Anonymous Access**: Public read access support
-
-### π Performance Features
-
-- **Signature Caching**: Avoid repeated signature calculations
-- **Batch Signing**: Sign multiple requests efficiently
-- **Streaming Support**: Sign data streams without buffering
-- **Hardware Acceleration**: Use hardware crypto when available
-
-### π§ Advanced Features
-
-- **Custom Headers**: Support for custom and vendor headers
-- **Regional Signing**: Multi-region signature support
-- **Clock Skew Handling**: Automatic time synchronization
-- **Signature Validation**: Server-side signature verification
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-signer = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Request Signing
-
-```rust
-use rustfs_signer::{Signer, SigningConfig, Credentials};
-use http::{Request, Method};
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Create credentials
- let credentials = Credentials::new(
- "AKIAIOSFODNN7EXAMPLE".to_string(),
- "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY".to_string(),
- None, // No session token
- );
-
- // Create signing configuration
- let config = SigningConfig {
- region: "us-east-1".to_string(),
- service: "s3".to_string(),
- credentials,
- ..Default::default()
- };
-
- // Create signer
- let signer = Signer::new(config);
-
- // Create request
- let request = Request::builder()
- .method(Method::GET)
- .uri("https://example-bucket.s3.amazonaws.com/example-object")
- .body(Vec::new())?;
-
- // Sign request
- let signed_request = signer.sign_request(request).await?;
-
- println!("Authorization header: {:?}", signed_request.headers().get("authorization"));
-
- Ok(())
-}
-```
-
-### Pre-signed URLs
-
-```rust
-use rustfs_signer::{Signer, PresignedUrlRequest};
-use std::time::Duration;
-
-async fn presigned_url_example() -> Result<(), Box> {
- let signer = Signer::new(signing_config);
-
- // Create pre-signed URL for GET request
- let presigned_request = PresignedUrlRequest {
- method: Method::GET,
- uri: "https://example-bucket.s3.amazonaws.com/example-object".parse()?,
- headers: Default::default(),
- expires_in: Duration::from_secs(3600), // 1 hour
- };
-
- let presigned_url = signer.presign_url(presigned_request).await?;
- println!("Pre-signed URL: {}", presigned_url);
-
- // Create pre-signed URL for PUT request
- let put_request = PresignedUrlRequest {
- method: Method::PUT,
- uri: "https://example-bucket.s3.amazonaws.com/upload-object".parse()?,
- headers: {
- let mut headers = HeaderMap::new();
- headers.insert("content-type", "text/plain".parse()?);
- headers
- },
- expires_in: Duration::from_secs(1800), // 30 minutes
- };
-
- let upload_url = signer.presign_url(put_request).await?;
- println!("Pre-signed upload URL: {}", upload_url);
-
- Ok(())
-}
-```
-
-### Streaming Upload Signing
-
-```rust
-use rustfs_signer::{StreamingSigner, ChunkedUploadSigner};
-use tokio::io::AsyncReadExt;
-
-async fn streaming_upload_example() -> Result<(), Box> {
- let signer = Signer::new(signing_config);
-
- // Create streaming signer
- let streaming_signer = StreamingSigner::new(signer, "s3".to_string());
-
- // Create chunked upload signer
- let mut chunked_signer = ChunkedUploadSigner::new(
- streaming_signer,
- "example-bucket".to_string(),
- "large-file.dat".to_string(),
- );
-
- // Initialize multipart upload
- let upload_id = chunked_signer.initiate_multipart_upload().await?;
- println!("Upload ID: {}", upload_id);
-
- // Upload chunks
- let mut file = tokio::fs::File::open("large-file.dat").await?;
- let mut chunk_buffer = vec![0u8; 5 * 1024 * 1024]; // 5MB chunks
- let mut part_number = 1;
- let mut etags = Vec::new();
-
- loop {
- let bytes_read = file.read(&mut chunk_buffer).await?;
- if bytes_read == 0 {
- break;
- }
-
- let chunk = &chunk_buffer[..bytes_read];
- let etag = chunked_signer.upload_part(part_number, chunk).await?;
- etags.push((part_number, etag));
-
- part_number += 1;
- }
-
- // Complete multipart upload
- chunked_signer.complete_multipart_upload(upload_id, etags).await?;
- println!("Upload completed successfully");
-
- Ok(())
-}
-```
-
-### Signature Validation
-
-```rust
-use rustfs_signer::{SignatureValidator, ValidationResult};
-use http::HeaderMap;
-
-async fn signature_validation_example() -> Result<(), Box> {
- let validator = SignatureValidator::new(signing_config);
-
- // Extract signature from request headers
- let headers = HeaderMap::new(); // Headers from incoming request
- let method = "GET";
- let uri = "/example-bucket/example-object";
- let body = b""; // Request body
-
- // Validate signature
- let validation_result = validator.validate_signature(
- method,
- uri,
- &headers,
- body,
- ).await?;
-
- match validation_result {
- ValidationResult::Valid { credentials, .. } => {
- println!("Signature valid for user: {}", credentials.access_key);
- }
- ValidationResult::Invalid { reason } => {
- println!("Signature invalid: {}", reason);
- }
- ValidationResult::Expired { expired_at } => {
- println!("Signature expired at: {}", expired_at);
- }
- }
-
- Ok(())
-}
-```
-
-### Batch Signing
-
-```rust
-use rustfs_signer::{BatchSigner, BatchSigningRequest};
-
-async fn batch_signing_example() -> Result<(), Box> {
- let signer = Signer::new(signing_config);
- let batch_signer = BatchSigner::new(signer);
-
- // Create multiple requests
- let requests = vec![
- BatchSigningRequest {
- method: Method::GET,
- uri: "https://bucket1.s3.amazonaws.com/object1".parse()?,
- headers: HeaderMap::new(),
- body: Vec::new(),
- },
- BatchSigningRequest {
- method: Method::PUT,
- uri: "https://bucket2.s3.amazonaws.com/object2".parse()?,
- headers: HeaderMap::new(),
- body: b"Hello, World!".to_vec(),
- },
- ];
-
- // Sign all requests in batch
- let signed_requests = batch_signer.sign_batch(requests).await?;
-
- for (i, signed_request) in signed_requests.iter().enumerate() {
- println!("Request {}: {:?}", i + 1, signed_request.headers().get("authorization"));
- }
-
- Ok(())
-}
-```
-
-### Custom Authentication
-
-```rust
-use rustfs_signer::{CustomAuthenticator, AuthenticationResult};
-use async_trait::async_trait;
-
-struct CustomAuth {
- // Custom authentication logic
-}
-
-#[async_trait]
-impl CustomAuthenticator for CustomAuth {
- async fn authenticate(&self, request: &Request>) -> Result> {
- // Custom authentication logic
- let auth_header = request.headers().get("authorization");
-
- if let Some(auth) = auth_header {
- // Parse and validate custom authentication
- let auth_str = auth.to_str()?;
-
- if auth_str.starts_with("Custom ") {
- // Validate custom token
- let token = &auth_str[7..];
- if self.validate_token(token).await? {
- return Ok(AuthenticationResult::Authenticated {
- user_id: "custom-user".to_string(),
- permissions: vec!["read".to_string(), "write".to_string()],
- });
- }
- }
- }
-
- Ok(AuthenticationResult::Unauthenticated)
- }
-}
-
-impl CustomAuth {
- async fn validate_token(&self, token: &str) -> Result> {
- // Implement token validation logic
- Ok(token.len() > 10) // Simple example
- }
-}
-```
-
-## ποΈ Architecture
-
-### Signer Architecture
-
-```
-Signer Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Signing API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β SigV4 β Pre-signed β Streaming β Batch β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Signature Calculation β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β HMAC-SHA256 β Canonicalization β String to Sign β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Cryptographic Primitives β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Signing Process
-
-| Step | Description | Purpose |
-|------|-------------|---------|
-| 1. Canonicalize | Format request components | Consistent representation |
-| 2. Create String to Sign | Combine canonicalized data | Prepare for signing |
-| 3. Calculate Signature | HMAC-SHA256 computation | Generate signature |
-| 4. Add Headers | Add signature to request | Complete authentication |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test signature generation
-cargo test signature
-
-# Test pre-signed URLs
-cargo test presigned
-
-# Test validation
-cargo test validation
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Dependencies**: Cryptographic libraries (ring, rustls)
-- **Compatibility**: AWS S3 API compatible
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS IAM](../iam) - Identity and access management
-- [RustFS Crypto](../crypto) - Cryptographic operations
+- AWS Signature Version 4 (SigV4) implementation
+- Pre-signed URL generation and validation
+- Multiple authentication methods (access key, STS token, IAM role)
+- Streaming upload signature support
+- Hardware-accelerated cryptographic operations
+- Multi-region signature support
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Signer API Reference](https://docs.rustfs.com/signer/)
-- [AWS S3 Compatibility](https://docs.rustfs.com/s3-compatibility/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/utils/README.md b/crates/utils/README.md
index 1b6607ee..3df737bf 100644
--- a/crates/utils/README.md
+++ b/crates/utils/README.md
@@ -1,6 +1,6 @@
[](https://rustfs.com)
-# RustFS Utils - Utility Library
+# RustFS Utils - Utility Functions
Essential utility functions and common tools for RustFS distributed object storage
@@ -17,378 +17,21 @@
## π Overview
-**RustFS Utils** is the utility library for the [RustFS](https://rustfs.com) distributed object storage system. It provides a comprehensive collection of utility functions, helper tools, and common functionality used across all RustFS modules, including system operations, cryptographic utilities, compression, and cross-platform compatibility tools.
-
-> **Note:** This is a foundational submodule of RustFS that provides essential utility functions for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Utils** provides essential utility functions and common tools for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π§ System Utilities
-
-- **Cross-Platform Operations**: Unified system operations across platforms
-- **Process Management**: Process spawning and management utilities
-- **Resource Monitoring**: CPU, memory, and disk usage monitoring
-- **Network Utilities**: Network interface and connectivity tools
-
-### π File System Utilities
-
-- **Path Manipulation**: Advanced path handling and normalization
-- **File Operations**: Safe file operations with atomic writes
-- **Directory Management**: Recursive directory operations
-- **Symbolic Link Handling**: Cross-platform symlink management
-
-### ποΈ Compression & Encoding
-
-- **Multiple Algorithms**: Support for gzip, zstd, lz4, and more
-- **Streaming Compression**: Memory-efficient streaming compression
-- **Base64 Encoding**: High-performance base64 operations
-- **URL Encoding**: Safe URL encoding and decoding
-
-### π Cryptographic Utilities
-
-- **Hash Functions**: MD5, SHA1, SHA256, XXHash implementations
-- **Random Generation**: Cryptographically secure random utilities
-- **Certificate Handling**: X.509 certificate parsing and validation
-- **Key Generation**: Secure key generation utilities
-
-### π Network Utilities
-
-- **HTTP Helpers**: HTTP client and server utilities
-- **DNS Resolution**: DNS lookup and resolution tools
-- **Network Interface**: Interface detection and configuration
-- **Protocol Utilities**: Various network protocol helpers
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-utils = "0.1.0"
-
-# Or with specific features
-rustfs-utils = { version = "0.1.0", features = ["compression", "crypto", "network"] }
-```
-
-### Feature Flags
-
-```toml
-[dependencies]
-rustfs-utils = { version = "0.1.0", features = ["full"] }
-```
-
-Available features:
-
-- `compression` - Compression and decompression utilities
-- `crypto` - Cryptographic functions and utilities
-- `network` - Network-related utilities
-- `path` - Advanced path manipulation tools
-- `system` - System monitoring and management
-- `full` - All features enabled
-
-## π§ Usage
-
-### File System Utilities
-
-```rust
-use rustfs_utils::fs::{ensure_dir, atomic_write, safe_remove};
-use rustfs_utils::path::{normalize_path, is_subdirectory};
-
-fn main() -> Result<(), Box> {
- // Ensure directory exists
- ensure_dir("/path/to/directory")?;
-
- // Atomic file write
- atomic_write("/path/to/file.txt", b"Hello, World!")?;
-
- // Path normalization
- let normalized = normalize_path("./some/../path/./file.txt");
- println!("Normalized: {}", normalized.display());
-
- // Check if path is subdirectory
- if is_subdirectory("/safe/path", "/safe/path/subdir") {
- println!("Path is safe");
- }
-
- Ok(())
-}
-```
-
-### Compression Utilities
-
-```rust
-use rustfs_utils::compress::{compress_data, decompress_data, Algorithm};
-
-fn compression_example() -> Result<(), Box> {
- let data = b"This is some test data to compress";
-
- // Compress with different algorithms
- let gzip_compressed = compress_data(data, Algorithm::Gzip)?;
- let zstd_compressed = compress_data(data, Algorithm::Zstd)?;
- let lz4_compressed = compress_data(data, Algorithm::Lz4)?;
-
- // Decompress
- let decompressed = decompress_data(&gzip_compressed, Algorithm::Gzip)?;
- assert_eq!(data, decompressed.as_slice());
-
- println!("Original size: {}", data.len());
- println!("Gzip compressed: {}", gzip_compressed.len());
- println!("Zstd compressed: {}", zstd_compressed.len());
- println!("LZ4 compressed: {}", lz4_compressed.len());
-
- Ok(())
-}
-```
-
-### Cryptographic Utilities
-
-```rust
-use rustfs_utils::crypto::{hash_data, random_bytes, generate_key};
-use rustfs_utils::crypto::HashAlgorithm;
-
-fn crypto_example() -> Result<(), Box> {
- let data = b"Important data to hash";
-
- // Generate hashes
- let md5_hash = hash_data(data, HashAlgorithm::MD5)?;
- let sha256_hash = hash_data(data, HashAlgorithm::SHA256)?;
- let xxhash = hash_data(data, HashAlgorithm::XXHash64)?;
-
- println!("MD5: {}", hex::encode(md5_hash));
- println!("SHA256: {}", hex::encode(sha256_hash));
- println!("XXHash64: {}", hex::encode(xxhash));
-
- // Generate secure random data
- let random_data = random_bytes(32)?;
- println!("Random data: {}", hex::encode(random_data));
-
- // Generate cryptographic key
- let key = generate_key(256)?; // 256-bit key
- println!("Generated key: {}", hex::encode(key));
-
- Ok(())
-}
-```
-
-### System Monitoring
-
-```rust
-use rustfs_utils::sys::{get_system_info, monitor_resources, DiskUsage};
-
-async fn system_monitoring_example() -> Result<(), Box> {
- // Get system information
- let sys_info = get_system_info().await?;
- println!("OS: {} {}", sys_info.os_name, sys_info.os_version);
- println!("CPU: {} cores", sys_info.cpu_cores);
- println!("Total Memory: {} GB", sys_info.total_memory / 1024 / 1024 / 1024);
-
- // Monitor disk usage
- let disk_usage = DiskUsage::for_path("/var/data")?;
- println!("Disk space: {} / {} bytes", disk_usage.used, disk_usage.total);
- println!("Available: {} bytes", disk_usage.available);
-
- // Monitor resources
- let resources = monitor_resources().await?;
- println!("CPU Usage: {:.2}%", resources.cpu_percent);
- println!("Memory Usage: {:.2}%", resources.memory_percent);
-
- Ok(())
-}
-```
-
-### Network Utilities
-
-```rust
-use rustfs_utils::net::{resolve_hostname, get_local_ip, is_port_available};
-
-async fn network_example() -> Result<(), Box> {
- // DNS resolution
- let addresses = resolve_hostname("example.com").await?;
- for addr in addresses {
- println!("Resolved address: {}", addr);
- }
-
- // Get local IP
- let local_ip = get_local_ip().await?;
- println!("Local IP: {}", local_ip);
-
- // Check port availability
- if is_port_available(8080).await? {
- println!("Port 8080 is available");
- } else {
- println!("Port 8080 is in use");
- }
-
- Ok(())
-}
-```
-
-### Certificate Utilities
-
-```rust
-use rustfs_utils::certs::{parse_certificate, validate_certificate_chain, CertificateInfo};
-
-fn certificate_example() -> Result<(), Box> {
- let cert_pem = include_str!("../test_data/certificate.pem");
-
- // Parse certificate
- let cert_info = parse_certificate(cert_pem)?;
- println!("Subject: {}", cert_info.subject);
- println!("Issuer: {}", cert_info.issuer);
- println!("Valid from: {}", cert_info.not_before);
- println!("Valid until: {}", cert_info.not_after);
-
- // Validate certificate chain
- let ca_certs = vec![/* CA certificates */];
- let is_valid = validate_certificate_chain(&cert_info, &ca_certs)?;
-
- if is_valid {
- println!("Certificate chain is valid");
- } else {
- println!("Certificate chain is invalid");
- }
-
- Ok(())
-}
-```
-
-### Encoding Utilities
-
-```rust
-use rustfs_utils::encoding::{base64_encode, base64_decode, url_encode, url_decode};
-
-fn encoding_example() -> Result<(), Box> {
- let data = b"Hello, World!";
-
- // Base64 encoding
- let encoded = base64_encode(data);
- let decoded = base64_decode(&encoded)?;
- assert_eq!(data, decoded.as_slice());
-
- // URL encoding
- let url = "https://example.com/path with spaces?param=value&other=data";
- let encoded_url = url_encode(url);
- let decoded_url = url_decode(&encoded_url)?;
- assert_eq!(url, decoded_url);
-
- println!("Base64 encoded: {}", encoded);
- println!("URL encoded: {}", encoded_url);
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Utils Module Structure
-
-```
-Utils Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Public API Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β File System β Compression β Crypto β Network β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β System Info β Encoding β Certs β Path Utils β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Platform Abstraction Layer β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Operating System Integration β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Feature Overview
-
-| Category | Features | Platform Support |
-|----------|----------|------------------|
-| File System | Atomic operations, path manipulation | All platforms |
-| Compression | Gzip, Zstd, LZ4, Brotli | All platforms |
-| Cryptography | Hashing, random generation, keys | All platforms |
-| System | Resource monitoring, process management | Linux, macOS, Windows |
-| Network | DNS, connectivity, interface detection | All platforms |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Run tests for specific features
-cargo test --features compression
-cargo test --features crypto
-cargo test --features network
-
-# Run tests with all features
-cargo test --features full
-
-# Run benchmarks
-cargo bench
-```
-
-## π Performance
-
-The utils library is optimized for performance:
-
-- **Zero-Copy Operations**: Minimize memory allocations where possible
-- **Lazy Evaluation**: Defer expensive operations until needed
-- **Platform Optimization**: Use platform-specific optimizations
-- **Efficient Algorithms**: Choose the most efficient algorithms for each task
-
-### Benchmarks
-
-| Operation | Performance | Notes |
-|-----------|-------------|-------|
-| Path Normalization | ~50 ns | Uses efficient string operations |
-| Base64 Encoding | ~1.2 GB/s | SIMD-optimized implementation |
-| XXHash64 | ~15 GB/s | Hardware-accelerated when available |
-| File Copy | ~2 GB/s | Platform-optimized copy operations |
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Architecture**: x86_64, aarch64, and others
-- **Dependencies**: Minimal external dependencies
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS ECStore](../ecstore) - Erasure coding storage engine
-- [RustFS Crypto](../crypto) - Cryptographic operations
-- [RustFS Config](../config) - Configuration management
+- Cross-platform system operations and monitoring
+- File system utilities with atomic operations
+- Multi-algorithm compression and encoding support
+- Cryptographic utilities and secure key generation
+- Network utilities and protocol helpers
+- Certificate handling and validation tools
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Utils API Reference](https://docs.rustfs.com/utils/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π§ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/workers/README.md b/crates/workers/README.md
index 86f41fcc..f86b9a95 100644
--- a/crates/workers/README.md
+++ b/crates/workers/README.md
@@ -1,6 +1,6 @@
[](https://rustfs.com)
-# RustFS Workers - Background Job Processing
+# RustFS Workers - Background Job System
Distributed background job processing system for RustFS object storage
@@ -17,446 +17,21 @@
## π Overview
-**RustFS Workers** provides a distributed background job processing system for the [RustFS](https://rustfs.com) distributed object storage system. It handles asynchronous tasks such as data replication, cleanup, healing, indexing, and other maintenance operations across the cluster.
-
-> **Note:** This is a core submodule of RustFS that provides essential background processing capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Workers** provides distributed background job processing capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π Job Processing
-
-- **Distributed Execution**: Jobs run across multiple cluster nodes
-- **Priority Queues**: Multiple priority levels for job scheduling
-- **Retry Logic**: Automatic retry with exponential backoff
-- **Dead Letter Queue**: Failed job isolation and analysis
-
-### π οΈ Built-in Workers
-
-- **Replication Worker**: Data replication across nodes
-- **Cleanup Worker**: Garbage collection and cleanup
-- **Healing Worker**: Data integrity repair
-- **Indexing Worker**: Metadata indexing and search
-- **Metrics Worker**: Performance metrics collection
-
-### π Scalability Features
-
-- **Horizontal Scaling**: Add worker nodes dynamically
-- **Load Balancing**: Intelligent job distribution
-- **Circuit Breaker**: Prevent cascading failures
-- **Rate Limiting**: Control resource consumption
-
-### π§ Management & Monitoring
-
-- **Job Tracking**: Real-time job status monitoring
-- **Health Checks**: Worker health and availability
-- **Metrics Collection**: Performance and throughput metrics
-- **Administrative Interface**: Job management and control
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-workers = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Worker Setup
-
-```rust
-use rustfs_workers::{WorkerManager, WorkerConfig, JobQueue};
-use std::time::Duration;
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Create worker configuration
- let config = WorkerConfig {
- worker_id: "worker-1".to_string(),
- max_concurrent_jobs: 10,
- job_timeout: Duration::from_secs(300),
- retry_limit: 3,
- cleanup_interval: Duration::from_secs(60),
- };
-
- // Create worker manager
- let worker_manager = WorkerManager::new(config).await?;
-
- // Start worker processing
- worker_manager.start().await?;
-
- // Keep running
- tokio::signal::ctrl_c().await?;
- worker_manager.shutdown().await?;
-
- Ok(())
-}
-```
-
-### Job Definition and Scheduling
-
-```rust
-use rustfs_workers::{Job, JobBuilder, JobPriority, JobQueue};
-use serde::{Deserialize, Serialize};
-use async_trait::async_trait;
-
-#[derive(Debug, Clone, Serialize, Deserialize)]
-pub struct ReplicationJob {
- pub source_path: String,
- pub target_nodes: Vec,
- pub replication_factor: u32,
-}
-
-#[async_trait]
-impl Job for ReplicationJob {
- async fn execute(&self) -> Result<(), Box> {
- println!("Starting replication job for: {}", self.source_path);
-
- // Perform replication logic
- for node in &self.target_nodes {
- self.replicate_to_node(node).await?;
- }
-
- println!("Replication job completed successfully");
- Ok(())
- }
-
- fn job_type(&self) -> &str {
- "replication"
- }
-
- fn max_retries(&self) -> u32 {
- 3
- }
-}
-
-impl ReplicationJob {
- async fn replicate_to_node(&self, node: &str) -> Result<(), Box> {
- // Implementation for replicating data to a specific node
- println!("Replicating {} to node: {}", self.source_path, node);
- tokio::time::sleep(Duration::from_secs(1)).await; // Simulate work
- Ok(())
- }
-}
-
-async fn schedule_replication_job() -> Result<(), Box> {
- let job_queue = JobQueue::new().await?;
-
- // Create replication job
- let job = ReplicationJob {
- source_path: "/bucket/important-file.txt".to_string(),
- target_nodes: vec!["node-2".to_string(), "node-3".to_string()],
- replication_factor: 2,
- };
-
- // Schedule job with high priority
- let job_id = job_queue.schedule_job(
- Box::new(job),
- JobPriority::High,
- None, // Execute immediately
- ).await?;
-
- println!("Scheduled replication job with ID: {}", job_id);
- Ok(())
-}
-```
-
-### Custom Worker Implementation
-
-```rust
-use rustfs_workers::{Worker, WorkerContext, JobResult};
-use async_trait::async_trait;
-
-pub struct CleanupWorker {
- storage_path: String,
- max_file_age: Duration,
-}
-
-#[async_trait]
-impl Worker for CleanupWorker {
- async fn process_job(&self, job: Box, context: &WorkerContext) -> JobResult {
- match job.job_type() {
- "cleanup" => {
- if let Some(cleanup_job) = job.as_any().downcast_ref::() {
- self.execute_cleanup(cleanup_job, context).await
- } else {
- JobResult::Failed("Invalid job type for cleanup worker".to_string())
- }
- }
- _ => JobResult::Skipped,
- }
- }
-
- async fn health_check(&self) -> bool {
- // Check if storage is accessible
- tokio::fs::metadata(&self.storage_path).await.is_ok()
- }
-
- fn worker_type(&self) -> &str {
- "cleanup"
- }
-}
-
-impl CleanupWorker {
- pub fn new(storage_path: String, max_file_age: Duration) -> Self {
- Self {
- storage_path,
- max_file_age,
- }
- }
-
- async fn execute_cleanup(&self, job: &CleanupJob, context: &WorkerContext) -> JobResult {
- println!("Starting cleanup job for: {}", job.target_path);
-
- match self.cleanup_old_files(&job.target_path).await {
- Ok(cleaned_count) => {
- context.update_metrics("files_cleaned", cleaned_count).await;
- JobResult::Success
- }
- Err(e) => JobResult::Failed(e.to_string()),
- }
- }
-
- async fn cleanup_old_files(&self, path: &str) -> Result> {
- let mut cleaned_count = 0;
- // Implementation for cleaning up old files
- // ... cleanup logic ...
- Ok(cleaned_count)
- }
-}
-```
-
-### Job Queue Management
-
-```rust
-use rustfs_workers::{JobQueue, JobFilter, JobStatus};
-
-async fn job_queue_management() -> Result<(), Box> {
- let job_queue = JobQueue::new().await?;
-
- // List pending jobs
- let pending_jobs = job_queue.list_jobs(JobFilter {
- status: Some(JobStatus::Pending),
- job_type: None,
- priority: None,
- limit: Some(100),
- }).await?;
-
- println!("Pending jobs: {}", pending_jobs.len());
-
- // Cancel a job
- let job_id = "job-123";
- job_queue.cancel_job(job_id).await?;
-
- // Retry failed jobs
- let failed_jobs = job_queue.list_jobs(JobFilter {
- status: Some(JobStatus::Failed),
- job_type: None,
- priority: None,
- limit: Some(50),
- }).await?;
-
- for job in failed_jobs {
- job_queue.retry_job(&job.id).await?;
- }
-
- // Get job statistics
- let stats = job_queue.get_statistics().await?;
- println!("Job statistics: {:?}", stats);
-
- Ok(())
-}
-```
-
-### Distributed Worker Coordination
-
-```rust
-use rustfs_workers::{WorkerCluster, WorkerNode, ClusterConfig};
-
-async fn distributed_worker_setup() -> Result<(), Box> {
- let cluster_config = ClusterConfig {
- node_id: "worker-node-1".to_string(),
- cluster_endpoint: "https://cluster.rustfs.local".to_string(),
- heartbeat_interval: Duration::from_secs(30),
- leader_election_timeout: Duration::from_secs(60),
- };
-
- // Create worker cluster
- let cluster = WorkerCluster::new(cluster_config).await?;
-
- // Register worker types
- cluster.register_worker_type("replication", Box::new(ReplicationWorkerFactory)).await?;
- cluster.register_worker_type("cleanup", Box::new(CleanupWorkerFactory)).await?;
- cluster.register_worker_type("healing", Box::new(HealingWorkerFactory)).await?;
-
- // Start cluster participation
- cluster.join().await?;
-
- // Handle cluster events
- let mut event_receiver = cluster.event_receiver();
-
- tokio::spawn(async move {
- while let Some(event) = event_receiver.recv().await {
- match event {
- ClusterEvent::NodeJoined(node) => {
- println!("Worker node joined: {}", node.id);
- }
- ClusterEvent::NodeLeft(node) => {
- println!("Worker node left: {}", node.id);
- }
- ClusterEvent::LeadershipChanged(new_leader) => {
- println!("New cluster leader: {}", new_leader);
- }
- }
- }
- });
-
- Ok(())
-}
-```
-
-### Job Monitoring and Metrics
-
-```rust
-use rustfs_workers::{JobMonitor, WorkerMetrics, AlertConfig};
-
-async fn job_monitoring_setup() -> Result<(), Box> {
- let monitor = JobMonitor::new().await?;
-
- // Set up alerting
- let alert_config = AlertConfig {
- failed_job_threshold: 10,
- worker_down_threshold: Duration::from_secs(300),
- queue_size_threshold: 1000,
- notification_endpoint: "https://alerts.example.com/webhook".to_string(),
- };
-
- monitor.configure_alerts(alert_config).await?;
-
- // Start monitoring
- monitor.start_monitoring().await?;
-
- // Get real-time metrics
- let metrics = monitor.get_metrics().await?;
- println!("Worker metrics: {:?}", metrics);
-
- // Set up periodic reporting
- tokio::spawn(async move {
- let mut interval = tokio::time::interval(Duration::from_secs(60));
-
- loop {
- interval.tick().await;
-
- if let Ok(metrics) = monitor.get_metrics().await {
- println!("=== Worker Metrics ===");
- println!("Active jobs: {}", metrics.active_jobs);
- println!("Completed jobs: {}", metrics.completed_jobs);
- println!("Failed jobs: {}", metrics.failed_jobs);
- println!("Queue size: {}", metrics.queue_size);
- println!("Worker count: {}", metrics.worker_count);
- }
- }
- });
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Workers Architecture
-
-```
-Workers Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Job Management API β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Scheduling β Monitoring β Queue Mgmt β Metrics β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Worker Coordination β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Job Queue β Load Balancer β Health Check β Retry β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Distributed Execution β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Worker Types
-
-| Worker Type | Purpose | Characteristics |
-|-------------|---------|----------------|
-| Replication | Data replication | I/O intensive, network bound |
-| Cleanup | Garbage collection | CPU intensive, periodic |
-| Healing | Data repair | I/O intensive, high priority |
-| Indexing | Metadata indexing | CPU intensive, background |
-| Metrics | Performance monitoring | Low resource, continuous |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test job processing
-cargo test job_processing
-
-# Test worker coordination
-cargo test worker_coordination
-
-# Test distributed scenarios
-cargo test distributed
-
-# Integration tests
-cargo test --test integration
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Network**: Cluster connectivity required
-- **Storage**: Persistent queue storage recommended
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS Common](../common) - Common types and utilities
-- [RustFS Lock](../lock) - Distributed locking
+- Distributed job execution across cluster nodes
+- Priority-based job scheduling and queue management
+- Built-in workers for replication, cleanup, healing, and indexing
+- Automatic retry logic with exponential backoff
+- Horizontal scaling with load balancing
+- Real-time job monitoring and administrative interface
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Workers API Reference](https://docs.rustfs.com/workers/)
-- [Job Processing Guide](https://docs.rustfs.com/jobs/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
diff --git a/crates/zip/README.md b/crates/zip/README.md
index 93a665db..ef71fb9a 100644
--- a/crates/zip/README.md
+++ b/crates/zip/README.md
@@ -17,391 +17,21 @@
## π Overview
-**RustFS Zip** provides high-performance compression and archiving capabilities for the [RustFS](https://rustfs.com) distributed object storage system. It supports multiple compression algorithms, streaming compression, and efficient archiving operations optimized for storage systems.
-
-> **Note:** This is a performance-critical submodule of RustFS that provides essential compression capabilities for the distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
+**RustFS Zip** provides high-performance compression and archiving capabilities for the [RustFS](https://rustfs.com) distributed object storage system. For the complete RustFS experience, please visit the [main RustFS repository](https://github.com/rustfs/rustfs).
## β¨ Features
-### π¦ Compression Algorithms
-
-- **Zstandard (Zstd)**: Fast compression with excellent ratios
-- **LZ4**: Ultra-fast compression for real-time applications
-- **Gzip**: Industry-standard compression for compatibility
-- **Brotli**: Web-optimized compression for text content
-
-### π Performance Features
-
-- **Streaming Compression**: Compress data on-the-fly without buffering
-- **Parallel Processing**: Multi-threaded compression for large files
-- **Adaptive Compression**: Automatic algorithm selection based on data
-- **Hardware Acceleration**: Leverage CPU-specific optimizations
-
-### π§ Archive Management
-
-- **ZIP Format**: Standard ZIP archive creation and extraction
-- **TAR Format**: UNIX-style tar archive support
-- **Custom Formats**: RustFS-optimized archive formats
-- **Metadata Preservation**: Maintain file attributes and timestamps
-
-### π Compression Analytics
-
-- **Ratio Analysis**: Detailed compression statistics
-- **Performance Metrics**: Compression speed and efficiency
-- **Content-Type Detection**: Automatic compression algorithm selection
-- **Deduplication**: Identify and handle duplicate content
-
-## π¦ Installation
-
-Add this to your `Cargo.toml`:
-
-```toml
-[dependencies]
-rustfs-zip = "0.1.0"
-```
-
-## π§ Usage
-
-### Basic Compression
-
-```rust
-use rustfs_zip::{Compressor, CompressionLevel, CompressionAlgorithm};
-
-#[tokio::main]
-async fn main() -> Result<(), Box> {
- // Create compressor
- let compressor = Compressor::new(CompressionAlgorithm::Zstd, CompressionLevel::Default);
-
- // Compress data
- let input_data = b"Hello, World! This is some test data to compress.";
- let compressed = compressor.compress(input_data).await?;
-
- println!("Original size: {} bytes", input_data.len());
- println!("Compressed size: {} bytes", compressed.len());
- println!("Compression ratio: {:.2}%",
- (1.0 - compressed.len() as f64 / input_data.len() as f64) * 100.0);
-
- // Decompress data
- let decompressed = compressor.decompress(&compressed).await?;
- assert_eq!(input_data, decompressed.as_slice());
-
- Ok(())
-}
-```
-
-### Streaming Compression
-
-```rust
-use rustfs_zip::{StreamingCompressor, StreamingDecompressor};
-use tokio::io::{AsyncReadExt, AsyncWriteExt};
-
-async fn streaming_compression_example() -> Result<(), Box> {
- // Create streaming compressor
- let mut compressor = StreamingCompressor::new(
- CompressionAlgorithm::Zstd,
- CompressionLevel::Fast,
- )?;
-
- // Compress streaming data
- let input = tokio::fs::File::open("large_file.txt").await?;
- let output = tokio::fs::File::create("compressed_file.zst").await?;
-
- let mut reader = tokio::io::BufReader::new(input);
- let mut writer = tokio::io::BufWriter::new(output);
-
- // Stream compression
- let mut buffer = vec![0u8; 8192];
- loop {
- let bytes_read = reader.read(&mut buffer).await?;
- if bytes_read == 0 {
- break;
- }
-
- let compressed_chunk = compressor.compress_chunk(&buffer[..bytes_read]).await?;
- writer.write_all(&compressed_chunk).await?;
- }
-
- // Finalize compression
- let final_chunk = compressor.finalize().await?;
- writer.write_all(&final_chunk).await?;
- writer.flush().await?;
-
- Ok(())
-}
-```
-
-### Archive Creation
-
-```rust
-use rustfs_zip::{ZipArchive, ArchiveBuilder, CompressionMethod};
-
-async fn create_archive_example() -> Result<(), Box> {
- // Create archive builder
- let mut builder = ArchiveBuilder::new("backup.zip".to_string());
-
- // Add files to archive
- builder.add_file("config.json", "config/app.json", CompressionMethod::Deflate).await?;
- builder.add_file("data.txt", "data/sample.txt", CompressionMethod::Store).await?;
-
- // Add directory
- builder.add_directory("logs/", "application_logs/").await?;
-
- // Create archive
- let archive = builder.build().await?;
-
- println!("Archive created: {}", archive.path());
- println!("Total files: {}", archive.file_count());
- println!("Compressed size: {} bytes", archive.compressed_size());
- println!("Uncompressed size: {} bytes", archive.uncompressed_size());
-
- Ok(())
-}
-```
-
-### Archive Extraction
-
-```rust
-use rustfs_zip::{ZipExtractor, ExtractionOptions};
-
-async fn extract_archive_example() -> Result<(), Box> {
- // Create extractor
- let extractor = ZipExtractor::new("backup.zip".to_string());
-
- // List archive contents
- let entries = extractor.list_entries().await?;
- for entry in &entries {
- println!("File: {} ({} bytes)", entry.name, entry.size);
- }
-
- // Extract specific file
- let file_data = extractor.extract_file("config.json").await?;
- println!("Extracted config.json: {} bytes", file_data.len());
-
- // Extract all files
- let extraction_options = ExtractionOptions {
- output_directory: "extracted/".to_string(),
- preserve_paths: true,
- overwrite_existing: false,
- };
-
- extractor.extract_all(extraction_options).await?;
- println!("Archive extracted successfully");
-
- Ok(())
-}
-```
-
-### Adaptive Compression
-
-```rust
-use rustfs_zip::{AdaptiveCompressor, ContentAnalyzer, CompressionProfile};
-
-async fn adaptive_compression_example() -> Result<(), Box> {
- // Create adaptive compressor
- let mut compressor = AdaptiveCompressor::new();
-
- // Configure compression profiles
- compressor.add_profile(CompressionProfile {
- content_type: "text/*".to_string(),
- algorithm: CompressionAlgorithm::Brotli,
- level: CompressionLevel::High,
- min_size: 1024,
- });
-
- compressor.add_profile(CompressionProfile {
- content_type: "image/*".to_string(),
- algorithm: CompressionAlgorithm::Lz4,
- level: CompressionLevel::Fast,
- min_size: 10240,
- });
-
- // Compress different types of content
- let text_content = std::fs::read("document.txt")?;
- let image_content = std::fs::read("photo.jpg")?;
-
- // Analyze and compress
- let text_result = compressor.compress_adaptive(&text_content, Some("text/plain")).await?;
- let image_result = compressor.compress_adaptive(&image_content, Some("image/jpeg")).await?;
-
- println!("Text compression: {} -> {} bytes ({})",
- text_content.len(), text_result.compressed_size, text_result.algorithm);
- println!("Image compression: {} -> {} bytes ({})",
- image_content.len(), image_result.compressed_size, image_result.algorithm);
-
- Ok(())
-}
-```
-
-### Parallel Compression
-
-```rust
-use rustfs_zip::{ParallelCompressor, CompressionJob};
-
-async fn parallel_compression_example() -> Result<(), Box> {
- // Create parallel compressor
- let compressor = ParallelCompressor::new(4); // 4 worker threads
-
- // Prepare compression jobs
- let jobs = vec![
- CompressionJob {
- id: "file1".to_string(),
- data: std::fs::read("file1.txt")?,
- algorithm: CompressionAlgorithm::Zstd,
- level: CompressionLevel::Default,
- },
- CompressionJob {
- id: "file2".to_string(),
- data: std::fs::read("file2.txt")?,
- algorithm: CompressionAlgorithm::Lz4,
- level: CompressionLevel::Fast,
- },
- CompressionJob {
- id: "file3".to_string(),
- data: std::fs::read("file3.txt")?,
- algorithm: CompressionAlgorithm::Gzip,
- level: CompressionLevel::High,
- },
- ];
-
- // Execute parallel compression
- let results = compressor.compress_batch(jobs).await?;
-
- for result in results {
- println!("Job {}: {} -> {} bytes",
- result.job_id, result.original_size, result.compressed_size);
- }
-
- Ok(())
-}
-```
-
-### Content Deduplication
-
-```rust
-use rustfs_zip::{DeduplicationCompressor, ContentHash};
-
-async fn deduplication_example() -> Result<(), Box> {
- // Create deduplication compressor
- let mut compressor = DeduplicationCompressor::new();
-
- // Add files for compression
- let file1 = std::fs::read("document1.txt")?;
- let file2 = std::fs::read("document2.txt")?;
- let file3 = std::fs::read("document1.txt")?; // Duplicate of file1
-
- // Compress with deduplication
- let result1 = compressor.compress_with_dedup("doc1", &file1).await?;
- let result2 = compressor.compress_with_dedup("doc2", &file2).await?;
- let result3 = compressor.compress_with_dedup("doc3", &file3).await?;
-
- println!("File 1: {} bytes -> {} bytes", file1.len(), result1.compressed_size);
- println!("File 2: {} bytes -> {} bytes", file2.len(), result2.compressed_size);
- println!("File 3: {} bytes -> {} bytes (deduplicated: {})",
- file3.len(), result3.compressed_size, result3.is_deduplicated);
-
- // Get deduplication statistics
- let stats = compressor.get_dedup_stats();
- println!("Deduplication saved: {} bytes", stats.bytes_saved);
- println!("Duplicate files found: {}", stats.duplicate_count);
-
- Ok(())
-}
-```
-
-## ποΈ Architecture
-
-### Zip Module Architecture
-
-```
-Zip Architecture:
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-β Compression API β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Algorithm β Streaming β Archive β Adaptive β
-β Selection β Compression β Management β Compressionβ
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Compression Engines β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Zstd β LZ4 β Gzip β Brotli β Custom β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
-β Low-Level Compression β
-βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
-```
-
-### Compression Algorithms
-
-| Algorithm | Speed | Ratio | Use Case |
-|-----------|-------|-------|----------|
-| LZ4 | Very Fast | Good | Real-time compression |
-| Zstd | Fast | Excellent | General purpose |
-| Gzip | Medium | Good | Web compatibility |
-| Brotli | Slow | Excellent | Text/web content |
-
-## π§ͺ Testing
-
-Run the test suite:
-
-```bash
-# Run all tests
-cargo test
-
-# Test compression algorithms
-cargo test algorithms
-
-# Test streaming compression
-cargo test streaming
-
-# Test archive operations
-cargo test archive
-
-# Benchmark compression performance
-cargo bench
-```
-
-## π Requirements
-
-- **Rust**: 1.70.0 or later
-- **Platforms**: Linux, macOS, Windows
-- **Dependencies**: Native compression libraries
-- **Memory**: Sufficient RAM for compression buffers
-
-## π Related Projects
-
-This module is part of the RustFS ecosystem:
-
-- [RustFS Main](https://github.com/rustfs/rustfs) - Core distributed storage system
-- [RustFS Rio](../rio) - High-performance I/O
-- [RustFS Utils](../utils) - Utility functions
+- Multiple compression algorithms (Zstd, LZ4, Gzip, Brotli)
+- Streaming compression for memory efficiency
+- Parallel processing for large files
+- Archive format support (ZIP, TAR, custom formats)
+- Adaptive compression with content-type detection
+- Compression analytics and performance metrics
## π Documentation
-For comprehensive documentation, visit:
-
-- [RustFS Documentation](https://docs.rustfs.com)
-- [Zip API Reference](https://docs.rustfs.com/zip/)
-- [Compression Guide](https://docs.rustfs.com/compression/)
-
-## π Links
-
-- [Documentation](https://docs.rustfs.com) - Complete RustFS manual
-- [Changelog](https://github.com/rustfs/rustfs/releases) - Release notes and updates
-- [GitHub Discussions](https://github.com/rustfs/rustfs/discussions) - Community support
-
-## π€ Contributing
-
-We welcome contributions! Please see our [Contributing Guide](https://github.com/rustfs/rustfs/blob/main/CONTRIBUTING.md) for details.
+For comprehensive documentation, examples, and usage guides, please visit the main [RustFS repository](https://github.com/rustfs/rustfs).
## π License
-Licensed under the Apache License, Version 2.0. See [LICENSE](https://github.com/rustfs/rustfs/blob/main/LICENSE) for details.
-
----
-
-
- RustFS is a trademark of RustFS, Inc.
- All other trademarks are the property of their respective owners.
-
-
-
- Made with π¦ by the RustFS Team
-
+This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.