rustfs/docs/PHASE4_GUIDE.md

# Phase 4: Full Integration Guide

## Overview

Phase 4 represents the final stage of the adaptive buffer sizing migration path. It provides a unified, profile-based implementation with deprecated legacy functions and optional performance metrics.

## What's New in Phase 4

### 1. Deprecated Legacy Function

The `get_adaptive_buffer_size()` function is now deprecated:

```rust
#[deprecated(
    since = "Phase 4",
    note = "Use workload profile configuration instead."
)]
fn get_adaptive_buffer_size(file_size: i64) -> usize
```

**Why Deprecated?**
- Profile-based approach is more flexible and powerful
- Encourages use of the unified configuration system
- Simplifies maintenance and future enhancements

**Still Works:**
- Function is maintained for backward compatibility
- Internally delegates to GeneralPurpose profile
- No breaking changes for existing code

### 2. Profile-Only Implementation

All buffer sizing now goes through workload profiles:

**Before (Phase 3):**
```rust
fn get_buffer_size_opt_in(file_size: i64) -> usize {
    if is_buffer_profile_enabled() {
        // Use profiles
    } else {
        // Fall back to hardcoded get_adaptive_buffer_size()
    }
}
```

**After (Phase 4):**
```rust
fn get_buffer_size_opt_in(file_size: i64) -> usize {
    if is_buffer_profile_enabled() {
        // Use configured profile
    } else {
        // Use GeneralPurpose profile (no hardcoded values)
    }
}
```

**Benefits:**
- Consistent behavior across all modes
- Single source of truth for buffer sizes
- Easier to test and maintain

### 3. Performance Metrics

Optional metrics collection for monitoring and optimization:

```rust
#[cfg(feature = "metrics")]
{
    metrics::histogram!("buffer_size_bytes", buffer_size as f64);
    metrics::counter!("buffer_size_selections", 1);
    
    if file_size >= 0 {
        let ratio = buffer_size as f64 / file_size as f64;
        metrics::histogram!("buffer_to_file_ratio", ratio);
    }
}
```

## Migration Guide

### From Phase 3 to Phase 4

**Good News:** No action required for most users!

Phase 4 is fully backward compatible with Phase 3. Your existing configurations and deployments continue to work without changes.

### If You Have Custom Code

If your code directly calls `get_adaptive_buffer_size()`:

**Option 1: Update to use the profile system (Recommended)**
```rust
// Old code
let buffer_size = get_adaptive_buffer_size(file_size);

// New code - let the system handle it
// (buffer sizing happens automatically in put_object, upload_part, etc.)
```

**Option 2: Suppress deprecation warnings**
```rust
// If you must keep calling it directly
#[allow(deprecated)]
let buffer_size = get_adaptive_buffer_size(file_size);
```

**Option 3: Use the new API explicitly**
```rust
// Use the profile system directly
use rustfs::config::workload_profiles::{WorkloadProfile, RustFSBufferConfig};

let config = RustFSBufferConfig::new(WorkloadProfile::GeneralPurpose);
let buffer_size = config.get_buffer_size(file_size);
```

## Performance Metrics

### Enabling Metrics

**At Build Time:**
```bash
cargo build --features metrics --release
```

**In Cargo.toml:**
```toml
[dependencies]
rustfs = { version = "*", features = ["metrics"] }
```

### Available Metrics

| Metric Name | Type | Description |
|------------|------|-------------|
| `buffer_size_bytes` | Histogram | Distribution of selected buffer sizes |
| `buffer_size_selections` | Counter | Total number of buffer size calculations |
| `buffer_to_file_ratio` | Histogram | Ratio of buffer size to file size |

### Using Metrics

**With Prometheus:**
```rust
// Metrics are automatically exported to Prometheus format
// Access at http://localhost:9090/metrics
```

**With Custom Backend:**
```rust
// Use the metrics crate's recorder interface
use metrics_exporter_prometheus::PrometheusBuilder;

PrometheusBuilder::new()
    .install()
    .expect("failed to install Prometheus recorder");
```

### Analyzing Metrics

**Buffer Size Distribution:**
```promql
# Most common buffer sizes
histogram_quantile(0.5, buffer_size_bytes)  # Median
histogram_quantile(0.95, buffer_size_bytes) # 95th percentile
histogram_quantile(0.99, buffer_size_bytes) # 99th percentile
```

**Buffer Efficiency:**
```promql
# Average ratio of buffer to file size
avg(buffer_to_file_ratio)

# Files where buffer is > 10% of file size
buffer_to_file_ratio > 0.1
```

**Usage Patterns:**
```promql
# Rate of buffer size selections
rate(buffer_size_selections[5m])

# Total selections over time
increase(buffer_size_selections[1h])
```

## Optimizing Based on Metrics

### Scenario 1: High Memory Usage

**Symptom:** Most buffers are at maximum size
```promql
histogram_quantile(0.9, buffer_size_bytes) > 1048576  # 1MB
```

**Solution:**
- Switch to a more conservative profile
- Use SecureStorage or WebWorkload profile
- Or create custom profile with lower max_size

### Scenario 2: Poor Throughput

**Symptom:** Buffer-to-file ratio is very small
```promql
avg(buffer_to_file_ratio) < 0.01  # Less than 1%
```

**Solution:**
- Switch to a more aggressive profile
- Use AiTraining or DataAnalytics profile
- Increase buffer sizes for your workload

### Scenario 3: Mismatched Profile

**Symptom:** Wide distribution of file sizes with single profile
```promql
# High variance in buffer sizes
stddev(buffer_size_bytes) > 500000
```

**Solution:**
- Consider per-bucket profiles (future feature)
- Use GeneralPurpose for mixed workloads
- Or implement custom thresholds

## Testing Phase 4

### Unit Tests

Run the Phase 4 specific tests:
```bash
cd /home/runner/work/rustfs/rustfs
cargo test test_phase4_full_integration
```

### Integration Tests

Test with different configurations:
```bash
# Test default behavior
./rustfs /data

# Test with different profiles
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /data

# Test opt-out mode
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
```

### Metrics Verification

With metrics enabled:
```bash
# Build with metrics
cargo build --features metrics --release

# Run and check metrics endpoint
./target/release/rustfs /data &
curl http://localhost:9090/metrics | grep buffer_size
```

## Troubleshooting

### Q: I'm getting deprecation warnings

**A:** You're calling `get_adaptive_buffer_size()` directly. Options:
1. Remove the direct call (let the system handle it)
2. Use `#[allow(deprecated)]` to suppress warnings
3. Migrate to the profile system API

### Q: How do I know which profile is being used?

**A:** Check the startup logs:
```
Buffer profiling is enabled by default (Phase 3), profile: GeneralPurpose
Using buffer profile: GeneralPurpose
```

### Q: Can I still opt-out in Phase 4?

**A:** Yes! Use `--buffer-profile-disable`:
```bash
export RUSTFS_BUFFER_PROFILE_DISABLE=true
./rustfs /data
```

This uses GeneralPurpose profile (same buffer sizes as PR #869).

### Q: What's the difference between opt-out in Phase 3 vs Phase 4?

**A:**
- **Phase 3**: Opt-out uses hardcoded legacy function
- **Phase 4**: Opt-out uses GeneralPurpose profile
- **Result**: Identical buffer sizes, but Phase 4 is profile-based

### Q: Do I need to enable metrics?

**A:** No, metrics are completely optional. They're useful for:
- Production monitoring
- Performance analysis
- Profile optimization
- Capacity planning

If you don't need these, skip the metrics feature.

## Best Practices

### 1. Let the System Handle Buffer Sizing

**Don't:**
```rust
// Avoid direct calls
let buffer_size = get_adaptive_buffer_size(file_size);
let reader = BufReader::with_capacity(buffer_size, file);
```

**Do:**
```rust
// Let put_object/upload_part handle it automatically
// Buffer sizing happens transparently
```

### 2. Use Appropriate Profiles

Match your profile to your workload:
- AI/ML models: `AiTraining`
- Static assets: `WebWorkload`
- Mixed files: `GeneralPurpose`
- Compliance: `SecureStorage`

### 3. Monitor in Production

Enable metrics in production:
```bash
cargo build --features metrics --release
```

Use the data to:
- Validate profile choice
- Identify optimization opportunities
- Plan capacity

### 4. Test Profile Changes

Before changing profiles in production:
```bash
# Test in staging
export RUSTFS_BUFFER_PROFILE=AiTraining
./rustfs /staging-data

# Monitor metrics for a period
# Compare with baseline

# Roll out to production when validated
```

## Future Enhancements

Based on collected metrics, future versions may include:

1. **Auto-tuning**: Automatically adjust profiles based on observed patterns
2. **Per-bucket profiles**: Different profiles for different buckets
3. **Dynamic thresholds**: Adjust thresholds based on system load
4. **ML-based optimization**: Use machine learning to optimize buffer sizes
5. **Adaptive limits**: Automatically adjust max_size based on available memory

## Conclusion

Phase 4 represents the mature state of the adaptive buffer sizing system:
- ✅ Unified, profile-based implementation
- ✅ Deprecated legacy code (but backward compatible)
- ✅ Optional performance metrics
- ✅ Production-ready and battle-tested
- ✅ Future-proof and extensible

Most users can continue using the system without any changes, while advanced users gain powerful new capabilities for monitoring and optimization.

## References

- [Adaptive Buffer Sizing Guide](./adaptive-buffer-sizing.md)
- [Implementation Summary](./IMPLEMENTATION_SUMMARY.md)
- [Phase 3 Migration Guide](./MIGRATION_PHASE3.md)
- [Performance Testing Guide](./PERFORMANCE_TESTING.md)