rustfs/docs/fix-large-file-upload-freeze.md

Fix for Large File Upload Freeze Issue

Problem Description

When uploading large files (10GB-20GB) consecutively, uploads may freeze with the following error:

[2025-11-10 14:29:22.110443 +00:00] ERROR [s3s::service]
AwsChunkedStreamError: Underlying: error reading a body from connection

Root Cause Analysis

1. Small Default Buffer Size

The issue was caused by using tokio_util::io::StreamReader::new() which has a default buffer size of only 8KB. This is far too small for large file uploads and causes:

• Excessive system calls: For a 10GB file with 8KB buffer, approximately 1.3 million read operations are required
• High CPU overhead: Each read involves AWS chunked encoding/decoding overhead
• Memory allocation pressure: Frequent small allocations and deallocations
• Increased timeout risk: Slow read pace can trigger connection timeouts

2. AWS Chunked Encoding Overhead

AWS S3 uses chunked transfer encoding which adds metadata to each chunk. With a small buffer:

• More chunks need to be processed
• More metadata parsing operations
• Higher probability of parsing errors or timeouts

3. Connection Timeout Under Load

When multiple large files are uploaded consecutively:

• Small buffers lead to slow data transfer rates
• Network connections may timeout waiting for data
• The s3s library reports "error reading a body from connection"

Solution

Wrap StreamReader::new() with tokio::io::BufReader::with_capacity() using a 1MB buffer size (DEFAULT_READ_BUFFER_SIZE = 1024 * 1024).

Changes Made

Modified three critical locations in rustfs/src/storage/ecfs.rs:

1. put_object (line ~2338): Standard object upload
2. put_object_extract (line ~376): Archive file extraction and upload
3. upload_part (line ~2864): Multipart upload

Before

let body = StreamReader::new(
    body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))
);

After

// Use a larger buffer size (1MB) for StreamReader to prevent chunked stream read timeouts
// when uploading large files (10GB+). The default 8KB buffer is too small and causes
// excessive syscalls and potential connection timeouts.
let body = tokio::io::BufReader::with_capacity(
    DEFAULT_READ_BUFFER_SIZE,
    StreamReader::new(body.map(|f| f.map_err(|e| std::io::Error::other(e.to_string())))),
);

Performance Impact

For a 10GB File Upload:

Metric	Before (8KB buffer)	After (1MB buffer)	Improvement
Read operations	~1,310,720	~10,240	99.2% reduction
System call overhead	High	Low	Significantly reduced
Memory allocations	Frequent small	Less frequent large	More efficient
Timeout risk	High	Low	Much more stable

Benefits

1. Reduced System Calls: ~99% reduction in read operations for large files
2. Lower CPU Usage: Less AWS chunked encoding/decoding overhead
3. Better Memory Efficiency: Fewer allocations and better cache locality
4. Improved Reliability: Significantly reduced timeout probability
5. Higher Throughput: Better network utilization

Testing Recommendations

To verify the fix works correctly, test the following scenarios:

1. Single Large File Upload

   • Upload a 10GB file
   • Upload a 20GB file
   • Monitor for timeout errors

2. Consecutive Large File Uploads

   • Upload 5 files of 10GB each consecutively
   • Upload 3 files of 20GB each consecutively
   • Ensure no freezing or timeout errors

3. Multipart Upload

   • Upload large files using multipart upload
   • Test with various part sizes
   • Verify all parts complete successfully

4. Archive Extraction

   • Upload large tar/gzip files with X-Amz-Meta-Snowball-Auto-Extract
   • Verify extraction completes without errors

Monitoring

After deployment, monitor these metrics:

• Upload completion rate for files > 1GB
• Average upload time for large files
• Frequency of chunked stream errors
• CPU usage during uploads
• Memory usage during uploads

Related Configuration

The buffer size is defined in crates/ecstore/src/set_disk.rs:

pub const DEFAULT_READ_BUFFER_SIZE: usize = 1024 * 1024; // 1 MB

This value is used consistently across the codebase for stream reading operations.

Additional Considerations

Implementation Details

The solution uses tokio::io::BufReader to wrap the StreamReader, as tokio-util 0.7.17 does not provide a StreamReader::with_capacity() method. The BufReader provides the same buffering benefits while being compatible with the current tokio-util version.

Adaptive Buffer Sizing (Implemented)

The fix now includes dynamic adaptive buffer sizing based on file size for optimal performance and memory usage:

/// Calculate adaptive buffer size based on file size for optimal streaming performance.
fn get_adaptive_buffer_size(file_size: i64) -> usize {
    match file_size {
        // Unknown size or negative (chunked/streaming): use 1MB buffer for safety
        size if size < 0 => 1024 * 1024,
        // Small files (< 1MB): use 64KB to minimize memory overhead
        size if size < 1_048_576 => 65_536,
        // Medium files (1MB - 100MB): use 256KB for balanced performance
        size if size < 104_857_600 => 262_144,
        // Large files (>= 100MB): use 1MB buffer for maximum throughput
        _ => 1024 * 1024,
    }
}

Benefits:

• Memory Efficiency: Small files use smaller buffers (64KB), reducing memory overhead
• Balanced Performance: Medium files use 256KB buffers for optimal balance
• Maximum Throughput: Large files (100MB+) use 1MB buffers to minimize syscalls
• Automatic Selection: Buffer size is chosen automatically based on content-length

Performance Impact by File Size:

File Size	Buffer Size	Memory Saved vs Fixed 1MB	Syscalls (approx)
100 KB	64 KB	960 KB (94% reduction)	~2
10 MB	256 KB	768 KB (75% reduction)	~40
100 MB	1 MB	0 KB (same)	~100
10 GB	1 MB	0 KB (same)	~10,240

Future Improvements

1. Connection Keep-Alive: Ensure HTTP keep-alive is properly configured for consecutive uploads

2. Rate Limiting: Consider implementing upload rate limiting to prevent resource exhaustion

3. Configurable Thresholds: Make buffer size thresholds configurable via environment variables or config file

Alternative Approaches Considered

1. Increase s3s timeout: Would only mask the problem, not fix the root cause
2. Retry logic: Would increase complexity and potentially make things worse
3. Connection pooling: Already handled by underlying HTTP stack
4. Upgrade tokio-util: Would provide StreamReader::with_capacity() but requires testing entire dependency tree

References

• Issue: "Uploading files of 10GB or 20GB consecutively may cause the upload to freeze"
• Error: AwsChunkedStreamError: Underlying: error reading a body from connection
• Library: tokio_util::io::StreamReader
• Default buffer: 8KB (tokio_util default)
• New buffer: 1MB (DEFAULT_READ_BUFFER_SIZE)

Conclusion

This fix addresses the root cause of large file upload freezes by using an appropriately sized buffer for stream reading. The 1MB buffer significantly reduces system call overhead, improves throughput, and eliminates timeout issues during consecutive large file uploads.