rustfs/docs/fix-nosuchkey-regression.md

Fix for NoSuchKey Error Response Regression (Issue #901)

Problem Statement

In RustFS version 1.0.69, a regression was introduced where attempting to download a non-existent or deleted object would return a networking error instead of the expected NoSuchKey S3 error:

Expected: Aws::S3::Errors::NoSuchKey
Actual: Seahorse::Client::NetworkingError: "http response body truncated, expected 119 bytes, received 0 bytes"

Root Cause Analysis

The issue was caused by the CompressionLayer middleware being applied to all HTTP responses, including S3 error responses. The sequence of events that led to the bug:

1. Client requests a non-existent object via GetObject
2. RustFS determines the object doesn't exist
3. The s3s library generates a NoSuchKey error response (XML format, ~119 bytes)
4. HTTP headers are written, including Content-Length: 119
5. The CompressionLayer attempts to compress the error response body
6. Due to compression buffering or encoding issues with small payloads, the body becomes empty (0 bytes)
7. The client receives Content-Length: 119 but the actual body is 0 bytes
8. AWS SDK throws a "truncated body" networking error instead of parsing the S3 error

Solution

The fix implements an intelligent compression predicate (ShouldCompress) that excludes certain responses from compression:

Exclusion Criteria

1. Error Responses (4xx and 5xx): Never compress error responses to ensure error details are preserved and transmitted accurately
2. Small Responses (< 256 bytes): Skip compression for very small responses where compression overhead outweighs benefits

Implementation Details

impl Predicate for ShouldCompress {
    fn should_compress<B>(&self, response: &Response<B>) -> bool
    where
        B: http_body::Body,
    {
        let status = response.status();
        
        // Never compress error responses (4xx and 5xx status codes)
        if status.is_client_error() || status.is_server_error() {
            debug!("Skipping compression for error response: status={}", status.as_u16());
            return false;
        }
        
        // Check Content-Length header to avoid compressing very small responses
        if let Some(content_length) = response.headers().get(http::header::CONTENT_LENGTH) {
            if let Ok(length_str) = content_length.to_str() {
                if let Ok(length) = length_str.parse::<u64>() {
                    if length < 256 {
                        debug!("Skipping compression for small response: size={} bytes", length);
                        return false;
                    }
                }
            }
        }
        
        // Compress successful responses with sufficient size
        true
    }
}

Benefits

1. Correctness: Error responses are now transmitted with accurate Content-Length headers
2. Compatibility: AWS SDKs and other S3 clients correctly receive and parse error responses
3. Performance: Small responses avoid unnecessary compression overhead
4. Observability: Debug logging provides visibility into compression decisions

Testing

Comprehensive test coverage was added to prevent future regressions:

Test Cases

1. test_get_deleted_object_returns_nosuchkey: Verifies that getting a deleted object returns NoSuchKey
2. test_head_deleted_object_returns_nosuchkey: Verifies HeadObject also returns NoSuchKey for deleted objects
3. test_get_nonexistent_object_returns_nosuchkey: Tests objects that never existed
4. test_multiple_gets_deleted_object: Ensures stability across multiple consecutive requests

Running Tests

# Run the specific test
cargo test --test get_deleted_object_test -- --ignored

# Or start RustFS server and run tests
./scripts/dev_rustfs.sh
cargo test --test get_deleted_object_test

Impact Assessment

Affected APIs

• GetObject
• HeadObject
• Any S3 API that returns 4xx/5xx error responses

Backward Compatibility

• No breaking changes: The fix only affects error response handling
• Improved compatibility: Better alignment with S3 specification and AWS SDK expectations
• No performance degradation: Small responses were already not compressed by default in most cases

Deployment Considerations

Verification Steps

1. Deploy the fix to a staging environment
2. Run the provided Ruby reproduction script to verify the fix
3. Monitor error logs for any compression-related warnings
4. Verify that large successful responses are still being compressed

Monitoring

Enable debug logging to observe compression decisions:

RUST_LOG=rustfs::server::http=debug

Look for log messages like:

• Skipping compression for error response: status=404
• Skipping compression for small response: size=119 bytes

Related Issues

• Issue #901: Regression in exception when downloading non-existent key in alpha 69
• Commit: 8618570383 (getobjectlength fix)

References

• AWS S3 Error Responses
• tower-http CompressionLayer
• s3s Library