rustfs/docs/cluster_recovery.md

Resolution Report: Issue #1001 - Cluster Recovery from Abrupt Power-Off

1. Issue Description

Problem: The cluster failed to recover gracefully when a node experienced an abrupt power-off (hard failure). Symptoms:

• The application became unable to upload files.
• The Console Web UI became unresponsive across the cluster.
• The rustfsadmin user was unable to log in after a server power-off.
• The performance page displayed 0 storage, 0 objects, and 0 servers online/offline.
• The system "hung" indefinitely, unlike the immediate recovery observed during a graceful process termination (kill).

Root Cause (Multi-Layered):

1. TCP Connection Issue: The standard TCP protocol does not immediately detect a silent peer disappearance (power loss) because no FIN or RST packets are sent.
2. Stale Connection Cache: Cached gRPC connections in GLOBAL_Conn_Map were reused even when the peer was dead, causing blocking on every RPC call.
3. Blocking IAM Notifications: Login operations blocked waiting for ALL peers to acknowledge user/policy changes.
4. No Per-Peer Timeouts: Console aggregation calls like server_info() and storage_info() could hang waiting for dead peers.

---

2. Technical Approach

To resolve this, we implemented a comprehensive multi-layered resilience strategy.

Key Objectives:

1. Fail Fast: Detect dead peers in seconds, not minutes.
2. Evict Stale Connections: Automatically remove dead connections from cache to force reconnection.
3. Non-Blocking Operations: Auth and IAM operations should not wait for dead peers.
4. Graceful Degradation: Console should show partial data from healthy nodes, not hang.

---

3. Implemented Solution

Solution Overview

The fix implements a multi-layered detection strategy covering both Control Plane (RPC) and Data Plane (Streaming):

1. Control Plane (gRPC):

   • Enabled http2_keep_alive_interval (5s) and keep_alive_timeout (3s) in tonic clients.
   • Enforced tcp_keepalive (10s) on underlying transport.
   • Context: Ensures cluster metadata operations (raft, status checks) fail fast if a node dies.

2. Data Plane (File Uploads/Downloads):

   • Client (Rio): Updated reqwest client builder in crates/rio to enable TCP Keepalive (10s) and HTTP/2 Keepalive (5s). This prevents hangs during large file streaming (e.g., 1GB uploads).
   • Server: Enabled SO_KEEPALIVE on all incoming TCP connections in rustfs/src/server/http.rs to forcefully close sockets from dead clients.

3. Cross-Platform Build Stability:

   • Guarded Linux-specific profiling code (jemalloc_pprof) with #[cfg(target_os = "linux")] to fix build failures on macOS/AArch64.

Configuration Changes

pub async fn storage_info<S: StorageAPI>(&self, api: &S) -> rustfs_madmin::StorageInfo {
    let peer_timeout = Duration::from_secs(2);
    
    for client in self.peer_clients.iter() {
        futures.push(async move {
            if let Some(client) = client {
                match timeout(peer_timeout, client.local_storage_info()).await {
                    Ok(Ok(info)) => Some(info),
                    Ok(Err(_)) | Err(_) => {
                        // Return offline status for dead peer
                        Some(rustfs_madmin::StorageInfo {
                            disks: get_offline_disks(&host, &endpoints),
                            ..Default::default()
                        })
                    }
                }
            }
        });
    }
    // Rest continues even if some peers are down
}

Fix 4: Enhanced gRPC Client Configuration

File Modified: crates/protos/src/lib.rs

Configuration:

const CONNECT_TIMEOUT_SECS: u64 = 3;      // Reduced from 5s
const TCP_KEEPALIVE_SECS: u64 = 10;        // OS-level keepalive
const HTTP2_KEEPALIVE_INTERVAL_SECS: u64 = 5; // HTTP/2 PING interval
const HTTP2_KEEPALIVE_TIMEOUT_SECS: u64 = 3;  // PING ACK timeout
const RPC_TIMEOUT_SECS: u64 = 30;          // Reduced from 60s

let connector = Endpoint::from_shared(addr.to_string())?
    .connect_timeout(Duration::from_secs(CONNECT_TIMEOUT_SECS))
    .tcp_keepalive(Some(Duration::from_secs(TCP_KEEPALIVE_SECS)))
    .http2_keep_alive_interval(Duration::from_secs(HTTP2_KEEPALIVE_INTERVAL_SECS))
    .keep_alive_timeout(Duration::from_secs(HTTP2_KEEPALIVE_TIMEOUT_SECS))
    .keep_alive_while_idle(true)
    .timeout(Duration::from_secs(RPC_TIMEOUT_SECS));

---

4. Files Changed Summary

File	Change
crates/common/src/globals.rs	Added evict_connection(), has_cached_connection(), clear_all_connections()
crates/common/Cargo.toml	Added tracing dependency
crates/protos/src/lib.rs	Refactored to use constants, added evict_failed_connection(), improved documentation
crates/protos/Cargo.toml	Added tracing dependency
crates/ecstore/src/rpc/peer_rest_client.rs	Added auto-eviction on RPC failure for server_info() and local_storage_info()
crates/ecstore/src/notification_sys.rs	Added per-peer timeout to storage_info()
crates/iam/src/sys.rs	Made notify_for_user(), notify_for_service_account(), notify_for_group() non-blocking

---

5. Test Results

All 299 tests pass:

test result: ok. 299 passed; 0 failed; 0 ignored

---

6. Expected Behavior After Fix

Scenario	Before	After
Node power-off	Cluster hangs indefinitely	Cluster recovers in ~8 seconds
Login during node failure	Login hangs	Login succeeds immediately
Console during node failure	Shows 0/0/0	Shows partial data from healthy nodes
Upload during node failure	Upload stops	Upload fails fast, can be retried
Stale cached connection	Blocks forever	Auto-evicted, fresh connection attempted

---

7. Verification Steps

1. Start a 3+ node RustFS cluster
2. Test Console Recovery:
   • Access console dashboard
   • Forcefully kill one node (e.g., kill -9)
   • Verify dashboard updates within 10 seconds showing offline status
3. Test Login Recovery:
   • Kill a node while logged out
   • Attempt login with rustfsadmin
   • Verify login succeeds within 5 seconds
4. Test Upload Recovery:
   • Start a large file upload
   • Kill the target node mid-upload
   • Verify upload fails fast (not hangs) and can be retried

---

8. Related Issues

• Issue #1001: Cluster Recovery from Abrupt Power-Off
• PR #1035: fix(net): resolve 1GB upload hang and macos build

9. Contributors

• Initial keepalive fix: Original PR #1035
• Deep-rooted reliability fix: This update