rustfs/docs/kms/test_suite_integration.md

KMS Test Suite Integration

RustFS ships with an extensive set of automated tests that exercise the KMS stack. This guide explains how to run them locally and in CI.

Crate Overview

• crates/kms – unit tests for configuration, caching, and backend adapters.
• crates/e2e_test/src/kms – end-to-end suites for Local and Vault backends, multipart uploads, edge cases, and fault recovery.
• crates/e2e_test/src/kms/common.rs – reusable test environments (spins up RustFS, configures Vault, manages buckets).

Prerequisites

Requirement	Purpose
vault binary (>=1.15)	Required for Vault end-to-end tests. Install from Vault releases.
awscurl (optional)	Debugging helper to hit admin endpoints.
openssl, md5	Used by SSE-C helpers during tests.
Local ports	Tests bind ephemeral ports (ensure 127.0.0.1:<random> is free).

Running Unit Tests

cargo test --workspace --exclude e2e_test

This covers the core KMS crate plus supporting libraries.

Running End-to-End Suites

All KMS Tests

NO_PROXY=127.0.0.1,localhost \
HTTP_PROXY= HTTPS_PROXY= \
cargo test -p e2e_test kms:: -- --nocapture --test-threads=1

• --nocapture streams logs to stdout for troubleshooting.
• --test-threads=1 ensures serial execution; most tests spawn standalone RustFS and Vault processes.

Local Backend Only

cargo test -p e2e_test kms::kms_local_test:: -- --nocapture --test-threads=1

Vault Backend Only

vault server -dev -dev-root-token-id=dev-root-token &
VAULT_PID=$!

cargo test -p e2e_test kms::kms_vault_test:: -- --nocapture --test-threads=1
kill $VAULT_PID

The tests can also start Vault automatically if the binary is found on PATH. When running in CI, whitelist the vault executable in the sandbox or mark the job as privileged.

Updating Fixtures

• Adjustment to SSE behaviour or multipart limits often requires touching crates/e2e_test/src/kms/common.rs. Keep helpers generic so multiple tests can reuse them.
• When fixing bugs, add targeted coverage in the relevant suite (e.g. kms_fault_recovery_test.rs).
• Vault-specific fixtures live in crates/e2e_test/src/kms/common.rs::VaultTestEnvironment.

Debugging Tips

• Use the CLAUDE DEBUG log lines (left intentionally verbose) to inspect the RustFS server flow during tests.
• If a test fails with Operation not permitted, rerun with sandbox overrides (cargo test ... with elevated permissions) as shown above.
• Attach RUST_LOG=rustfs::kms=debug to surface detailed backend interactions.

CI Recommendations

• Split KMS tests into a dedicated job so slower suites (Vault) do not gate unrelated changes.
• Cache the Vault binary and reuse it across runs to minimise setup time.
• Surface logs and target/debug/e2e_test-* binaries as artifacts when failures occur.

For API usage examples and configuration reference, consult http-api.md and dynamic-configuration-guide.md.