# Append Write Design

This document captures the current design of the append-write feature in RustFS so that new contributors can quickly understand the moving parts, data flows, and testing expectations.

## Goals & Non-Goals

### Goals
- Allow clients to append payloads to existing objects without re-uploading the full body.
- Support inline objects and spill seamlessly into segmented layout once thresholds are exceeded.
- Preserve strong read-after-write semantics via optimistic concurrency controls (ETag / epoch).
- Expose minimal S3-compatible surface area (`x-amz-object-append`, `x-amz-append-position`, `x-amz-append-action`).

### Non-Goals
- Full multipart-upload parity; append is intentionally simpler and serialized per object.
- Cross-object transactions; each object is isolated.
- Rebalancing or background compaction (future work).

## State Machine

Append state is persisted inside `FileInfo.metadata` under `x-rustfs-internal-append-state` and serialized as `AppendState` (`crates/filemeta/src/append.rs`).

```
Disabled --(initial PUT w/o append)--> SegmentedSealed
Inline --(inline append)--> Inline / InlinePendingSpill
InlinePendingSpill --(spill success)--> SegmentedActive
SegmentedActive --(Complete)--> SegmentedSealed
SegmentedActive --(Abort)--> SegmentedSealed
SegmentedSealed --(new append)--> SegmentedActive
```

Definitions:
- **Inline**: Object data fully stored in metadata (`FileInfo.data`).
- **InlinePendingSpill**: Inline data after append exceeded inline threshold; awaiting spill to disk.
- **SegmentedActive**: Object data lives in erasure-coded part(s) plus one or more pending append segments on disk (`append/<epoch>/<uuid>`).
- **SegmentedSealed**: No pending segments; logical content equals committed parts.

`AppendState` fields:
- `state`: current state enum (see above).
- `epoch`: monotonically increasing counter for concurrency control.
- `committed_length`: logical size already durable in the base parts/inline region.
- `pending_segments`: ordered list of `AppendSegment { offset, length, data_dir, etag, epoch }`.

## Metadata & Storage Layout

### Inline Objects
- Inline payload stored in `FileInfo.data`.
- Hash metadata maintained through `append_inline_data` (re-encoding with bitrot writer when checksums exist).
- When spilling is required, inline data is decoded, appended, and re-encoded into erasure shards written to per-disk `append/<epoch>/<segment_id>/part.1` temporary path before rename to primary data directory.

### Segmented Objects
- Base object content is represented by standard erasure-coded parts (`FileInfo.parts`, `FileInfo.data_dir`).
- Pending append segments live under `<object>/append/<epoch>/<segment_uuid>/part.1` (per disk).
- Each append stores segment metadata (`etag`, `offset`, `length`) inside `AppendState.pending_segments` and updates `FileInfo.size` to include pending bytes.
- Aggregate ETag is recomputed using multipart MD5 helper (`get_complete_multipart_md5`).

### Metadata Writes
- `SetDisks::write_unique_file_info` persists `FileInfo` updates to the quorum of disks.
- During spill/append/complete/abort, all mirrored `FileInfo` copies within `parts_metadata` are updated to keep nodes consistent.
- Abort ensures inline markers are cleared (`x-rustfs-internal-inline-data`) and `FileInfo.data = None` to avoid stale inline reads.

## Request Flows

### Append (Inline Path)
1. Handler (`rustfs/src/storage/ecfs.rs`) validates headers and fills `ObjectOptions.append_*`.
2. `SetDisks::append_inline_object` verifies append position using `AppendState` snapshot.
3. Existing inline payload decoded (if checksums present) and appended in-memory (`append_inline_data`).
4. Storage class decision determines whether to remain inline or spill.
5. Inline success updates `FileInfo.data`, metadata, `AppendState` (state `Inline`, lengths updated).
6. Spill path delegates to `spill_inline_into_segmented` (see segmented path below).

### Append (Segmented Path)
1. `SetDisks::append_segmented_object` validates state (must be `SegmentedActive` or `SegmentedSealed`).
2. Snapshot expected offset = committed length + sum of pending segments.
3. Payload encoded using erasure coding; shards written to temp volume; renamed into `append/<epoch>/<segment_uuid>` under object data directory.
4. New `AppendSegment` pushed, `AppendState.epoch` incremented, aggregated ETag recalculated.
5. `FileInfo.size` reflects committed + pending bytes; metadata persisted across quorum.

### GET / Range Reads
1. `SetDisks::get_object_with_fileinfo` inspects `AppendState`.
2. Reads committed data from inline or erasure parts (ignoring inline buffers once segmented).
3. If requested range includes pending segments, loader fetches each segment via `load_pending_segment`, decodes shards, and streams appended bytes.

### Complete Append (`x-amz-append-action: complete`)
1. `complete_append_object` fetches current `FileInfo`, ensures pending segments exist.
2. Entire logical object (committed + pending) streamed through `VecAsyncWriter` (TODO: potential optimization) to produce contiguous payload.
3. Inline spill routine (`spill_inline_into_segmented`) consolidates data into primary part, sets state `SegmentedSealed`, clears pending list, updates `committed_length`.
4. Pending segment directories removed and quorum metadata persisted.

### Abort Append (`x-amz-append-action: abort`)
1. `abort_append_object` removes pending segment directories.
2. Ensures `committed_length` matches actual durable data (inline length or sum of parts); logs and corrects if mismatch is found.
3. Clears pending list, sets state `SegmentedSealed`, bumps epoch, removes inline markers/data.
4. Persists metadata and returns base ETag (multipart MD5 of committed parts).

## Error Handling & Recovery

- All disk writes go through quorum helpers (`reduce_write_quorum_errs`, `reduce_read_quorum_errs`) and propagate `StorageError` variants for HTTP mapping.
- Append operations are single-threaded per object via locking in higher layers (`fast_lock_manager` in `SetDisks::put_object`).
- On spill/append rename failure, temp directories are cleaned up; operation aborts without mutating metadata.
- Abort path now realigns `committed_length` if metadata drifted (observed during development) and strips inline remnants to prevent stale reads.
- Pending segments are only removed once metadata update succeeds; no partial deletion is performed ahead of state persistence.

## Concurrency

- Append requests rely on exact `x-amz-append-position` to ensure the client has an up-to-date view.
- Optional header `If-Match` is honored in S3 handler before actual append (shared with regular PUT path).
- `AppendState.epoch` increments after each append/complete/abort; future work may expose it for stronger optimistic control.
- e2e test `append_segments_concurrency_then_complete` verifies that simultaneous appends result in exactly one success; the loser receives 400.

## Key Modules

- `crates/ecstore/src/set_disk.rs`: core implementation (inline append, spill, segmented append, complete, abort, GET integration).
- `crates/ecstore/src/erasure_coding/{encode,decode}.rs`: encode/decode helpers used by append pipeline.
- `crates/filemeta/src/append.rs`: metadata schema + helper functions.
- `rustfs/src/storage/ecfs.rs`: HTTP/S3 layer that parses headers and routes to append operations.

## Testing Strategy

### Unit Tests
- `crates/filemeta/src/append.rs` covers serialization and state transitions.
- `crates/ecstore/src/set_disk.rs` contains lower-level utilities and regression tests for metadata helpers.
- Additional unit coverage is recommended for spill/append failure paths (e.g., injected rename failures).

### End-to-End Tests (`cargo test --package e2e_test append`)
- Inline append success, wrong position, precondition failures.
- Segmented append success, wrong position, wrong ETag.
- Spill threshold transition (`append_threshold_crossing_inline_to_segmented`).
- Pending segment streaming (`append_range_requests_across_segments`).
- Complete append consolidates pending segments.
- Abort append discards pending data and allows new append.
- Concurrency: two clients racing to append, followed by additional append + complete.

### Tooling Considerations
- `make clippy` must pass; the append code relies on async operations and custom logging.
- `make test` / `cargo nextest run` recommended before submitting PRs.
- Use `RUST_LOG=rustfs_ecstore=debug` when debugging append flows; targeted `info!`/`warn!` logs are emitted during spill/abort.

## Future Work

- Streamed consolidation in `complete_append_object` to avoid buffering entire logical object.
- Throttling or automatic `Complete` when pending segments exceed size/quantity thresholds.
- Stronger epoch exposure to clients (header-based conflict detection).
- Automated cleanup or garbage collection for orphaned `append/*` directories.

---

For questions or design discussions, drop a note in the append-write channel or ping the storage team.