Failure Modes

Read

A gateway outage returns a connection error to the client. SDKs do not retry directly against the backing store. Check gateway health and connectivity; store failures remain visible through the gateway response.

The document cache is stateless and can scale to zero with no disruption: document fetches fall through to origin (turbopuffer, or S3 for snapshots) on a miss or cache outage, so a cache failure degrades latency, not availability.

Write

Successful writes require success from the backing store. If the gateway is unreachable, the client returns the connection error. Cache warming is best-effort and does not replace the durable store write.

Pipeline stop-writes

The primary failure mode for writes through a healthy gateway is Aerospike stop-writes during a multi-stage pipeline job: staged documents stay warm in the cache but carry no vector data yet, and once that data exceeds the Aerospike drive allocation the cache rejects further writes.

The pipeline does not stall. Each stage persists its chunk bodies to S3 before it touches the cache, and pipeline state lives in PostgreSQL, so the Aerospike write is best-effort: on stop-writes the gateway logs the skipped write and the stage still completes. Downstream chunk reads degrade to the S3 backing for as long as the cache is rejecting writes.

Recovery is automatic. The Helm document cache restarts on stop-writes by default (documentCache.autoRestartOnStopWrites: true) and clears its Aerospike backing file on pod start (documentCache.storage.resetOnStart: true); the gateway reconnects in the background and refills the cache from S3 on demand. No pipeline work is lost — S3 and PostgreSQL are the durable recovery boundary and must stay healthy.

Operator signals:

• layer_aerospike_op_duration_seconds{status="aerospike_stop_writes"} — the stop-writes condition itself, the same series the dashboard charts.
• hevlayer_cache_cold_responses_total — reads being served from S3 backing instead of the cache while it recovers.
• hevlayer_document_cache_cold_starts_total and hevlayer_document_cache_cold_start_seconds — the demand-triggered reconnect-and-refill cycle after the cache restarts.
• Gateway warn logs Aerospike chunk write failed (best-effort) and Aerospike chunk read failed; falling back to S3 backing.

Client failures

When the gateway is unreachable, the SDKs return the gateway connection error. They do not retry directly against a backing store. That keeps the client surface backend-neutral: a namespace may be backed by turbopuffer or another store selected by VectorStore, and callers should not need a store-specific credential or fallback path.