CortexDB 0.11.0 — every write through a replicated log, cluster mode, and a one-way data upgrade

0.11.0 is a feature release on top of 0.10.5, the last published release. Everything in RELEASE_NOTES_0.10.5.md (and the notes before it) still applies; this page lists what changed since.

It is a one-way upgrade. Its first start migrates the data dir, a 0.10.x binary cannot open the dir afterwards, and the way back is a restore. Read "Before you upgrade" first. The one-click updater does not do this upgrade: do it by hand ("Upgrade").

For clients nothing is removed: the routes, the request and response shapes and the 11 capabilities are 0.10.5's. What a client can see is listed under "What changes for a single node".

Summary

• Every write goes through a replicated log. The event log is now the state of one Raft group (openraft) per shard, written only by that group's apply. A plain cortexdb runs one such group with one member (RF=1): a write is acknowledged once its log entry is fsynced, committed and applied, and the indexer builds every view from the log. Plain cortexdb is still one process, one data dir and zero config.
• Cluster mode (new, opt-in). cortexdb node --config FILE starts a metadata, data or ingress node. A cluster holds shards of 1 to 5 replicas; a shard stays available while a majority of its replicas is up, only its leader serves (reads are linearizable), a lagging or replacement replica catches up by a streamed snapshot, a scope root moves between shards on the operator's command under load, and a shard that lost its quorum is restored from one replica's backup. Nodes talk over mutual TLS with pinned keys. An operator places everything with cortexdb admin; nothing places, splits, merges or rebalances on its own. On a cluster a shard's leader embeds each write once and replicates the vectors through the log, so embedding spend is one embedding per chunk per shard at any replica count. Limits under "Cluster mode".
• One-way data upgrade. The first start keeps the event log as it was in pre_stage2/ and runs four one-time migrations ("Before you upgrade"). Two ways back to 0.10.x: the cold backup you take before the upgrade, or pre_stage2/ ("Downgrade to 0.10.1 through 0.10.5"); both were drilled with 0.10.5. cortexdb/cortexdb:latest now names 0.11.0: pin v0.10.5 until you upgrade on purpose.
• The legacy dual WAL is removed, as 0.10.x announced: CORTEX_UNIFIED_WAL=0, or a data dir that still holds v1_wal/, exits 78 LEGACY_WAL_RETIRED before any store opens.
• Writes on a single node. Three new retriable 503s (NOT_LEADER, WAL_UNAVAILABLE, WAL_OUTCOME_UNKNOWN); a replay returns the original wal_offset; a failed write keeps its key; a cancelled write answers 409 CANCELLED; the status probe fails closed; a capture record above 4 MiB less 64 KiB is refused; CORTEX_WAL_SYNC=0 no longer weakens an acknowledged write.
• Forgets and erasures go through the log (Erase entries of up to 4,096 events, applied by the server's erase tail before the request answers), and the raw-residual pass now also clears the event log, the Raft log and its snapshots.
• Fixes for defects 0.10.5 still has. A write acknowledged indexed could be lost from the full-text index by a kill within about 100 ms; a server killed inside a forget's search-index commit could never start again; the index audit counted every memory twice; a layer build could stall the server behind a queued scope-registry change; a scope read racing an erasure could fail SCOPE_INDEX_CORRUPT.
• Performance and memory. Measured against 0.10.5 on a quiet machine with jemalloc: every write and read latency within +5 % of 0.10.5's at p50 and p99, or within 0.10.5's own spread between rounds; default-wait write p99 24-28 % lower (9 % with a 300 ms embedder), a forget by id 24-30 % faster, the same bulk throughput, a restart at 100,000 events ready 10-17 % sooner, peak memory 1-9 % lower and disk per event within +1 %. One fixed cost: about 13 MiB more memory at ready on an empty data dir (99-100 against 86-87 MiB, +15 %).
• Clients. No client is re-released with 0.11.0: every client published with 0.10.5 (Python SDK cortexdbai 0.12.0, TypeScript SDK cortexdbai 0.10.1, cortexdb-mcp 0.8.0, cortexdb-cli 0.6.0, cortexdb-connectors 0.2.22) works against 0.11.0 unchanged.

Before you upgrade

• The first start migrates the data dir, one way. Before it opens the event log, 0.11.0 copies the event log (unified_wal/) and the old idempotency store (v1_idempotency/) into <data_dir>/pre_stage2/. Then it runs four migrations, each once:

  • the server's own derived rows move out of the event log into <data_dir>/derived_wal/, the event log's erasure index is rebuilt for the captured rows it keeps, and the event log is then compacted whole once, so this start takes longer on a large dir (plan for up to two or three times the size of unified_wal/ on disk while pre_stage2/ is kept);
  • the unexpired idempotency records move into the event log;
  • vocabularies.json and temporal_phrases.json move into the event log;
  • the retention policies move from the governance store into the event log.

  The event log gains column families that a 0.10.x binary cannot open: 0.10.5 started on the migrated dir exits at once (Column families not opened: ...). After the first start the way back is a restore ("Downgrade to 0.10.1 through 0.10.5"). Take a cold backup first.

• cortexdb/cortexdb:latest is 0.11.0 now. A deployment that pulls latest (the Compose file's default) and is restarted, or recreated by docker compose up, upgrades one way, with no backup unless you took one. Pin cortexdb/cortexdb:v0.10.5 until you upgrade on purpose.

• The one-click updater does not do this upgrade. The Updates page of a 0.10.3 to 0.10.5 server offers 0.11.0 like any newer release, and its updater sidecar (built from a 0.10.x checkout) knows no one-way boundary: it would upgrade without the one-way consent, and if 0.11.0 did not report healthy in time (the first start migrates before it serves), it would start 0.10.x again on the migrated dir, which 0.10.x cannot open; only the sidecar's backup would then bring it back. Do not use "Update now" for 0.11.0. Upgrade by hand ("Upgrade"), then run the updater overlay from a 0.11.0 checkout for later updates. A sidecar from 0.11.0 or later refuses this upgrade (ONE_WAY_CONSENT_REQUIRED) before CortexDB stops and changes nothing, because a 0.10.x console cannot send the consent a one-way upgrade needs.

• The legacy dual WAL is gone. CORTEX_UNIFIED_WAL=0 (or false, off), or a data dir that still holds v1_wal/, logs code=LEGACY_WAL_RETIRED and exits 78 before any store opens. Stop the server and run cortexdb migrate-unified-wal --data-dir DIR --confirm first.

• CORTEX_WAL_SYNC=0 no longer weakens acknowledged writes. Every write is fsynced in the Raft log before it is acknowledged, whatever the variable says, and =0 logs one warning at start. It still skips the fsync of the node-local derived_wal/ appends, which can be rebuilt.

• A storage or fsync error stops the process. The shard stops, writes answer 503 WAL_UNAVAILABLE, and the process logs code=WAL_FATAL and exits 74. Run it under a supervisor that restarts it (Docker restart: unless-stopped, as docker-compose.yml sets; systemd Restart=always).

• Every write can answer three new retriable 503s (retriable: true): experience, bulk, cancel, forget, erasure, the retention sweep, and the vocabulary, temporal-phrase and retention-policy changes, among others:

  • NOT_LEADER: never on a single node once it serves.
  • WAL_UNAVAILABLE: the shard stopped on a storage error, or is shutting down. Nothing was written.
  • WAL_OUTCOME_UNKNOWN: the write was submitted and may have committed, for example when it did not commit within 10 s. Retry with the same idempotency key, which replays the original receipt if it did.

  A client that retries on retriable: true needs no change. The other changes a client can see are under "What changes for a single node", "Writes".

• Restoring a cold backup needs a server binary. cold_backup.py restore now runs cortexdb restore-check on any archive that holds an event log (unified_wal/), 0.10.x's included, and refuses the archive when no binary can check it. Pass --cortexdb PATH, a 0.11.0 binary: its --help must name restore-check. With Docker, docker cp cortexdb:/usr/local/bin/cortexdb . copies the image's. A single node's archive always passes the check.

• CORTEX_BULK_DEFER_VECTORS is retired. Raw-first bulk ingest (index lexically first, add the vectors later from a background queue) does not exist any more: set truthy, the variable is ignored with one warning at start. Remove it from your env file. A dir whose raw-first queue still holds rows at the upgrade drains them in the background from the first start, through the index repair ("Vectors", below). CORTEX_DRAIN_EMBED_CONCURRENCY and CORTEX_V1_MIRROR_MAX_IN_FLIGHT are retired with the paths they tuned and are ignored.

• A single node keeps its vectors where 0.10.5 keeps them. The vector index (vector/) holds each chunk's vector, rows written before the upgrade keep theirs, and the first start embeds nothing ("Vectors", below). Only cluster data nodes keep vectors in the event log.

• Back up the new directories. shard/ (the Raft log and snapshots) and derived_wal/ are part of the data. cold_backup.py and the updater's backup archive the whole data dir, so their archives already hold them.

• Run the bare binary on jemalloc, as the container image does ("Operations", "Memory").

Upgrade

From 0.10.5

1. Read "Before you upgrade". Make sure nothing restarts the server on a newer image while you work (pin cortexdb/cortexdb:v0.10.5).
2. Stop the server and give it the full stop grace (120 s): docker stop -t 120 cortexdb, or systemctl stop with TimeoutStopSec=120.
3. Take a cold backup of the stopped data dir: python cold_backup.py backup DATA_DIR ARCHIVE, or a plain tar of the directory. It is the clean way back.
4. If the dir still holds v1_wal/, or you run with CORTEX_UNIFIED_WAL=0, migrate first: cortexdb migrate-unified-wal --data-dir DATA_DIR --confirm, and drop the variable.
5. Start 0.11.0 on the same dir, with the same environment (image cortexdb/cortexdb:v0.11.0).
   • DATA_DIR/pre_stage2/ appears: the rollback point. The log names it: Stage 2 rollback point taken, a warning under target cortex_wal, which the default RUST_LOG of the binary and of the image include (a RUST_LOG of your own shows it only with cortex_wal=warn in it).
   • The log reports each import: imported v1_idempotency claims into the shard, imported tenant maps into the shard and imported retention policies into the shard.
   • On a large dir this start takes longer; GET /v1/admin/ready answers 200 once it serves.
6. Check:
   • GET /v1/admin/version: crate_version 0.11.0 and the git_sha you deployed; the same 11 capabilities as 0.10.5.
   • GET /v1/admin/ready: ready: true.
   • Before any write or recall, GET /v1/admin/metrics: vectors.embed_inputs is 0 (the first start embedded nothing; every write and every recall query adds to it).
   • A memory you know is found by GET /v1/events/{id} and by /v1/recall, and a retry of a recent write with its idempotency key replays it.
7. Once you trust the release, delete DATA_DIR/pre_stage2/. It holds the event log as it was at the upgrade, erased events included, and every erasure reports log_residual_pending with the reason pre_stage2_retained until it is gone.

Docker Compose: set the image to cortexdb/cortexdb:v0.11.0 (or pull latest) only at step 5, and keep the pin afterwards.

From 0.10.4 or earlier

0.11.0 contains everything in 0.10.5. Read RELEASE_NOTES_0.10.5.md, "Upgrade from 0.10.4" (and from there the earlier notes) for what those releases change for clients and operators; then take the steps above, straight to 0.11.0. The way back is to the release you came from. The upgrade and both ways back were drilled from 0.10.1, 0.10.2 (pre_stage2/ only) and 0.10.5 data dirs.

The one-click updater after the upgrade

Once 0.11.0 serves, run the updater overlay from a 0.11.0 checkout, so the sidecar is rebuilt (docker compose -f docker-compose.yml -f docker-compose.updater.yml up -d --build). For a later one-way release, which its image declares (the io.cortexdb.one-way-below label; 0.11.0's image declares 0.11.0), the sidecar refuses the job with ONE_WAY_CONSENT_REQUIRED before CortexDB stops, the Updates page then asks for the operator's consent (accept_one_way), the backup is required, the first start's migration is waited for, and the old container is never started again on the migrated data.

Downgrade to 0.10.1 through 0.10.5

A 0.10.x binary cannot open a data dir 0.11.0 has served. There are two ways back, to the 0.10.x release you upgraded from. Both lose every write made since the upgrade and bring back every event forgotten or erased since it, so issue those forgets and erasures again on 0.10.x:

• The cold backup (preferred). Restore the archive from "Upgrade" step 3 into a fresh dir and start 0.10.x on it. A plain tar archive is extracted as it is. An archive from cold_backup.py is restored with python cold_backup.py restore ARCHIVE NEW_DIR --cortexdb PATH_TO_0.11.0_BINARY: the check needs a binary with restore-check, even when you restore for 0.10.x, and the restore is refused without one. You get the data dir exactly as it was before the upgrade.

• pre_stage2/. Stop 0.11.0, then:

  1. Replace unified_wal/ and v1_idempotency/ with the copies in pre_stage2/.
  2. Delete derived_wal/, shard/ and pre_stage2/. vector/ stays: a single node keeps its vectors there in 0.10.x's format.
  3. Start 0.10.x. pre_stage2/ holds its event log as it was, erasure index, stamp and idempotency reverse rows included, so 0.10.x reads its index at once and catches up nothing.

  Only the event log and the idempotency store go back to the upgrade point; the other stores keep what 0.11.0 wrote to them. What a drill with 0.10.5 showed (a 0.10.5 dir with single, bulk and multi-chunk writes and an erasure, upgraded, written to, forgotten from and erased from under 0.11.0, then taken back this way):

  • Events written since the upgrade are gone: not found by id, and their idempotency keys are free again (a resend stores a new event). Keys written before the upgrade replay their events.
  • Every pre-upgrade event is found by id and by recall, the multi-chunk one included, and the index audit reports nothing missing.
  • An event forgotten since the upgrade is back in the event log, payload included. 0.10.x does not list or recall it (the view stores keep their own markers), but GET /v1/events/{id} returns it with its payload, and 0.10.x does not redact it by itself. The same forget, issued again on 0.10.x, removes it.
  • Erasure jobs and receipts written before and after the upgrade are still served, and their proofs (/verify) were clean. An event written before the upgrade and erased under 0.11.0 is back in the event log after these steps, as a forgotten one is: erase it again.
  • Vocabularies, temporal phrases and retention policies are their pre-upgrade versions (drilled with 0.10.1).

  0.11.0 refuses to start on a dir where derived_wal/ or shard/ survived next to a restored event log, and its error names the directories to delete. Resuming into derived_wal/ would bring back rows 0.10.x erased since, and the Raft log in shard/ would replay the writes 0.11.0 made before the downgrade, or install its old snapshot over the event log and lose what 0.10.x wrote since.

After a downgrade, starting 0.11.0 again upgrades the dir again from the start, with a new pre_stage2/ (drilled with 0.10.5).

0.10.2 to 0.10.5 store what 0.10.1 stores, plus 0.10.4's erasure receipts (erasure_job/ keys in v1_layers/) and cohort_alias.key. 0.11.0 reads both as they are; its receipts add one optional field, which 0.10.5 ignores. Both ways back were drilled with 0.10.1 and 0.10.5, and the pre_stage2/ way with 0.10.2 too.

What changes for a single node

Against 0.10.5. "0.10.x" below means every 0.10 release, 0.10.5 included.

Writes

• The acknowledgement point. A write is acknowledged once its log entry is fsynced, committed and applied. The row and its idempotency record are written in one batch.
  • ?wait=indexed waits for the indexer, which reads the log. There is no inline indexing path any more.
  • Bulk writes go to the log in chunks of about 1 MiB, one fsync per chunk. A bulk 503 WAL_OUTCOME_UNKNOWN, or a 207, can follow chunks that committed: resend the batch with the same keys, and the committed items replay.
  • A same-key replay with ?wait=indexed that the indexer has not reached answers 408 WAIT_TIMEOUT once the write's wait budget runs out (0.10.x replays did not wait on the indexer).
  • Measured against 0.10.5 on a quiet machine (jemalloc as in the image; an embedding stub at 10 ms and 300 ms, and the mock): every write and read latency within +5 % of 0.10.5's at p50 and p99, or within 0.10.5's spread between rounds. Closest: writes paced at 20 a second after a restart at 100,000 events, p50 +4.0 % and +4.2 %, p99 3.1 % lower (stub) and +5.3 % (mock, inside 0.10.5's 5.9 % spread). The default-wait p99 is 24-28 % lower (9 % with the 300 ms stub), a forget by id 24-30 % faster, bulk throughput equal (within 1 %), a restart at 100,000 events ready 10-17 % sooner, and disk per event within +1 %.
• A write acknowledged indexed survives a crash in the full-text index. The indexer records its progress only past full-text documents that a commit has written (the group commit, CORTEX_BM25_COMMIT_INTERVAL_MS), and after a crash it re-checks the writes since then. 0.10.x recorded it first, so a kill -9 within about 100 ms of indexing could lose writes acknowledged indexed from the full-text index for good. A data dir 0.10.x ran through a crash may hold such writes: the startup audit logs INDEX LOSS DETECTED, and POST /v1/admin/index-audit with {"repair": true} re-indexes them. The startup audit also logs that line after any crash under load, before the indexer has re-checked the last writes; audit again once the server serves. The deferred-enrichment cursor (CORTEX_ENRICHMENT_DELAY_SECONDS) likewise waits for the full-text commit of the facts it passes; the audit does not check those.
• The index audit counts each memory once. POST /v1/admin/index-audit and the startup INDEX LOSS DETECTED line report scanned, missing, missing_total, missing_vectors* and repaired per memory. 0.10.x counted each applied memory twice (its Captured row and its Derived twin), so its counts were up to double.
• Replays return the original wal_offset. 0.10.x answered a replay with 0. New offsets are dense.
• A failed write keeps its key. INDEXING_FAILED (502) is retriable: false: the same key replays the failure, so resend the data under a new key. 0.10.x dropped the key and re-processed a same-key resend. GET /v1/experience/status says failed with that note.
• Cancelled writes answer 409 CANCELLED (retriable: false, details.event_id). This is the answer to a replay, or a ?wait=indexed or consolidated, of a write whose cancel committed, before and after a restart, and it holds per item in a bulk. The status probe says cancelled, and the key stays claimed. 0.10.x answered the same replay 200 with status: "indexed", although the write is never recalled, and its probe said failed, with a note to resend under the same key.
• The status probe fails closed. GET /v1/experience/status answers 503 WAL_UNAVAILABLE (retriable) when it cannot read the write's records. 0.10.x read an unreadable record as absent and answered found: false, which tells a client a resend is safe.
• Size limits. The log takes commands up to 4 MiB.
  • A capture whose encoded record is above 4 MiB less 64 KiB is 422 INVALID_ENVELOPE. In a bulk, that item fails alone and the others are written. 0.10.x took such a record up to the 32 MiB body limit, as long as its indexable text stayed within 1 MiB.
  • A vocabulary, temporal phrase or retention policy above 4 MiB is 413 WAL_TOO_LARGE (retriable: false). 0.10.x stored it.
• Vocabularies, temporal phrases and retention policies live in the event log. A read that fails on storage answers 500 TENANT_MAP_STORAGE. A retention-policy change whose outcome is unknown still writes its audit entry, marked "outcome unknown", and answers the 503.
• Erasures on long-lived stores stay fast. 0.10.5 made them fast by bounding its reads of the idempotency store. Here the event log finds an event's idempotency claim with a point read, and the status probe's scan of the claims and each write's expiry of the claims older than 24 h are bounded the same way.

Vectors

• As in 0.10.5. A single node embeds and indexes each write itself, and its vector index (vector/) keeps each chunk's vector, in 0.10.x's format. ?wait=indexed waits for that indexing alone, and recall reads the vectors from vector/. Cluster data nodes do it differently: their shard's leader embeds each write once and replicates the vectors ("Cluster mode", below). Every single-node data dir holds an empty file, local_chunk_rows, which records that vector/ needs no conversion; a 0.10.x dir gets it at its first start. (A data dir that a pre-release 0.11.0 build had given vectors in the event log is converted at its first start under this release, with no provider call.)
• Chunk ids are derived from the event id and the chunk's position (they were random for chunks after the first).
• Blob captures (blob_ref) still chunk and embed on the node that stores the blob.
• Rejected inputs. With a remote embedding provider (not mock or another local one), an input the provider rejects alone costs one more provider call (a probe input, embedding probe N) and then fails only the embed request that holds it; 0.10.x failed every request of the same coalesced flush. If the probe fails too, the flush fails as before.
• GET /v1/admin/metrics gains vectors (on a single node only embed_inputs, every input this process handed its embedder, moves) and indexer.vectors_waiting (never set on a single node); it loses deferred_vectors, with raw-first bulk.

Forget, erasure and retention

• Every forget and erasure goes through the log. It commits Erase entries of up to 4,096 events each. This server's erase tail then removes those events from the views: content, search index, vectors and derived rows. The request answers only after the tail has applied its entries.
  • Each entry adds one governance audit row (purge, reason shard-erase) and one search-index commit. 0.10.x committed the index once per request, so a large scope erasure commits it more often.
  • /v1/erasures deletes its event rows in these entries (0.10.x wrote one sync per 256).
  • Forget and erasure timings were not re-measured against 0.10.5.
• Counts. purge on /v1/forget and summary.purge on /v1/erasures are the sums of what the erase tail removed for the request's entries. The event counts (deleted.events, deleted_events, redacted_events) are the entries' own.
• The retention sweep report has no per-scope purge object any more. expired, erased and derived stay.
• A redact_events forget by query deletes its events. A forget with about_subject, about_entity or predicate, the events layer and cascade: "redact_events" now deletes the matched events from the log, freeing their keys and scrubbing references to them. 0.10.x purged their documents from the views but kept their rows in the event log.
• Named ids that are no event. Under redact_events, a memory_ids entry that names no event of the scope in the log (a fact_ or ent_ id, or an id that does not exist) purges nothing and counts nothing. 0.10.x passed every such id to the storage engine's purge, and answered 502 when the scope was under a legal hold. Name facts and entities under derived_only.
• Failures after the commit. A forget or erasure whose log entry committed but whose view purge failed answers the erase tail's error (502), and so does a retry, until the tail, which retries the entry on its own, has purged the views. This covers the orphan-document sweep and the fact cascade, which in 0.10.x ran and failed before anything was erased. A forget or erasure the log refuses answers its retriable 503 and records it on its audit row (details.audit_id).
• The forget cutoff. 0.10.x's rule holds: an event recorded at or before a forget's cutoff is forgotten, one recorded after it is kept. Two things differ:
  • The cutoff is the later of the server's clock and the latest time recorded on an event the log already holds. On a single node that differs from the clock only after the clock stepped back. While it differs, the log records new captures at that latest time too, so a capture made during the forget request, before its tombstone, falls inside the cutoff and is forgotten.
  • A capture admitted before the cutoff that reaches the log only after the forget's tombstone is recorded after the cutoff, so it is kept. 0.10.x erased it in its end-of-request sweep.
• Erasure receipts and proof (0.10.4's). /verify re-reads the event log's rows and its idempotency claims (there is no v1_idempotency/ store), and all_ok waits for the log residual (below). A receipt records the job's place in the log (residual.feed), so a job rebuilt from its receipt after a restart still reports the log's residual. The redacted event-log rows lose their labels, intent and location, as on 0.10.5.
• Raw residual. The background pass still compacts erased keys out of content/, derived_wal/ and session_fragments/, and merges the Tantivy segments that still hold erased documents. It now also compacts them out of the event log (unified_wal/). After each erasure the server then builds a shard snapshot, purges the Raft log through the erasure and compacts it, and drops its older snapshots. GET /v1/erasures/{id} reports log_residual_cleared_at once all of that is done, as 0.10.x did. New log_residual_pending reason: pre_stage2_retained: pre_stage2/ still holds an erased row. The job clears once you delete pre_stage2/, so delete it once the upgrade is final.
• Retired lint. The startup lint ERASURE_RESIDUAL_LEGACY_WAL is gone: the configuration it warned about now refuses to start.
• A race is fixed. A forget, move or retention scan that read a scope while an erasure deleted one of its rows could fail SCOPE_INDEX_CORRUPT. It no longer does.
• Blob-link repair goes through the log. POST /v1/admin/index-audit/source-blob-links applies its links through the log. A link is added only to an event that is still live, unredacted and under the scope, so an event erased in between stays erased.

Operations

• Data dir layout. New:

  • shard/raft_log/ and shard/snapshots/: the log and its snapshots. A snapshot is taken every 10,000 entries (on a single node also every 32 MiB of log) and after each erasure, and the log is purged behind it.
  • derived_wal/.
  • pre_stage2/: delete it once you will not go back.
  • local_chunk_rows (an empty file, above).

  v1_idempotency/, vocabularies.json and temporal_phrases.json are read once and then left unused. If shard/raft_log/ is lost, the server restarts from its applied state.

• Memory. Against 0.10.5 on a quiet machine with jemalloc (the same runs as the latencies above): the same writes peaked 1.4-2.7 % lower, 100,000 events peaked 1.1-8.7 % lower and a restart there settled 3.7-4.0 % lower, and a start on an empty dir holds about 13 MiB more at ready (99-100 against 86-87 MiB, +15 %): the code and state of the replicated log. Run the bare binary on jemalloc, as the container image does: LD_PRELOAD naming libjemalloc.so.2 (package libjemalloc2) and the image's MALLOC_CONF (background_thread:true,dirty_decay_ms:5000,muzzy_decay_ms:5000); in a systemd unit, two Environment= lines. Under glibc's malloc a long write-and-forget load grows the memory from fragmentation until a restart (2,233 MiB after 299,700 captures written and forgotten, 886 MiB with jemalloc), as on 0.10.x.

• GET /v1/admin/wal describes the event log, which after the upgrade holds only captured events, so live_count drops at the first start. Its consumer floors come from derived_wal/.

• POST /v1/admin/wal/compact and the daily dead-row pass (CORTEX_WAL_COMPACT_INTERVAL_SECS) compact derived_wal/, bounded by its head, not by the indexer watermark.

• GET /v1/admin/metrics. indexer.inflight, oldest_inflight_secs, barriers and reclaimed_inflight_total are always zero or null, because the inline indexing path is gone. indexer.held is never set on a single node.

• GET /v1/admin/ready. checks.commitments.reason no longer reports unified_wal_off.

• The default log filter adds cortex_wal=warn, the binary's (when RUST_LOG is unset) and the image's RUST_LOG: the event log's warnings and errors reach the log, among them the first start's rollback point and a row quarantined because it fails to decode. 0.10.x dropped them.

• The cluster-flag refusal is simpler. Every refused cluster flag or variable now logs code=CLUSTER_MODE_UNSUPPORTED. 0.10.x also used CLUSTER_FLAGS_INCOMPLETE, CLUSTER_ENV_RENAMED, CLUSTER_RF_MISMATCH, CLUSTER_NODE_ID_INVALID and NATIVE_HOST_CLUSTER_UNSUPPORTED.

  • Any non-empty CORTEX_CLUSTER_* variable now counts, CORTEX_CLUSTER_ID alone included.
  • The exit code is still 78, and the message now adds "For a cluster, use cortexdb node --config".

• Cluster dirs are refused. Plain cortexdb refuses a cluster data node's dir: exit 78 CLUSTER_DIR.

• New subcommands. cortexdb node and cortexdb admin run a cluster (below). cortexdb restore-check --data-dir DIR [--reincarnate NODE] is the archive check cold_backup.py runs.

• Cold backups of 0.11.0 dirs. A shard snapshot hard-links the event log's files, and cold_backup.py backup archives such a file as a regular file under each name, which every restore accepts.

• A wedge is fixed. A layer build could stall the server while every worker waited behind a queued scope-registry change. It no longer does.

• A crash loop is fixed. A server killed inside a forget's search-index commit could never start again: at start it re-runs the pending forget, whose commit wrote a file the killed one had left, and the index refused it (forget incomplete … FileAlreadyExists(…del)) on every start. The search index now removes the files its last commit does not name when it opens. 0.10.5 has the same defect.

• Removed. The dormant cluster crates (cortex-rpc, cortex-gossip, cortex-replication, cortex-control-plane) and the orphan cortex-storage are gone, and so is docker-compose.cluster-experimental.yml. Nothing needs protoc to build any more.

• The image carries the label io.cortexdb.one-way-below="0.11.0", which the updater sidecar reads after the pull.

• A documentation fix. POST /v1/auth/revoke answers 204, as it always did. The OpenAPI document said 200.

Cluster mode (new)

cortexdb node --config FILE starts one process in one of three roles:

• metadata: a voter of the three-voter metadata Raft group, which holds the nodes, shards, scope placement and the token-revocation set.
• data: one replica of one shard, running the unmodified single-node engine.
• ingress: the public plain-HTTP API. It routes each request by its scope root to the shard's leader.

Nodes talk over mutual TLS with pinned keys. cortexdb admin places everything: it bootstraps the group, adds nodes, creates shards with 1 to 5 replicas, assigns scope roots, moves a root between shards (scope move), replaces a replica or a metadata voter, and restores a shard that lost its quorum. The cluster operator guide (docs/generated/cluster.md in the source tree) has the configs, the admin verbs, the error and exit codes, and the runbooks.

What it does:

• A shard stays available while a majority of its replicas is up.
• Only the leader serves. Reads are linearizable, and the ingress follows a leader change.
• A lagging replica catches up by a streamed snapshot, and a lost disk is repaired by remove, wipe, add.
• A scope root moves between shards on the operator's command, under load.
• A shard that lost its quorum restarts from one replica's backup (admin shard restore), and a metadata voter that lost its disk rejoins (admin meta remove|add, node --join).
• A shard's leader embeds each committed write once, after it is acknowledged, and replicates the chunks and their vectors through the shard's log; every replica indexes the vectors the log carries, and the vector index keeps no second copy of them.

Its limits in this release:

• Nothing automatic. Nothing places, splits, merges or rebalances on its own, and there is one shard per data node, so three replicas take three data nodes.

• Routing. A request whose scope roots are on different shards is refused (422 CROSS_SHARD_REQUEST). The ingress refuses with 501 CLUSTER_ROUTE_UNSUPPORTED:

  • the blob routes, keyed uploads included;
  • by-id reads without ?scope=;
  • cross-shard listings;
  • audit, import and export;
  • most admin routes, including the re-derivation repair routes and the blob-link repair, and the cohort status route (ask a data node).

  A data node refuses a capture of binary blob bytes (400 BLOB_UNSUPPORTED_CLUSTERED); a text blob is inlined and accepted. A body the ingress cannot read the scope from is 422 INVALID_BODY when a routing key has the wrong type and 400 when it is not JSON.

• Leader-local state. LLM-derived rows (enrichment facts, beliefs, concepts), blob bytes, erasure jobs and receipts, the usage ledger, audit rows, config files, the answer cache and the cohort projections, read-all cache and cohort_alias.key are local to each node, not replicated. An erasure's job and receipt are kept on the node that ran it (the leader of the scope's shard at the time); another node answers 404 for them. The erasure feed is never truncated.

• Configuration. A data owner clears every inherited CORTEX_* variable and reads no override file: set engine variables (the answer cache, the cohort flags, the label limits, the request log, CORTEX_RECALL_BUDGET_EXCERPTS) in each data node's [engine] table. The ingress answers GET /v1/admin/version from its own environment, so set CORTEX_RECALL_BUDGET_EXCERPTS there too. /internal/ui/* (the console's control plane and one-click updates) answers 403 on a data node and is not routed by an ingress.

• A large forget or erasure delays its whole shard. While a replica applies an erasure entry (up to 4,096 events, about 3 s for short captures), it holds every scoped or by-id request on the shard, whatever its scope root: after 2 s it answers 503 SHARD_CATCHING_UP, which the ingress asks again within resend_window_s. In a test a 12,000-event forget delayed requests on another root by up to 5.4 s, with no error. A replica that starts serving, or becomes leader, with several entries still to apply holds requests until it has applied them all. The owner's /v1/admin/metrics shows the wait as edge.erase_floor_wait_seconds.

• Erasure clear time. log_residual_cleared_at waits for the shard's current voters only (replica_unreachable names a voter that did not answer). It does not cover a learner, a replica removed from the shard, or a scope move's source shard (a move's cleanup is not an erasure).

• Cost. Only a shard's leader embeds: each write once, and the queries of the reads it serves. Embedding spend is one embedding per chunk per shard at any replica count, and a replica added back, a replacement or a scope move's target embeds nothing. Measured at RF=3 with OpenAI embeddings: each shard's leader spent the tokens a single node spends on the same writes (730,506 for 1,500 multi-chunk passages), and the other replicas spent none on writes.

• The leader's provider indexes the shard. While the leader's embedding provider fails, no replica indexes new writes: writes are still acknowledged and reads serve what is indexed, but ?wait=indexed answers 408, and a new leader's reads answer SHARD_CATCHING_UP until the backlog is embedded. cortexdb admin status fails VECTORS_STALLED after 120 s; fix that node's provider, or move leadership with admin shard transfer.

• wait=indexed has a longer tail on a cluster. Because only the shard leader embeds and the vectors travel through the shard's log, a wait=indexed write at RF=3 with a real provider had a p99 about 24 % above the build before vector replication (p50 +4 %), following the provider's own latency jitter; with a steady stand-in provider it was faster. Default-wait writes and reads are not affected, and a plain single node does not take this path.

• The forget cutoff after a failover or a move. The times a shard records never go back. After a failover from a leader whose clock ran ahead, or a move from a shard whose leader's did, the cutoff is ahead of the new leader's clock until that clock catches up. Meanwhile a capture made during a forget request is forgotten with it (see "The forget cutoff" above).

• One binary per shard. Every log entry carries its apply format, and this release's entries carry format 5. A replica on an older binary stops on a newer entry (SHARD_FORMAT_UNSUPPORTED) rather than apply it differently, so a rolling upgrade across a format change stalls the replicas left behind: upgrade a shard's replicas together.

• Pins and transport. openraft is pinned at =0.10.0-alpha.34. The public ingress port has no TLS: put a terminator in front of it.

• Measured on one region. Six hosts in three AWS availability zones of one region, and a real embedding provider (OpenAI) at RF=3, passed the cluster demos: a new leader 3.1-3.7 s after the old one was killed, and across zones a capture's p50 about 13-15 ms, wait=indexed 22-24 ms and a recall 12-13 ms (measured before the vectors moved into the log; the RF=3 demo passed again after, on six such hosts with one shard's links shaped to 1 and 0.5 MB/s, with 384-dimension mock vectors). Another region, a WAN or a VPN has not been measured.

• A slow link between replicas. A follower that needs a snapshot streams it from the leader on a connection of its own, which the transfer's progress keeps alive, and a retry starts again from zero. With every flow into the follower shaped to 0.2 MB/s on one host, snapshots of 4 and 6.5 MB installed at the link rate in 10 of 10 runs, with no leader change. The transfer's deadlines count the bytes the leader hands its socket, not those that cross the link, so below about 0.2 MB/s a snapshot of several MB can fail every attempt: keep a shard's replicas on links that stay well above that. The shard's log also carries each chunk's vector and text (about 3.5 KB per chunk at 1,536 dimensions), so its snapshots are larger: across six hosts with the links shaped, a 20 MB snapshot installed in 22 s at 1 MB/s and a 42 MB one in 87 s at 0.5 MB/s. Size the links for the log's byte rate, several times the capture bytes.

• Run the nodes on jemalloc. The container image preloads jemalloc; start a data node from the bare binary the same way (LD_PRELOAD naming libjemalloc.so.2, and the image's MALLOC_CONF, both of which cortexdb node keeps). Under glibc's allocator a shard leader's memory passed 2 GiB within an hour of a soak's load, from fragmentation, and fell back only at a restart; with jemalloc it stayed under 1 GiB.

A single-node data dir cannot be adopted as a cluster shard: cortexdb node refuses it (exit 78 CLUSTER_DIR_FOREIGN). The one-click updater manages single-node Compose deployments only; a cluster is upgraded node by node, one binary per shard.

The 0.10.x features on the replicated log and on a cluster

Every feature of 0.10.1 to 0.10.5 is in this release and behaves on a single node as it does on 0.10.5, except where "What changes for a single node" says otherwise. Where they work differently here:

• Erasure index (0.10.1). The index follows the rows: the coordinator's derived_wal/ keeps its own (backfilled in the background), and the event log's is replicated state that the log's apply writes with each row, built for the upgraded rows by the first start's derived-row move. 0.10.x's idempotency reverse rows are not needed. GET /v1/admin/metrics erasure_index gains derived_wal and source_wal, and its wal is ready only when both are. Erasures of events the log no longer holds (0.10.1's DB-1) ride the Erase entry, so every replica removes what cites them.
• directives.extract is stored on the captured event and honoured on single, bulk and /v1/import/jsonl writes, as on 0.10.5, and on every replica of a shard.
• The answer cache (CORTEX_ANSWER_CACHE=1, opt-in). In memory on each server, never written to disk or replicated. On a cluster any erasure, cancel, eviction or registry change a replica applies, and any row its indexer indexes in the scope (another leader's writes and a move's imports included), invalidates that replica's entries; an entry is bound to the root's placement, so a root moved away and back serves nothing cached; a new leader starts cold; the ingress passes Cache-Control through.
• One-click updates refuse a container that runs a cortexdb subcommand (node, admin, ...) and know one-way upgrades ("Upgrade").
• One durable write per batch (0.10.2) lands on derived_wal/ (one write and one sync per indexer page); a bulk capture is already one replicated log entry per 1 MiB chunk.
• The request log (0.10.2). wal_append_ms is the write's replicated log entry, committed and applied (a bulk item's, its chunk's). On a cluster the data owner that served the request writes the line, under the X-Request-Id the ingress minted or forwarded, which the owner's own refusals carry too.
• Embedding batches (0.10.2). A cortexdb node data owner clears the image's CORTEX_* values, so it runs at the binary's CORTEX_EMBEDDING_MAX_BATCH_ITEMS of 2048, clamped to the response bound, unless its [engine] sets one.
• Cross-customer cohort lanes (0.10.4, 0.10.5; experimental, off by default). The exact lane's projection is built from the shard's event log, which every replica holds. On a cluster set the flags in each data node's [engine]; the ingress routes the cohort routes and the warm-up by the scope's root (a parent and its customers share a root, so its shard holds every row the lanes read). A stable alias changes when another node starts serving the root.
• Enrichment (0.10.4). The enrichment loops, and CORTEX_ENRICHMENT_SCOPE_PREFIXES, run on the shard's leader only (always this node at RF=1). GET /v1/admin/layers/stats, belief_reconcile included, is the leader's.
• Label limits (0.10.5; announced, off unless CORTEX_LABEL_LIMITS=on). Checked by the write pipeline before a capture is proposed, never by the log's apply.
• Settings page (0.10.5). Single node only.

Environment variables

No new variable is needed on a single node.

Error and exit codes

INDEXING_FAILED (502) is now retriable: false. The cluster's own codes (SHARD_CATCHING_UP, SCOPE_MOVING, CROSS_SHARD_REQUEST, CLUSTER_ROUTE_UNSUPPORTED, ...) are in the cluster operator guide.

Measurements

• Upgrade and downgrade, with 0.10.5. A 0.10.5 data dir (six notes, one multi-chunk document, a bulk of four and an erasure, written by the published 0.10.5 binary) was cold-backed up, upgraded by a candidate of this release (the same 11 capabilities, pre_stage2/ taken, nothing embedded at the first start, every event readable and recalled, a 0.10.5 key replayed, the 0.10.5 erasure's job, receipt and proof served), written to, forgotten from and erased from; 0.10.5 then refused the migrated dir, and served it again after the pre_stage2/ steps and after a restore of the cold backup (refused without a binary, restored with 0.11.0's) and of a plain tar, as "Downgrade to 0.10.1 through 0.10.5" says; 0.11.0 then upgraded the downgraded dir again.
• Single node against 0.10.5 (a candidate of this release with 0.10.3 to 0.10.5 merged in, against 0.10.5 built from its tag with the same compiler; one quiet 32-vCPU machine, jemalloc, the two binaries interleaved, five rounds of the light suites and three of the heavy ones): every metric within +5 % of 0.10.5's or 0.10.5's spread between rounds, but memory at ready on an empty dir; see "Writes" and "Operations", "Memory".
• Cluster (RF=1 and RF=3, real hosts in three availability zones of one region, a real embedding provider and shaped links): see "Cluster mode".

Known issues

• About 13 MiB more memory at ready on an empty data dir than 0.10.5 (99-100 against 86-87 MiB), the code and state of the replicated log. It does not grow with the data, and peak memory under load is lower than 0.10.5's.
• The way back loses the writes since the upgrade and brings back what was forgotten or erased since ("Downgrade to 0.10.1 through 0.10.5"). Issue those forgets and erasures again on 0.10.x.
• The one-click updater of a 0.10.x server must not be used for this upgrade ("Before you upgrade").
• Cluster mode's limits are under "Cluster mode": operator placement only, single-shard requests, leader-local derived state, a large erasure delays its whole shard, one binary per shard, measured in one region.
• Everything listed under "Known issues" in RELEASE_NOTES_0.10.5.md still applies.

Clients

No client is re-released with 0.11.0. Every client published with 0.10.5 works against 0.11.0 unchanged, and the new codes carry retriable, which the SDKs already pass through.

The source tree carries documentation-only edits to the Python SDK (docstrings of write_status, wal_status, wal_compact, forget_preview, erase and experience()), the TypeScript SDK (the ExperienceItem.labels doc comment) and cortexdb-mcp (memory_status's not-found text and the memory_store tags description). They ship with each client's next release.

Notes

POST /v1/experience?wait=settled is still accepted but cannot succeed and returns 503 SETTLEMENT_UNAVAILABLE. Use wait=indexed.

Single node remains the default topology; cluster mode is new and opt-in.