CortexDB 0.12.0 — a full disk or full memory no longer kills the server, and old data can move to an object store

0.12.0 is a feature release on top of 0.11.0. Everything in RELEASE_NOTES_0.11.0.md (and the notes before it) still applies; this page lists what changed since.

Upgrading from 0.11.0 is reversible as long as you do not turn on the two new opt-in features that move data: a keep_learned retention rule and the segment store (CORTEX_ICE_URL). Both write log entries of shard apply format 6, which 0.11.0 cannot read; take a cold backup before turning either on. Nothing else changed in the log or the replicated state, and the one-click updater's one-way boundary stays at 0.11.0.

Getting 0.12.0. This release is for pilots and customers: download it from https://customer.cortexdb.ai (the Linux amd64 and arm64 tarballs, a docker load image file per platform, and the image in the private registry it names). It is not on Docker Hub or the public GitHub releases, so the console's "Update now", which pulls from Docker Hub, does not install it: upgrade by hand. A 0.12.0 server checks https://customer.cortexdb.ai/api/latest-version for newer releases.

For clients nothing is removed: the routes, the request and response shapes and the capabilities are 0.11.0's, plus the additions below. The published clients need no release.

Summary

• A full disk no longer kills a default server. At 95% used or under 4 GiB available on the data directory's disk, new writes get a retriable 503 DISK_FULL (Retry-After: 30) and the background jobs that create data pause; searches, forgets, erasures and retention keep working, and writes resume on their own once space is freed. Through 0.11.0 the next write failed and the process exited (WAL_FATAL, exit 74). On a cluster the shard leader refuses writes while any voter's disk is critical.
• Full memory no longer kills it either. Vectors in RAM are capped by default (an unset CORTEX_VECTOR_RESIDENT_MAX now means auto, sized from the process's memory limit); the least recently used scopes move to a compact cold tier that is still searched, so a scoped search returns what an all-in-RAM index does. All RocksDB stores share one memory budget. On Linux a memory-pressure monitor turns heavy requests away with a retriable 503 MEMORY_PRESSURE before memory runs out, while single writes, ordinary recalls, answers, forgets and erasures are always served.
• Old data can move to an object store (opt-in). With CORTEX_ICE_URL (S3, GCS, Azure Blob or a local path) and a node key, the server seals its oldest events, by scope root, into encrypted segment objects while the data disk is above its target, on a single server and on a cluster. Reads never fail: recall searches the segments within a per-request budget, and what was learned from sealed events stays. Erasures, forgets and retention reach sealed events, and an erased event's segment is rewritten under a new key.
• Retention can keep what was learned (opt-in). A retention rule's new mode keep_learned removes the raw events past their age and keeps the facts, beliefs and other knowledge learned from them; erase (what every existing rule means) removes both.
• Reads no longer fail while a scope is created next to them. A read that raced a scope registry change used to answer AUTHORIZATION_STATE_CHANGED; it now runs again from a fresh authorization snapshot, and only a change that concerns the scope it reads reaches it.

Disk

At 95% used or under 4 GiB available (CORTEX_DISK_CRITICAL_PERCENT, CORTEX_DISK_CRITICAL_FREE_MB; warn at CORTEX_DISK_WARN_PERCENT, 85%) every command that adds data (a capture, a bulk chunk, an import, a tenant map) is refused before it is proposed, writing nothing: 503 DISK_FULL, retriable, Retry-After: 30, details.node naming the node. Forgets, erasures, retention and cancels are never refused. The enrichment scanner, the auto layers, the artifact workers, the compaction pass and snapshot saves pause at critical and resume on their own. /v1/admin/ready stays ready: true and sets degraded; /v1/admin/metrics disk, GET /metrics cortexdb_disk_* and a cluster node's /node/v1/status disk report it. CORTEX_DISK_GUARD=off only reports.

Memory

• Vectors in RAM. CORTEX_VECTOR_RESIDENT_MAX unset now means auto: half of this process's memory limit (the smallest of the host's memory, the container's cgroup limit and CORTEX_MEMORY_LIMIT_MB) less a 1.5 GiB reserve, over the bytes a vector takes in RAM, and at least 50,000 vectors (CORTEX_VECTOR_RAM_FRACTION, CORTEX_VECTOR_RAM_RESERVE_MB). Past it the least recently used scopes move to the cold tier, which now keeps the same compact codes as RAM (548 bytes a vector at 1,536 dimensions under TQ2, instead of 6 KB) and is searched before the rescore. A restart loads at most about the cap. off keeps every vector in RAM, as before.
• RocksDB. Every store shares one block cache and one write-buffer manager, a tenth of the memory limit by default (128 MiB to 4 GiB, CORTEX_ROCKSDB_MEMORY_MB; off restores each store's own sizing), and each database's write-ahead log is bounded.
• Memory pressure (Linux). Every 2 s the server reads its memory against the same limit. At 85% it answers bulk writes and imports (before their body is read), subtree recalls, recalls over 16,000 tokens and fresh cohort read-alls with a retriable 503 MEMORY_PRESSURE (Retry-After: 10), runs /v1/recall at half its permits, pauses the jobs that create data and sheds vectors to the cold tier; it leaves the state once three readings in a row are at or below 75%. CORTEX_MEMORY_PRESSURE=off turns it off.

Running several servers on one host without containers: give each CORTEX_MEMORY_LIMIT_MB, or every one sizes itself from the whole host.

The segment store (opt-in)

Set CORTEX_ICE_URL (an s3://, gs://, az:// or file:// URL; CORTEX_ICE_ENDPOINT, CORTEX_ICE_REGION and the CORTEX_ICE_* credentials for S3-compatible stores) and CORTEX_ICE_KEY_FILE, a node key that cortexdb ice-key --out FILE makes. Keep the key file with your backups: losing it loses every sealed event.

• Sealing. While the data disk is above CORTEX_ICE_TARGET_PERCENT (75%), or the disk guard reads warn, the server seals its oldest events past CORTEX_ICE_HOT_DAYS (7) into segments of CORTEX_ICE_SEGMENT_MB (48): each event's row, text, vectors and index entries leave the local disk for one encrypted object (a key of its own per segment, wrapped by the node key), and the log keeps a small marker. An event enrichment has not learned from is held back, and is learned from its segment later if it had to go.
• Reading. Recall searches the segments of the scope root it reads as two more ranked lists, within CORTEX_ICE_READ_DEADLINE_MS (1,500 ms), _READ_MAX_MB and _READ_MAX_REQUESTS a request, through a disk cache and a memory cache (CORTEX_ICE_CACHE_GB, _CACHE_MB). What it cannot read in time is left out and the response says segments_unreachable in warnings; the request still answers. GET /v1/events/{id} serves a sealed event from its segment (503 SEGMENT_UNAVAILABLE, retriable, when the store does not answer), GET /v1/events lists sealed events beside live ones, POST /v1/export exports them, and GET /v1/experience/status?idempotency_key= finds a sealed write.
• Deleting. Erasures, forgets and retention reach sealed events. The segment is marked at once (the event is never read again) and rewritten under a new key within a pass, the old object deleted; the erasure receipt names the segments and /verify checks them. A server journals every erasure into the store, and a data dir restored from a backup (cortexdb restore-check) applies again every erasure made since the backup before it serves.
• Ownership. Only the server that owns a bucket location deletes there; a copied data dir or a restored backup deletes nothing until an operator adopts or forks the location (POST /v1/admin/ice/owner).
• Clusters. Every data node takes the same CORTEX_ICE_URL and key in its [engine]. Each shard's leader seals for its fullest replica's disk; every replica detaches what a seal commits and reads the same segments. A scope root moved between shards carries its segments by reference (the objects stay where they are), and a new leader warms its caches. A shard restored with restore-check --reincarnate needs the ice settings in the environment.
• AWS KMS. With CORTEX_ICE_KMS_KEY_ID the key file holds only the node key's KMS ciphertext (cortexdb ice-key --kms KEY_ID --out FILE), decrypted at boot by one Decrypt call: no disk holds the key, and disabling the KMS key makes every sealed segment unreadable.
• Operating it. GET /v1/admin/ice/status, POST /v1/admin/ice/seal and POST /v1/admin/ice/owner (trusted operator), /v1/admin/metrics ice and cortexdb_ice_*. A log that holds segments refuses to boot without CORTEX_ICE_URL or with another node key (exit 78 ICE_REQUIRED, ICE_KEY_MISMATCH).

Not yet: the cross-scope ranked subtree recall does not search segments (its response says subtree_segments_not_searched), and a root's many small segments are not merged.

Retention: keep_learned

PUT /v1/admin/retention/policies takes mode: keep_learned (the default for a rule stored without one) or erase. Under keep_learned the sweep removes each expired event's raw data and keeps an expired marker and everything learned from it, which recall, citations and /v1/beliefs/why keep serving; with enrichment on, an event not yet learned from is learned from first, within a daily budget. erase removes both, as every rule did before. A rule stored before this release, and the deployment default, mean erase unless CORTEX_RETENTION_DEFAULT_MODE=keep_learned; along a scope path erase wins. /v1/erasures still removes everything, expired markers and what was learned included. The Python SDK and the CLI take the mode in their next releases; the HTTP API takes it now.

Clusters

• A shard's leader learns from each event once (one model call) and replicates the result, so every replica holds the same facts under the same ids, and a new leader asks the model nothing its predecessor learned.
• keep_learned retention and the segment store run on data nodes.

Environment variables

New: CORTEX_DISK_GUARD, CORTEX_DISK_{WARN,CRITICAL}_PERCENT, CORTEX_DISK_CRITICAL_FREE_MB, CORTEX_DISK_CHECK_INTERVAL_SECS, CORTEX_VECTOR_RAM_FRACTION, CORTEX_VECTOR_RAM_RESERVE_MB, CORTEX_VECTOR_EVICT_BATCH_PER_TICK, CORTEX_MEMORY_LIMIT_MB, CORTEX_MEMORY_PRESSURE, CORTEX_MEMORY_PRESSURE_{HIGH,LOW,INTERVAL_MS,SHED_FRACTION}, CORTEX_ROCKSDB_MEMORY_MB, CORTEX_RETENTION_DEFAULT_MODE, CORTEX_ICE_URL, CORTEX_ICE_KEY_FILE, CORTEX_ICE_KMS_{KEY_ID,REGION,ENDPOINT}, CORTEX_ICE_TARGET_PERCENT, CORTEX_ICE_HOT_DAYS, CORTEX_ICE_SEGMENT_MB, CORTEX_ICE_READ_{DEADLINE_MS,MAX_MB,MAX_REQUESTS}, CORTEX_ICE_CACHE_{GB,MB} and the store's CORTEX_ICE_* credentials. Changed: CORTEX_VECTOR_RESIDENT_MAX unset now means auto (set off for the old behaviour). docs/generated/env-vars.md describes each.

Error and exit codes

New: DISK_FULL and MEMORY_PRESSURE (503, retriable, with Retry-After); SEGMENT_UNAVAILABLE (503, retriable) on event reads; ICE_OFF (409) and ICE_FAILED (503, retriable) on the ice routes; the warnings segments_unreachable and subtree_segments_not_searched; boot exits 78 MEMORY_CONFIG, ICE_CONFIG, ICE_REQUIRED, ICE_KEY_MISMATCH and ICE_RESTORE_REPLAY.

Measurements

Soaks on spot machines, mock embeddings, no paid API (docs/RESOURCE_GUARDS_PLAN.md and docs/ICE_TIER_PLAN.md, "Soaks"): a server filling a 10 GB disk now keeps serving every read instead of dying; with the segment store on over an S3 API it wrote 161,800 items, served every read (p50 350 ms, p99 1.3 s) and found 100% of recent and 99.94% of old items while sealing 86,655 events; RF=3 clusters with the segment store on passed an hour and half an hour of leader kills, partitions, replacements and moves with every operation linearizable.

Known issues

• On a cluster, after an erasure of an event whose fact superseded an older one, a replica that never applied the erased event's learned record may still show the older fact as current.
• GET /v1/events lists sealed events from the newest 100,000 markers of a scope root only; POST /v1/export exports all of them.

Clients

No client release is needed: every new error is a retriable 503 or an opt-in route the clients already handle. The Python SDK's and CLI's retention mode option ships with their next releases.