CortexDB · docs

CortexDB 0.10.0 — multilingual memory, erasure that reaches every copy, and derived-data repair

Never published; superseded by 0.10.1. 0.10.0 was prepared but never tagged or published. Everything on this page ships in 0.10.1, together with 0.10.1's own changes (budget excerpts, the context receipt and the answer-context merge). Read RELEASE_NOTES_0.10.1.md, which covers every change since v0.9.13; this page is kept for reference only.

This release captures and recalls memories in the language they were written in, makes /v1/forget and /v1/erasures reach the uploaded blob and the bytes left in the storage files, repairs derived facts and beliefs that earlier enrichment-on builds got wrong, and restores the /v1/answer context quality that whole-event contexts had cost.

The measured launch set is English plus Latin-script Hinglish. Every other script is captured, tagged and recalled, but recall across scripts (a Latin query over a Devanagari memory) is not yet at the quality bar; that is the next release's work.

Before you upgrade

Read this section even if you only use English. Several contracts change.

Multilingual capture and recall

Nothing is translated. You send the original text plus hints, and the server keeps both.

Erasure

Facts and beliefs (enrichment-on deployments)

Nothing here runs unless enrichment is on (CORTEX_ENRICHMENT_DELAY_SECONDS set).

Repairing derived data written by earlier builds

Enrichment-on stores written by v0.9.13 and earlier can hold facts with a lost negation, facts of different people merged, names in two scripts merged, colliding entity ids, and beliefs counting facts of the opposite polarity. New trusted-operator routes repair them from the event log, without wipes and without re-ingesting anything:

Follow docs/DERIVED_REDERIVE_RUNBOOK.md step by step: audit, quiesce ingestion, cold backup, dry runs, apply, re-audit. It also says what the audit cannot see (a denial an older extractor stored as an affirmative) and how to check for it by hand. An apply is one-way for a rollback to v0.9.13: after it, going back means restoring the cold backup.

Answer context

/v1/answer contexts are built from whole events. This release fixes the packing that change had broken: derived fact rows behind a timestamp were ranked as structure and cut first, assistant replies carrying an echo of the user were counted as user evidence, and one oversized event ended the context. Dialogue with named speakers ("Caroline: ...") ranks level with fact rows, and in a congested multi-topic answer each topic's focused dialogue and fact rows are packed before the rest. Each fix has a switch, and the documented set in PRODUCTION_CONFIG.md ("Answer context") restores the previous behaviour: CORTEX_EVIDENCE_ROLE_CLASSIFIER, CORTEX_CONTEXT_STRIP_ECHO, CORTEX_CONTEXT_RENDER_SKIP_OVERSIZED, CORTEX_FACET_PACK_POLICY, CORTEX_CONTEXT_ASSISTANT_EXCERPTS (off by default) and CORTEX_ABSTAIN_SELF_RESCUE (the default is the previous pass; guarded is opt-in).

/v1/answer and /v1/compose report diagnostics.degraded_sources as stable codes only (coordinator_recall, wal_supplement, derived_<lane>, artifact_search, index_divergence, derived_backfill, facet_plan_unavailable, facet_recall_failed, ms_executor_fallback, else unknown); error text stays in the server log. /v1/answer also detects "no such information" abstentions, no longer treats general nouns ("issue", "problem") as unsupported premises, and reports premise_unsupported_terms and extractive_retry_outcome.

LongMemEval-S (full 500 questions, blind server harness, enrichment off): this release was checked for non-inferiority against our previous baseline run under the same configuration, and passed. That is a regression check, not a published score. The run predates the answer-stage changes in the paragraph above, which came in with the final merge.

Also in this release

New environment variables

All are in docs/generated/env-vars.md (§4.2, §9.1, §11.4, §12) and the operator-facing ones in PRODUCTION_CONFIG.md: CORTEX_DESTRUCTIVE_LEASE_WAIT_SECS, CORTEX_SHUTDOWN_DESTRUCTIVE_WAIT_SECS, the CORTEX_ERASURE_RESIDUAL* family and CORTEX_ERASURE_PERIODIC_COMPACTION_SECS, CORTEX_CHUNK_FRAGMENTS, CORTEX_POLARITY_RECHECK, CORTEX_POLARITY_QUOTE_TRIGGER_EN, CORTEX_ENRICHMENT_SCOPE_QUIET_MS and ..._MAX_MS, CORTEX_ENRICHMENT_SKIP_NON_EN, the answer-context switches above (plus CORTEX_CONTEXT_ASSISTANT_EXCERPT_MAX_BYTES), CORTEX_QUERY_VARIANTS_FUSION and CORTEX_QUERY_VARIANTS_DEPTH (evaluation), CORTEX_QUERY_EMBED_CACHE_CAP, CORTEX_EMBEDDING_CACHE_DISABLE (measurement only), CORTEX_CODE_REPO_ROOTS, and the erasure index switches CORTEX_ERASURE_INDEX, CORTEX_ERASURE_INDEX_BACKFILL, CORTEX_ERASURE_INDEX_BACKFILL_ROWS and CORTEX_ERASURE_INDEX_BACKFILL_PAUSE_MS. Never set a CORTEX_EVAL_* variable on a production server.

Known issues

Found in the pilot-upgrade drills on enrichment-on stores and the reviews after them. They affect enrichment-on deployments only; with enrichment off (the default) there are no facts or beliefs to get wrong.

Upgrade

  1. Read PRODUCTION_CONFIG.md: the enrichment spend switches, "Erasure raw residual", "Answer context" and the 120 s stop grace.
  2. Take a cold backup (scripts/cold_backup.py), then upgrade. Expect the one-time code-plane re-index and, on incremental-layer stores, the belief sweep on the first start. The swept beliefs are rebuilt by the auto-layer ticks, at most 2,000 topics per scope per tick with one embedding call per belief, so on a large store the reconcile outlasts the first tick: it is finished when GET /v1/admin/layers/stats reports belief_reconcile.done: true (dirty_topics counts what is still queued). done needs CORTEX_V1_LAYERS_AUTO on (without it nothing is seeded or swept: rebuild with POST /v1/beliefs/build); it never turns true while a CORTEX_V1_LAYERS_AUTO_EXCLUDE_SCOPES prefix covers a scope with facts (the sweep empties that scope's beliefs and no tick rebuilds them: build it by hand); and a build that fails drops its topics, so compare by_scope.<scope>.belief_errors_total before and after. If you keep the old container for a rollback, set its restart policy to no (docker update --restart=no): with always, a Docker daemon restart starts it again beside the new one, on the same port and data dir. The first start also backfills the erasure index in the background: no downtime and no extra start time on a small store (a 1M-event store's 2M WAL rows took 6.6-7.1 s and a 5M-event store's 10M took 53 s, both overlapped with the start), and until erasure_index.wal.ready is true on GET /v1/admin/metrics erasures read the whole store as before. The backfill resumes where it stopped after a restart or a crash. Every start also rebuilds the erasure retry candidates from the forgotten markers and the idempotency reverse index from the live records, both in the background.
  3. For each tenant that uploaded text blobs, run the source-blob-links backfill, then review orphan-blobs (a blob inlined before 0.10.0 and not backfilled is listed even when its event is live).
  4. Enrichment-on deployments: run the audit and, where it or the runbook's manual check finds damage, the repair in docs/DERIVED_REDERIVE_RUNBOOK.md.
  5. Clients: gate the new fields on capabilities[].

Rollback

A v0.9.13 binary opens and serves a data dir 0.10.0 has used: no column family was added (the erasure index lives in an existing family, under a key range earlier binaries never read). Three things end or limit that:

After upgrading again:

  1. Delete the orphan blobs of events erased during the window (GET /v1/admin/index-audit/orphan-blobs, then DELETE /v1/blobs/{id}; check each first, the report also lists blobs of live events written before 0.10.0), and set CORTEX_ERASURE_PERIODIC_COMPACTION_SECS for a while to reach the raw residual.

  2. Facts: a fact 0.10.0 erased before the rollback is dropped again at boot. A fact that v0.9.13 erased or forgot during the window and that a v0.9.13 crash brought back is not: it is still served. Keep, for every forget and erasure you send during a rollback window, its body, the time you sent it and its response; the v1 audit log (CORTEX_V1_AUDIT_PATH) records each request's endpoint, actor, time, status and a peppered body hash, not the body, so it tells you how many there were and when, not what they named. After upgrading again:

    • Re-issue as it was only a request that names memory ids (selector.memory_ids of evt_ or fact_ ids), without its from_preview_id and idempotency_key: it names exactly what it named before. An erasure re-issued this way matches facts by the events they came from even though the events are gone, and reports them in summary.deleted_facts.
    • Never re-issue any other request as it was: a scope-wide forget, an about_subject, about_entity or predicate forget, an ent_ id, a whole-scope erasure. None has a time bound, so re-issued later it also deletes what was written after the original, in the window and since the upgrade; a scope-wide redaction or erasure also moves the scope's forget tombstone, which hides every earlier event of the subtree for good. Remove instead exactly what came back: the facts in the request's scope that its selector matches, that were created before it was sent and that no later capture supports, forgotten by fact_ id; for an erasure, also re-issue it by the ids of their source events that no longer exist, which names no live event. Take the time of the request from the server, never from the client: the response's audit_id or erasure_id is a UUIDv7 that carries it. Leave facts recorded within a couple of minutes of it to a person's review rather than a script, and review the list item by item when the response was not kept. docs/ERASURE_COVERAGE.md ("Rollback window") has the procedure.
    • Never turn a forget into an erasure: a forget that named only layers: ["facts"] left its event live on purpose, and an erasure of that id deletes the event.

    What cannot be recovered this way: a request whose body or time was not kept; a fact built after the request from one that came back (a consolidation merge), which the time bound keeps; an about_entity that names an entity-graph id. Better still, hold the requests that do not name memory ids until the upgraded build serves again: then there is nothing to re-issue.

  3. Beliefs: on incremental-layer deployments the belief sweep runs by itself; elsewhere rebuild them with POST /v1/beliefs/build for each scope.

  4. The erasure index catches up the rows the earlier binary appended during the window, in the background, before erasures read it again.

docs/ERASURE_COVERAGE.md ("Rollback window") has the details.

Clients

Client versions prepared with this release and released with 0.10.1: cortexdbai 0.12.0 on PyPI and 0.10.0 on npm (version(), capabilities(), the capture and recall language, time and variant parameters, delete_blob / deleteBlob, event_chunks / eventChunks, recall diagnostics types and matched-fragment helpers); cortexdb-mcp 0.8.0 (matched document spans in memory_search output, clearer code-plane tool output); cortexdb-cli 0.6.0 (cortexdb auth tokens --scope is now where your auth.mint is checked, and the new --confine CAP:PREFIX carries the token's scopes); cortexdb-connectors 0.2.22 (GitHub: full PR, issue, comment and push bodies, REST-list backfill with GITHUB_BACKFILL_DAYS, object-identity record keys; read its CHANGELOG before upgrading, because GitHub records are re-keyed).

Notes

Single-node remains the only supported topology.

Erasure and forget were measured up to 5,000,000 events; 25,000,000 was not measured. What still grows with the store is listed in docs/ERASURE_COVERAGE.md ("What an erasure reads"): a scope-wide erasure's pass over the content store, the bitemporal closure when bitemporal records are erased, and, on the legacy dual WAL (CORTEX_UNIFIED_WAL=0), every full scan this release removed from the unified WAL.

POST /v1/experience?wait=settled is still accepted but cannot succeed and returns 503 SETTLEMENT_UNAVAILABLE. Use wait=indexed.