CortexDB · docs

CortexDB 0.10.2 — indexed captures that do not queue behind disk syncs, one wait deadline, forgets that survive scope registrations, and the captured speaker on every recalled event

0.10.2 is a fix release on top of 0.10.1, the last published release. Everything in RELEASE_NOTES_0.10.1.md still applies; this page lists only what changed since. It adds no column family, directory or on-disk format, and nothing is migrated: a 0.10.1 server opens and serves a data dir 0.10.2 has used (see "Rollback").

Summary

Upgrade from 0.10.1

Before you upgrade: what clients and operators will notice

Upgrade steps

Nothing is migrated and there is nothing to run afterwards. Replace the binary or image and restart as for any release, giving the server 120 s to stop (as in 0.10.1: it waits for running forgets and erasures and persists its snapshot). Then check:

  1. GET /v1/admin/version: crate_version 0.10.2, the git_sha you deployed, and event_scope_filter_v1 in capabilities[].
  2. GET /v1/admin/ready: ready: true.
  3. The log: if you set CORTEX_EMBEDDING_MAX_BATCH_ITEMS yourself, a CORTEX_EMBEDDING_MAX_BATCH_ITEMS=<n> clamped to <m> warning tells you the value in effect. At 3072 dimensions the image's own 256 is above the bound, so the warning appears with no setting of yours; it is harmless.
  4. If the request log is too verbose for your log pipeline, set CORTEX_REQUEST_LOG=off.

Upgrading from v0.9.13

0.10.2 contains everything in 0.10.1. Read RELEASE_NOTES_0.10.1.md first ("Upgrade from v0.9.13", "Plan the first start", "Upgrade steps"): its first start, background work and upgrade steps are the same for 0.10.2, and this page's "Before you upgrade" applies on top.

Performance

How it was measured: one host, one server container at a time (4 CPUs, 3 GB of memory, mock embeddings, a fresh data dir), 0.10.1 and 0.10.2 alternated with the same client script. The disk matters: the gains come from fewer disk syncs, so they show on a disk whose sync takes milliseconds (here EBS gp3, 2.9 ms per 4 KiB synced write) and vanish on a fast one (NVMe, 0.13 ms). Your absolute numbers depend on your disk; the shape (latency that no longer grows with the size of the burst ahead of you) is what carries over.

Indexed captures into new scopes

Each round, four clients seed 400 memories into one scope while six clients capture with wait=indexed into brand-new scopes, pausing 0.25 s between captures (the "no pause" row drops the pause). In the rows "with forgets", a forget-all of the seeded scope runs during the seed and another after it:

wait=indexed capture, EBS0.10.10.10.2
p50 / p90 / p99 / max (ms), with forgets (10 min)66 / 546 / 1120 / 140372 / 156 / 276 / 460
rounds whose p99 passed 1 s47 of 1220 of 105
captures over 500 ms / over 1 s, with forgets813 / 1630 / 0
p50 / p90 / p99 / max (ms), no forgets (6 min)94 / 1119 / 1569 / 184152 / 119 / 178 / 278
captures over 1 s, no forgets3940
p50 / p90 / p99 / max (ms), with forgets, no pause (5 min)161 / 282 / 794 / 1372159 / 276 / 371 / 992
indexer cost per event in a page5.2 ms0.47 ms

With real disk syncs, 0.10.2's tail (p99 178 ms, no forgets) is below what either build reaches with disk syncs turned off (CORTEX_WAL_SYNC=0, a diagnostic setting that gives up durability: p99 264 ms on 0.10.1, 294 ms on 0.10.2). On NVMe the two builds measure the same (p50 15 ms, p99 95 vs 93 ms).

Bulk captures

500-item /v1/experience/bulk, idle server, EBS, 6 requests0.10.10.10.2
no wait (ms)2953-3158240-412
wait=indexed (ms)3157-3222425-509

A bulk's log write is made while the indexer holds back every newer write, so the other clients' wait=indexed captures used to wait out one disk sync per bulk item. Six new-scope wait=indexed writers beside two clients sending bulks back to back (90 s per bulk size; the first column is a 0.10.2 candidate that already had the indexer change but not yet the bulk one):

Bulk sizeSingles p50 / p99 (ms), indexer change onlySingles p50 / p99 (ms), 0.10.2Bulk log write while the indexer waits, p50 / max (ms)
none24 / 3223 / 31-
50200 / 34468 / 181284 / 327 → 5 / 15
200794 / 1534284 / 439719 / 1149 → 7 / 13
50048 / 371554 / 9942520 / 2795 → 12 / 15

What is left of the singles' tail is the bulk's own indexing (its embedding call and index writes: about 1 s for 500 items here), because wait=indexed waits for the whole log (see "Known issues").

Wait deadline

With an embedding provider that answers 503 to every call, a wait=indexed capture:

0.10.10.10.2
answer408 at 60.0 s, message "timed out after 30s"408 at 30.0 s, waited_ms: 30003, waiting_on: "indexer"

A bulk wait=indexed fails fast with 502 INDEXING_FAILED on both.

Forget and erasure

Forgets and erasures sent while other clients registered new scopes:

0.10.10.10.2
forget-alls beside 12 clients registering new scopes in the same organisation39 of 52 refused 40341 of 41 done at the first attempt (p50 374 ms)
forget-alls beside writers with no pause (arm above)26 403s; 10 of 70 never succeeded in 2 attempts50 of 50 done at the first attempt
forgets beside about 6.5 new child scopes per second inside the forgotten subtree283 of 283287 of 287 at the first attempt
POST /v1/erasures beside the registrations20 refused 40367 accepted, 2 409 (retriable)
GET /v1/erasures/{id} polls11 spurious 404s in 5400 in 670

Forget-all latency (ms, answered 200) has a trade-off on a slow disk: the indexer no longer stalls, so a forget's search-index commit now competes with it for the index writer.

Forget-all p50 / p90 / p99 / max (ms)0.10.10.10.2
EBS, beside the seeding writers108 / 209 / 301 / 341221 / 310 / 372 / 408
EBS, beside writers with no pausep50 115p50 299
EBS, no concurrent writers151 / 210 / 224 / 382155 / 227 / 258 / 383
NVMe, beside the seeding writers50 / 69 / 86 / 10251 / 79 / 96 / 108

Long documents and embedding calls

One 1,000,000-byte event (2,061 chunks) at 1536 dimensions, against an OpenAI-compatible endpoint:

0.10.1 (batch 2048)0.10.2 (image default 256)0.10.2 with 2048 set (clamped to 340)
provider calls110, each response 64.6 MB and rejected9, largest response 8.1 MB7, largest response 10.7 MB
capture408 at 60 s; the indexer kept retrying200 in 1.6 s200 in 1.6 s

What changed

Capture and indexing: one durable write per batch

A wait=indexed capture waits until the indexer has applied every write logged before it, in every scope, not only its own event. The indexer applies the log in pages, and each page wrote its derived rows with one disk sync per event: a 256-event page cost 256 syncs. On a disk with millisecond syncs, a capture that landed behind a burst of writes (a seed, a connector backfill) waited seconds for them. /v1/experience/bulk did the same with its captured events, while holding back the indexer from every newer write.

Both now write a batch with one durable write: one write of the whole batch, then one sync, before anything is acknowledged. A crash keeps either the whole batch or none of it, and offsets are never reused or skipped. The stored rows are the same as before.

wait=indexed is still a wait for the whole log: a capture that lands behind a bulk waits for that bulk's indexing, its embedding call included. Use wait=accepted and poll GET /v1/experience/status when you do not need read-your-writes.

One wait deadline

Forget and erasure: a scope registration no longer refuses a forget

A forget, an erasure and their previews check the caller's authorization once, then take a lease on their footprint. The lease used to be refused whenever anything in the scope registry had changed in between, and every capture that registers a brand-new scope, from any tenant, makes such a change. The request then answered 403 POLICY_DENIED, with nothing done, and the same race made GET /v1/erasures/{id} and its cancel answer a spurious 404.

The lease now compares the plan's own authorization boundaries, so a scope registered outside the footprint does not matter. When a boundary appears or vanishes inside it, the request re-plans on the current state (a caller the new boundary denies still gets 403, a job it cannot see still 404) and tries again, up to 5 attempts, then answers 409 AUTHORIZATION_STATE_CHANGED (retriable, Retry-After: 1). Registry errors that are not a denial keep their own status instead of a 403. Each refusal is logged at warn with its reason. An erasure refused this way marks its job failed with nothing erased; a retry with the same idempotency_key restarts that job and runs it, while a retry without one leaves a failed job record per refused attempt.

Diagnostics:

Recall: the captured identity for every event

Some recall paths built a layers.events item from an older internal record that holds only the capturing caller, while GET /v1/events/{id} read the captured event:

Those items reported the caller as observed_actor, {id: "unknown"} as subject, the scope as scope_key and a per-partition sequence as wal_offset, and a filters.metadata.observed_actor recall dropped them. Every item now carries the captured event's identity. A capture whose stored copy is redacted is never served from the older record instead, and a cancelled capture now leaves the holistic, exhaustive and recency packs too. context_block bytes are unchanged. This had been wrong since v0.7.0.

Checked on a 13,080-byte event captured on behalf of another person and hit at deep chunks, under granular and holistic recall, with prefer_exhaustive, order=recency, a 90-event holistic scope and an observed_actor filter: every item's caller, observed_actor, subject, scope_key, scope and wal_offset equal GET /v1/events/{id}. On 0.10.1 all eight checks fail. The correct identity is also served for long events a 0.10.1 server wrote.

Recall: candidates dropped for want of content

A search-index hit carries no text; its text comes from the content store. A forget deletes the content first and publishes the search-index delete at the end, so a recall running meanwhile can find a hit with no content. It drops it (nothing is ever served) and used to warn "dropped candidates with missing or tenant-mismatched content". Now:

GET /v1/admin/metrics counts them since process start under recall_empty_content_drops (absent_during_erasure, absent_orphan, foreign_tenant).

Recall: after a store grows past the scope-map threshold while serving

A store whose log crossed 250,000 entries while serving, with no reader having built the scope-partition map while it was smaller, never got the map until the next restart, so the holistic view's descendant events came back empty (8 of 8 recalls over 35 s, found on a bulk re-ingest into a fresh data dir). The first check past the threshold now builds the map in the background (about 1 s for 266,000 entries); only the recalls in that window are degraded, and every later one serves the descendant events.

GET /v1/events/{id}?scope= (event_scope_filter_v1)

?scope=<scope> returns the event only when its own scope is exactly that scope. An event in a sibling, ancestor or descendant scope is the same 404 NOT_FOUND as a missing id, byte for byte. An empty, blank or malformed value is 422 INVALID_SCOPE_GRAMMAR, and a repeated scope parameter is refused with a plain-text 400: never a read without the filter. Without the parameter nothing changes; note that the static operator key and an auth-disabled server read every scope of every tenant by id. A 0.10.1 server ignores the parameter, so send it only when capabilities[] lists event_scope_filter_v1, or compare the returned scope yourself. It does not apply to GET /v1/facts/{id}.

Capabilities

GET /v1/admin/version lists 11 names: 0.10.1's ten plus event_scope_filter_v1. budget_excerpt_v1 is the only name an operator switch withdraws: it is not listed while CORTEX_RECALL_BUDGET_EXCERPTS is 0, off or false, the setting that restores the 0.10.0 render, and the startup config lint logs RECALL_BUDGET_EXCERPTS_DISABLED. In 0.10.1 the name stayed listed with the switch off, so a client that trusted it expected contexts inside their budget and did not get them. Every other name stays listed whatever the switches say.

NameNew inWhat it tells a client
event_scope_filter_v10.10.2GET /v1/events/{id} honours ?scope= (exact match)
budget_excerpt_v10.10.1No context entry passes budgets.max_tokens; now absent while the switch is off
the other nine0.10.1Unchanged (RELEASE_NOTES_0.10.1.md, "Capabilities")

Request log and request ids

CORTEX_REQUEST_LOG (off, slow or all; default slow) writes one info line per request under the target cortex_api::request, which needs cortex_api=info in RUST_LOG (the image's default has it):

request request_id=req_0197… method=POST route=/v1/experience status=408 total_ms=30009
  admission_ms=4 lease_wait_ms=0 wal_append_ms=2 indexer_wait_ms=30000 queue_depth=412
  outcome="timeout"

The same id is now on the audit rows the request writes: the authentication layer's row and the gdpr: true row of POST /v1/forget. A forget's audit trail, its log line and your own report can be joined on it. (Before, the gdpr row always, and the authentication layer's row on a successful request, carried a fresh id that appeared nowhere else.)

On the measured runs the line accounted for where a slow capture spent its time: in the seeding test above, 0.10.2's slowest 1% of captures spent a median 245 ms in indexer_wait_ms and 43 ms in admission_ms, and client and server times agreed within 2 ms at p50.

Embedding batches within the response cap

With CORTEX_EMBEDDING_MAX_BATCH_ITEMS=2048 (the 0.10.1 image default), one call for a long document asked for more than the embedding client accepts in one response (a 630-chunk event at 1536 dimensions is about 21 MB of formatted numbers against a 16 MiB cap). The call failed and was retried unchanged, and indexing stalled on that event. Now:

Admin console

The readiness pill said "degraded (mock embeddings)" for every degraded state, including a failing embedding provider (for example an expired key answering 401). It now names the reason (mock embeddings, or embedding calls failing) and shows the server's detail on hover.

API contract text

/v1/openapi.json descriptions are rewritten for API users (no internal project codes, ticket ids or storage internals). POST /v1/forget now references ForgetRequest like its preview does (only scope is required; reason is an alias of audit_note; from_preview_id is documented; unknown fields are refused, as the server always did), which tightens clients generated from the spec. POST /v1/answer marks question_type deprecated and ignored and documents skip_answer_llm, conversation_id, view, order, budgets and include. The new 408s and 409s are declared, with Retry-After on the 409s, and GET /v1/events/{id} documents ?scope=.

Environment variables

VariableChange
CORTEX_REQUEST_LOGNew. off, slow (default) or all; see "Request log and request ids".
CORTEX_REQUEST_LOG_SLOW_MSNew. The slow threshold, default 1000 ms.
CORTEX_EMBEDDING_MAX_BATCH_ITEMSThe image default is 256 (was 2048; the binary's own default stays 2048); the OpenAI-compatible client clamps it to its response bound.
CORTEX_RECALL_BUDGET_EXCERPTS0/off/false now also withholds budget_excerpt_v1 and logs a startup lint.
RUST_LOGThe image default adds cortex_embedding=warn (so does the binary's built-in default).

docs/generated/env-vars.md has the full rows.

Error codes

HTTPCodeWhereChangeRetriable
408WAIT_TIMEOUTPOST /v1/experience, /v1/experience/bulk with ?wait=One budget per request (about 30 s single; min(30 s + 1 s per item, 600 s) bulk, inline indexing included). details.waited_ms, details.waiting_on; bulk adds event_ids. The legacy dual-WAL mirror's wait timeout is a 408, not a 500.yes: poll, or resend with the same keys
409AUTHORIZATION_STATE_CHANGED/v1/forget, /v1/forget/preview, /v1/erasures, /v1/erasures/preview, GET /v1/erasures/{id} and its cancelNew on these routes. Replaces a spurious 403 POLICY_DENIED or 404. Retry-After: 1, details.retry_after_seconds. (On /v1/recall and /v1/answer the same code remains a 503.)yes
422INVALID_SCOPE_GRAMMARGET /v1/events/{id}New there, for an empty or malformed ?scope=.no

Startup warnings

Rollback

0.10.2 adds no column family, directory or on-disk format; its batched log writes store the same rows a one-by-one write stores. To roll back, stop 0.10.2 with the usual 120 s grace and start the 0.10.1 build on the same data dir. A rollback brings back 0.10.1's behaviour, including the issues this release fixes (and, on the 0.10.1 image, its embedding batch default of 2048). A client that gates on capabilities[] stops sending ?scope= by itself; one that does not keeps sending it, and 0.10.1 ignores it.

What was checked, on one data dir, with a graceful stop between each step:

  1. 0.10.1 wrote English, Devanagari and Hinglish captures, a 32 KB turn, a 13 KB event captured on someone else's behalf and a 200-item bulk, and ran a forget-all and a forget by id.
  2. 0.10.2 read all of it (a budget excerpt, ?scope= giving 200 for the event's own scope and 404 for an ancestor, and the correct identity for the long event 0.10.1 wrote, which 0.10.1 itself had served with the caller's identity), wrote single captures, a 300-item bulk and an 11-chunk Hindi document, and ran a forget-all and a forget by id.
  3. 0.10.1 again opened the store and served everything 0.10.2 had written (the bulk samples, the long document by id); what 0.10.2 had forgotten stayed gone. It wrote, bulk-wrote 100 items and forgot the document by id.
  4. 0.10.2 again read everything, including the writes made during the rollback, and the forgotten items answered 404.

Every phase stopped cleanly (exit 0) and logged no error. After each phase the column families of every RocksDB store (from its latest options file) and the data dir's top-level entries were listed: 0.10.2 added none, before or after the rollback. The final 0.10.2 build passed all 72 checks.

A rollback from 0.10.2 straight to v0.9.13 was not re-run. 0.10.2 stores nothing 0.10.1 does not, so RELEASE_NOTES_0.10.1.md, "Rollback", describes it, with its limits.

Known issues

New or changed in 0.10.2:

Carried over from 0.10.1 (still present in 0.10.2):

Clients

No client changes: no client version is bumped, and cortexdbai 0.12.0 (PyPI) and 0.10.0 (npm), cortexdb-mcp 0.8.0, cortexdb-cli 0.6.0 and cortexdb-connectors 0.2.22 work against 0.10.2 unchanged. Every change is additive or on an error path. What clients will see:

Notes

Single-node remains the only supported topology.

The single-capture, forget and registration figures above were measured on a 0.10.2 candidate before its last change, which touched only the bulk capture write and the request ids; the bulk figures, the upgrade and rollback check and the smoke tests were run on the final build. The recall-cost figures (+11% and +23%) come from the change's own measurement and were not re-run on the final build.