Keep retrieval current when the source changes

A search index is a data structure prepared to find information efficiently. It is not the authority on whether that information is still correct. Think of a printed handbook: editing the master document does not change the copies already on desks. A search service has the same maintenance problem.

Indexing prepares or updates that searchable representation. Retrieval asks it for candidates. Context assembly chooses what a model will receive. These are different steps. A result can match your question beautifully and still describe a withdrawn procedure.

As an FDE, you need to explain who owns the original, how changes reach each copy, and what the application does while they disagree. This applies to keyword indexes, vectors, graph relationships, summaries and cached answers. A more capable model does not repair a stale source pipeline.

Our fictional procedure P-17 moves through revisions 1, 2 and 3. A revision is a version number assigned by the source. Revision 2 changes the handoff instructions. Revision 3 records deletion. The source retains a tombstone: an ID and deletion version that help reject old work. It no longer retains the procedure text in its active row.

The visual follows three distinct places: the authoritative source, its search copy, and the context packet. Read the words and revision labels in each. The arrows describe work between stores; they do not imply that updating one store instantly changes another.

Start with both copies at revision 1. Move to Correct the source. Search still matches the word “handoff” in revision 1, but the source says revision 2. The application withholds that candidate. After Refresh the index, the current passage can enter context again.

Continue through the delayed job and deletion states. A revision 1 job arriving late must not roll the copy backwards. A deletion marker must stop use even before the search row is removed. Retrying a revision 2 job after deletion must not resurrect it. Explain each result before checking your reasoning.

These panels deliberately expose synthetic internal debug data so you can compare the copies. In a real service, do not expose unauthorized excerpts in diagnostics, search previews, rerankers or model input.

Download fde-index-lifecycle.py below. Open a terminal in its folder and use your installed Python 3.11+ command; python3 is the example. The program creates a real in-memory SQLite database and full-text search table, runs the lifecycle, then closes the database.

python3 fde-index-lifecycle.py run --stage corrected
python3 fde-index-lifecycle.py run --stage refreshed
python3 fde-index-lifecycle.py run --stage late-job
python3 fde-index-lifecycle.py run --stage deleted
python3 fde-index-lifecycle.py run --stage purged
python3 fde-index-lifecycle.py run --stage replayed
python3 fde-index-lifecycle.py test

Each run reconstructs the same history from scratch and prints the selected state. source is the current authoritative row; indexed is the derived copy; matches lists candidate IDs, versions and decisions; context contains only admitted source text. The browser replays exported results from this exact program. It does not execute SQL in your browser.

SQLite's FTS5 module provides full-text search. Here a normal FTS5 table stores a searchable copy independently of the sources table. The lab deliberately updates them at different steps. A standard build of Python often includes FTS5, but support depends on the installed SQLite build; the script reports a clear error if it is unavailable.

Open apply_job. A job names a source ID, revision and operation. The worker compares that revision with the authoritative row before writing. If they differ, it ignores the job. If the operation is appropriate, it reads the source's current content, removes the previous search row and inserts the replacement in one local transaction. Applying the same current job twice leaves one copy.

Open retrieve. It first gets IDs and revisions from the index, then checks the source. Missing or deleted sources produce no context. A version mismatch also produces no context. A current match contributes the source's text. Bound SQL parameters separate data from the query, and the search phrase is quoted as literal FTS text.

SELECT source_id, revision
FROM search
WHERE search MATCH ?
ORDER BY rowid

This query finds candidates; the Python source checks decide whether their content may be used. Query success alone does not establish currency. Current source text still needs task relevance, citation fidelity and permission checks in a deployed application.

The source and index share one local process here. A real remote service introduces races between reading a version and writing a copy. Define conditional writes or per-source ordering, retry behavior, an authoritative version boundary and reconciliation. Do not assume this SQLite transaction can span a document store and an independent managed search service.

Predict what happens if you search for a word that exists only in the old revision:

python3 fde-index-lifecycle.py run --stage corrected --query old
python3 fde-index-lifecycle.py run --stage refreshed --query old
python3 fde-index-lifecycle.py run --stage refreshed --query revised

Before refresh, “old” finds a stale candidate that must be withheld. After refresh, that term no longer matches. “Revised” then matches current evidence. This distinguishes a real change to the search contents from a filter that merely hides every result.

For a code experiment, save a separate copy of the script. Change NEW_TEXT so it no longer contains “handoff.” Run the refreshed state with the default query, then search for a word you added. Explain why an up-to-date index can legitimately return no match. Keep the original file and record your diff. The original tests describe the original fixture; do not edit their expectations merely to conceal a changed result.

For review, ask a receiving engineer to choose a stage without telling you which. Use the JSON to name the authoritative revision, the search revision, the admitted context and the next operational action. A useful answer explains the state, not just the command's exit code.

Create a lifecycle table for every derived copy: parser output, chunks, vectors, keyword index, graph edges, summaries and answer cache. Name its owner, source/version key, update path, deletion path and reconciliation check. A completed queue delivery is not proof that all of those copies changed.

An idempotent update can be repeated without producing another effect. An out-of-order event arrives after a newer event even though it describes an older version. Reconciliation compares the intended source state with the derived state and repairs differences. Treat those as separate responsibilities.

Withholding stale context protects the evidence boundary but can reduce availability. Agree what the user sees during rebuilding: a current direct-source read, a clearly unavailable answer, or an approved manual route. Never silently answer from withdrawn material to make availability look better.

Logical deletion removes a row from normal query results. It does not prove that bytes disappeared from every storage page, backup, log or replica. This in-memory lab establishes query and replay behavior; retention, secure erasure and compliance require separate environment-specific evidence. Current document authorization and citation-opening checks also remain necessary.

Keep your predictions, three changed-query outputs, changed-copy diff and lifecycle table. Add a short handoff: which behavior ran, which copy was withheld, which retry was rejected, and what the local experiment cannot establish about distributed concurrency or access control.

Original lesson and examples by LockedIn Labs. Primary references checked on 23 October 2026:

• SQLite FTS5: full-text queries, stored content, consistency and deletion behavior.
• Azure AI Search: update or rebuild an index: incremental updates, document identity, batch results and asynchronous indexing.

Continue to evaluation and include stale, missing and deleted sources in the expected-evidence cases. Return to data, infrastructure and recovery to examine what survives a restore. A restored index can be consistent with its backup and still miss later business changes.