Data 101: a document, a record and a searchable copy

Object storage keeps named objects: bytes, a key and metadata. A procedure PDF, image or JSON document can be an object. A relational database stores records in tables with named columns. A search index is a representation prepared to help find material. These are responsibilities, not a requirement to buy three separate products. A database may provide search as well as record queries.

Think of a shared drive with manuals, a spreadsheet tracking tickets and a catalog that helps find a manual. The analogy gets you started. A database also provides queries, constraints and transactions; a cloud object service has its own versioning and access behavior. The catalog is a copy, so it can lag behind a changed or deleted manual.

A row is one record. A primary key identifies it, such as ticket T-104. SQL, Structured Query Language, expresses queries and changes to relational data; SQLite and PostgreSQL are database software. A transaction groups database operations so they can commit together or be rolled back together. It does not automatically include a separate file, object service or search service.

An authoritative source owns a particular fact. In this example, procedure P-17 owns procedure content, while the ticket table owns T-104’s status. Ingestion builds the searchable copy from the source. Provenance keeps the source ID, revision and origin with that copy. A plausible match is a candidate; it is not proof of current content, permission or a correct answer.

An FDE needs this map before connecting customer data to an agent. If a user asks whether a ticket is open, query its current record. If they need the review procedure, find an eligible passage and check it before supplying it to a model. The model should not guess either the current ticket state or the caller’s permission.

P-17 is a JSON document at revision 2. T-104 is an open ticket that references it. A search table initially contains revision 1. The program’s training scope is a supplied trusted-context fixture, not a real login. Each case starts from a fresh temporary source file and fresh SQLite database; changes do not carry into the next case.

On a narrow screen, scroll the table sideways. Before each run, predict the ticket state, event count and context-entry count. A permitted procedure is not proof that a ticket update committed; a successful ticket update is not permission to use a procedure.

Download fde-data-foundations.py and keep it beside your working copy. Use Python 3.11 or newer with its standard-library SQLite module. If a distributor omitted SQLite, use a Python installation that includes it. No package installation is required.

python3 --version
python3 -c "import sqlite3; print(sqlite3.sqlite_version)"
python3 fde-data-foundations.py compare
python3 fde-data-foundations.py record-update
python3 fde-data-foundations.py record-rollback
python3 fde-data-foundations.py source-deleted

The program actually writes and reads a JSON file in its own temporary directory, executes SQLite queries and commits or rolls back a transaction. The directory is removed after each case. The database is in memory. It makes no network requests and does not touch your documents, run a model, verify an identity or provision vendor storage.

The search copy is a real SQLite row built from the source. LIKE performs a small text-pattern lookup; it is not vector search, semantic ranking or a production search index. Every baseline case finds one candidate. Only current-copy, record-update and record-rollback admit one context entry. The other four admit none. All cases report zero model calls.

In source-deleted, compare source_exists: false, index_rows: 1 and the empty context list. The copy really remains in this local database until cleanup. That is why removing a source alone cannot establish that all derived copies are gone.

The ticket lookup is short enough to read as a sentence:

SELECT ticket_id, status, procedure_id
FROM tickets
WHERE ticket_id = ? AND scope = ?

SELECT chooses columns, FROM names the table and WHERE filters rows. The question marks are parameter placeholders; Python passes values separately from SQL text. Both ticket identity and the supplied trusted scope constrain this query. Missing or invisible data returns no row, not a guessed ticket status. This one path does not establish complete tenant isolation.

The search query uses the ticket’s procedure ID, the derived row’s scope and the query text. Before admitting the result, inspect loads the current source and checks existence, current scope, source ID, revision and content. A stale scope in the copy cannot overrule the source’s current restriction. Comparing full text works for this tiny whole-document example; real chunk pipelines need their own trusted provenance and integrity design.

update_ticket executes BEGIN, changes the ticket and inserts an event before COMMIT. The success case leaves a resolved ticket and one event. The failure case raises an exception after the update but before the event insert, then explicitly executes ROLLBACK. The query afterwards sees an open ticket and no event. The failure is deliberately injected; the database operations and rollback are real.

The code never writes P-17 during that ticket transaction. Its unchanged content is a separate observation, not proof that SQLite can undo object writes. Likewise, an event row is an ordinary row here, not a tamper-proof audit log. This exercise does not test process crashes, disk durability, concurrent changes or a cloud service’s transaction behavior.

Make two separate copies of the Python file. Keep the original and its output.

Experiment A: a search miss is not a missing source. Replace QUERY = "review" with QUERY = "renewal". Run current-copy in that copy. Expect matching_candidates: 0, evidence_reason: "no-search-match" and empty context. The source still exists and index_rows is still 1. Explain why “we found nothing using this query” is different from “there is no procedure.” Do not change the document merely to make a test pass; propose a fresh query and a new test case.

Experiment B: inspect state instead of trusting the status label. In a separate disposable copy, deliberately replace db.execute("ROLLBACK") with db.execute("COMMIT"). Run record-rollback. The ticket is now resolved with zero event rows. The returned transaction label still says rolled-back because you have broken the implementation without changing its label. Read the actual rows: the operation left partial business state. Restore the original file afterwards; this is a local fault experiment, not a recovery recommendation.

Write why a receiving engineer needs observed data as well as an application message. Propose a check that compares the ticket and event after failure. Do not infer that a correct local rollback covers an external object store, a send operation or recovery after power loss.

These examples identify roles, not interchangeable APIs or a complete architecture. Choose a search service or database index from the workload’s query, update, access and operational requirements. Azure AI Search is one example of a separate search service. The local SQLite exercise is not a provisioned version of any of these products.

Prepare one provider-specific data map: source owner, location, read identity, revision scheme, update trigger, derived copies, deletion path, retention rules and an observation to verify each boundary. An object service’s consistency guarantee does not make your separately maintained search copy current. Handle indexing failures and reconcile changes deliberately. Cache invalidation, revocation races, backups and retention remain separate operating concerns.

Continue with the existing protected lessons:

• Google Cloud data lesson
• AWS data lesson
• Azure data lesson

You need existing training access for those lessons. This guide and its local exercise stay free. For the next conceptual step, open RAG: from documents to model context in the agent engineering course. Retrieval-augmented generation connects eligible evidence to a model call; it does not replace these storage responsibilities.

Keep the seven predictions and outputs, both changed-copy outputs and a diagram identifying which store owns each fact. Add one update and deletion route for each copy. Ask a colleague to explain the rollback case and the search miss without your narration.

A useful handover says, “T-104’s status is in the ticket table; P-17 owns the procedure; search supplies candidates checked against the current source. Here is the observed failure behavior, and here are the cloud and concurrency checks we have not performed.” It does not claim that a diagram or a local exercise certifies a deployment.

For the implementation concepts, see Python SQLite and SQLite transactions, checked on 21 October 2026. Object and search references checked the same day: Cloud Storage, Amazon S3 and Azure AI Search indexes. Related service references: Cloud SQL, Amazon RDS, Blob Storage and Azure SQL Database.