# Data, infrastructure and recovery: an FDE systems workshop

An AI service is software running on compute, using data and other services. A model does not replace the database, network, access policy or recovery process. Start with one request and follow what it depends on.

This original workshop uses synthetic tickets. The Python exercise uses real SQLite queries, indexes, backups and restores in temporary folders. The Terraform exercise uses a built-in local state resource. Neither creates cloud infrastructure, accesses customer data or proves production recovery objectives.

## 1. Name the system before selecting the tools

A server provides a service. A virtual machine supplies an operating-system environment. A container packages a process and its dependencies. An image is used to start that process. A relational database keeps records and enforces configured constraints and transactions. An object store keeps named bytes such as documents. A search index prepares a way to find information; it may live inside a database or in a separate service.

Draw the fictional path: caller → API → authorized ticket/procedure read → optional model call → reviewed draft. Mark identity, network connections and every piece of state that must survive a restart. Use the [vendor courses](/training/vendors) to map it to Google Cloud, AWS or Azure.

## 2. Choose retrieval from the question

Retrieval-augmented generation (RAG) supplies external evidence to a model. It is a pattern, not a database or proof of truth. An exact current ticket status may need only a scoped SQL lookup.

| Need | Starting approach | Question to test |
| --- | --- | --- |
| Exact record or identifier | Structured lookup / lexical search | Does the exact permitted fact appear? |
| Paraphrased procedure | Embedding retrieval | Does similarity retrieve support rather than a contradiction? |
| Codes and paraphrases together | Hybrid lexical + vector search | Does combining rankings fix a measured miss? |
| Better order among candidates | Reranking | Was the needed evidence in the candidate set? |
| Relationships or corpus themes | Graph-based retrieval / GraphRAG | Are represented relationships supported and kept current? |
| Adaptive multi-step research | Agentic retrieval | Does each extra search justify its cost and authority? |

Hybrid search combines search signals. Agentic retrieval lets a model influence retrieval steps. A knowledge graph represents entities and relationships; an HNSW index helps approximate vector lookup. They are not interchangeable meanings of “graph.”

Keep a simple baseline and compare exact IDs, paraphrases, no-answer questions, stale sources and denied access. Use the [real embedding lab](/training/guides/embeddings) before choosing more machinery. Apply authorization before private passages enter model-based reranking or answer generation.

Run the [hybrid search proof lab](/training/guides/hybrid-search) to follow actual keyword counts and recorded embedding scores into a computed fused ranking. Change candidate depth, omit a source and inspect the no-answer case before proposing a more complex retrieval system.

Run the [knowledge-graph retrieval lab](/training/guides/graph-retrieval) to trace source-backed dependency paths in actual SQLite. Compare a limited traversal, a withdrawn source, an unverified claim and a cycle before asking a model to explain the result.

## 3. Run an actual database index experiment

Download [the Python lab](/templates/fde-systems-recovery.py). Use Python 3.11 or newer. It uses only the standard library and temporary synthetic databases; no installation, account or network is needed.

```shell
python3 fde-systems-recovery.py
```

Before running, predict whether adding an index changes the returned IDs. Inspect `before_plan`, `after_plan` and `same_result`.

The fixture contains 2,000 rows. Both scoped reads return 143 North/open tickets. The unscoped comparison returns 285 open tickets across both tenants. The supplied tenant value is training context, not an authenticated identity. The index supports lookup; it does not add a missing access predicate.

The observed SQLite plans should move from a scan to an indexed search; exact wording can vary by version. This is not a latency benchmark. In a copy, change only the status parameter from `open` to `closed`, rerun and explain why the plan can remain indexed while the result changes.

## 4. Understand both recovery clocks

RTO, Recovery Time Objective, limits the interruption before an agreed useful service returns. RPO, Recovery Point Objective, limits the age of recoverable data relative to the disruption. They are targets, not guaranteed features.

For this fictional scenario, RPO is five minutes and RTO is fifteen. Minute zero means 10:00. T-1 and T-2 exist by then; T-3 commits at 10:03 and T-4 at 10:06. The declared interruption starts at 10:08.

| Case | Recovery point | Accepted service | Expected result |
| --- | --- | --- | --- |
| Older backup | 10:00 | 10:20 | T-3/T-4 missing; 8-minute gap misses RPO; 12-minute interruption meets RTO |
| Fresher backup | 10:06 | 10:20 | All four recovered; 2-minute gap and 12-minute interruption meet the targets |
| Slow service acceptance | 10:06 | 10:26 | Data target met; 18-minute interruption misses RTO |
| Incompatible schema | 10:06 | Not accepted | Integrity is okay, but the declared application schema check fails |

The file restore and row checks actually execute. The business timestamps are supplied scenario values. `measured_local_restore_and_check_seconds` measures local restore/check work only; it is **not** production RTO. The missing-ID list reports lost records, while the recovery-point gap reports time. Keep both units visible.

The program rebuilds a search table from restored rows. It cannot recover facts absent from the backup. An index rebuild is not a substitute for authoritative data recovery.

## 5. Change one cause at a time

Before running, copy the downloaded file to `fde_systems_recovery.py` in this disposable folder; Python module imports use underscores. Save this next to the downloaded Python lab as `compare.py` and run it with Python 3.11+:

```python
from fde_systems_recovery import recovery_experiment

earlier = recovery_experiment(backup_minute=3)
later = recovery_experiment(backup_minute=6)
print(earlier["missing_ids"], earlier["recovery_point_gap_minutes"])
print(later["missing_ids"], later["recovery_point_gap_minutes"])
```

The earlier case should recover T-3 but miss T-4, with a five-minute gap. The later case has no missing IDs and a two-minute gap. Both hold the supplied acceptance time constant.

Next change only `acceptance_minute` to 24. The interruption is sixteen minutes, so the RTO target is missed even with the fresher data. Explain why moving the backup cannot repair that delay.

## 6. Run Terraform without a cloud account

Terraform is an infrastructure-as-code tool. HCL configuration describes intent, a provider communicates with its target, state records managed object bindings, and a plan proposes changes. Apply executes them.

Use an organization-approved Terraform 1.4+ installation within major version 1. Create a **new empty disposable folder**, download [the local HCL fixture](/templates/fde-terraform-local.tf) into it and rename that file to `main.tf`. Read it first. Do not run these commands in an existing infrastructure workspace.

```shell
terraform version
terraform init -input=false
terraform validate
terraform plan -out=lesson.tfplan
terraform show lesson.tfplan
# After reviewing the local-only plan:
terraform apply lesson.tfplan
terraform output -json
terraform plan
```

The built-in `terraform_data` resource stores a fictional contract with `replicas = 2`. It creates **no servers** and needs no external provider. The second plan should report no changes. Applying a saved plan does not ask for approval again; inspect and authorize that exact artifact first.

Now change only the input:

```shell
terraform plan -var='replicas=3' -out=changed.tfplan
terraform show changed.tfplan
# After reviewing the updated local contract:
terraform apply changed.tfplan
terraform output -json
```

Finally run `terraform plan` without that input override. Predict why it proposes returning the contract to the configured default of two. Inspect it before any apply. A command-line variable was not a persistent edit to `main.tf`.

Keep state and binary plans local. They can contain sensitive values in real projects. This fixture has no credentials, network resources or cloud cleanup. Its temporary local files are the only artifacts to discard when finished.

## 7. Distinguish restore, failover and failback

A backup preserves a recoverable point. A replica follows changes and may copy mistakes too. Failover moves service to a replacement; failback is a later move back. Neither guarantees a particular RPO or RTO without a workload-specific contract and rehearsal.

Promotion must prevent the old owner from continuing authoritative writes. Traffic redirection alone may not do that. Define fencing, retry/idempotency behavior and reconciliation of uncertain writes. For restoration, include compatible software, access, keys, networking and derived indexes alongside data.

The local exercise did not simulate a network partition, promote a remote standby or test power-loss durability. Identify those as separate staging observations if your design depends on them.

## 8. Keep a receiving packet

Retain the system map, query plans, actual restored IDs, controlled comparisons and local Terraform outputs. Add the authoritative data owner, index refresh/deletion policy, write owner, accepted-service definition, recovery targets and untested dependencies.

The protected [eight-lesson systems course](/app/academy?course=production-systems&lesson=systems-101&view=focus) adds worked decisions, review criteria and a failover handoff capstone. Course completion does not itself establish field qualification.

## Primary references

Checked 23 October 2026. These support terminology; all exercise scenarios and target values are original teaching examples.

- [PostgreSQL: indexes](https://www.postgresql.org/docs/current/indexes-intro.html)
- [SQLite: query plans](https://www.sqlite.org/eqp.html)
- [Python: backup API](https://docs.python.org/3/library/sqlite3.html#sqlite3.Connection.backup)
- [Microsoft: hybrid search](https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview)
- [Microsoft GraphRAG: query methods](https://microsoft.github.io/graphrag/query/overview/)
- [HashiCorp: terraform_data](https://developer.hashicorp.com/terraform/language/resources/terraform-data)
- [HashiCorp: planning](https://developer.hashicorp.com/terraform/cli/commands/plan)
- [AWS: recovery objectives](https://aws.amazon.com/what-is/recovery-time-objective/)