Data, infrastructure and recovery: an FDE systems workshop

A server provides a service. A virtual machine supplies an operating-system environment. A container packages a process and its dependencies. An image is used to start that process. A relational database keeps records and enforces configured constraints and transactions. An object store keeps named bytes such as documents. A search index prepares a way to find information; it may live inside a database or in a separate service.

Draw the fictional path: caller → API → authorized ticket/procedure read → optional model call → reviewed draft. Mark identity, network connections and every piece of state that must survive a restart. Use the vendor courses to map it to Google Cloud, AWS or Azure.

Retrieval-augmented generation (RAG) supplies external evidence to a model. It is a pattern, not a database or proof of truth. An exact current ticket status may need only a scoped SQL lookup.

Hybrid search combines search signals. Agentic retrieval lets a model influence retrieval steps. A knowledge graph represents entities and relationships; an HNSW index helps approximate vector lookup. They are not interchangeable meanings of “graph.”

Keep a simple baseline and compare exact IDs, paraphrases, no-answer questions, stale sources and denied access. Use the real embedding lab before choosing more machinery. Apply authorization before private passages enter model-based reranking or answer generation.

Download the Python lab. Use Python 3.11 or newer. It uses only the standard library and temporary synthetic databases; no installation, account or network is needed.

python3 fde-systems-recovery.py

Before running, predict whether adding an index changes the returned IDs. Inspect before_plan, after_plan and same_result.

The fixture contains 2,000 rows. Both scoped reads return 143 North/open tickets. The unscoped comparison returns 285 open tickets across both tenants. The supplied tenant value is training context, not an authenticated identity. The index supports lookup; it does not add a missing access predicate.

The observed SQLite plans should move from a scan to an indexed search; exact wording can vary by version. This is not a latency benchmark. In a copy, change only the status parameter from open to closed, rerun and explain why the plan can remain indexed while the result changes.

RTO, Recovery Time Objective, limits the interruption before an agreed useful service returns. RPO, Recovery Point Objective, limits the age of recoverable data relative to the disruption. They are targets, not guaranteed features.

For this fictional scenario, RPO is five minutes and RTO is fifteen. Minute zero means 10:00. T-1 and T-2 exist by then; T-3 commits at 10:03 and T-4 at 10:06. The declared interruption starts at 10:08.

The file restore and row checks actually execute. The business timestamps are supplied scenario values. measured_local_restore_and_check_seconds measures local restore/check work only; it is not production RTO. The missing-ID list reports lost records, while the recovery-point gap reports time. Keep both units visible.

The program rebuilds a search table from restored rows. It cannot recover facts absent from the backup. An index rebuild is not a substitute for authoritative data recovery.

Before running, copy the downloaded file to fde_systems_recovery.py in this disposable folder; Python module imports use underscores. Save this next to the downloaded Python lab as compare.py and run it with Python 3.11+:

from fde_systems_recovery import recovery_experiment

earlier = recovery_experiment(backup_minute=3)
later = recovery_experiment(backup_minute=6)
print(earlier["missing_ids"], earlier["recovery_point_gap_minutes"])
print(later["missing_ids"], later["recovery_point_gap_minutes"])

The earlier case should recover T-3 but miss T-4, with a five-minute gap. The later case has no missing IDs and a two-minute gap. Both hold the supplied acceptance time constant.

Next change only acceptance_minute to 24. The interruption is sixteen minutes, so the RTO target is missed even with the fresher data. Explain why moving the backup cannot repair that delay.

Terraform is an infrastructure-as-code tool. HCL configuration describes intent, a provider communicates with its target, state records managed object bindings, and a plan proposes changes. Apply executes them.

Use an organization-approved Terraform 1.4+ installation within major version 1. Create a new empty disposable folder, download the local HCL fixture into it and rename that file to main.tf. Read it first. Do not run these commands in an existing infrastructure workspace.

terraform version
terraform init -input=false
terraform validate
terraform plan -out=lesson.tfplan
terraform show lesson.tfplan
# After reviewing the local-only plan:
terraform apply lesson.tfplan
terraform output -json
terraform plan

The built-in terraform_data resource stores a fictional contract with replicas = 2. It creates no servers and needs no external provider. The second plan should report no changes. Applying a saved plan does not ask for approval again; inspect and authorize that exact artifact first.

Now change only the input:

terraform plan -var='replicas=3' -out=changed.tfplan
terraform show changed.tfplan
# After reviewing the updated local contract:
terraform apply changed.tfplan
terraform output -json

Finally run terraform plan without that input override. Predict why it proposes returning the contract to the configured default of two. Inspect it before any apply. A command-line variable was not a persistent edit to main.tf.

Keep state and binary plans local. They can contain sensitive values in real projects. This fixture has no credentials, network resources or cloud cleanup. Its temporary local files are the only artifacts to discard when finished.

A backup preserves a recoverable point. A replica follows changes and may copy mistakes too. Failover moves service to a replacement; failback is a later move back. Neither guarantees a particular RPO or RTO without a workload-specific contract and rehearsal.

Promotion must prevent the old owner from continuing authoritative writes. Traffic redirection alone may not do that. Define fencing, retry/idempotency behavior and reconciliation of uncertain writes. For restoration, include compatible software, access, keys, networking and derived indexes alongside data.

The local exercise did not simulate a network partition, promote a remote standby or test power-loss durability. Identify those as separate staging observations if your design depends on them.

Retain the system map, query plans, actual restored IDs, controlled comparisons and local Terraform outputs. Add the authoritative data owner, index refresh/deletion policy, write owner, accepted-service definition, recovery targets and untested dependencies.

The protected eight-lesson systems course adds worked decisions, review criteria and a failover handoff capstone. Course completion does not itself establish field qualification.

Checked 23 October 2026. These support terminology; all exercise scenarios and target values are original teaching examples.

• PostgreSQL: indexes
• SQLite: query plans
• Python: backup API
• Microsoft: hybrid search
• Microsoft GraphRAG: query methods
• HashiCorp: terraform_data
• HashiCorp: planning
• AWS: recovery objectives