# NVIDIA infrastructure and operations workbook

Original LockedIn Labs exercise · reviewed 20 October 2026.
Companion: /training/vendors#nvidia.

## Start with the component map

Explain each item before choosing a product:

| Responsibility | Terms to understand | Evidence to request |
| --- | --- | --- |
| Numerical execution | GPU, Tensor Cores, CUDA | Exact GPU and supported kernel/runtime |
| Memory | HBM, weight format, request state | Full workload memory budget |
| Communication | NVLink, NVSwitch, InfiniBand, Ethernet | Actual topology and workload transfer pattern |
| Model | Nemotron, model revision, license | Exact artifact and task evaluation |
| Development | NeMo components | Specific tool and its output |
| Serving | NIM, container runtime, API | Supported model/runtime configuration |
| Scheduling | Kubernetes, Slurm, GPU Operator | Resource request and allocation evidence |
| Operations | DCGM, logs, metrics, errors | Observation window and recovery procedure |

Nemotron is not a server. NeMo is not the GPU. NIM is not customer authorization.
An FDE connects these responsibilities to an owned application.

New to CUDA execution? Begin with the [CPU-to-GPU walkthrough](/training/guides/nvidia-systems#3-inspect-an-environment-without-pretending-to-have-a-gpu)
and the [free CPU-only trace lab](/templates/cuda-execution-lab.py). Explain host,
device, kernel, grid, block and thread, then trace one valid write and one guarded
skip before diagnosing a cluster. No GPU or cloud account is needed for that lab.

## Lab A: budget without buying hardware

Download the existing capacity lab from /training/guides/nvidia-systems and run:

```bash
python3 nvidia-capacity-lab.py memory --concurrency 4
python3 nvidia-capacity-lab.py memory --concurrency 10
python3 nvidia-capacity-lab.py test
```

Record the inputs, reserve, overhead and concurrency assumption. Explain the
refused case. These are synthetic budgeting inputs, not measurements of a
named GPU or Nemotron deployment. Distinguish GB from GiB and total stored
weights from active compute in a mixture-of-experts model.

## Lab B: diagnose one missing GPU

Fictional incident evidence:

- The host lists a supported GPU and installed driver.
- The container starts successfully.
- The application reports zero usable GPUs.
- No container device exposure or scheduler-allocation evidence is provided.

Draw six layers: device → driver → container integration → device discovery →
scheduler allocation → application execution.

Write a table with observation, hypothesis, next check and responsible owner.
Do not infer the failure's cause from the four facts alone. The best next step
is to collect the missing runtime/allocation evidence, not install a different
driver or buy another GPU immediately.

On an authorized host, these are read-only starting observations:

```bash
nvidia-smi --query-gpu=name,driver_version,memory.total --format=csv
```

On an authorized Kubernetes cluster, inspect only the agreed node and namespace:

```bash
kubectl config current-context
kubectl get nodes
kubectl describe node "$LAB_NODE"
kubectl get pods --namespace "$LAB_NAMESPACE"
kubectl describe pod "$LAB_POD" --namespace "$LAB_NAMESPACE"
```

Confirm the cluster/context before inspection. Inspect allocatable resources,
the workload's GPU request, placement and events. Do not copy cluster secrets
or private payloads into evidence. This exercise does not install GPU Operator,
change MIG partitions, restart services or launch a benchmark.

No hardware access? Complete the synthetic diagnosis and mark it as a rehearsal.
That is useful learning, but not a hardware-validation result.

## Lab B2: read the instruments before diagnosing

Telemetry means measurements reported by a system. Think of your laptop's task
manager: an application can hold memory while doing little processing. A GPU's
memory occupancy and activity are separate too. `nvidia-smi` is NVIDIA's System
Management Interface, a command-line utility; it is not a model or a benchmark.

First, explain these fields to a colleague:

| Field | Question it answers | Unit |
| --- | --- | --- |
| `memory.used` / `memory.total` | How much device memory is reported used, compared with its total? | MiB; 1,024 MiB = 1 GiB |
| `utilization.gpu` | During what share of the sample was at least one GPU kernel executing? | Percent of sampled time |
| `utilization.memory` | During what share of the sample was device memory read or written? | Percent of sampled time |

A kernel is a function that executes on the GPU. Neither activity percentage
counts busy cores, and memory activity is not occupied VRAM or achieved bandwidth.
The sample period depends on the device. Activity can overlap: do not add these
percentages together. Device memory here is the discrete GPU's VRAM, not host RAM.

Use these **authored synthetic snapshots**, not measurements from named hardware:

| Scenario | Used / total MiB | GPU activity | Memory activity |
| --- | --- | --- | --- |
| Between requests; model allocations remain | 16,384 / 24,576 | 0% | 0% |
| During a request | 16,384 / 24,576 | 82% | 36% |
| More allocations; no error evidence supplied | 23,552 / 24,576 | 58% | 29% |
| Reporting setup lacks utilization readings | 16,384 / 24,576 | N/A | N/A |

For each row, write: **observation → supported conclusion → missing evidence →
next read-only check**. Answer before opening the review:

1. Why can the first two rows hold the same memory and have different activity?
2. Does 82% mean 82% of CUDA cores were occupied? What does 36% describe?
3. Does the third row prove an out-of-memory error, or safe extra concurrency?
4. Why must N/A remain different from zero?

<details>
<summary>Review your interpretation</summary>

1. Retained allocations do not require continuous execution. The first row
   reports no kernel activity during its sample, not across the whole day.
   Correlate request timestamps with device samples and app logs.
2. Both percentages concern sampled time. They do not establish core occupancy,
   bandwidth saturation, throughput or the application's bottleneck. Compare
   request latency and queueing; use approved profiling for a deeper diagnosis.
3. Neither conclusion follows. Inspect allocation failures, peak workload demand
   and runtime overhead. Total minus used is not guaranteed allocatable headroom;
   actual reservations and memory accounting vary.
4. N/A is unavailable evidence. Verify the exact device, driver and reporting
   mode. MIG partitions supported GPUs into instances; some utilization queries
   are unsupported in that mode. Choose supported monitoring rather than
   converting unknown activity into an idle reading.

</details>

Optional hardware practice, **only on an authorized host**:

```bash
nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu,utilization.memory --format=csv
```

Record the device identity, timestamp, workload and reporting mode alongside the
output. Compare permitted observations before and during an approved workload.
This workbook does not authorize launching a job, changing settings or running a
load test. Without hardware, submit the four-row reasoning table as a rehearsal.

References checked 20 October 2026:
[NVIDIA System Management Interface](https://docs.nvidia.com/deploy/nvidia-smi/index.html),
[NVML utilization definitions](https://docs.nvidia.com/deploy/nvml-api/latest/api/structnvmlUtilization__t.html),
[DCGM monitoring](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html).

## Lab C: performance is a question, not a logo

Suppose a fictional request takes 600 ms retrieving evidence and 150 ms in
generation. Even eliminating generation time leaves 600 ms. First investigate
the dominant measured path.

Now suppose retrieval is fast but throughput falls under concurrency. Propose
measurements that separate queueing, memory pressure, compute and communication.
Use the same model revision and request set. Name the instrumentation and
warm/cold conditions. Do not invent benchmark outputs.

## Lab D: a current procurement decision

Compare a supported Blackwell/Blackwell Ultra system with a supported Vera Rubin
offering only after obtaining evidence for the exact configurations.

Record:

1. GPU SKU, memory capacity/format support and measured workload.
2. Server/rack, CPU, interconnect, storage and network topology.
3. Driver, container, framework and model support matrix.
4. Power, cooling, facility installation and operating ownership.
5. Dated supplier availability, warranty, service and full price.
6. Rent-versus-own assumptions, utilization and the cost of idle capacity.
7. Acceptance test, fallback configuration and the evidence that would change
   your recommendation.

Architecture generation is not a quote or a delivery commitment. Peak vendor
figures are not your measured application performance.

## Choose the right NVIDIA exam route

- **NCA-AIIO:** begin with accelerated-computing and infrastructure foundations.
- **NCP-AII:** add real bring-up, physical infrastructure, installation and
  validation work.
- **NCP-AIO:** add real Base Command Manager, Slurm, Kubernetes, monitoring and
  recovery work. The current vendor exam includes hands-on work.

For each objective, record: introduced here / demonstrated in a real lab /
still needs practice. No completion score in this workbook is a vendor credential.

Official preparation:

- [NCA-AIIO course and blueprint](https://www.nvidia.com/en-us/learn/certification/ai-infrastructure-operations-associate/)
- [NCP-AII workshop and blueprint](https://www.nvidia.com/en-us/learn/certification/ai-infrastructure-professional/)
- [NCP-AIO workshop and blueprint](https://www.nvidia.com/en-us/learn/certification/ai-operations-professional/)
- [GPU Operator documentation](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/overview.html)
- [DCGM documentation](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/getting-started.html)
- [Vera Rubin architecture](https://developer.nvidia.com/blog/nvidia-vera-rubin-pod-seven-chips-five-rack-scale-systems-one-ai-supercomputer/)