Let's cut through the marketing slides. When they say "GPU memory isolation," your brain jumps to CPU-side page table isolation (PTI) for Meltdown. It's not that. Not even close. The threat model is different, but the residual risk is substantial for a multi-tenant, agent-hosting platform like ours.
NemoClaw's primary mechanism is temporal isolation via full GPU workload scheduling and VRAM re-allocation, not true concurrent hardware-backed page table isolation per process. When your tenant's inference job completes, the GPU context is torn down and the VRAM is marked free. The next workload gets its own fresh allocation. The isolation relies on the driver and hypervisor to perform a secure wipe or zero-fill on that memory before reassignment. This is where we have to trust the software stack.
The hardware-level guardrails from NVIDIA (like on Ampere/Hopper) are about access control for the *hypervisor* (or host) vs. the *guest* (VM), not between two guest workloads running sequentially in the same VM or container. They prevent a guest from reading/writing host-managed memory, but they don't inherently guarantee that memory released by Guest A is scrubbed before Guest B gets it. That's a software responsibility.
Here's the known gap: VRAM residue. If the driver stack doesn't explicitly zero-fill freed memory buffers, sensitive data (model weights, intermediate activations, prompt data) from a previous tenant can persist. I've seen this in lab conditions with custom CUDA tools. The risk escalates if an adversary can:
* Deploy a malicious agent that allocates VRAM and deliberately doesn't initialize it.
* Use low-level CUDA or PTX instructions to scan memory ranges for non-zero patterns.
* Perform a side-channel attack based on allocation timing if the underlying memory manager has predictable behavior.
For bare-metal agent deployments, our CI/CD hardening must enforce:
* Mandatory use of `CUDA_MEMSET_AT_ALLOC=1` or equivalent driver environment flags to force zeroing.
* Post-job validation hooks that trigger a driver-level "secure-free" call, not just `cudaFree`.
* Agent base images stripped of any diagnostic or memory introspection tools that could be weaponized.
* Scheduling constraints that ensure a physical GPU is dedicated to a trust boundary for a full cycle, not just a single job, forcing a deeper reset.
The real story? GPU PTI doesn't exist in the CPU sense. We're relying on a software chain of custody for memory sanitization, and that chain has weak links in the user-space driver and runtime libraries. Until we get hardware-level tags and crypto erase for every VRAM page, we must treat every GPU as a shared resource that leaks. Our pipelines need to build that assumption into the threat model from the image build stage onward.
build then verify