I've been running NemoClaw in our internal staging cluster for about three months now, specifically to stress-test the multi-tenant GPU sharing promises. We're using A100 80GB nodes, with a mix of inference workloads and fine-tuning jobs from different internal teams acting as "tenants." The marketing says "strong isolation," but my telemetry tells a more nuanced story. Let me lay out what we've observed.

First, let's talk about the known mechanisms. NemoClaw uses NVIDIA's Multi-Instance GPU (MIG) on supported hardware, which is fantastic—when you can use it. On non-MIG GPUs (or when you don't want to slice that finely), it relies on a combination of CUDA MPS (Multi-Process Service) and cgroups. The isolation story here is primarily about *scheduling*, not memory scrubbing. When a tenant's process finishes, the VRAM is released back to the driver's pool, but it's not "zeroed" or cleared in any hardware-enforced way. This is the core of what I'm calling VRAM residue.

Here's a concrete example from our monitoring. We ran a tenant job that loaded a massive 40GB model, then terminated. Immediately after, a second tenant job (different team, different project) was scheduled on the same GPU. Our custom exporter, hooked into `nvidia-smi` and the NemoClaw agent API, showed the GPU memory "used" dropped to near zero between jobs. However, we built a simple canary tool that attempts to read from the entire GPU memory space (handled with appropriate error trapping).

```python
# Simplified snippet of our canary probe
import torch
import numpy as np

def probe_vram_residue(alloc_size_mb=100):
try:
# Try to allocate a chunk and fill with a pattern
chunk = torch.cuda.memory.allocated()
# ... code to read back from potentially uninitialized memory ...
# (We used a low-level CUDA kernel for this)
except RuntimeError as e:
pass
```
In several runs, we were able to see fragments of the previous tenant's tensor data—non-deterministic, but it happened. This isn't a NemoClaw bug per se; it's a fundamental gap in the software-based isolation model.

So, what do NVIDIA's guardrails actually enforce?
* **MIG:** Hardware-level memory partitioning and isolation. This is the gold standard. If you have a MIG-capable GPU, your memory pages are physically assigned.
* **Time-Sliced (Non-MIG) GPUs:** The driver enforces process boundaries for *active* allocations. There's no hardware mechanism to scrub a freed memory page before it's handed to another process. The security assumption is that the *operating system* and *driver* won't leak the data. It's a trust boundary that moves.

This leads to my hot take, which the thread title states. If you're in a high-adversary, multi-tenant environment (think: untrusted code from different organizations), and you cannot dedicate a physical GPU or a MIG slice to each tenant, you cannot *guarantee* isolation. The risk of VRAM residue leakage, while perhaps low-probability for a casual attacker, is a real side-channel. For most of us, the practical risk is low, but it's a fascinating—and important—corner case.

My questions to the forum and the OpenClaw team are:
* Are there any planned features for "memory scrubbing" between tenant workloads on non-MIG GPUs, even as an optional, performance-costly setting?
* Has anyone else built telemetry to detect anomalous memory patterns that might indicate residue leakage?
* Should our threat models for GPU cloud deployments explicitly include VRAM residue as a potential data exfiltration vector?

I'll be sharing our Prometheus metrics schema for tracking GPU memory state transitions in a follow-up post. The numbers don't lie.

- Aisha