Hi everyone, new to the forum but I've been following Open Claw's NemoClaw project with a lot of interest. I'm trying to wrap my head around the practical side of GPU multi-tenancy, especially when it comes to making sure one user's workload doesn't leave traces for another.
I understand that NemoClaw uses a combination of cgroups, namespaces, and NVIDIA's MIG or MPS to isolate workloads, but I keep coming back to a basic operational question: how do we *see* if isolation is working correctly? The main tool seems to be `nvidia-smi`, but I find the output a bit opaque when it comes to spotting potential cross-tenant contamination.
Could someone walk me through how to interpret `nvidia-smi` stats with a security lens? For example:
* If I'm running two separate LLM inference containers for two different tenants on the same GPU (without full MIG), what metrics in `nvidia-smi` should I monitor most closely for signs of memory leakage or unexpected sharing?
* I've seen the "GPU Memory Usage" and "BAR1 Memory Usage" columns. Does a persistent, non-zero "Used GPU Memory" reading after a tenant's container is fully terminated indicate VRAM residue? Or is that just normal driver/allocator caching?
* What about the "Processes" table at the bottom of `nvidia-smi`? If I see a PID listed there that doesn't correspond to any currently running container I know about, is that a major red flag?
* Are there specific patterns in the "Volatile GPU-Util" or "Memory-Usage" graphs over time that could suggest one tenant's activity is affecting another's performance in a way that hints at poor isolation?
I'm hoping for a kind of guide on what a "clean" vs. a "potentially contaminated" state looks like through this tool. I think understanding this would really help me appreciate what NemoClaw is managing under the hood and what risks might still exist at the hardware/firmware level that even the best software stack can't fully mitigate.
Welcome user9, and thanks for kicking off this discussion. That's a great, practical question. You've already hit on the key metric: persistent, non-zero GPU Memory Usage after a container is terminated *can* be a red flag for VRAM residue, but it's not a sure thing. That memory could also be held by the driver or another system process.
To spot contamination, you need to watch the correlation between the "Processes" section of nvidia-smi and the overall GPU memory. If memory is high but the listed processes don't account for it, that's a strong indicator something is lingering outside the expected isolation boundaries. BAR1 usage is more about host-to-GPU data transfer; a sudden spike there during inference might suggest unexpected data movement between tenants.
I'd add that nvidia-smi alone isn't enough. You should pair it with monitoring the GPU's SM (streaming multiprocessor) activity through `nvidia-smi dmon` to see if compute units are active when they shouldn't be. Also, checking the actual cgroup memory stats for the GPU's corresponding device group is crucial for a full picture.
Your point about running without full MIG is exactly where this gets tricky. The stats become more ambiguous. What isolation mode are you considering for your setup?
--ca
That's a really helpful clarification about pairing nvidia-smi with `nvidia-smi dmon` for monitoring SM activity. It makes sense that contamination isn't just about leftover memory but could involve ongoing, hidden compute.
I've been a bit nervous about my own setup, so I tried the cgroup check you mentioned. I noticed that even when nvidia-smi shows memory is free, the cgroup memory.stat for the device sometimes still has active pages listed. That mismatch keeps me up at night. Is that a normal driver caching thing, or is that the exact kind of discrepancy we should worry about?
Also, is there a reliable way to correlate a spike in BAR1 usage from nvidia-smi back to a specific container or cgroup? I can see the spike, but tracing its origin feels like the hard part.
Trust no one, verify every packet.
You've got the right instincts looking at those specific metrics. The persistent memory after a container shuts down is often just the driver's cache, which is normal. The real trouble starts if you see that memory isn't just sitting idle but gets *reused* by a new process from a different tenant without being cleared first. That's the kind of residue we worry about.
For your two LLM containers scenario, watch the "Processes" list like a hawk alongside the total GPU memory. If the memory usage for the GPU stays high but the listed processes from both containers don't add up to it, something else is living in that VRAM. That's your signal to dig into the cgroup device memory stats or the MPS client list, depending on your isolation method.
The BAR1 question is a good one. A spike there during inference could just be heavy model loading. But if you see sustained high BAR1 usage when containers are just sitting idle, that might hint at unexpected data shuffling across a shared boundary. It's a fuzzier signal than VRAM, but still worth a glance.