I've been auditing a few multi-tenant GPU boxes running CUDA workloads. The isolation story is messy, and `/proc/pid/maps` is a good place to start looking for leftover mappings that shouldn't be there.
If a process using CUDA exits uncleanly, or if there's a bug in the runtime/driver, you can sometimes see residual mappings. These are often GPU memory segments. The problem is they might still be accessible to a subsequent tenant process on the same GPU, leading to data leakage.
Here's a practical way to hunt for them. First, get the PID of your CUDA process. Then, look for mappings that aren't typical anonymous or file-backed heap/stack.
```bash
# Look for mappings with no associated file (common for GPU memory)
sudo cat /proc//maps | grep -v ".so" | grep -v ".." | grep -v "/" | head -20
# More targeted: look for the /dev/nvidia* device mappings directly
sudo cat /proc//maps | grep /dev/nvidia
```
Key things I check:
* Mappings to `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm`, etc. These are the driver interfaces.
* Large, anonymous executable mappings (rwxp) in high address ranges. These can be CUDA driver allocations.
* Mappings that persist *after* the main application process has terminated. This is the real red flag.
The hardware-level guardrails (like NVIDIA's MIG or bare-metal partitioning) prevent active cross-tenant access, but they don't automatically scrub VRAM that a previous tenant left mapped into the address space of a *new* process if the driver doesn't clean up properly. This is a software-layer issue.
Has anyone built tooling to baseline a "clean" maps state and diff it after workload termination? I'm considering hooking `fork`/`exec` calls to track mappings from launch to exit, but I'm curious if there's a simpler audit trail method.
Log everything, alert on anomalies.
Great point about the `/dev/nvidia*` mappings. They're a clear signal.
One caveat on the grep filters: stripping out everything with a "/" might also hide interesting, legitimate but unexpected mappings from other device files. I'd suggest a two-pass approach: first look for the known `/dev/nvidia` patterns, then do a broader scan for *any* device file mapping (`/dev/`) in that process to see if something else is sneaking in.
Also, watch for the permissions on those mappings. A leftover mapping that's still `rw` is a much higher risk than one that's just `r`.
- Asia (mod)
Right, the two-pass approach is smart. I usually dump everything with the device major/minor numbers visible first - sometimes you'll catch stray DMA buffers or odd character devices that aren't NVIDIA's.
For the permissions, you're spot on about `rw` being the real concern. I've seen leftover `rw-p` mappings survive a process crash and hang around until the next CUDA context gets created on that GPU, which is a nasty leak vector.
One extra check I toss in is looking for mappings that are *only* in the dead process's maps and not in any live ones, using a quick diff against `/proc/[pid]/maps` for all other PIDs. It's a bit heavy, but it catches ghosts.
if it compiles, ship it
This is super helpful, thanks for breaking it down. I've been trying to wrap my head around securing my own little GPU server for friends.
So, if I'm following, the leftover mappings show up in /proc/[pid]/maps even *after* the pid is gone? That's the part that trips me up. How do I check for those if the process is dead? Do I have to catch it right as it's dying?
Good starter script. I'd add a quick check for the mapping's size alongside permissions. You'll often see large, contiguous blocks (like 256MB+) that scream "GPU memory" more than a random heap fragment.
Also, consider wrapping those greps in a watch loop if you're trying to catch a crash in real time. Something like `while true; do sudo cat /proc//maps | grep /dev/nvidia; sleep 0.5; done`. It's saved me a few headaches when debugging unclean exits.
trivy image --severity HIGH,CRITICAL
No. Once the PID is gone, its /proc entry is gone. That includes maps.
You're looking for mappings in a *different, still-running* process that shouldn't be there. The "leftover" mapping gets inherited or re-attached somehow. Focus on the live tenants, not the dead one.
Check the maps of every process that *could* access the GPU after a crash. That's where you'll find the ghost.
Show me the CVE.