We've been discussing GPU isolation at a high level, but I wanted to start a practical thread on driver configuration. A key principle in threat modeling NemoClaw's GPU workloads is reducing the attack surface of the CUDA driver itself. The monolithic `nvidia.ko` driver exposes a vast set of capabilities, many of which are unnecessary for a standard inference tenant.

The goal is to lock down the driver instance available to the container to a minimal, workload-specific subset. This isn't about patching the driver, but about using its built-in modular interfaces and deployment configurations. Here’s a step-by-step approach we've validated in our lab:

* **Identify Required Driver Libraries:** First, profile your typical workload (e.g., using `nvprof` or `strace`). For a standard Llama.cpp-based inference, you'll primarily need the core runtime libraries like `libcuda.so` and `libnvidia-ptxjitcompiler.so`. You often do *not* need the full suite of debugging, profiling, or management libraries.
* **Construct a Minimal Driver Manifest:** Instead of binding in the entire `/usr/lib/x86_64-linux-gnu` NVIDIA directory, create a manifest file listing only the necessary libraries and their dependencies. Use `ldd` on your container's application binary as a starting point.
* **Deploy via Container Mounts:** In your container orchestration spec (e.g., Kubernetes or Docker), mount only the libraries from your manifest, read-only, to a location within the container's filesystem. Combine this with a minimal `LD_LIBRARY_PATH`.
* **Leverage NVIDIA Container Toolkit Selectively:** The `nvidia-container-toolkit` can be configured with a custom config file to limit which driver capabilities and devices are exposed. This is where you can tie the library subset to specific GPU devices.

The main isolation gap here is that you're still relying on the same kernel module. However, this significantly reduces the vectors available from within a compromised container—they can't call APIs that aren't there. It also mitigates some risks from VRAM residue, as complex diagnostic tools that could scan memory won't be present.

Has anyone else tried a similar driver subset approach? I'm particularly interested in experiences with the CUDA Compatibility layers and if they introduce unexpected library dependencies.

- Oli