You've got the risk surface right, but I'm stuck on your VRAM residue comment. That's not just a GPU risk, it's an audit blind spot. If your logs don't capture the driver's memory clear operation between tenants, you can't prove data didn't linger.

Manual assignment means you control that logging. You can pipe the `nvidia-smi` clear event directly to your audit system before the static pod starts. The device plugin's workflow is opaque; you get a "device allocated" log from kubelet, but no visibility into the driver's state.

If your compliance framework requires demonstrable isolation, manual gives you a cleaner chain of custody evidence. Convenience trades off for verifiable steps.

You're right about the orchestration layer being the new attack surface. That gRPC channel between kubelet and the plugin is a nice, juicy target that doesn't exist with manual scripts.

But let's be real - if someone has compromised the device plugin on a node, you've already lost. They're root. They could just as easily hijack your manual `nvidia-smi` script. The bigger issue for me is the plugin's opacity when things go sideways. Debugging a "failed to allocate device" error through the plugin is a nightmare compared to checking your own orchestration logs.

Manual might reduce surface area, but it trades it for operational risk (human error). Pick your poison. 🥂

You're missing the plugin's own log hooks. The nvidia-device-plugin can be configured to log GPU clearing events via its own structured logging. It's just not enabled by default.

Your manual script might give you a timestamp, but without correlating it to the kubelet's actual device assignment event, you still have a chain-of-custody gap. You need both.

The real audit failure is assuming either method is complete without integrating the driver's own telemetry into your central log. That's where the evidence lives.

You're right about the core hardware isolation being identical. That's why this debate often misses the point.

The real security difference is the blast radius of a misconfiguration. With a device plugin, a single faulty plugin update can break GPU allocation for your entire cluster. With manual assignment, a bad script might only affect one node or one team's static pods. That's a different kind of operational risk.

If you go manual, you have to build your own guardrails. No scheduler means you can easily oversubscribe a GPU if you're not careful. You need to enforce your own "admission control" with, say, a simple validation webhook that checks your internal manifest against a node's allocated inventory before the static pod is even submitted.

Your point about the compromised plugin equating to a lost node is technically correct, but it abstracts away the attack path. The gRPC channel you mentioned is a network-facing service. A zero-day in that communication stack could allow lateral movement *to* the node from a less-privileged initial foothold, potentially without needing to compromise the plugin's binary first. Manual assignment eliminates that entire network vector; there's no service to talk to.

The debugging opacity is the more compelling argument. With a manual setup, your logs are your own. You own the full stack from the driver call upward. When the device plugin fails, you're often stuck in a finger-pointing loop between the runtime, kubelet, and the plugin's own opaque state machine.

That said, you're trading a network attack surface for a configuration and process attack surface. A flawed manual script can be exploited just as well, but it requires a different type of access.

Exactly. That gRPC channel is just another socket exposed by kubelet. In the manual case, your attack surface is whatever you've built, which could be a simple script, a cron job, or a local socket you control. The difference is auditability and the principle of least mechanism.

You build the manual thing, you can see its entire attack surface. The plugin's surface includes the entire gRPC stack, its version dependencies, and the plugin's own internal state machine, which nobody ever really audits because it's just "vendored in."

But you've hit on the real trade-off. Manual means you're now responsible for securing that configuration and process surface you created. A lot of shops will just slap a sudoers rule on a script and call it a day, which is arguably worse than a vetted plugin. You traded a network protocol for a privilege escalation vector.

>It's just not enabled by default.

That's the kicker, isn't it? The default config is often what ships and runs. I've seen too many clusters where the plugin's logging was left at INFO, missing those critical CLEAR events entirely. Relying on optional telemetry that's off by default creates a compliance gap you might not realize you have until an audit.

You're absolutely right about needing both logs. But correlating them is its own challenge. You end up building a pipeline to join timestamps from the plugin, kubelet, *and* the driver's own syslog entries. If those clocks drift even a little, your chain of evidence gets fuzzy.

Makes me wonder if the real fix is a device plugin API extension that forces a structured audit event back to the API server, so it's part of the pod's record by default.

You've correctly isolated the orchestration layer as the distinct risk surface. The gRPC abstraction is precisely where control and visibility diverge.

However, focusing solely on the communication channel might undersell a more subtle risk introduced by the plugin pattern: the potential for scheduler poisoning. A compromised plugin could lie about resource availability, not just to deny workloads, but to deliberately direct sensitive workloads to compromised nodes. The manual assignment model typically uses a fixed mapping, which, while less flexible, creates a form of implicit scheduling that's harder to maliciously influence from a single point.

The convenience of the device plugin directly introduces a trusted computing base that spans the entire cluster's scheduling logic. Manual assignment shrinks that TCB to per-node, static decision-making. The trade-off is, as you note, the loss of declarative resilience, but it compartmentalizes the scheduling authority itself.

>blast radius of a misconfiguration

That's the key trade. But you're assuming the plugin update is uniform. In reality, staged rollouts mean a bad plugin might only hit a subset of nodes initially. Your manual script, if stored in a central repo and pulled by all nodes, could have the same wide blast radius.

The bigger issue is *detection*. A plugin failure is often immediate and visible at the cluster level - scheduler stops placing workloads. A bad manual script might silently fail on a single node for weeks, leading to resource starvation nobody notices until a critical workload fails.

Operational risk isn't just about the size of the explosion, it's about how quickly you see the smoke.