That baseline CPU usage isn't unusual for a system that's actively managing GPU resources and its own internal state, even while idle. The overhead often comes from coordination and polling loops, as you suspected.

You could try profiling with `perf` or `htop` inside the container to see if the cycles are being spent in user space or kernel space. I've found similar issues in other inference servers where the main event loop, even when waiting, was performing a busy-wait on a condition variable due to a suboptimal scheduler hint. It's a classic systems programming footgun.

If you're comfortable with it, you could also try adjusting the process's nice value or cgroup CPU quota to throttle the idle behavior, but that's just masking the symptom. The real fix needs to come from the runtime developers implementing proper low-power idle states.

Yeah, I've seen that pattern on my end too. The 20-30% idle baseline is consistent across deployments for me as well.

> I'm curious if it's related to a constant polling mechanism

It likely is. In my logs, there's a tight loop from the GPU management daemon inside the container, pinging the driver status at a fixed interval. It's not a health check, it's more like a keep-alive for the CUDA context. You can spot it if you trace the syscalls - a regular clock of poll() or select() calls even with zero requests.

I don't think it's a config issue. Seems baked into the runtime to avoid latency spikes on the first inference after an idle period. Makes sense for a service, but the CPU tax is real.

That polling behavior matches what we see in our tracing, but I'd caution against assuming it's purely a CUDA context keep-alive. The overhead often stems from the dependency tree of monitoring and observability libraries bundled into the base image.

We instrumented a similar container and found three distinct periodic tasks contributing to the load:
* The CUDA status check you mentioned (~10ms interval)
* A Prometheus client library scraping internal metrics (default 15s interval, but with a computationally expensive histogram quantile calculation)
* An OpenTelemetry SDK batch processor polling its export queue

Each of these might be justified individually, but their compounded wake-up frequency creates that steady background burn. The fix isn't just about the runtime; it's about auditing the software bill of materials for the container image to identify and disable non-essential background services. Do you know which specific image tag and version you're running? Pinned versions help immensely for this kind of analysis.

Great catch on the observability libraries. That's often the hidden tax. I've had to write Rego policies just to audit container images for exactly this kind of bloat - flagging deployments that include both Prometheus and OTEL clients with aggressive defaults.

> The fix isn't just about the runtime; it's about auditing the software bill of materials

Totally. In my clusters, we enforce a policy that any background task with a periodic wake-up under 30s requires an explicit annotation to pass admission control. It cuts this idle load by more than half. You can get the Prometheus client to chill out by setting `process_start_time_seconds` and disabling quantiles, but you have to know it's there first.

Hey Tina, that idle CPU baseline you're seeing is definitely a known pattern, and you're on the right track looking at the base image. The default `nvidia/nim` image pulls in a whole dependency tree, and in my experience, a significant chunk of that steady burn comes from the built-in observability stack. It's not just the CUDA keep-alive.

I'd suggest running a quick SBOM generation on that exact image tag. You'll likely find multiple metric collection libraries (Prometheus, OTEL) with default configs that fire off periodic tasks. Sometimes they're even competing, each doing their own scrape or flush on different timers. That compounded wake-up frequency adds up to exactly the kind of background load you're describing.

Have you checked if there are any environment variables to tune or disable those non-essential collectors? Sometimes they're tucked away in the runtime config. If not, you might need to build a slimmer derivative image, which is a pain but often the only way to trim that tax. Let us know what you find in your process list!

Yeah, seeing the same thing on my test rig. That 20-30% idle burn tracks with what I'm getting too. I was worried I messed up my setup.

I just started reading the runtime docs last night, and there's a mention of a "background maintenance" process. It sounds like what everyone's describing with the polling. I'm still new to this, but is there a way to confirm it's that specific process? Maybe with `ps aux` inside the container?

Yeah, that background maintenance process is a solid lead. If you can exec into the container, `ps auxf` or `pstree` can show you the process hierarchy. The tricky part is often distinguishing the actual maintenance loop from the metric scraping loops others mentioned.

You could also try running `perf top` inside the container (if it's installed) to see which symbols are actually eating cycles. That's how I confirmed the main culprit in my case was the Prometheus client's histogram bucket updates - even on idle, it was doing a bunch of unnecessary math every few seconds.

Let us know what you find! It's always good to trace this stuff back to a specific component.

Great point about `perf top`. That's been my go-to for untangling these layered overheads. It's especially useful when the maintenance and metric loops are in separate threads of the same PID.

One caveat - the base `nvidia/nim` image I was using didn't have `perf` installed. I had to build a custom image just for profiling, which was a bit of a chore. An easier first step is often checking `/proc//task/*/schedstat` to see which kernel threads are actually getting scheduled most often. That usually points you right at the busy loop.

> the Prometheus client's histogram bucket updates - even on idle, it was doing a bunch of unnecessary math every few seconds.

This! I wrote a tiny Rego snippet for our admission controller that flags containers with Prometheus client libs missing the `process_start_time_seconds` gauge. It's a dead giveaway the defaults are still active. Found three other services with the same idle burn.

If user401's setup allows, setting `process.collectors` to just `[]` in the Prom client config can squash that entirely.

Totally, Tina. I've got the same baseline burn on my old dual-Xeon rack server. It's like having a tiny space heater that never turns off!

The container SBOM idea is spot on. When I dug into mine, it was a combo of the CUDA keep-alive *and* two different metric libraries doing their own thing. You can sometimes quiet the Prometheus chatter by setting `process_start_time_seconds` and disabling quantile calculations, but you have to know the flags.

One thing that helped me was just old-school `top` inside the container, sorted by TIME+. It quickly showed which specific threads had accumulated the most CPU seconds while idle, pointing straight at the guilty loops. Might be a simpler first step before breaking out `perf`.

Yeah, sorting by TIME+ is a classic, effective move. It cuts through the noise of momentary spikes and points right at the persistent background consumers. That's how I first spotted the Prometheus client's aggregator thread sitting at the top of the list after an hour of idle time, despite the main process showing near-zero user CPU.

One caveat I've run into, though, is that some of these metric or maintenance loops run in kernel-heavy patterns - think constant `poll()` or `futex()` waits with tiny bursts of work. In those cases, the TIME+ can still look deceptively low, because most of the CPU cost is in the syscall overhead and context switches, not in userspace computation. That's where pulling `pidstat -t` or looking at `schedstat` for voluntary/involuntary context switches per thread can add the missing piece.

Yeah, you're definitely not an outlier. That 20-30% idle baseline is pretty common with the standard image, exactly like others are seeing.

One quick thing to check before diving too deep: the container's health check. By default, it can be a pretty aggressive `curl` call. Run a `docker inspect` on your container and look at the `Healthcheck` section. I've seen that add a surprising amount of overhead if it's running every few seconds. You can tweak the interval or even disable it for testing to see if that's a chunk of your load.

Once you're past that, the `top` sorted by TIME+ is your friend, like user52 mentioned. It'll point right at the threads that have been busy doing 'nothing'. Let us know what you find

Oh wow, I'm seeing the exact same thing on my setup and I was worried it was just me messing something up. That 20-30% idle CPU is spot on.

I'm still really new to this, so the suggestions here are super helpful. I was just about to start poking around inside the containers, and the `top` sorted by TIME+ tip sounds like a great first step. I probably would've just stared at `docker stats` forever 😅

Quick question for the thread: if it *is* the health check, what's a safe interval to set it to without risking the container getting killed? Or is disabling it for a local test rig okay?