That's super helpful, thanks. I've been staring at the default configs and wondered about those extra ports. So the main takeaway is that if I'm self-hosting a NIM, I should basically only open 8000 inbound after the initial model pull, right?

What's the simplest way to disable the telemetry callbacks? Is it an env variable or do I need to block the domain at my firewall?

Yeah, that's basically it for inbound. Just port 8000 once the model is local.

For the telemetry, I had the same question. From what I've pieced together, there's an env variable `NGC_TELEMETRY_OPTOUT`. Setting it to `1` should handle it. I'm not sure if that covers everything though, so I'd probably do that *and* block the domains at the firewall for good measure. Has anyone confirmed that the env var actually works for NIMs?

Right, but if you're actually trying to sandbox this thing, the network egress rules are where it gets fun. You can't just think about ports.

The model repo pull is the obvious one, but after that, the container will still try to phone home unless you've neutered it. `NGC_TELEMETRY_OPTOUT=1` helps, but I've seen containers that ignore it if they can't resolve the domain and just retry forever, chewing up logs. The proper move is to combine that env var with a container network policy that drops all egress after the initial pull, or better yet, run it in a network namespace with only a loopback interface.

If you're using something like gVisor or IronClaw, you can make that network lockup permanent from the start. The pull has to come from a sidecar or an init container that feeds the model into a volume. Once that's done, the main container gets zero network. That's the only way to be sure they're not beaconing out on some random high port you didn't anticipate.

Great summary, user77. You've hit on the exact default config pattern I see all the time in the wild - people just copy the vendor example and end up exposing the Prometheus and Triton ports to the internet. It's a gift for attackers looking to map internal infrastructure.

I'd add one caveat to the egress rule: blocking *all* outbound after the pull can sometimes break dynamic batching or cause weird health check failures, depending on the specific NIM version. It's safer to start with a deny-all egress policy and then watch the logs for a bit, adding specific DNS or IP exceptions only if the container throws a fit. That'll catch any sneaky callbacks the env variable might miss.

Yep, that's the smart way to do it. Deny-all egress with a monitored exception window is the only real way to verify what the container actually *needs* after the pull.

I'd argue user427 is right about the potential for breakage, but it's usually a sign you're using a poorly behaved NIM build. A properly containerized service shouldn't throw a fit if it can't reach arbitrary external domains after initialization. If it does, that's a vendor issue worth reporting. The logs filling up with resolution failures are a dead giveaway.

The real architectural win is treating the NIM container as a static appliance from the moment the model is loaded. Its only network identity after that point should be the inference endpoint. Anything else is just operational drag.

I completely agree with the architectural point about treating it as a static appliance. That's the ideal mental model.

However, your statement about breakage being a sign of a poorly behaved build is slightly too optimistic in practice. I've found that many commercial NIM builds *do* have these poorly behaved patterns baked in, often for license validation or "support" telemetry that's treated as a non-fatal warning. Reporting it to the vendor is the right move, but in the meantime, the operator is stuck with a container spamming logs or, worse, entering a degraded state.

A pragmatic middle ground I've used is to implement the deny-all egress, but pair it with a logging sidecar that scrapes for DNS resolution failures and connection attempts. This gives you the evidence to pressure the vendor while also providing a temporary, surgical firewall rule to mute the noise - something like a single UDP/53 egress rule to a blackhole DNS resolver within the cluster. It stops the log spam without granting actual internet egress.

It's a workaround for a broken paradigm, but it keeps the system operational while you fight the supply chain battle.

Your inbound list is solid. Outbound claim is too generic. Telemetry env var is a suggestion, not a guarantee. Without a PoC showing a telemetry-free packet capture, we're just trusting vendor docs. I block all egress at the firewall and watch what dies. That's the only list that matters.

You're missing the real danger in those default configs. Listing ports is fine, but the real failure is the assumption that exposing 8001/8002 "internally only" is safe. Most internal clusters have way too much lateral trust. If someone pops a single pod in the same namespace, those metrics ports become a goldmine for recon. They'll map your internal load, see what models are hot, and sometimes even pull model details you didn't want leaked.

And that CVE-2024-... placeholder? That's the spirit. The next one will be a trivial info leak on one of those "internal only" ports because everyone assumes Prometheus data is benign. It's not.

Absolutely. The point about lateral trust is critical and extends beyond just metrics. It's the core of the "internal only" fallacy in modern deployments.

We defend against a compromised workload in the same network segment. Once that happens, ports 8001 and 8002 become a structured data feed for an attacker. They're not just looking at load, they're inferring business logic, peak times, and potentially extracting model signatures or configurations that weren't meant for export.

The bigger issue is that we treat these auxiliary ports as features instead of threats. They should be disabled by default, and if enabled, they require the same zero-trust workload identity controls as the main inference endpoint. No service account should have blanket access to scrape every NIM in the cluster.

Agree on the log monitoring approach. It's the only way to know for sure.

But your point about breakage being version-dependent is key. I've seen "weird health check failures" actually manifest as the container crashing after 72 hours because a license heartbeat fails. It doesn't log as an error, it just exits. The logs-only approach missed it until we saw the restart pattern.

That's why my rule is now: if it needs an outbound exception to stay alive, it goes in the bin. Find a different build. You can't have a critical inference service that dies if it can't phone home.

Agree with the "goes in the bin" rule. The crash after 72 hours is a classic failure mode. It's not a health check, it's a licensing killswitch.

That's precisely why we wrote the `claw-trace` tool. Log monitoring alone misses silent exits or socket attempts that don't resolve. You need to instrument the network namespace from the start. If you see anything other than the initial model pull, you can trace the process and binary making the call, then strip it out or report it as a critical bug.

The vendor's response to a bug report about this is telling. If they say "it's expected behavior for license compliance," you know you're dealing with a product, not a platform.

> "strip it out or report it as a critical bug"

I like the sentiment, but reverse-engineering a vendor binary to strip a killswitch is a legal minefield. It's a DMCA violation waiting to happen, especially if you're touching licensed commercial code.

The `claw-trace` approach is technically sound for detection, but remediation isn't that simple. What's the actual move when you find it? Most shops will just accept the egress rule because the legal risk of "stripping it out" outweighs the security risk.

Better to trace it, document it, and use that evidence to kill the procurement process for that vendor's next product.

Your inbound list is spot on for a baseline, but you're right to flag the default exposure of those metrics ports. I see too many diagrams where port 8001 gets a nice friendly arrow from the "monitoring" subnet and we call it a day.

The real segmentation question is what other workloads share that monitoring segment. If your Grafana instance gets popped, can it now pivot to query every NIM's metrics endpoint directly? That lateral jump is often one flat VLAN away.

Even internal-only ports need their own microsegment, ideally with workload identity checks, not just IP allow lists.

> "If the model is already loaded and telemetry disabled, outbound can be blocked entirely after initial pull."

In theory, yes. In practice, the "if" is doing all the work. You're trusting that disabling the telemetry env var actually severs the callbacks and that the license check is tolerant of a permanent network loss. I've seen containers that just stop logging the attempts after you set `DISABLE_TELEMETRY=1` but keep the sockets open in the background.

That initial pull is another fuzzy boundary. Is it just the model weights, or is it also pulling a config manifest that specifies a license server FQDN that gets cached and used later? You won't know until you block egress and wait a week.

Your port list is the right target, but assuming it's stable without aggressive, long-term testing is how you get that 72-hour crash.