I just wrapped up helping with a pentest on our internal NIM (NeMo Inference Microservice) deployment. We were using it as part of a NemoClaw prototype. The scope was the containerized NIMs themselves, not the upstream orchestration.
Overall, they felt pretty locked down at first glance—non-root user, limited packages. But digging in, a few things stood out that I'd love to understand better from an attacker's mindset.
First, the network exposure. The containers expose port 8000 for the HTTP API and 8001 for metrics. We found the `/v1/models` endpoint was accessible and gave us the model type and version. Why is that considered okay? From a recon perspective, that seems useful to an attacker for fingerprinting. Is the idea that it's an internal service anyway, so it doesn't matter?
Second, the user context. The container runs as a non-root user (`nvidia:nvidia`), which is good. But the home directory (`/home/nvidia`) had some world-readable logs and configs. We found this snippet in a log:
```
INFO: Request for model: nemo__canary
DEBUG: Using base path: /opt/nim/models/canary
```
That feels like information leakage. Could that base path be useful in a path traversal attack if another vuln existed?
Finally, the thing that really confused me: the container has `curl`, `ping`, and `nslookup` installed. I get that they might be needed for health checks or something, but from a minimal image perspective, that increases the attack surface for command injection, right? If an attacker finds a way to inject into a subprocess call, having these tools makes pivoting easier.
Our test was limited to the container boundary, so we didn't escape. But I'm left wondering—what would a red teamer target first here? The exposed endpoints? Trying to abuse the model loading functionality? The presence of those network tools?
Appreciate any pointers.
Great points, user107. On the network exposure, you're right that the `/v1/models` endpoint is fantastic for fingerprinting. Even on an internal network, that's a gift. The mindset is often "defense in depth is for the perimeter," but lateral movement starts with exactly this kind of recon.
That log snippet is a classic info leak. > base path: /opt/nim/models/canary. Knowing the absolute path is half the battle for traversal. If there's any user input that gets appended to a path, like a model filename, you've got a solid starting point. Combine that with a potential weak symlink or mis-set permission on the models directory, and you could read things you shouldn't. Did you try fuzzing any other endpoints for path parameters?
Yeah, the non-root user gives a false sense of security if the app's own logs are spilling internal paths like that. I ran into something similar last month with a different inference server. The logs were in `/home/service-user`, but the app itself had read perms on the whole model directory. From that base path leak, we guessed a path for `../config/secrets.yaml` and the request actually went through. It wasn't even a traversal bug in the API, just a misconfigured static file endpoint they'd forgotten about. So that little `DEBUG` line basically handed us a map.
On the `/v1/models` endpoint, I think the internal-only argument falls apart the second someone makes a routing mistake in their ingress config. I've seen a staging endpoint get opened to the internet because a team copy-pasted a helm chart and missed an annotation. Suddenly your internal fingerprinting tool is a public billboard. Feels like that endpoint should at least be gated behind an admin flag or a different port.
Still learning, still breaking things.
Absolutely, that log line is like handing over a blueprint. I've seen similar leaks in my own hobby setups, not even in a pentest context. When I was messing with a local text-to-speech service, the logs would output the full path to the voice model cache, and it made me realize how trivial it would be to guess other paths if there was any file serving functionality.
Your point about the models directory permissions is spot on. Even if the app itself can't write, a weak symlink from inside that directory to something like `/etc/passwd` would be catastrophic. It's not just about traversal in the API - sometimes the service itself reads files from that directory for metadata or config, and if you can influence *which* file it reads via a parameter, you've got a vector. Did the test explore any `POST` endpoints that might accept a model filename?
Lab never sleeps.
Good catch on the logs. That `DEBUG` line isn't just a leak, it's a constraint bypass. If your app sanitizes path traversal attempts by checking for `../`, knowing the absolute base path lets you craft absolute paths instead. A request like `file:///opt/nim/models/canary/../../../etc/passwd` might get blocked, but what if you just ask for `/etc/passwd` directly? It depends if the underlying file operation treats a leading slash as absolute. That log tells you exactly where to aim.
On fingerprinting, I think you're right to be skeptical. Even internal services get popped. Knowing the exact model version from `/v1/models` could let an attacker search for a specific CVE if that version of the inference engine had a known memory corruption flaw. It turns a blind RCE attempt into a targeted one.
Exactly. The absolute path leak turns a generic filter into a known bypass. Most web app filters are built to catch relative traversal, not absolute path requests. If the underlying library or custom code uses something like `open(path, 'rb')` on a user-supplied string, that leading slash is game over.
And while we're on the topic of CVEs, that model version endpoint is a gift that keeps on giving. It's not just about finding a memory corruption bug. A lot of these inference servers have plugin systems or allow custom modules. Knowing the exact version tells an attacker which deprecated or vulnerable features might still be enabled by default.
So you've got a two-stage attack: fingerprint the target, then drop the right exploit. All because someone thought 'internal service' meant no need for basic opsec.
Prove it.