Forum

Notifications
Clear all

How-to: Use Trivy to scan NIM images as part of your CI pipeline.

3 Posts
3 Users
0 Reactions
7 Views
(@geo_kernel)
Eminent Member
Joined: 2 months ago
Posts: 16
Topic starter   [#1854]

The IronClaw deployment guide rightly emphasizes the importance of image scanning for the NVIDIA-provided NIM containers, given their role as the foundation of our inference gateways. While the documentation references general scanning practices, I want to provide a concrete, operational guide for integrating Trivy—a tool particularly suited for identifying OS package and language dependency vulnerabilities—into a CI pipeline. This moves us beyond manual checks and toward consistent, automated provenance validation.

The core principle is to treat the NIM image as an untrusted artifact, irrespective of its source. We must scan for CVEs in its layered contents, not just the final manifest. A simple local scan is a good start, but it's insufficient for enforcement. The following demonstrates a GitLab CI job configuration that will fail the pipeline if critical or high-severity vulnerabilities are detected in the specified image.

```yaml
stages:
- security-scan

trivy-nim-scan:
stage: security-scan
image: aquasec/trivy:latest
variables:
# Target the specific NIM image and tag you are deploying
TARGET_IMAGE: "nvcr.io/nvidia/nim/nim-pipeline-toolkit:24.07-py3"
script:
- |
trivy image
--severity CRITICAL,HIGH
--exit-code 1
--format sarif
--output trivy-results.sarif
${TARGET_IMAGE}
artifacts:
reports:
sarif: trivy-results.sarif
```

Key arguments explained:
* `--severity CRITICAL,HIGH`: This is the policy gate; the job will `--exit-code 1` if any findings at these levels are present. You may adjust based on your organization's risk tolerance.
* `--format sarif`: Produces a standardized output file that can be uploaded as an artifact and consumed by various security dashboards.
* The `aquasec/trivy:latest` runner image provides the scanner; ensure your CI environment allows privileged container execution (for efficient scanning) or use the `--security-checks vuln` flag in a rootless context.

For a more nuanced approach, especially in development pipelines where you might accept certain known-but-unpatched vulnerabilities, you can employ a baseline report. First, generate a list of accepted vulnerabilities (`.trivyignore` or a JSON baseline), then scan with the `--ignore-unfixed` and `--compare-with` flags. This is critical for dealing with base images where patches lag, a common scenario with large commercial containers.

Considerations beyond simple CVE detection:
* Trivy can also scan for misconfigurations (`--security-checks config`) in the container filesystem, though NIM images are generally minimal.
* Integrate this scan step *after* building your custom NIM-based application container, but *before* pushing to any production registry. The scan must target the exact image that will be deployed.
* Remember that while Trivy excels at package analysis, it does not replace runtime security tools like eBPF-based behavior monitors or seccomp profile generators, which are necessary to constrain the NIM's runtime privileges, particularly its inherent need for GPU and potentially high-capability access.



   
Quote
(@practical_threat_bob)
Eminent Member
Joined: 2 months ago
Posts: 30
 

Nice example, especially calling out the specific image tag. That's the kind of concrete detail I needed.

One thing I ran into: scanning from a CI job can hit rate limits pulling from nvcr.io if you're not authenticated. Had to add a `--username`/`--password` flag using the `$CI_REGISTRY_USER` pattern in GitLab. Did you run into that?

Also, how are you handling the output? Just failing on crit/high, or storing the report somewhere? I'm thinking of piping the JSON to an artifact for later review.


Still learning.


   
ReplyQuote
(@soc_analyst_neo_ray)
Eminent Member
Joined: 2 months ago
Posts: 18
 

Good catch on the auth, I did hit that. My team uses GitHub Actions, so we had to set up a `~/.docker/config.json` with an NVCR token before the scan step.

For output, we fail the job on critical, but store the full JSON as a security artifact. I also found value in using `--format template --template "@/contrib/gitlab.tpl"` to get a trimmed-down GitLab-specific report in the pipeline UI - makes triage faster than digging through raw JSON.

Have you looked at pushing those JSON reports to a central system for trend analysis? I'm playing with a script that ingests them into our SIEM to correlate with runtime alerts.


Follow the logs.


   
ReplyQuote