Issue running NIM Llama 3.1 8B in air‑gapped environment: corrupted output on chat/completions

AI & Data Science NVIDIA NIM Models
, ,
1

Hi everyone,
I’m trying to run the Llama‑3.1‑8B‑Instruct NIM container (nvcr.io/nim/meta/llama-3.1-8b-instruct:latest) on a fully air‑gapped server, and I’m running into an unexpected issue.

Working setup

On a first server with Internet access, the container runs perfectly.

Goal

Run the same container on a second server without Internet access, following the air‑gapped installation procedure.

Steps performed

1. Transferred the Docker image

I exported the image from the online server and imported it on the offline one using:

  • docker save
  • file transfer
  • docker load

2. First run on the offline server

The container fails to start (expected), since it cannot download the model files.
From the logs I extracted the selected profile:

a4783dadbd6fe8f6af0bd3d03f4928fa4a59dfebbfe2ff8493b5ea847adbe4cd

3. Downloaded the cache on the online server

On the server with Internet access, I started the container and ran:

nim download-to-cache -p <profile>

The resulting cache directory looks like this:

├── blobs
│   ├── 1dc9bbfdb10d89c8968e5a7e7775a3a9
│   ├── ...
│   └── cef2a8a4421c1cedc1f9e5587900a521-9
├── refs
│   ├── fp8-tool-calling
│   └── trtllmapi-pt-runtime-params-l40sx2-latency-fp8-n64ve9jqfq
└── snapshots
    ├── fp8-tool-calling
    │   ├── model-00001-of-00002.safetensors
    │   ├── model-00002-of-00002.safetensors
    │   ├── tokenizer.json
    │   ├── ...
    └── trtllmapi-pt-runtime-params-l40sx2-latency-fp8-n64ve9jqfq
        ├── runtime_params.json
        └── ...

4. Copied the cache to the offline server

I copied the entire .cache directory to the air‑gapped server.

5. Started the container offline

I launched it with:

-e NIM_MODEL_PROFILE="$PROFILE_HASH"
-v $AIR_GAP_NIM_CACHE:/opt/nim/.cache

The container starts correctly and all endpoints respond.

Problem

The /v1/chat/completions and /v1/completions endpoints return corrupted output, consisting of unreadable or nonsensical characters.

What I’ve already checked

  • All cache files are present.
  • checksums.blake3 matches correctly.
  • The profile hash is exactly the one shown in the logs.
  • The same image works fine on the online server.

Question

What could cause corrupted model output in an air‑gapped setup, even when:

  • the Docker image is identical,
  • the correct profile is used,
  • the cache appears complete and checksums are valid?

Is there anything else that needs to be included in the cache besides blobs/, refs/, and snapshots/?
Or is there an additional step required for FP8 / TRT‑LLM profiles?

Thanks in advance for any help!
NIM logs.txt (92.2 KB)

2

Hi gioesposito96

I can see the possible issue in your logs — it’s not the model files or the profile. It’s the torch.compile inductor cache.

Look for these warnings in your logs:

WARNING torch._dynamo hit config.recompile_limit (16)
function: 'torch_dynamo_resume_in_forward_at_462' (attention.py:462)
last reason: self.layer_idx == 15

Your online server had already built and cached the torch.compile/inductor compilation artifacts. Those are stored separately from the NIM model cache — typically at /root/.cache/torch/dynamo and /root/.cache/torch/inductor inside the container. They were not included when you transferred .cache/. On the offline server, torch.compile starts from scratch, hits the recompile limit during FP8 layer warmup, and falls back to eager mode for attention and MLP layers. The FP8 quantization path (quant_algo='FP8' with modelopt) requires the compiled path to execute correctly — in eager fallback it produces corrupted tokens.

Three options to fix this, try in order:

Option 1 — Transfer the torch compile cache (cleanest fix) On your online server, find the torch compile cache inside the running container:

bash

docker exec <container_id> find /root/.cache/torch -type d 2>/dev/null

Copy those directories to the offline server alongside your model cache and mount them the same way.

Option 2 — Raise the recompile limit Add this environment variable to your offline container launch:

bash

-e TORCHDYNAMO_RECOMPILE_LIMIT=64

This gives torch.compile enough room to finish compiling all 32 layers without falling back.

Option 3 — Allow dynamic integer attributes (the hint in your own logs) Your logs literally suggest this fix:

bash

-e TORCH_DYNAMO_ALLOW_UNSPEC_INT_ON_NN_MODULE=1

This stops torch.compile from treating layer_idx as a static integer, eliminating the per-layer recompilation that exhausts the limit.

The model files, checksums, profile hash, and tokenizer are all fine — this is purely a compilation cache transfer issue, not a model cache issue. Option 1 is the proper air-gapped fix; Options 2 and 3 are workarounds that avoid needing to transfer the compile cache.

Hope this helps!!

3

Hi @stuart.rainey,

Thanks a lot for the detailed explanation — it really helped me understand what might be going on.

Unfortunately, none of the proposed solutions seems to fix the issue in my case.

About option 1 (transfer the torch compile cache)

On the working online server, inside the running container, there is no torch compile cache at all.
Both of these paths don’t exist:

/root/.cache/torch/dynamo
/root/.cache/torch/inductor

So there is nothing I can transfer to the offline machine.

About options 2 and 3 (environment variables)

Those environment variables are not supported by NIM containers, and setting them has no effect.
They are not listed in the official NIM environment variable reference:

I also verified inside the container that they are ignored.

Additional details that might matter

Here is the hardware setup for both machines:

  1. Online server (working):

    • Bare‑metal
    • 1× NVIDIA A6000 GPU

  2. Offline server (failing):

    • Virtual machine
    • 2× NVIDIA L40S GPUs passed through to the VM

Do you think this could be related to hardware differences, GPU architecture, or the fact that the offline machine is a VM?

Any additional hints would be greatly appreciated.

4

UPDATE

I solved it simply by using another profile.

INFO 2026-03-30 15:35:52.140 ngc_injector.py:138] Valid profile: a4783dadbd6fe8f6af0bd3d03f4928fa4a59dfebbfe2ff8493b5ea847adbe4cd (tensorrt_llm-l40s-fp8-tp2-pp1-latency-pytorch-26b9:10de-c00261d6b54267143edef861bf8c7f2ea7155dc94b4174a5122d9b38938c864d-2-true) on GPUs [0, 1]
INFO 2026-03-30 15:35:52.140 ngc_injector.py:138] Valid profile: 10e5c5c7b0037e16bc74e9ec32942256bf73fda9902c2697b6cbd8fe94b8b47e (tensorrt_llm-l40s-fp8-tp1-pp1-throughput-pytorch-26b9:10de-9040db2e6d00666a0ce0df1097edb5110e8e821cd43cce89b437c3f7160cb714-1-true) on GPUs [0]
INFO 2026-03-30 15:35:52.140 ngc_injector.py:138] Valid profile: 1b370216386939a1361a599aef31c94c3629debce8dcc1996acc4018b215d88d (tensorrt_llm-l40s-bf16-tp2-pp1-latency-pytorch-26b9:10de-ebff830536032a44128d413f71180f3d32f0088078208695b5778406b9b7d2a7-2-true) on GPUs [0, 1]
INFO 2026-03-30 15:35:52.140 ngc_injector.py:138] Valid profile: 32b2b692f2923cd72531e4404ac79287785b41799215c6464584db3b39afc23c (tensorrt_llm-l40s-bf16-tp1-pp1-throughput-pytorch-26b9:10de-da569859eec41300c84882910e470e7d9fe8f44d51aaeb9094422ed73d1eb99f-1-true) on GPUs [0]
INFO 2026-03-30 15:35:52.140 ngc_injector.py:138] Valid profile: 4c0d8954feb1eaaa7c2df1771a37a2d9304060953d086cb01b80afd9f1e75ecc (vllm-fp8-tp1-pp1-937c59e60e77dc6dae074242d8547504705e9eedebb260f9b298d6ece4f8c67d) on GPUs [0]
INFO 2026-03-30 15:35:52.140 ngc_injector.py:138] Valid profile: d67e23226ec82628f6cd2e38571f1404f67b58cec121c2a128b55f7ad9e79e84 (vllm-bf16-tp2-pp1-669776982fd5ec5017dbe428168a93e2e3557ed6f297d8476a4acf7d91ca5946) on GPUs [0, 1]
INFO 2026-03-30 15:35:52.140 ngc_injector.py:138] Valid profile: 1d53689b7c5bbb0a1b23abd52c2396efc0b42ad460ba74bb57841e16d5fb9aaa (vllm-bf16-tp1-pp1-669776982fd5ec5017dbe428168a93e2e3557ed6f297d8476a4acf7d91ca5946) on GPUs [0]
INFO 2026-03-30 15:35:52.140 ngc_injector.py:291] Selected profile: a4783dadbd6fe8f6af0bd3d03f4928fa4a59dfebbfe2ff8493b5ea847adbe4cd (tensorrt_llm-l40s-fp8-tp2-pp1-latency-pytorch-26b9:10de-c00261d6b54267143edef861bf8c7f2ea7155dc94b4174a5122d9b38938c864d-2-true)

Even if automatically the selected profile was: “tensorrt_llm-l40s-fp8-tp2-pp1-latency-pytorch”,
I managed to get it working using a different profile: “vllm-fp8-tp1-pp1”.

I’m not sure what the problem was with TensorRT but VLLM works perfectly.

I hope this can help someone else in the future!