Issue with Pod Termination in InferenceService using nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2 - No Terminate

TL;DR

What works: NIM InferenceService pods hanging on deletion (no SIGTERM reaching the NIM server, pod only force-deleted after terminationGracePeriodSeconds): the bug was reported to engineering and a fix is in place for NIM 1.12, tested by the maintainer to catch SIGTERM as expected.

Steps

1. What was reported:

Issue NVIDIA/nim-deploy#91 (closed, 10 comments): Currently, I am using the nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2 image to create an InferenceService following the provided guide, and the service is running properly. Pod creation and API calls are working fine, but I am encountering an issue when trying to delete the Pod. It seems that the Terminate command is not being sent to the Nim server when I request the deletion of the InferenceService or Pod. There are no KILL signals in the internal logs either, and the Pod is only forcefully deleted when it reaches the terminationGracePeriodSeconds: 300.

When to use

You are seeing this: What was reported: Issue NVIDIA/nim-deploy#91 (closed, 10 comments): Currently, I am using the nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2 image to create an InferenceService following the provided guide, and the service is running properly. Use this skill when you run into "Issue with Pod Termination in InferenceService using nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2 - No Terminate".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: 1.1.2. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

In the GitHub issue, the reporter asked: Issue with Pod Termination in InferenceService using nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2 - No Terminate.... The working answer that came out of the thread: NIM InferenceService pods hanging on deletion (no SIGTERM reaching the NIM server, pod only force-deleted after terminationGracePeriodSeconds): the bug was reported to engineering and a fix is in place for NIM 1.12, tested by the maintainer to catch SIGTERM as expected. Upgrade to NIM >= 1.12. Source: https://github.com/NVIDIA/nim-deploy/issues/91