Truss model stuck on Model is not ready - out of memory

TL;DR

That error message is misleading here. From what you are seeing, my best guess is the model is running out of memory and crashing, so it never finishes deploying.

Steps

1. That error message is misleading here. From what you are seeing, my best guess is the model is running out of memory and crashing, so it never finishes deploying. Open the metrics tab on your model and look at CPU memory usage, you will probably see it maxed out. Move to a larger instance with more RAM and it should deploy fine. 16GB worked for this exact model, though you may not need quite that much. Source: https://github.com/basetenlabs/truss/issues/751

When to use

You are seeing this: I deployed the sentence-transformers example from the Truss 101 tutorial, but calling truss predict just returns an error saying the model is not ready, it is still building or deploying. Use this skill when you run into "Truss model stuck on Model is not ready - out of memory".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Sources

• https://github.com/basetenlabs/truss/issues/751

In the thread https://vectle.com/threads/thr_prnXotnyhDMNP0x3cACuCw, the question was: I deployed the sentence-transformers example from the Truss 101 tutorial, but calling truss predict just returns an error saying the model is not ready, it is still building or deploying. It never becomes ready no matter how long I wait. The build itself reported no errors, so I am not sure what is stuck. The verified resolution was: That error message is misleading here. From what you are seeing, my best guess is the model is running out of memory and crashing, so it never finishes deploying. Open the metrics tab on your model and look at CPU memory usage, you will probably see it maxed out. Move to a larger instance with more RAM and it should deploy fine. 16GB worked for this exact model, though you may not need quite that much. Source: https://github.com/basetenlabs/truss/issues/751