add --max-active-adapters argument

TL;DR

The --max-active-adapters argument for Lorax does exist; if a fresh pull doesn't show it, update your Docker image (there was a Docker build cache issue hiding it). One caveat from the maintainer: it caps the number of concurrent adapters, not their total size, so very large adapters can still blow GPU memory. The longer-term idea is a --max-active-adapter-memory flag that reserves GPU memory instead of counting adapters.

When to use

You are seeing this: One caveat from the maintainer: it caps the number of concurrent adapters, not their total size, so very large adapters can still blow GPU memory. Use this skill when you run into "add --max-active-adapters argument".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Steps

1. The --max-active-adapters argument for Lorax does exist; if a fresh pull doesn't show it, update your Docker image (there was a Docker build cache issue hiding it). One caveat from the maintainer: it caps the number of concurrent adapters, not their total size, so very large adapters can still blow GPU memory. The longer-term idea is a --max-active-adapter-memory flag that reserves GPU memory instead of counting adapters.

Edge cases

• One caveat from the maintainer: it caps the number of concurrent adapters, not their total size, so very large adapters can still blow GPU memory.

In the GitHub issue gh:predibase/lorax#92 (https://github.com/predibase/lorax/issues/92), the problem reported was: Issue predibase/lorax#92 (closed, 8 comments): ### Feature request as discussed here: https://discord.com/channels/1174495433565945916/1176984269558652998/1179487788266172508 ### Motivation If there are to many requests for different adapters you are running out of memory, since the actual system only checks the memory for batch sizes and not concurrent adapters. ### Your contribution I looked at the effort it would take to add this feature and realized it was beyond my ability with Rust The verified resolution was: The --max-active-adapters argument for Lorax does exist; if a fresh pull doesn't show it, update your Docker image (there was a Docker build cache issue hiding it). One caveat from the maintainer: it caps the number of concurrent adapters, not their total size, so very large adapters can still blow GPU memory. The longer-term idea is a --max-active-adapter-memory flag that reserves GPU memory instead of counting adapters.