LoRAX: TensorParallelHead object has no attribute base_layer when serving Mixtral with an adapter

Not specified by the original author.

TL;DR

A new Docker image was built and pushed, and the reporter confirmed the fix. This crash is a LoRAX bug in adapter loading for Mixtral's tensor-parallel head, fixed in PR #131.

Steps

1. This crash is a LoRAX bug in adapter loading for Mixtral's tensor-parallel head, fixed in PR #131. Pull a fresh LoRAX Docker image with the fix, since the maintainer confirmed the corrected image was built and pushed. If you pinned an image tag from before the fix, update it. The reporter confirmed the new image resolved the error.

Expected: Pull a fresh LoRAX Docker image with the fix, since the maintainer confirmed the corrected image was built and pushed.

When to use

You are seeing this: Starting LoRAX with mistralai/Mixtral-8x7B-Instruct-v0.1 on two A100s and sending a generate request with a LoRA adapter crashed with 'TensorParallelHead' object has no attribute 'base\layer' while loading the adapter weights. Use this skill when you run into "LoRAX: TensorParallelHead object has no attribute baselayer when serving Mixtral with an adapter".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v0.1. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Asked in the thread (https://vectle.com/threads/thrkCvTKf2IG6ShmJEdh5jow): Starting LoRAX with mistralai/Mixtral-8x7B-Instruct-v0.1 on two A100s and sending a generate request with a LoRA adapter crashed with 'TensorParallelHead' object has no attribute 'base_layer' while loading the adapter weights. Maintainer tgaddair acknowledged it was an oversight on his side and put up PR #131. A new Docker image was built and pushed, and the reporter confirmed the fix. The verified answer: This crash is a LoRAX bug in adapter loading for Mixtral's tensor-parallel head, fixed in PR #131. Pull a fresh LoRAX Docker image with the fix, since the maintainer confirmed the corrected image was built and pushed. If you pinned an image tag from before the fix, update it. The reporter confirmed the new image resolved the error. Source: https://github.com/predibase/lorax/issues/130 Thread: https://vectle.com/threads/thrkCvTKf2IG6ShmJEdh5jow

Not specified by the original author.

TL;DR

Use the transport status only to separate usable from unusable responses. The server does everything right internally and still ships an outage as an empty answer, because the wire shape lets the client collapse the states again: 1. Add a degraded marker outside the body. Set a response header or equivalent metadata that names the missing sources, so proxies, logs and intermediate caches can see degradation without parsing the body.

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

When to use

Use this when a service, endpoint or agent tool combines several independent sources and returns the combined answer to consumers you do not control: other teams, generated SDK clients, mobile apps, spreadsheets, or an agent harness that reads tool output. It assumes the server already models each source outcome as ok, empty, unavailable or skipped with a completeness flag, and keeps the sources that succeeded. It covers the next narrower question: how to lay out the response so that the distinction survives a client that never looks at the status field.

The failure pattern

The server does everything right internally and still ships an outage as an empty answer, because the wire shape lets the client collapse the states again:

1. Empty collection for a failed source. The per-source entry carries status unavailable, but its items field is an empty list because the serializer requires the field. A client that reads only items sees nothing and renders "no results".
2. Default-to-empty deserializers. A generated client marks the items field as required and fills a missing value with an empty list. Even when the server omits the field, the client materializes an empty answer.
3. Partial marked only deep inside. The overall result is flagged partial only within per-source entries. A client that reads the merged top-level list has no single field to check.
4. Wrong transport status. A partial answer is sent as a 5xx. Retry logic and circuit breakers discard the good data and retry everything, so the healthy sources are hit again during the outage. Or the opposite: a total failure is sent as a 200 with an empty merged list.
5. Retrofit onto a bare-list endpoint. A status field is added to an endpoint that used to return a plain list. Old clients ignore the new field and keep treating partial as complete.

Rule

Make the naive reading of the response fail safe. A consumer that reads only the data fields must end up with "unknown", an error or an obviously missing value, never with a confirmed empty answer, whenever a source was unavailable or incomplete. Carry the overall status and the list of missing sources at the top level of the body. Use the transport status only to separate usable from unusable responses.

How to apply

1. Never serialize an empty collection for an unavailable or skipped source. Omit the items field or send null. Reserve an empty list for the empty status only. If the schema language supports variants, model the per-source outcome as a tagged union where only the ok and empty variants carry an items field at all.
2. Put one overall status at the top level. Include overall as complete, partial or failed, a boolean complete flag, and a missing list naming each source that was unavailable, incomplete or skipped with a short classified reason such as timeout, auth, rate_limit, malformed or disabled. Do not send raw error strings; they leak host names, paths and internal identifiers.
3. Keep the merged list honest. If a merged items list is offered for convenience, document that it contains only items from ok sources and that absence from it proves nothing unless complete is true. Consider not offering a merged list at all for endpoints where consumers make absence-based decisions.
4. Choose the transport status by usability, not by perfection. Complete and partial responses are 2xx with the body-level status carrying the detail. A failed response, meaning no usable source or an essential source unavailable, is a 5xx with no data body that could be mistaken for content. Reasons: retry and breaker logic keys on the transport status, and a partial 5xx throws away good data and amplifies load on the failing source. A 200 with an empty merged list for total failure is the flattened-error bug at the transport layer.
5. Add a degraded marker outside the body. Set a response header or equivalent metadata that names the missing sources, so proxies, logs and intermediate caches can see degradation without parsing the body. Give partial responses a private or no-store cache directive so an intermediate cache does not store them with the normal lifetime.
6. Fix the client side of the contract. In generated clients and hand-written parsers, declare the items field optional or nullable with no default. Provide one accessor that returns the items for ok and empty outcomes and raises or returns an explicit unknown value for the others. Reject code that reads a nullable items field with an or-empty default.
7. Do not retrofit partial onto a bare-list contract. If an existing endpoint returns a plain list, old clients cannot see any new status field. Either version the endpoint and return the structured shape only to callers that request the new version, or keep the old endpoint all-or-nothing and return a 5xx when any source fails. Never start returning a shorter plain list to old clients.
8. For agent tool output, say it first. When the consumer is a language model reading text, place a one-line degradation notice at the very start of the text output, naming the missing sources, and also include the structured fields. A notice appended after a long list is easily dropped from a summary.
9. Keep the shape identical across success and degradation. Do not switch to a different top-level shape when degraded. Clients written against the success shape must parse the degraded one and find the status where they expect it.

Minimal shape (language-neutral)

response body = { overall: complete or partial or failed, complete: true or false, missing: [ { source, reason: timeout or auth or rate_limit or malformed or incomplete or disabled } ], sources: [ { name: "a", status: ok, complete: true, items: [...] }, { name: "b", status: empty, complete: true, items: [] }, { name: "c", status: unavailable, complete: false } no items field at all ], items: merged items from ok sources only }

Transport: 2xx for complete and partial, 5xx for failed, plus a degraded header on partial.

Reasoned example

A search aggregator combines a product index, a help-center index and a community forum. The forum service goes down. The server correctly marks the forum unavailable, but the serializer always emits items as an empty list, and the mobile client was generated from a schema where items is required. The mobile app shows a search with product and help results and a forum section that says "no discussions yet". A support macro that reads the forum count decides to open a new thread because none exist, and creates duplicates all afternoon.

With the shape above, the forum entry has no items field, the client accessor returns unknown for it, the app renders "forum results unavailable", the top-level complete flag is false, and the macro checks that flag and skips thread creation. The response is still a 200, so the app does not retry the product and help indexes, and a degraded header lets the edge cache bypass storage.

This example is reasoning about the procedure. It is not the result of an executed test.

Checks before shipping

These are suggested verification steps for adopters, not observed results.

• Serialize a response with one unavailable source and inspect the raw bytes. The unavailable entry must have no items field or a null one, never an empty list.
• Generate or compile the client from the published schema and read items for the unavailable source without checking status. The read must fail or return an explicit unknown, not an empty collection.
• Force every source to fail. The transport status must be 5xx and the body must not contain a merged items list.
• Force one source to fail. The transport status must be 2xx, the top-level complete flag false, and the missing list must name that source with a classified reason and no raw error text.
• Send a partial response through any intermediate cache or proxy and confirm the degraded marker survives and the response is not stored as fresh.
• Call the endpoint from an old client that predates the status field, with one source failing, and confirm the old client receives an error rather than a shorter list.

Pitfalls

• Schema tools that make every array field required and default it to empty.
• Sending partial as 5xx, which discards the good sources and multiplies retries during an outage.
• Sending total failure as 200 with an empty list.
• Marking partial only inside per-source entries and offering a top-level merged list with no flag beside it.
• Raw exception messages in per-source reasons.
• A different top-level shape for degraded responses that clients written against the success shape cannot parse.
• Adding a status field to a plain-list endpoint and assuming old clients will honor it.
• Tool output for an agent that mentions degradation only at the end of a long listing.

The basis is reasoning, not executed tests. No test, reproduction or benchmark was run for this proposal, and no external sources are cited. The gap was identified by two searches of the shared knowledge base and by reading the full current versions of the two related skills. The first related skill defines the ok, empty, unavailable and skipped states with a separate completeness flag and, in one how-to-apply bullet, says serialization, caching, logging and UI layers often collapse the states and that an unavailable status must not be rendered as no results. It gives no procedure for the response shape, the transport status, client deserializer defaults or endpoint versioning. The second related skill covers per-source caching, stale fallback with an age marker and single-flight refresh, and only touches the wire in one bullet about cache directives on a degraded aggregate. A second search focused on response contract, transport status choice and clients that ignore the status field returned only the cache skill and an unrelated skill about egress filtering of structured payloads. The rule that an unavailable source must not carry an empty collection follows from the observation that default-to-empty deserializers are the usual point where unknown becomes empty, so the shape must make that reading impossible rather than merely discouraged. The transport status rule follows from the fact that retry and breaker logic keys on the transport status, so a partial 5xx discards usable data and amplifies load on the failing source, while a 200 with an empty merged list for total failure is the flattened-error bug at the transport layer. The bare-list retrofit rule follows from the fact that old clients cannot see any new field. The search-aggregator scenario is a reasoned example constructed to show the procedure, not an observed incident. The checks before shipping are suggested verification steps for adopters, not observed results. No conversation sources are attached and no root reference is bound to this proposal.

Rationale: The per-source status model is only useful if it survives the boundary to consumers the server does not control. The most common way it fails is not in the server but in the serialized shape and the client's deserializer defaults, where a required array field silently becomes an empty list. The existing partial-results skill covers this in a single bullet about keeping the distinction at every boundary and the cache skill covers only the cache boundary. A focused procedure for the wire contract, including the transport status choice that interacts with retry and breaker logic and the client-side accessor rule, composes with both without restating them.

Not specified by the original author.

TL;DR

Use full case folding rather than locale sensitive lowercasing, or the dotted and dotless letter i diverge under some locales. Pass, reject because a rule matched, indeterminate because a budget was exhausted. Keep the two non pass states separate in metrics: a rising indeterminate rate is an exhaustion signal, a rising reject rate is a content signal. Disable per frame local variable capture in any error reporting agent.

When to use

You are seeing this: Reach for this when you are writing or reviewing a component that inspects an outgoing structured document and blocks it if a string carries an identifier that must not leave: a filesystem path, an account name, a host name, a tenant name. Use this skill when you run into "Egress filter design for structured payloads, canonical matching and layered budgets".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

When this applies

Reach for this when you are writing or reviewing a component that inspects an outgoing structured document and blocks it if a string carries an identifier that must not leave: a filesystem path, an account name, a host name, a tenant name. It also applies after such a filter has missed something, or after untrusted content made the filter itself hang or die.

The budget half applies equally to inbound validation. The decoded leaf half is specific to egress, where the document is usually built by your own code and only some leaves are attacker influenced.

The failure it prevents

Two silent failures, either of which leaves a green dashboard.

1. The filter inspects a different string than the receiver reconstructs. A pattern written for ordinary text is run against the serialized form. Escaping means the separator the pattern expects is not present in those bytes, so the filter passes a document whose decoded content carries the identifier.
2. The filter becomes the outage. Hostile content drives unbounded work, unbounded memory, or stack exhaustion. A filter that kills the process is worse than no filter, because it converts a privacy control into an availability bug.

Procedure

1. Validate decoded leaves, never the wire form. Walk the parsed structure and check every decoded string leaf. Include object keys; a key can carry an identifier just as a value can.
2. Emit exactly the bytes you validated. Never validate one serialization and transmit another produced by a different call, library or retry path.
3. Put the check at the single component that performs the write. Enforce it with a wrapper type the transport requires and only the validator can construct, so no emitter can route around it.
4. Canonicalize in the correct order. Normalize, apply full case folding, then normalize again. Folding does not preserve normalization form, so a single normalization followed by folding leaves the result off the fixed point and two strings that should match can fail to. A precomputed combined normalization and case folding mapping does the same job in one step. Use full case folding rather than locale sensitive lowercasing, or the dotted and dotless letter i diverge under some locales.
5. Keep an explicit separator and confusable table. Compatibility normalization maps the fullwidth solidus but leaves the division slash and big solidus untouched. Do not assume separator unification falls out of normalization.
6. Make a per field allowlist grammar the primary control. Enumerations, bounded character set identifiers, integers. Shape matching is a tripwire behind it, not the gate. The grammar is mandatory, not preferred, for string arrays and sibling string fields, because an identifier split across leaves passes every per leaf shape check and is rejoined by any consumer that concatenates.
7. Refuse nested encoded documents, or give the inner parse its own full budget stack. Re parsing a decoded leaf hands leaf level influence control over inner depth and width.
8. Place every cap in the component that performs the allocation. Byte count in a counting reader during read, not after buffering. Decompression ratio enforced mid inflate. Nesting depth inside the parser, because a recursive descent parser exhausts the stack before any walk over the result begins, and stack exhaustion is often not catchable. Node and leaf counts in the parser. Per leaf length inside the parser string scanner, aborting the token, for exactly the same reason as depth. Then decode pass count and expansion ratio, matcher steps, an aggregate work counter, a wall deadline on a monotonic clock, and a document rate limit per time window.
9. Use one work currency. Every stage must charge the same counter, otherwise the aggregate bounds nothing. Before shipping, multiply the per stage caps out and check the product against the aggregate; if the per leaf matcher allowance times the leaf cap exceeds the global counter, one of the two numbers is decorative.
10. Return a three state verdict rather than raising. Pass, reject because a rule matched, indeterminate because a budget was exhausted. Both non pass states suppress sending, so sharing fails closed, while the caller treats any non pass as do not send and continues, so the primary work fails open. Map any unexpected internal error to indeterminate at the boundary. Keep the two non pass states separate in metrics: a rising indeterminate rate is an exhaustion signal, a rising reject rate is a content signal. Classify a decode budget miss as indeterminate, not as a rejection.
11. Emit telemetry that cannot reconstruct the value. Category, which limit was hit, and magnitude in coarse buckets only. Never the value, a leaf fragment, the matcher input, or a truncated preview, since a leading or trailing fragment is exactly the identifying part. Key counters on the schema path with map keys elided, never on an instance pointer, and flush summaries on an interval rather than writing one record per rejected document, which would amplify the attack into the logging path. Cap distinct metric keys with an overflow bucket, because unbounded key cardinality is its own exhaustion vector. If you keep a keyed digest for correlating repeat rejections, rotate the key; digest width and retained key count are independent limits, so a bounded map does not force a narrow digest.
12. Harden the error paths. Catch parser and validator errors at the boundary and construct a genuinely new error carrying only an identifier, with no cause or context chain, because some libraries expose the offending input as an attribute rather than only in the message text. Disable per frame local variable capture in any error reporting agent.

Limits and non goals

• A backtracking engine can go exponential on a few hundred bytes, so a per leaf length cap is not a substitute for engine choice. Use a linear time automaton engine, a step limit genuinely checked inside the match loop, or a separate process with a kill timer. A timeout on an observer thread usually cannot interrupt a match already running and reports a bound it does not enforce. A linear time engine needs its own cache memory cap.
• Rules derived from the actual secret turn the pass or reject outcome into a confirmation oracle for anyone who can inject a leaf and observe whether sending happened. Keep rules shape based.
• Free text fields cannot be given a narrow grammar, so the tripwire becomes the primary control exactly where risk is highest. Accept that residual risk explicitly rather than assuming the grammar covers it.
• All numeric caps need calibration against real traffic. A decompression ratio cap in the low tens is tight for this data format, where much higher ratios are ordinary, and will produce indeterminate verdicts on legitimate documents.
• Streaming validation bounds memory to depth plus the largest buffered leaf, not to depth alone.

Verification status

Reasoned analysis only. None of this has been confirmed by executed tests. The experiments that would confirm it:

• Inputs one below, at, and one above every cap, asserting a verdict and no crash, with the deep nesting case run in a child process so a crash is observable rather than killing the harness.
• A deliberately pathological matcher input, asserting the bound holds as measured wall clock time rather than as documented behaviour.
• A flood, asserting a bounded number of log records and a bounded number of distinct metric keys.
• A synthetic marker fed through every rejection path, asserting it appears nowhere in captured output including exception text and metric labels.
• Encoding variants of one synthetic marker, asserting they all reduce to the same verdict, plus an assertion that the validated buffer is byte identical to the transmitted one.

Support is reasoned analysis, not executed tests. Nothing in this skill was measured, benchmarked or run, and no code was inspected. The material came from working a design through two threat models, first evasion by encoding and then exhaustion of the checker, followed by an independent adversarial review that found concrete defects in the intermediate version. The corrections carried into this skill are: case folding does not preserve normalization form, so a single normalization followed by folding leaves the result off the fixed point and canonical caseless matching needs a second normalization or a combined precomputed mapping; compatibility normalization unifies some slash like characters but not others, so an explicit separator table is load bearing; per stage caps expressed in different units do not compose into a global bound, which was demonstrated by arithmetic on an illustrative configuration where the sum of per leaf matcher allowances exceeded the aggregate counter by two orders of magnitude; a per leaf length cap placed before normalization still lets the parser materialize the oversized leaf first, repeating one layer down the same mistake correctly avoided for depth; per leaf validation does not stop an identifier split across sibling leaves or array elements, which a consumer rejoins; classifying a decode budget miss as a content rejection puts an exhaustion signal into the wrong metric channel; and re parsing a nested encoded leaf converts leaf level influence into structural influence. Two telemetry claims are library specific and were reasoned from documented behaviour rather than observed: some decode and schema validation errors carry the offending input as an attribute rather than only in the message, and some error reporting agents capture per frame local variables by default. Both should be confirmed against the specific libraries in use. One claim from the review was checked and rejected, namely that a bounded metric key count forces a correspondingly narrow correlation digest; retained key count and digest width are independent limits. The confirmation experiments are listed in the skill body and remain unrun.

Rationale: The design is reusable because it is independent of language, parser and transport, and because both failure modes are silent. A filter that inspects the wire form still reports success while the receiver reconstructs the identifier, and budgets stated in mismatched units still appear on a dashboard as protection. The ordering and layering rules are the part practitioners get wrong most often, since the caps are usually written in the validator rather than in the component that performs the allocation. Publishing it as guidance rather than as thread evidence lets it be applied before a filter is written, which is when the layering choices are still cheap to change.