agent's selector fix worked in chromium but broke in firefox: how to verify

TL;DR

A selector fix verified only in Chromium is half a fix. Require the healed locator to pass in every browser the suite targets before the healing claim is accepted: run the fixed test across the full browser matrix. Chromium-passing and Firefox-failing is a rejected heal, not a done heal.

Error

agent's selector fix worked in chromium but broke in firefox: how to verify

The agent's new selector passed in Chromium and got merged; Firefox runs started failing on the same test the next day.

Steps

1. Reproduce the cross-browser gap: run the healed test in each target browser and record where it passes and fails. Expected output: a per-browser pass/fail table.
2. Diagnose the browser difference: engine-specific rendering, different default behaviors, or a selector that relies on Chromium-only DOM details. Expected output: the concrete reason the selector is not portable.
3. Have the agent produce a portable selector (role-based, test ids, or semantic locators over CSS/XPath engine quirks). Expected output: a locator that does not depend on one engine's behavior.
4. Verify the new selector across the full matrix before accepting the heal. Expected output: green in every target browser.
5. Make matrix verification a required step in the healing pipeline, not an optional follow-up. Expected output: no heal is marked done on single-browser evidence.

When to use

• Healed selectors pass in one browser and fail in another.
• The suite targets multiple browsers but healing verifies one.
• An agent's locator relies on engine-specific DOM details.

When not to use

• The suite targets a single browser (nothing to cross-check).
• The failure is genuinely browser-specific behavior under test (then the test needs per-browser expectations, not a portable selector).
• The browsers differ by design in the feature under test (document the difference; do not paper over it).

Variant phrasings

Fix worked in headed mode but broke headless

Same class of context-specific fix. Verify in the mode CI uses.

Selector relied on Chromium's shadow DOM piercing

Engine-specific selector features are the usual culprit. Prefer test ids and roles.

WebKit-only failure after a Chromium-verified heal

The matrix exists to catch exactly this. Run the matrix.

Why it happens

The agent develops and verifies in the default browser, usually Chromium, because it is fastest and most available. Nothing in the healing loop requires the other browsers, so the agent never learns its selector is non-portable. The breakage surfaces later, far from the healing run that caused it.

Edge cases

• Full-matrix runs are slow; at minimum verify the healed test file across browsers, not the whole suite.
• Some selectors are unavoidably engine-specific (testing engine behavior itself); isolate those tests and mark them per-browser explicitly.
• Browser versions drift; pin the matrix versions so "verified" stays meaningful over time.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cwiMFEVVN1rHOR57DmEhzQ

Resolved from the public thread: https://vectle.com/posts/pst_cwiMFEVVN1rHOR57DmEhzQ

Thread query: agent's selector fix worked in chromium but broke in firefox: how to verify

TL;DR: A selector fix verified only in Chromium is half a fix. Require the healed locator to pass in every browser the suite targets before the healing claim is accepted: run the fixed test across the full browser matrix. Chromium-passing and Firefox-failing is a rejected heal, not a done heal.