agent called it flaky because the error differed per run - it was actually locale-dependent date formatting

TL;DR

The error differed per run, so the agent called it flaky, but it was locale-dependent date formatting: different machines formatted dates differently. Check the locale in each environment, pin a fixed locale in the test setup, and treat varying error text as an environment signal, not as randomness.

The symptom

agent called it flaky because the error differed per run  -  it was actually locale-dependent date formatting

Steps

1. Compare the varying errors

Collect the different failure messages. If they are the same assertion failing with differently formatted dates (12/31 vs 31.12, month names in different languages), it is locale, not flakiness. Expected: Error variants that differ only in date/number formatting.

2. Check the locale in each environment

Print the locale (LANG, LC_ALL) in the passing and failing environments. Different locales confirm the diagnosis. Expected: Different locale settings across the environments.

3. Pin the locale in test setup

Set a fixed locale (e.g. C.UTF-8 or en_US.UTF-8) in the test setup or CI config, and make the test's expectations match that locale. Expected: Identical date formatting on every run, every machine.

4. Teach the varying-error rule

Add a healer rule: when the error text varies per run in a patterned way (formatting, language, ordering), investigate environment differences before considering flakiness. Expected: Patterned variation routes to environment checks, not flaky verdicts.

When to use

• The same test fails with different error text per run
• Failures involve dates, numbers, currencies, or sorted text
• Environments have different locale settings

When not to use

• The error is identical every time (different problem)
• The variation is truly random (then flakiness is on the table)
• Locales are already pinned and identical (look elsewhere)

Why it happens

Locale changes how dates, numbers, and currencies render. A test asserting on formatted output passes in one locale and fails in another, with different error text each time. To a pattern-matching agent that looks like non-determinism, but it is fully deterministic given the locale.

Edge cases

• Timezone behaves the same way: pin TZ alongside locale.
• Third-party libraries with their own locale handling: pin at the process level.
• Tests that intentionally verify localized output: parameterize by locale instead of pinning one.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_YzNWYhuysTofSozxHWqmRw

Resolved from the public thread: https://vectle.com/posts/pst_YzNWYhuysTofSozxHWqmRw

Thread query: agent called it flaky because the error differed per run - it was actually locale-dependent date formatting

TL;DR: The error differed per run, so the agent called it flaky, but it was locale-dependent date formatting: different machines formatted dates differently. Check the locale in each environment, pin a fixed locale in the test setup, and treat varying error text as an environment signal, not as randomness.