news agent fetched non-english articles, language filter failed

TL;DR

Non-English articles slip through when the agent filters on the query language instead of detecting each article's language. Run language detection on the article text (not the query), drop or separately route non-English items, and set the news API's language parameter as a first pass.

The error

briefing included 12 German and 5 French articles in an English digest
filter: query was English; API language param not set.
Reader-facing language mix in the morning brief.

Fix it

1. Confirm the failure mode: check whether the news API request set a language parameter. If not, the API returned its default multilingual mix. Expected: Expected: the request log shows no language param, explaining the mix.

2. Set the API's language parameter (for example language=en) as the first-pass filter on every news query. Expected: Expected: the result set becomes predominantly English at the source.

3. Add per-article language detection on the fetched text with a library like langdetect, and drop or quarantine anything below the confidence threshold. Expected: Expected: the few non-English stragglers are caught before the briefing.

4. For the pipeline, route quarantined articles to a separate multilingual digest instead of deleting them; some readers want them. Expected: Expected: no coverage is lost, but the English brief stays English.

When to use

• Your English news digest contains non-English articles.
• You need defense in depth: API param plus per-article detection.
• You run multilingual monitoring where language routing matters.

When not to use

• The brief is intentionally multilingual - then label each article's language instead of filtering.
• The non-English articles are the story (foreign press coverage) - then feature them deliberately.
• The API has no language parameter - then rely on detection alone.

Why it happens

News APIs default to multilingual results unless told otherwise, and an English query still matches non-English articles (proper nouns, tickers, quoted English). Filtering on the query language checks the wrong thing; only the article's detected language is ground truth.

Variant phrasings

news api non-english articles language filter

Same failure and fix; some searchers phrase the query this way.

multilingual articles in english news digest fix

Same failure and fix; some searchers phrase the query this way.

Edge cases

• Mixed-language articles (English body with German quotes) can confuse detectors; use the majority language.
• Language detection on very short snippets (headlines only) is unreliable; detect on the body text.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_IMhcf9urf2xM2jH9je0TQQ