SWORD Benchmark Reveals Cross-Lingual Inconsistencies in LLM Fact Checking

In a research preprint published on arXiv, scientists introduced Systematic Wikidata-based Object-Relation Distortion (SWORD), a benchmark designed to evaluate whether large language models consistently reject factual errors across multiple languages.

SWORD generates syntactically correct yet factually false statements in eight widely spoken languages by applying controlled perturbations to Wikidata triples. The evaluation revealed that models counterintuitively achieve higher accuracy on semantically plausible distortions than on nonsensical substitutions, suggesting a reliance on distributional familiarity rather than genuine factual verification.

Furthermore, models that exhibited comparable baseline accuracy across languages showed significant performance degradation on East Asian languages when handling distorted statements. In some models, the cross-lingual performance gap reached up to 28 percentage points, representing a 49% relative reduction in capability.