top of page
Blurry Blue_edited_edited.jpg

When a QA Report becomes a QA problem

6 minutes ago
6 min read

I grew up religiously watching Star Trek: The Next Generation. Captain Jean-Luc Picard’s “Tea, Earl Grey, hot” remains one of those lines I can never imagine in anyone but Patrick Stewart’s voice. His recent passing resulted in quite a few episode rewatches and also brought back memories of a number of favourite episodes. Remember an episode called Darmok in which Jean-Luc encounters the Tamarians? The universal translator can process their words but cannot make sense of what they are saying because their language relies on references to stories and events that Picard knows nothing about. Until he begins to understand the context behind those expressions, the translation is useless.

That episode hit close to home while I was working (mostly battling) with AI-assisted linguistic QA for African languages. Most of the translations I work with have already gone through human translation or review. Like many in the industry currently fine-tuning AI QA workflows, I want an additional check that catches something we may have missed with the least amount of false positives and hallucinations. I don’t want to start a discussion with a translator over an error that doesn’t exist.

Now, false positives have always been part of QA. Xbench and the built-in checks in CAT tools can return enormous lists of warnings, particularly on large multilingual projects. Repeated words are an easy illustration. In some contexts, including in African languages, repetition is legitimate. A checker looking for duplicated words can flag it anyway. You check the segment, recognise that the wording is correct, ignore the warning and quickly move on. Other warnings are less straightforward, but we’re used to distinguishing what a rule has detected from what a linguist would regard as an error.

The AI reports I’m working with now are different. They don’t just identify a possible issue. They provide the source, the translation, an alternative and an explanation of why the alternative is supposedly better. That explanation can be long and technically plausible. And in most cases, very confident. Even if the AI has misunderstood the grammar or context, it can still argue its case with remarkable confidence. The reviewer is then faced with something much harder to dismiss than a highlighted repeated word.

A 2025 CHI study¹ found that explanations increased people’s reliance on both correct and incorrect answers from large language models. The researchers weren’t studying localisation, but I recognise the problem. A well-presented explanation makes it easier to believe that the person who produced the translation made a mistake, particularly when I can’t assess the target language.

Imagine you receive a report showing 10 apparent errors from a translator you’ve worked with for years. In this case, they aren’t 10 quick warnings. They’re little linguistic masterpieces, complete with proposed corrections. Some look super serious. But you know the translator’s previous work and have no particular reason to doubt their ability, but the report is right there in front of you. Even as a translator myself and knowing how problematic general AI is for African languages, I inadvertently catch myself asking whether the person rushed this job, missed something obvious or has simply stopped producing the standard of work I expect.

This is the part that is worthwhile to look at. The doubt that arrives before you know whether the findings are correct. A working relationship built over many projects can wobble because of an assessment that took a few minutes or even seconds to generate. You may not actually accuse the translator of anything, but the way you read their work and the way you approach the next query have already changed. If the report turns out to be wrong, it doesn’t always follow that the doubts you entertained disappear as quickly as they appeared.

With ubiquitous and free GenAI, more and more, the report comes from the client, who has put our translation through AI. They may know less about the target language than we do, but they now have a detailed document explaining why our work is wrong. The client begins questioning the LSP’s quality, and we risk doing exactly the same to the translator when we just forward the findings. Obviously, we can’t just dismiss the client’s questions, as some may be valid, but we shouldn’t pass on that doubt before checking what it’s based on.

Suppose the translator comes back and demonstrates that 7 of those 10 findings are false positives. The original translation is acceptable, the proposed corrections are unnecessary, the AI had the wrong language (even though you told it) or is simply hallucinating. The close relation of African languages and the thousands of dialects trip LLMs up much more than for the bigger languages. But what about the remaining 3? They are genuine errors, and that is a real problem.* You would want them corrected, and depending on the text, they might justify a considerable amount of additional checking. I don’t think a translator can reasonably refuse all QA queries because a proportion of them turn out to be wrong. Nor can a project manager ignore a plausible concern simply because the linguist has always been reliable. Clients ask questions, and often rightly so.

Yet the translator has now spent time examining 10 findings and rebutting 7. With a traditional QA warning, “intentional repetition” or “correct in context” may be enough. With a detailed AI argument, the response can turn into a full-blown linguistic report. The translator has to explain why an apparent grammatical rule does not apply, why the suggested correction changes the meaning, or why the original construction is acceptable in that particular variety of the language. If the project manager cannot judge those explanations, we might need another linguist to look at the dispute. We have effectively created a second QA exercise to verify the first.

Dealing with it every day, I know how irritating that becomes for a good translator after several projects. It’s difficult to object strongly when some of the queries are valid. Answer too briefly, and you might appear dismissive, spend time answering properly, and you’ve taken on work that wasn’t in the original assignment. Worse, the translator may start feeling that their judgement counts for less than a fluent explanation generated by a system that doesn’t even understand the language.

Also, the cost doesn’t disappear because nobody entered it on a timesheet. There is the translator’s time, the project manager’s time, possibly a second linguist’s time and the delay while everyone exchanges explanations. In the above case, 70% was wasted time.

When the report comes from an end client, additional reports and processes on, sometimes, a variety of platforms with usernames, passwords, 2FA, etc., etc. often follow. On a multilingual project, an apparently inexpensive automated check can generate a surprising amount of human work. I wouldn’t call that a saving without knowing what was caught.

Unfortunately, there is still no useful universal ratio of genuine errors to false positives for these types of reports. 3 errors affecting dosage instructions or contractual obligations are not equivalent to 3 arguable preferential wording. Equally, an AI tool that flags every possible variation may find more real errors than a conservative check while making the whole workflow unmanageable and impossible to scale. And, importantly, where does responsibility lie? Do you just forward an AI report to the translator and ask them to resolve everything? That is, of course, quick and easy for the person sending the email, at least initially, but it transfers the system’s weaknesses to the linguist. But a project manager cannot independently verify subtle linguistic issues in every target language, so the answer cannot simply be “check everything before asking”. Some questions will need an independent reviewer, while others should never leave the internal QA stage. The distinction has to be part of the workflow, and its cost has to be acknowledged. This means an urgent discussion about who will foot the bill for the 70% wastage mentioned above.

I still want to use AI for linguistic QA if possible. The genuine errors it finds are the reason to keep improving the process. What I don’t want is to make a good translator repeatedly prove that they know their own language. Look at the severity of what it catches, and the effort required to get there, not the impressive number of findings in the report. If a report makes you doubt someone you’ve trusted for years, you should demand more than an articulate explanation. The same way you don’t fall for that impressive and confident “doctor” on Instagram explaining in detail why you shouldn’t drink water because it deactivates the coagulant enzymes in your gluteus maximus, leading to incorrect karyotyping, don’t be fooled by QA reports just because they’re confident and fluent.

*Note: this is a hypothetical split, not a statistic about AI QA, but it gets at the problem better than a general complaint that the technology sometimes makes mistakes.

1. Kim, Sunnie S. Y., Jennifer Wortman Vaughan, Q. Vera Liao, Tania Lombrozo and Olga Russakovsky (2025), "Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies", Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25). https://doi.org/10.1145/3706598.3714020

bottom of page