How does Language I/O evaluate and take action on quality?

What are Quality Estimates?

The Quality Estimate is a range of new features that allow organizations to define quality thresholds for Translation Optimization and Machine Translation, then evaluate their content according to these thresholds.

Why does it matter?

By defining a threshold for quality and evaluating outgoing messages against this threshold, the Language I/O QE processes help make sure that your agents and end users get the best translation quality available at that time, whether through translation optimization, or by ensuring that the translation meets a predefined standard of quality.

Because Quality Estimate runs on top of the translation processes, there might be an increased latency cost associated with using this feature.

What is "quality"?

It is important to specify what exactly is evaluated when talking about Quality Estimate. Language I/O QE processes evaluate quality in terms of contextual similarity. 

Here's an example with a message that contains an accidental typo that is also a real word (your/you're):

• Original message: "Hi, this is John. I'm going to look at you're chat with our chatbot so I can understand what's going on. That way, you won't have to repeat yourself. Please give me a few seconds and I'll be right back."
• MT1: "Bonjour, c'est John. Je vais voir si tu parles avec notre chatbot pour que je puisse comprendre ce qui se passe. Comme ça, tu n'auras pas à te répéter. Donnez-moi quelques secondes et je reviens tout de suite."

The first translation engine is thrown off by the typo and misses the intent. It also mixes formal and informal styles, so the resulting translation falls below the TQE threshold. As a result, Quality Failover kicks in and requests a second translation:

• MT2: "Bonjour, c'est John. Je vais regarder votre chat avec notre chatbot pour comprendre ce qui se passe. Ainsi, vous n'aurez pas à vous répéter. Accordez-moi quelques secondes et je reviens tout de suite".

The second translation engine does detects the meaning behind the typo and fixes the translation accordingly, so its score is higher. This is the translation that is returned to the agent.

To summarize, Language I/O's approach prioritizes the intent of the text and the preservation of meaning in the translation, over word choice or style.

How does it work?

For Optimization

When Optimization QE is enabled, outgoing messages pass through a Large Language Model routine that lightly alters the source content to increase the likelihood of a high-quality machine translation. If after analysis, the optimized message falls below defined quality thresholds, the optimization operation is cancelled and the original, non-optimized message is sent for translation instead.

Example

Suppose that an agent enters the long introduction below:

• Original message: "Hi, this is John, Let me give a quik look over on your chat with our Chatbot you had earlier so I can get up to speed, and that way you won’t have to give me all the details you already provided. Please give me just a few seconds and I’ll be right back to contniue our conversation."

Note that in addition to being verbose, the message also contains two typos. Using translation optimization, the source is processed before translation, resulting in this version:

• Optimized message:"Hi, this is John. I'm going to look at your chat with our chatbot so I can understand what's going on. That way, you won't have to repeat yourself. Please give me a few seconds and I'll be right back."

The system compares the optimized version against the original and determines that the optimized version is the one that should be translated. As a result, the translated message not only retains the original's intent, it is also much more straightforward:

• Translation: "Bonjour, c'est John. Je vais regarder votre conversation avec notre chatbot pour comprendre ce qui se passe. Ainsi, vous n'aurez pas à vous répéter. S'il vous plaît, donnez-moi quelques secondes et je reviens tout de suite."

For more information, see Using the Optimization Quality Estimate (OQE).

For Translation

For Translation QE, the system assesses the quality score of the translation. If it falls below the defined threshold:

1. If Quality Failover is enabled, the system automatically generates a second translation using an alternative machine translation engine. The second translation is scored and the score is compared to that of the original translation. The translation that scores higher is delivered.
2. If Human Translation is enabled and available in your Language I/O application, the agent receives a prompt to either send the content for Human Translation or to proceed with the translation that was delivered. The Human review step is entirely optional.

For more information, see Using the Translation Quality Estimate (TQE).