How Alexa learned French

Since launching in France, Alexa has played music by "Trois Cafés Gourmands" and "Amel Bent," made calls to family members, set timers, shared the weather forecast, and much more. To say that Alexa is a polyglot would be an understatement – prior to learning French, Alexa already spoke five variants of English (U.S., U.K., Canada, India, Australia), German, and Japanese.

The team working on Alexa’s launch in France asked, "Can we use Alexa’s knowledge of all these other languages to help her learn French faster?" The answer was a resounding "oui."

The research scientists on the Alexa team used a machine learning technique called transfer learning to help Alexa learn to identify words in requests that people made to Alexa. Transfer learning involves drawing on knowledge in one area, and using that knowledge to train a model in a different area.

“Humans are inherently good when it comes to transfer learning,” says Max Bisani, Sr. Manager, Machine Learning, Alexa AI-Natural Understanding. He leads a machine learning team in Aachen, Germany, responsible for teaching Alexa new languages. “For example, if you learn the word “bread”, you know how to pronounce letters like 'b' and 'r'. You then know how to use these letters in other words. What’s more, now suppose, you come across a familiar word ‘croissant’. You don’t have to relearn that croissant is a noun, or that it is similar to bread, or that it probably tastes good with butter.”

The machine learning team worked together to draw on Alexa's knowledge of English and also identify words in French.

When you speak to Alexa, your command (or request, if you’re being polite) is broken up into segments that are each 25 milliseconds long, and the amount of energy in each frequency is calculated. A neural network – made up of several layers-- estimates how likely each sound in a catalogue of sounds is present in that segment. By piecing these likelihood for how the individual sounds come together, Alexa identifies words and phrases.

“When we teach Alexa French, we drew on a massive catalog of examples from other languages, because recognizing the energy patterns of an “m” or “s” sound in English is a good starting point for identifying 'm' and 's' sounds in French," said Bisani. "Leveraging our neural networks from English, German, and other languages have resulted in a tenfold reduction in the amount of data we need to teach Alexa French”.

Recognizing words is one thing. Understanding what they mean is entirely another. French and English have significant differences that make this task difficult.

For example, in English, you would address both your child and an elderly person with the pronoun “you.” However, in French you would likely use “tu” for the former and “vous” for the latter. One of many fundamental differences is that the French language has more variations in sentence structure than English – words are commonly expressed in the passive voice or in an order that wouldn’t make sense in English (example: Harry Potter was the book the best I have read).

These differences could lead to difficulties if applying the same systems to comprehend them both. Alexa uses a Natural Language Understanding (NLU) system to understand the intent behind a user’s request. Given the fundamental differences between the two languages, the NLU system used for English would fail miserably at understanding human intent in French. So Alexa’s scientists used two different approaches to overcome this difficulty.

One approach was to automatically translate the command from U.S. English, run it through a French NLU system, and translate the statement back into English. The translation is then scored according to how well the new semantic tags line up with those of the original. Poor translations are discarded, while accurate ones are used to drive improvements to the system.

“We know you cannot simply take ‘an American Alexa’ and expect that experience to resonate with customers in France,” says Nicolas Maynard, Alexa country manager, France. “We have different ways of asking the same question with only the pronunciation varying depending on whether you’re trying to be polite or not. Alexa needs to understand both.”

Given this, the second approach to overcoming the potential NLU issues required a human touch.

The French version of Alexa was beta tested with Amazon employees in its fulfillment centers and corporate office locations. These conversations helped Alexa understand the nuances of French accents by region, the kind of questions that were commonly asked (for example, cooking and soccer came up an awful lot), and expressions for commonly requested tasks like setting a timer or an alarm.

The rapid uptake of Alexa in France has been impressive, but there's much work to do as the worlds a lot larger than one country, or language.

“It’s still day one when it comes to teaching Alexa new languages,” says Bisani. “One problem we’re working on now is to enable developers who make new skills for Alexa. Take the example of someone who has made a skill in French. How can that developer get that skill understood all over the world, in every language, with one click?”

Restez à l'écoute pour la suite.