“Hola! ¿Hablas inglés?” I asked the woman who answered the phone in the Barcelona restaurant.
I was calling in a futile attempt to make a reservation for the CNET team dinner during Mobile World Congress this year. Unfortunately, I don’t know Spanish (I learned French and German at school). And as it turned out, she didn’t speak English either.
“No!” she said, and brusquely hung up.
What I needed in that moment was the kind of AI call translation feature that’s becoming increasingly prevalent on phones – including Samsung and Google devices, and, from next week, Honor.
When Honor unveils its Magic V5 foldable at a launch event on Aug. 28 in London, it will come with what the company is calling “the industry’s first on-device large speech model,” which will allow live AI call translation to take place on device, with no cloud processing.
Currently the phone supports six languages – English, Chinese, French, German, Italian and Spanish. For the aforementioned reasons, I can’t test all of these, but I’ve already had a play around with the feature and can confirm it did a very effective job of translating my garbled messages into French. I only wish I’d had it available to me in Spain when I needed it.
The model Honor has deployed was designed by the company in collaboration with Shanghai Jiao Tong University, based on the open-source Whisper model, said Fei Fang, president of product at Honor in an interview. It’s been optimized for streaming speech recognition, automatic language detection and translation inference acceleration (that’s speed and efficiency, to you and I).
According to Fang, Honor’s user experience studies have shown that as long as translation occurs within 1.5 seconds, it doesn’t “induce waiting anxiety,” in anyone attempting to use AI call translation. As such, it’s made sure to keep the latency to within these parameters so you won’t get anxious waiting for the translation to kick in.
“We also work together with industry language experts to consistently and comprehensively evaluate the accuracy of our output,” she added. “The assessment is primarily based on five metrics: accuracy, logical coherence, readability, grammatical correctness and conciseness.”
In addition to Honor’s AI model, live translation is being powered by Qualcomm’s Snapdragon 8 Elite chip. The 8 Elite’s NPU allows multimodal generative AI applications to be integrated onto the device, allowing. Honor’s algorithms work together with the NPU to keep power consumption as low as possible while maintaining the required accuracy of the translations, said Christopher Patrick, SVP of mobile handsets at Qualcomm.
There are a number of benefits to having the AI model embedded on the Magic V5, but perhaps the most compelling is the privacy it guarantees. It means that everything is processed locally and your calls will therefore remain completely confidential. The fact that the model lives on device and you don’t need to download voice packages also reduces its storage needs.
Another benefit of running the model on the phone itself is “offline usability,” said Patrick. “All conversation information is stored directly on-device and users can access it anytime, anywhere, without network restrictions.”
The work Honor has done on AI call translation is set to be recognized at the upcoming Interspeech conference on speech science and tech. But already, Honor is thinking about how this use of AI can be used to enable other new and exciting features for the people who buy its phones.
“Beyond the essential user scenario of call translation, Honor’s on-device large speech
model will also be deployed in scenarios such as face-to-face translation [and] AI subtitles,” said Fang. The process of developing the speech model has allowed Honor’s AI team to gain extensive experience of model optimization, which it will use to develop other AI applications, she added.
“Looking ahead, we will continue to expand capabilities in areas such as emotion recognition and health monitoring, further empowering voice interactions with your on-device AI assistant,” she said.
