Zuckerberg Launches AI Dubbing and Speech Translation at Meta Developer Conference

Social media giant Meta, the parent company of Facebook, WhatsApp, and Instagram, announced at Meta Connect 2024 the addition of a number of new AI-enabled features to its products. Among the most talked-about are those that help users do the talking — more specifically, automatic dubbing for Instagram Reels audio.

“With automatic dubbing and lip-syncing, Meta AI will simulate the speaker’s voice in another language and sync their lips to match,” a September 25, 2024 press release explained. “We’re starting with small tests on Instagram and Facebook, translating some creators’ videos from Latin America and the US in English and Spanish, and we plan to expand this to more creators and languages.”

Meta’s news comes on the heels of YouTube’s September 18, 2024 announcement that hundreds of thousands of creators will gain access to automatic dubbing capabilities “in the coming months.”

Notably, Meta’s current tests are limited to Spanish-English, while YouTube is reportedly expanding the number of supported languages “from Spanish and Portuguese to French, Italian and more.” Moreover, YouTube noted how many users would be able to try out the tool, whereas Meta did not mention any estimates on who, or how many, can do so.

Meta’s developer conference also heralded real-time speech translation capabilities for Ray-Ban Meta smart glasses, originally launched in 2021.

“Soon, your glasses will be able to translate speech in real time. When you’re talking to someone speaking Spanish, French or Italian, you’ll hear what they say in English through the glasses’ open-ear speakers,” a press release stated, adding, “We plan to add support for more languages in the future to make this feature even more useful.”

Slator 2024 Language Industry Market Report — Language AI Edition

The 140-page flagship report features in-depth market analysis, language AI opportunities, survey results, and much more.

$970 BUY NOWIncluded in our Growth, Pro, and
Enterprise plans. Subscribe now!

Most functional commands for the smart glasses — for instance, instructing the glasses to take a photo — already operate in English, French, and Italian, although Meta AI assistance on the device is only available in English. The glasses are currently available for purchase in Canada, the US, Australia, and a dozen European countries, with prices ranging from USD 299-429 depending on the style.

While Meta did not specify the exact technology behind the speech translation and automatic dubbing features, the company has long publicized its investments in speech translation R&D.

SeamlessM4T, a model released in August 2023, is one of the recent milestone developments; it handles different combinations of text and speech translation for dozens of languages. It was followed in short order by the Seamless suite of models introduced in December 2023, including SeamlessStreaming, which was designed for automatic speech recognition and speech-to-text translation for nearly 100 languages.