Alibaba, the renowned Chinese multinational corporation, is making waves in the tech world with its latest innovation. Known for its e-commerce prowess, Alibaba is also a significant player in technological advancements. The company’s Institute for Intelligent Computing recently unveiled EMO, a groundbreaking AI video generator.
EMO, short for Emote Portrait Alive, is a state-of-the-art framework that breathes life into static images. It uses a single reference image and vocal audio to create an animated avatar video, complete with facial expressions and poses.
The team demonstrated EMO’s capabilities with a variety of examples. One such instance involved an AI-generated woman from OpenAI’s Sora debut, who was animated to sing Dua Lipa’s “Don’t Start Now”. Another demonstration featured the iconic Mona Lisa, animated to perform Miley Cyrus’s “Flowers”, as covered by YUQI.
One of EMO’s standout features is its ability to synchronize lip movements in the animated video with the actual audio. This allows the model to support songs in multiple languages. EMO is versatile, working with various artistic styles, including photographs, paintings, and anime-style cartoons. It can also handle different types of audio inputs, such as regular speech.
Interestingly, the audio input doesn’t necessarily have to be original. As demonstrated by Adobe’s new generative AI platform, music can be created from text prompts. This means that generating realistic-sounding voices, as Taylor Swift can attest, is quite straightforward.
EMO, built on a Stable Diffusion backbone, is not the first of its kind, but it’s arguably the most effective. Despite some initial imperfections, such as a noticeable softening effect on skin and occasional awkward mouth movements, the overall accuracy of the lip movements in response to the input audio is impressive.
