AI Powered VR Avatars Are INSANE | RTX 4080 Giveaway!!

I’m no game designer, but I made a VR character that kind of looks like my wife Chary and speaks naturally like her, in less than an hour, with NVIDIA’s AI tools. When I asked her who she was, she said: “How could you forget that I’m your wife?” This is a more experimental video about AI-powered avatars: how I made her, and what this tech could mean for VR games soon.

NVIDIA sponsored this video, which also let us run an RTX 4080 giveaway at the time. They didn’t direct what I say about their tools.

Why NVIDIA is leading on AI

In early 2023 artificial intelligence is the talk of the year: ChatGPT, Bing AI, Synthesia and image generators do in seconds what usually takes a lot of time. They paint whatever complex picture you have in mind, help you code, write a video script, even generate whole music videos. Researching this video blew my mind, and it made me think about how AI could change VR and gaming in general.

One company is leading the way, and you can probably guess it: NVIDIA. You probably know them from their gaming GPUs, but they saw AI coming and have invested in it for over a decade, back when nobody believed it was a sustainable bet. Now every chip they make is focused on AI, and they power ChatGPT, which reached 100 million active users two months after launch.

Omniverse and ACE

NVIDIA’s latest project is Omniverse ACE, short for Avatar Cloud Engine: a set of developer tools for 3D game characters and avatars. It brings text-based AI like ChatGPT to life with animated faces that act a lot like humans. NVIDIA believes this is how we’ll talk to AI in the future, and if that works out, it may also be how we customise our own appearance in VR. It isn’t far off, either: it’s here now and free, and anyone can use it.

I know an AI with a face sounds a bit Terminator, but NVIDIA isn’t Skynet. These tools are meant to make it easier for developers, or anyone really, to work on metaverse projects.

What Omniverse is

Omniverse is a real-time 3D design collaboration and digital twin platform. That’s a mouthful, but it’s basically what you’d expect a metaverse to be: not a walled garden, but something more like Ready Player One. It’s one virtual universe that connects to virtual worlds owned by others, and it’s open, so things work together.

That’s possible because it uses USD, an open standard first invented by Pixar (yes, Disney’s animation studio). USD connects different tools, assets and projects, so people can collaborate remotely without being stuck on one platform. The idea is that everyone can build complex 3D worlds and connect them to ACE-powered AI. That’s what NVIDIA is after, and I think it’s pretty cool.

How I made my AI-powered avatar of Chary

1. Audio2Face, an AI service in ACE, generates facial motion and lip sync in real time from audio. It even matches expressions and emotions to the words, and I have full control: I can make her look amazed, angry, happy or sad.
2. MetaHuman in Unreal Engine is where I connected it to make her look beautiful. Because it’s Omniverse, I could have used Unity or Ready Player Me (the popular avatar creator for VRChat) instead. It doesn’t matter where you design: you can take your avatar from one tool to another and keep working on it.

If I can do this without being a game designer, imagine what game designers can do.

What else these AI tools can do

• Conversation. Add conversational AI like Riva speech so the character holds a natural conversation or answers complex questions.
• Custom voices. It can create a natural custom voice from only 30 minutes of voice data, so I could speak and make it sound like Chary, or create a new voice to narrate a story.
• Storylines. Omniverse has an app to generate storylines based on existing game content.
• Motion. Give the AI a random video of someone moving, and it estimates the physics-based motion and puts it on your character.
• Variations. It can generate variations of your avatar from a single 2D example image.
• Live translation. Translate your voice into several languages in real time, even during a video call.

What AI-powered avatars could mean for VR games

One of the biggest things AI can do is make games more engaging and challenging based on your personality and skill. For example:

• Enemies that adapt and learn from your actions, so they’re harder to beat.
• NPCs that look and react like real people, perceive the game world and act on your requests, like carrying out a battle strategy or showing you the way somewhere.
• Dialogue generated on the fly, or a storyline that adjusts to your choices, for more surprising stories.

And that’s only avatars, which scratches the surface. AI apps are popping up like mushrooms, and plenty of them aren’t NVIDIA’s.

AI has real dangers too. With this tech people could easily impersonate others, and it’s important to be aware of that risk. But NVIDIA is making it easier and more accessible in an open way, which I can stand behind, so I hope more developers start using it.

How to try Omniverse ACE

Omniverse ACE is free to use, but when we made this video in March 2023 it was in early access: you had to sign up for an account and get approved before the public release. I think they only ask for some technical ability, so it should be an easy entry. You need an RTX GPU to run Omniverse.

The RTX 4080 giveaway and GTC

To go with this, NVIDIA let us give away an RTX 4080 graphics card to our viewers. To enter you had to subscribe, comment what you think about NVIDIA’s AI, register for NVIDIA’s free GTC developer conference with our link, and watch at least one virtual session (not the keynote) during GTC, which ran online from March 20 to 23, 2023. You needed a permanent address in Europe, the Middle East or Africa. The giveaway is over now.

GTC is NVIDIA’s free conference about breakthroughs in AI, the metaverse and beyond, with online training, insights and access to experts on generative AI and GPU programming, including talks about Omniverse ACE.

Why I’m excited

I’m proud to partner with NVIDIA. I’ve used their GPUs all my life, and they’ve brought me a lot of joy because they let me play games. Making a talking, emoting version of Chary in under an hour, without any game design skills, showed me how close AI characters in VR games already are. As long as it stays open and people stay aware of the risks, I hope a lot more developers start using it.