Why Will GameNGen Change How You Play Video Games Forever

PLUS: Turn Your Dull Pitches into Unforgettable Stories

AuthorAuthor

Tezan Sahu & Sandra Anil
September 3rd, 2024

Howdy fellas!

Today, Spark & Trouble have the perfect blend of creative inspiration and tech innovation.

Hereā€™s a sneak peek into todayā€™s edition šŸ‘€

  • 3 jaw-dropping AI tools that you just canā€™t miss šŸŖ„

  • Convert your boring pitches into memorable stories using our fresh prompt templatešŸ§ā€ā™€ļø

  • DOOM reborn: How AI is revolutionizing video games šŸŽ®

Time to jump in!šŸ˜„

PS: Got thoughts on our content? Share 'em through a quick survey at the end of every edition It helps us see how our product labs, insights & resources are landing, so we can make them even better.

10x Your Workflow with AI šŸ“ˆ

Work smarter, not harder! In this section, youā€™ll find prompt templates šŸ“œ & bleeding-edge AI tools āš™ļø to free up your time.

3 AI Tools You JUST Can't Miss šŸ¤©

  • šŸ—ŗļø Guidde: Gen AI platform to create video documentation 11x faster

  • šŸ“¹ Minvo: Create stunning pro-clips from YouTube videos with 3 clicks

  • āœ’ļø GravityWrite: Create Blogs, Videos, Ad copies, Social Media Posts, Images and more in seconds!

Fresh Prompt Alert!šŸšØ

You know that moment when youā€™ve got a killer product idea, but itā€™s just a jumble of thoughts and possibilities?

Time to turn those sparks into a story! This week's Fresh Prompt Alert is your go-to for crafting a compelling storyboard. Imagine your product coming to life, panel by panel, showing exactly how it will revolutionize your userā€™s world.

Ready to weave some magic? Grab this prompt, and letā€™s start telling stories that sell šŸ‘‡

Act as a seasoned marketer & a compelling storyteller.

To help understand the product and how it would work in reality, tell a compelling story about [the product] in the form of a storyboard.

Hereā€™s a description of the product: [product description]

Generate a [number of panels]-panel storyboard outline using the [storytelling framework] framework to explain how this product will transform the userā€™s life.

Represent this storyboard outline in a tabular format with columns for:

- Panel number
- Stage of the storytelling framework
- Description of the visual
- Detailed prompt that I can use in a text-to-image generation tool to obtain the visual
- Voice-over for the panel

* Replace the content in brackets with your details

We tried this template for ā€œThe Vision, Debuggedā€ā€¦
And hereā€™s the result: Check it out

Cranking this up a notch, weā€™ve also generated the images using the exact image prompts provided in the responseā€¦and thatā€™s resulted in this cool storyboard! šŸ‘‡

Meet Sarah, a tech enthusiast who wants to stay updated in the fast-paced world of AI, but she's drowning in information overload

But then Sarah discovers 'The Vision Debugged'ā€”a concise, engaging newsletter that decodes the latest in AI, making it easy to stay informed.

Equipped with cutting-edge insights and case studies from 'The Vision Debugged,' Sarah now brings game-changing ideas to the table.

With every edition, Sarah evolves from a tech enthusiast to an AI thought leader, recognized for her expertise and innovative ideas.

Not only has Sarah transformed her career, but sheā€™s also paying it forwardā€”mentoring the next generation with insights from 'The Vision Debugged.'

Transform your career, one insightful edition at a time. Subscribe to 'The Vision Debugged' and stay ahead in the world of AI.

Ainā€™t that amazing!? šŸ˜

Now itā€™s your turn!
Try it out for any product of your choice & share your stories with us!

Hot off the Wires šŸ”„

We're eavesdropping on the smartest minds in research. šŸ¤« Don't miss out on what they're cooking up! In this section, we dissect some of the juiciest tech research that holds the key to what's next in tech.āš”

Remember the first time you fired up Breath of the Wild (The Legend of Zelda series), marvelling at its vast open world? Or perhaps you've lost yourself driving through the intricate cityscape of GTA V?

As gamers, we've witnessed the jaw-dropping evolution of video game graphics over the years. From the pixelated charm of Super Mario to the photorealism in Asphalt 9: Legends, game engines have come a long way in crafting immersive virtual worlds.

But what if I told you that the next revolution in gaming isn't about more polygons or ray tracing, but about AI taking the wheel?

Enter GameNGen: The AI-Powered Game Engine. Researchers from Google Research, Tel Aviv University, and DeepMind have just dropped this game-changer (pun intended). It's the first game engine fully powered by AI that lets players interact with complex environments in real time, maintaining high quality even during marathon gaming sessions.

To put it in perspective, GameNGen can simulate the iconic first-person shooter "DOOM (the 1993 edition)" at over 20 frames per second. That's right, we're talking about AI rendering a 3D game environment on the fly!

Sample gameplay of DOOM (1993)

Forging the fundamentals

But before we dive deeper, let's break down some gamer lingo for our non-button-mashing friends:

šŸŽ® Game Loop: The heartbeat of every game, consisting of three steps: (1) gather user inputs, (2) update the game state, and (3) render it to screen pixels. This loop, running at high frame rates, creates the illusion of an interactive virtual world.

šŸ–„ļø Game Engine: The software that processes scenes and renders a stream of images in response to a player's actions. It's the behind-the-scenes maestro that keeps track of player position, movement, objects, and game logic. Think Unity or Unreal Engine.

šŸ¤– Reinforcement Learning (RL) Agents: AI programs that learn by interacting with an environment, much like how we humans learn through trial and error.

šŸŒ€ Diffusion Models: A type of AI model that learns to generate data by gradually denoising random noise (missed our earlier edition where weā€™ve talked in detail about diffusion? No worries - here you go!)

šŸ–¼ļø DDIM sampling: DDIM Sampling is a method that quickly generates high-quality images by efficiently removing noise step-by-step. Its speciality lies in producing great results with fewer steps, making it faster than other methods. Hereā€™s a deep dive for the enthusiastic fellas!

šŸ§  TPU (Tensor Processing Unit): It is a specialized chip designed specifically for AI tasks, particularly machine learning. Unlike a GPU, which is versatile for graphics and various computations, a TPU is optimized solely for the specific mathematical operations common in AI, making it faster and more efficient for these tasks.

So, whatā€™s new?

Currently, all game engines rely on a game loop & a set of manual rules, painstakingly programmed by developers, to handle everything from physics simulations to character behaviors. Also, despite the impressive advancements in AI-generated videos, they've remained too slow for real-time applications like gaming.

GameNGen flips this script entirely, using AI to handle both the game state updates and rendering in real time. By using an augmented version of Stable Diffusion v1.4 it is able to generate frames of the DOOM game at an acceptable rate, with quality comparable to the original game.

But how does it work? Glad you asked!

Under the hoodā€¦

At its core, GameNGen is powered by an augmented version of Stable Diffusion v1.4, a powerful AI model typically used for generating images from text descriptions.

However, the researchers had to overcome some significant hurdles to make this work for real-time gaming:

  1. Training Data: You can't just feed a regular diffusion model some game footage and expect it to spit out playable levels. The researchers needed vast amounts of diverse gameplay data.
    Their solution? Train a reinforcement learning (RL) agent to play the game, focusing not on high scores, but on exploring as many different scenarios as possible (think of it as an AI tourist in the world of DOOM, taking snapshots of everything it encounters). These snapshots (about 900 million frames!) were then used to train a special diffusion model.

  2. From Text to Action: Unlike standard Stable Diffusion, which generates images based on text prompts, GameNGen needed to respond to player actions. The team cleverly encoded these actions (like key presses) into "action embeddings" that the model could understand.

  3. Memory and Context: To maintain consistency, the model considers the previous 64 frames along with recent actions when generating each new frame. These observed frames are encoded in a latent space & concatenated to capture their essence. Also, some noise was added to these previous frames & the model was incentivized to correct these frames sequentially

  4. Speed Optimization: Using a technique called DDIM sampling for 4 steps and leveraging the power of TPUs (Tensor Processing Units), the team managed to generate each frame in about 50 milliseconds, achieving that crucial 20 fps benchmark.

Thatā€™s what the overall training architecture looked like (source: GameNGen paper)

Why does this matter?

GameNGen isn't just a tech demo ā€“ it's a proof-of-concept that could revolutionize how we think about game development. This technology opens up exciting possibilities:

  • Infinitely varied game worlds that never play the same way twice

  • NPCs with truly dynamic behaviours, learning and evolving based on player actions

  • Reduced development time for creating expansive game environments

  • Easier customization and modding capabilities for players

And it's not just about gaming. This technology could revolutionize fields like:

šŸ•¶ļø Virtual Reality: More responsive and realistic VR environments.
šŸš— Autonomous Vehicles: Better real-time simulation for training and testing.
šŸ™ļø Smart Cities: Dynamic urban planning and traffic management simulations.

Of course, we're not quite ready to toss out our PS5s just yet. GameNGen still has some hurdles to overcome:

Limited memory: The current 3-second "memory" might not be sufficient for more complex games.

AI vs. Human Play: There may be discrepancies between how the RL agent plays versus real human players, potentially leading to unexpected behaviors.

Computational Requirements: Running these models in real-time requires some serious processing power.

While it might be a while before we're playing "Grand Theft Auto: AI Edition," GameNGen proves that AI-powered game engines are more than just a pipe dream.

The next time you boot up your favourite game, take a moment to appreciate the intricate code making it all possible ā€“ because someday soon, that code might just be replaced by a very clever AI.

Spark 'n' Trouble Shenanigans šŸ˜œ

Scrolling through LinkedIn, we came across this unique fusion of the 2024 Paris Olympics with the extraordinary world of Marvel & DCā€¦with superheroes featuring as athletes in real-life events.

Check out this fascinating work by the creative wiz Tapan Aslot:

Virtual Heroes Celebrating the Real Superheroes - AI Artwork by Tapan Aslot (source: LinkedIn)
Tap the image to view the original post & artworks

Well, thatā€™s a wrap!
Thanks for reading šŸ˜Š

See you next week with more mind-blowing tech insights šŸ’»

Until then,
Stay CuriousšŸ§  Stay AwesomešŸ¤©

PS: Do catch us on LinkedIn - Sandra & Tezan

Reply

or to participate.