Microsoft Research has introduced Muse, a generative AI model designed to simulate video game visuals and player actions. Published in Nature, this research presents the World and Human Action Model (WHAM), which enables AI to generate game environments and predict player movements. The project is a collaboration between Microsoft Research Game Intelligence, Teachable AI Experiences (Tai X), and Xbox Game Studios’ Ninja Theory.
To support further research and development, Microsoft is open-sourcing the model weights, sample data, and an interactive prototype called the WHAM Demonstrator. These resources are available on Azure AI Foundry.
Understanding Muse’s Capabilities
Muse can generate extended gameplay sequences based on an initial prompt—such as a short clip of human gameplay and corresponding controller inputs. It operates in a “world model mode,” predicting how the game environment will evolve. The goal is to create AI models that accurately reflect real gameplay dynamics.
Key Features:
- Generates visuals and actions for simulated gameplay.
- Produces long, coherent game sequences.
- Learns from real player interactions.
Motivation Behind the Research
The inspiration for Muse began in late 2022, following the public release of ChatGPT. The research team explored how similar AI techniques could be applied to video games, leveraging gameplay data from Bleeding Edge, a multiplayer game developed by Ninja Theory.
The dataset includes game visuals and player actions, collected with user consent. The collaboration ensured compliance with ethical and legal standards for data use.
Training and Development
Muse was trained using over 1 billion images and controller actions, equivalent to seven years of continuous human gameplay. The process involved:
- Scaling up model training from V100 GPUs to H100 GPUs.
- Improving the representation of controller actions and game visuals.
- Refining the model through iterative updates to enhance accuracy.
A key breakthrough was the model’s ability to generate consistent gameplay visuals over time, improving with each training iteration.
Evaluating Muse’s Performance
To ensure practical applications, researchers identified three key evaluation criteria:
- Consistency – The model must follow game physics and mechanics, ensuring actions like movement and interactions align with real gameplay.
- Diversity – Muse should generate varied gameplay scenarios in response to the same prompt, reflecting different possibilities.
- Persistency – User modifications (such as adding a new character) should be retained in subsequent generated sequences.
The WHAM Demonstrator provides an interactive way to test and refine these capabilities.
Future Applications and Open Research
Muse represents a step toward AI-assisted game development, offering potential applications in:
- Game ideation – Helping developers visualize new gameplay scenarios.
- AI-driven game testing – Simulating how a game might evolve under different player behaviors.
- Creative tools – Allowing game designers to explore interactive AI-generated content.
By open-sourcing Muse, Microsoft encourages collaboration and further advancements in AI-generated gameplay.
