Playtesting

AI Ran Out of Internet. Now It's Learning by Playing Games Again.

Progress in AI depends on finding new data and new challenges for AI

Welcome to the first edition of Playtesting, our new column in which Alex Duffy explores how games can help make AI smarter and more beneficial for people. In addition to being a contributing writer, Alex is the cofounder of Good Start Labs, a company dedicated to using games to improve AI that was incubated at Every.—Kate Lee

Was this newsletter forwarded to you? Sign up to get it in your inbox.

Earlier this year a version of Gemini won a gold medal at the International Mathematical Olympiad (IMO). Some models can diagnose certain medical conditions as well as human physicians. And AI is helping us forecast weather faster and more accurately without the need for supercomputers.

But in other areas, performance is just plain bad. AIs used in legal research, for example, have fabricated hundreds of facts and even whole cases that don't exist, leaving attorneys who've trusted models facing fines and other liabilities.

It doesn't have to be this way. It's all down to the data. This era of generative AI was trained on publicly available information scraped from the internet—a biased dataset rich in some domains of knowledge, and wanting in others. And now that they've hoovered that up, it will take years to generate more high-quality knowledge to ingest and create more reliable outputs for every user. AI models are, as a result, incredibly good at tasks where they have lots of high-quality examples, and weak at those they don't. This phenomenon is often described using the metaphor of a jagged frontier.

AI could answer routine medical questions, triage symptoms, or explain test results while doctors focus on complex problems and novel treatments. But we can't deploy it in critical areas such as healthcare, legal research, or financial advising if it's not reliable. If we want to realize the promise of AI, this jagged frontier needs to be filled in. Games can help make that happen.

Build a project in a single day

AI can feel abstract until you build something real—and tools "made for programmers" can stop you before you start. Spend one day with Every's Dan Shipper to get acquainted with the best tools on the market, assign your AI agents real tasks, and ship something live. Claude Code for Beginners runs Nov. 19, live on Zoom. You'll leave with a project, a reusable workflow, and a head start on building the agentic apps that will populate the future.

Save your seat now

Want to sponsor Every? Click here.

In a game, we can create any scenario—a negotiation, a crisis, a moral dilemma, a portfolio to manage—and watch exactly how the AI responds.

I've learned so much from games. Runescape taught me how to type, how markets work, and how to not get scammed. Redstone in Minecraft taught me about circuits long before my electrical and computer engineering degree. League of Legends taught me collaboration under pressure and awareness, and almost everyone I've asked has similar stories about games.

We can iterate with games until the model does what you need. Maybe you want a model to lie less, get better at using many different tools, or be funnier. These synthetic playgrounds are how this generation of AI grows up and works better for people.

Games teach us what AI can and cannot do, so it can learn to do more things for us that fit our preferences. After years playing with AI and watching AI play, I've learned why...

Become a paid subscriber to Every to unlock this piece and learn about:

The data crisis forcing AI labs to rethink how models learn
What happened to professional Go players after AI crushed the world's best in 2016
Why Alex's startup is using a Cards Against Humanity-style game to solve AI's alignment problem