Tag Archives: Turing Test

Years ago, I found myself caught up in the first Pokémon craze. And I am not ashamed to admit that I fueled that craze in my own special way.

For a couple of years, I put played the game for hours on end on my Gameboy, so much so that I would buy packs of AA batteries to ensure I never ran out. I was not just content with beating the game. I wasn’t even content trying to “catch ’em all.” From red to blue to yellow to silver to gold, I wanted to master every last bit of those games.

Even after all these years, I regret nothing. Those were hours well spent.

But I’m not just bringing up Pokémon for the sake of nostalgia. Recently, it became relevant for an unexpected, but intriguing reason. And it has to do with artificial intelligence.

I know AI has been in the news a lot in recent years, for better and for worse. But it’s definitely newsworthy because this technology is advancing at an incredible pace. It’s no longer this novelty gizmo that can win at games like chess, Go, or pong. Even if it doesn’t have human level intelligence, AI is changing the world. And the pace of that change is likely to accelerate.

It’s now at a point where gauging the advancing capabilities of AI is increasingly difficult. For years, the Turing Test was considered the primary means of testing an AI. But even that has proven limited and incomplete.

That’s where Pokémon comes in. I know that sounds like a weird segway, but it’s actually appropriate. I would even argue that this is a better way to measure the capabilities of an AI.

At this very moment, an AI model called Claude is being tasked with playing the classic version of Pokémon Red. There are even occasional livestreams of it. Having played that game multiple times and beaten it in every way possible, I am genuinely in favor of this. The fact that Claude has been struggling to beat this game, sometimes in hilarious ways, only proves that test has merit.

That’s because Pokémon, as a game, is very different from other games that AI has played. It’s not linear in that the point is to get a character from Point A to Point B, like in Mario, Metroid, or Donkey Kong. It also doesn’t have straightforward rules like Jeopardy, Go, or chess. Pokémon games are a lot more open ended. They’re also a lot more random.

There’s no one way to assemble, train, and develop a team of Pokémon that help you beat every gym and defeat the Elite 4. There are also multiple options on how to start the game. Your first choice is to pick a starter Pokémon in Squirtle, Charmander, or Bulbasaur.

From there, you deal with numerous random encounters in certain areas. And if you try rushing into a match against a gym leader or Team Rocket, you’re likely to lose, even if you use perfect strategy.

It’s also not enough to simply know the map and understand where everything is located. It also takes a measure of planning, patience, and resource management. You need to know which Pokémon to train, which skills to teach, and which matchups are most advantageous.

These are all capabilities that AI has not developed beyond a certain point. There’s a complexity to the game and its mechanics that it has yet to grasp. It shows in just how much the AI has struggled thus far. As I’m writing this, no AI has been able to beat Pokemon in terms of defeating the Elite Four. At times, it has even gotten completely stuck.

That doesn’t mean the AI is a failure or in some ways flawed. It just means that it’s incomplete. It’s not yet at a level where it can process tasks of a certain scope and compelxity. It’s like a child that has learned to walk, but can’t run at a sustained pace. That’s likely to change and change quickly. It’s only a matter of time before an AI like Claude finally beats Pokemon. But after that, what next?

That’s a difficult question to answer at the moment. There are plenty of other games out there that have more compelx mechancis than classic versions of Pokemon. But at some point, playing games just isn’t going to be enough for an AI. Even if it gets to a point where it can beat any video game, that doesn’t mean we’ve achieved a superhuman level of AI. It just means we’ve got an AI that is capable of more complex tasks.

That has many possibilities and implications. But for now, we can only speculate. In the meantime, we can also watch as our best AI tries to catch ’em all. Some might be cheering for it. Others may hope it keeps failing. We don’t yet know when or whether an AI will achieve the coveted title of Pokemon Master. But if it ever does, then chances are it’ll be ready to become a master at many other challenges.