LAST year Artur Filipowicz, a computer scientist at Princeton University, had a stop-sign problem. Dr Filipowicz is teaching cars how to see and interpret the world, with a view to them being able to drive themselves around unaided. One quality they will need is an ability to recognise stop signs. To that end, he was trying to train an appropriate algorithm. Such training meant showing this algorithm (or, rather, the computer running it) lots of pictures of lots of stop signs in lots of different circumstances: old signs and new signs; clean signs and dirty signs; signs partly obscured by lorries or buildings; signs in sunny places, in rainy places and in foggy ones; signs in the day, at dusk and at night.
Obtaining all these images from photo libraries would have been hard. Going out into the world and shooting them in person would have been tedious. Instead, Dr Filipowicz turned to “Grand Theft Auto V”, the most recent release of a well-known series of video games. “Grand Theft Auto V” is controversial because of its realistic portrayal of crime and violence—but from Dr Filipowicz’s point of view it was ideal, because it also features realistic stop signs. By tinkering with the game’s software, he persuaded it to spit out thousands of pictures of these signs, in all sorts of situations, for his algorithm to digest.
Dr Filipowicz’s stop signs are one instance of the fondness that students of artificial intelligence (AI, of which machine vision is an example) have for video games. There are several reasons for this popularity. Some people, such as Dr Filipowicz, use games as training grounds for the real world. Others, observing that different games require different cognitive skills, think games can help them understand how the problem of intelligence may be broken down into smaller, more manageable chunks. Others still, building on these two observations, think games can help them develop a proper theory of artificial (and perhaps even natural) intelligence.
For all of this to happen, though, the games themselves have first to be tweaked so that they can be played directly by another computer program, rather than by a human being watching the action on a screen. “Grand Theft Auto V”, for instance, can be turned from a source of pictures of road signs into a driving simulator for autonomous vehicles by bolting onto it a piece of software called “Deep Drive”. This lets the driving and navigation programs of such vehicles take control—a cheaper and safer way of testing driving software than letting it loose on roads.
Games companies are beginning to understand this. In June 2015, for instance, Microsoft started Project Malmo, an AI-development platform based on a popular “world-building” game called “Minecraft” that it had recently purchased. In November 2016 Activision Blizzard, owners of “Starcraft II”, a science-fiction strategy game in which players build and command human and alien armies, announced something similar in collaboration with DeepMind, an AI firm owned by Alphabet, Google’s holding company.
The following month, with the permission of the owners involved, a privately financed research group in San Francisco, called OpenAI, released “Universe”. This is a piece of software, free for all to use, which features hundreds of games presented in ways that mean they can be played directly by appropriate programs. The offerings in “Universe” range from bestselling, big-budget titles such as “Portal 2” (a physics-based puzzle game) to cheap-and-cheerful web games like “Bubble Hit Pony Parade” and “James the Space Zebra”.
One of Microsoft’s hopes in starting Project Malmo was to teach AI software to collaborate with people. To this end, Katja Hofman, the project’s head, is trying to use “Minecraft” to create an advanced personal assistant. Her goal is software that can anticipate what its human operator wants, and help him achieve it.
