Nvidia AI research points to an evolution of the chip business
What happens as more of the world’s computer tasks get handed over to neural networks?
That’s an intriguing prospect, of course, for Nvidia, a company selling a whole heck of a lot of chips to train neural networks.
The prospect cheers Bryan Catanzaro, who is the head of applied deep learning research at Nvidia.
“We would love for model-based to be more of the workload,” Catanzaro told ZDNetthis week during an interview at Nvidia‘s booth at the NeurIPS machine learning conference in Montreal. Catanzaro was the first person doing neural network work at Nvidia when he took a job there in 2011 after receiving his PhD from the University of California at Berkeley in electrical engineering and computer science.
Model-based is a shorthand for replacing some of what used to be explicit programming with neural networks that infer the way to solve a computing problem. There were examples of that shift at Nvidia‘s booth that have broad implications for computing.
An example is a paper presented at the conference this week called Video to Video Synthesis, authored by himself and colleagues, along with a researcher from MIT‘s Computer Science and Artificial Intelligence Lab. The work seeks to synthesize videos of a street scene by “predicting future frames of video.”
Traditionally, such tasks would be programmed by hand, the task of “rendering” being a laborious one. “Today video is triangle by triangle, and can cost a million dollars” or more, says Catanzaro.
Instead, the new approach takes videos of street scenes and feeds some of the frames to a generative adversarial network, or GAN, which then predicts frames of video. It builds upon prior work that synthesized images of things given a few samples.
The important part is that the GAN is figuring out the task of rendering, replacing laborious physics specifications in the traditional approach.
Nvidia has taken the approach to video game-style simulations, with some interesting results. In the Nvidia booth at NeurIPS, an arcade-style driver’s seat was set up and visitors had the chance to drive through a simulated street scene. The street scene in this case showed some of the current limitations of the state of the art. Rather than looking photo-realistic, it had the feel of a water color painting, with colors and textures of buildings and cars shifting as one drove through the simulation.
Those artifacts, says Catanzaro, are a reflection of issues between translating from the “Unreal Engine 4” 3D system that is training the GAN in the case of the driving simulation. “When Unreal gives us the rendered world to begin with, and it doesn’t do the full rendering, it just creates a sketch, it’s too precise and everything is perfect.
“But lines and sketches from real videos are actually better, they are a little wavy; we need the rendering to be more like real video.
