Algorithm Clones Van Gogh’s Artistic Style and Pastes It onto Other Images, Movies
A deep neural network has learned to cut and paste artistic styles onto other images.
May 10, 2016
The nature of artistic style is something of a mystery to most people. Think of Vincent Van Gogh’s Starry Night, Picasso’s work on cubism, or Edvard Munch’s The Scream. All have a powerful, unique style that humans recognize easily.
But what of machines? Deep neural networks are revolutionizing the way machines recognize and interpret the world. Machine vision now routinely outperforms humans at tasks such as object and face recognition, something that was unimaginable just a few years ago.
Recently, these devices have taken the first tentative steps toward recognizing artistic style and even reproducing it. Just how far this kind of work can go hasn’t been clear. For example, is it possible to copy and paste an artistic style from one image onto an entire video, without producing artefacts that ruin the visual experience?
Today, Manuel Ruder and pals at the University if Freiburg in Germany show that exactly this is possible. These guys take famous works of art such as Starry Night and The Scream and transfer their style to a range of video sequences taken from movies such as Ice Age and TV programs such as Miss Marple. The result is an impressive rerendering of videos and the possibility of doing it in almost any style imaginable.
Deep neural networks consist of many layers that each extract information from an image then pass on the leftover data to the next layer. The first layers extract broad patterns such as color and the deeper layers extract progressively more detail, which allow object recognition.
The information extracted by the deeper layers is important. It is essentially the content of an image minus the contextual information such as color, texture, and so on. In a way, it is the computer equivalent of a line drawing.
Last year, Leon Gatys at the University of Tubingen and a few pals began studying artistic style in this way. They discovered that it is possible to capture artistic style by looking not at the information in each layer but at the correlations between layers. So the way an artist reproduces a face is correlated with the way he or she reproduces a tree or a house or the sun. Capturing this correlation also captures the style.
But their key discovery was that the content of an image can be completely separated from the artistic style. What’s more, they found they could take this artistic style and copy and paste it onto the content of any other image.
All of a sudden it becomes possible to capture the abstract style of a Kandinsky and paste it onto a picture of your cat. That’s a lot of fun.