From abstract to concrete, researchers use neural networks to transform sketches into photos

The application of the recent fire Prisma may have hit many people's weaknesses. The feeling of transforming ordinary photos into works of art is truly extraordinary. However, anyone who adds a filter will be, but it's not easy to make the abstract concrete. Right now, with the efforts of Dutch researchers, this concept is becoming a reality.

Four neuroscientists at the University of Nijmegen in the Netherlands are using deep neural networks to create new models that allow users to turn ordinary facial sketches into realistic images. The study, called Convolutional Sketch Inversion, began accessing the online paper preprinting platform arXiv (unpublished paper), but it has now received approval from the European Computer Vision Conference.

Neuroscientists have expressed that their models are compatible with various works of art and convert self-portraits into ordinary photos. However, I am afraid that its biggest beneficiary is the police, because originally unclear eyewitness accounts can finally become normal pictures.

“We also made this model inspired by the neural style (an algorithm that lets the machine imitate the painting style of an existing painting to redraw a picture),” says Yağmur Güçlütürk and Umut Güçlü, neuroscientists involved in the study.

Previously, they had converted a landscape map of the city of Tübingen, Germany, into a painting similar to Van Gogh's famous "Starry Night" style. “This experience has given us new thoughts. What will the transition be? What will van Gogh's Starry Night be like?” wrote Güçlütürk and Güçlü.

In research papers, Güçlütürk and Güçlü explained how their neural network software works:

"Next, let's explain how the next neural network transforms sketches into ordinary photos. First, I need to build a huge data set and fill it with various sketches and photos (in pairs). Then I started Show the sketch to the neural network and ask it to convert it into a photo.At first, the accuracy of the neural network is normal.After comparing the converted image with the database, I will point out the mistakes of the neural network.Based on these feedbacks, The network needs to constantly adjust the strategy and eventually give an accurate conversion picture."

Figure Note: The work of the neural network, the left picture is the original, the middle is a sketch, the right is the image converted by the algorithm, the accuracy is indeed quite high.

Practice makes perfect, as long as the repetition is repeated, the model learns to match sketches and pictures.

"We keep repeating the last two steps," said Güçlütürk and Güçlü. "Ultimately, the neural network can finally transform the perfect picture. If all goes well, the future neural network can not only convert the sketches seen, but also have a very high degree of accuracy in the conversion of a new sketch."

In order to continuously train and test this algorithm, neuroscientists also used computer-generated sketches based mainly on the CelebA dataset (large-scale celebrity face-marking dataset containing 200,000+ celebrity pictures) and LFW datasets ( 13000+ network picture).

In addition to experimenting with his own face sketches, neuroscientists also used three self-portraits by famous Dutch artists (Rembrandt, Van Gogh, and Escher).

Figure note: Conversion of self-portraits by three famous artists

Right now, Güçlütürk and Güçlü are considering how to bring this technology to market. It is reported that the art market and the courtroom may be the first areas to bathe in new technologies. Güçlütürk and Güçlü stated: "Our subsidiary has already started preparations and hopes to bring the technology to market as soon as possible."