Damaged Rembrandt painting gets restored by AI

17th-century famous Dutch painter Rembrandt Harmenszoon van Rijn's famed painting The Night Watch, which disfigured after the artist's death, has been made complete through the use of artificial intelligence.

The Rijksmuseum in the Netherlands has owned The Night Watch since it opened in 1885 and considers it one of the best-known paintings in its collection. In 2019, the museum embarked on a multi-year, multi-million-dollar restoration project, referred to as Operation Night Watch, to recover the painting, reports Art News. 

The effort marks the 26th restoration of the work over the span of its history.

In the beginning, restoring The Night Watch to its original size hadn't been considered until the eminent Rembrandt scholar Erst van der Wetering suggested it in a letter to the museum, noting that the composition would change dramatically. The museum tapped its senior scientist, Rob Erdmann, to head the effort using three primary tools: the remaining preserved section of the original painting, a 17th-century copy of the original painting attributed to Gerrit Lundens that had been made before the cuts, and AI technology. 

"There's nothing wrong with having an artist recreate [the missing pieces] by looking at the small copy, but then we'd see the hand of the artist there. Instead, we wanted to see if we could do this without the hand of an artist. That meant turning to artificial intelligence," Erdmann said. 

AI was used to solve a set of specific problems, the first of which was that the copy made by Lundens is one-fifth the size of the original, which measures almost 12 feet in length.

The other issue was that Lundens painted in a different style than Rembrandt, which raised the question of how the missing pieces could be restored to an approximation of how Rembrandt would have painted them. Erdmann created three separate neural networks, a type of machine learning technology that trains computers to learn how to do specific tasks to address the problems.

"The first [neural network] was responsible for identifying shared details. It found more than 10,000 details in common between The Night Watch and Lundens's copy." Erdmann said.

Erdmann made a test for the neural network, similar to flashcards, by splitting up the painting into thousands of tiles and placing matching tiles from both the original and the copy side-by-side. The AI then had to create an approximation of those tiles in the style of Rembrandt. Erdmann graded the approximations—and if it painted in the style of Lundens, it failed. After the program ran millions of times, the AI was ready to reproduce tiles from the Lundens copy in the style of Rembrandt. 

The AI's reproduction was printed onto canvas and lightly varnished, and then the reproduced panels were attached to the frame of The Night Watch over top the fragmented original. The reconstructed panels do not touch Rembrandt's original painting and will be taken down in three months out of respect for the Old Master. 

"It already felt to me like it was quite bold to put these computer reconstructions next to Rembrandt," Erdmann added.

As for the original painting by Rembrandt, it may receive conservation treatment depending on the conclusions of the research being conducted as part of Operation Night Watch.

The painting has sustained damaged that may warrant additional interventions. In 1975, the painting was slashed several times, and, in 1990, it was splashed with acid. The reconstructed painting went on view at the Rijksmuseum on Wednesday and will remain into September.