Researchers Employ AI to Transform 134-Year-Old Picture into 3D Reconstruction of Missing Temple Relief Detail
A group of computer scientists recently created 3D recreations of misplaced relief panels at a UNESCO World Heritage Site using artificial intelligence.
The scientists constructed a neural network that can transform a single 2D photo of a 3D object into a digital 3D reconstruction. Essentially, they invented a 21st-century version of a stereoscope. They presented this prototype at the 32nd ACM Multimedia conference last month.
For their research, the experts employed pictures of reliefs from Indonesia's Borobudur temple, a UNESCO World Heritage Site. The temple is adorned with 2,672 reliefs, making it the largest collection of Buddhist reliefs worldwide. In the late 19th century, the temple's bottom portion was reinstalled, concealing 156 reliefs behind stone walls, and they remain hidden now. However, grayscale photos were taken of each panel before they were buried. Surprisingly, the team's neural network effectively reconstructed one of those now-hidden reliefs utilizing a 134-year-old black-and-white photo.
[Image: Pan et al. 2024]
Earlier attempts to restore the reliefs failed to replicate their fine details because of the loss of depth value compression. This means that these 3D reliefs have intricate details from the closest carvings to the viewer and the farthest from the viewer. Previous reconstruction attempts flattened out these varying depth details. The team referred to these fine details as "soft edges" and mapped them by calculating the curvature changes in the 3D space.
The team concluded that the edge map was reducing the model's accuracy as it failed to convey the changes in 3D curvature appropriately. The way it was incorporated into the network also minimized its effect on estimating depth in physical objects.
A soft-edge map (left) and semantic map (right) of the 2D relief image were depicted in the new paper.
"Although we achieved 95% reconstruction accuracy, finer details such as human faces and decorations were still missing," said Satoshi Tanaka, a researcher at Ritsumeikan University in Japan and co-author of the study, in a university release. "This was due to the high compression of depth values in 2D relief images, making it difficult to extract depth variations along edges. Our new method addresses this by enhancing depth estimation, specifically along soft edges, using a new edge-detection approach."
The images above represent the team's best experimental results for a soft-edge map (left) and a semantic map (right) of the sample relief, compared to the original data (top row). The edge map tracks the points where curves in the relief give it depth, which confuses earlier models.
The semantic map, reminiscent of Ellsworth Kelly's Blue Green Red, shows how the model's knowledge base associates related concepts. In this image, the model distinguishes the foreground features (blue), human figures (red), and background. The researchers also included how their model compared with other state-of-the-art models in relation to the original images.
AI has its drawbacks but has proven helpful in resolving issues in image recognition and preserving cultural heritage. In September, another team utilized a neural network to identify previously unseen details in Raphael's panels, and another team used a convolutional neural network to nearly double the number of known Nazca lines—famous Peruvian geoglyphs.
The new model can process multi-modal understanding, meaning it can take in different types of data to understand its target object better. In this case, the soft-edge detector utilized to measure curves in the relief doesn't only use slight brightness changes to perceive depth but also screams and walls. Using both types of data allowed the new model to create a more detailed 3D reconstruction of the relief than previous attempts.
"Our technology has vast potential for preserving and sharing cultural heritage," Tanaka said. "It opens up new possibilities not only for archaeologists but also for immersive virtual experiences through VR and metaverse technologies, preserving global heritage for future generations."
Culture needs to be preserved, but some cultural artifacts are particularly at risk. While AI-generated reconstructions can't replace the true originals, they can have their uses. Neural networks like the one described in the paper could potentially resurrect lost artifacts like the Bamiyan Buddhas, which were destroyed by the Taliban in 2001, if only in an augmented or virtual reality environment.
The models could also be used to preserve cultural artifacts on the brink of destruction, such as centuries-old indigenous carvings on boab trees in Australia's Tanami Desert.
Cultural heritage shapes who we are by reflecting the communities and cultures that came before us. If these AI models aid art historians and preservationists in saving even one historical artifact, they've achieved something worthwhile. However, it's worth noting that AI models require a considerable amount of energy, which could indirectly contribute to the loss of cultural heritage. But using technology for positive ends is still the right thing to do—especially when it comes to preserving irreplaceable artifacts.
The advancements in tech and AI have the potential to revolutionize the future of artificial reconstruction in preserving cultural heritage. For instance, the newly developed technology can not only create 3D replicas of ancient artifacts but also enhance their fine details, such as human faces and decorations, that were previously missed.
In the realm of artificial intelligence, the future of technology lies in its ability to process multi-modal understanding, utilizing various types of data to improve image recognition and reconstruction results, as demonstrated by the recent success in recreating the Borobudur temple reliefs.
