OUR Shurijo: Rebuilding Shurijo Castle with Memories

OUR Shurijo Project

About a year ago on October 31, 2019, a fire destroyed the Shuri Castle, an icon of Okinawan culture in Japan. I am a computer vision researcher who has worked in cultural heritage preservation, and I launched a project soon after this tragedy with a group of people who gathered on social media. The members include researchers, students, engineers, artists, and writers, who are involved with computer vision, human-computer interaction, virtual/augmented reality, and Okinawan history.

The project is referred to as OUR Shurijo: Shurijo Castle Digital Reconstruction project, which aims to collect photographs and videos of the castle along with the memories from the public, and to digitally reconstruct 3D models of the castle so that they can be used as alternative touristic resources. We are particularly interested in sharing and visualizing memories using the 3D models, since adding memories to a 3D model can give a hint of human presence and their lives. Namely, we aim to comprehensively involve the person who provided the photographs, appeared in the photographs, and who will see the photographs in the experience of the 3D model. Therefore, we appealed to the public to provide not only photographs but also information such as the shooting date, memories of that day, age, gender, and countries and regions of the providers.

The data has been collected through a website in 6 languages, with about 40,000 users accessing the site in a year, and about 85,000 images (including those extracted from videos) from 35 countries and regions have been received. There are about 3,000 unique contributors, half of whom sent their memories, with an average of about 100 characters per person. 67% of the contributions came from Japan, and 20% came from Taiwan.

Collected memories

The texts that have been sent to us so far contain various memories, such as school trips, honeymoon trips, and trips with friends and family. I have looked at the data many times, and it is like seeing people’s lives on a runway.

We have received many wonderful memories but let me introduce two of them. The group photo on the left of the last two photographs is from a woman in her 20s from Saitama Prefecture, who posted as follows: “I participated in a peace seminar where students from China, Taiwan, Germany, and Japan gathered to reflect on history and exchange ideas for peace. Through visiting various places and interacting with local people, we learned the importance of talking directly with others regardless of their historical background, respecting them even if you don’t understand what they say, and reaching out to them when they need help. We took this picture at the end of the seminar when we all went sightseeing and gained important insights.”

The family photo on the right is from a man in his 40s from Osaka Prefecture who provided the following memory: “Every holiday since 2018, my family has been traveling to the 100 greatest castles in Japan, and in April 2019, at the end of the Heisei era, we visited Shurijo Castle to conclude our trip. At Shurijo Castle, we celebrated by breaking the mini kusudama that we had brought with us, and the people around us applauded us, making it a wonderful memory.”

Creation and publication of the models

The models we have created so far are now available on the Sketchfab platform. We have created the Main Hall, the King’s Throne, the King’s Crown, and Shurijo Castle Park. Most of them are available for download and can be used with proper citation.

The Main Hall

The King’s Throne

The King’s Crown

Shurijo Castle Park

Shurijo Castle Park, version 2

The data processing flow in our project is as follows. We use the Django framework to realize the image collection form, and collected information is stored on the SQL database. We extract 3 frames per second from videos, and obscure personal identities using human-body segmentation and face detection based on deep neural networks. The images are then classified into 20 classes using a customized image classification network using the Custom Vision API of Microsoft Azure cognitive services. The classified images are input to photogrammetry software RealityCapture and manually cleaned with Cinema4D.

Thanks to Sketchfab, we use its viewer to show 3D models on our website. We taught students from Konan School in Okinawa how to use Sketchfab’s viewer combined with cardboard goggles for a VR experience. They are volunteer tour guides in Shurijo Castle Park for students from outside of Okinawa. Our model of the Main Hall was used for a guided tour of Shurijo Castle by the students of Konan School to more than 100 students in February 2020.

On October 30, 2020, Google Arts and Culture and Okinawa Prefecture’s “Reconstruction of Shuri Castle” content was released, and our content is available on their platform as well. The content is curated by Okinawa Prefecture, and the page about Ryukyuan architecture and how it was influenced by Japanese and Chinese architecture was personally interesting to me. Integration with the entire Shuri Castle model is also in progress through our collaboration with NHK Global Media Services. It is partially visible on Sketchfab.

Shurijo Castle Park Digital Reconstruction with NHK Global Media Services

Release of our data

We are closing the data collection at the end of 2020 and planning to release the data on our website, as soon as the obscuration of personal identities is finished. We are grateful if the collected data can be used by many people for many applications and research purposes. Image matching and text analysis would be interesting research topics. Our story is available as a 25-minute English TV program created by NHK World.

The reconstructed Main Hall from a part of the collected data and the small square pyramids representing each camera look as if the cameras are wishing for a rebuilt Shurijo Castle. Let us hope the result will be a restoration even better than the last one.

 

Visit OUR Shurijo on Sketchfab