Overview

The growing desire among consumers for novel and realistic experiences has driven the continuous growth and advancement of immersive technologies, particularly within the gaming and entertainment sectors. However, these technologies are still in the early stages of adoption in the realm of the arts. Museums, traditionally focused on preserving history and maintaining a sense of tradition, now face the challenge of engaging a growing generation of digital natives. It is therefore crucial for us to pioneer innovative experiences that can attract a wider and more diverse audience and foster a renewed appreciation for the arts.

To achieve this, a deep understanding of the needs and motivations of individuals who may be less appreciative of the arts is paramount. NCS, in partnership with the National Gallery of Singapore, has developed a mixed reality experience that is tailored to meet the preferences and expectations of this target audience. This proof of concept leverages the advanced mixed reality capabilities of Meta Quest Pro to seamlessly blend virtual and augmented realities in a physical museum space.

How it began: The visitor journey map

In 2021, NCS became the official innovation partner of the National Gallery of Singapore. The NCS NEXT Digital Experience team collaborated with the Gallery to understand the then current visitor challenges so that a ‘user-first’ approach could be taken to co-create an immersive visitor journey. The Gallery’s Visitor Journey Map served as a foundation for crafting connected, immersive, and innovative next-generation experiences.

One of the key challenges identified to ensuring broader appeal of the museum experience was making art more accessible to visitors who may not be familiar with art appreciation. To elevate the visitor experience using innovative technology, a holist approach that considered customer/user experience, and included behavioural science, was adopted.

Critical challenges

NCS’ initial research revealed challenges faced by individuals less familiar with appreciating the arts, including:

• Limited ways to access and understand the stories behind works of art: Visitors who were unfamiliar with art had a strong desire to understand the reasons that a piece of art had been created. However, the existing written captions below the static paintings fell short of capturing that essence of creation and limited the opportunities for visitors to develop meaningful connections with both the art and the artists.
• Lack of guidance about how to appreciate art: Visitors who were unfamiliar with art felt discouraged and excluded as they believed they required knowledge of art techniques and context to appreciate the work. They desired more guidance, and a barrier-free way to experience, consume, and appreciate the art.

Based on those findings, a team comprised of designers, developers, product managers, and UI/UX (User Interface/User Experience) experts from NCS NEXT Digital Experience and NCS Xperience Works was formed to create a mixed reality immersive experience that would bring the artworks to life for visitors.

The experience

In the Gallery, visitors don a mixed reality headset while seated in a physical space. Using hand gestures, visitors can interact with the art and learn about context and back stories in an intuitive and engaging way. The novel and immersive nature of this mixed reality experience appeals to visitors, creates excitement, and increases engagement with Gallery exhibits.

Immersion-realistic environments:
Through an immersive encounter, a life-sized adult tiger moves in close proximity, roars, and transports the visitor into the heart of the artwork. This realistic spatial experience, enriched with spatial sound and 3D elements, serves as a pivotal factor in elevating the perceived value of the visit and increases the visitor’s willingness to pay.

Narration delivered by the artist:

Through an animated portrait, the artist unfolds his past, leading visitors on a visual journey, while sharing his influences, and explaining his artistic process. With narratives about the artist and his inspirations, and information about the meaning of the artwork, the immersive experience allows visitors to develop a deeper emotional connection with both the art and the artist.

Interactive artwork:

The user interface style is in keeping with that of the artwork and the magnifying glass feature allows visitors to explore the painting and to zoom in. The ability to highlight elements of the artwork in greater detail helps visitors make sense of what they are seeing and enriches their overall engagement with the artistic content.

‘Gale’ the Gen AI guide:

A friendly guide, powered by Gen AI, helps to answer any questions visitors may have when viewing the artwork, personalising the visit.

The technology

The mixed reality immersive experience focuses on a few key components:

• Animation of the artwork:
  The focal point of Raden Saleh's "Forest Fire" has been given a new dimension through animation overlaid on the actual artwork. The previously static figures of the tiger, ox, and deer are animated, moving as if truly alive. The effect is heightened by a dynamic visual portrayal of the blazing fire, and the accompanying crackling sound effects, paired with a voiceover of the forest fire background story. The addition of motion and sound transforms the scene into a lifelike spectacle. The result is a more immersive and engaging viewing experience of "Forest Fire," as the animation breathes new life into the classic piece.
   
• Spatial design:
  The spatial design of the mixed reality environment has been carefully crafted to ensure seamless interaction between the virtual elements and the physical space. Bezel, a collaborative spatial design platform for teams, was used for advanced mapping. This tool accurately measures and recreates the dimensions and features of the physical room within the virtual environment. This method guarantees a coherent experience, where virtual and real elements coexist naturally, preserving depth perception and spatial awareness for the user.
   
• Development of realistic 3D models:
  The creation of realistic 3D models is crucial for an immersive experience. High-quality 3D models of key elements, such as the forest fire, tiger, and Gale, the docent ambassador, were developed with an emphasis on detail and realism. Techniques were used to deliver intricate details and lifelike realism, including realistic lighting and shadows, to enhance the authenticity of these characters. Additionally, photogrammetry techniques were utilised to capture textures and details from real-world objects. These elements were meticulously refined and optimised for rendering in the mixed reality environment, to ensure that the models are visually compelling and true to life.

• Interactive elements:
  Interactive elements in our mixed reality environment were designed to enable natural and intuitive user engagement. During the development process, focus was put on selecting user-friendly gestures and commands. The interactive design was built on a blend of touch controls, eye tracking, and gestures. An iterative approach was taken to identify the most effective 'tappable' buttons for the menu panel to ensure realistic interactions with virtual objects. Additionally, eye tracking technology intuitively follows the user's gaze, enhancing interactions with Gale, the ambassador. The combination of eye tracking with touch and gesture-based interactive methods was designed to produce experiences that feel natural and responsive, while also being deeply immersive. The aim was to enhance the overall user experience and reduce any frustration that might result from elements that fail to respond in a realistic manner.

User testing: Paving the way to the future of museum visits

The team conducted iterative user testing where we went through a series of in-person sessions. The team spoke to eight users directly within the exhibition spaces, where users interacted with a prototype designed to immerse them in the authentic gallery environment.

The primary objective was to assess the appeal and desirability of an immersive mixed reality experience for individuals who were new to art appreciation. Testing focused on the extent to which the incorporation of innovative immersive technologies increased users’ appreciation for art and had a positive impact on their overall gallery experience.

The user testing phase spanned a comprehensive two-week period and resulted in the following findings:

• Strong positive sentiment to the Mixed Reality (MR) immersive experience:
  • Average score of 80% for user intent to revisit the gallery 
  • Average score of 90% for user intent to recommend the MR Immersive to others  
  • Average score of 71% for ease-of-use of the MR immersive headset
• Novelty of immersive experiences in museums is a strong pull factor: Users noted the unique appeal of an experience that is not yet widely available in traditional art settings in Singapore. User sentiment reflected excitement with the ability to interact with immersive elements, such as the virtual tiger, outside of real-world constraints. The willingness of users to pay up to SG$10 for such engaging encounters underscores the strong attraction and perceived value of the mixed reality experience.
• Sensorial experiences increase accessibility to the arts: The sensorial, immersive experience, from the collective of 3D interactive elements, to the spatial surround sounds, was noted as an effortlessly enjoyable entry point by users who are new to art appreciation.
• Narration of the artwork increases emotional connection: The introduction of narration as well as information pertaining to the meaning of the artwork deepen user engagement. The animated talking paintings and interactive art controls contribute to a richer understanding and make the art more accessible and meaningful for users.
• Gen AI guide helps personalise museum visits:  Users commented that, while still conceptual, the Gen AI guide’s ability to answer questions and assist with navigation helped to personalise their museum visit.

These findings not only affirm the revolutionary potential of the MR experience but also highlight the ability to redefine how individuals connect with, and enjoy, cultural narratives within a museum setting with the addition of mixed reality elements. This phase has opened a pathway to a more immersive journey through art and history.