PRISM: Performance Recording & Integrated Synchronized Multimodal System

How do people use their bodies when they perform, and how do those movements lead to the sound they produce? To gain deeper insight into instrumental performance, including piano playing, we are developing PRISM (Performance Recording & Integrated Synchronized Multimodal System), a platform that records and analyzes a wide range of performance-related data. By capturing and visualizing the full process—from human movement to sound production—PRISM enables a richer understanding of performance from multiple perspectives. Through the use of this system and co-creation with academic researchers and the broader music community, we aim to contribute to a range of fields, including music education and performance science, that seek to better understand performers.

Enabling performers to engage more deeply with music

Our goal is to support performance experiences that help each performer engage with music more freely and more deeply. This is not only about improving technique. It is also about helping more people build a lasting, fulfilling relationship with their instruments and creating an environment in which they do not have to face the uncertainty and frustration of improvement on their own.

Playing an instrument is not simply a matter of following a formula. It is a deeply personal and creative act in which performers engage in an ongoing dialogue with music through their bodies and sensibilities. At the same time, many people become discouraged when their playing does not go as intended, and some may even begin to feel that music is not for them. We believe that if performers can objectively understand what is happening in their own performance, unnecessary anxiety can be reduced, making it easier to embrace the process of experimentation and growth.

Applying Measurement-Based Scientific Support to Musical Performance

In sports, tools such as high-speed cameras, sensors, and AI are widely used to visualize movement and physical condition, providing support for improvement that goes beyond intuition alone. In music, by contrast, performers may record themselves to check posture or technique, but there are still limited ways to objectively capture the relationship between movement and sound. As a result, it can be difficult to identify the source of performance issues, and the path to improvement can become indirect and inefficient. We aim to bring the kind of scientific support that has advanced in sports into the field of musical performance, helping performers engage more deeply with their art while gaining a clearer understanding of their own performance state.

In musical performance, sound is shaped by a complex interplay of factors, including the performer’s movements, the condition and individual characteristics of the instrument, and the acoustics of the space. Even when performers feel they are playing the same way as usual, subtle changes in physical condition, force, or environment can alter the resulting sound in ways that are difficult for them to perceive. This is why it is essential to measure and visualize what is happening during performance as accurately as possible. In particular, to support the reproducibility of performance, we aim to record the conditions surrounding both successful and unsuccessful performances and provide feedback that enables objective reflection.

PRISM: Performance Recording & Integrated Synchronized Multimodal System

We are developing PRISM, a system that records and analyzes a wide range of data generated during musical performance. By capturing the entire process—from a performer’s physical movements to the production of sound—through multiple forms of synchronized data, PRISM enables playback and analysis from a variety of perspectives. Through this multifaceted view of the relationship between movement and sound, we aim not only to advance research in performance science, but also to provide new insights for performers and educators.

PRISM can synchronously measure and record a broad range of performance-related data, including:

・MIDI data
・Audio data
・Center-of-gravity and balance data
・Performance video
・Breathing data

By combining these measurements, PRISM can capture musical performance from multiple perspectives. In piano performance, for example, the system can record video and audio using multiple cameras and microphones while simultaneously capturing MIDI data. This makes it possible to observe the entire process in detail—from the performer’s movements to the generation of sound—alongside the performance data itself.

Measuring Piano Performance with Yamaha’s Proprietary Sensor Technology

For piano performance measurement, Yamaha’s Disklavier ENSPIRE PRO can capture detailed data not only from the performer, but also from the instrument itself. In a grand piano, pressing a key triggers a complex action mechanism that causes a hammer to strike a string and produce sound. The Disklavier ENSPIRE PRO is equipped with built-in sensors that can precisely measure the motion of both keys and hammers, and we have developed a system that can acquire this sensor data directly. One of the key strengths of this approach is that it enables highly accurate measurement without compromising the playability of the grand piano.

This system allows us to observe in detail how a pianist moves the keys and hammers during performance. When combined with a motion capture system, it also makes it possible to follow the entire chain of events—from the pianist’s body movements and key actions, to the resulting hammer motion, and ultimately to the sound that reaches the listener. In this way, one of PRISM’s defining features is its ability to capture, in an integrated manner, the performer’s physical movement, the instrument’s internal behavior, and the resulting sound.

Future Outlook

PRISM has made it possible to capture musical performance as data, but we are still exploring how best to make use of the data we collect. Musical instruments and playing techniques are remarkably diverse, and there are many aspects that may be overlooked from the perspective of a single company. It is also difficult to determine in advance which types of measurement will truly contribute to performance improvement. That is why we believe this field must be developed in collaboration with researchers and performers around the world. To support such collaboration, we are considering making the PRISM system open source and sharing the data it records. By advancing this research together with partner worldwide, we hope to help establish new forms of scientific support for performers.