Data Measurement System for Musical Performances

Understand music performance holistically through a multimodal recording system

We are developing a system that collects and analyzes diverse data during musical performances to gain deeper insights into the movements involved in playing instruments, such as the piano. This system captures a variety of data involved in the entire performance process, from movement to sound production. Synchronizing and analyzing these datasets enables a multifaceted understanding of how physical movements of the body and the instrument correlate with the resulting sounds. Using this system, we aim to facilitate and accelerate research in performance science, to deliver valuable insights to musicians, educators, and researchers.

Our system can synchronously record the following modalities:

・MIDI data
・Audio data
・Center of gravity (balance) of the player sitting on a piano chair
・Video data

For example, when measuring piano performance, it is possible to record video from multiple angles using cameras, capture audio with microphones, and collect MIDI data. This enables the accurate measurement of the entire process—from the performer’s movements to the resulting sound—along with detailed performance data.

Measure various data in sync with the performance

When analyzing piano performance, we utilize Yamaha’s automatic playing piano, the Disklavier ENSPIRE™ PRO, to measure not only the data mentioned above but also the intricate movements of the instrument itself. In a grand piano, sound is produced through a sophisticated mechanism: when a performer presses the keys, hammers strike the strings via a complex action system. The Disklavier ENSPIRE™ PRO is equipped with advanced sensors that can precisely measure the movements of both the keys and hammers. We have developed a system that enables us to obtain data directly from these sensors. Notably, this system achieves high-precision measurements without sacrificing the natural playability of the grand piano.

Review the performance from various modalities

This system is capable of recording various modalities, such as video and sensor data, with high precision and synchronization. This allows a performance to be understood from multiple modalities. Additionally, the ability to replay the recorded data will enable performers to better grasp how their physical movements translate into sound, offering new opportunities for insight and growth.

We envision utilizing this system across a broad range of research fields, such as performance science, to deepen our understanding of the mechanics and artistry of musical expression.