AI Performance Expression Rendering Technology
AI Reproduces the Expression of a Human Performance
Natural, Human-like Performance
This technology allows AI to generate natural, human-like piano performances for various music scores. Furthermore, by utilizing a hybrid player piano such as the Disklavier™ *¹, it is possible to generate realistic performances on a real piano.
*¹ The Disklavier is a reproducing piano equipped with an automatic performance function that accurately reproduces the movements of the keyboard and pedals.
Perform Freely with Piano Accompaniment by Your Favorite Artists
Combining this technology with AI music ensemble technology provides the experience of allowing anyone to play with one’s favorite artists, anytime. We hope to offer a new way to enjoy music, making it easier for people to enjoy ensemble music at home.
AI Analysis and Learning
WWith this technology, AI (deep neural network) learns to associate the characteristics and expressions of the human pianist’s performance with the corresponding music score data. Once trained, the AI can generate natural performances with human-like characteristics and expressions for any given score. In addition, by training on performances played by a specific performer, the AI will be able to generate a performance that is unique to that person, even for a piece that the performer has never played before.
To train the AI, we prepare “performance data” recorded by a human playing a keyboard instrument and “score data” of the corresponding performance.
Performance data is the control that is input on the keyboard, including the velocity and timing of key presses and releases. Since this kind of information cannot be obtained directly from an audio recording of a piano performance, we infer the performance data that best recreates the audio recording on a Disklavier or a specific digital piano from the recorded sound source.
During training, a deep neural network is used to learn the mapping between the performance data and the music score data. By learning to map between local features in the human performance and local features in the music score, it becomes possible to generate a performance for a music score that has never been played by a human.
The AI also discovers the space of musical expression for a given music score. This is important because a pianist can play the same piece of music very differently each time – it’s insufficient to simply learn a one-to-one mapping between music score and performance. The AI learns to explain variations not written in the music score in the space of latent variables. This allows the AI to generate a variety of performances for a given music score and a human to manipulate the musical expression.
Application Examples
Dear Glenn
We exhibited “Dear Glenn”, a project that pursues the possibility of co-creation between AI and humans, at the Ars Electronica Festival 2019 media arts festival.
“Dear Glenn” uses AI piano performance expression technology to learn and reproduce the performances of Glenn Gould, the legendary pianist who died in 1982. In addition, by fusing this technology with AI music ensemble technology , Dear Glenn is able to reproduce Glenn Gould’s touch on the piano and perform ensembles with artists who are connected to him.
