Dear Glenn

An A.I. system crafted by love.

GLENN GOULD AS A.I. GLENN GOULD AS A.I.

Project

This is a project dubbed Dear Glenn, a project inspired by his unique creative style and launched to explore the future of music through the use of artificial intelligence.
Glenn Gould was known for his devotion to recording with digital media and an interest in rethinking the relationship between performer and audience. The project to develop this system has been dubbed “Dear Glenn” as a tribute to the artist’s attitude, which was the inspiration for the idea behind the project.

Glenn Gould as A.I.

With the full support of the Glenn Gould Foundation, Yamaha analyzed over 100 hours of Gould’s performance recordings to develop an understanding of his playing style and employed deep learning algorithms based on the data collected to create the AI system. In addition to Gould’s audio recordings, AI learning data included human input in the form of performances by multiple pianists who were admirers of Gould and intimately familiar with his performance style, raising the quality of reproduction to new levels. Near-instant performance analysis of fellow human players enables the AI to play predictively while interacting with human musicians. More than simply an automated performance, the AI reproduces the masterful touch of Glenn Gould to provide an inspiring and interactive experience of co-creation between an AI pianist and human musicians.

The profile of Glenn Gould

Born in Toronto, Canada in 1932, Glenn Gould was a legendary pianist who passed away in 1982 at the young age of 50. Gould has received extraordinary praise and is known for his masterful performances of J.S. Bach’s music, beginning with his debut album Bach: The Goldberg Variations, which was released in 1956. In 1964, Gould announced the end of his concert career and began to concentrate on recording, devoting himself to digital media releases. Gould was also known for his unconventional and unique performance habits, which included sitting on a low chair and leaning over the piano keyboard, as well as humming while playing, even during recordings. In his later years, Gould recorded three albums, including Bach: The Goldberg Variations, on a Yamaha concert piano.

Concert & Event

GLENN GOULD AS A.I. GLENN GOULD AS A.I.

Yamaha’s Glenn Gould AI performs for the first time ever at the 2019 Ars Electronica Festival.

  • PIANO: GLENN GOULD, FRANCESCO TRISTANO
  • ARTIFICIAL INTELLIGENCE: CRAFTES BY Yamaha
PROGRAM

GOLDBERG VARIATIONS, BWV ARIA J.S.BACH

ART OF FUGUE, BWV1080/18 J.S.BACH

The concert was held at St. Florian Monastery on September 7, the third day of the Ars Electronica festival. In addition to a piano solo, the AI system performed a piano duet with Francesco Tristano and a wind trio with members of the Bruckner Orchestra Linz (violin and flute) for a performance “with contemporary artists that transcended space and time.” Neither of the ensemble pieces performed were included in the machine learning data, so audience members listened with great interest to see how well the AI system could reproduce Gould’s musicality without any recording data to rely on and how well it could cooperate and interact with human players while playing together in ensemble.

Keynote Session

Date Saturday, 7 September 2019 19:00 - 19:30
Venue Altomonte Saal, St. Florian Monastery (Linz, Austria)
Speakers Francesco Tristano (Pianist), Brian M. Levine (Glenn Gould Foundation), Norbert Trawöger(Bruckner Orchestra Linz), Akira Maezawa (R&D, Yamaha Corporation)
Theme Possibilities of Co-Creation Between Artists and AI

Concert (AI Unveiling)

Date Saturday, 7 September 2019 20:30 - 21:00
Venue Basilika, St. Florian Monastery (Linz, Austria)
Performers Piano:Glenn Gould, Francesco Tristano
Flute,Violin:Norbert Trawöger, Maria Elisabeth Köstler(Bruckner Orchestra Linz)
Program J.S. Bach’s Goldberg Variations (BWV 988), Fuga inversa a 2 Clavicembali from The Art of the Fugue (BWV 1080/18)*, Trio Sonata G Major (BWV 1038)* *Never performed by Glenn Gould

About the Ars Electronica Festival

ARS ELECTRONICA Image

The Ars Electronica Festival is a festival that centers on the latest in art, technology, and science and is held by Ars Electronica, a media art research institute based in Linz, Austria. First held in 1979, the festival has since become one of the largest media art events in the world. During the festival, experts gather from around the world at exhibitions, performances, events, and international conferences in venues around Linz to engage in discussion and foster new connections.

ARS ELECTRONICA Image

Research and Development

Training of the AI

STEP 01. Analysis

Extract music performance data from music audio recordings by Glenn Gould, such as the velocity and the timing fluctuations of key depression.

STEP 02. Training

Use an AI (deep neural network) to train a mapping between the music score and the performance data, so that the AI could generate performance data to any music score data. For the performance data, we use (1) the analyzed results from step 01 and (2) performances by Glenn Gould experts who were asked to play in the style of Glenn Gould.

STEP 03. Performance

For a given music score data, the trained AI from step 02 generates a corresponding music performance in the style of Glenn Gould.

Dear Glenn - TRAINING OF THE AI

Training of the AI

Dear Glenn - TRAINING OF THE AI

STEP 01. Analysis

Extract music performance data from music audio recordings by Glenn Gould, such as the velocity and the timing fluctuations of key depression.

STEP 02. Training

Use an AI (deep neural network) to train a mapping between the music score and the performance data, so that the AI could generate performance data to any music score data. For the performance data, we use (1) the analyzed results from step 01 and (2) performances by Glenn Gould experts who were asked to play in the style of Glenn Gould.

STEP 03. Performance

For a given music score data, the trained AI from step 02 generates a corresponding music performance in the style of Glenn Gould.

Music ensemble with AI and human musicians

Dear Glenn - Music ensemble with AI and human musicians Dear Glenn - Music ensemble with AI and human musicians

This system also allows a Disklavier player piano to play in the style of Glenn Gould while synchronizing with human musicians. This capability is achieved by controlling the playback based on the analysis of the sound and the motion generated by human musicians.

Interviews

Bruce Brubaker

Bruce Brubaker

Pianist

What are your feelings regarding Gould in the context of your work?
Even though we can do many things technologically now that Gould could not, some of the ideas he had—particularly in regard to post-production art—are extremely relevant to the way I’d like to make recordings now, and to the way I’d like to think about recording and what it is. I think that Gould really was a role model for me in that respect.
What do you think about the relationship between Glenn Gould and technology?
For Gould, the technology of the recording studio could be used to create an experience that was more personal, more intimate, and also which contained a greater range of expressiveness than could ever happen in a single performance. He was finding ways to maneuver through what he had made, to assemble something that was driven by his imagination rather than by the physical realities of the hand or the concert room.
What is the musicality of Glenn Gould?

I think that as a part of the history of music we have to understand that the invention of sound recording near the end of the 19th century was a significant stage in the development of music. You could almost say that music before sound recording was one single, long period, and that with sound recording something else began. In our discussion of artificial intelligence, sound recording may have been the first phase of that development with regard to music. Recording allowed us as musicians to have some awareness of ourselves, and of the art and the sound that we make aside from our body. It was like inventing a mirror, an ability to perceive something that we couldn’t hear before.

Gould and his work were part of that development. He was one of the few people in the classical world who recognized that everything had changed, and that because of recording, the way performers perform also needed to change. The way concert halls were made changed—everything changed, and instead of denying that I think he was interested in trying to embrace that present moment. The fact is that the music created through the agency of sound recordings actually does change. Not because the recordings change — but because our ears, and the context for the recordings continue to change.

In Toronto we interviewed people who had a direct connection with Glenn Gould. When we let them listen to this AI, their first reaction was that is seemed somewhat strange. But many areas of this music made them feel good, and some even cried. It is kind of fascinating, that an AI could evoke an emotional response.

We are now at a kind of cusp, as we begin to understand what it is about human music-making that really matters in terms of the experience that listeners have. In the past it may have seemed that there was some kind of magic or incomprehensible ether around the performer or the performance that caused listeners to have an emotional or even spiritual reaction. Increasingly, as cognitive psychologists are studying micro-timings, we are starting to find ways to look at what it is that live musicians really do. We’re coming to realize that our reaction to music is a beautiful, human, and often an unrepeatable-seeming thing, but at the same time, something that is the result of certain very specific things that happen. These might have seemed to be uniquely the product of an individual, but actually, the trigger for those responses is something that we are beginning to understand.

What’s interesting is that this offers those of us who are still making music with our bodies a more nuanced understanding of what it is that we do. I’ve often said, for example, that no two beats are created equal. It’s sort of a joke, I suppose, but when played by a human being, it’s true! And yet, how many not-so-well-informed music teachers are telling their students to play exactly equal beats?! That’s not possible, and it’s not desirable! The threshold at which we human listeners perceive equality of beat is fascinating. I think that it has actually changed considerably since the 1980s when we started to hear music that really did have exactly equal beats—music made in the studio—where you play equal beats quite easily. Now it’s quite prevalent, and in certain kinds of pop music we have exactly equal beats.

Do you have any desires or opinions to offer Yamaha on the development of AI?

The thing that is particularly fascinating to me about the Glenn Gould project, is really not the project itself, but how much it tells us about what the future may hold. AI can be used to capture the artistic personality and ethos of a human player.

I don’t know how you see it, but I don’t really feel that AI is a completely new development. I think it’s an extension of some things which have already been happening, and it may be more stunningly obvious to people that it is something important, but I think that many of these trends and many of these ways of thinking have already been going on for a while.

You can imagine that an AI could come up not only with a recording (a sound realization), but also a nuanced understanding of the subject, which might give rise to all kinds of artistic applications. We have to come to terms with the idea that the world of human performance is going to continue to change very rapidly, and that at some point the value and the meaning of what it is to be a human performer will once again come into question, as happened after sound recording became prevalent in the 20th century. I am sure that is going to happen—we see it happening in other fields and I think it’s probably coming sooner than we think in music.

For this particular project, what do you think was achieved in terms of creating a performance, and what do you think remains to be done?

In its best moments, the project offers the sense that the playing we hear is expressive and does have a very strong connection to some sort of humanity rather than feeling machine-like. Of course, this raises the question of what exactly expressiveness is, because we used to imagine that humans had some ideas or feelings, or thoughts, and that somehow those things were then transformed into a pattern of sounds that were received by another human who received an experience something like the first person had.

As to the future, many things remain to be done. For example, how the pianist uses their foot on the damper pedal (right pedal) is a complex phenomenon that is difficult to decipher when you listen to sound recordings, and certainly needs to be investigated. The specific nuances of rhythmic timing are still an area that merits a lot of study, although in this project that begins to happen. Initially I thought it would be challenging to achieve a way of playing unequal rhythm that would feel similar to what a human really would do, but I think we are coming close. It bears more study, as it is extremely interesting, and I think that is one of the most personal aspects of each individual human being’s performance. Those slight proportional differences in rhythmic expression have a lot to do with our rhythmic or expressive signature as musicians, and I believe that this is an area that will continue to become clearer.

Do you think that Glenn Gould, if he were here, would appreciate our efforts?

Oh, I think so. I look at it in two ways. It’s possible that on one hand he might be horrified, but then on the other I think he would be really thrilled—probably a little of both.

I say “horrified” only because it would take the making of the sounds out of his control, and, of course, “thrilled” because it allows something new to happen. Both of those things are true.

Collaborators

"Dear Glenn" Ambassador and AI Collaborator

  • Francesco Tristano (LU):Pianist and Composer

AI Collaborators and Advisors

  • Brian Levine (CA):Executive Director of Glenn Gould Foundation
  • Norbert Trawöger (AT):Flutist and Artistic Director of Bruckner Orchestra Linz
  • Maria Elisabeth Köstler (AT/DE):Violinist from Bruckner Orchestra Linz
  • Bruce Brubaker (US):Pianist
  • Adam Sherkin (CA):Pianist
  • Kevin Ahfat (CA):Pianist

Pianists from Tokyo University of the Arts

  • Sho Okuya (JP)
  • Genki Takai (JP)
  • Shimon Ono (JP)

Special Thanks
Glenn Gould's Dear Friends

  • Adele Armin (CA)
  • Janet Somerville (CA)
  • Lorne Tulk (CA)
  • Roxolana Roslak (CA)

Official Partner

  • Glenn Gould Foundation

Supported Partner

  • Tokyo University of the Arts

Contact

For technical inquiries, please access to the inquiry form page.

For press inquiries, please access to the inquiry form page.

THE GLENN GOULD FOUNDATION
Tokyo University of the Arts
JAPAN-AUSTRIA 1869-2019