Global Internship：Research & Development Division

Outline (2024 Season)

Fields

• AI-powered solo detection for musical instruments

  
  Mixing engineers perform a variety of tasks depending on different situations during a live performance. Therefore, it is important to understand the situations that can occur during a live performance in order to develop a mixing console that can assist engineers. Detection of solo sections is a particularly important issue for this context.
  In this internship, as a member of a team dealing with machine learning and audio signal processing, the intern will be expected to develop a technology for detecting solo sections from the input, mainly based on machine learning technology.
  The intern will also be expected to perform more advanced tasks, such as identifying not only solo sections, but also other sections of a live performance at the same time.

  SKILL

  • Required: Expertise in programming (Python, C++), audio signal processing, and machine learning
  • Welcomed: Experience in playing musical instruments or mixing

• Sound field reproduction of musical instruments when performing

  
  Even in a virtual space, unique radiation characteristics might be required when performing musical instruments. This could contribute to the "reality of the musical performance".
  In this internship, as a member of a team dealing with audio signal processing and spatial audio, the intern will be expected to use multiple ViReal Mics (https://www.yamaha.com/en/about/research/technologies/vireal/) to develop a method for the measurement of the radiation characteristics of musical instruments when performing, and to evaluate the developed method in a real environment. The intern will also be expected to model the recorded source as objects with directivity to auralize through binaural reproduction.

  SKILL

  • Required: Expertise in sound field theory in the spherical harmonic domain, array signal processing, and programming (Python, MATLAB)
  • Welcomed: Experience in sound field recording and the measurement of radiation characteristics

• Immersive Virtual Backgrounds for Enhanced Remote Communication with 3D audio production

  
  Virtual background technologies are implemented in remote communication applications, such as Teams or Zoom. However, traditional virtual backgrounds lack naturalness, remaining static even when there is camera movement. Additionally, the acoustic characteristics of various environments are not simulated accurately. In this internship, you will have the opportunity to develop 3D rendering techniques for dynamically adapting virtual backgrounds to camera movements. Furthermore, the task includes context-driven virtual room acoustics such as the speaker's voice enhancement. Our goal is to create software that revolutionizes remote communication by delivering more realistic virtual backgrounds and immersive audio experiences, raising the bar for industry standards.

  SKILL

  • Required: Proficiency in programming languages such as Python and C++
  • Welcomed: Experience in audio signal processing, computer vision, and image processing

• AI-powered agent for music ensemble

  
  Understanding and generating a music performance by a computer enables a wide variety of applications that add value to the experience of playing a musical instrument. In this internship, we will work on machine learning techniques related to music performance and/or motion generation or recognition, for the purpose of providing an interactive music performance agent. We will provide two tracks for this internship:
  (1) The intern will be expected to apply machine learning to create an interactive virtual musician.
  (2) The intern will be expected to create an interactive virtual musician system for a music ensemble.

  SKILL

  (1)

  • Required: Expertise in programming (Python,C++), music information retrieval, and machine learning
  • Welcomed: Expertise in human-computer interaction, experience in handling human motion data (e.g. BVH data)

  (2)

  • Required: Experience in creating interactive systems (experience in programming languages e.g., Max/MSP/Jitter, TouchDesigner, Unity, UE, HTML+JS), experience with machine learning frameworks (e.g. torch, TF, CoreML)
  • Welcomed: Expertise in human-computer interaction or music informatics research, knowledge of character animation, experience in creating VR/XR systems

• Psychophysical modeling of the auditory spatial impression in the context of musical instruments and audio equipment

  
  At Yamaha, we consider two factors when seeking to achieve high-quality sound through the research and development of musical instruments and audio equipment: tone quality or timbre, and the spatial features of sound. The way humans form an "auditory spatial impression" from such spatial features is an active topic of research with many open questions. During this internship, you will formulate new psychophysical models of the auditory spatial impression of musical instruments and audio equipment. To this end, you will gather sensory data from psychophysical experiments, and apply data analysis techniques to identify the physical features of sound sources and fields that can best explain the experimental results.

  SKILL

  • Required: Expertise in Python programming, knowledge of statistics and data analysis
  • Welcomed: Knowledge of physical acoustics, psychophysics and array signal processing