Welcome to the Kyoto University Music Robots page! Our research group focuses on the development of theremin-playing robots that interact with human musicians.

The theremin-playing robot system, initially developed by Takeshi Mizumoto in 2009, has now been ported to 4 different robot platforms. We showcase some of them here.

HRP-2 robot (solo): 1-DOF player, 2-DOF player, with vibrato, with expression

HRP-2 robot (ensemble): tempo detection, phase sync, flute cue detection, drum & flute trio, score following, audio-visual guitar tracking

HIRO robot: 1-DOF with vibrato

NAO robot: 4-DOF with vibrato new!, interactive beat tracking new!

Publications explaining these demos can be found here, and on the group members’ web pages: Takeshi Mizumoto, Takuma Otsuka, Angelica Lim, Tatsuhiko Itohara under the supervision of Dr. Hiroshi G. Okuno.

This version added another degree of freedom to increase the speed at which it could play, and  reduce wobbling from side-to-side.

Song: Eine Kleine Nachtmusik

Here we add vibrato to increase the expression in the piece.

Song: Polonaise

Music is always more fun with friends! Here, HRP-2 listens for percussive beats with a microphone. As it plays, it matches its co-player’s speed, even if the speed changes mid-song. Here we see that tempo detection works well, but can you tell the synchronization is off?

Song: Auralee

Challenges: Matching tempo is not enough to synchronize.

How do musicians stay in sync? This system models human-robot synchronization with the Kuramoto coupled oscillator model. The same model has been used to describe the synchronizing calling patterns among Japanese tree frogs. As you can notice, the beats and notes now match!

Song: Auralee

Challenges: Starting/stopping is still manual.

In this video, we combine all our previous techniques to create a trio. The robot can detect both visual and audio cues: 1) It watches the flutist for a cue to start 2) listens to the drummer to stay in sync, and 3) ends with the flutist’s end cue.

Song: Minuet

Challenges: 3-way interaction. Who’s following who?

Music robots would be useful to play music with  at home when others aren’t available -- the NAO robot is compact and practical for home and for gigs. We implement a robust pitch detection method here to deal with noise from head microphones. Seventy-five points were used for calibration.

Song: Theme from Star Trek

Challenges: How can we make the performance more expressive and human-like?

The future of music robot companions is near! In this video, the robot listens to a egg shaker to set its tempo throughout the piece. This shows the accuracy of the beat tracker even for complex signals. Twenty points were used for pitch calibration.

Song: Hey Jude

Challenges: Can the robot not only follow, but  inspire the humans, too?

Musicians often use visual cues to communicate with each other. For example, a head nod might be a cue to start a song. Here, the robot recognizes cues given by a flutist to automatically start the song, change tempo, and cut off the end note. Vocaloid was used to make HRP-2 sing the 3rd part.

Song: Les Anges Dans Les Campagnes

Challenges: Visual cues are impractical to use throughout the entire song.

Drums are one thing; acoustic instruments like guitar are harder for robots to track. Here, HRP-2 both listens to the guitar and watches the player’s hand cross the strings. With this “multi-modal fusion”, HRP-2 can stay in sync with his human co-player.

Song: Auralee

HIRO’s fast actuators are made for precise manipulation on the assembly line. It works well for theremin, too!

Song: Ave Maria

Score following means that the robot listens to each note its accompanist plays, and finds where it is in the score. This is a difficult problem, but very useful for instruments where the ‘beat’ is not obvious.

Song: Minuet