The virtual rakugo theater system being developed in our laboratory aims to enable even beginners to enjoy the experience of speaking rakugo as if it were karaoke through the use of VR technology.
System
We create CG renderings of the rakugo stage, audience characters, and a seated rakugo performer-style avatar. The rakugo performer avatar synchronizes with the movements of the user wearing the HMD and motion controllers.
A model video is presented in front of the user so that they can speak rakugo by imitating hand gestures and lines. The model video shows a professional rakugo performer speaking rakugo specifically for this system, and the lines to be spoken are also presented with subtitles.
In rakugo, there is a gesture called “kamishimo” (upper and lower), in which characters are played differently by the direction of the face and body. By changing the presentation position of the model video to the direction in which the face should be turned according to the professional rakugo performer’s “kamishimo,” users can naturally imitate the “kamishimo” gesture simply by focusing on the video.
Audience characters clap, laugh, and look away in response to the flow of the story and the user’s movements. The better the user imitates the movements of the professional rakugo performer in the model video, the more the audience characters laugh, aiming to reproduce the feeling of “making them laugh” that comes from speaking well.
