Virtual Puppetry with Spiraling Interaction


The  virtual interactive puppet performances, called “YONG-SHIN-GUD (calling-dragon-spirit),consists of the artist’s performance with live music, storytelling and 3D motion graphics, and the audience’s participation. “Yong-Shin-Gud” is a Korean word, named for a special shamanic ritual that evokes dragon spirit. Same with original meaning, my live puppetry is a special ceremony to embody the dragon spirit in computer interactivity, shaping the constant process of transformation and spiraling phases of interaction.


Driven by storytelling and musical instrumental play, Yong-Shin-Gud starts with 3 components: the virtual world, the real space of the performance room, and a sound activated puppet, “Virtual Shaman,” who mediates between the virtual and the real, in the same way of Korean Shaman ritual, “Gud”, composed of the binary pairs and Shaman. The virtual sound activated puppet eventually acquires the ultimate trans-state of shaman, by spiraling interactive dialogues with the real puppeteer.



The Virtual Shaman : Sound Activated Puppet


Virtual Shaman: sound activated puppet comes after the primitive expression of Korean traditional puppet. Lacking realistic details, virtual shaman’s appearance connects us with the remote time of totem pole. It stimulates the viewer’s perception of the figure in question and encourages the act of imagination based on the abstracted signs of life. Virtual puppet leads viewers to imagine. The viewer’s imagination completes the figure and life of the virtual puppet.




Virtual Shaman: Sound Activated Puppet

Unlike traditional puppet, Virtual Shaman is motivated by sound, not by string, rod or hand triggers.Any sound input coming into the microphone, such as musical instrument sound and storytelling motivate the mouth, body and facial expression of 3D virtual puppet on screen, in real time. The virtual puppet constantly speaks and sings back to the puppeteer, like real-time echoes and mirror reflections. Whereas the traditional puppeteer requires to have a certain trained skill to manipulate the puppet, my virtual puppet provides us immediate knowledge of how the system works. In this complete hands-off experience, the puppeteer transformed to the puppet in a second, with strong attention to what puppet speaks about and how. 


Virtual Shaman was created by using open source, Antoine Schmitt asFFTXtra in 3D interactive application. The asFFT Xtra computes a power spectrum using a FFT (Fast Fourier Transform) on the input sound samples. This live sound spectrum was applied to the 3D puppet’s mouth, body and texture with different type of sensitivity and controls. Most of all, real time mouth interaction has a significant role to make virtual shaman alive. The sign system of speech is vital importance to puppetry, allowing people to imagine the puppet as having life. Even though primitive up and down mouth response, it makes virtual puppet look exactly lip-syncs with the puppeteer’s voice and musical instrument play. However, what truly makes the puppet alive is not the technical control over the puppet’s physical movement, but the user (puppeteer)’s spiritual engagement and thewilling suspension of disbelief towards the presence of the virtual puppet. The audience’s willingness makes the leap from the perception of representative abstractions to the imagination of life. The audience chooses to imagine and complete the puppetry. This willingness only exists when the distinction between puppet and puppeteer is abolished. It came true by performing the ritual of interaction.


Process of Ritual


YONG-SHIN-GUD shapes the ongoing real –time lip-sync process and eventually erase the line between puppet and puppeteer. As yin and yang continuously exchange their position side by side without hierarchy, the real puppeteer becomes the virtual puppet and the virtual puppet becomes the real puppeteer in cycles.  This dynamic demonstrates my perspective of the body as a strong spiritual entity that is continuously transformed into our binary reflections, seen as objects, images, and puppets. The puppetry starts with a simple physical motivational element (the lip-syncing) with which to interact; and then the whole interactive process becomes more and more powerful by the puppeteer’s increasing spiritual engagement with the puppet. This back and forth between physical motivation and spiritual activation constitutes the magical part of puppetry: The puppeteer’s voice motivates the puppet’s mouth.  The puppet lip-syncs with the puppeteer.  The puppeteer becomes the puppet and talk through the puppet. The puppet’s voice motivates the puppeteer’s mouth. The puppeteer lip-syncs with the puppet. The puppet becomes the puppeteer and speaks through the puppeteer. The puppeteer’s voice motivates the puppet’s mouth 


Virtual puppet and puppeteer telling a story together

For this process, my project requires that the performer watch carefully her puppet’s continuous response on screen, whereas it is not necessary for the sound performer to watch the image unfolding on the screen in the most of audio-visual interactive performances. The performer should face her puppet on the screen, as the puppet’s response will gradually motivate the performer’s musical or storytelling inspiration. The audience should be located between the performer and the puppet, thus placed physically, and spiritually, between this spiraling inner dialogue of the puppet and the puppeteer.


It is magical how much the puppeteer merges into the puppet on the screen. Without seeing the puppet, the puppeteer can’t possibly play the music with the same energy. The melody she plays is not generated by her brain and consciousness—it’s not her melody at all. Rather, the melody is created between the puppet and her. Her sound inspires the puppet and its movement inspires her in return. They continue to inspire each other and play a melody that is neither solo nor duet. The puppeteer falls in love with her puppet during this cycle. The more she loves the puppet, the more the puppet loves her. The more her puppet loves her, the more she loves her puppet. As a Shaman begins to build its relationship with an object, so too does the puppeteer build its relationship with the virtual puppet. The relationship begins in a microscopic way, but it soon expands step by step, becoming a gigantic scale that fills up the entire world. The puppeteer communicates with the puppet, with themselves, with the audience, with the spiritual world and with the whole universe. The Inner space communicates with the outer space. The energy of ritual spirals up and up.


The Role of Puppetry

Puppetry is one of the most ancient forms of performance in the world. In Korean traditional puppet play, puppeteers all belongs to the lowest social class who were oppressed, mistreated or even persecuted. Puppet plays were vehicles for their satire and criticism in society. Transformed to the puppet, the puppeteer discovers to speak something that was not revealed in his/her everyday life. It can be criticism or free imagination. Puppet has been functioned as a revolutionary interface to bring people’s free speech and imaginative storytelling. 


Yong-Shin-Gud is also the public puppet performance forum to bring people’s true voice andimagination. It awakes our forgotten tradition of oral storytelling and reinstalls it in our visually oriented contemporary cyber-culture. “Once upon a time, long long time ago . . . . ”  This is the opening sentence given to audiences when they interact with the Virtual Shaman before and after the artist’s musical and storytelling performance. It is the secret interface to open the door of inner communication and improvisation. People’s speech through the virtual puppet’s mouth is unexpected. It is unexpected hybrid created between their puppet and themselves. Virtual shaman helps people to tell and listen their story.

“…Puppet could tell the story. Puppet could tell him how to tell stories. So the big man and the puppet could tell the stories together. And this is how the big man learned to tell the stories…” —a part of stories told by an audience through the virtual puppet.


The Virtual Shaman is designed for anybody, not only for the trained artist and the technician. It uses simple technology that works perfectly in every environment. Everyone who enjoys puppetry can come to the microphone and be a storyteller, a puppeteer and, eventually, his/her puppet. The dynamic dialogue between puppet and puppeteer brings forth the energy of free imagination and improvisation in this performance forum. As in the shamanic ritual, such a spiraling interaction ultimately erases the line of distinction between all the binary pairs, such as artist/audience, reality/virtuality, mundane/spirit, east/west and puppet/puppeteer.  YONG-SHIN-GUD, as a contemporary shamanic ritual, frees all those who are oppressed in our predominantly structure- oriented society.