Ritualizing Interactive Media, from Motivation to Activation
- Journal Technoetic Arts 3.2. Intellect, Bristol, UK, 2005. pp.105-123
Excerpt from Introduction:
“...Despite its contemporary omni-presence, however, the idea and desire for interaction can’t simply be new and contemporary. Looking back on history, human desire for interaction has been continuously manifested from the day of primitive ritual to contemporary cyberspace. Our interactive routines have continued, from micro to macro scale, in order to confirm our existence in everyday life. This universal repetitive patterns of human activity is “the Ritual.” It is ritual because it is the archetypes of cycles driven by human instinct, regardless of their cultural and historical period. Based on Korean shaman ritual, yin/yang process, I am defining “The Form of Ritual”, to explain this fundamental human process of interaction and becoming.
Interactive technology is an ongoing expression of human desire. Its essential value could be found in understanding human being, nature and cosmos. Revealing its hidden essence, historical presence and spiritual value would be the next paradigm of interactive art practice.”
CoPuppet: Collaborative Interaction in Virtual Puppetry
- Book-Transdisciplinary Digital Art: Sound, Vision and the New Screen. Communications in Computer and Information Science (CCIS) Vol. 7. Eds. Randy Adams, Steve Gibson, and Stefan M. Arisona. Berlin, Germany: Springer, 2007. pp. 326-34. Co-Authors: Paolo Bottoni, Anna Labella, Stefano Faralli, Mario Pierro, Alessio Malizia, at Multimedia Lab, Univ. of Rome, La Sapienza, Italy
Puppetry is one of the most ancient forms of representation, diffused all over the world in different shapes, degrees of freedom in movements and forms of manipulation. Puppets make an ideal environment for creative collaboration, inspiring the development of supporting technologies (from carpentry to digital worlds). The CoPuppet project explores the possibilities offered by multimodal and cooperative interaction, in which performers, or even audience members, are called to affect different parts of a puppet through gestures and voice. In particular, we exploit an existing architecture for the creation of multimodal interfaces to develop the CoPuppet framework for designing, deploying and interacting during performances in which virtual puppets are steered by multiple multimodal controls. The paper illustrates the steps needed to define performances, also showing the integration of digital and real puppetry for the case of wayang shadowplay.
From Traditional to Virtual Interactive Puppetry: A Comprehensive Approach
- The proceedings of ISEA 2008, 14th International Symposium on Electronic Arts, Singapore, 2008. Co- authors: Stefano Faralli, Paolo Bottoni, Anna Labella at Univ. of Rome, La Sapienza, Italy
Excerpt from Introduction:
“...Deeply rooted in the Cartesian hierarchy of separation between subject and object, in the digital media culture a puppet is considered as something to be manipulated and controlled, ignoring its transformative relationship with the user. In the digital translation of puppetry, we are interested in how interactive technology will support ancient wisdoms of ritual, revealing transformative relationships of puppet and puppeteer, resulting in performances of great excitement, public engagement and reflection of the community, creating rich layers of mixed reality environments. In fact, Wayang Kulit offers complicate layers of mixed reality, which allow viewers to walk around the screen, watching the real puppet and its virtual form as a shadow at the same time. Mixed reality does not only happen in viewer/puppeteer’s consciousness at the moment the shadow becomes alive with its own spirit, but also in the viewer’s perception struggling between real and virtual forms of presentation, walking around the screen. This kind of setting creates rich platforms for discussing reality, virtuality and mixed reality all together, which we adopt as our methodology in order to explore the full potential of virtual puppetry.”
Redefining Puppet: Paradoxical Relationship between Human & Object
- The Proceedings of the 9th International Research Conference, Consciousness Reframed: a transdisciplinary inquiry into art, science, technology and society, New Realities: Being Syncretic, Eds. Roy Ascott, Gerald Bast, Wolfgang Fiel., organized by University of Applied Arts, Vienna, Austria.: SpringerWienNewYork, Jul. 3-5, 2008
Excerpt from Abstract:
“...Puppet could be redefined, emphasizing its relationship with the pup- peteer, especially in contemporary culture, where interaction is a significant theme in expressions of body and identity. Puppet is an ancient prototype of dynamic human interaction with object. Puppetry has a close connec- tion with shaman ritual. A ritual object such as a mask or puppet, used by shaman, brings a trance state of consciousness, resulting in a performance of great excitement, public engagement and re ection of community. Its im- portant role would be activating continuous layers of transformative process, transforming itself to something else and transforming shaman to someone else. Just as with ritual objects, puppet could be understood as a continuous transformative entity, by having dynamic relationship with the puppeteer, and therefore its de nition could be free from culturally rooted misunder- standing; it is not puppet as a fixed entity, to be controlled and manipu- lated, but rather puppet as a source of energy continuously reconstructing its physicality and tactility, arisen from relationship. By redefining puppet, this paper will explore the digital translation of puppetry, in a paradoxical rela- tionship between puppeteer and puppet, shaman and ritual object, and user and avatar, which will drive never-ending process of transformation. I call this “ritual.” This relationship is a key aspect of puppet, creating playfulness in layers of mixed reality, which will resonate into daily life.”
Shin-Myung: Playfulness Emerging from Oppression
- The Book Catalogue-HOMO LUDENS LUDENS: Locating Play in Contemporary Culture and Society, LABoral Centre for Art and Creative Industries, Gijon, Spain, 2008
There is enigmatic quality of playfulness when child play with puppets, and when puppeteer play with puppet. It seems to me such joyful play is originated from unavoidable distance between puppet and puppeteer. Distance that cannot be reached, emerging desire to overcome, phases of expanding body, gradually developing imaginary senses, nerves and physicality… This is enigma, caused by extreme condition of paradox, which will bring play after play until the moment of fullness. Structure of paradox is deeply embedded in Korean sub-consciousness in every aspects of culture. Korean understanding of playfulness lies on this paradoxical context; Ultimate state of playfulness called “신명 SHIN-MYEONG” is arisen from its opposite state of tremendous grief & anger called “한 HAN.” Real potential of playfulness cannot be achieved without oppression, which is fundamental of Yin/Yang relativity, as well as Korean shaman ritual. Korean shaman ritual “굿 GUD” drives incredible swirls of transformation, which initiates from the tremendous state of oppression, from there, jumps up to the ultimate state of playfulness. This paper will explain Shin-Myung and Han in Korean historical and cultural contexts, and examine how these opposite states contrast and balance each other, shaping the ongoing process of ritual, from daily life to sacred ceremony. Such paradoxical structure of play can be initiated from relationship between shaman/shaman object, and puppeteer /puppet, which could be re-enacted and augmented in a relationship between human/computer, and real/virtual, in my ongoing virtual interactive puppetry projects.
Searching for Love Impossible
- Technoetic Arts 8.2.: Intellect Ltd, Bristol, UK, 2010
We live in layers of mixed realities with continuous conflicts, negotiation and
becoming. I find it interesting to look at our situation as a continuous struggle in
the fusion of virtual/actual presences, and machine/human. However, we seem to be
far from understanding these relationships. Maybe the problem lies in the questions
themselves, promoting unidirectional preconceptions. By reversing the questions, we
might be able to identify something that has been missing in previous discussions:
Can we talk about disconnection to further discuss connection?
Can we talk about distancing to further discuss the process of becoming?
Can we talk about love impossible, to further discuss love?
This paradoxical journey would maintain us in the continuum of struggle and mas-
sive doubts, and encounter deep aspects of human being. In this article, I examine
the Korean cultural psyche called han in the context of a paradoxical state of con-
sciousness and the story of love impossible, and argue that han is a driving force
of transformative ritual in human consciousness. Beyond cultural boundaries, this
article also positions han in a microscopic view of contemporary life. Han underlies
paradoxical human relationships with objects, and with my virtual puppets, in the
context of distance, disconnection and love impossible.
Virtual puppet, my love impossible
- Metaverse, Creativity, Intellect ltd, Bristol, United Kingdom, 2010. Editor: Elif Ayiter/ Yacov Sharir, The Editorial board includes: Beth Harris, the director of digital studies at MOMA, Steven Zucker, the dean of graduate studies at FIT, Howard Rheingold, Stanford University
This article expounds upon the convergence of two seemingly disparate psychic manifestations; namely the mode of Han, a mindset deeply embedded in traditional Korean culture and the contemporary relationship of a human handler to his or her avatar in three dimensional virtual environments. As an artist whose artistic medium is virtual puppetry performed through three dimensional media, the author has found an extreme state of paradox as a key aspect of her own Korean culture, embodied by the concept of Han as the paradoxical state of the human psyche, initiated from the micro-politics of body. This article investigates the potential relation between human and virtual bodies, and avatars and their users in a paradoxical manner: this is a story of the love impossible.
Crying with the Virtual
- The Proceedings of ISEA 2011, 16th International Symposium on Electronic Arts, Istanbul, Turkey, 2011)
You look as big as a moon, a star, a butterfly, a firefly, and disappearing beyond the horizon. I don’t see your shadow. I don’t see even a bit of shadow.
This paper will deal with the ontological issue of the virtual body, exploring the rich meta sensory experience of the user with an avatar. Historically there are evidences of individuals inhabiting an alternative body in many ritual and performative contexts. Becoming a ritual body, or leaving/transcending the physical body, demonstrates the continuous emotional development and perceptual changes of body and mind over time. It provides a platform for understanding the potential of digital bodies in virtual space; what we call "avatars." This paper will explore the sensory dimension, emotional flow, and transcendental quality of the virtual body in the digital age. This will be investigated in the context of Korean experiential reality and the concurrent process of gradually transforming emotional psyches.
Virtual space affords an infinite depth along the Z-axis, bearing both physical and psychic dimensions. Virtual bodies traveling on such a vast Z-axis constitute new ritual bodies parting on Z, becoming increasingly remote, intangible, flexible, deconstructed, multiplied, and fragmented. Virtual bodies present us with a new challenge and distance to overcome in the process of interacting, communing, spiraling, coping or conflicting with the actual body. This paper introduces a paradoxical inquiry of seeking the full potentiality of being and experience between multiplied selves assisted by digital technology.
Korean experiential reality runs on infinite paradoxical loops, swirling like a tornado, deeply touching the human psyche and emotion. It integrates body and mind, creating a meta layer of experience and body, continuously negotiating the state of "neither-nor": neither physical nor mental, neither actual nor virtual. The emotional flow of the meta body will be explored through the perspective of Korean experiential reality—a paradoxical process of constant becoming—and through the "virtual" as the potentiality of being and experience in the digital age.
Virtual puppetry assisting the elder's life review
- The proceedings of ISEA 2013, 18th International Symposium on Electronic Arts, The university of Sydney Press, Australia, 2013
This paper will describe “VoicingElder”, an expressive storytelling platform for the senior population, using virtual puppets. VoicingElder combines reminiscent oral storytelling with virtual puppetry to develop a deeper understanding of self-worth. This form of storytelling through puppets is inspired by traditional Korean shamanistic ritual and the philosophy of han. VoicingElder aims to fuse Korean shamanistic ritual and twenty-first century virtual interactive technology to examine the current state of elder hood in Western culture, and to contribute in positive ways to the new elder hood that we face today.
VoicingElder: Avatar platform for the older adults, informed by multiple therapeutic traditions
- The Proceedings of ISEA 2014, 20th International Symposium on Electronic Arts, Dubai, 2014
In the United States and elsewhere, we are being challenged to think of a new phase of life called 'elder hood.' It is critical that we think about how to shape elder hood in ways that are beneficial to both the individual and community. VoicingElder is a reminiscence storytelling platform designed for older adults, using an avatar and facial recognition software. Storytelling and reminiscence are enormously important processes in old age because they nurture intergenerational sharing and communication, and allow seniors to express and strengthen their identities as they review and share their memories. In VoicingElder, the puppeteer (the senior) controls an on-screen avatar through facial recognition and voice input. The avatar may represent the senior's younger self, a relative, or an imaginary character. The software instantly translates the senior's facial movements into the facial movements of the avatar. The senior can thus emotionally embody the avatar, bringing a transformative and emotional character to their oral storytelling process. VoicingElder is a hybrid therapeutic tool that embraces several therapeutic traditions. Research in drama therapy, therapeutic puppetry, patient centered therapy, and avatar therapy all bring different ways to explore the user’s psyche, emotion and engagement in depth.
Book Chapter-Sensing without Sensing: could virtual reality support Korean rituals?
- The Point of Being. Editor: Derrick de Kerckhove, Cristina Miranda de Almeida: Publisher- Cambridge Scholars, UK, 2014. pp. 165 -196
“Sensing without sensing” explores the new dimension of sensorial experience shaped and suggested by virtual interactive technology. The core principle of “sensing without sensing” is the lightening of weights of actuality, in order to open the infinite space of meta realms, awakening underlying perceptive layers of our bodies, transforming them into the intangible, indefinable, and spiritual. It is a Korean experiential reality, continuously conflicting between actual and virtual layers: the chaotic state of being neither here nor there, in a quantum state of paradox. Sensing without sensing is a Korean ontological journey of taking off to the sky. It is quite an emotional journey, from extreme grief to joy, continuously dreaming, desiring, and eventually transcending the sky. It is the dynamic swirling dance of Korean cosmology, Sam-Taeguk, where emerge gradually developing emotional psyches such as Jung, Han, and Shin-Myeong.
The mode of sensing without sensing is further connected to the ontological issue of the virtual, supporting the paradoxical state of human experience in intervals between multiple realities. Technology has supported the distance between human and alternative bodies, using devices such as the rod and the stick, wireless and virtual, constantly increasing the state of paradox in the psychical dimension of the “active void”. Sensing without sensing contributes to the construction of the paradigm and concept of Point of Being by analysing the relationship between the virtual/potential and the actual. It suggests Point of Being in ongoing human endeavour to maximize intervals between multiple dimensions of body, and as a way to deal with it in a psychical and emotional dimension, searching for an alternative mode of senses, nerves and tactility.
Avatar Life-Review: Seniors Reminiscing Through Virtual Bodies
- The Proceedings of HCI INTERNATIONAL, Communications in Computer and Information Science (CCIS), Springer, Berlin, 2017
This paper will discuss experiences of older adult participants and the artist as a mediator and participant in ten avatar life-review sessions held at a senior living facility in Richmond, Virginia. The avatar system uses lip-syn- chronization, motion detection, and sentiment analysis to playfully engage the user’s life-review process. The system offered female and male interactive ava- tars for four developmental stages: childhood, adolescence, young adulthood, and elderhood. Participants’ experiences are studied anthropologically as a mode of personal and communal ritual and person-centered care in a common theatrical environment.
Avatar life-review: Virtual bodies in a dramatic paradox
- Ryu, S. (2017). Avatar life-review: Virtual bodies in a dramatic paradox. Virtual Creativity, 7:2. Bristol, UK: Intellect Ltd, pp.121–31. DOI: 10.1386/vcr.7.2.121_1
This article will examine ongoing avatar life-review projects, in the light of drama therapy concepts and methods, exploring a hybrid model of avatar/drama therapy in a virtually mediated environment. The avatar life-review platform will incorporate techniques/methods of drama therapy and psychodrama such as role playing, role- reversal, doubling and mirroring as a hybrid therapeutic model between VR and theater. It will address multiple states of self in dramatic paradox, especially for people with traumatic memories, disabilities, memory loss or mental health complications.
Avatar, mixed reality, drama therapy, life-review, storytelling, role, paradox
PERFORMING VIRTUAL BODIES
- Ryu, S. (2020). Performing virtual bodies. International Journal of Performance Arts and Digital Media, Taylor & Francis, 1-17. DOI: 10.1080/14794713.2020.1778899
- SJR: Visual Arts and Performing Arts Q1 (2020)
Following the technical advancement of cross-reality platforms – including virtual, mixed and augmented reality, the term ‘virtual bodies’ has become widespread. Despite its increased use, however, there has not been much investigation of ‘virtual bodies’ taking into account a cross-cultural approach that links traditional methods and digital outcomes. This article will demonstrate the methods of experiencing virtual bodies, in the context of Korean healing ritual, along with discussions of my avatar life-review projects, in the phase of wishing, facing, creating and performing virtual bodies. Since 2016, I have been working on avatar life- review platforms that have a mirrored body movement and lip- synchronization of live speech. This platform was implemented in two projects: in the Voicing Elder (2016), the avatar life-review for older adults at Gayton Terrace Senior Living, Richmond, VA; The Voicing Han (2019), the avatar life-review for cancer patients, at VCU palliative care, Massey cancer center, Richmond, VA. These avatar life-review sessions will be discussed as examples of wishing, facing, creating and performing virtual bodies, to proactively experiment, design and facilitate emotional and therapeutic experiences. It will enrich the potential of technically mediated bodies in digital space, supported by the constant development of virtual, interactive and simulation technologies.
Avatar; virtual bodies; Korean shamanism; healing ritual; life-review; art and medicine; mixed reality
Feasibility of Delivering an Avatar Facilitated Life-review Intervention for Patients with Cancer
- Dang, M., Noreika, D., Ryu, S., Sima, A., Ashton, H., Ondris, B., Coley, F., Nestler, J., & Del Fabbro, E. (2020). Feasibility of Delivering an Avatar-Facilitated Life Review Intervention for Patients with Cancer. Journal of Palliative Medicine, Vol. 24, No. 4. pp.1-7. Apr 2021.520-526. DOI: 10.1089/jpm.2020.0020
- SJR: Medicine Q1 (2020) / JCR: Clinical Medicine Q1 (2020)
Background: Life review, a narrative-based intervention, helps individuals organize memories into a meaningful whole, providing a balanced view of the past, present, and future. Examining how the content of memories contributes to life's meaning improves some clinical outcomes for oncology patients. Combining life review with other modalities may enhance therapeutic efficacy. We hypothesized a life review intervention might be enhanced when combined with a kinetic, digital representation (avatar) chosen by the patient. Our goal was to determine the feasibility of an avatar-based intervention for facilitating life review in patients with advanced cancer.
Methods: We conducted an observational, feasibility trial in a supportive care clinic. Motion capture technology was used to synchronize voice and movements of the patient onto an avatar in a virtual environment. Semistructured life review questions were adapted to the stages of child, teenager, adult, and elder. Outcome measures included adherence, recruitment, comfort of study procedure, patients' perceived benefits, and ability to complete questionnaires, including the Edmonton Symptom Assessment System (ESAS) and Functional Assessment of Chronic Illness Therapy-Spiritual Well-Being Scale (FACIT-Sp).
Results: Seventeen patients were approached, with 11/12 completing the intervention. The total visit time of a single intervention averaged 67 minutes. The post-intervention survey found all patients agreed or strongly agreed (Likert Scale 1–5) they would participate again, would recommend it to others, and found the experience beneficial. After one month, ESAS scores were either unchanged or improved in 80% of patients.
Conclusion: An avatar-facilitated life review was feasible with a high rate of adherence, completion, and acceptability by patients. The findings support the need for a clinical trial to test the efficacy of this novel intervention. Clinical Trial Number NCT03996642.
Avatar, digital representation, feasibility, life review, motion capture, virtual reality
Embodied storytelling and meaning-making at the end of life: VoicingHan avatar life-review for palliative care in cancer patients.
- Ryu, S., & Price, S. (2021). Embodied storytelling and meaning-making at the end of life: VoicingHan avatar life-review for palliative care in cancer patients. Arts & Health: An International Journal for Research, Policy and Practice. Taylor & Francis, pp.1-15. DOI: 10.1080/17533015.2021.1942939
- SJR: Arts and Humanities Q2 (2021)
This article presents the VoicingHan project as a new form of life-review mediated by a virtual body-digital avatar promoting the reconstruction of self and identity through play and performativity. Whereas traditional life-review uses interview as the primary means of the therapeutic process, VoicingHan life-review is mediated by a virtual body avatar with self-guided participation in one’s own life stories through embodied storytelling performance. The VoicingHan study enrolled 12-patients receiving outpatient palliative care at VCU Massey cancer center, from June to Aug 2019. The storytelling performances were recorded via avatar video format and distributed to participants for review and/or sharing. The present study considered the avatar videos as qualitative data emerging from these individually constructed life review narratives. In this article, the benefits of VoicngHan life-review were demonstrated by analyzing the avatar video narratives, based on theoretical and developmental frameworks. Through the lens of Social Constructivism, this qualitative study deepens our understanding of human-avatar interactions to engage life review processes and the role of creativity, play and performativity within the liminal space of a life-threatening illness.
life review, end of life care, digital media, avatar, virtual reality, mixed reality, drama therapy, qualitative data analysis, social constructivism, embodiment