DX Portfolio
A Fan into Opera
Interactive Installation | Independent Project
This interactive installation converts Cantonese Opera from a passive viewing experience into an embodied, participatory one. Inspired by the "Fragrant Lament" scene from The Flower Princess, it invites two participants to jointly perform a moment of the opera, one controlling character movement through fan gestures, and the other animating the falling blossoms on stage. The work asks: what if heritage wasn't something you watched, but something you did, together, with another person.
TOOLS
Audrino, Gyroscope Sensors, Ultrasonic Distance Sensor, LED Lighting, TouchDesigner, Prusa 3D Printer, Autodest Fusion
INSTRUCTOR
Professor Matthew Francisco, Megan McNames
TIME
March 2026 - May 2026
AFFILIATION
Indiana University Bloomington, Informatics Department
BACKGROUND & MOTIVATION
Cantonese Opera is both a performing art and a living form of cultural heritage. Unlike heritage that can be preserved in static museum archives, performance art heritage is dynamic and multisensory. It should be re-performed to stay alive. In a traditional opera performance, the audience is positioned as a spectator, watching a fixed narrative unfold on stage.
This prototype draws on Nina Simon's argument that cultural institutions should treat audiences as active contributors to meaning-making, not passive viewers (Simon, 2010). It asks how a centuries-old art form can shift from something watched to something enacted.
This question is not only theoretical. For Chinese immigrant communities, Cantonese Opera has historically been a way of building community abroad, as a means of connecting with compatriots, maintaining a shared identity, and passing down a "living" heritage across generations. The art form has always functioned as a bridge between people.
This installation tries to translate that social function into the interaction design itself. It is designed for two people acting together, so that the act of "performing" the opera also becomes an act of connecting with another person.
THE CONCEPT
The work is inspired by a moment from the scene "Fragrant Lament" (香夭) in The Flower Princess (帝女花), in which the princess and her lover choose to die together beneath falling blossoms. The installation uses it as an emotional and structural reference. Participants perform within its emotional register, tension, hesitation, resolve, using the fan as the interface.
The fan was chosen deliberately as the tangible interface because it is a core expressive tool in Cantonese Opera. Repurposing it as a sensor-based controller keeps the interaction rooted in the vocabulary of the art form itself, rather than introducing an unrelated input device.
EXPERIENCE DESIGN
Cantonese Opera is a broad concept, and translating it into a single installation meant narrowing it down. I chose The Flower Princess (帝女花) as the source play partly because it is one of the most celebrated and widely known works in the Cantonese Opera repertoire. Therefore, it can be a recognizable entry point even for people unfamiliar with the art form, and within it, one scene as the moment that could best represent the art form within the scope of an installation.
At the same time, because performance art is fundamentally about bodily expression, the experience needed to involve movement. That led me to draw the interaction from a prop that already belongs to Cantonese Opera itself, namely the fan. Using the opera's own prop as the interface meant participants could physically hold and move the same object performers use, so they could feel something of Cantonese Opera through their own body.
The sketch of the stage
The sketch of interaction
PROTOTYPING
The earliest prototyping started with paper. I made a minimal paper puppet to explore how it could move, initially imagining it worn directly on the hand like a glove puppet. However, given the difficulty of fabricating a detailed Cantonese Opera character puppet by hand, and weighing that against what the project actually needed to communicate, I shifted to 3D printing instead. The next low-fidelity prototype was a 3D-printed fan, made to explore how the fan itself could move and be held, drawing on how fans are traditionally used in Cantonese Opera performance. In parallel, I worked through the underlying technical questions: how the sensors should be used, and how to resolve the connection between Arduino and TouchDesigner.
The Prototype of Fan and Puppet
The Sketch of the Stage
The 3D printed fan and the paper puppet
The Sensers Contacting with the Motor and with TouchDesigner
Testing the Serial Communication between the Sensor and TouchDesigner
MODELS & 3D PRINTING
The character figures started from reference images of the characters from this scene of the opera, collected to capture their costumes, poses, and silhouettes. These images were then fed into MeshyAI, an AI-driven 3D model generation tool, to produce 3D models of the characters. The resulting models were printed on a Prusa 3D printer, producing physical figures that could later be mounted onto the servo motors.
The Model of the Female Figure Generated from the Image
The Model of the Male Figure Generated from the Image
The Printed Models
INTERACTION CALIBRATION
Calibrating the gesture recognition was an iterative process. I tested different threshold values for the gyroscope's X, Y, and Z axes, since the same physical gesture, tilting the fan vertically, for example, didn't always produce identical readings. Using the Arduino Serial Monitor, I tracked the live orientation and raw sensor values in real time, which made it possible to see exactly where a gesture's data range began and ended, rather than guessing at numbers. While testing, I also paid attention to how the fan was naturally held and moved, the range of motion, the speed, the angle, and adjusted the detection thresholds to match those natural movements rather than forcing users to perform an unnaturally "correct" gesture. In parallel, I tuned the servo motors' rotation ranges and movement angles, testing fast, repeated, and simultaneous inputs from both fans to find safe angle limits that kept the two figures from colliding with each other or the stage structure.
Moving the Fan to Trigger Characters' Shaking
The Characters Moving with the Rotation of the Fan
Different Orientations in Arduino
The Record of Rotation Degrees
Calibrating the gesture recognition was an iterative process. I tested different threshold values for the gyroscope's X, Y, and Z axes, since the same physical gesture
The Diagram of Fan Movement with Character Movement (AI-assisted)
ITERATION
The way the characters were mounted went through a key iteration. In the first version, the two characters were printed as a single connected piece and mounted on top of two stacked servo motors, with the goal of giving them freer, more layered movement. In practice, this setup made it difficult to control each motor independently. The motors were stacked on the same axis, rotating one always pulled the other along with it, so both characters ended up turning to the same angle no matter which motor was actually driven. To fix this, the second version separated the design into two distinct characters, each mounted on its own independent motor, with each fan's distinct gesture vocabulary mapped to a different motor. This gave each character its own axis of movement, and finally let the two participants' fans control their characters independently.
The Second Version of Character Models Attached on the Motors
The First Version of Character Models Attached on the Motors
Early testing showed that participants enjoyed the effects produced by the fan but couldn't connect their gestures to a clear meaning, since most lacked background knowledge of The Flower Princess or Cantonese Opera, which led to introducing Xi Qiao that gives story and cultural context plus interaction instructions before the experience begins. Xi Qiao is originally a newspaper page advertising a Cantonese Opera performance, introducing its actors, song lyrics, and story. Here it is adapted to introduce the background story of this installation, which can be placed on the sensor box, activating the lighting and music for the installation.
The Final Design of Xiqiao (AI-assisted)
FABRICATION
The stage was built as a simple physical set using wood board, black cardstock, and a few 3D-printed elements quickly modeled in Fusion 360. Two slender printed strips were glued directly onto the servo motors, and the character figures were then mounted onto these strips, which gives each character a clean lever arm that translated the motor's rotation into visible movement. To complete the scene, a 3D-printed moon was placed toward the back of the stage. A small box sat at the very front, with a folding screen placed on top of it and the character figures positioned behind. All the wiring was hidden inside this front box, keeping the visible stage clean of cables. A small stand holding Xi Qiao was folded out of paper and placed at the very front.
The Process of the Stage and Characters Assumbly
The Final Prototype
The Sensor Attached to the Fan
THE DEMO
The demo presents a complete interaction cycle, demonstrating the relationship between user input and system output. It shows two stages of interaction, namely the use of tickets to trigger visual and musical elements, and the use of fan gestures to control character movement and stage effects. The fan functions as a storytelling medium that enables embodied interaction and performative engagement. Therefore, the interaction with the fans is essential to immerse participants physically and emotionally within the experience.
The Experience Diagram (AI-assisted)
Technical Connections
The Installation
The Effect with Projected Falling Flowers
The Xi Qiao
REFLECTION
The most important aspect that brings my design into life is to establish a meaningful relationship between the interaction methods and the expressive experience I want the participants to have through the installation. A key challenge was designing an interaction system that could effectively translate elements of Cantonese Opera into an interactive installation while maintaining a coherent experience.
The final design remained conceptually close to my original plan, although changes occurred throughout development. Initially, I planned to incorporate several embodied interaction methods using traditional opera props such as fans, water sleeves, and musical instruments. However, during the prototyping process, I decided to focus solely on the fan. Concentrating on a single prop allowed me to explore embodied interaction more deeply, rather than creating superficial interactions with multiple objects. Another major change involved the interaction logic itself. My original idea was for fan movement to manipulate opera music, with the musical data triggering character movements. However, coding experiments revealed that the music data was too unpredictable to reliably control the motors. As a result, I changed the interaction design. The fan can directly control character movements. This adjustment created a more responsive relationship between participants’ gestures and the system feedback.
Sketching, 3D printing, and coding experiments were the most valuable methods during the development. Sketching allowed me to quickly capture ideas as they emerged, and then compare or further develop some of them. 3D printing helped me test the dimensions, assembly, and physical relationships between stage components and characters. Handling physical prototypes also allowed me to better imagine user interaction within the installation. Coding experiments were important for developing the interaction system. Through iterative testing, I explored how 3D-printed characters could move using motors and how fan gestures could trigger different movements. The interaction evolved from simple orientation-based controls to more refined angle recognition that enabled different preset actions and expressive possibilities. Research into Cantonese Opera performances also informed the design process. I studied specific opera stories, fan choreography, and stage elements to better understand how embodied gestures contribute to narrative expression in opera.
The primary interaction in the installation is the relationship between fan gestures, character movement, and stage effects. According to Shedroff’s Experience Cube (2014), this design strongly relates to the dimensions of control and feedback. The installation only becomes active when participants place the ticket in the designated area and begin interacting with the fan. Without user participation, the installation remains inactive. Participants’ gestures directly influence the system’s responses in real time. Through embodied interaction, users physically engage with the installation and shape the performance. This creates a higher level of interactivity compared to traditional opera performances. The project also connects to the creative dimension of interaction design. Instead of watching a predetermined performance, participants use the fan to generate their own stories with character movement. In this way, users actively participate in reimagining performance-art heritage. This participatory structure reflects some thoughts of the contemporary performance theory, in which the binary opposition of performers and audience is blurring, and audience’s action becomes part of the performative event (Fischer-Lichte, 2008).
The installation has several affordance, including embodied affordance, cultural affordance, and feedback affordance.
-
Embodied affordance encourages participants to use bodily movement rather than button-based interaction. This is inspired by embodied interaction put forward by Dourish (2001), who emphasize the physical engagement with the interactive system. Users wave and rotate the fan to influence narrative and visual elements of the system.
-
Cultural affordance is communicated through the use of opera fans, opera character models, and tickets. These elements carry recognizable cultural meanings associated with Cantonese Opera.
-
Feedback affordance is achieved through the system’s responses to user gestures. Fan movements trigger visual effects and character motion in real time, helping participants understand the relationship between their actions and the system.
These affordances are visible in the installation. The fan functions as both a cultural symbol and an interaction device. Together with the user flow and instructions, the installation encourages participants to experiment with the fan and create their own operatic puppet performance.
THE FURTHER STEP
The next phase of playtest will focus from the system to the relationship between the two participants, since that relationship is impotant for this project.
The next iteration should focus on improving the robustness of gesture recognition and reducing false triggers, which currently undermine participants' sense of control over the characters. This refinement happens on two levels. Motion refinement addresses how the characters move over time, so a motion reads as one continuous expressive gesture rather than a sudden jump between states. Structural refinement addresses how that movement is physically produced at the mechanical level, such as servo response speed and range of motion，so the characters can move in ways that feel more expressive and natural rather than constrained by the hardware.
Moonlight on Mott Street
Projection Mapping | Independent Project
This projection mapping project is a continuation of the interactive Cantonese Opera installation. Two 3D-printed opera figures stand on the stage, and the video projection serves as the scenic backdrop. The visual content is based on the streetview of Manhattan’s Chinatown, NYC. The front stage features moving streetscapes and large-scale urban scenes, transitioning from historical imagery to contemporary views of the same area. Two side walls display animations created from Gaussian Splats of iconic Chinatown signs and neon lights. Through both macro and micro perspectives of the neighborhood, the work reveals different aspects of everyday life in this community.
By integrating the projected environment with Cantonese Opera elements, the installation expressed the role of opera performance in shaping Chinatown’s community life, expressing cultural identity, and connecting memories of the homeland with contemporary lived experience.
TOOLS
Scaniverse 3D Scan, Seedance 2.0, CapCut, Madmapper
INSTRUCTOR
Professor Katina Bitsicas
TIME
June 2026
AFFILIATION
Indiana University Bloomington, Media School
FOOTAGE COLLECTION
The Footage of Streetview Generated by Seedance 2.0 and a prompt
An Archival Image of Mott Street, Chinatown
Scanning and Animating with Gaussian Splats in Scaniverse