FaceGesture AI Twister: A Multimodal AI Physical Puzzle
GitHub: https://github.com/ystyao/facegesture-ai-twister
FaceGesture AI Twister is a Raspberry Pi-powered game that turns on-device AI into a hilarious, fast-paced test of human coordination. Instead of a traditional controller, your face, fingers, and spatial awareness are the inputs. Players have just five seconds to contort themselves into a randomly generated combination, like holding a “disgusted” face, flashing a peace sign, and keeping one hand within 20 cm of a sensor, then freeze for three agonizing seconds while the camera captures the chaos.
The system runs an int8-quantized EfficientNet-B0 model (TensorFlow Lite) for on-device emotion recognition alongside MediaPipe hand-gesture classification. I led the overall game design and hardware architecture, laying out the display and sensors and integrating the final physical build. Instead of a pass/fail buzzer, each round ends with an AI commentator: a cloud LLM reacts to the results, and Piper TTS speaks it aloud, cheering or playfully roasting the player’s blunders.
Course: AIPI590 Advanced Topics in AI for Product Innovation - AI in the Physical World (2026)
Team: Diya Mirji, Ishan Madan, Shuangting Yao
My Role: Overall game design, hardware design & integration
Weaving Mountain Memories
- A Blockchain Product for Preserving Yunnan Ethnic-Minority Intangible Cultural Heritage Crafts
Weaving Mountain Memories is a design proposal for a digital collaboration and archival platform focused on preserving the intangible cultural heritage (ICH) handicrafts of ethnic minorities in Yunnan, China. Addressing the structural dilemmas of traditional archiving—where craftspeople are reduced to "objects of record" and digital memories risk loss under single-institution control—this project proposes a redesign oriented toward cultural sovereignty. Integrating the FISCO BCOS consortium blockchain, the system uses NFC tags placed beside physical exhibits as anchors. This seamlessly extends offline exhibitions into a 3D digital space, allowing visitors to immersively engage with craft memories and directly support the creators.
Distinct from markets that over-financialize heritage, the core of this design leverages media technologies to reshape ownership structures. Through the pioneering "Proof of Cultural Contribution (PoCC)" multi-party co-signing mechanism, the proposal grants heritage inheritors a primary participatory veto over their own archives at the smart contract level. By combining non-transferable ERC-721 authenticity credentials bound to physical works with an automated, creator-favored revenue-splitting protocol, the project dismantles institutional monopolies. Ultimately, it establishes a transparent, equitable creator ecosystem with cross-cycle vitality for traditional handicrafts.
Course: FINTECH 564 Blockchain (2026)
The Living Marketplace: A Visual Ethnography of Cultural Exchange in Xizhou Market
This project presents a visual ethnography of the vibrant, everyday transactions and embodied practices within the periodic market of Xizhou, Yunnan, China. Developed as a web-based 3D Virtual Gallery using Three.js, the project functions as a "living archive" that records local knowledge and social relations through an immersive, first-person roaming experience. By moving beyond top-down institutional heritage narratives, the gallery utilizes a thematic Cluster Tree to reorganize 56 field photographs into networks of local culture—ranging from traditional Bai headwear to regional medicinal herbs. This spatialized narrative encourages "slow looking" and treats digital media not just as a recording tool, but as a critical intervention that reclaims the marketplace as a social space full of memory, emotion, and lived history
Course: ARTHIST 580S Digital Humanities Proseminar (2025)
Adam & Susie’s Face
It is an educational digital game designed for children aged seven and above, specifically targeting those with Autism Spectrum Disorder (ASD). The game aims to help children recognize and appropriately respond to facial expressions in specific real-world scenarios, fostering social skills development in a fun, supportive environment. Developed in collaboration with an Applied Behavior Analysis therapist, the game was tested with two children with ASD to ensure it was therapeutic and engaging.
The game features two key mechanics:
Facial Expression Matching – Players drag faces showing three emotions (happy, angry, sad) and match them with the character, Adam. After each match, prompts appear to explain the emotion and its social implications.
Interactive Dialogue – Players type sentences to Adam, who responds by changing his facial expression. It helps children practice recognizing facial cues and formulating appropriate verbal responses.
Built on the Scratch platform, with elements of machine learning, the game provides a dynamic and adaptive learning experience tailored to the emotional and social needs of children with ASD.
Course: MSTU 6000 Advanced Design Educational Games (2018)
Echo
It is a portable journal app designed to help users reflect on their daily lives and enhance mindfulness. Developed using Android Studio and programing language Java, with the interface designed in Figma, Echo encourages users to not only record their experiences but also reflect on and learn from their past.
I was responsible for both the interface design and app programming. The app's unique mechanics support users in their mindfulness practice, making it more than just a journal but a tool for personal growth.
Course: MSTU 5031 Object-Oriented Theory/Programmng II (2019)
Team Member: Leina Lei