FaceGesture AI Twister: A Multimodal AI Physical Puzzle

GitHub: https://github.com/ystyao/facegesture-ai-twister

FaceGesture AI Twister is a Raspberry Pi-powered game that turns on-device AI into a hilarious, fast-paced test of human coordination. Instead of a traditional controller, your face, fingers, and spatial awareness are the inputs. Players have just five seconds to contort themselves into a randomly generated combination, like holding a “disgusted” face, flashing a peace sign, and keeping one hand within 20 cm of a sensor, then freeze for three agonizing seconds while the camera captures the chaos.

The system runs an int8-quantized EfficientNet-B0 model (TensorFlow Lite) for on-device emotion recognition alongside MediaPipe hand-gesture classification. I led the overall game design and hardware architecture, laying out the display and sensors and integrating the final physical build. Instead of a pass/fail buzzer, each round ends with an AI commentator: a cloud LLM reacts to the results, and Piper TTS speaks it aloud, cheering or playfully roasting the player’s blunders.

Course: AIPI590 Advanced Topics in AI for Product Innovation - AI in the Physical World (2026)
Team: Diya Mirji, Ishan Madan, Shuangting Yao
My Role: Overall game design, hardware design & integration