Actual Catmoji v1.9.1 panel, cropped from the application. Camera off; no gesture predictions are shown.
Make a model feel like play.
A prediction is just a label until someone can do something with it. I built Catmoji to connect a hand gesture to an immediate character response, then turned the same input into a timed challenge.
Hand landmarks
MediaPipe supplies 21 points. Their coordinates become a normalized 63-value input.
A steadier label
Distance-weighted kNN votes across seven neighbors; a seven-frame window smooths the result.
A visible response
The chosen label drives a Moji sticker and optional voice, or matches the current Challenge target.
The five emotion names are gesture labels, not a measurement of how someone feels or a facial-expression diagnosis.
The decisions behind it.
Make the gesture understandable.
The Challenge guide maps one to five fingers to happy, sad, angry, surprised and excited. A visible target, four-second round and three lives give the same classifier a concrete job beyond showing a probability bar.
Stabilize before reacting.
Camera predictions can change between frames. Temporal voting softens that flicker; Challenge also requires a matching gesture to hold for 650 milliseconds. These are interaction safeguards, not a measured accuracy guarantee.
Keep the vision path in the browser.
Hand landmarks and kNN inference run locally in the page. Voice is separate: Gemini TTS uses a server proxy, with browser speech as a fallback. Keeping those paths explicit avoids calling the whole product offline or entirely on-device.
From classifier to playable product.
A live, open-source product with Normal, Intro and Challenge modes, a 2,460-sample gesture model, and an installable PWA shell. I built the interaction, classifier integration, Moji presentation and voice flow as one solo project.
- JavaScript
- MediaPipe
- kNN
- PWA
Recognition depends on camera access, lighting and the hand pose. Models, libraries and optional cloud voice still need network access. These captures document the interface, not a recognition benchmark; a real-camera recording is not included yet.
Try Catmoji