Humane opened up the AI Pin’s gesture recognition system to third-party developers in July, releasing a public API that lets developers define custom hand gestures and bind them to application logic. The API provides real-time hand pose tracking (26-point skeleton), supports single-hand and two-hand gestures, and works alongside voice commands and on-device LLM inference to enable multimodal interaction.
Watch: Humane AI Pin — CEO Shows Us What It Can Do (YouTube)
Custom gesture vocabularies for domain-specific work
Until now, the AI Pin supported a fixed set of gestures: pinch to select, swipe to scroll, open palm to dismiss. Developers can now define their own — a photographer could use a two-hand frame gesture to crop an image, a musician could define hand positions that map to MIDI pitches, a surgeon could have gesture commands that don’t risk contaminating sterile hands by triggering a nearby touchscreen. The gesture space is now programmable.
Multimodal fusion at the gesture level
The API lets gestures fire events that trigger voice transcription, vision analysis, or local model inference in sequence. A “measure” gesture could capture the scene, run dimension estimation through a local vision model, and prompt the on-device LLM to convert measurements into a scale drawing. The device becomes a tool platform rather than an interface fixed at the factory.