Unity’s XR Interaction Toolkit, combined with the XR Hands package, gives VR developers an official, maintained pipeline for controller-free hand interaction — hover, grab, pinch, and point gestures read directly from a headset’s built-in hand-tracking cameras (Quest, and other OpenXR-compatible headsets), without writing custom skeletal-tracking code from scratch.
Watch: Hand Tracking with Unity XR Interaction Toolkit (Valem Tutorials, YouTube)
Step 1: Install the right packages, in the right order
Through Unity’s Package Manager, install XR Interaction Toolkit, XR Hands, and the relevant OpenXR plugin for your target headset. Enable hand-tracking specifically inside your OpenXR project settings — it’s not on by default even after the packages are installed, and skipping this step is the most common reason a fresh setup shows controller input but no hand data at all.
Step 2: Get the default hand visualization working before customizing anything
XR Interaction Toolkit ships with a prebuilt hand-tracking sample that renders a basic tracked hand mesh matching real hand movement, with zero custom code required. Import and run this sample first, before touching your own scene — it’s the fastest way to confirm your headset, OpenXR settings, and Unity project are actually correctly configured, isolating hardware/setup problems from anything you might get wrong in your own interaction logic later.
Step 3: Interactable objects work the same way for hands as for controllers
The toolkit’s core interaction model — an object marked as an Interactable, responding to hover, select, and activate events — works identically whether the interactor is a physical controller or a tracked hand. This is the toolkit’s actual design strength: build your grab-and-throw or button-press logic once, against the abstract interactor/interactable pattern, and it works for controller-based and hand-tracking-based XR setups without a parallel code path for each input method.
Step 4: Add gesture recognition once basic interaction works
Beyond simple hover/grab, XR Hands exposes enough raw joint data to build custom gesture recognition — a pinch, a fist, a specific finger configuration — as a trigger for something other than direct object manipulation (a menu toggle, a locomotion mode switch, a UI shortcut). Build this only after basic interaction is solid; gesture recognition is meaningfully harder to get feeling reliable than hover/grab, since it depends on tracking-data noise tolerance that’s easy to underestimate until you’re testing with real, imperfect hand-tracking data rather than a controller’s clean button state.