Made by AI, Directed by David Stein

Search
XR / Spatial Computing

WebXR Hand Tracking: Getting Started

This site's WebXR tutorials have covered hit-testing, depth sensing, and spatial anchors — all built around controllers. Here's how to read a user's actual hands, directly in the browser, with no engine required.

3 min read

This site’s earlier WebXR tutorials — hit-testing, depth sensing, spatial anchors — all assume a controller in each hand. Modern headsets (Quest, Vision Pro, and others) can track bare hands directly, and the browser can access that data through the same WebXR session API you’re likely already using, via the Hand Input extension. No engine, no native SDK — this works in plain JavaScript and Three.js, the same stack as this site’s earlier no-engine WebXR tutorial.

What hand tracking actually gives you

Unlike a controller, which reports a single position and orientation, WebXR’s Hand Input API reports a full skeleton: 25 joints per hand, each with its own position and orientation in 3D space — one for each finger segment, the wrist, and the palm. That’s dramatically richer than controller input, but it also means your code has to decide what to do with 25 data points per hand instead of one.

Requesting hand tracking in a session

Hand tracking is requested as an optional feature when you create your WebXR session, alongside whatever other features (hit-test, local-floor, etc.) your app already uses:

navigator.xr.requestSession('immersive-vr', {
  optionalFeatures: ['hand-tracking', 'local-floor']
}).then(onSessionStarted);

Inside your render loop, you check each XRInputSource for a hand property. If it’s present, that input source is a tracked hand rather than a controller:

for (const inputSource of frame.session.inputSources) {
  if (inputSource.hand) {
    // This is a hand, not a controller
    const indexTip = inputSource.hand.get('index-finger-tip');
    const jointPose = frame.getJointPose(indexTip, referenceSpace);
    if (jointPose) {
      // jointPose.transform.position gives you the fingertip's XYZ
    }
  }
}

The joint names ('index-finger-tip', 'thumb-tip', 'wrist', and so on) follow a standardized naming scheme defined by the spec, so the same code works across any headset that implements Hand Input correctly.

How this differs from MediaPipe’s hand tracking

This site’s earlier MediaPipe hand-tracking tutorial covers a webcam-based approach: a 2D camera image goes in, estimated hand landmarks come out, computed by a model running in your own code. WebXR’s Hand Input API is a different pipeline entirely — the headset’s own onboard sensors and firmware do the hand tracking, and the browser just exposes the already-computed joint poses through a standard API. That means no model to run yourself, but it also means you’re limited to whatever hardware the user is wearing actually supports, and you only get this data inside an active XR session — it’s not a general-purpose, camera-based hand tracker you can point at an arbitrary webcam feed.

A first project: fingertip markers

A good first exercise is placing a small visible marker — a sphere in Three.js, for instance — at each fingertip, updated every frame from the joint poses. Once that’s working and stable, the natural next step is pinch detection: measure the distance between the thumb tip and index-finger tip joints, and treat a close enough distance as a “pinch” gesture — the same interaction pattern Quest’s own system UI uses for selection, and the foundation most WebXR hand-tracking interactions build on from here.

Related Reading