7 months ago
895e131Introduce webcam-based hand tracking using @mediapipe/tasks-vision.
hands.js wraps the MediaPipe HandLandmarker API, processing video
frames at ~30fps and emitting normalized gesture events (pan position,
pinch state, hand size, handedness).
Gesture mapping:
- Left hand movement: orbit rotation (anchor-based, not absolute)
- Left hand proximity: zoom (hand size maps to orbit distance)
- Right hand open + move: orbit rotation (same as left)
- Right hand pinch + move: pan (shifts the look-at target)
The gesture handler disables OrbitControls while tracking, and
smoothly interpolates camera position via exponential smoothing
(factor 0.1) to avoid jitter. When no hand is detected, standard
mouse/trackpad controls re-enable with auto-rotation.
A toggle button ("Hands [off/on]") in the top-right corner lazy-loads
the MediaPipe model on first activation. This avoids the ~5MB model
download until the user opts in.
Anchor-based rotation is the right approach here because absolute
hand-to-angle mapping causes jumps when the hand re-enters frame.
By storing the initial hand position as an anchor and computing
deltas, the rotation is always relative and smooth.Parent2195561