Home

verycracked / starbase

publicverycracked/Starbase
Overview Code History Branches Pull requestsIssuesInsights
main
HomeOverview Code PRsIssues

Add hand gesture control via MediaPipe vision

7 months ago

895e131
Authored
V.C. Billingsley2/20/2026, 5:52:23 AM
Introduce webcam-based hand tracking using @mediapipe/tasks-vision.
hands.js wraps the MediaPipe HandLandmarker API, processing video
frames at ~30fps and emitting normalized gesture events (pan position,
pinch state, hand size, handedness).

Gesture mapping:
- Left hand movement: orbit rotation (anchor-based, not absolute)
- Left hand proximity: zoom (hand size maps to orbit distance)
- Right hand open + move: orbit rotation (same as left)
- Right hand pinch + move: pan (shifts the look-at target)

The gesture handler disables OrbitControls while tracking, and
smoothly interpolates camera position via exponential smoothing
(factor 0.1) to avoid jitter. When no hand is detected, standard
mouse/trackpad controls re-enable with auto-rotation.

A toggle button ("Hands [off/on]") in the top-right corner lazy-loads
the MediaPipe model on first activation. This avoids the ~5MB model
download until the user opts in.

Anchor-based rotation is the right approach here because absolute
hand-to-angle mapping causes jumps when the hand re-enters frame.
By storing the initial hand position as an anchor and computing
deltas, the rotation is always relative and smooth.

Parent2195561

2 files changed
  • src/hands.js+301−0
  • src/main.js+169−0