MUSE GESTURES logoMUSEGESTURES
01
The Installation

Conduct sound
with your hands.

A real-time sound installation where movement becomes music. A webcam tracks your hands, a custom engine translates gestures into a living pad, a pentatonic laser harp, and a vocal ostinato — all in the browser.

DIVE IN
02
How it works

Three layers,
two hands,
one instrument.

Every gesture maps to a specific sound parameter. The left hand sculpts the atmosphere — a drone pad, a vocal ostinato, the reverb amount. The right hand plays the melody.

The audio engine is fully synthesized in the browser through the Web Audio API. No backend roundtrip, no latency beyond the camera frame.

Tracking

MediaPipe Hands

Google on-device hand landmarker. 21 keypoints per hand, stable inference, GPU-accelerated. Confidence thresholds and a temporal consistency filter reject false positives.

Engine

Web Audio API

A custom graph of oscillators, filters, convolution reverb, and feedback delay — all running natively in the browser, sample-accurate and zero-latency.

Synthesis

Three voices

An evolving pad inspired by the "Forsitan Modulare" - Draen module, a monophonic sample-based laser harp, and a looped vocal ostinato.

03
The gestures

Six gestures.
Two roles.

The left hand controls the atmosphere. The right hand plays the melody. Each gesture below is wired to a specific parameter of the audio engine — move your hand, hear the change instantly.

Hold your hands in the frame, in front of the camera. Smooth movements produce smooth sound.

Fig. 1 — Six gestures, two hands. The left hand sculpts the drone; the right hand plays the melody.

Fig. 1 — Six gestures, two hands. The left hand sculpts the drone; the right hand plays the melody.

Open PDF
04
Begin

Ready to play?

You will be asked for camera access. The audio engine starts on your input only — we require a user gesture before any sound can play.