Gesture is a voice and hand controller for macOS, built in a single sitting. The webcam watches for hand positions, the microphone listens for commands, and the two threads run side by side, so an action can be triggered either by saying "open Safari" or by a gesture at the screen. It is a proof of concept.
The whole program is one app.py file with two functions running on separate threads. Voice control uses SpeechRecognition with Google's recognizer for transcription, then shells out to osascript or the macOS say command to carry out the action. Two commands are handled: open Safari, and report the time. Everything else falls through to "Command not recognized." The two handlers are a scaffold for adding more, and no more were ever added.
Gesture control uses OpenCV plus MediaPipe to pull hand landmarks from the webcam feed, and pyautogui to fire keystrokes. The current gesture is a check of whether the tip of the index finger (landmark 8) is above the wrist (landmark 0), and if so, a press of spacebar. That is enough to pause a YouTube video by pointing up at the screen. A one-second time.sleep after each trigger keeps it from firing fifty times per raised finger.
index_tip = hand_landmarks.landmark[8]
wrist = hand_landmarks.landmark[0]
if index_tip.y < wrist.y:
pyautogui.press('space')
time.sleep(1)
There is also a stubs.py, which exists for the sole purpose of forcing modulegraph to detect rubicon.objc when bundling the app into a .app with py2app. macOS is not accommodating about shipping a Python app that wants webcam and microphone access.
What does not work: there is no real command grammar, just two hardcoded if branches; no gesture vocabulary beyond "finger up = space"; and no error recovery if the webcam is not accessible. A wake word would stop it transcribing every word said all day. Swapping Google's recognizer for something local such as Vosk would keep every utterance off Google's API.
It shows how few moving parts the basic version of this needs. GitHub Repo.