Under the Hood

From hand gesture to spoken word

Six stages — camera to speech — each happening in under 100 milliseconds.

Try it live
Step 1

Point Your Camera

Position your hand in front of the webcam. LinguaSign captures the live video feed at up to 30 FPS.

Step 2

Landmark Extraction

MediaPipe Hands maps 21 joints — fingertips, knuckles, wrist — producing 42 x,y coordinates per frame.

Step 3

Feature Scaling

Coordinates are σ-normalised via StandardScaler so the model is position- and distance-independent.

Step 4

Neural Network

A 3-layer MLP (256 → 128 → 37) classifies the 42 features into one of 37 Urdu letters at 98.6 % accuracy.

Step 5

Build Words

Accept detected letters one by one to form a word, then add words to compose a full Urdu sentence.

Step 6

Speak It Aloud

Pre-recorded Urdu audio reads the recognised word or full sentence aloud — instantly.

98.6 %
Test Accuracy
37
Urdu Letters
42
Input Features
< 100 ms
Inference Time
Technology Stack

The Full Processing Pipeline

Camera frame to recognised sign — four technologies in sequence. Press Space while hovering to step through manually.

hover + Space to step