Signmaster

Class Embedded Systems
Project Duration 2 weeks
Team Size 2 members

My Role
Circuit Design
Physical Prototyping
Algorithm Design
Presentation

What is Signmaster?

Signmaster is a glove that detects hand gestures of the American Sign Language Alphabet and converts them into text for display on a screen.

Choosing a Project and Scope

My teammate and I both had an interest in doing something that would help a disadvantaged population. We settled on reducing the communication barrier between the deaf and hearing communities. At first, we wanted to convert sign language sentence gestures into written words. However, this would have been too complex for a two week project. Full sentences in sign language are comprised of hand gestures, arm movements, and even facial expressions. We decided to tackle a subset of sign language called fingerspelling.

User and Domain Research

We interviewed two students who took sign language classes at the university. They were not deaf themselves, but had recently gone through the learning curve for beginners in sign language. We learnt that beginners typically start by learning fingerspelling. Each letter of the English alphabet has a corresponding hand gesture and fingerspelling means that a user spells out the letters of a word, one hand gesture at a time.

Building the Glove

Designing the Algorithm

We created the glove knowing that we could determine the intended gesture with a bottom-up approach, by determining the "bent-ness" of each finger and mapping it onto a letter. 

Introducing states

Studying the gesture alphabet showed that it was only necessary to know whether each finger is BENT, UNBENT, or SOMEWHAT BENT, rather than working with the exact degree of "bent-ness". So, several times a second, we took a snapshot of resistance values from each flex sensor, and characterized each finger in one of these three states. Once we had the state of each finger, we tested to see if the set matched any of the theoretical letter sets. 

Process of elimination

Rather than iterating through each theoretical alphabet for a match, we optimized our algorithm by eliminating certain letters right off the bat, based on key indicators. For example, certain letters required a bent wrist. We therefore introduced another flex sensor at the wrist. If this sensor were in the BENT state, then the intended letter could only be one of four letters (G, H, P, or Q), rather than one of 26 letters. 

The need for machine learning

There is a lot of variability in the characterization of finger "bent-ness". Every time a user puts on the glove, her fingers are in a slightly different position than they were the last time, and the resistance ranges for each state changes slightly. Furthermore, environmental factors such as temperature can also affect the resistance readings. To handle this variability, we developed an algorithm that would dynamically change the acceptable range for each state as new values came in. This is the defining characteristic of a k-means clustering algorithm. The centroid of each cluster (where cluster represents one of the three states of "bent-ness") was the average of the last 10 values in that cluster. When a new value was accepted into a cluster, that cluster's centroid was recalculated. In this manner, the clusters are malleable and therefore accepting of variability.

The Results

Within our two-week project period, we were able to successfully decode 24 out of the 26 alphabet letters. Our average speed of display was roughly one letter every 30 seconds. While this is slow, we are confident that further optimizations could easily be made to enhance the speed. You can watch the demonstration of our product below.