Develop Sign Language Translator with Python

Published on 14 May 2022. Written by Shirin Tikoo

Sign languages (also known as signed languages) are languages that utilize the visual-manual methodology to pass on importance. Language is communicated through the manual sign stream in the mix with non-manual components.

Read more..

The primary issue is that the hard of hearing like deaf and dumb people cannot communicate easily with normal people since persons other than disabled persons do not learn how to communicate in sign language with each other. The solution is to create a translator which can detect sign language performed by a disabled person and then that sign will be fed to a machine-learning algorithm performed by a neural network which is then detected by the neural network and translated on display so that a normal person can understand what the sign conveys.

Project Description

To build a Sign language translator we will need four things:

Data Gathering

A separate directory is created with a collection of various sub-directories labelled as A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V, W, X, Y, Z. Each sub-directory contains various images of the hand gesture of a user displaying a sign language for the corresponding label i.e. Folder A will be having 20-30 images of the sign language of A displayed and saved by the user. Similarly, Folder B will be having 20-30 images of the sign language of B displayed and saved by the user and similar steps needed to perform for other alphabets.

Creating Model

A model is created using TensorFlow which will recognize the real-time input of the hand signs of the user and will display the corresponding letter of the sign language. This TensorFlow model is a GPU model that compares the real-time image with the data that is already stored in the database with the help of the data gathering process.

Training Model

The model which is created in the previous method of the project using TensorFlow is being trained with some samples of data from the database of the images of the words from each word. This step will help the model to get a hold of how a letter is being represented using a hand gesture to perform sign language. This model is trained for some random amount of time in order to get a better and efficient accuracy while performing the actual test with the model.

Testing Model

After the model is trained it can be tested to observe the actual working of the system. The user shows a hand gesture of any word based on the sign language data which is being provided in the model and also used for the training of the model. When the user creates a hand sign for any specific letter the model recognizes the gesture and displays the corresponding letter in the bottom left side of the console.

Project Implementation:

1. Firstly we need to download a dataset. We can use the original MNIST (Modified National Institute of Standards and Technology) image dataset for our project or any other dataset that is available on the internet.
2. After gathering the dataset, we will build and train the model, construct the model, we will require CNN (Convolutional Neural Network) to perceive the letters in order and some Keras to build the CNN.
3. To build a model, we need tensorflow which will recognize the real-time input of the hand signs of the user and will display the corresponding letter of the sign language.
4. The next step is to train the data; this step will help to remember how the letter is being represented using a hand gesture to perform sign language.
5. Presently we have a prepared model, the next step is to create a window to take the input image from our webcam.
6. At last, our model is prepared for testing and the user can create a hand sign for any specific letter the model recognizes the gesture and displays the corresponding letter in the bottom left side of the console.

Software Requirements:

Programming language - OpenCV, TensorFlow, Keras, Python

Kit required to develop Develop Sign Language Translator with Python:

• No kit

Technologies you will learn by working on Develop Sign Language Translator with Python:

• Python
• Machine Learning
• Computer Vision

Any Questions?