Sign language uses hand shapes and movements to share information, but a computer can't understand these signs just by looking at a picture. We picked this topic to find out how AI identifies different hand shapes and if using a more advanced model is always better than using simpler techniques.
In this project, we looked at different models to see which one works best for recognizing ASL letters. Our aim was to test a basic model, two techniques that combine multiple models, and a Convolutional Neural Network (CNN) all on the same set of data.
Dataset and preprocessing
We used the Sign Language MNIST dataset, which contains 27,455 training images and 7,172 test images. Each image is 28 × 28 pixels in grayscale. The dataset has 24 letter classes; J and Z are excluded because they involve movement.
We assigned numbers from 0 to 23 to the labels and divided each pixel value by 255 to bring them into the 0 to 1 range. We divided the original training data into 23,336 images for training and 4,119 images for validation. The test set was kept apart to be used for the final evaluation.
The CNN used the original 28 x 28 images. For the other models, we took each 4 x 4 pixel block and averaged them to get 49 features for each image.
Models and evaluation
We compared four models:
• Logistic Regression as the baseline.
• Random Forest as a bagging ensemble.
• Histogram Gradient Boosting as a boosting ensemble.
• CNN as the image recognition model developed earlier in the project.
Our main measure was macro F1, which treats each letter as equally important. We also looked at accuracy, the classification report, and the confusion matrix.
All four models were tested using three-fold stratified cross-validation on the 23,336 training images. A new CNN was created for every fold.
Hyperparameter tuning and model comparison
We adjusted the Random Forest model by trying different numbers of trees, maximum depth, and minimum samples per leaf. The best setup used 120 trees, a maximum depth of 20, and required at least one sample per leaf. After tuning, the validation macro F1 score reached 98.70%.
The CNN achieved the best validation score and was chosen as the final model. In the recorded test run, it got 97.80% accuracy and 97.80% macro F1 score using 7,172 test images.
Confusion matrix and analysis
The confusion matrix shows how many images were classified correctly and which letters were mixed up. Most CNN predictions land on the diagonal, which shows that the classifications are correct. The classification report also shows where performance can improve: recall was about 0.88 for T, 0.91 for S, and 0.92 for I.
Both ensemble methods outperformed the baseline. Random Forest worked better than Histogram Gradient Boosting, but adjusting the settings only made a small difference. The CNN had the best scores, but the way it got its input was different: it used the actual images, while the other three models used 49 averaged pixel features.
Conclusion and limitations
In this experiment, the CNN performed best during both cross-validation and validation. Random Forest was the better of the two ensemble methods. The CNN test results showed about 97.80% accuracy and 97.80% macro F1 score.
These results come from one image dataset. They haven't figured out how well the model performs with different users, lighting conditions, backgrounds, or camera angles yet. The dataset includes only static letters, so this model cannot understand full sign language communication.
Colab notebook: ASL_Ensemble
References: Sign Language MNIST·From EDA to ML Model: The Step-by-Step Guide Most Tutorials Skip
Testing Video: Testing
