Sign languages are the primary means of communication for deaf communities, but the development of effective automatic recognition systems remains a significant challenge. In this work, we focus on the task of Isolated Sign Language Recognition (ISLR) using a multimodal approach grounded in a Large Language Model (LLM) architecture. We merge modalities, including visual characteristics into the linguistic representation space of LLMs, and perform ablation studies to evaluate the individual contributions of each visual modality to the recognition performance. Experiments are conducted on the AVASAG100 dataset, where our method achieves a weighted F1-score (W-F1) of 70.36±3.00 and a macro F1-score (MF1) of 62.34±3.18 projecting landmarks extracted from the pose into the LLM’s emebdding-space. These results underscore the value of multimodal integration in ISLR and provide guidelines for future research directions.
Sign languages are the primary means of communication for deaf communities, but the development of effective automatic recognition systems remains a significant challenge. In this work, we focus on the task of Isolated Sign Language Recognition (ISLR) using a multimodal approach grounded in a Large Language Model (LLM) architecture. We merge modalities, including visual characteristics into the linguistic representation space of LLMs, and perform ablation studies to evaluate the individual contributions of each visual modality to the recognition performance. Experiments are conducted on the AVASAG100 dataset, where our method achieves a weighted F1-score (W-F1) of 70.36±3.00 and a macro F1-score (MF1) of 62.34±3.18 projecting landmarks extracted from the pose into the LLM’s emebdding-space. These results underscore the value of multimodal integration in ISLR and provide guidelines for future research directions. Read More
Related posts:
Automatic Speech Recognition Advancements for Indigenous Languages of the Americas
Synthesizing olfactory understanding: multimodal language models for image-text smell matching
Parameter-efficient adaptation of large vision—language models for video memorability prediction
Design, Development and Validation of Multi-Modal Deep Learning based Models for Memorability Prediction in Short Videos
Segmentación e identificación de gestos en imágenes y vídeos: aplicación en la traducción de la Lengua de Signos Española (LSE)
Multilingual Speech Emotion Recognition combining Audio and Text Information using Small Language Models