Recent advancements in deep learning have shown significant potential in developing barrier-free environments for the hearing-impaired. In particular, progress in computer vision task has accelerated research on sign language recognition and translation for the hearing-impaired. However, the lack of diverse and specialized datasets limits the development of high-level services, such as museum exhibitions, that require interaction beyond simply recognizing and translating sign language.

To address this, we introduce KSL-Ex, a Korean Sign Language (KSL) dataset designed to support real-world interactive exhibition services. Our dataset comprises 29,574 sign language video samples, including isolated words, continous KSL sentences, question answering (QA) pairs, and detailed annotations.

We also propose an interactive sign language QA framework that leverages state-of-the-art large language models which have demonstrated outstanding performance in question answering tasks. The proposed framework consists of four key components: the sign language recognition model, the answering module, the refinement module, and the 3D sign language animation module. These components effectively process sign language queries, generate and refine appropriate answers, and finally produce sign language animation, thereby providing an interactive QA interface for the hearing-impaired.

Experimental results demonstrate the effectiveness of both KSL-Ex and the proposed framework, indicating strong potential for delivering interactive sign language services in real-world environments.