Acquisition of multimodal data corpus for automatic sign language processing

Jakub Gałka, Przemysław Węgrzynowicz, Mariusz Mąsior


This paper presents the creation of a Polish Sign Language corpus suitable for recognition research and automatic translation of sign language. The recording approach used and the captured data modalities are presented, as well as the description of the acquisition system implementation. The evaluation of the collected corpus is presented and compared to other available resources.


sign language recognition; data analysis; data collection; image processing

Full Text:



World Federation of the Deaf: Sign Language. Available at: (accessed 26 January 2016).

Stokoe W.C.: Sign Language Structure: An Outline of the Visual Communication Sys-tem of the American Deaf. J. Deaf Stud. Deaf Educ.. Vol. 10(1), 1960, p. 3÷37.

Vogler C., Metaxas D.: A Framework for Recognizing the Simultaneous Aspects of American Sign Language. Computer Vision and Image Understanding, Vol. 81, No. 3, 2001, p. 358÷384.

Theodorakis S., Pitsikalis V., Maragos, P.: Dynamic–static unsupervised sequentiality, statistical subunits and lexicon for sign language recognition. Image and Vision Com-puting, Vol. 32, No. 8, 2014, p. 533÷549.

Von Agris U., Zieren J., Canzler U., Bauer B., Kraiss K.-F.: Recent developments in visual sign language recognition. Universal Access in the Information Society, Vol. 6, No. 4, 2008, p. 323÷362.

Kadous M.W.: Temporal Classification: Extending the Classification Paradigm to Mul-tivariate Time Series. PhD dissertation, School of Computer Science and Engineering, Univ. of New South Wales, Australia 2002.

Lichman M.: UCI Machine Learning Repository []. Univer-sity of California, School of Information and Computer Science, Irvine, CA 2013.

Escalera S. et al.: Multi-modal Gesture Recognition Challenge 2013: Dataset and Re-sults. ICMI, 2013.

Horn B.K.B., Schunck B.G.: Determining optical flow. Artificial Intelligence, Vol. 17, 1981, p. 185÷203.

Barczewska K.: The Automatic Recognition of Isolated Sign Language Signs Based on Gesture Components and DTW Algorithm. Challenges of Modern Technology, Vol. 5, No. 3, 2014, p. 1÷8.

Tsai D.M., Chiu W.Y., Lee M.H.: Optical flow-motion history image for action recog-nition. Signal, Image and Video Processing, Vol. 9, No. 8, 2014, p. 1897÷1906.

Yeo H.S., Lee B.G., Lim H.: Hand Tracking and Gesture Recognition System for Hu-man-Computer Interaction Using Low-Cost Hardware. Journal of Multimedia Tools and Applications, Vol. 74, No. 8, 2013, p. 2687÷2715.