Unsupervised Feature Learning for Visual Sign Language Identification

The relationship between forms and meanings are not totally arbitrary [17]. Both signed and spoken languages manifest iconicity, that is forms of words or signs are somehow motivated by the meaning of the word or sign. While sign languages show a lot of iconicity in the lexicon [22], this has not led to a universal sign language. The same concept can be iconically realised by the manual articulators in a way that conforms to the phonological regularities of the languages, but still lead to different sign forms.

Iconicity is also used in the morphosyntax and discourse structure of all sign languages, however, and there we see many similarities between sign languages. Both real-world and imaginary objects and locations are visualised in the space in front of the signer, and can have an impact on the articulation of signs in various ways. Also, the use of constructed action appears to be used in many sign languages in similar ways. The same holds for the rich use of non-manual articulators in sentences and the limited role of facial expressions in the lexicon: these too make sign languages across the world very similar in appearance, even though the meaning of specific articulations may differ [7].

Just as speakers have different voices unique to each individual, signers have also different signing styles that are likely unique to each individual. Signers’ uniqueness results from how they articulate the shapes and movements that are specified by the linguistic structure of the language. The variability between signers either in terms of physical properties (hand sizes, colors, etc) or in terms of articulation (movements) is such that it does not affect the understanding of the sign language by humans, but that it may be difficult for machines to generalize over multiple individuals. At present we do not know whether the differences between signers using the same language are of a similar or different nature than the differences between different languages. At the level of phonology, there are few differences between sign languages, but the differences in the phonetic realization of words (their articulation) may be much larger.

The visual ’activity’ of signing comes in a context of a specific environment. This environment can include the visual background and camera noises. The background objects of the video may also include dynamic objects – increasing the ambiguity of signing activity. The properties and configurations of the camera induce variations of scale, translation, rotation, view, occlusion, etc. These variations coupled with lighting conditions may introduce noise. These challenges are by no means specific to sign interaction, and are found in many other computer vision tasks.

Given samples of sign language videos (unknown sign language with one signer per video), our system performs the following steps to learn a feature representation (note that these video samples are separate from the video samples that are later used for classifier learning or testing):

1. 1.

   Extract patches. Extract small videos (hereafter called patches) randomly from anywhere in the video samples. We fix the size of the patches such that they all have $r$ rows, $c$ columns and $f$ frames and we extract patches $m$ times. This gives us $𝑿=\left\{x^{\left(1\right)},x^{\left(1\right)},\mathrm{\dots },x^{\left(m\right)}\right\}$, where $x^{\left(i\right)}\in R^N$ and $N=r*c*f$ (the size of a patch). For our experiments, we extract 100,000 patches of size $15*15*1$ (2D) and $15*15*2$ (3D).

2. 2.

   Normalize the patches. There is evidence that normalization and whitening [13] improve performance in unsupervised feature learning [4]. We therefore normalize every patch $x^{\left(i\right)}$ by subtracting the mean and dividing by the standard deviation of its elements. For visual data, normalization corresponds to local brightness and contrast normalization.

3. 3.

   Learn a feature-mapping. Our unsupervised algorithm takes in the normalized and whitened dataset $𝑿=\left\{x^{\left(1\right)},x^{\left(1\right)},\mathrm{\dots },x^{\left(m\right)}\right\}$ and maps each input vector $x^{\left(i\right)}$ to a new feature vector of $K$ features ($f:R^N\to R^K$). We use two unsupervised learning algorithms a) K-means b) sparse autoencoders.

   1. (a)

      K-means clustering: we train K-means to learns $K$ $c^{\left(k\right)}$ centroids that minimize the distance between data points and their nearest centroids [5]. Given the learned centroids $c^{\left(k\right)}$, we measure the distance of each data point (patch) to the centroids. Naturally, the data points are at different distances to each centroid, we keep the distances that are below the average of the distances and we set the other to zero:

      $$f_k\left(x\right)=max\left\{0,\mu \left(z\right)-z_k\right\}$$

      (1)

      where $z_k={||x-c^{\left(k\right)}||}^2$ and $\mu \left(z\right)$ is the mean of the elements of $z$.

   2. (b)

      Sparse autoencoder: we train a single layer autoencoder with $K$ hidden nodes using backpropagation to minimize squared reconstruction error. At the hidden layer, the features are mapped using a rectified linear (ReL) function [15] as follows:

      $$f\left(x\right)=g\left(Wx+b\right)$$

      (2)

      where $g\left(z\right)=max\left(z,0\right)$. Note that ReL nodes have advantages over sigmoid or tanh functions; they create sparse representations and are suitable for naturally sparse data [11].

From K-means, we get $K$ $R^N$ centroids and from the sparse autoencoder, we get $W\in R^{KxN}$ and $b\in R^K$ filters. We call both the centroids and filters as the learned features.

Given the learned features, the feature mapping functions and a set of labeled training videos, we extract features as follows:

1. 1.

   Convolutional extraction: Extract features from equally spaced sub-patches covering the video sample.

2. 2.

   Pooling: Pool features together over four non-overlapping regions of the input video to reduce the number of features. We perform max pooling for K-means and mean pooling for the sparse autoencoder over 2D regions (per frame) and over 3D regions (per all sequence of frames).

3. 3.

   Learning: Learn a linear classifier to predict the labels given the feature vectors. We use logistic regression classifier and support vector machines [16].

The extraction of classifier features through convolution and pooling is illustrated in figure 1.

Our experimental data consist of videos of 30 signers equally divided between six sign languages: British sign language (BSL), Danish (DSL), French Belgian (FBSL), Flemish (FSL), Greek (GSL), and Dutch (NGT). The data for the unsupervised feature learning comes from half of the BSL and GSL videos in the Dicta-Sign corpus²²http://www.dictasign.eu/. Part of the other half, involving 5 signers, is used along with the other sign language videos for learning and testing classifiers.

For the unsupervised feature learning, two types of patches are created: 2D dimensions ($15*15$) and 3D ($15*15*2$). Each type consists of randomly selected 100,000 patches and involves 16 different signers. For the supervised learning, 200 videos (consisting of 1 through 4 frames taken at a step of 2) are randomly sampled per sign language per signer (for a total of 6,000 samples).

The data preprocessing stage has two goals.

First, to remove any non-signing signals that remain constant within videos of a single sign language but that are different across sign languages. For example, if the background of the videos is different across sign languages, then classifying the sign languages could be done with perfection by using signals from the background. To avoid this problem, we removed the background by using background subtraction techniques and manually selected thresholds.

The second reason for data preprocessing is to make the input size smaller and uniform. The videos are colored and their resolutions vary from $320*180$ to $720*576$. We converted the videos to grayscale and resized their heights to $144$ and cropped out the central $144*144$ patches.

We evaluate our system in terms of average accuracies. We train and test our system in leave-one-signer-out cross-validation, where videos from four signers are used for training and videos of the remaining signer are used for testing. Classification algorithms are used with their default settings and the classification strategy is one-vs.-rest.