Does the Phonology of L1 Show Up in L2 Texts?

The task of Native Language Identification (NLI) is to determine the first language of the writer of a text in another language. In a ground-breaking paper, Koppel et al. (2005) propose a set of features for this task: function words, character $n$-grams, rare part-of-speech bigrams, and various types of errors. They report 80% accuracy in classifying a set of English texts into five L1 languages using a multi-class linear SVM.

The First Shared Task on Native Language Identification [24] attracted submissions from 29 teams. The accuracy on a set of English texts representing eleven L1 languages ranged from 31% to 83%. Many types of features were employed, including word length, sentence length, paragraph length, document length, sentence complexity, punctuation and capitalization, cognates, dependency parses, topic models, word suffixes, collocations, function word $n$-grams, skip-grams, word networks, Tree Substitution Grammars, string kernels, cohesion, and passive constructions [1, 17, 3, 5, 7, 9, 11, 12, 4, 16, 18, 19, 20, 21, 22, 23, 26]. In particular, word $n$-gram features appear to be particularly effective, as they were used by the most competitive teams, including the one that achieved the highest overall accuracy [13]. Furthermore, the most discriminative word $n$-grams often contained the name of the native language, or countries where it is commonly spoken [8, 19, 21]. We refer to such words as toponymic terms.

There is no doubt that the toponymic terms are useful for increasing the NLI accuracy; however, from the psycho-linguistic perspective, we are more interested in what characteristics of L1 show up in L2 texts. Clearly, L1 affects the L2 writing in general, and the choice of words in particular, but what is the role played by the phonology? Tsur and Rappoport (2007) observe that limiting the set of features to the relative frequency of the 200 most frequent character bigrams yields a respectable 66% accuracy on a 5-language classification task. The authors propose the following hypothesis to explain this finding: “the choice of words [emphasis added] people make when writing in a second language is strongly influenced by the phonology of their native language”. As the orthography of alphabetic languages is at least partially representative of the underlying phonology, character bigrams may capture these phonological preferences.

In this paper, we provide evidence against the above hypothesis. We design an algorithm to identify the most discriminative words and the character bigrams that are indicative of such words, and perform two experiments to quantify their impact on the NLI task. The results of the first experiment demonstrate that the removal of a relatively small set of discriminative words from the training data significantly impairs the accuracy of a bigram-based classifier. The results of the second experiment reveal that the most indicative bigrams are quite similar across different language sets. We conclude that character bigrams are effective in determining L1 of the author because they reflect differences in L2 word usage that are unrelated to the phonology of L1.

Tsur and Rappoport (2007) report that character bigrams are more effective for the NLI task than either unigrams or trigrams. We are interested in identifying the character bigrams that are indicative of the most discriminative words in order to quantify their impact on the bigram-based classifier.

We follow both Koppel et al. (2005) and Tsur and Rappoport (2007) in using a multi-class SVM classifier for the NLI task. The classifier computes a weight for each feature coupled with each L1 language by attempting to maximize the overall accuracy on the training set. For example, if we train the classifier using words as features, with values representing their frequency relative to the length of the document, the features corresponding to the word China might receive the following weights:

These weights indicate that the word provides strong positive evidence for Chinese as L1, as opposed to the other four languages.

We propose to quantify the importance of each word by converting its SVM feature weights into a single score using the following formula:

where $N$ is the number of languages, and $w_{ij}$ is the feature weight of word $i$ in language $j$. The formula assigns higher scores to words with weights of high magnitude, either positive or negative. We use the Euclidean norm rather than the sum of raw weights because we are interested in the discriminative power of the words.

We normalize the word scores by dividing them by the score of the 200th word. Consequently, only the top 200 words have scores greater than or equal to $1.0$. For our previous example, the 200${}^{th}$ word has a word score of 1493, while China has a word score of 1930, which is normalized to $1930/1493=1.29$. On the other hand, the 1000${}^{th}$ word gets a normalized score of 0.43.

[t] Computing the scores of words and bigrams in the data. {algorithmic}[1] \STATEcreate list of words in training data \STATEtrain SVM using words as features \FORALLwords i \STATE$WordScor{e}_i=\sqrt{{\sum }_{j=1}^{N}w_{ij}{}^2}$ \ENDFOR\STATEsort words by WordScore \STATENormValue = WordScore${}_{200}$ \STATEcreate list of 200 most frequent bigrams \FORbigrams k = 1 to 200 \STATE$BigramScor{e}_k={\prod }_{k\in i}\frac{WordScor{e}_i}{NormValue}$ \ENDFOR\STATEsort character bigrams by BigramScore

In order to identify the bigrams that are indicative of the most discriminative words, we promote those that appear in the high-scoring words, and downgrade those that appear in the low-scoring words. Some bigrams that appear often in the high-scoring words may be very common. For example, the bigram an occurs in words like Japan, German, and Italian, but also by itself as a determiner, as an adjectival suffix, and as part of the conjunction and. Therefore, we calculate the importance score for each character bigram by multiplying the scores of each word in which the bigram occurs.

Algorithm 2 summarizes our method of identifying the discriminative words and indicative character bigrams. In line 2, we train an SVM on the words encountered in the training data. In lines 3 and 4, we assign the Euclidean norm of the weight vector of each word as its score. Starting in line 7, we determine which character bigrams are representative of high scoring words. In line 10, we calculate the bigram scores.

In this section, we describe two experiments aimed at quantifying the importance of the discriminative words and the indicative character bigrams that are identified by Algorithm 2.

We have provided experimental evidence against the hypothesis that the phonology of L1 strongly affects the choice of words in L2. We showed that a small set of high-frequency function words have disproportionate influence on the accuracy of a bigram-based NLI classifier, and that the majority of the indicative bigrams appear to be independent of L1. This suggests an alternative explanation of the effectiveness of a bigram-based classifier in identifying the native language of a writer — that the character bigrams simply mirror differences in the word usage rather than the phonology of L1.

Our explanation concurs with the findings of Daland (2013) that unigram frequency differences in certain types of phonological segments between child-directed and adult-directed speech are due to a small number of word types, such as you, what, and want, rather than to any general phonological preferences. He argues that the relative frequency of sounds in speech is driven by the relative frequency of words. In a similar vein, Koppel et al. (2005) see the usefulness of character $n$-grams as “simply an artifact of variable usage of particular words, which in turn might be the result of different thematic preferences,” or as a reflection of the L1 orthography.

We conclude by noting that our experimental results do not imply that the phonology of L1 has absolutely no influence on L2 writing. Rather, they show that the evidence from the Native Language Identification task has so far been inconclusive in this regard.

We thank the participants and the organizers of the shared task on NLI at the BEA8 workshop for sharing their reflections on the task. We also thank an anonymous reviewer for pointing out the study of Daland (2013).

This research was supported by the Natural Sciences and Engineering Research Council of Canada and the Alberta Innovates Technology Futures.