Weak semantic context helps phonetic learning
in a model of infant language acquisition

Both the LD and TLD models are computational-level models of phonetic (specifically, vowel) categorization where phones (vowels) are presented to the model in the context of words.²²For a related model that also tackles the word segmentation problem, see Elsner et al. (10). In a model of phonological learning, Fourtassi and Dupoux (submitted) show that semantic context information similar to that used here remains useful despite segmentation errors. The task is to infer a set of phonetic categories and a set of lexical items on the basis of the data observed for each word token $x_i$. In the original LD model, the observations for token $x_i$ are its frame $f_i$, which consists of a list of consonants and slots for vowels, and the list of vowel tokens ${𝒘}_i$. (The TLD model includes additional observations, described below.) A single vowel token, $w_{ij}$, is a two dimensional vector representing the first two formants (peaks in the frequency spectrum, ordered from lowest to highest). For example, a token of the word kitty would have the frame $f_i=\mathtt{\text{k\_t\_}}$, containing two consonant phones, /k/ and /t/, with two vowel phone slots in between, and two vowel formant vectors, $w_{i0}=\left[464,2294\right]$ and $w_{i1}=\left[412,2760\right]$.³³In simulations we also experiment with frames in which consonants are not represented perfectly.

Given the data, the model must assign each vowel token to a vowel category, $w_{ij}=c$. Both the LD and the TLD models do this using intermediate lexemes, $\mathbf{\ell }$, which contain vowel category assignments, $v_{\mathrm{\ell }j}=c$, as well as a frame $f_{\mathrm{\ell }}$. If a word token is assigned to a lexeme, $x_i=\mathrm{\ell }$, the vowels within the word are assigned to that lexeme’s vowel categories, $w_{ij}=v_{\mathrm{\ell }j}=c$.⁴⁴The notation is overloaded: $w_{ij}$ refers both to the vowel formants and the vowel category assignments, and $x_i$ refers to both the token identity and its assignment to a lexeme. The word and lexeme frames must match, $f_i=f_{\mathrm{\ell }}$.

Lexical information helps with phonetic categorization because it can disambiguate highly overlapping categories, such as the ae and eh categories in Figure 1. A purely distributional learner who observes a cluster of data points in the ae-eh region is likely to assume all these points belong to a single category because the distributions of the categories are so similar. However, a learner who attends to lexical context will notice a difference: contexts that only occur with ae will be observed in one part of the ae-eh region, while contexts that only occur with eh will be observed in a different (though partially overlapping) space. The learner then has evidence of two different categories occurring in different sets of lexemes.

Simulations with the LD model show that using lexical information to constrain phonetic learning can greatly improve categorization accuracy (11), but it can also introduce errors. When two word tokens contain the same consonant frame but different vowels (i.e., minimal pairs), the model is more likely to categorize those two vowels together. Thus, the model has trouble distinguishing minimal pairs. Although young children also have trouble with minimal pairs (41; 45), the LD model may overestimate the degree of the problem. We hypothesize that if a learner is able to associate words with the contexts of their use (as children likely are), this could provide a weak source of information for disambiguating minimal pairs even without knowing their exact meanings. That is, if the learner hears k$V_{\mathrm{1}}$t and k$V_{\mathrm{2}}$t in different situational contexts, they are likely to be different lexical items (and $V_1$ and $V_2$ different phones), despite the lexical similarity between them.

To demonstrate the benefit of situational information, we develop the Topic-Lexical-Distributional (TLD) model, which extends the LD model by assuming that words appear in situations analogous to documents in a topic model. Each situation $h$ is associated with a mixture of topics ${\theta }_h$, which is assumed to be observed. Thus, for the $i$th token in situation $h$, denoted $x_{hi}$, the observed data will be its frame $f_{hi}$, vowels ${𝒘}_{hi}$, and topic vector ${\theta }_h$.

From an acquisition perspective, the observed topic distribution represents the child’s knowledge of the context of the interaction: she can distinguish bathtime from dinnertime, and is able to recognize that some topics appear in certain contexts (e.g. animals on walks, vegetables at dinnertime) and not in others (few vegetables appear at bathtime). We assume that the child would learn these topics from observing the world around her and the co-occurrences of entities and activities in the world. Within any given situation, there might be a mixture of different (actual or possible) topics that are salient to the child. We assume further that as the child learns the language, she will begin to associate specific words with each topic as well.

Thus, in the TLD model, the words used in a situation are topic-dependent, implying meaning, but without pinpointing specific referents. Although the model observes the distribution of topics in each situation (corresponding to the child observing her non-linguistic environment), it must learn to associate each (phonetically and lexically ambiguous) word token with a particular topic from that distribution. The occurrence of similar-sounding words in different situations with mostly non-overlapping topics will provide evidence that those words belong to different topics and that they are therefore different lexemes. Conversely, potential minimal pairs that occur in situations with similar topic distributions are more likely to belong to the same topic and thus the same lexeme.

Although we assume that children infer topic distributions from the non-linguistic environment, we will use transcripts from childes to create the word/phone learning input for our model. These transcripts are not annotated with environmental context, but Roy et al. (37) found that topics learned from similar transcript data using a topic model were strongly correlated with immediate activities and contexts. We therefore obtain the topic distributions used as input to the TLD model by training an LDA topic model (5) on a superset of the child-directed transcript data we use for lexical-phonetic learning, dividing the transcripts into small sections (the ‘documents’ in LDA) that serve as our distinct situations $𝒉$. As noted above, the learned document-topic distributions $𝜽$ are treated as observed variables in the TLD model to represent the situational context. The topic-word distributions learned by LDA are discarded, since these are based on the (correct and unambiguous) words in the transcript, whereas the TLD model is presented with phonetically ambiguous versions of these word tokens and must learn to disambiguate them and associate them with topics.

Following previous models of vowel learning (8; 50; 26; 9) we assume that vowel tokens are drawn from a Gaussian mixture model. The Infinite Gaussian Mixture Model (IGMM) (35) includes a DP prior, as described above, in which the base distribution $H_C$ generates multivariate Gaussians drawn from a Normal Inverse-Wishart prior.⁵⁵This compound distribution is equivalent to
${\mathrm{\Sigma }}_c\sim \text{𝐼𝑊}({\mathrm{\Sigma }}_0,{\nu }_0),{\mu }_c|{\mathrm{\Sigma }}_c\sim N({\mu }_0,\frac{{\mathrm{\Sigma }}_c}{{\nu }_0})$ Each observation, a formant vector $w_{ij}$, is drawn from the Gaussian corresponding to its category assignment $c_{ij}$:

The above model generates a category assignment $c_{ij}$ for each vowel token $w_{ij}$. This is the baseline IGMM model, which clusters vowel tokens using bottom-up distributional information only; the LD model adds top-down information by assigning categories in the lexicon, rather than on the token level.

In the LD model, vowel phones appear within words drawn from the lexicon. Each such lexeme is represented as a frame plus a list of vowel categories ${𝒗}_{\mathrm{\ell }}$. Lexeme assignments for each token are drawn from a DP with a lexicon-generating base distribution $H_L$. The category for each vowel token in the word is determined by the lexeme; the formant values are drawn from the corresponding Gaussian as in the IGMM:

$H_L$ generates lexemes by first drawing the number of phones from a geometric distribution and the number of consonant phones from a binomial distribution. The consonants are then generated from a DP with a uniform base distribution (but note they are fixed at inference time, i.e., are observed categorically), while the vowel phones ${𝒗}_{\mathrm{\ell }}$ are generated by the IGMM DP above, $v_{\mathrm{\ell }j}\sim G_C$.

Note that two draws from $H_L$ may result in identical lexemes; these are nonetheless considered to be separate (homophone) lexemes.

Each vowel in the lexicon must be assigned to a category in the IGMM. The posterior probability of a category assignment is composed of the DP prior over categories and the likelihood of the observed vowels belonging to that category. We use ${𝒘}_{\mathrm{\ell }j}$ to denote the set of vowel formants at position $j$ in words that have been assigned to lexeme $\mathrm{\ell }$. Then,

The first (DP prior) factor is defined as:

where $n_c$ is the number of other vowels in the lexicon, ${𝒗}^{\setminus lj}$, assigned to category $c$. Note that there is always positive probability of creating a new category.

The likelihood of the vowels is calculated by marginalizing over all possible means and variances of the Gaussian category parameters, given the NIW prior. For a single point (if $\left|{𝒘}_{\mathrm{\ell }j}\right|=1$), this predictive posterior is in the form of a Student-$t$ distribution; for the more general case see Feldman et al. (11), Eq. B3.

We jointly sample $𝒙$ and $𝒛$, the variables assigning tokens to tables and topics. Resampling the table assignment includes the possibility of changing to a table with a different lexeme or drawing a new table with a previously seen or novel lexeme. The joint conditional probability of a table and topic assignment, given all other current token assignments, is:

The first factor, the prior probability of topic $k$ in document $h$, is given by ${\theta }_{hk}$ obtained from the LDA. The second factor is the prior probability of assigning word $x_i$ to table $t$ with lexeme $\mathrm{\ell }$ given topic $k$. It is given by the HDP, and depends on whether the table $t$ exists in the HDP topic-lexicon for $k$ and, likewise, whether any table in the topic-lexicon has the lexeme $\mathrm{\ell }$:

Here $n_{kt}$ is the number of other tokens at table $t$, $n_k$ are the total number of tokens in topic $k$, $m_{\mathrm{\ell }}$ is the number of tables across all topics with the lexeme $\mathrm{\ell }$, and $m$ is the total number of tables.

The third factor, the likelihood of the vowel formants ${𝒘}_{hi}$ in the categories given by the lexeme ${𝒗}_l$, is of the same form as the likelihood of vowel categories when resampling lexeme vowel assignments. However, here it is calculated over the set of vowels in the token assigned to each vowel category (i.e., the vowels at indices where $v_{{\mathrm{\ell }}_t\cdot }=c$). For a new lexeme, we approximate the likelihood using 100 samples drawn from the prior, each weighted by $\alpha /100$ (28).

The three hyperparameters governing the HDP over the lexicon, ${\alpha }_l$ and ${\alpha }_k$, and the DP over vowel categories, ${\alpha }_c$, are estimated using a slice sampler. The remaining hyperparameters for the vowel category and lexeme priors are set to the same values used by Feldman et al. (11).

We test our model on situated child directed speech, taken from the C1 section of the Brent corpus in childes (6; 20). This corpus consists of transcripts of speech directed at infants between the ages of 9 and 15 months, captured in a naturalistic setting as parent and child went about their day. This ensures variability of situations.

Utterances with unintelligible words or quotes are removed. We restrict the corpus to content words by retaining only words tagged as adj, n, part and v (adjectives, nouns, particles, and verbs). This is in line with evidence that infants distinguish content and function words on the basis of acoustic signals (38). Vowel categorization improves when attending only to more prosodically and phonologically salient tokens (1), which generally appear within content, not function words. The final corpus consists of 13138 tokens and 1497 word types.

The transcripts do not include phonetic information, so, following Feldman et al. (11), we synthesize the formant values using data from Hillenbrand et al. (17). This dataset consists of a set of 1669 manually gathered formant values from 139 American English speakers (men, women and children) for 12 vowels. For each vowel category, we construct a Gaussian from the mean and covariance of the datapoints belonging to that category, using the first and second formant values measured at steady state. We also construct a second dataset using only datapoints from adult female speakers.

Each word in the dataset is converted to a phonemic representation using the CMU pronunciation dictionary, which returns a sequence of Arpabet phoneme symbols. If there are multiple possible pronunciations, the first one is used. Each vowel phoneme in the word is then replaced by formant values drawn from the corresponding Hillenbrand Gaussian for that vowel.

The Arpabet encoding used in the phonemic representation includes 24 consonants. We construct datasets both using the full set of consonants—the ‘C24’ dataset—and with less fine-grained consonant categories. Distinguishing all consonant categories assumes perfect learning of consonants prior to vowel categorization and is thus somewhat unrealistic (29), but provides an upper limit on the information that word-contexts can give.

In the ‘C15’ dataset, the voicing distinction is collapsed, leaving 15 consonant categories. The collapsed categories are B/P, G/K, D/T, CH/JH, V/F, TH/DH, S/Z, SH/ZH, R/L while HH, M, NG, N, W, Y remain separate phonemes. This dataset mirrors the finding in Mani and Plunkett (22) that 12 month old infants are not sensitive to voicing mispronunciations.

The ‘C6’ dataset distinguishes between only 6 coarse consonant phonemes, corresponding to stops (B,P,G,K,D,T), affricates (CH,JH), fricatives (V, F, TH, DH, S, Z, SH, ZH, HH), nasals (M, NG, N), liquids (R, L), and semivowels/glides (W, Y). This dataset makes minimal assumptions about the category categories that infants could use in this learning setting.

Decreasing the number of consonants increases the ambiguity in the corpus: bat not only shares a frame (b_t) with boat and bite, but also, in the C15 dataset, with put, pad and bad (b/p_d/t), and in the C6 dataset, with dog and kite, among many others (STOP_STOP). Table 1 shows the percentage of types and tokens that are ambiguous in each dataset, that is, words in frames that match multiple wordtypes. Note that we always evaluate against the gold word identities, even when these are not distinguished in the model’s input. These datasets are intended to evaluate the degree of reliance on consonant information in the LD and TLD models, and to what extent the topics in the TLD model can replace this information.

The input to the TLD model includes a distribution over topics for each situation, which we infer in advance from the full Brent corpus (not only the C1 subset) using LDA. Each transcript in the Brent corpus captures about 75 minutes of parent-child interaction, and thus multiple situations will be included in each file. The transcripts do not delimit situations, so we do this somewhat arbitrarily by splitting each transcript after 50 CDS utterances, resulting in 203 situations for the Brent C1 dataset. As well as function words, we also remove the five most frequent content words (be, go, get, want, come). On average, situations are only 59 words long, reflecting the relative lack of content words in CDS utterances.

We infer 50 topics for this set of situations using the mallet toolkit (25). Hyperparameters are inferred, which leads to a dominant topic that includes mainly light verbs (have, let, see, do). The other topics are less frequent but capture stronger semantic meaning (e.g. yummy, peach, cookie, daddy, bib in one topic, shoe, let, put, hat, pants in another). The word-topic assignments are used to calculate unsmoothed situation-topic distributions $𝜽$ used by the TLD model.

We evaluate against adult categories, i.e., the ‘gold-standard’, since all learners of a language eventually converge on similar categories. (Since our model is not a model of the learning process, we do not compare the infant learning process to the learning algorithm.) We evaluate both the inferred phonetic categories and words using the clustering evaluation measure V-Measure (VM; 36).⁶⁶Other clustering measures, such as 1-1 matching and pairwise precision and recall (accuracy and completeness) showed the same trends, but VM has been demonstrated to be the most stable measure when comparing solutions with varying numbers of clusters (7). VM is the harmonic mean of two components, similar to F-score, where the components (VC and VH) are measures of cross entropy between the gold and model categorization. For vowels, VM measures how well the inferred phonetic categorizations match the gold categories; for lexemes, it measures whether tokens have been assigned to the same lexemes both by the model and the gold standard. Words are evaluated against gold orthography, so homophones, e.g. hole and whole, are distinct gold words.

We compare all three models—TLD, LD, and IGMM—on the vowel categorization task, and TLD and LD on the lexical categorization task (since IGMM does not infer a lexicon). The datasets correspond to two sets of conditions: firstly, either using vowel categories synthesized from all speakers or only adult female speakers, and secondly, varying the coarseness of the observed consonant categories. Each condition (model, vowel speakers, consonant set) is run five times, using 1500 iterations of Gibbs sampling with hyperparameter sampling. Overall, we find that TLD outperforms the other models in both tasks, across all conditions.

Vowel categorization results are shown in Figure 3. IGMM performs substantially worse than both TLD and LD, with scores more than 30 points lower than the best results for these models, clearly showing the value of the protolexicon and replicating the results found by Feldman et al. (11) on this dataset. Furthermore, TLD consistently outperforms the LD model, finding better phonetic categories, both for vowels generated from the combined categories of all speakers (‘all’) and vowels generated from adult female speakers only (‘w’), although the latter are clearly much easier for both models to learn. Both models perform less well when the consonant frames provide less information, but the TLD model performance degrades less than the LD performance.

Both the TLD and the LD models find ‘supervowel’ categories, which cover multiple vowel categories and are used to merge minimal pairs into a single lexical item. Figure 4 shows example vowel categories inferred by the TLD model, including two supervowels. The TLD supervowels are used much less frequently than the supervowels found by the LD model, containing, on average, only two-thirds as many tokens.

Figure 5 shows that TLD also outperforms LD on the lexeme/word categorization task. Again performance decreases as the consonant categories become coarser, but the additional semantic information in the TLD model compensates for the lack of consonant information. In the individual components of VM, TLD and LD have similar VC (“recall”), but TLD has higher VH (“precision”), demonstrating that the semantic information given by the topics can separate potentially ambiguous words, as hypothesized.

Overall, the contextual semantic information added in the TLD model leads to both better phonetic categorization and to a better protolexicon, especially when the input is noisier, using degraded consonants. Since infants are not likely to have perfect knowledge of phonetic categories at this stage, semantic information is a potentially rich source of information that could be drawn upon to offset noise from other domains. The form of the semantic information added in the TLD model is itself quite weak, so the improvements shown here are in line with what infant learners could achieve.