Can You Repeat That?
Using Word Repetition to Improve Spoken Term Detection

The spoken term detection task arises as a key subtask in applying NLP applications to spoken content. Tasks like topic identification and named-entity detection require transforming a continuous acoustic signal into a stream of discrete tokens which can then be handled by NLP and other statistical machine learning techniques. Given a small vocabulary of interest (1000-2000 words or multi-word terms) the aim of the term detection task is to enumerate occurrences of the keywords within a target corpus. Spoken term detection converts the raw acoustics into time-marked keyword occurrences, which may subsequently be fed (e.g. as a bag-of-terms) to standard NLP algorithms.

Although spoken term detection does not require the use of word-based automatic speech recognition (ASR), it is closely related. If we had perfectly accurate ASR in the language of the corpus, term detection is reduced to an exact string matching task. The word error rate (WER) and term detection performance are clearly correlated. Given resource constraints, domain, channel, and vocabulary limitations, particularly for languages other than English, the errorful token stream makes term detection a non-trivial task.

In order to improve detection performance, and restricting ourselves to an existing ASR system or systems at our disposal, we focus on leveraging broad document context around detection hypotheses. ASR systems traditionally use N-gram language models to incorporate prior knowledge of word occurrence patterns into prediction of the next word in the token stream. N-gram models cannot, however, capture complex linguistic or topical phenomena that occur outside the typical 3-5 word scope of the model. Yet, though many language models more sophisticated than N-grams have been proposed, N-grams are empirically hard to beat in terms of WER.

We consider term detection rather than the transcription task in considering how to exploit topic context, because in evaluating the retrieval of certain key terms we need not focus on improving the entire word sequence. Confidence scores from an ASR system (which incorporate N-gram probabilities) are optimized in order to produce the most likely sequence of words rather than the accuracy of individual word detections. Looking at broader document context within a more limited task might allow us to escape the limits of N-gram performance. We will show that by focusing on contextual information in the form of word repetition within documents, we obtain consistent improvement across five languages in the so called Base Phase of the IARPA BABEL program.

We seek a workable definition of broad document context beyond N-gram models that will improve term detection performance on an arbitrary set of queries. Given the rise of unsupervised latent topic modeling with Latent Dirchlet Allocation [] and similar latent variable approaches for discovering meaningful word co-occurrence patterns in large text corpora, we ought to be able to leverage these topic contexts instead of merely N-grams. Indeed there is work in the literature that shows that various topic models, latent or otherwise, can be useful for improving language model perplexity and word error rate (Khudanpur and Wu, 1999; Chen, 2009; Naptali et al., 2012). However, given the preponderance of highly frequent non-content words in the computation of a corpus’ WER, it’s not clear that a 1-2% improvement in WER would translate into an improvement in term detection.

Still, intuition suggests that knowing the topic context of a detected word ought to be useful in predicting whether or not a term does belong in that context. For example, if we determine the context of the detection hypothesis is about computers, containing words like ‘monitor,’ ‘internet’ and ‘mouse,’ then we would be more confident of a term such as ‘keyboard’ and less confident of a term such as ‘cheese board’. The difficulty in this approach arises from the variability in word co-occurrence statistics. Using topic information will be helpful if ‘monitor,’ ‘keyboard’ and ‘mouse’ consistently predict that ‘keyboard’ is present. Unfortunately, estimates of co-occurrence from small corpora are not very consistent, and often over- or underestimate concurrence probabilities needed for term detection.

We illustrate this variability by looking at how consistent word co-occurrences are between two separate corpora in the same language: i.e., if we observe words that frequently co-occur with a keyword in the training corpus, do they also co-occur with the keywords in a second held-out corpus? Figure 1, based on the BABEL Tagalog corpus, suggests this is true only for high frequency keywords.

Each point in Figure 1 represents one of 355 Tagalog keywords used for system development by all BABEL participants. For each keyword $k$, we count how often it co-occurs in the same conversation as a vocabulary word $w$ in the ASR training data and the development data, and designate the counts $T\left(k,w\right)$ and $D\left(k,w\right)$ respectively. The $x$-coordinate of each point in Figure 1 is the frequency of $k$ in the training data, and the $y$-coordinate is the correlation coefficient ${\rho }_k$ between $T\left(k,w\right)$ and $D\left(k,w\right)$. A high ${\rho }_k$ implies that words $w$ that co-occur frequently with $k$ in the training data also do so in the search collection.

To further illustrate how Figure 1 was obtained, consider the high-frequency keyword bukas (count $=\mathrm{𝟖𝟕𝟗}$) and the low-frequency keyword Davao (count $=\mathrm{𝟏𝟏}$), and plot $T\left(k,\cdot \right)$ versus $D\left(k,\cdot \right)$, as done in Figure 4. The correlation coefficients ${\rho }_{\text{𝑏𝑢𝑘𝑎𝑠}}$ and ${\rho }_{\text{𝐷𝑎𝑣𝑎𝑜}}$ from the two plots end up as two points in Figure 1.

Figure 1 suggests that $\left(k,w\right)$ co-occurrences are consistent between the two corpora (${\rho }_k>0.8$) for keywords occurring 100 or more times. However, if the goal is to help a speech retrieval system detect content-rich (and presumably infrequent) keywords, then using word co-occurrence information (i.e. topic context) does not appear to be too promising, even though intuition suggests that such information ought to be helpful.

In light of this finding, we will restrict the type of context we use for term detection to the co-occurrence of the term itself elsewhere within the document. As it turns out this ‘burstiness’ of words within documents, as the term is defined by Church and Gale in their work on Poisson mixtures (1995), provides a more reliable framework for successfully exploiting document context.

To this point we have assumed an implicit property of low-frequency words which Church and Gale state concisely in their 1999 study of inverse document frequency:

Low frequency words tend to be rich in content, and vice versa. But not all equally frequent words are equally meaningful. .

The typical use of Document Frequency ($\mathrm{DF}$) in information retrieval or text categorization is to emphasize words that occur in only a few documents and are thus more “rich in content”. Close examination of $\mathrm{DF}$ statistics by Church and Gale in their work on Poisson Mixtures (1995) resulted in an analysis of the burstiness of content words.

In this section we look at $\mathrm{DF}$ and burstiness statistics applying some of the analyses of to the BABEL Tagalog corpus. We observe, in 648 Tagalog conversations, similar phenomena as observed by Church and Gale on 89,000 AP English newswire articles. We proceed in this fashion to make a case for why burstiness ought to help in the term detection task.

For the Tagalog conversations, as with English newswire, we observe that the document frequency, ${\mathrm{DF}}_w$, of a word $w$ is not a linear function of word frequency $f_w$ in the log domain, as would be expected under a naive Poisson generative assumption. The implication of deviations from a Poisson model is that words tend to be concentrated in a small number of documents rather than occurring uniformly across the corpus. This is the burstiness we leverage to improve term detection.

The first illustration of word burstiness can be seen by plotting observed inverse document frequency, ${\mathrm{IDF}}_w$, versus $f_w$ in the log domain (Figure 7). We use the same definition of ${\mathrm{IDF}}_w$ as :

where $N$ is the number of documents (i.e. conversations) in the corpus.

There is good linear correlation ($\rho =0.73$) between $\log f_w$ and ${\mathrm{IDF}}_w$. Yet, visually, the relationship in Figure 7 is clearly not linear. In contrast, the AP English data exhibits a correlation of $\rho =0.93$ []. Thus the deviation in the Tagalog corpus is more pronounced, i.e. words are less uniformly distributed across documents.

A second perspective on word burstiness that follows from is that a Poisson assumption should lead us to predict:

For the AP newswire, Church and Gale found the largest deviation between the predicted $\widehat{{\mathrm{IDF}}_w}$ and observed ${\mathrm{IDF}}_w$ to occur in the middle of the frequency range. We see a somewhat different picture for Tagalog speech in Figure 7. Observed ${\mathrm{IDF}}_w$ values again deviate significantly from their predictions (2), but all along the frequency range.

There is a noticeable quantization effect occurring in the high $\mathrm{IDF}$ range, given that our $N$ is at least a factor of 100 smaller than the number of AP articles they studied: 648 vs. 89,000. Figure 8 also shows the difference between and observed ${\mathrm{IDF}}_w$ and Poisson estimate ${\widehat{\mathrm{IDF}}}_w$ and further illustrates the high variance in ${\mathrm{IDF}}_w$ for low frequency words.

Two questions arise: what is happening with infrequent words, and why does this matter for term detection? To look at the data from a different perspective, we consider the random variable $k$, which is the number of times a word occurs in a particular document. In Figure 9 we plot the following ratio, which define as burstiness :

as a function of $f_w$. We denote this as $E\left[k\right]$ and can interpret burstiness as the expected word count given we see $w$ at least once.

In Figure 9 we see two classes of words emerge. A similar phenomenon is observed concerning adaptive language models []. In general, we can think of using word repetitions to re-score term detection as applying a limited form of adaptive or cache language model []. Likewise, Katz attempts to capture these two classes in his G model of word frequencies (1996).

For the first class, burstiness increases slowly but steadily as $w$ occurs more frequently. Let us label these Class A words. Since our corpus size is fixed, we might expect this to occur, as more word occurrences must be pigeon-holed into the same number of documents

Looking close to the $y$-axis in Figure 9, we observe a second class of exclusively low frequency words whose burstiness ranges from highly concentrated to singletons. We will refer to these as Class B words. If we take the Class A concentration trend as typical, we can argue that most Class B words exhibit a larger than average concentration. In either case we see evidence that both high and low frequency words tend towards repeating within a document.

We summarize our re-scoring of repeated words with the observation: given a correct detection, the likelihood of additional terms in the same documents should increase. When we observe a term detection score with high confidence, we boost the other lower-scoring terms in the same document to reflect this increased likelihood of repeated terms.

For each term $t$ and document $d$ we propose interpolating the ASR confidence score for a particular detection $t_d$ with the top scoring hit in $d$ which we’ll call ${\widehat{t}}_d$.

We will we develop a principled approach to selecting $\alpha$ using the adaptation property of the corpus. However to verify that this approach is worth pursuing, we sweep a range of small $\alpha$ values, on the assumption that we still do want to mostly rely on the ASR confidence score for term detection. For the Tagalog data, we let $\alpha$ range from 0 (the baseline) to 0.4 and re-score each term detection score according to (6). Table 1 shows the results of this parameter sweep and yields us 1 to 2% absolute performance gains in a number of term detection metrics.

The primary metric for the BABEL program, Actual Term Weighted Value (ATWV) is defined by NIST using a cost function of the false alarm probability $P\left(\mathrm{FA}\right)$ and $P\left(\mathrm{Miss}\right)$, averaged over a set of queries []. The manner in which the components of ATWV are defined:

implies that cost of a miss is inversely proportional to the frequency of the term in the corpus, but the cost of a false alarm is fixed. For this reason, we report both ATWV and the $P\left(\mathrm{Miss}\right)$ component. A decrease in $P\left(\mathrm{Miss}\right)$ reflects the fact that we are able to boost correct detections of the repeated terms.

Now that we have tested word repetition-based re-scoring on a small Tagalog development set we want to know if our approach, and particularly our $\widehat{\alpha }$ estimate is sufficiently robust to apply broadly. At our disposal, we have the five BABEL languages — Tagalog, Cantonese, Pashto, Turkish and Vietnamese — as well as the development data from the NIST 2006 English evaluation. The BABEL evaluation query sets contain roughly 2000 terms each and the 2006 English query set contains roughly 1000 terms.

The procedure we follow for each language condition is as follows. We first estimate adaptation probabilities from the ASR training transcripts. From these we take the weighted average as described previously to obtain a single interpolation weight $\widehat{\alpha }$ for each training condition. We train ASR acoustic and language models from the training corpus using the Kaldi speech recognition toolkit [] following the default BABEL training and search recipe which is described in detail by . Lastly, we re-score the search output by interpolating the top term detection score for a document with subsequent hits according to Equation 6 using the $\widehat{\alpha }$ estimated for this training condition.

For each of the BABEL languages we consider both the FullLP (80 hours) and LimitedLP (10 hours) training conditions. For the English system, we also train a Kaldi system on the 240 hours of the Switchboard conversational English corpus. Although Kaldi can produce multiple types of acoustic models, for simplicity we report results using discriminatively trained Subspace Gaussian Mixture Model (SGMM) acoustic output densities, but we do find that similar results can be obtained with other acoustic model configurations.

Using our final algorithm, we are able to boost repeated term detections and improve results in all languages and training conditions. Table 3 lists complete results and the associated estimates for $\widehat{\alpha }$. For the BABEL languages, we observe improvements in ATWV from 0.7% to 1.3% absolute and reductions in the miss rate of 0.8% to 1.9%. The only test for which $P\left(\mathrm{Miss}\right)$ did not improve was the Vietnamese Limited LP setting, although overall ATWV did improve, reflecting a lower $P\left(\mathrm{FA}\right)$.

In all conditions we also obtain $\alpha$ estimates which correspond to our expectations for particular languages. For example, adaptation is lowest for the agglutinative Turkish language where longer word tokens should be less likely to repeat. For Vietnamese, with shorter, syllable length word tokens, we observe the lowest adaptation estimates.

Lastly, the reductions in $P\left(\mathrm{Miss}\right)$ suggests that we are improving the term detection metric, which is sensitive to threshold changes, by doing what we set out to do, which is to boost lower confidence repeated words and correctly asserting them as true hits. Moreover, we are able to accomplish this in a wide variety of languages.

Leveraging the burstiness of content words, we have developed a simple technique to consistently boost term detection performance across languages. Using word repetitions, we effectively use a broad document context outside of the typical 2-5 N-gram window. Furthermore, we see improvements across a broad spectrum of languages: languages with syllable-based word tokens (Vietnamese, Cantonese), complex morphology (Turkish), and dialect variability (Pashto).

Secondly, our results are not only effective but also intuitive, given that the interpolation weight parameter matches our expectations for the burstiness of the word tokens in the language on which it is estimated.

We have focused primarily on re-scoring results for the term detection task. Given the effectiveness of the technique across multiple languages, we hope to extend our effort to exploit our human tendency towards redundancy to decoding or other aspects of the spoken document processing pipeline.

This work was partially supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Defense U.S. Army Research Laboratory (DoD / ARL) contract number W911NF-12-C-0015. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoD/ARL, or the U.S. Government.

Insightful discussions with are also gratefully acknowledged.