Distributed Representations of Geographically Situated Language

The vast textual resources used in NLP – newswire, web text, parliamentary proceedings – can encourage a view of language as a disembodied phenomenon. The rise of social media, however, with its large volume of text paired with information about its author and social context, reminds us that each word is uttered by a particular person at a particular place and time. In short: language is situated.

The coupling of text with demographic information has enabled computational modeling of linguistic variation, including uncovering words and topics that are characteristic of geographical regions [], learning correlations between words and socioeconomic variables []; and charting how new terms spread geographically []. These models can tell us that hella was (at one time) used most often by a particular demographic group in northern California, echoing earlier linguistic studies [], and that wicked is used most often in New England []; and they have practical applications, facilitating tasks like text-based geolocation []. One desideratum that remains, however, is how the meaning of these terms is shaped by geographical influences – while wicked is used throughout the United States to mean bad or evil (“he is a wicked man”), in New England it is used as an adverbial intensifier (“my boy’s wicked smart”). In leveraging grounded social media to uncover linguistic variation, what we want to learn is how a word’s meaning is shaped by its geography.

In this paper, we introduce a method that extends vector-space lexical semantic models to learn representations of geographically situated language. Vector-space models of lexical semantics have been a popular and effective approach to learning representations of word meaning [, inter alia]. In bringing in extra-linguistic information to learn word representations, our work falls into the general domain of multimodal learning; while other work has used visual information to improve distributed representations [], this work generally exploits information about the object being described (e.g., strawberry and a picture of a strawberry); in contrast, we use information about the speaker to learn representations that vary according to contextual variables from the speaker’s perspective. Unlike classic multimodal systems that incorporate multiple active modalities (such as gesture) from a user [], our primary input is textual data, supplemented with metadata about the author and the moment of authorship. This information enables learning models of word meaning that are sensitive to such factors, allowing us to distinguish, for example, between the usage of wicked in Massachusetts from the usage of that word elsewhere, and letting us better associate geographically grounded named entities (e.g, Boston) with their hypernyms (city) in their respective regions.

The model we introduce is grounded in the distributional hypothesis [], that two words are similar by appearing in the same kinds of contexts (where “context” itself can be variously defined as the bag or sequence of tokens around a target word, either by linear distance or dependency path). We can invoke the distributional hypothesis for many instances of regional variation by observing that such variants often appear in similar contexts. For example:

• •

  my boy’s wicked smart

• •

  my boy’s hella smart

• •

  my boy’s very smart

Here, all three variants can often be seen in an immediately pre-adjectival position (as is common with intensifying adverbs).

Given the empirical success of vector-space representations in capturing semantic properties and their success at a variety of NLP tasks [], we use a simple, but state-of-the-art neural architecture [] to learn low-dimensional real-valued representations of words. The graphical form of this model is illustrated in figure 1.

This model corresponds to an extension of the “skip-gram” language model [] (hereafter SGLM). Given an input sentence $𝒔$ and a context window of size $t$, each word $s_i$ is conditioned on in turn to predict the identities of all of the tokens within $t$ words around it. For a vocabulary $V$, each input word $s_i$ is represented as a one-hot vector ${𝒘}_i$ of length $\left|V\right|$. The SGLM has two sets of parameters. The first is the representation matrix $W\in {ℝ}^{\left|V\right|\times k}$, which encodes the real-valued embeddings for each word in the vocabulary. A matrix multiply $𝒉={𝒘}^{\top }W,\in {ℝ}^k$ serves to index the particular embedding for word $𝒘$, which constitutes the model’s hidden layer. To predict the value of the context word $y$ (again, a one-hot vector of dimensionality $\left|V\right|$), this hidden representation $𝒉$ is then multiplied by a second parameter matrix $X\in {ℝ}^{\left|V\right|\times k}$. The final prediction over the output vocabulary is then found by passing this resulting vector through the softmax function $𝒐=\text{softmax}\left(X𝒉\right)$, giving a vector in the $\left|V\right|$-dimensional unit simplex. Backpropagation using (input $x$, output $y$) word tuples learns the values of $W$ (the embeddings) and $X$ (the output parameter matrix) that maximize the likelihood of $y$ (i.e., the context words) conditioned on $x$ (i.e., the $s_i$’s). During backpropagation, the errors propagated are the difference between $𝒐$ (a probability distribution with $k$ outcomes) and the true (one-hot) output $y$.

Let us define a set of contextual variables $𝒞$; in the experiments that follow, $𝒞$ is comprised solely of geographical state ${𝒞}_{\mathrm{𝑠𝑡𝑎𝑡𝑒}}=\left\{\text{AK},\text{AL},\mathrm{\dots },\text{WY}\right\}$) but could in principle include any number of features, such as calendar month, day of week, or other demographic variables of the speaker. Let $\left|𝒞\right|$ denote the sum of the cardinalities of all variables in $𝒞$ (i.e., $51$ states, including the District of Columbia). Rather than using a single embedding matrix $W$ that contains low-dimensional representations for every word in the vocabulary, we define a global embedding matrix $W_{\mathrm{𝑚𝑎𝑖𝑛}}\in {ℝ}^{\left|V\right|\times k}$ and an additional $\left|𝒞\right|$ such matrices (each again of size $\left|V\right|\times k$, which capture the effect that each variable value has on each word in the vocabulary. Given an input word $𝒘$ and set of active variable values $𝒜$ (e.g., $𝒜=\{\mathrm{𝑠𝑡𝑎𝑡𝑒}=\text{MA}\}$), we calculate the hidden layer $𝒉$ as the sum of these independent embeddings: $𝒉={𝒘}^{\top }{W}_{\mathrm{𝑚𝑎𝑖𝑛}}+{\sum }_{a\in 𝒜}{𝒘}^{\top }{W}_a$. While the word wicked has a common low-dimensional representation in $W_{\mathrm{𝑚𝑎𝑖𝑛},\mathrm{𝑤𝑖𝑐𝑘𝑒𝑑}}$ that is invoked for every instance of its use (regardless of the place), the corresponding vector $W_{\text{MA},\mathrm{𝑤𝑖𝑐𝑘𝑒𝑑}}$ indicates how that common representation should shift in $k$-dimensional space when used in Massachusetts. Backpropagation functions as in standard SGLM, with gradient updates for each training example $\left\{x,y\right\}$ touching not only $W_{\mathrm{𝑚𝑎𝑖𝑛}}$ (as in SGLM), but all active $W_{𝒜}$ as well.

The additional $W$ embeddings we add lead to an increase in the number of total parameters by a factor of $\left|𝒞\right|$. To control for the extra degrees of freedom this entails, we add squared ${\mathrm{\ell }}_2$ regularization to all parameters, using stochastic gradient descent for backpropagation with minibatch updates for the regularization term. As in Mikolov et al. [], we speed up computation using the hierarchical softmax [] on the output matrix $X$.

This model defines a joint parameterization over all variable values in the data, where information from data originating in California, for instance, can influence the representations learned for Wisconsin; a naive alternative would be to simply train individual models on each variable value (a “California” model using data only from California, etc.). A joint model has three a priori advantages over independent models: (i) sharing data across variable values encourages representations across those values to be similar; e.g., while city may be closer to Boston in Massachusetts and Chicago in Illinois, in both places it still generally connotes a municipality; (ii) such sharing can mitigate data sparseness for less-witnessed areas; and (iii) with a joint model, all representations are guaranteed to be in the same vector space and can therefore be compared to each other; with individual models (each with different initializations), word vectors across different states may not be directly compared.

We evaluate our model by confirming its face validity in a qualitative analysis and estimating its accuracy at the quantitative task of judging geographically-informed semantic similarity. We use 1.1 billion tokens from 93 million geolocated tweets gathered between September 1, 2011 and August 30, 2013 (approximately 127,000 tweets per day evenly sampled over those two years). This data only includes tweets that have been geolocated to state-level granularity in the United States using high-precision pattern matching on the user-specified location field (e.g., “new york ny” $\to$ NY, “chicago” $\to$ IL, etc.). As a preprocessing step, we identify a set of target multiword expressions in this corpus as the maximal sequence of adjectives + nouns with the highest pointwise mutual information; in all experiments described below, we define the vocabulary $V$ as the most frequent 100,000 terms (either unigrams or multiword expressions) in the total data, and set the dimensionality of the embedding $k=100$. In all experiments, the contextual variable is the observed US state (including DC), so that $\left|𝒞\right|=51$; the vector space representation of word $𝒘$ in state $s$ is ${𝒘}^{\top }{W}_{\mathrm{𝑚𝑎𝑖𝑛}}+{𝒘}^{\top }{W}_s$.

We introduced a model for leveraging situational information in learning vector-space representations of words that are sensitive to the speaker’s social context. While our results use geographical information in learning low-dimensional representations, other contextual variables are straightforward to include as well; incorporating effects for time – such as time of day, month of year and absolute year – may be a powerful tool for revealing periodic and historical influences on lexical semantics.

Our approach explores the degree to which geography, and other contextual factors, influence word meaning in addition to frequency of usage. By allowing all words in different regions (or more generally, with different metadata factors) to exist in the same vector space, we are able compare different points in that space – for example, to ask what terms used in Chicago are most similar to hot dog in New York, or what word groups shift together in the same region in comparison to the background (indicating the shift of an entire semantic field). All datasets and software to support these geographically-informed representations can be found at: http://www.ark.cs.cmu.edu/geoSGLM.

The research reported in this article was supported by US NSF grants IIS-1251131 and CAREER IIS-1054319, and by an ARCS scholarship to D.B. This work was made possible through the use of computing resources made available by the Open Cloud Consortium, Yahoo and the Pittsburgh Supercomputing Center.