Tutorials Description:

Matrix and Tensor Factorization Methods for Natural Language Processing

Guillaume Bouchard (Xerox), Jason Naradowsky (UCL), Sebastian Riedel (UCL), Tim Rocktaschel (UCL) and Andreas Vlachos (UCL)

[PDF]

Tensor and matrix factorization methods have attracted a lot of attention recently thanks to their successful applications to information extraction, knowledge base population, lexical semantics and dependency parsing. In the first part, we will first cover the basics of matrix and tensor factorization theory and optimization, and then proceed to more advanced topics involving convex surrogates and alternative losses. In the second part we will discuss recent NLP applications of these methods and show the connections with other popular methods such as transductive learning, topic models and neural networks. The aim of this tutorial is to present in detail applied factorization methods, as well as to introduce more recently proposed methods that are likely to be useful to NLP applications.

Outline:

Part I: Matrix/Tensor Factorization Basics (90 minutes)
- Matrix factorization basics (40 min): bilinear forms, spectral representations, low rank approximations theorems, optimization with stochastic gradient descent, losses
- Tensor factorization basics (20 minutes): representations,notation decompositions (Tucker etc.)
- Advanced topics (30 minutes): convex surrogates, L1 regularization, alternative losses (ranking loss, logistic loss)

Break (15 minutes)

Part II: Applications in NLP (75 minutes)
- Information extraction, knowledge base population with connections to transductive learning and multitask learning (35 minutes)
- Lexical semantics with connections to neural networks, latent semantic analysis and topic models (30 minutes)
- Structured prediction (10 minutes)

Presenters:

Guillaume Bouchard is a senior researcher in statistics and machine learning at Xerox, focusing on statistical learning using low-rank model for large relational databases. His research includes text understanding, user modeling, and social media analytics. The theoretical part of his work is related to the efficient algorithms to compute high dimensional integrals, essential to deal with uncertainty (missing and noisy data, latent variable models, Bayesian inference). The main application areas of his work includes the design of virtual conversational agents, link prediction (predictive algorithms for relational data), social media monitoring and transportation analytics. His web page is available at www.xrce.xerox.com/people/bouchard.

Jason Naradowsky is a postdoc at the Machine Reading group at UCL. Having previously obtained a PhD at UMass Amherst under the supervision of David Smith and Mark Johnson, his current research aims to improve natural language understanding by performing task-specific training of word representations and parsing models. He is also interested in semi-supervised learning, joint inference, and semantic parsing. His web page is available at http://narad.github.io/.

Sebastian Riedel is a senior lecturer at University College London and an Allen Distinguished Investigator, leading the Machine Reading Lab. Before, he was a postdoc and research scientist with Andrew McCallum at UMass Amherst, a researcher at Tokyo University and DBCLS with Tsujii Junichi, and a PhD student with Ewan Klein at the University of Edinburgh.
He is interested in teaching machines how to read and works at the intersection of Natural Language Processing (NLP) and Machine Learning, investigating various stages of the NLP pipeline, in particular those that require structured prediction, as well as fully probabilistic architectures of end-to-end reading and reasoning systems. Recently he became interested in new ways to represent textual knowledge using low-rank embeddings and how to reason with such representations. His web page is available at http://www.riedelcastro.org/.

Tim Rocktaschel is a PhD student in Sebastian Riedel's Machine Reading group at University College London. Before that he worked as research assistant in the Knowledge Management in Bioinformatics group at Humboldt-Universit{\"a}t zu Berlin, where he also obtained his Diploma in Computer Science. He is broadly interested in representation learning (e.g. matrix/tensor factorization, deep learning) for NLP and automated knowledge base completion, and how these methods can take advantage of symbolic background knowledge. His webpage is available at http://rockt.github.io/.

Andreas Vlachos is postdoc at the Machine Reading group at UCL working with Sebastian Riedel on automated fact-checking using low-rank factorization methods.
Before that he was a postdoc at the Natural Language and Information Processing group at the University of Cambridge and at the University of Wisconsin-Madison. He is broadly interested in natural language understanding (e.g. information extraction, semantic parsing) and in machine learning approaches that would help us towards this goal. He has also worked on active learning, clustering and biomedical text mining. His web page is available at http://sites.google.com/site/andreasvlachos/.

[back to tutorials list]