• Home
• Papers
• GP Tutorial
• Teaching
• Team
• CV

Trevor Cohn

Senior Lecturer
ARC Future Fellow

Computing and Information Systems
The University of Melbourne
Language Technology and Knowledge Discovery research groups

Google Scholar

DBLP

Understanding of human language by computers has been a central goal of Artificial Intelligence since its beginnings, with massive potential to improve communication, provide better information access and automate basic human tasks. My research focuses on technologies for automatic processing of human language, with several applications including automatic translation (akin to Google and Bing's translation tools). My core focus is on probabilistic machine learning modelling of language applications, particularly handling uncertain or partly observed data and structured prediction problems.

News

• Melbourne will host ACL in July 2018, at the Melbourne Convention Centre, with Tim Baldwin, Karin Verspoor and myself serving as the local chairs.
• Co-organising the ALTA 2016 workshop to be held in Monash Caulfield campus in Melbourne, on the 5th - 7th of December 2016. Please submit your papers by 30th September.
• I'll be giving a tutorial on succinct data structures for NLP at COLING 2016 in Osaka, Japan on 12th December, 2016.

Current Projects

• Efficient storage and access to text count data: An application to unlimited order language modelling. 2016 – 2017. Google Research Award, $US 85k.
• Learning Deep Semantics for Automatic Translation between Human Languages. 2016 – 2019. ARC Discovery with Reza Haffari, $450k.
• Ariel: Analysis of Rare Incident-Event Languages. 2015 – 2018. DARPA LORELEI (sub-contract), $300k.
• Adaptive Context-Dependent Machine Translation for Heterogeneous Text. 2014 – 2018. ARC Future Fellowship, $730k.
• Pheme: Computing Veracity Across Media, Languages, and Social Networks. 2014 – 2017. EU FP7 with Kalina Bontcheva and others, £494k.

Selected Papers

Fast, Small and Exact: Infinite-order Language Modelling with Compressed Suffix Trees
Ehsan Shareghi, Matthias Petri, Gholamreza Haffari and Trevor Cohn. In Transactions of ACL (TACL) (to be presented at EMNLP-16), 2016.
Abstract

×

Fast, Small and Exact: Infinite-order Language Modelling with Compressed Suffix Trees

Efficient methods for storing and querying are critical for scaling high-order n-gram language models to large corpora. We propose a language model based on compressed suffix trees, a representation that is highly compact and can be easily held in memory, while supporting queries needed in computing language model probabilities on-the-fly. We present several optimisations which improve query runtimes up to 2500x, despite only incurring a modest increase in construction time and memory usage. For large corpora and high Markov orders, our method is highly competitive with the state-of-the-art KenLM package. It imposes much lower memory requirements, often by orders of magnitude, and has runtimes that are either similar (for training) or comparable (for querying).

PDF Code

Learning Crosslingual Word Embeddings without Bilingual Corpora
Long Duong, Hiroshi Kanayama, Tengfei Ma, Steven Bird and Trevor Cohn. In Proceedings of EMNLP, 2016.
Abstract

×

Learning Crosslingual Word Embeddings without Bilingual Corpora

Crosslingual word embeddings represent lexical items from different languages in the same vector space, enabling transfer of NLP tools. However, previous attempts had expensive resource requirements, difficulty incorporating monolingual data or were unable to handle polysemy. We address these drawbacks in our method which takes advantage of a high coverage dictionary in an EM style training algorithm over monolingual corpora in two languages. Our model achieves state-of-the-art performance on bilingual lexicon induction task exceeding models using large bilingual corpora, and competitive results on the monolingual word similarity and cross-lingual document classification task.

PDF Code

Learning a Lexicon and Translation Model from Phoneme Lattices
Oliver Adams, Graham Neubig, Trevor Cohn, Steven Bird, Quoc Truong Do and Satoshi Nakamura. In Proceedings of EMNLP, 2016.
Abstract

×

Learning a Lexicon and Translation Model from Phoneme Lattices

Language documentation begins by gathering speech. Manual or automatic transcription at the word level is typically not possible because of the absence of an orthography or prior lexicon, and though manual phonemic transcription is possible, it is very slow. On the other hand, translations of the minority language into a major language are more easily acquired. We propose a method to harness such translations to improve automatic phoneme recognition. The method assumes no prior lexicon or translation model, instead learning them from phoneme lattices and translations of the speech being transcribed. Experiments demonstrate phoneme error rate improvements against two baselines and the model's ability to learn useful bilingual lexical entries.

Richer Interpolative Smoothing Based on Modified Kneser-Ney Language Modeling
Ehsan Shareghi, Trevor Cohn and Gholamreza Haffari. In Proceedings of EMNLP (short papers), 2016.
Abstract

×

Richer Interpolative Smoothing Based on Modified Kneser-Ney Language Modeling

In this work we present a natural extension of the widely used Modified Kneser-Ney interpolative smoothing technique which provides richer smoothing via more discount parameters. We provide mathematical underpining for the estimators of new discount parameters, and showcase the utility of our rich MKN language models on data from multiple languages with varying size. The empirical results reveal that our language model leads to significant reduction in perplexity, particularly for small data regime.

Learning Robust Representations of Text
Yitong Li, Trevor Cohn and Timothy Baldwin. In Proceedings of EMNLP (short papers), 2016.
Abstract

×

Learning Robust Representations of Text

Deep neural networks have achieved remarkable results across many language processing tasks, however these methods are highly sensitive to noise and adversarial attacks. We present a regularization based method for limiting network sensitivity to its inputs, inspired by ideas from computer vision, thus learning models that are more robust. Empirical evaluation over a range of sentiment datasets with a convolutional neural network shows that, compared to a baseline model and the dropout method, our method achieves superior performance over noisy inputs and out-of-domain data.

PDF Code

Learning a Translation Model from Word Lattices
Oliver Adams, Graham Neubig, Trevor Cohn and Steven Bird. In Proceedings of Interspeech-16, 2016.
Abstract

×

Learning a Translation Model from Word Lattices

Translation models have been used to improve automatic speech recognition when speech input is paired with a written translation, primarily for the task of computer-aided translation. Existing approaches require large amounts of parallel text for training the translation models, but for many language pairs this data is not available. We propose a model for learning lexical translation parameters directly from the word lattices for which a transcription is sought. The model is expressed through composition of each lattice with a weighted finite-state transducer representing the translation model, where inference is performed by sampling paths through the composed finitestate transducer. We show consistent word error rate reductions in two datasets, using between just 20 minutes and 4 hours of speech input, additionally outperforming a translation model trained on the 1-best path.

PDF Code

Exploiting Tree Kernels for high performance Chemical Induced Disease relation extraction
Nagesh C. Panyam, Karin Verspoor, Trevor Cohn and Kotagiri Ramamohanarao. In Proceedings of International Symposium on Semantic Mining in Biomedicine (SMBM), 2016.
Abstract

×

Exploiting Tree Kernels for high performance Chemical Induced Disease relation extraction

Machine learning approaches based on supervised classification have emerged as effective methods for Biomedical relation extraction such as the Chemical-Induced-Disease (CID) task. These approaches owe their success to a rich set of features crafted from the lexical and syntactic regularities in the text. Kernel methods are an effective alternative to manual feature engineering and have been successfully used in similar tasks such as text classification. In this paper, we study the effectiveness of tree kernels for Chemical-Disease relation extraction. Our experiments demonstrate that subset tree kernels increase the F-score to 61.7% as compared to 57.9% achieved with simple feature engineering. We also describe the strengths and short-comings of tree kernel approaches for the CID relation extraction task.

PDF

SeeDev Binary Event Extraction using SVMs and a Rich Feature Set
Nagesh C. Panyam, Gitansh Khirbat, Karin Verspoor, Trevor Cohn and Kotagiri Ramamohanarao. In Proceedings of BioNLP-ST, 2016.
Abstract

×

SeeDev Binary Event Extraction using SVMs and a Rich Feature Set

This paper describes the system details and results of the participation of the team from the University of Melbourne in the SeeDev binary event extraction of BioNLP-Shared Task 2016. This task addresses the extraction of genetic and molecular mechanisms that regulate plant seed development from the natural lan- guage text of the published literature. In our submission, we developed a system1 using a support vector machine classifier with linear kernel powered by a rich set of features. Our system achieved an F1-score of 36.4%.

PDF Code

Learning when to trust distant supervision: An application to low-resource POS tagging using cross-lingual projection
Meng Fang and Trevor Cohn. In Proceedings of CoNLL-16, 2016.
Abstract

×

Learning when to trust distant supervision: An application to low-resource POS tagging using cross-lingual projection

Cross lingual projection of linguistic annotation suffers from many sources of bias and noise, leading to unreliable annotations that cannot be used directly. In this paper, we introduce a novel approach to sequence tagging that learns to correct the errors from cross-lingual projection using an explicit debiasing layer. This is framed as joint learning over two corpora, one tagged with gold standard and the other with projected tags. We evaluated with only 1,000 tokens tagged with gold standard tags, along with more plentiful parallel data. Our system equals or exceeds the state-of-the-art on eight simulated low-resource settings, as well as two real low-resource languages, Malagasy and Kinyarwanda.

PDF

Exploring Prediction Uncertainty in Machine Translation Quality Estimation,
Daniel Beck, Lucia Specia and Trevor Cohn. In Proceedings of CoNLL-16, 2016.
Abstract

×

Exploring Prediction Uncertainty in Machine Translation Quality Estimation,

Machine Translation Quality Estimation is a notoriously difficult task, which lessens its usefulness in real-world translation environments. Such scenarios can be improved if quality predictions are accompanied by a measure of uncertainty. However, models in this task are traditionally evaluated only in terms of point estimate metrics, which do not take prediction uncertainty into account. We investigate probabilistic methods for Quality Estimation that can provide well-calibrated uncertainty estimates and evaluate them in terms of their full posterior predictive distributions. We also show how this posterior information can be useful in an asymmetric risk scenario, which aims to capture typical situations in translation workflows.

PDF

Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning
Ekaterina Vylomova, Laura Rimell, Trevor Cohn and Timothy Baldwin. In Proceedings of ACL-16, 2016.
Abstract

×

Take and Took, Gaggle and Goose, Book and Read: Evaluating the Utility of Vector Differences for Lexical Relation Learning

Recent work has shown that simple vector subtraction over word embeddings is surprisingly effective at capturing different lexical relations, despite lacking explicit supervision. Prior work has evaluated this intriguing result using a word analogy prediction formulation and hand-selected relations, but the generality of the finding over a broader range of lexical relation types and different learning settings has not been evaluated. In this paper, we carry out such an evaluation in two learning settings: (1) spectral clustering to induce word relations, and (2) supervised learning to classify vector differences into relation types. We find that word embeddings capture a surprising amount of information, and that, under suitable supervised training, vector subtraction generalises well to a broad range of relations, including over unseen lexical items.

PDF

pigeo: A Python Geotagging Tool
Afshin Rahimi, Trevor Cohn and Timothy Baldwin. In Proceedings of ACL-16 (Demonstrations), 2016.
Abstract

×

pigeo: A Python Geotagging Tool

We present pigeo, a Python geolocation prediction tool that predicts a location for a given text input or Twitter user. We discuss the design, implementation and application of pigeo, and empirically evaluate it. pigeo is able to geolocate informal text and is a very useful tool for users who require a free and easy-to-use, yet accurate geolocation service based on pre-trained models. Additionally, users can train their own models easily using pigeo’s API.

PDF Code

Hawkes Processes for Continuous Time Sequence Classification: An Application to Rumour Stance Classification in Twitter
Michal Lukasik, P.K. Srijith, Duy Vu, Kalina Bontcheva, Arkaitz Zubiaga and Trevor Cohn. In Proceedings of ACL-16 (Short papers), 2016.
Abstract

×

Hawkes Processes for Continuous Time Sequence Classification: An Application to Rumour Stance Classification in Twitter

Classification of temporal textual data sequences is a common task in various domains such as social media and the Web. In this paper we propose to use Hawkes Processes for classifying sequences of temporal textual data, which exploit both temporal and textual information. Our experiments on rumour stance classification on four Twitter datasets show the importance of using the temporal information of tweets along with the textual content.

PDF Code

Incorporating Structural Alignment Biases into an Attentional Neural Translation Model
Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vylomova, Kaisheng Yao, Chris Dyer and Gholamreza Haffari. In Proceedings of NAACL-16, 2016.
Abstract

×

Incorporating Structural Alignment Biases into an Attentional Neural Translation Model

Neural encoder-decoder models of machine translation have achieved impressive results, rivalling traditional translation models. However their modelling formulation is overly simplistic, and omits several key inductive biases built into traditional models. In this paper we extend the attentional neural translation model to include structural biases from word based alignment models, including positional bias, Markov conditioning, fertility and agreement over translation directions. We show improvements over a baseline attentional model and standard phrase-based model over several language pairs, evaluating on difficult languages in a low resource setting.

PDF Code

An Attentional Model for Speech Translation Without Transcription
Long Duong, Antonios Anastasopoulos, Steven Bird, David Chiang and Trevor Cohn. In Proceedings of NAACL-16, 2016.
Abstract

×

An Attentional Model for Speech Translation Without Transcription

For many low-resource languages, spoken language resources are more likely to be annotated with translations than transcriptions. This bilingual speech data can be used for word-spotting, spoken document retrieval, and even for documentation of endangered languages. We experiment with the neural, attentional model applied to this data. On phone-to-word alignment and translation reranking tasks, we achieve large improvements relative to several baselines. On the more challenging speech-to-word alignment task, our model nearly matches GIZA++’s performance on gold transcriptions, but without recourse to transcriptions or to a lexicon.

PDF Code

Incorporating Context into Recurrent Neural Network Language Models
Cong Duy Vu Hoang, Gholamreza Haffari and Trevor Cohn. In Proceedings of NAACL-16 (short), 2016.
Abstract

×

Incorporating Context into Recurrent Neural Network Language Models

Recurrent neural network language models (RNNLM) have recently demonstrated vast potential in modelling long-term dependencies for NLP problems, ranging from speech recognition to machine translation. In this work, we propose methods for conditioning RNNLMs on external side information, e.g., metadata such as keywords or document title. Our experiments show consistent improvements of RNNLMs using side information over the baselines for two different datasets and genres in two languages. Interestingly, we found that side information in a foreign language can be highly beneficial in modelling texts in another language, serving as a form of cross-lingual language modelling.

PDF

Document Context Language Models
Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, Jacob Eisenstein. In Proceedings of ICLR-16 Workshop, 2016.
Abstract

×

Document Context Language Models

Text documents are structured on multiple levels of detail: individual words are related by syntax, but larger units of text are related by discourse structure. Existing language models generally fail to account for discourse structure, but it is crucial if we are to have language models that reward coherence and generate coherent texts. We present and empirically evaluate a set of multi-level recurrent neural network language models, called Document-Context Language Models (DCLM), which incorporate contextual information both within and beyond the sentence. In comparison with word-level recurrent neural network language models, the DCLM models obtain slightly better predictive likelihoods, and considerably better assessments of document coherence.

PDF

Convolution Kernels for Discriminative Learning from Streaming Text
Michal Lukasik and Trevor Cohn. In Proceedings of AAAI-16, 2016.
Abstract

×

Convolution Kernels for Discriminative Learning from Streaming Text

Time series modeling is an important problem with many applications in different domains. Here we consider discriminative learning from time series, where we seek to predict an output response variable based on time series input. We develop a method based on convolution kernels to model discriminative learning over streams of text. Our method outperforms competitive baselines in three synthetic and two real datasets, rumour frequency modeling and popularity prediction tasks.

PDF