H. Schmid. Probabilistic part-of-speech tagging using decision trees. In Proceedings of International Conference on New Methods in Language Processing, September 1994.[url]

——————–

In this paper, a new probabilistic tagging method is presented which avoids problems that Markov models based tagger face, when they have to estimate transition probabilities from sparse data.

In this tagging method, transition probabilities are estimate using a decision tree. Based on this method, a part-of-speech tagger (called TreeTagger) has been implemented which achieves 96.36% of accuracy on Penn-Treebank data which is better than that of a trigram tagger on the same data.

Tags: statistics, text data mining