Abstract: We propose a novel distribution that generalizes the Multinomial distribution to enable dependencies between the dimensions. Our novel distribution is based on the parametric form of the Poisson MRF model [Yang, 2012] but is fundamentally different because of the domain restriction to a fixed-length vector like in a Multinomial where the number of trials is fixed or known. Thus, we propose the Fixed-Length Poisson MRF (LPMRF) distribution. We develop methods to estimate the likelihood and log partition function (i.e. the log normalizing constant), which was not previously possible with the Poisson MRF model. In addition, we create mixture and topic models that use LPMRF as a base distribution and discuss the similarities and differences with previous topic models such as the recently proposed Admixture of Poisson MRFs [Inouye, 2014]. Finally, we show the effectiveness of our LPMRF distribution over Multinomial models by evaluating the test set perplexity on a dataset of abstracts. Qualitatively, we show that the positive dependencies discovered by LPMRF are interesting and intuitive.
- Topics:
- Graphical Models
- Topic Models
Download: pdf, poster, code
Citation
- Fixed-Length Poisson MRF: Adding Dependencies to the Multinomial (pdf, poster, software, code)
D. Inouye, P. Ravikumar, I. Dhillon.
In Neural Information Processing Systems (NIPS), pp. 3195-3203, December 2015.
Bibtex: