MADE: Masked Autoencoder for Distribution Estimation. There has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder’s parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.

References in zbMATH (referenced in 12 articles , 1 standard article )

Showing results 1 to 12 of 12.
Sorted by year (citations)

  1. Papamakarios, George; Nalisnick, Eric; Rezende, Danilo Jimenez; Mohamed, Shakir; Lakshminarayanan, Balaji: Normalizing flows for probabilistic modeling and inference (2021)
  2. Tajnafoi, Gabor; Arcucci, Rossella; Mottet, Laetitia; Vouriot, Carolanne; Molina-Solana, Miguel; Pain, Christopher; Guo, Yi-Ke: Variational Gaussian process for optimal sensor placement. (2021)
  3. Arenz, Oleg; Zhong, Mingjun; Neumann, Gerhard: Trust-region variational inference with Gaussian mixture models (2020)
  4. Brehmer, Johann; Louppe, Gilles; Pavez, Juan; Cranmer, Kyle: Mining gold from implicit models to improve likelihood-free inference (2020)
  5. Harshvardhan, G. M.; Gourisaria, Mahendra Kumar; Pandey, Manjusha; Rautaray, Siddharth Swarup: A comprehensive survey and analysis of generative models in machine learning (2020)
  6. Tan, Linda S. L.; Bhaskaran, Aishwarya; Nott, David J.: Conditionally structured variational Gaussian approximation with importance weights (2020)
  7. Vergari, Antonio; Di Mauro, Nicola; Esposito, Floriana: Visualizing and understanding sum-product networks (2019)
  8. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le: XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019) arXiv
  9. Goessling, Marc: Logitboost autoregressive networks (2017)
  10. Mocanu, Decebal Constantin; Mocanu, Elena; Nguyen, Phuong H.; Gibescu, Madeleine; Liotta, Antonio: A topological insight into restricted Boltzmann machines (2016)
  11. Uria, Benigno; Côté, Marc-Alexandre; Gregor, Karol; Murray, Iain; Larochelle, Hugo: Neural autoregressive distribution estimation (2016)
  12. Mathieu Germain, Karol Gregor, Iain Murray, Hugo Larochelle: MADE: Masked Autoencoder for Distribution Estimation (2015) arXiv