Hyperband

Hyperband: a novel bandit-based approach to hyperparameter optimization. Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration nonstochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.


References in zbMATH (referenced in 20 articles )

Showing results 1 to 20 of 20.
Sorted by year (citations)

  1. Chawshin, Kurdistan; Berg, Carl Fredrik; Varagnolo, Damiano; Lopez, Olivier: Automated porosity estimation using CT-scans of extracted core data (2022)
  2. Hertel, Lars; Baldi, Pierre; Gillen, Daniel L.: Reproducible hyperparameter optimization (2022)
  3. Lakhmiri, Dounia; Le Digabel, Sébastien: Use of static surrogates in hyperparameter optimization (2022)
  4. Ozaki, Yoshihiko; Tanigaki, Yuki; Watanabe, Shuhei; Nomura, Masahiro; Onishi, Masaki: Multiobjective tree-structured Parzen estimator (2022)
  5. Zhen Wang, Weirui Kuang, Yuexiang Xie, Liuyi Yao, Yaliang Li, Bolin Ding, Jingren Zhou: FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learning (2022) arXiv
  6. Binder, Martin; Pfisterer, Florian; Lang, Michel; Schneider, Lennart; Kotthoff, Lars; Bischl, Bernd: mlr3pipelines -- flexible machine learning pipelines in R (2021)
  7. Chen, Xiaoli; Duan, Jinqiao; Karniadakis, George Em: Learning and meta-learning of stochastic advection-diffusion-reaction systems from sparse measurements (2021)
  8. De Loera, Jesús A.; Haddock, Jamie; Ma, Anna; Needell, Deanna: Data-driven algorithm selection and tuning in optimization and signal processing (2021)
  9. Grosnit, Antoine; Cowen-Rivers, Alexander I.; Tutunov, Rasul; Griffiths, Ryan-Rhys; Wang, Jun; Bou-Ammar, Haitham: Are we forgetting about compositional optimisers in Bayesian optimisation? (2021)
  10. Lakhmiri, Dounia; Digabel, Sébastien Le; Tribes, Christophe: HyperNOMAD. Hyperparameter optimization of deep neural networks using mesh adaptive direct search (2021)
  11. Yang, Zebin; Zhang, Aijun: Hyperparameter optimization via sequential uniform designs (2021)
  12. Zöller, Marc-André; Huber, Marco F.: Benchmark and survey of automated machine learning frameworks (2021)
  13. Chamakh, Linda; Gobet, Emmanuel; Szabó, Zoltán: Orlicz random Fourier features (2020)
  14. Kandasamy, Kirthevasan; Vysyaraju, Karun Raju; Neiswanger, Willie; Paria, Biswajit; Collins, Christopher R.; Schneider, Jeff; Poczos, Barnabas; Xing, Eric P.: Tuning hyperparameters without grad students: scalable and robust Bayesian optimisation with Dragonfly (2020)
  15. Ribeiro, Rita P.; Moniz, Nuno: Imbalanced regression and extreme value prediction (2020)
  16. Altschuler, Jason; Brunel, Victor-Emmanuel; Malek, Alan: Best arm identification for contaminated bandits (2019)
  17. Candelieri, A.; Perego, R.; Archetti, F.: Bayesian optimization of pump operations in water distribution systems (2018)
  18. Chan, Shing; Elsheikh, Ahmed H.: A machine learning approach for efficient uncertainty quantification using multiscale methods (2018)
  19. Li, Lisha; Jamieson, Kevin; DeSalvo, Giulia; Rostamizadeh, Afshin; Talwalkar, Ameet: Hyperband: a novel bandit-based approach to hyperparameter optimization (2018)
  20. Zhu, Yinhao; Zabaras, Nicholas: Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quantification (2018)