A disciplined approach to neural network hyper-parameters
Recommendations on how to optimize learning rate, weight decay, momentum and batch size.
ArXiV: https://arxiv.org/pdf/1803.09820.pdf
#nn #hyperopt
Recommendations on how to optimize learning rate, weight decay, momentum and batch size.
ArXiV: https://arxiv.org/pdf/1803.09820.pdf
#nn #hyperopt
🤓Interesting note on weight decay vs L2 regularization
In short, the was difference when moving from caffe (which implements weight decay) to keras (which implements L2). That led to different results on the same net architecture and same set of hyperparameters.
Link: https://bbabenko.github.io/weight-decay/
#DL #nn #hyperopt #hyperparams
In short, the was difference when moving from caffe (which implements weight decay) to keras (which implements L2). That led to different results on the same net architecture and same set of hyperparameters.
Link: https://bbabenko.github.io/weight-decay/
#DL #nn #hyperopt #hyperparams
bbabenko.github.io
weight decay vs L2 regularization
one popular way of adding regularization to deep learning models is to include a weight decay term in the updates. this is the same thing as adding an $L_2$ ...
HiPlot: High-dimensional interactive plots made easy
Interactive parameters' performance #visualization tool. This new Facebook AI's release enables researchers to more easily evaluate the influence of their hyperparameters, such as learning rate, regularizations, and architecture.
Link: https://ai.facebook.com/blog/hiplot-high-dimensional-interactive-plots-made-easy
Github: https://github.com/facebookresearch/hiplot
Demo: https://facebookresearch.github.io/hiplot/_static/demo/demo_basic_usage.html
Pip:
#hyperopt #facebook #opensource
Interactive parameters' performance #visualization tool. This new Facebook AI's release enables researchers to more easily evaluate the influence of their hyperparameters, such as learning rate, regularizations, and architecture.
Link: https://ai.facebook.com/blog/hiplot-high-dimensional-interactive-plots-made-easy
Github: https://github.com/facebookresearch/hiplot
Demo: https://facebookresearch.github.io/hiplot/_static/demo/demo_basic_usage.html
Pip:
pip install hiplot#hyperopt #facebook #opensource