🤓Interesting note on weight decay vs L2 regularization

In short, the was difference when moving from caffe (which implements weight decay) to keras (which implements L2). That led to different results on the same net architecture and same set of hyperparameters.

Link: https://bbabenko.github.io/weight-decay/

#DL #nn #hyperopt #hyperparams

Implementing a ResNet model from scratch.

Well-written and explained note on how to build and train a ResNet model from ground zero.

Link: https://towardsdatascience.com/implementing-a-resnet-model-from-scratch-971be7193718

#ResNet #DL #CV #nn #tutorial

Understanding Convolutional Neural Networks through Visualizations in PyTorch

Explanation of how #CNN works

Link: https://towardsdatascience.com/understanding-convolutional-neural-networks-through-visualizations-in-pytorch-b5444de08b91

#PyTorch #nn #DL

Use Machine Learning to filter messages in the browser

Article on filtering messages during #Twitch streaming with #tensorflow.

Link: https://dev.to/embiem/use-machine-learning-to-filter-messages-in-the-browser-4i19

#DL #NN #NLP #tensorflowjs

​​Neural Networks seem to follow a puzzlingly simple strategy to classify images

Interesting article on how actually #NN see images and what helps to distinct different classes.

Link: https://medium.com/bethgelab/neural-networks-seem-to-follow-a-puzzlingly-simple-strategy-to-classify-images-f4229317261f

#BagNet #ResNet #Dl #CV

One of the most complete lists of #NN types.

Link: http://www.asimovinstitute.org/neural-network-zoo/

Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screening

A deep convolutional neural network for breast cancer screening exam classification, trained and evaluated on over 200,000 exams (over 1,000,000 images). #nn achieves an #AUC of 0.895 in predicting whether there is a cancer in the breast, when tested on the screening population.

Link: https://arxiv.org/abs/1903.08297

#cv #dl #cancer #objectdetection

A Recipe for Training Neural Networks by Andrej Karpathy

New article written by Andrej Karpathy distilling a bunch of useful heuristics for training neural nets. The post is full of real-world knowledge and how-to details that are not taught in books and often take endless hours to learn the hard way.

Link: https://karpathy.github.io/2019/04/25/recipe/

#tipsandtricks #karpathy #tutorial #nn #ml #dl

The lottery ticket hypothesis: finding sparse, trainable neural networks

Best paper award at #ICLR2019 main idea: dense, randomly-initialized, networks contain sparse subnetworks that trained in isolation reach test accuracy comparable to the original network. Thus compressing the original network up to 10% its original size.

Paper: https://arxiv.org/pdf/1803.03635.pdf

#nn #research

Yet another good intro into difference between artificial neural network and biological one.

If you're getting started in Data Science, you need to start with the basic building building block of Neural Networks - a Perceptron. To understand what it is, there's this good link to get started with.

Link: https://towardsdatascience.com/the-differences-between-artificial-and-biological-neural-networks-a8b46db828b7

#nn #entrylevel #beginner

​​🥇Parameter optimization in neural networks.

Play with three interactive visualizations and develop your intuition for optimizing model parameters.

Link: https://www.deeplearning.ai/ai-notes/optimization/

#interactive #demo #optimization #parameteroptimization #novice #entrylevel #beginner #goldcontent #nn #neuralnetwork

The HSIC Bottleneck: Deep Learning without Back-Propagation

An alternative to conventional backpropagation, that has a number of distinct advantages.

Link: https://arxiv.org/abs/1908.01580

#nn #backpropagation #DL #theory

​​And the Bit Goes Down: Revisiting the Quantization of Neural Networks

Researchers at Facebook AI Research found a way to compress neural networks with minimal sacrifice in accuracy.

Works only on fully connected and CNN only for now.

Link: https://arxiv.org/abs/1907.05686

#nn #DL #minimization #compresson

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

How normalization applied to layers helps to reach faster convergence.

ArXiV: https://arxiv.org/abs/1502.03167

#NeuralNetwork #nn #normalization #DL

Classification and Loss Evaluation - Softmax and Cross Entropy Loss

Nice notes on softmax cross entropy loss and how to implement it in numpy.

Link: https://deepnotes.io/softmax-crossentropy

#nn #entrylevel #wheretostart

27,600 V100 GPUs, 0.5 PB data, and a neural net with 220,000,000 weights

If you wonder, it all was used to address scientific inverse problem in materials imaging.

ArXiV: https://arxiv.org/pdf/1909.11150.pdf

#ItIsNotAboutSize #nn #dl

Applying deep learning to Airbnb search

Story of how #Airbnb research team moved from using #GBDT (gradient boosting) to #NN (neural networks) for search, with all the metrics and hypothesises.

Link: https://blog.acolyer.org/2019/10/09/applying-deep-learning-to-airbnb-search/

Two papers stating random architecture search is a competitive (in some cases superior) baseline for NAS methods.

These are papers demonstrating that Neural Architecture Search can be stohastic.

Paper 1: https://arxiv.org/abs/1902.08142
Paper 2: https://arxiv.org/abs/1902.07638

#NAS #nn #DL

All the vector algebra you need for understanding neural networks

Article contains great explanations and description of matrix calculus you need to know and understand to really grok neural networks.

Link: https://explained.ai/matrix-calculus/index.html

#WhereToStart #entrylevel #novice #base #DL #nn

Free eBook from Stanford: Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares

Base material you need to understand how neural networks and other #ML algorithms work.

Link: https://web.stanford.edu/~boyd/vmls/

#Stanford #MOOC #WhereToStart #free #ebook #algebra #linalg #NN