Forwarded from Tensorflow(@CVision)
چرا استفاده از تابع فعالسازی Relu به جای sigmoid در شبکه های عمیق رایج است؟ چه مزایایی دارد؟
pic: http://jmbeaujour.com/pics/posts_pics/udacity-DL/reLu_function.png
مزایا:
- تنها 50 درصد مواقع این تابع فعال میشود و در نتیجه از لحاظ پردازشی صرفه جویی میشود.
- جلوی انفجار گرادیان یا محو شدن آن را میگیرد. (عجب ترجمه ای!)
- با یافته های بیولوژیکی توسط نوروساینتیست ها مطابقت بیشتری دارد (علوم شناختی)
#ReLU provides some important benefits that might not be obvious at first glance:
✔️during the initialization process of a Neural Network model, weights are distributed at random for each unit. ReLU will only activate approximately 50% of the time, which actually saves some processing
power.
✔️The ReLU structure also solves the 'Vanishing Gradient' and 'Exploding Gradient' problems,both of which are well-known issues with the training process.
✔️ as a marginal benefit, this type of activation achieves 'Biological Plausibility', because it's directly relatable to the common biological model of a neuron.
pic: http://jmbeaujour.com/pics/posts_pics/udacity-DL/reLu_function.png
مزایا:
- تنها 50 درصد مواقع این تابع فعال میشود و در نتیجه از لحاظ پردازشی صرفه جویی میشود.
- جلوی انفجار گرادیان یا محو شدن آن را میگیرد. (عجب ترجمه ای!)
- با یافته های بیولوژیکی توسط نوروساینتیست ها مطابقت بیشتری دارد (علوم شناختی)
#ReLU provides some important benefits that might not be obvious at first glance:
✔️during the initialization process of a Neural Network model, weights are distributed at random for each unit. ReLU will only activate approximately 50% of the time, which actually saves some processing
power.
✔️The ReLU structure also solves the 'Vanishing Gradient' and 'Exploding Gradient' problems,both of which are well-known issues with the training process.
✔️ as a marginal benefit, this type of activation achieves 'Biological Plausibility', because it's directly relatable to the common biological model of a neuron.
Forwarded from Tensorflow(@CVision)
#Relu
[109] Nair, V. and G. E. Hinton. 2010. “Rectified Linear Units Improve Restricted Boltzmann Machines,” Proc. 27th Int. Conf. Mach. Learn., no. 3, pp. 807–814.
[109] Nair, V. and G. E. Hinton. 2010. “Rectified Linear Units Improve Restricted Boltzmann Machines,” Proc. 27th Int. Conf. Mach. Learn., no. 3, pp. 807–814.