Colab SeedBank

- TF is everywhere (naturally) - but at least they use keras
- On the other hand - all of the files are (at least now) downloadable via .ipynb or .py
- So - it may be a good place to look for boilerplate code

Also interesting facts, that are not mentioned openly
- Looks like they use Tesla K80s, which practically are 2.5-3x slower than 1080Ti
(https://medium.com/initialized-capital/benchmarking-tensorflow-performance-and-cost-across-different-gpu-options-69bd85fe5d58)
- Full screen notebook format is clearly inspired by Jupyter plugins
- Ofc there is a time limit for GPU scripts and GPU availability is not guaranteed (reported by people who used it)
- Personally - it looks a bit like slow instances from FloydHub - time limitations / slow GPU etc/etc

In a nutshell - perfect source of boilerplate code + playground for new people.

#deep_learning

Lazy failsafe in PyTorch Data Loader

Sometimes you train a model and testing all the combinations of augmentations / keys / params in your dataloader is too difficult. Or the dataset is too large, so it would take some time to check it properly.

In such cases I usually used some kind of failsafe try/catch.
But looks like even simpler approach works:

if img is None:
    # do not return anything
     pass
else:
    return img

#deep_learning
#pytorch

Playing with focal loss for multi-class classification

Playing with this Loss
https://gist.github.com/snakers4/5739ade67e54230aba9bd8a468a3b7be

If anyone has a better option - please PM me / or comment in the gist.

#deep_learning
#data_science

Playing with open-images

Did a benchmark of multi-class classification models and approaches useful in general with multi-tier classificators.

The basic idea is - follow the graph structure of class dependencies - train a good multi-class classifier => train coarse semseg models for each big cluster.

What worked
- Using SOTA classifiers from imagenet
- Pre-training with frozen encoder (otherwise the model performes worse)
- Best performing architecture so far - ResNet152 (a couple of others to try as well)
- Different resolutions => binarise them => divide into 3 major clusters (2:1,1:2,1:1)
- Using adaptive pooling for different aspect ratio clusters

What did not work or did not significantly improve results
- Oversampling
- Using modest or minor augs (10% or 25% of images augmented)

What did not work
- Using 1xN + Nx1 convolutions instead of pooling - too heavy
- Using some minimal avg. pooling (like 16x16), then using different 1xN + Nx1 convolutions for different clusters - performed mostly worse than just adaptive pooling

Yet to try
- Focal loss
- Oversampling + augs

#deep_learning

2018 DS/ML digest 18

Highlights of the week
(0) RL flaws
https://thegradient.pub/why-rl-is-flawed/
https://thegradient.pub/how-to-fix-rl/
(1) An intro to AUTO-ML
http://www.fast.ai/2018/07/16/auto-ml2/
(2) Overview of advances in ML in last 12 months
https://www.stateof.ai/

Market / applied stuff / papers
(0) New Nvidia Jetson released
https://www.phoronix.com/scan.php?page=news_item&px=NVIDIA-Jetson-Xavier-Dev-Kit
(1) Medical CV project in Russia - 90% is data gathering
http://cv-blog.ru/?p=217
(2) Differentiable architecture search
https://arxiv.org/pdf/1806.09055.pdf
-- 1800 GPU days of reinforcement learning (RL) (Zoph et al., 2017)
-- 3150 GPU days of evolution (Real et al., 2018)
-- 4 GPU days to achieve SOTA in CIFAR => transferrable to Imagenet with 26.9% top-1 error
(3) Some basic thoughts about hyper-param tuning
https://engineering.taboola.com/hitchhikers-guide-hyperparameter-tuning/
(4) FB extending fact checking to mark similar articles
https://www.poynter.org/news/rome-facebook-announces-new-strategies-combat-misinformation
(5) Architecture behind Alexa choosing skills https://goo.gl/dWmXZf
- Char-level RNN + Word-level RNN
- Shared encoder, but attention is personalized
(6) An overview of contemporary NLP techniques
https://medium.com/@ageitgey/natural-language-processing-is-fun-9a0bff37854e
(7) RNNs in particle physics?
https://indico.cern.ch/event/722319/contributions/3001310/attachments/1661268/2661638/IML-Sequence.pdf?utm_campaign=Revue%20newsletter&utm_medium=Newsletter&utm_source=NLP%20News
(8) Google cloud provides PyTorch images
https://twitter.com/i/web/status/1016515749517582338

NLP
(0) Use embeddings for positions - no brainer
https://twitter.com/i/web/status/1018789622103633921
(1) Chatbots were a hype train - lol
https://medium.com/swlh/chatbots-were-the-next-big-thing-what-happened-5fc49dd6fa61

The vast majority of bots are built using decision-tree logic, where the bot’s canned response relies on spotting specific keywords in the user input.

Interesting links
(0) Reasons to use OpenStreetMap
https://www.openstreetmap.org/user/jbelien/diary/44356
(1) Google deployes its internet ballons
https://goo.gl/d5cv6U
(2) Amazing problem solving
https://nevalalee.wordpress.com/2015/11/27/the-hotel-bathroom-puzzle/
(3) Nice flame thread about CS / ML is not science / just engineering etc
https://twitter.com/RandomlyWalking/status/1017899452378550273

#deep_learning
#data_science
#digest

My post on open images stage 1

For posterity
Please comment
https://spark-in.me/post/playing-with-google-open-images

#deep_learning
#data_science

New Keras version

https://github.com/keras-team/keras/releases/tag/2.2.1

No real major changes...

#deep_learning

The reality of human face recognition

There is a lot of hype related to the surveillance state / 1984 / Chinese offline cameras.

Cannot help but feature this amazing article from Russian engineers (RU):
https://habr.com/company/recognitor/blog/418127/

#deep_learning

Airbus ship detection challenge

On a surface this looks like a challenging and interesting competition:
- https://www.kaggle.com/c/airbus-ship-detection
- Train / test sets - 14G / 12G
- Downside - Kaggle and very fragile metric
- Upside - a separate significant price for fast algorithms!
- 768x768 images seem reasonable

#deep_learning
#data_science

Some interesting NLP related ideas from ACL 2018

http://ruder.io/acl-2018-highlights/

Overall
- bag-of-embeddings is surprisingly good at capturing sentence-level properties, among other results
- language models are bad at modelling numerals and propose several strategies to improve them
- current state-of-the-art models fail to capture many simple inferences
- LSTM representations, even though they have been trained on one task, are not task-specific. They are often predictive of unintended aspects such as demographics in the data
- Word embedding-based methods exhibit competitive or even superior performance

Four common ways to introduce linguistic information into models:
- Via a pipeline-based approach, where linguistic categories are used as features;
- Via data augmentation, where the data is augmented with linguistic categories;
- Via multi-task learning;

#nlp

2018 DS/ML digest 19

Market / data / libraries
(0) 32k lesions image dataset open-sourced
- https://goo.gl/CUQwnv
- https://nihcc.app.box.com/v/DeepLesion
(1) A new Distill article about Differentiable Image Parameterizations
- Usually images are parametrized as RGB values (normalized)
- Idea - use different (learnable) parametrization
- https://distill.pub/2018/differentiable-parameterizations/
- Parametrizing resulting image with fourier transform enables to use different architectures with style transfer https://distill.pub/2018/differentiable-parameterizations/#figure-style-transfer-diagram
- Working with transparent images
(2) Lip reading with 40% Word Error Rate https://arxiv.org/pdf/1807.05162.pdf
(3) Joing auto architecture + hyper param search https://arxiv.org/pdf/1807.06906.pdf (*)
(4) https://rl-navigation.github.io/deployable/
(5) New CNN architectures from ICML https://www.facebook.com/icml.imls/videos/429607650887089/%20 (*)
(6) Jupiter notebook widget for text annotaion https://github.com/natasha/ipyannotate
(7) A bit more debunking of auto-ml by fast.ai http://www.fast.ai/2018/07/23/auto-ml-3/
(8) A small intro to Bayes methods https://alexanderdyakonov.wordpress.com/2018/07/30/%d0%b1%d0%b0%d0%b9%d0%b5%d1%81%d0%be%d0%b2%d1%81%d0%ba%d0%b8%d0%b9-%d0%bf%d0%be%d0%b4%d1%85%d0%be%d0%b4/
(9) Criminal face recognition 20% false positives - https://www.nytimes.com/2018/07/26/technology/amazon-aclu-facial-recognition-congress.html?
(10) Denoising images wo noiseless ground-truth https://news.developer.nvidia.com/ai-can-now-fix-your-grainy-photos-by-only-looking-at-grainy-photos/?ncid=--45511

NLP
(0) Autoencoders for text https://habr.com/company/antiplagiat/blog/418173/ - no clear conclusion?
(1) RNN use cases overview https://indico.cern.ch/event/722319/contributions/3001310/attachments/1661268/2661638/IML-Sequence.pdf
(2) ACL 2018 notes http://ruder.io/acl-2018-highlights/

Hardware
(0) Edge embeddable TPU devices https://aiyprojects.withgoogle.com/edge-tpu ?
(1) GeForce 11* finally coming soon? Prices for 1080Ti are falling now...

#digest
#deep_learning

Autofocus for semseg?

http://arxiv.org/abs/1805.08403
I have not seen people for whom DeepLab worked...and in my tests dilated convolutions were the same...though some claim they help with high-res images with small objects...

Ideas:
(0) Autofocus layer, a novel module that enhances the multi-scale processing of CNNs by learning to select the ‘appropriate’ scale for identifying different objects in an image
(1) Layer description
https://pics.spark-in.me/upload/2f562fb9d12d76c36fa8777713de9716.jpg
(2) Implementation https://github.com/yaq007/Autofocus-Layer/blob/master/models.py

I believe this will work best for 3D images

#deep_learning