Running mean programming pattern in PyTorch

Sometimes you just need to apply exponential weighting:
(0) When tracking some metric
(1) When weighting a loss
(2) When applying something inspired by Adam

I used to do it in a quite an ugly way:
(0) Feed a list => calculate averages
(1) Do the same, but using a separate class

Found out from my colleague that it can be done in PyTorch using torch.nn.register_buffer
Very cool

Like this post or have something to say => tell us more in the comments or donate!

Google Duplex

In a nutshell - a combination of ML (RNN + speech recognition + WaveNet + Tacotron) that can call a human and pretend to be a human. Only works for narrow specific domains (call to a restaurant).

Links:
(0) Blog post
(1) MKBHD video

Usually I include such links into digests, but this time it looks like insanity:
(0) It looks heavily doctored, but believable
(1) All the components are kind of known to be at least 90-95% as good as presented
(2) Once again - it is very domain focused

One of the key research insights was to constrain Duplex to closed domains, which are narrow enough to explore extensively.
Duplex can only carry out natural conversations after being deeply trained in such domains.
It cannot carry out general conversations.

Like this post or have something to say => tell us more in the comments or donate!

2018 DS/ML digest 10

Market
(0) Some moonshots by Google in working with electronic health records
(1) Google duplex - a narrow domain bot that makes calls for you
(2) Nature wants to make its ML journal ... paid
(3) Standford DawnBench - training Imagenet encoders as quickly and cheaply as possible
(4) Facebook achieves 85% on Imagenet by training on 1bn images in 336 GPUs in a week
(5) Learning the models of the surrounding world based on a DOOM like game

Practice / libraries / code
(0) A smarter and new way to ensemble CNNs
- Traditional approach - ensemble CNNS with different architecture - and just vote / average / apply linear regression on top
- Newer approach - use Cyclic Learning rate
- Even newer approach - model snapshot ensembling
- Stochastic Weight Averaging
-- store running average of the models
-- train one model with CLR
-- at the end of each lr update (or epoch) - do a running average of the models with some weights
-- the gist of the method is located on this line
-- I do understand why the update bnorm params, but I do not understand why it cannot be done just running 1 train epoch
- Papers on CNN ensembling 1 2 3
(1) (RU) Small amount of technocal details, but face-detection + face hashing works in retail (+human operator) given an HD camera
(2) (RU) Pose estimation
(3) Numpy autograd

"New" papers worth mentioning
(0) SqueezeNext
- Module comparsion
- Key changes
(i) more aggressive channel reduction by incorporating a two-stage squeeze module
(ii separable 3 × 3 convolutions
(iii) element-wise addition skip co

nection similar to ResNet
- Performance
(1) GANs to generate full-body anime characters in different poses

Visualizations:
(0) (does not work in Firefox) Visualizing encoder-decoder networks for translation

#data-science
#deep-learning
#digest

Like this post or have something to say => tell us more in the comments or donate!

How to show links?
anonymous poll

Full if short, hide if long – 29
👍👍👍👍👍👍👍 58%

Hide them behind markup – 11
👍👍👍 22%

Post full links – 9
👍👍 18%

Shorten them – 1
▫️ 2%

👥 50 people voted so far.

A great presentation about current state of particle tracking + ML

Also Kaggle failed to share this for some reason
https://indico.cern.ch/event/702054/attachments/1606643/2561698/tr180307_davidRousseau_CERN_trackML-FINAL.pdf

Key problem - current algorithm - Kalman filter faces time constaints

#data_science

Internet / tech

- Google I/O news https://goo.gl/1FszFA
- MS to give custom voice option to its apps - https://goo.gl/5e2oMw
- Katzenberg (former Disney executive) raises US$800m to make YouTube like short series - https://goo.gl/wLpTGi => Internet + Video is a commodity now?
- _Reportedly_ Lyft has 35% market share in the USA https://goo.gl/tvzDTu
- Google becoming evil and doing military contracts - https://goo.gl/3HjYDg - wtf?
- Apple autonomous drive fleet is _repotedly_ now at 55 https://goo.gl/94tfkk

#internet

Playing with 3D interactive scatter plots

Turns out you can do this using ipython widgets + ipyvolume.

Best example:
- Playing with particle data (https://nbviewer.jupyter.org/urls/gist.githubusercontent.com/maartenbreddels/04575b217aaf527d4417173f397253c7/raw/926a0e57403c0c65eb55bc52d5c7401dc1019fdf/trackml-ipyvolume.ipynb)

All of this looks kind of wobbly / new and a bit useless, but it works, is free and fast.

I was also trying to assing each point a colour like here

But in the end a much more simple approach just worked

fig = ipv.figure()

N = len(hits.volume_id.unique())
cmap = matplotlib.cm.get_cmap("tab20", N)
colors = cmap(np.linspace(0, 1.0, N))
colors = ["#%02x%02x%02x"  % tuple([int(k*255) for k in matplotlib.colors.to_rgb(color)[:3]]) for color in colors]

for i in range(0,N):
    hits_v = hits[hits.volume_id == list(hits.volume_id.unique())[i]]
    scatter = ipv.scatter(hits_v.x, hits_v.y, hits_v.z, marker="diamond", size=0.1, color=colors[i])

ipv.show()

Like this post or have something to say => tell us more in the comments or donate!

Using ncdu with exclude

A really good extension of standard du

sudo ncdu --exclude /exclude_folder /

Useful when something is mounted in /media or /mnt

#linux

A thorough and short guide to Matplotlib API

A bit of history, small look under the hood and logical explanation of how to use it best:

https://realpython.com/python-matplotlib-guide/

#data_science

2018 DS/ML digest 11

Cool thing this week
(0) ML vs. compute stidy since 2012 - chart / link

Market
(0) Once again about Google Duplex
(1) Google announcements from Google IO
-- Email autocomplete

  
We encode the subject and previous email by averaging the word embeddings in each field. We then join those averaged embeddings, and feed them to the target sequence RNN-LM at every decoding step, as the model diagram below shows.  

-- Learning Semantic Textual Similarity from Conversations blog, paper. Something in the lines of Sentence2Vec, but for conversations, self-supervised, uses attention and embedding averaging
-- Google Clips device + interesting moment estimation on the device. Looks like MobileNet distillation into a small network with some linear models on top

Libraries / tools / papers
(0) SaaS NLP annotation tool
(1) CNNs allegedly can reconstruct low light images? Blog, paper, Looks cool AF
(2) Cool thing to try in a new project - postgres restful API wrapper - such things require a lot of care though, but can elimininate a lot of useless work for small projects.

For my blog I had to write a simple business tier layer myself. I doubt that I could use this w/o overengineering because I constructed open-graph tags for example in SQL queries for example

Job / job market
(0) (RU) Realistic IT immigration story

Datasets
(0) Last week open images dataset was updated. I downloaded the small one for the sake of images. Though the download process itself is a bit murky

#machine-learning
#digest
#deep-learning

Like this post or have something to say => tell us more in the comments or donate!

Using groupby in pandas in multi-thread fashion

Sometimes you just need to use all of your CPUs to process some nasty thing in pandas (because you are lazy to do it properly) quick and dirty.

Pandas' GroupBy: Split, Apply, Combine seems to have been built exactly for that, but there is also a lazy workaround.

Solution I googled
- https://gist.github.com/tejaslodaya/562a8f71dc62264a04572770375f4bba

My lazy way using tqdm + Pool
- https://gist.github.com/snakers4/b246de548669543dc3b5dbb49d4c2f0c

(Savva, if you read this, I know that your version is better, you can also send it to me to share xD)

#ds

New competitions on Kaggle

Kaggle has started a new competition with video ... which is one of those competitions (read between the lines - blatant marketing)
https://www.kaggle.com/c/youtube8m-2018

I.e.
- TensorFlow Record files
- Each of the top 5 ranked teams will receive $5,000 per team as a travel award - no real prizes
- The complete frame-level features take about 1.53TB of space (and yes, these are not videos, but extracted CNN features)

So, they are indeed using their platform to promote their business interests.
Released free datasets are really cool, but only when you can use then for transfer learning, which implies also seeing the underlying ground level data (i.e. images of videos).

#data_science
#deep_learning

A couple of neat tricks in PyTorch to make code more compact and more useful for hyper-param tuning

You may have seen that today one can use CNNs even for tabular data.
In this case you may to resort to a lot of fiddling regarding model capacity and hyper-params.
It is kind of easy to do so in Keras, but doing this in PyTorch requires a bit more fiddling.

Here are a couple of patterns that may help with this:

(0) Clever use of nn.Sequential()

self.layers = nn.Sequential(*[
    ConvLayer(in_channels=channels,
              out_channels=channels,
              kernel_size=kernel_size,
              activation=activation,
              dropout=dropout)
    for _ in range(blocks)
])

(1) Clever use of lists (which is essentially the same as above)
Just this construction may save a lot of space and give a lot of flexibility

modules = [] 
modules.append(...) 
self.classifier = nn.Sequential(*modules) 

(2) Pushing as many hyper-params into flags for console scripts
You can even encode something like 1024_512_256 to be passed as list to your model constructor, i.e.
1024_512_256 => 1024,512,256 => an MLP with corresponding amount of neurons

(3) (Obvious) Using OOP where it makes sense

Example I recently used for one baseline

#deep_learning
Like this post or have something to say => tell us more in the comments or donate!

Dealing with class imbalance with CNNs

For small datasets / problems - oversampling works best, for large dataset - it's unclear
- http://arxiv.org/abs/1710.05381

Interestingly enough, they did not test oversampling + augmentations.

Transforms in PyTorch

The added a lot of useful stuff lately:
- https://pytorch.org/docs/master/torchvision/transforms.html

Basically this enables to build a decent pre-processing out-of box for simple tasks (just images)

I believe it will be much slower that OpenCV, but for small tasks it's ideal, if you do no look under the hood

#deep_learning