How to show links?
anonymous poll

Full if short, hide if long – 29
👍👍👍👍👍👍👍 58%

Hide them behind markup – 11
👍👍👍 22%

Post full links – 9
👍👍 18%

Shorten them – 1
▫️ 2%

👥 50 people voted so far.

A great presentation about current state of particle tracking + ML

Also Kaggle failed to share this for some reason
https://indico.cern.ch/event/702054/attachments/1606643/2561698/tr180307_davidRousseau_CERN_trackML-FINAL.pdf

Key problem - current algorithm - Kalman filter faces time constaints

#data_science

Internet / tech

- Google I/O news https://goo.gl/1FszFA
- MS to give custom voice option to its apps - https://goo.gl/5e2oMw
- Katzenberg (former Disney executive) raises US$800m to make YouTube like short series - https://goo.gl/wLpTGi => Internet + Video is a commodity now?
- _Reportedly_ Lyft has 35% market share in the USA https://goo.gl/tvzDTu
- Google becoming evil and doing military contracts - https://goo.gl/3HjYDg - wtf?
- Apple autonomous drive fleet is _repotedly_ now at 55 https://goo.gl/94tfkk

#internet

Playing with 3D interactive scatter plots

Turns out you can do this using ipython widgets + ipyvolume.

Best example:
- Playing with particle data (https://nbviewer.jupyter.org/urls/gist.githubusercontent.com/maartenbreddels/04575b217aaf527d4417173f397253c7/raw/926a0e57403c0c65eb55bc52d5c7401dc1019fdf/trackml-ipyvolume.ipynb)

All of this looks kind of wobbly / new and a bit useless, but it works, is free and fast.

I was also trying to assing each point a colour like here

But in the end a much more simple approach just worked

fig = ipv.figure()

N = len(hits.volume_id.unique())
cmap = matplotlib.cm.get_cmap("tab20", N)
colors = cmap(np.linspace(0, 1.0, N))
colors = ["#%02x%02x%02x"  % tuple([int(k*255) for k in matplotlib.colors.to_rgb(color)[:3]]) for color in colors]

for i in range(0,N):
    hits_v = hits[hits.volume_id == list(hits.volume_id.unique())[i]]
    scatter = ipv.scatter(hits_v.x, hits_v.y, hits_v.z, marker="diamond", size=0.1, color=colors[i])

ipv.show()

Like this post or have something to say => tell us more in the comments or donate!

Using ncdu with exclude

A really good extension of standard du

sudo ncdu --exclude /exclude_folder /

Useful when something is mounted in /media or /mnt

#linux

A thorough and short guide to Matplotlib API

A bit of history, small look under the hood and logical explanation of how to use it best:

https://realpython.com/python-matplotlib-guide/

#data_science

2018 DS/ML digest 11

Cool thing this week
(0) ML vs. compute stidy since 2012 - chart / link

Market
(0) Once again about Google Duplex
(1) Google announcements from Google IO
-- Email autocomplete

  
We encode the subject and previous email by averaging the word embeddings in each field. We then join those averaged embeddings, and feed them to the target sequence RNN-LM at every decoding step, as the model diagram below shows.  

-- Learning Semantic Textual Similarity from Conversations blog, paper. Something in the lines of Sentence2Vec, but for conversations, self-supervised, uses attention and embedding averaging
-- Google Clips device + interesting moment estimation on the device. Looks like MobileNet distillation into a small network with some linear models on top

Libraries / tools / papers
(0) SaaS NLP annotation tool
(1) CNNs allegedly can reconstruct low light images? Blog, paper, Looks cool AF
(2) Cool thing to try in a new project - postgres restful API wrapper - such things require a lot of care though, but can elimininate a lot of useless work for small projects.

For my blog I had to write a simple business tier layer myself. I doubt that I could use this w/o overengineering because I constructed open-graph tags for example in SQL queries for example

Job / job market
(0) (RU) Realistic IT immigration story

Datasets
(0) Last week open images dataset was updated. I downloaded the small one for the sake of images. Though the download process itself is a bit murky

#machine-learning
#digest
#deep-learning

Like this post or have something to say => tell us more in the comments or donate!

Using groupby in pandas in multi-thread fashion

Sometimes you just need to use all of your CPUs to process some nasty thing in pandas (because you are lazy to do it properly) quick and dirty.

Pandas' GroupBy: Split, Apply, Combine seems to have been built exactly for that, but there is also a lazy workaround.

Solution I googled
- https://gist.github.com/tejaslodaya/562a8f71dc62264a04572770375f4bba

My lazy way using tqdm + Pool
- https://gist.github.com/snakers4/b246de548669543dc3b5dbb49d4c2f0c

(Savva, if you read this, I know that your version is better, you can also send it to me to share xD)

#ds

New competitions on Kaggle

Kaggle has started a new competition with video ... which is one of those competitions (read between the lines - blatant marketing)
https://www.kaggle.com/c/youtube8m-2018

I.e.
- TensorFlow Record files
- Each of the top 5 ranked teams will receive $5,000 per team as a travel award - no real prizes
- The complete frame-level features take about 1.53TB of space (and yes, these are not videos, but extracted CNN features)

So, they are indeed using their platform to promote their business interests.
Released free datasets are really cool, but only when you can use then for transfer learning, which implies also seeing the underlying ground level data (i.e. images of videos).

#data_science
#deep_learning

A couple of neat tricks in PyTorch to make code more compact and more useful for hyper-param tuning

You may have seen that today one can use CNNs even for tabular data.
In this case you may to resort to a lot of fiddling regarding model capacity and hyper-params.
It is kind of easy to do so in Keras, but doing this in PyTorch requires a bit more fiddling.

Here are a couple of patterns that may help with this:

(0) Clever use of nn.Sequential()

self.layers = nn.Sequential(*[
    ConvLayer(in_channels=channels,
              out_channels=channels,
              kernel_size=kernel_size,
              activation=activation,
              dropout=dropout)
    for _ in range(blocks)
])

(1) Clever use of lists (which is essentially the same as above)
Just this construction may save a lot of space and give a lot of flexibility

modules = [] 
modules.append(...) 
self.classifier = nn.Sequential(*modules) 

(2) Pushing as many hyper-params into flags for console scripts
You can even encode something like 1024_512_256 to be passed as list to your model constructor, i.e.
1024_512_256 => 1024,512,256 => an MLP with corresponding amount of neurons

(3) (Obvious) Using OOP where it makes sense

Example I recently used for one baseline

#deep_learning
Like this post or have something to say => tell us more in the comments or donate!

Dealing with class imbalance with CNNs

For small datasets / problems - oversampling works best, for large dataset - it's unclear
- http://arxiv.org/abs/1710.05381

Interestingly enough, they did not test oversampling + augmentations.

Transforms in PyTorch

The added a lot of useful stuff lately:
- https://pytorch.org/docs/master/torchvision/transforms.html

Basically this enables to build a decent pre-processing out-of box for simple tasks (just images)

I believe it will be much slower that OpenCV, but for small tasks it's ideal, if you do no look under the hood

#deep_learning

MobileNetv2
New light-weight architecture from Google with 72%+ top1

(0)
Performance https://goo.gl/2czk9t
Link http://arxiv.org/abs/1801.04381

Pre-trained implementation
- https://github.com/tonylins/pytorch-mobilenet-v2
- but this one took much more memory that I expected
- did not debug it

(1)
Gist - new light-weight architecture from Google with 72%+ top1 on Imagenet
Ofc Google promotes only its own papers there
No mention of SqueezeNet
This is somewhat disturbing

(2)
Novel ideas
- the shortcut connections are between the thin bottleneck layers
- the intermediate expansion layer uses lightweight depthwise convolutions
- it is important to remove non-linearities in the narrow layers in order to maintain representational power

(3)
Very novel idea - it is argued that non-linearities collapse some information.
When the dimensionality of useful information is low, you can do w/o them w/o loss of accuracy

(4) Building blocks
- Recent small networks' key features (except for SqueezeNet ones) - https://goo.gl/mQtrFM
- MobileNet building block explanation
- https://goo.gl/eVnWQL https://goo.gl/Gj8eQ5
- Overall architecture - https://goo.gl/RRhxdp

#deep_learning

https://github.com/Randl/MobileNetV2-pytorch
My implementation of MobileNetV2 - currently top low computational model - on PyTorch 0.4. RMSProp didn't work (I have feeling there are issues with it in PyTorch), so training is with SGD (scheme similar to ShuffleNet's one - reducing lr after 200 and 300 epochs). The results are a bit better that claimed in paper and achieved by other repos - 72.1% top1. Supports any scaling factor/input size (divisible by 32) as described in paper.

Some insights about why recent TF speech recognition challenge dataset was so poor in quality:
- https://petewarden.com/2018/05/28/why-you-need-to-improve-your-training-data-and-how-to-do-it/

Cool ideas
+ a cool idea - use last layer in CNN as an embedding in TB visualization + how to

#deep_learning