My foray into the STT Dark Forest

My tongue-in-cheek article on ML in general, and how to make your STT model train 3-4x faster with 4-5x less weights with the same quality

https://spark-in.me/post/stt-dark-forest

#data_science
#deep_learning
#stt

Poor man's ensembling techniques

So you want to improve your model's performance a bit.
Ensembling helps. But as is ... it's useful only on Kaggle competitions, where people stack over9000 networks trained on 100MB of data.

But for real life usage / production, there exist ensembling techniques, that do not require significant computation cost increase (!).
All of this is not mainstream yet, but it may work on you dataset!
Especially if your task is easy and the dataset is small.

- SWA (proven to work, usually used as a last stage when training a model);
- Lookahead optimizer (kind of new, not thoroughly tested);
- Multi-Sample Dropout (seems like a cheap ensemble, should work for classification);

Applicability will vary with your task.
Plain vanilla classification can use all of these, s2s networks probably only partially.

#data_science
#deep_learning

Playing with name NER

Premise

So, I needed to separate street names that are actual name + surname. Do not ask me why.
Yeah I know that maybe 70% of streets are human names more or less.
So you need 99% precision and at least 30-40% recall.
Or you can imagine a creepy soviet name like Трактор.

So, today making a NER parser is easy, take out our favourite framework (plan PyTorch ofc) of choice.
Even use FastText or something even less true. Add data and boom you have it.

The pain

But not so fast. Turns our there is a reason why cutting out proper names is a pain.
For Russian there is the natasha library, but since it works on YARGY, it has some assumptions about data structure.
I.e. names should be capitalized, come in pairs (name - surname), etc etc - I did not look their rules under the hood, but I would write it like this.

So probably this would be a name - Иван Иванов
But this probably would not ванечка иванофф

Is it bad?
Ofc no, it just assumes some stuff that may not hold for your dataset.
And yeah it works for streets just fine.

Also recognizing a proper name without context does not really work. And good luck finding (or generating) corpora for that.

Why deep learning may not work

So I downloaded some free databases with names (VK.com respects your secutity lol - the 100M leaked database is available, but useless, too much noise) and surnames.
Got 700k surnames of different origin, around 100-200k male and female names. Used just random words from CC + wiki + taiga for hard negative mining.
Got 92% accuracy on 4 classes (just word, female name, male name, surname) with some naive models.

... and it works .... kind of. If you give it 10M unique word forms, it can distinguish name-like stuff in 90% of cases.
But for addresses it is useless more or less and heuristics from natasha work much better.

The moral

- A tool that works on one case may be 90% useless on another;
- Heuristics have very high precision, low recall and are fragile;
- Neural networks are superior, but you should match your artifically created dataset to the real data (it may take a month to pull off properly);
- In any case, properly cracking both approaches may take time, but both heuristics and NNs are very fast to create, but sometimes 3 plain rules give you 100% precision with 10% recall and sometimes generating a fake dataset that matches your domain is a no-brainer. It depends.

#data_science
#nlp
#deep_learning

Easiest solutions to manage configs for ML models

When you have a lot of experiments, you need to minimize your code bulk and manage model configs concisely.
(This also kind of can be done via CLI parameters, but usually these things complement each other)

I know 3 ways:

(0) dicts + kwargs + dotdicts
(1) [attr](https://github.com/python-attrs/attrs)
(2) new python 3.7 [DataClass](https://docs.python.org/3/library/dataclasses.html) (which is very similar to attr)

Which one do you use?

#data_science

Streamlit vs. viola vs. panel vs. dash vs. bokeh server

TLDR - make scientific web-apps via python only w/o any web-programming (i.e. django, tornado).

Dash
- Mostly for BI
- Also a paid product
- Looks like the new Tableau
- Serving and out-of-the-box scaling options

Bokeh server
- Mostly plotting (very flexible, unlimited capabilities)
- High entry cost, though bokeh is kind of easy to use
- Also should scale well

Panel
- A bokeh server wrapper with a lot of capabilities for geo + templates

Streamlit
- The nicest looking app for interactive ML apps (maybe even annotation)
- Has pre-built styles and grid
- Limited only to its pre-built widgets
- Built on tornado with a very specific data model incompatible with the majority of available widgets
- Supposed to scale well - built on top of tornado

Viola
- If it runs in a notebook - it will run in viola
- Just turns a notebook into a server
- The app with the most promise for DS / ML
- Scales kind of meh - you need to run a jupyter kernel for each user - also takes some time to spin up a kernel
- Fully benefits from a rich ecosystem of jupyter / python / widgets
- In theory has customizable grid and CSS, but does not come pre-built with this => higher barrier to entry

Also most of these apps have no authentication buil-in.

More details:

- A nice summary here;
- A very detailed pros and cons summary of Streamlit + Viola. Also a very in-depth detailed discussion;
- Also awesome streamlit boilerplate is awesome;

#data_science

Collapsible Headings now in JupyterLab

- https://github.com/aquirdTurtle/Collapsible_Headings

The only thing that kept me from switching!
Please share your favourite plugins for JupyterLab!

#data_science

Using Viola With The Power of Vue.js

Remember the last post about python dashboard / demo solutions?

Since then we tried Viola and Streamlit.
Streamlit is very cool, but you cannot build really custom things with it. You should have 100% support of widgets that you need, and there are always issues with caching.
Also it is painful to change the default appearance.

Remember that viola "In theory has customizable grid and CSS"?
Of course someone took care of that!

Enter ipyvuetify + voila-vuetify.
TLDR - this allows you to have viola demos with Vue UI Library, all in 100% python.

Problems:

- No native table widget / method to show and UPDATE pandas tables. There are solutions that load Vue UI tables, but no updates out-of-the-box
- All plotting libraries will work mostly fine
- All jupiter widgets will work fine. But when you will need a custom widget - you will have to either code it, or find some hack with js-links or manual HTML manipulations
- Takes some time to load, will NOT scale to hundreds / thousands of concurrent users
- ipyvuetify is poorly documented, not very intuitive examples

Links:

- https://github.com/voila-dashboards/voila-vuetify
- https://github.com/mariobuikhuizen/ipyvuetify

#data_science

Some Proxy Related Tips and Hacks ... Quick Ez and in Style =)

DO is not cool anymore

First of all - let's get the Elephant out of the room. Remember I recommended Digital Ocean?
Looks like they behave like a f**g corporation now. They require you selfie with a passport now.
F**k this. Even AWS does not need this.


Why would you need proxies?

Scraping mostly. Circumventing anal restrictions.
Sometimes there are other legit use cases like proxying your tg api requests.


Which framework to use?

None.
One of our team tried scrapy, but there is too much hassle (imho) w/o any benefits.
(apart from their corporate platform crawlera, but I do not still understand why it exists, enterprise maybe)
Just use aiohttp, asyncio, bs4, requests, threading and multiprocessing.
And just write mostly good enough code.
If you do not need to scrape 100m pages per day or use selenium to scrape JS, this is more than enough.
Really. Do not buy-in into this cargo cult stuff.


Video

For video-content there are libraries:

- youtube-dl - lots of features, horrible python API, nice CLI, it really works
- pytube - was really cool and pythonic, but author abandoned it. Most likely he just wrote a ton of regexp that he decided not to support. Some methods still work though

Also remember that many HTTP libraries have HTTP / SOCK5 support.
If the libraries are old, this may be supported via env variables.


Where to get proxies?

The most interesting part.
There are "dedicated" services (e.g. smartproxy / luminati.io / proxymesh / bestproxy.ru / mobile proxy services).
And they probably are the only option when you scrape Amazon.
But if you need high bandwidth and many requests - such proxies usually have garbage speed.
(and morally - probably 50% of them are hacked routers)
Ofc there is scrapoxy.io/ - but this is just too much!


Enter Vultr

They have always been a DO look-alike serice.

I found a simple hacky way to get 10-20-40 proxies quickly.
You can just use Vultr + Ubuntu 18.04 docker image + write a plain startup script.
That is is. Literally.

With Docker already installed your script may looks something like this:

docker run -d --name socks5_1 -p 1080:1080 -e PROXY_USER=user -e PROXY_PASSWORD=password serjs/go-socks5-proxy && \
docker run -d --name socks5_2 -p 1081:1080 -e PROXY_USER=user -e PROXY_PASSWORD=password serjs/go-socks5-proxy

There are cheaper hosting alternatives - Vultr is quite expensive.
But his script feature + Docker images really save time.
But now they have the nice features - backups / snapshots / startup scripts / ez scaling / etc / etc w/o the corporate bs.
Also use my Give $100, Get $25 link!

Beware - new accounts may be limited to 5 servers per account.
You may want to create several accounts at first.

#data_science
#scraping

Finally Migrating to JupyterLab?

TLDR - there is no killer feature, most likely this is just future-proofing.

With these plugins (some of which even work with the latest version of JupyterLab) you can finally migrate:

- jupyterlab_filetree
- toc
- collapsible_headings

Extensions can be installed using jupyter labextension install. Depending on your conda installation, sometimes you can even install them in your JupyterLab UI.

Migration from Notebook

Just add this and replace how you run your notebook:

RUN conda install -c conda-forge jupyterlab && \
          conda install nodejs
...

CMD jupyter lab --port=8888 --ip=0.0.0.0 --no-browser

Obvious Downsides

- Looks like it is slower than notebooks (most annoying factor)
- No clear UX improvement, text editors are worse than IDEs, notebooks are the same
- Terminal is much less useful than standard Linux terminal or Putty
- With larger / more structured notebooks it crashes
- Most likely JupyterHub will continue to work with notebooks


I understand that pushing code to tested modules / having more smaller notebooks is preferable, but now when I have given this a test, most likely I will migrate only when forced to.


#data_science

Pandas Official Guide

Pandas now has a human readable best practices guide!

https://pandas.pydata.org/pandas-docs/stable/user_guide/

#data_science

Building Hyper Professional Looking PDFs in One Shell Command

You know, there are 2 types of people - those who value form over substance and substance over form.

I really like writing my documents in markdown and using CVS to store them, but many people do not understand this.

Enter Pandoc

You can build very professional-looking, whitepaper almost quality PDF documents with a single shell command using pandoc.

Its original template kind of sucks (do not also get me started on Latex and its witnesses) and shows its age. But I found a perfect solution - Eisvogel pandoc template.

It takes some fiddling with pandoc params, but in the end it is worth the effort.

- https://github.com/Wandmalfarbe/pandoc-latex-template
- https://pandoc.org/MANUAL.html

With this, you command may look like this:

pandoc \
meeting.md -o \
meeting.pdf \
--from markdown \
--template eisvogel \
--latex-engine=xelatex \
--highlight-style pygments

And viola, you have a perfect investment bank looking document.

Enjoy!

#data_science

Notebooks + Spreadsheets

Notebooks and spreadsheets (Excel or Google Sheets) have always been two most useful and helpful instruments I have ever used. Whole companies were built based on pseudo-relational Excel databases (this is ofc does not scale well).

Now there is a new library in python that integrates some JS tables library seamlessly with ipywidgets and notebooks. It is news and predictably sucks a little bit (as most of interactive tables in JS).

It goes without saying that it opens up a lot of possibilities for ML annotation - you can essentially combine tables and ipywidgets easily.

As far as I see It does not have an option to embed some HTML code, but recently there just appeared and Audio widget in ipywidgets (buried in the release notes somewhere)

So you can just use this to load audio into ipysheet:

wavb = open('test.wav', "rb").read()
audio = Audio(value=wavb,
              format='wav',
              autoplay=False)

#data_science

Excel in Notebooks

This notebook tool looks awesome.
It enables you to essentially replicate most useful excel functionality within notebook pandas dataframes:

- https://github.com/quantopian/qgrid

Was looking for something like this for a long time!

Similar tools I saw before required some fiddling (just search spreadsheets / excel on the channel), this one just works with an existing pandas dataframe! You can also hack in HTML elements and there are ample callbacks for your custom functionality.

#data_science

Compressed Feather in Pandas

A nifty feature in pandas I totally missed - saving not only .csv data frames compressed, but also .feather ones. Reduces files size 4-5x for repetitive data.

- Pandas to feather doc
- Pyarrow to feather doc

#data_science

Serialization of Standard Python Data Type in PyArrow

Some time ago, when I tried to save a pandas dataframe to feather or parquet and I had something like:

[1, 2, 3]

in some cells, pyarrow broke and refused to work.

Now it works at least with python lists and dicts.
It is very cool. Combined with compression='zstd' it is also fast and provides some really needed compression for text data.

The downside that is for example instead of list of lists you get a nested numpy array after reading the file back, but this is a small price to pay, right?

#data_science