Spark in me via @vote
Given the current situation ... which post / guide would you like next?
DS / ML related (back log of hobby projects)! – 47
👍👍👍👍👍👍👍 69%
OpenVPN + Docker – 12
👍👍 18%
Dante proxy + Arubacloud + DigitalOcean + Vultr + Docker – 9
👍 13%
👥 68 people voted so far.
DS / ML related (back log of hobby projects)! – 47
👍👍👍👍👍👍👍 69%
OpenVPN + Docker – 12
👍👍 18%
Dante proxy + Arubacloud + DigitalOcean + Vultr + Docker – 9
👍 13%
👥 68 people voted so far.
Useful Python abstractions / sugar / patterns
I already shared a book about patterns, which contains mostly high level / more complicated patters. But for writing ML code sometimes simple imperative function programming style is ok.
So - I will be posting about simple and really powerful python tips I am learning now.
This time I found out about
Map
#python
#data_science
I already shared a book about patterns, which contains mostly high level / more complicated patters. But for writing ML code sometimes simple imperative function programming style is ok.
So - I will be posting about simple and really powerful python tips I am learning now.
This time I found out about
map
and filter
, which are super useful for data preprocessing:Map
items = [1, 2, 3, 4, 5]Filter
squared = list(map(lambda x: x**2, items))
number_list = range(-5, 5)Also found this book - http://book.pythontips.com/en/latest/map_filter.html
less_than_zero = list(filter(lambda x: x < 0, number_list))
print(less_than_zero)
#python
#data_science
A note on CDNs and protecting your website against censorship
TLDR
- https://goo.gl/47UqyZ
- Using a free / cheap CDN service can enable you to protect your domain hosted resource from censorship
- Unless CDN servers will be blocked (but I guess the CDN has more servers, than you, right?)
So, I host spark-in.me on Digital Ocean. And I do not want to move or start a CDN by myself. I read news, that Google abandoned some of its proxying tools because of such censorship events...interesting.
I knew that services like Cloudflare (**CDN**) forward your traffic somehow, but I was not sure what IP is actually seen by the user and whether all of the traffic is forwarded. Then I read their FAQ
- https://goo.gl/uHPLjW
It says
So I tried their free-tier service (paid service starts from US$20-200, which is too steep) and it just works, though SSL certificates were issued ~90 mins after I changed my nameservers. It is as easy as:
- Backup your DNS settings somewhere
- Import to CloudFlare
- Change name servers in your domain registrar cabinet
- 90 mins and ... profit
Now I cannot see my direct DO server IP when I resolve my DNS:
#internet
#security
TLDR
- https://goo.gl/47UqyZ
- Using a free / cheap CDN service can enable you to protect your domain hosted resource from censorship
- Unless CDN servers will be blocked (but I guess the CDN has more servers, than you, right?)
So, I host spark-in.me on Digital Ocean. And I do not want to move or start a CDN by myself. I read news, that Google abandoned some of its proxying tools because of such censorship events...interesting.
I knew that services like Cloudflare (**CDN**) forward your traffic somehow, but I was not sure what IP is actually seen by the user and whether all of the traffic is forwarded. Then I read their FAQ
- https://goo.gl/uHPLjW
It says
After a visitor's browser has done the initial DNS lookup, it begins making requests to retrieve the actual content of a website. These requests are directed to the IP address that was returned from the DNS lookup. Before Cloudflare, that address would have been 198.51.100.1. With Cloudflare as the authoritative nameserver, the new address is 203.0.113.1. Cloudflare’s data center at 203.0.113.1 will serve as much of your website as it can from its local storage, and ask your web server at 198.51.100.1 for any part of your website it doesn’t already have stored locally. The Cloudflare data center at 203.0.113.1 will then provide your complete website to the visitor, so the visitor never talks directly to your web server at 198.51.100.1.
So I tried their free-tier service (paid service starts from US$20-200, which is too steep) and it just works, though SSL certificates were issued ~90 mins after I changed my nameservers. It is as easy as:
- Backup your DNS settings somewhere
- Import to CloudFlare
- Change name servers in your domain registrar cabinet
- 90 mins and ... profit
Now I cannot see my direct DO server IP when I resolve my DNS:
$ dig +short spark-in.me
104.27.142.65
104.27.143.65
#internet
#security
Forwarded from Админим с Буквой (bykva)
Немного о баш скриптовании
Порою возникает необходимость записать какие-то данные в фай с сохранением переноса строк и отступов. сделать это можно несколькими способами.
1) с помощью echo
Такой способ имеет только единственный плюс - он однострочник. В реальности он впринципе не читабельный.
Разбить этот однострочник можно, конечно, на несколько строк:
И такой способ впринципе хорош до тех пор, пока вам не придется экранировать кавычки в тексте.
2) с помощью cat
В отличии от первого способа переменные по-прежнему интерпретируются, а кавычки экранировать не нужно.
#bash_tips_and_tricks
Порою возникает необходимость записать какие-то данные в фай с сохранением переноса строк и отступов. сделать это можно несколькими способами.
1) с помощью echo
echo -e 'This is first string\nAnd this is second' > /path/to/file
Такой способ имеет только единственный плюс - он однострочник. В реальности он впринципе не читабельный.
Разбить этот однострочник можно, конечно, на несколько строк:
echo "
str1
$variable - why not?
str N
"
И такой способ впринципе хорош до тех пор, пока вам не придется экранировать кавычки в тексте.
2) с помощью cat
cat > ceph.conf « EOF
[global]
mon_host = xx.xx.xx.xx:6789
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
EOF
В отличии от первого способа переменные по-прежнему интерпретируются, а кавычки экранировать не нужно.
#bash_tips_and_tricks
For windows users, that use their legacy machine as thin client to access Linux servers
Old habits die slowly. I use and old, but powerful Windows machine and so far doing everything on remote servers was ok, until I needed to commit to
But my key is stored locally and I do not want to use git bash or any windows based software, because it sucks. Also having a single source of truth on a remote Linux machine is better anyway. But I cannot store my key on the remote machine.
There is a solution -
- Install pageant, add your identity locally (.ppk private key file)
- Check
- Follow the below guides to check that all works
-
https://www.digitalocean.com/community/tutorials/how-to-use-pageant-to-streamline-ssh-key-authentication-with-putty
https://developer.github.com/v3/guides/using-ssh-agent-forwarding/
#linux
Old habits die slowly. I use and old, but powerful Windows machine and so far doing everything on remote servers was ok, until I needed to commit to
github
using ssh agent forwarding
.But my key is stored locally and I do not want to use git bash or any windows based software, because it sucks. Also having a single source of truth on a remote Linux machine is better anyway. But I cannot store my key on the remote machine.
There is a solution -
ssh-agent forwarding
. In a nutshell:- Install pageant, add your identity locally (.ppk private key file)
- Check
allow agent forwarding
in Putty- Follow the below guides to check that all works
-
profit
https://www.digitalocean.com/community/tutorials/how-to-use-pageant-to-streamline-ssh-key-authentication-with-putty
https://developer.github.com/v3/guides/using-ssh-agent-forwarding/
#linux
Digitalocean
How To Use Pageant to Streamline SSH Key Authentication with PuTTY | DigitalOcean
Pageant is a PuTTY authentication agent. It holds your private keys in memory so that you can use them whenever you are connecting to a server. It eliminates…
Dmytro's top4 solution on DS Bowl
https://www.kaggle.com/c/data-science-bowl-2018/discussion/55118
#deep_learning
#data_science
https://www.kaggle.com/c/data-science-bowl-2018/discussion/55118
#deep_learning
#data_science
SOCK5 proxy configuration on Vultr
As of now, Vult was not (yet) blocked, probably because it is less known in the CIS. If you missed our Digital Ocean sock5 configuration guide, then you can follow this guide.
For us, both DO and Vulture work as of now.
- https://spark-in.me/post/vultr-sock5-proxy-server
You can use our referral links to create accounts
- https://m.do.co/c/6f8e77dddc23
- https://www.vultr.com/?ref=7402755
If you like the above guides, consider buying us a coffee
- https://www.buymeacoffee.com/8oneCIN
#linux
#digital_freedom
As of now, Vult was not (yet) blocked, probably because it is less known in the CIS. If you missed our Digital Ocean sock5 configuration guide, then you can follow this guide.
For us, both DO and Vulture work as of now.
- https://spark-in.me/post/vultr-sock5-proxy-server
You can use our referral links to create accounts
- https://m.do.co/c/6f8e77dddc23
- https://www.vultr.com/?ref=7402755
If you like the above guides, consider buying us a coffee
- https://www.buymeacoffee.com/8oneCIN
#linux
#digital_freedom
Spark in me
Playing with a simple SOCKS5 proxy server on Vultr and Ubuntu 16.04
Start your own proxy server
Статьи автора - http://spark-in.me/author/yara_tchk
Блог - http://spark-in.me
Статьи автора - http://spark-in.me/author/yara_tchk
Блог - http://spark-in.me
Readable list comprehensions in Python
My list and dictionary comprehensions usually look like s**t
https://gist.github.com/IaroslavR/7dcb54830242a22de1869f6fd05a8d7e
#python
My list and dictionary comprehensions usually look like s**t
https://gist.github.com/IaroslavR/7dcb54830242a22de1869f6fd05a8d7e
#python
Gist
Examples of readable comprehension formatting from SO
Examples of readable comprehension formatting from SO - examples.py
DWT article on habrahabr.ru
DS Bowl article is live on habrahabr.ru
- https://habrahabr.ru/post/354040/
Please support us with your likes.
#data_science
DS Bowl article is live on habrahabr.ru
- https://habrahabr.ru/post/354040/
Please support us with your likes.
#data_science
Хабр
Применяем Deep Watershed Transform в соревновании Kaggle Data Science Bowl 2018
Применяем Deep Watershed Transform в соревновании Kaggle Data Science Bowl 2018 Представляем вам перевод статьи по ссылке и оригинальный докеризированный код. Да...
Predictive models running on clientside?
Looks like soon you will have not only to use AdBlock but also restrict WebGL usage
https://youtu.be/Nc8kZABv-KE
#deep_learning
Looks like soon you will have not only to use AdBlock but also restrict WebGL usage
https://youtu.be/Nc8kZABv-KE
#deep_learning
YouTube
Tensorflow.js Explained
Tensorflow.js is Google's new Javascript verison of its popular Machine Learning library Tensorflow. This allows developers, hobbyists, and researchers to build & train AI models in the browser! It allows for both training and inference to happen entirely…
Using a subset of GPUs without restarting the docker container
If you have multiple
Profit.
#deep_learning
If you have multiple
CUDA
GPUs visible within your container, but you do not want to use them all (and PyTorch API is wobbly there now), then the docs advise you toimport os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="2,3"
Profit.
#deep_learning
Stupid errors
Found out why my model on DS Bowl generalized poorly.
I forgot to re-create and instance of optimizer after unfreezing the encoder.
https://github.com/snakers4/ds_bowl_2018/commit/1d8c0a22a72cbca18c7eccc661c9e9bf344cff26
#deep_learning
Found out why my model on DS Bowl generalized poorly.
I forgot to re-create and instance of optimizer after unfreezing the encoder.
optimizer = torch.optim.Adam(filter(lambda p: p.requires_grad, model.parameters()),
# Only finetunable params
lr=args.lr)
https://github.com/snakers4/ds_bowl_2018/commit/1d8c0a22a72cbca18c7eccc661c9e9bf344cff26
#deep_learning
GitHub
Add optimizer re-creation on model encoder unfreeze · snakers4/ds_bowl_2018@1d8c0a2
ds_bowl_2018 - Kaggle Data Science Bowl 2018
Internet digest
(0) Ben Evans - https://goo.gl/72b4pm
ML / industry
(1) FB to design its own FPGAs / ML chips - https://goo.gl/nh2Wph ?
(2) Google willing to replicate iMessage, again https://goo.gl/MtwCet
-- No mention of Telegram - but all Google's attempts are aeons behind Telegram
-- Google willing to go the hardest route - a standard enforced on the carrier + replace the messenging app
-- All of the previous attempts kind of did not work
(3) Facebook media backlash - https://goo.gl/rKd9E5
(4) Who makes LIDARs - https://goo.gl/uD5qc5
(5) Tesla over automation - https://goo.gl/1WBMj3
Telecom
(1) British Telecom to switch to VOIP - https://goo.gl/MCbZgq
(2) Flickr purchased - https://goo.gl/AMcE6f
#internet
(0) Ben Evans - https://goo.gl/72b4pm
ML / industry
(1) FB to design its own FPGAs / ML chips - https://goo.gl/nh2Wph ?
(2) Google willing to replicate iMessage, again https://goo.gl/MtwCet
-- No mention of Telegram - but all Google's attempts are aeons behind Telegram
-- Google willing to go the hardest route - a standard enforced on the carrier + replace the messenging app
-- All of the previous attempts kind of did not work
(3) Facebook media backlash - https://goo.gl/rKd9E5
(4) Who makes LIDARs - https://goo.gl/uD5qc5
(5) Tesla over automation - https://goo.gl/1WBMj3
Telecom
(1) British Telecom to switch to VOIP - https://goo.gl/MCbZgq
(2) Flickr purchased - https://goo.gl/AMcE6f
#internet
TechCrunch
Facebook has a new job posting calling for chip designers
Facebook has posted a job opening looking for an expert in ASIC and FPGA, two custom silicon designs that companies can gear toward specific use cases — particularly in machine learning and artific…
PyTorch 0.4 released
https://github.com/pytorch/pytorch/releases/tag/v0.4.0
Key
(1) Tensor / Variable merged
(2) Zero-dimensional Tensors
(3) dtypes
(4) migration guide http://pytorch.org/2018/04/22/0_4_0-migration-guide.html
#pytorch
https://github.com/pytorch/pytorch/releases/tag/v0.4.0
Key
(1) Tensor / Variable merged
(2) Zero-dimensional Tensors
(3) dtypes
(4) migration guide http://pytorch.org/2018/04/22/0_4_0-migration-guide.html
#pytorch
GitHub
Release Trade-off memory for compute, Windows support, 24 distributions with cdf, variance etc., dtypes, zero-dimensional Tensors…
PyTorch 0.4.0 release notes
Table of Contents
Major Core Changes
Tensor / Variable merged
Zero-dimensional Tensors
dtypes
migration guide
New Features
Tensors
Full support for advanced indexi...
Table of Contents
Major Core Changes
Tensor / Variable merged
Zero-dimensional Tensors
dtypes
migration guide
New Features
Tensors
Full support for advanced indexi...
On the surface looks like an interesting competition
Well, I said that about Power Laws - but then it turned out otherwise.
So far I can see CV, NLP and tables in one mix.
https://www.kaggle.com/c/avito-demand-prediction/
#data_science
Well, I said that about Power Laws - but then it turned out otherwise.
So far I can see CV, NLP and tables in one mix.
https://www.kaggle.com/c/avito-demand-prediction/
#data_science
Kaggle
Avito Demand Prediction Challenge
Predict demand for an online classified ad
Forwarded from Админим с Буквой (bykva)
Релиз дистрибутива Ubuntu 18.04 LTS
Состоялся релиз дистрибутива Ubuntu 18.04 "Bionic Beaver", который отнесён к категории выпусков с длительным сроком поддержки (LTS), обновления для которых формируются в течение 5 лет. Установочные образы созданы для Ubuntu Desktop, Ubuntu Server, Ubuntu Cloud, Kubuntu, Ubuntu Budgie, Lubuntu, Ubuntu Studio, Ubuntu Kylin, Ubuntu MATE и Xubuntu.
Состоялся релиз дистрибутива Ubuntu 18.04 "Bionic Beaver", который отнесён к категории выпусков с длительным сроком поддержки (LTS), обновления для которых формируются в течение 5 лет. Установочные образы созданы для Ubuntu Desktop, Ubuntu Server, Ubuntu Cloud, Kubuntu, Ubuntu Budgie, Lubuntu, Ubuntu Studio, Ubuntu Kylin, Ubuntu MATE и Xubuntu.