​​Cross-Lingual Ability of Multilingual BERT: An Empirical Study to #ICLR2020

In this work, the authors provide a comprehensive study of the contribution of different components in multilingual #BERT (M-BERT) to its cross-lingual ability.
They study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages – #Spanish, #Hindi, & #Russian – & using two conceptually different #NLP tasks, textual entailment & #NER.

Also, they construct a new corpus – Fake-English (#enfake), by shifting the Unicode of each character in English Wikipedia text by a large constant so that there is strictly no character overlap with any other Wikipedia text.
And, in this work, they consider Fake-English as a different language.

Among their key conclusions are the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it.

paper: https://arxiv.org/abs/1912.07840

#DS #ML #NN #ebooks

https://blog.paralleldots.com/data-science/24-best-and-free-books-to-understand-machine-learning/

In python 3.9, PEP-616 introduced str.removeprefix and str.removesuffix methods:

'abcd'.removeprefix('ab')
# 'cd'

'abcd'.removeprefix('fg')
# 'abcd'

The implementation is simple (it's implemented on C, of course, but the idea is the same):

def removeprefix(self: str, prefix: str) -> str:
if self.startswith(prefix):
return self[len(prefix):]
return self

​​Exploring Transfer Learning with T5: the Text-To-Text Transfer Transformer

tl;dr:
- 11 billion parameters
- encoder-decoder models generally outperformed “decoder-only” language models
- fill-in-the-blank-style denoising objectives worked best;
- the most important factor was the computational cost;
- training on in-domain data can be beneficial but that pre-training on smaller datasets can lead to detrimental overfitting;
- multitask learning could be close to competitive with a pre-train-then-fine-tune approach but requires carefully choosing how often the model is trained on each task

The model can be fine-tuned on smaller labeled datasets, often resulting in (far) better performance than training on the labeled data alone.
Present a large-scale empirical survey to determine which transfer learning techniques work best and apply these insights at scale to create a new model that we call the T5. Also, introduce a new open-source pre-training dataset, called the Colossal Clean Crawled Corpus (C4).

The T5 model, pre-trained on C4, achieves SOTA results on many NLP benchmarks while being flexible enough to be fine-tuned to a variety of important downstream tasks.


blog post: https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html
paper: https://arxiv.org/abs/1910.10683
github (with pre-trained models): https://github.com/google-research/text-to-text-transfer-transformer
colab notebook: https://colab.research.google.com/github/google-research/text-to-text-transfer-transformer/blob/master/notebooks/t5-trivia.ipynb

#nlp #transformer #t5

Interesting paper bout reproducibility in AI/ML from Dr. Edward Raff is a Chief Scientist at Booz Allen Hamilton. He analyzed 255 papers, and successfully reproduce 162 from them.

A 62% success rate is higher than many meta-analyses from other sciences, and I suspect my 62% number is lower than reality

Interesting facts:
1. Having fewer equations per page makes a paper more reproducible.
2. Empirical papers may be more reproducible than theory-oriented papers.
3. Sharing code is not a panacea
4. Having detailed pseudo code is just as reproducible as having no pseudo code.
5. Creating simplified example problems do not appear to help with reproducibility.
6: Please, check your email (papers of people who answer on emails is more reproducible)

https://thegradient.pub/independently-reproducible-machine-learning/

Leave a comment on how do you feel about this.

https://github.com/ignani/PythonMLDataScience

Have just uploaded some samples. Will keep adding as and when I get some free time. You can use it for learning.

Let me know what view. 😊

Nextstrain build for novel coronavirus (nCoV)

https://github.com/nextstrain/ncov

For all those who are looking for help on learning Python, or looking for practice projects to work on... check out this topic https://t.me/python_programmers_club/121484

Data structures are used to organize and store data efficiently in a computer so that they can be accessed and modified efficiently. Examples of data structures include arrays, linked lists, stacks, queues, trees, and graphs.

Algorithms are step-by-step procedures for solving a problem or performing a task. Algorithms can be implemented using data structures to improve their efficiency and performance. Examples of algorithms include sorting algorithms (such as quicksort and mergesort), search algorithms (such as binary search), and graph algorithms (such as Dijkstra's shortest path algorithm).

Both data structures and algorithms are fundamental concepts in computer science and are widely used in various applications, such as databases, operating systems, computer networks, and software engineering. The choice of data structure and algorithm can greatly impact the efficiency and performance of a software system.

In the Python programming language, there are several built-in data structures such as lists, tuples, sets, and dictionaries. Lists are ordered collections of elements and can be modified, while tuples are ordered, immutable collections. Sets are unordered collections of unique elements, and dictionaries are unordered collections of key-value pairs.

Python also has a rich set of libraries for implementing various algorithms, including NumPy and SciPy for numerical computing, and pandas for data analysis.

For sorting, the "sorted" function can be used, which returns a sorted list, while the "sort" method can be used to sort a list in place. For searching, the "in" operator can be used to check if an element is in a list or a dictionary, or the "index" method can be used to find the first occurrence of an element in a list.

In addition, Python has a large community of users and developers who have contributed many open-source packages that implement a wide variety of algorithms, making it easier for developers to incorporate these algorithms into their own projects.