Top 8 trends from ICLR 2019

Overview of trends on #ICLR2019:
1. Inclusivity
2. Unsupervised representation learning & transfer learning
3. Retro ML
4. RNN is losing its luster with researchers
5. GANs are still going on strong
6. The lack of biologically inspired deep learning
7. Reinforcement learning is still the most popular topic by submissions
8. Most accepted papers will be quickly forgotten

Link: https://huyenchip.com/2019/05/12/top-8-trends-from-iclr-2019.html

#ICLR #overview

​​FreeLB: Enhanced Adversarial Training for Language Understanding

The authors propose a novel adversarial training algorithm – FreeLB, that promotes higher robustness and invariance in the embedding space, by adding adversarial perturbations to word embeddings and minimizing the resultant adversarial risk inside different regions around input samples, applied to Transformer-based models for NLU & commonsense reasoning tasks.

Experiments on the GLUE benchmark show that when applied only to the finetuning stage, it is able to improve the overall test scores:
* of BERT-based model from 78.3 -> 79.4
* RoBERTa-large model from 88.5 -> 88.8

The proposed approach achieves SOTA single-model test accuracies of 85.44% and 67.75% on ARC-Easy and ARC-Challenge.

paper: https://arxiv.org/abs/1909.11764

#nlp #nlu #bert #adversarial #ICLR

​​Top Trends of Graph Machine Learning in 2020

In this blogpost the author shares an overview of ICLR 2020 papers on Graph Machine Learning and highlights several trends:

1. More solid theoretical understanding of GNN:
* the dimension of the node embeddings should be proportional to the size of the graph if we want GNN being able to compute a solution to popular graph problems
* under certain conditions on the weights, GCNs cannot learn anything except node degrees and connected components when the number of layers grows
* a certain readout operation after neighborhood aggregation could help capture different types of node classification

2. New cool applications of GNN:
* a way to detect and fix bugs simultaneously in Javascript code
* inferring the types of variables for languages like Python or TypeScript
* reasoning in IQ-like tests (Raven Progressive Matrices (RPM) and Diagram Syllogism (DS)) with GNNs
* an RL algorithm to optimize the cost of TensorFlow computation graphs

3. Knowledge graphs become more popular:
* an idea to embed a query into a latent space not as a single point, but as a rectangular box
* a way to work with numerical entities and rules
* the re-evaluation of the existing models and how do they perform in a fair environment

4. New frameworks for graph embeddings:
* a way to improve running time and accuracy in node classification problem for any unsupervised embedding method
* a simple baseline that does not utilize a topology of the graph (i.e. it works on the aggregated node features) performs on par with the SOTA GNNs

blog post:
https://towardsdatascience.com/top-trends-of-graph-machine-learning-in-2020-1194175351a3

#ICLR #gnn #graphs

Survey of machine-learning experimental methods at NeurIPS2019 and ICLR2020

Good thread about what ML scientists do experiments on their papers


twitter: https://twitter.com/deliprao/status/1235697595919421440
report: https://hal.archives-ouvertes.fr/hal-02447823/document

#Survey #NeurIPS #ICLR #Experiments #ml