#academia
This is not only Julia for biologists. It is for everyone who is not using Julia.
Roesch, Elisabeth, Joe G. Greener, Adam L. MacLean, Huda Nassar, Christopher Rackauckas, Timothy E. Holy, and Michael P. H. Stumpf. 2021. “Julia for Biologists.” ArXiv [q-Bio.QM]. arXiv. http://arxiv.org/abs/2109.09973.
This is not only Julia for biologists. It is for everyone who is not using Julia.
Roesch, Elisabeth, Joe G. Greener, Adam L. MacLean, Huda Nassar, Christopher Rackauckas, Timothy E. Holy, and Michael P. H. Stumpf. 2021. “Julia for Biologists.” ArXiv [q-Bio.QM]. arXiv. http://arxiv.org/abs/2109.09973.
#visualization #art #fun
More like a blog post…
But the visualisation is cool. I posted it as a comment.
[2109.15079] Asimov's Foundation -- turning a data story into an NFT artwork
https://arxiv.org/abs/2109.15079
More like a blog post…
But the visualisation is cool. I posted it as a comment.
[2109.15079] Asimov's Foundation -- turning a data story into an NFT artwork
https://arxiv.org/abs/2109.15079
#ML
Duan T, Avati A, Ding DY, Thai KK, Basu S, Ng AY, et al. NGBoost: Natural Gradient Boosting for probabilistic prediction. arXiv [cs.LG]. 2019. Available: http://arxiv.org/abs/1910.03225
(I had it on my reading list for a long time. However, I didn't read it until today because the title and abstract are not attractive at all.)
But this is a good paper. It goes deep to dig out the fundamental reasons why some methods work and others don't.
When inferring probability distributions, it is straightforward to come up with methods with parametrized distributions (statistical manifolds). Then, by tuning the parameters, we adjust the distribution to fit our dataset the best.
The problem is the choice of the objective function and optimization methods. This paper mentioned a most generic objective function and a framework to optimize the model along the natural gradient instead of just the gradient w.r.t. the parameters.
Different parametrizations of the objective is like coordinate transformations and chain rule only works if the transformations are in a "flat" space but such "flat" space is not necessarily a good choice for a high dimensional problem. For a space that is approximately flat in a small region, we can define distance like what we do in differential geometry[^1]. Meanwhile, just like "covariant derivatives" in differential geometry, some kind of covariant derivative can be found on statistical manifolds and they are called "natural derivatives".
Descending in the direction of natural derivatives is navigating the landscape more efficiently.
[^1]: This a Riemannian space
Duan T, Avati A, Ding DY, Thai KK, Basu S, Ng AY, et al. NGBoost: Natural Gradient Boosting for probabilistic prediction. arXiv [cs.LG]. 2019. Available: http://arxiv.org/abs/1910.03225
(I had it on my reading list for a long time. However, I didn't read it until today because the title and abstract are not attractive at all.)
But this is a good paper. It goes deep to dig out the fundamental reasons why some methods work and others don't.
When inferring probability distributions, it is straightforward to come up with methods with parametrized distributions (statistical manifolds). Then, by tuning the parameters, we adjust the distribution to fit our dataset the best.
The problem is the choice of the objective function and optimization methods. This paper mentioned a most generic objective function and a framework to optimize the model along the natural gradient instead of just the gradient w.r.t. the parameters.
Different parametrizations of the objective is like coordinate transformations and chain rule only works if the transformations are in a "flat" space but such "flat" space is not necessarily a good choice for a high dimensional problem. For a space that is approximately flat in a small region, we can define distance like what we do in differential geometry[^1]. Meanwhile, just like "covariant derivatives" in differential geometry, some kind of covariant derivative can be found on statistical manifolds and they are called "natural derivatives".
Descending in the direction of natural derivatives is navigating the landscape more efficiently.
[^1]: This a Riemannian space
#visualization
"Fail"
When visualizing data, the units being used have to be specified for any values shown.
But the style of the charts is attractive. :)
By chungischef
Available at:
https://www.reddit.com/r/dataisbeautiful/comments/q958if/recreation_of_a_classic_population_density_map/
"Fail"
When visualizing data, the units being used have to be specified for any values shown.
But the style of the charts is attractive. :)
By chungischef
Available at:
https://www.reddit.com/r/dataisbeautiful/comments/q958if/recreation_of_a_classic_population_density_map/
#ML
(I am experimenting with a new platform. This post is also available at: https://community.kausalflow.com/c/ml-journal-club/how-do-neural-network-generalize )
There are somethings that are quite hard to understand in deep neural networks. One of them is how the network generalizes.
[Zhang2016] shows some experiments about the amazing ability of neural networks to learn even completely random datasets. But they can not generalize as the data is random. How to understand generalization? The authors mentioned some theories like VC dimension, Rademacher complexity, and uniform stability. But none of them is good enough.
Recently, I found the work of Simon et al [Simon2021]. The authors also wrote a blog about this paper [Simon2021Blog].
The idea is to simplify the problem of generalization by looking at how a neural network approximates a function f. This is approximate vectors in Hilbert space. Thus we are looking at the similarity of the vectors f, and its neural network approximation f'. The similarity of these two vectors is related to the eigenvalues of the so-called “neural tangent kernel” (NTK).
Using NTK, they derived an amazingly simple quantity, learnability, which can measure how Hilbert space vectors align with each other, that is, how good the approximation using the neural network is.
[Zhang2016]: Zhang C, Bengio S, Hardt M, Recht B, Vinyals O. Understanding deep learning requires rethinking generalization. arXiv [cs.LG]. 2016. Available: http://arxiv.org/abs/1611.03530
[Simon2021Blog]: Simon J. A First-Principles Theory of NeuralNetwork Generalization. In: The Berkeley Artificial Intelligence Research Blog [Internet]. [cited 26 Oct 2021]. Available: https://bair.berkeley.edu/blog/2021/10/25/eigenlearning/
[Simon2021]: Simon JB, Dickens M, DeWeese MR. Neural Tangent Kernel Eigenvalues Accurately Predict Generalization. arXiv [cs.LG]. 2021. Available: http://arxiv.org/abs/2110.03922
(I am experimenting with a new platform. This post is also available at: https://community.kausalflow.com/c/ml-journal-club/how-do-neural-network-generalize )
There are somethings that are quite hard to understand in deep neural networks. One of them is how the network generalizes.
[Zhang2016] shows some experiments about the amazing ability of neural networks to learn even completely random datasets. But they can not generalize as the data is random. How to understand generalization? The authors mentioned some theories like VC dimension, Rademacher complexity, and uniform stability. But none of them is good enough.
Recently, I found the work of Simon et al [Simon2021]. The authors also wrote a blog about this paper [Simon2021Blog].
The idea is to simplify the problem of generalization by looking at how a neural network approximates a function f. This is approximate vectors in Hilbert space. Thus we are looking at the similarity of the vectors f, and its neural network approximation f'. The similarity of these two vectors is related to the eigenvalues of the so-called “neural tangent kernel” (NTK).
Using NTK, they derived an amazingly simple quantity, learnability, which can measure how Hilbert space vectors align with each other, that is, how good the approximation using the neural network is.
[Zhang2016]: Zhang C, Bengio S, Hardt M, Recht B, Vinyals O. Understanding deep learning requires rethinking generalization. arXiv [cs.LG]. 2016. Available: http://arxiv.org/abs/1611.03530
[Simon2021Blog]: Simon J. A First-Principles Theory of NeuralNetwork Generalization. In: The Berkeley Artificial Intelligence Research Blog [Internet]. [cited 26 Oct 2021]. Available: https://bair.berkeley.edu/blog/2021/10/25/eigenlearning/
[Simon2021]: Simon JB, Dickens M, DeWeese MR. Neural Tangent Kernel Eigenvalues Accurately Predict Generalization. arXiv [cs.LG]. 2021. Available: http://arxiv.org/abs/2110.03922
Kausalflow
How do neural networks generalize? | kausalflow
There are somethings that are quite hard to understand in deep neural networks. One of them is how the network generalizes.
[Zhang2016] shows some experiments about the amazing ability to learn even completely random datasets but can not generalize as...
[Zhang2016] shows some experiments about the amazing ability to learn even completely random datasets but can not generalize as...
#ML
https://www.microsoft.com/en-us/research/blog/turing-bletchley-a-universal-image-language-representation-model-by-microsoft/
https://www.microsoft.com/en-us/research/blog/turing-bletchley-a-universal-image-language-representation-model-by-microsoft/
Microsoft Research
Turing Bletchley: A Universal Image Language Representation model by Microsoft - Microsoft Research
Today, the Microsoft Turing team (opens in new tab) is thrilled to introduce Turing Bletchley, a 2.5-billion parameter Universal Image Language Representation model (T-UILR) that can perform image-language tasks in 94 languages. T-Bletchley has an image encoder…
#ML
( I am experimenting with a new platform. This post is also available at: https://community.kausalflow.com/c/ml-journal-club/probably-approximately-correct-pac-learning-and-bayesian-view )
The first time I read about PAC was in the book The Nature of Statistical Learning Theory by Vapnik [^Vapnik2000].
PAC is a systematic theory on why learning from data is even feasible [^Valiant1984]. The idea is to quantify the errors when learning from data and we find that is is possible to have infinitesimal error under some certain codnitions, e.g., large datasets. Quote from Guedj [^Guedj2019]:
> A PAC inequality states that with an arbitrarily high probability (hence "probably"), the performance (as provided by a loss function) of a learning algorithm is upper-bounded by a term decaying to an optimal value as more data is collected (hence "approximately correct").
Bayesian learning is an very important topic in machine learning. We implement Bayesian rule in the components of learning, e.g., postierior in loss function. There also exists a PAC theory for Bayesian learning that explains why Bayesian algorithms works. Guedj wrote a primer on this topic[^Guedj2019].
[^Vapnik2000]: Vladimir N. Vapnik. The Nature of Statistical Learning Theory. 2000. doi:10.1007/978-1-4757-3264-1
[^Valiant1984]: Valiant LG. A theory of the learnable. Commun ACM. 1984;27: 1134–1142. doi:10.1145/1968.1972
[^Guedj2019]: Guedj B. A Primer on PAC-Bayesian Learning. arXiv [stat.ML]. 2019. Available: http://arxiv.org/abs/1901.05353
[^Bernstein2021]: Bernstein J. Machine learning is just statistics + quantifier reversal. In: jeremybernste [Internet]. [cited 1 Nov 2021]. Available: https://jeremybernste.in/writing/ml-is-just-statistics
( I am experimenting with a new platform. This post is also available at: https://community.kausalflow.com/c/ml-journal-club/probably-approximately-correct-pac-learning-and-bayesian-view )
The first time I read about PAC was in the book The Nature of Statistical Learning Theory by Vapnik [^Vapnik2000].
PAC is a systematic theory on why learning from data is even feasible [^Valiant1984]. The idea is to quantify the errors when learning from data and we find that is is possible to have infinitesimal error under some certain codnitions, e.g., large datasets. Quote from Guedj [^Guedj2019]:
> A PAC inequality states that with an arbitrarily high probability (hence "probably"), the performance (as provided by a loss function) of a learning algorithm is upper-bounded by a term decaying to an optimal value as more data is collected (hence "approximately correct").
Bayesian learning is an very important topic in machine learning. We implement Bayesian rule in the components of learning, e.g., postierior in loss function. There also exists a PAC theory for Bayesian learning that explains why Bayesian algorithms works. Guedj wrote a primer on this topic[^Guedj2019].
[^Vapnik2000]: Vladimir N. Vapnik. The Nature of Statistical Learning Theory. 2000. doi:10.1007/978-1-4757-3264-1
[^Valiant1984]: Valiant LG. A theory of the learnable. Commun ACM. 1984;27: 1134–1142. doi:10.1145/1968.1972
[^Guedj2019]: Guedj B. A Primer on PAC-Bayesian Learning. arXiv [stat.ML]. 2019. Available: http://arxiv.org/abs/1901.05353
[^Bernstein2021]: Bernstein J. Machine learning is just statistics + quantifier reversal. In: jeremybernste [Internet]. [cited 1 Nov 2021]. Available: https://jeremybernste.in/writing/ml-is-just-statistics
Kausalflow
Probably Approximately Correct (PAC) learning and bayesian view | kausalflow
The first time I read about PAC was in the book The Nature of Statistical Learning Theory by Vapnik [^Vapnik2000].
PAC is a systematic theory on why learning from data is even feasible [^Valiant1984]. The idea is to quantify the errors when learning f...
PAC is a systematic theory on why learning from data is even feasible [^Valiant1984]. The idea is to quantify the errors when learning f...
#DS #ML
Microsoft created two depositories for Machine Learning and Data Science beginners. They created many sketches. I love this style.
https://github.com/microsoft/Data-Science-For-Beginners
https://github.com/microsoft/ML-For-Beginners
Microsoft created two depositories for Machine Learning and Data Science beginners. They created many sketches. I love this style.
https://github.com/microsoft/Data-Science-For-Beginners
https://github.com/microsoft/ML-For-Beginners
#ML
(See also https://bit.ly/3F1Kv2F )
Centered Kernel Alignment (CKA) is a similarity metric designed to measure the similarity of between representations of features in neural networks[^Kornblith2019].
CKA is based on the Hilbert-Schmidt Independence Criterion (HSIC). HSIC is defined using the centered kernels of the features to compare[^Gretton2005]. But HSIC is not invariant to isotropic scaling which is required for a similarity metric of representations[^Kornblith2019]. CKA is a normalization of HSIC.
The attached figure shows why CKA makes sense.
CKA has problems too. Seita et al argues that CKA is a metric based on intuitive tests, i.e., calculate cases that we believe that should be similar and check if the CKA values is consistent with this intuition. Seita et al built a quantitive benchmark[^Seita].
[^Kornblith2019]: http://arxiv.org/abs/1905.00414
[^Gretton2005]: https://link.springer.com/chapter/10.1007%2F11564089_7
[^Seita]: https://bair.berkeley.edu/blog/2021/11/05/similarity/
(See also https://bit.ly/3F1Kv2F )
Centered Kernel Alignment (CKA) is a similarity metric designed to measure the similarity of between representations of features in neural networks[^Kornblith2019].
CKA is based on the Hilbert-Schmidt Independence Criterion (HSIC). HSIC is defined using the centered kernels of the features to compare[^Gretton2005]. But HSIC is not invariant to isotropic scaling which is required for a similarity metric of representations[^Kornblith2019]. CKA is a normalization of HSIC.
The attached figure shows why CKA makes sense.
CKA has problems too. Seita et al argues that CKA is a metric based on intuitive tests, i.e., calculate cases that we believe that should be similar and check if the CKA values is consistent with this intuition. Seita et al built a quantitive benchmark[^Seita].
[^Kornblith2019]: http://arxiv.org/abs/1905.00414
[^Gretton2005]: https://link.springer.com/chapter/10.1007%2F11564089_7
[^Seita]: https://bair.berkeley.edu/blog/2021/11/05/similarity/
#DS #news
This is a post about Zillow's Zetimate Model.
Zillow (https://zillow.com/ ) is an online real-estate marketplace and it is a big player. But last week, Zillow withdrew from the house flipping market and planned to layoff a handful of employees.
There are rumors indicating that this action is related to their machine learning based price estimation tool, Zestimate ( https://www.zillow.com/z/zestimate/ ).
At a first glance, Zestimate seems fine. Though the metrics shown on the website may not be that convincing, I am sure they've benchmarked more metrics than those shown on the website.
There are some discussions on reddit.
Anyways, this is not the best story for data scientists.
1. News: https://www.reddit.com/r/MachineLearning/comments/qlilnf/n_zillows_nnbased_zestimate_leads_to_massive/
2. This is Zestimate: https://www.zillow.com/z/zestimate/
3. https://www.wired.com/story/zillow-ibuyer-real-estate/
This is a post about Zillow's Zetimate Model.
Zillow (https://zillow.com/ ) is an online real-estate marketplace and it is a big player. But last week, Zillow withdrew from the house flipping market and planned to layoff a handful of employees.
There are rumors indicating that this action is related to their machine learning based price estimation tool, Zestimate ( https://www.zillow.com/z/zestimate/ ).
At a first glance, Zestimate seems fine. Though the metrics shown on the website may not be that convincing, I am sure they've benchmarked more metrics than those shown on the website.
There are some discussions on reddit.
Anyways, this is not the best story for data scientists.
1. News: https://www.reddit.com/r/MachineLearning/comments/qlilnf/n_zillows_nnbased_zestimate_leads_to_massive/
2. This is Zestimate: https://www.zillow.com/z/zestimate/
3. https://www.wired.com/story/zillow-ibuyer-real-estate/
Zillow
Zillow: Real Estate, Apartments, Mortgages & Home Values
The leading real estate marketplace. Search millions of for-sale and rental listings, compare Zestimate® home values and connect with local professionals.
#fun
Lol, thank you Mr Lossfunction. But, which sanitizer are you using?
https://www.reddit.com/r/learnmachinelearning/comments/qpolnw/data_cleaning_is_so_must/
Lol, thank you Mr Lossfunction. But, which sanitizer are you using?
https://www.reddit.com/r/learnmachinelearning/comments/qpolnw/data_cleaning_is_so_must/
#ML #news
1. https://ai.googleblog.com/2021/11/model-ensembles-are-faster-than-you.html
2. Wang X, Kondratyuk D, Christiansen E, Kitani KM, Alon Y, Eban E. Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models. arXiv [cs.CV]. 2020. Available: http://arxiv.org/abs/2012.01988
Most companies probably have several models to solve the same problem. There are model A, model B, even model C. The final result is some kind of aggregation of the three models. Or the models are cascaded like what's shown in the figure. But it takes a lot of computing resources to run the features through the three models.
Wang et al shows that ensembles are not more resource demanding than big models with similar performance in CV tasks.
1. https://ai.googleblog.com/2021/11/model-ensembles-are-faster-than-you.html
2. Wang X, Kondratyuk D, Christiansen E, Kitani KM, Alon Y, Eban E. Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models. arXiv [cs.CV]. 2020. Available: http://arxiv.org/abs/2012.01988
Wang et al shows that ensembles are not more resource demanding than big models with similar performance in CV tasks.
This media is not supported in your browser
VIEW IN TELEGRAM
#ML #fun
animegan v2! (I stole this animation from reddit. https://www.reddit.com/r/MachineLearning/comments/qo4kp8/r_p_animeganv2_face_portrait_v2/ )
Try it out:
1. Telegram bot (works pretty well): https://t.me/face2stickerbot
2. Dashboard (sometimes it doesn't work): https://huggingface.co/spaces/akhaliq/AnimeGANv2
Code: https://github.com/bryandlee/animegan2-pytorch
Redditors made some funny photos too.
https://www.reddit.com/r/MachineLearning/comments/qo4kp8/r_p_animeganv2_face_portrait_v2/
—
This post is also available here: https://community.kausalflow.com/c/ml-applications/animeganv2
animegan v2! (I stole this animation from reddit. https://www.reddit.com/r/MachineLearning/comments/qo4kp8/r_p_animeganv2_face_portrait_v2/ )
Try it out:
1. Telegram bot (works pretty well): https://t.me/face2stickerbot
2. Dashboard (sometimes it doesn't work): https://huggingface.co/spaces/akhaliq/AnimeGANv2
Code: https://github.com/bryandlee/animegan2-pytorch
Redditors made some funny photos too.
https://www.reddit.com/r/MachineLearning/comments/qo4kp8/r_p_animeganv2_face_portrait_v2/
—
This post is also available here: https://community.kausalflow.com/c/ml-applications/animeganv2
#DS #Visualization
Okay, I'll tell you the reason I wrote this post. It is because xkcd made [this](https://xkcd.com/2537/).
---
Choosing proper colormaps for our visualizations is important. It's almost like shooting a photo using your phone. Some phones capture details in every corner, while some phones give us overexposed photos and we get no details in the bright regions.
A proper colormap should make sure we see the details we need to see. To address the importance of colormaps, we use the two examples shown on the website of colorcet[^colorcet]. The two colormaps, hot, and fire, can be found in matplotlib and colorcet, respectively.
I can not post multiple images in one message, please see the full post for the comparisons of the two colormaps. Really, it is amazing. Find the link below:
https://github.com/kausalflow/community/discussions/20
It is clear that "hot" brings in some overexposure. The other colormap, "fire", is a so-called perceptually uniform colormap. More experiments are performed in colorcet. Glasbey et al showed some examples of inspecting different properties using different colormaps[^Glasbey2007].
One of the methods to make sure the colormap shows enough details is to use perceptually uniform colrmaps[^Kovesi2015]. Kovesi provides a method to validate if a color map has uniform perceptual contrast[^Kovesi2015].
---
References and links mentioned in this post:
[^colorcet]: Anaconda. colorcet 1.0.0 documentation. [cited 12 Nov 2021]. Available: https://colorcet.holoviz.org/
[^colorcet-github]: holoviz. colorcet/index.ipynb at master · holoviz/colorcet. In: GitHub [Internet]. [cited 12 Nov 2021]. Available: https://github.com/holoviz/colorcet/blob/master/examples/index.ipynb
[^Kovesi2015]: Kovesi P. Good Colour Maps: How to Design Them. arXiv [cs.GR]. 2015. Available: http://arxiv.org/abs/1509.03700
[^Glasbey2007]: Glasbey C, van der Heijden G, Toh VFK, Gray A. Colour displays for categorical images. Color Research & Application. 2007. pp. 304–309. doi:10.1002/col.20327
[^matplotlib-colormaps]: Choosing Colormaps in Matplotlib — Matplotlib 3.4.3 documentation. [cited 12 Nov 2021]. Available: https://matplotlib.org/stable/tutorials/colors/colormaps.html
Okay, I'll tell you the reason I wrote this post. It is because xkcd made [this](https://xkcd.com/2537/).
---
Choosing proper colormaps for our visualizations is important. It's almost like shooting a photo using your phone. Some phones capture details in every corner, while some phones give us overexposed photos and we get no details in the bright regions.
A proper colormap should make sure we see the details we need to see. To address the importance of colormaps, we use the two examples shown on the website of colorcet[^colorcet]. The two colormaps, hot, and fire, can be found in matplotlib and colorcet, respectively.
I can not post multiple images in one message, please see the full post for the comparisons of the two colormaps. Really, it is amazing. Find the link below:
https://github.com/kausalflow/community/discussions/20
It is clear that "hot" brings in some overexposure. The other colormap, "fire", is a so-called perceptually uniform colormap. More experiments are performed in colorcet. Glasbey et al showed some examples of inspecting different properties using different colormaps[^Glasbey2007].
One of the methods to make sure the colormap shows enough details is to use perceptually uniform colrmaps[^Kovesi2015]. Kovesi provides a method to validate if a color map has uniform perceptual contrast[^Kovesi2015].
---
References and links mentioned in this post:
[^colorcet]: Anaconda. colorcet 1.0.0 documentation. [cited 12 Nov 2021]. Available: https://colorcet.holoviz.org/
[^colorcet-github]: holoviz. colorcet/index.ipynb at master · holoviz/colorcet. In: GitHub [Internet]. [cited 12 Nov 2021]. Available: https://github.com/holoviz/colorcet/blob/master/examples/index.ipynb
[^Kovesi2015]: Kovesi P. Good Colour Maps: How to Design Them. arXiv [cs.GR]. 2015. Available: http://arxiv.org/abs/1509.03700
[^Glasbey2007]: Glasbey C, van der Heijden G, Toh VFK, Gray A. Colour displays for categorical images. Color Research & Application. 2007. pp. 304–309. doi:10.1002/col.20327
[^matplotlib-colormaps]: Choosing Colormaps in Matplotlib — Matplotlib 3.4.3 documentation. [cited 12 Nov 2021]. Available: https://matplotlib.org/stable/tutorials/colors/colormaps.html
xkcd
Painbow Award
#visualization
Nicolas P. Rougier released his book on scientific visualization. He made some aesthetically pleasing figures. And the book is free.
https://github.com/rougier/scientific-visualization-book
Nicolas P. Rougier released his book on scientific visualization. He made some aesthetically pleasing figures. And the book is free.
https://github.com/rougier/scientific-visualization-book