#entropy #antropy #stats #informationtheory

Ботанам, которые любят считать энтропию, пригодится!

https://raphaelvallat.com/antropy/build/html/index.html

#statistics #informationtheory #entropy #python #featureselection #featureengineering

Ну да ладно, пока просто в личном блоге опубликую, оказывается, вроде потом можно будет статью прикрепить к паблику.

https://medium.com/@fingoldo/15819b261de0

#ml #masters #ensembling #featureengineering #entropy

Продолжаем.

"A common procedure is to train several competing models on the same training set and then choose the best performer for deployment. However, it is usually advantageous to use all of the competing models and intelligently combine their predictions or class decisions to produce a consensus opinion."

"It is not widely known that the entropy of a predictor variable can have a profound impact on the ability of many models to make effective use of the variable. Responsible researchers will compute the entropy of every predictor and take remedial action if any predictor has low entropy."

Первая идея не нова, в соревах все стэкают модели. Но опять-таки, это до сих пор не стандарт в МЛ, и тот же sklearn просто отбрасывает все модели за исключением "лучшей", там даже опции нет сохранить остальные, или, упаси Боже, совместно их использовать.

А вот энтропийный подход к выбору и предобработке предикторов оригинален, такой идеи я нигде не встречал больше. Что нам предлагает классика? Генерить побольше потенциальных признаков произвольной природы, пока Ваша модель не захлебнётся по ресурсам. Но ведь можно действовать умнее. Эту идею можно использовать при комбинации нескольких признаков: к примеру, оставлять только те комбинации, чья энтропия превышает энтропии родителей.

#masters #ml #entropy #mutualinformation #featureselection #pld #fleuret

Читаю последние главы книги "Assessing and Improving Prediction and Classification" Тима Мастерса, и прямо хочется взять и самому реализовать расписанные там алгоритмы отбора предикторов на основе прямого последовательного включения (forward stepwise selection) и теории информации. Тем более что у меня же есть в планах большое сводное тестирование методов FS. Почему прямого - он самый быстрый, следовательно, в современных реалиях, когда предикторов десятки тысяч, самый практичный.

Самый простой вариант называется PLD, когда на каждом шаге в набор активных предикторов S выбирается фича, имеющая самую высокую взаимную информацию (ВИ/MI) с таргетом минус среднюю ВИ с уже зафиксированными переменными S.

Улучшенная версия Fleuret ещё более интеллектуальна, она смотрит на условную ВИ зафиксированных переменных и таргета при данном кандидате:

Suppose X is a candidate for inclusion and Z is a variable that is already in S, the set of predictors chosen so far. The conditional mutual information of X and Y given Z measures how much the candidate X contributes to predicting Y above and beyond what we already get from Z. A good candidate will have a large value of I(X;Y|Z) for every Z in S. If there is even one variable Z in S for which I(X;Y|Z) is small, there is little point in including this candidate X, because it contributes little beyond what is already contributed by that Z.

#entropy #mutualinformation #prediction #finance #trading #causality

https://www.youtube.com/watch?v=tdWh7agQfTk

#information #entropy #maxwellsdemon #szillardsengine

Шикарно иллюстрированная, глубокая статья. Всем рекомендую к прочтению.

https://medium.com/swlh/a-deep-conceptual-guide-to-mutual-information-a5021031fad0

#poetry #rap #battle #celebrities #fun #gpt #information #entropy

"Verse 1 - Alan Turing:

I'm Turing, the father of all computation,
Built a machine that helped end a nation.
Your information theory, it's imitation,
I laid the foundation, you just gave it notation.

Broke the Enigma, saved millions of lives,
While you're playing with switches, just trying to survive,
And to think they said, you're the brightest of minds,
But not even your own theory could define your lines.

Verse 2 - Claude Shannon:

It's Shannon, the master of communication,
Your work is nothing without my information.
You can break the codes, but can't handle the truth,
I took your machine, gave it substance and proof.

You're binary, predictable, no real surprise,
But I'm riding entropy, in disguise.
Gave birth to the bit, the byte, the baud,
Without my blueprint, you'd still be flawed.

Verse 3 - Alan Turing:

Nice try Shannon, but you're no match,
Your logic circuit's flickering, I'm about to catch,
Your "bit" and "byte" are just a ruse,
Without my machine, they're of no use.

My Turing machine is universal,
Your theory is just controversial.
I'm the mastermind of artificial intelligence,
Your work is vital, but lacks relevance.

Verse 4 - Claude Shannon:

You're living in a theoretical existence,
I'm empirical, practical, I go the distance.
Your machine's universal, but I'm not impressed,
It's a theoretical model, while mine's manifest.

I'm the juggler, the unicyclist, the eclectic soul,
You're a one-track mind, stuck in a hole.
So, Turing, it's time you faced your defeat,
Cause, in this rap battle, you just can't compete."

#poetry #rap #battle #celebrities #fun #gpt #information #entropy

Ну что, продолжаем рэп-битву титанов информатики?

"Verse 5 - Tsallis:

Hold up, let Tsallis step into the scene,
With a power so fierce, it's like nothing you've seen.
Turing and Shannon, your theories are neat,
But my entropy's the game-changer, can't be beat.

Non-extensive stats, breaking the mold,
In the realm of complexity, my story is told.
Turing, your universality is fine,
But Tsallis entropy brings a paradigm.

Shannon, entropy's not just your domain,
Tsallis q-entropy's here to reign.
I generalize, beyond your scope,
In the statistical world, I'm the new hope.

Verse 6 - Alan Turing:

Tsallis, your entropy's just a tweak,
In the world of computation, I'm the peak.
Universal computation, that's my claim,
Your q-entropy's just a fleeting flame.

Breaking codes, saving lives with precision,
While you play with parameters, in a theoretical vision.
I laid the groundwork for AI's inception,
Your entropy's just a mathematical perception.

Verse 7 - Claude Shannon:

Tsallis, your stats might have a twist,
But in the world of communication, I persist.
I ride the waves of information flow,
Your q-entropy, in comparison, is slow.

Entropy's dance, a symphony I conduct,
In the realm of signals, my theories erupt.
Turing, you may compute, Tsallis may tweak,
But in the real world, my information's what they seek.

Verse 8 - Tsallis:

In the dance of complexity, I bring the beat,
Tsallis entropy, making theories complete.
Turing and Shannon, take a back seat,
I transcend limits, my stats can't be beat.

Non-extensive, a paradigm shift,
In the rap battle of minds, my verses uplift.
So whether it's Turing, Shannon, or the rest,
Tsallis entropy reigns supreme, the best."

#physics #entropy #people #boltzmann

https://youtu.be/InP-f6l3vmU?si=6jtoixf5M4xehbVt