Aspiring Data Science

#ml #feedback #trubrics #monitoring

Впервые вижу. Можно централизованно собирать обратную связь по вашим МЛ моделям. Развёрнутым, к примеру, в streamlit.`

Why should you monitor usage of your models?
🚨 Identify bugs - users are constantly running inference on your models, and may be more likely to find bugs than your ML monitoring system
🧑‍💻 Fine tune - users often hold domain knowledge that can be useful to fine tune models
👥 Align - identifying user preferences will help you to align models to your users

https://github.com/trubrics/trubrics-sdk

GitHub

GitHub - trubrics/trubrics-sdk: The first user analytics platform for AI models

The first user analytics platform for AI models. Contribute to trubrics/trubrics-sdk development by creating an account on GitHub.

102 viewsAnatoly Alekseev, 15:53

Aspiring Data Science

#mlops #monitoring #drift #evidently #greatexpectations

Мониторить нужно:
1) распределение входных признаков
2) распределение предсказаний
3) невязку моделей
4) неуверенность моделей (trust score; conformal?)
5) прокси-метрики, если метки созревают медленно

Держать теневые (shadow) модели.
Записывать боевые предикты.
Interleaving deploy of ML models instead of full A/B.
Устаревание модели можно тоже моделировать.

https://www.youtube.com/watch?v=rD2Ydyr3Sdc

YouTube

Устойчивость ML-моделей. Датазавтрак в Екатеринбурге

Устойчивость ML-моделей Как часто дообучать модель и как делать это правильно. О том, что модели нужно не только делать, но и поддерживать + пара советов на этом пути. Слайды тут https://kolodezev.ru/ekaterinburg2023.html

93 viewsAnatoly Alekseev, edited 17:55

Aspiring Data Science

#drift #jsd #monitoring #mlops

https://towardsdatascience.com/how-to-understand-and-use-jensen-shannon-divergence-b10e11b03fd6

Medium

How to Understand and Use the Jensen-Shannon Divergence

A primer on the math, logic, and pragmatic application of JS Divergence — including how it is best used in drift monitoring

114 viewsAnatoly Alekseev, 18:17

Aspiring Data Science

#mlops #drift #monitoring #mlflow

https://www.youtube.com/watch?v=tGckE83S-4s

YouTube

Drifting Away: Testing ML Models in Production

Deploying machine learning models has become a relatively frictionless process. However, properly deploying a model with a robust testing and monitoring framework is a vastly more complex task. There is no one-size-fits-all solution when it comes to productionizing…

77 viewsAnatoly Alekseev, edited 01:33

Aspiring Data Science

#drift #monitoring

https://www.youtube.com/watch?v=uOG685WFO00

YouTube

ML Drift: Identifying Issues Before You Have a Problem

Over time, our AI predictions degrade. Full Stop.

Whether it’s concept drift, where the relationships of our data to what we’re trying to predict has changed, or data drift, where our production data no longer resembles the historical training data, identifying…

81 viewsAnatoly Alekseev, 02:21

Aspiring Data Science

#monitoring

Интересна идея, что в ML мониторить нужно 3 разных класса

1) данные - признаки и метки
2) предсказания модели
3) ML-метрики

https://www.youtube.com/watch?v=oUcuilWWX78

YouTube

Monitoring ML Models in Production

What happens after you deploy a machine learning model? How do you make sure that your model performance doesnot degrade as data and the world change?
In this talk, Danny D. Leybzon will explain how monitoring ML models in production is key to deriving value…

👍1

91 viewsAnatoly Alekseev, 02:26

Aspiring Data Science

#mlops #monitoring

Putting fears into the metrics, add metrics to the dashboard & alerts, интересно.

https://www.youtube.com/watch?v=wWxqnZb-LSk

YouTube

Lina Weichbrodt: What I learned from monitoring more than 30 Machine Learning Use Cases

Speaker:: Lina Weichbrodt

Track: General: Production
This talk summarizes the most important insights I gained from running more than 30 machine learning use cases in production. We will take a loan prediction model as an example use case and cover questions…

90 viewsAnatoly Alekseev, edited 05:40

Aspiring Data Science