⚡️ I have run a Proof of Concept with Redshift Serverless

And here's the summary:

— Reserving Redshift nodes is the best option given specific usage patterns.
— Reserved nodes are 30% cheaper than on-demand ones.
— Serverless doesn't show stable and production-ready results: faced with multiple query failures during data refresh.
— Moreover, Serverless will definitely cost more with the same usage patterns.

Read more →

#dbt #redshift #database #serverless

🌐 @data_apps | Навигация по каналу

Why I use dev containers?

How to package dbt and dependencies to a reproducible container

– works almost instanty for you and team members
– latest stable dbt + adapter versions
– easy inividual secrets set up
– configuring multiple dbt targets
– fancy terminal: zsh + git plugin + shell history

+ 30 second gif live demo

Read more →

#dbt #dev #container

🌐 @data_apps | Навигация по каналу

What I value most in BI tool

❄️ I use Looker and totally satisfied with it.

Here's my list of features labeled with (H)igh, (M)edium and (L)ow priority and grouped by:
— Developer experience
— User experience
— Administration
— Monitoring, System activity
— Integrations & Apps

I used this list to compare different solutions against what I currently use to see if I can benefit from switching to another tool.

You can use it as reference to populate BI tool comparison matrix (included).

➡️ https://telegra.ph/What-I-value-most-in-BI-tool-03-14 →

#bi #tool #solution

🌐 @data_apps | Навигация по каналу

Аналитика ключевых метрик компании: dbt Metrics / Cube.js / LookML

Привет! 25 апреля Вт в 20:00 проведу вебинар на YouTube.

Рассмотрим на практике грани интересной темы:
🔸 Что такое семантический слой, и в чем разница между Метрикой и Витриной данных
🔸 Пути формирования метрик: SQL (ad-hoc), dbt Metrics package, Cube.js, LookML (Looker)
🔸 Примеры декларативной конфигурации dbt metrics в yaml
🔸 Как использовать продвинутые возможности dbt Metrics (derived metrics, secondary calculations)

Важные выводы и идеи при выборе решения:
🔸 Open source solutions vs. Vendor lock-in
🔸 Flexibility vs. Speed tradeoff (online queries vs. pre-materialized)
🔸 Access vs. Expressiveness (in-depth features vs. ease of use)

Если тема интересна, приходите на транcляцию - буду отвечать на вопросы, можно будет подискутировать.

❗Ссылка будет в этом канале за 10-15 минут до начала трансляции.

#dbt #metrics #bi

🌐 @data_apps | Навигация по каналу

❗Ссылка на трансляцию: https://www.youtube.com/live/S7U2VTe06tE?feature=share

❓Вопросы задавайте в комментарии к этому посту либо через чат:
https://otus.ru/lessons/event-chat/2847/

Семантический слой для Аналитики ключевых метрик – dbt Metrics vs. Cube

🔸 Что такое семантический слой, и в чем разница между Метрикой и Витриной данных.
🔸 Пути формирования метрик: SQL (ad-hoc), UI, dbt Metrics, Cube.
🔸 Примеры декларативной конфигурации в YAML и использования метрик.
🔸 На что обращать внимание при выборе решения: Стек технологий, Доступность vs. Выразительнось, Гибкость vs. Скорость.

В продолжение последней темы - для тех, кто любит читать.

Читать на Habr →

🌐 @data_apps | Навигация по каналу

Having extensive experience with Looker and dbt I've got several thoughts on

⚡️ Replacing Looker with Cube + Superset ⚡️

🔸 Future of BI and Analytics revolves around declarative metrics store and polyglot serving
— You set your metrics simple way, in a single store and version control them
— You then consume these metrics from wherever needed (BI, SQL, Apps, REST API)

🔸 Cube is a powerful rival to Looker
— Both support: Declarative Metrics, Dynamic SQL generation, Caching, RBAC
— Cube is Open Source while Looker is a vendor lock-in (which is tying your options and opportunities)
— Cube offers freemium plan while Looker is top-tier in terms of pricing

🔸 Cube still has got many things to improve
— Looker has fancy browser IDE with code-completion, syntax checks and built-in docs
— Cube is of a higher entry level in terms of: deployment, git, development, updates, etc.
— Looker supports flexible templating with Liquid: variables, dynamic joins, filters etc. enabling otherwise impossible opportunities
— Looker has impressive list of supported data sources and dialects

Apart from this Cube is not a visualization tool, so you will definitely need one on top of Cube (e.g. Superset, Metabase).
Looker has everything to be named top grade BI tool: rich dashboards, scheduling, alerting, API.

🔸 I would go on with Looker
— In case you have invested much into your LookML project and using it for a long period of time
— I would strongly consider using Cube with OSS BI tool if you are searching for a new stack or want to switch

#bi #looker #cube #metrics

⬇️ Share your thought if you've ever used Looker or Cube.

🌐 @data_apps | Навигация по каналу

⚙️ Clickhouse is advancing at data integration

Apart from having fully-functional DB links to PostgreSQL and MySQL,

Clickhouse is also able to fetch data from any URL:

— No need for 3rd party tools (ELT)
— Extensive configurations for data formats and compression
— Do it SQL declarative way
— Works best with dbt modeling

Now I give you a brief example of how to sync data from external API (e.g. gSheets) to Clickhouse:

1. You create a table with URL engine pointing to external API either manually or using dbt macro
2. You register this table as a dbt source
3. You then reference this source anywhere in your dbt project

See the code →

#elt #integration #clickhouse #dbt

🌐 @data_apps | Навигация по каналу

Major improvements to my Analytics Engineering Lab on Github

🚀 Airbyte Data Pipelines Lab

– Configuring Data Pipelines with Airbyte
– Deploying Infrastructure as Code with Terraform and Yandex.Cloud
– Instant development with Github Codespaces
– Assignment checks with Github Actions

⚙️ What you will do:

– Configure Developer Environment: GitHub Codespaces or devcontainer
– Deploy Infrastructure with Terraform: VM with Airbyte installed, S3 Bucket, Clickhouse
– Configure Data Pipelines: Postgres Source, Clickhouse Destination, S3 Destination
– Test your Pipeline with dbt: locally and triggering automated testing with Github Actions

Open Lab on Github →

#airbyte #terraform #yc #lab #elt #data_integration

🌐 @data_apps | Навигация по каналу