Nafig
77 subscribers
1 photo
1 file
70 links
Cool modern tech. Python, ML, Big Data, Linux stuff.

by @danielgafni
Download Telegram
Astral.sh are killing it.

Probably the biggest innovation in Python packaging over the last 5 years.

So long, Poetry, and thanks for all the fish! Dependency resolution & installation are going to be 10-100x faster now.
Just found a really useful feature of Docker Compose called watch.

It allows executing actions against the running container during development, such as syncing source code or rebuilding the image on file changes.

This is more versatile than mounting large directories into the running container. The feature supports multiple conditions and exclusion of certain files.

Example with uv
Recently I’ve spent some time working on dagster-ray.

Pretty happy with the result: now there is a bunch of ways to kick off computations on Ray from Dagster pipelines, all with their own advantages and tradeoffs, including two types of Pipes clients (Dagster Pipes has been my main project for the last 3 months).

It's still early but anyway, it's probably the most complicated and feature-rich of all my personal projects. I know there has been demand for a proper Ray integration in the Dagster community. Hopefully, folks will have more options going forward.
Mind blowing amount of optimizations for faster dev environment startup mentioned in this blog post by GitPod.
Intuitively it makes sense that Kubernetes isn't the best platform for dev environments, and they also go to great lengths in order to support this claim.
Pretty nice read.
Somehow it's the first time I'm hearing about Burrito - think ArgoCD for Terraform.
Looking really great, should make terraform development much easier. Might try for me personal infra management soon.
It's happening! Astral is building a Python type checker in Rust.

Can't wait to get my hands on it.

https://x.com/charliermarsh/status/1884651482009477368
Please open Telegram to view this post
VIEW IN TELEGRAM
I decided to post an original piece of content instead of doing another effortless repost.

You can read my take on organizing and building Python monorepos efficiently with uv and Dagger on my website.

Teaser:

In this post, I am going to share an approach to building Python monorepos that solves these issues in a very elegant way. The benefits of this approach are:
- it works with any `uv` project (even yours!)
- it needs little to zero maintenance and boilerplate
- it provides end-to-end pipeline caching --- including steps downstream to building the image (like running linters and tests), which is quite rare
- it's easy to run locally and in CI



uv init --package --lib weird-location/nested/lib-three
uv add --package lib-three lib-one lib-two
dagger call build-project --root-dir . --project lib-three

✔️ connect 0.2s
✔️ load module 5.4s
✔️ parsing command line arguments 2.0s

✔️ monorepoDagger: MonorepoDagger! 2.1s
✔️ .buildProject(
│ │ debugSleep: 0.000000
│ │ project: "lib-three"
│ │ rootDir: no(digest: "sha256:7112225e5254a6bc947b4ce9318d5ed7e8e5a713df2bb1acefa52bbd739077ce"): Missing
│ ): Container! 8.2s
✔️ .defaultArgs: [String!]! 0.0s

✔️ Container.mounts: [String!]! 0.0s

✔️ Container.entrypoint: [String!]! 0.0s

✔️ Container.platform: Platform! 0.0s

✔️ Container.user: String! 0.0s

✔️ Container.workdir: String! 0.0s

_type: Container
defaultArgs:
- python3
entrypoint: []
mounts: []
platform: linux/amd64
user: ""
workdir: /src
Extremely excited about Dagster Components being released!

This is going to dramatically improve UX with Dagster development (it's already high so I guess the new bar is just unimaginable) and significantly smoothen the learning curve. I imagine even people who can't really code (like your typical data analysts) will be able to have impact with this system!