https://pypi.org/project/parquet-tools/

#parquet #data_engineering

Do it on your own:

https://gist.github.com/egorsmkv/ea16104f4e167702f3a7f65a442eed6f

HF dataset viewer support chat conversations

Example: https://huggingface.co/datasets/HuggingFaceTB/smoltalk

#hf

My time to use dask, because 16 GB jsonl with non-trivial structure is not fittable into 50 gb of memory...

#data_engineering

Ohhh, it does not save original structure... List became "Large String"...

Splitted the original file into smaller lines using split and then processed using polars

datasets 4.0.0 rocks now

https://huggingface.co/docs/datasets/en/use_with_polars

#kek_moment in training neural networks

it should learn more, obviously

Google's Veo model is good in generation videos

#genai

Why good training matters

Full log of those translations

https://huggingface.co/docs/trl/en/sft_trainer#trl.SFTConfig.dataset_num_proc

What saves my time and time of a ML lab

#ml

computers see a language with their own understanding

#nlp

emotional swings in ML

https://huggingface.co/spaces/Yehor/en-uk-translator

Now you can play with the model

I will update the checkpoint sometimes

#nlp #llm

Я тут дізнався, що можу по секрету розповісти про новий НАЙБІЛЬШИЙ корпус української мови з існуючих!
На 60 BILLION tokens!

КОБЗА - https://huggingface.co/datasets/Goader/kobza

Це не просто корпус - це надія, родючий чорнозем, з якого дадуть паростки численні українські ШІ продукти.

Людина яка подарувала нам цей скарб - Mykola Haltiuk (@xgoader), ще і наш недавній підписник!

Updated the checkpoint

Happy 20th Birthday, Django
Article, Comments