doing something

Here's a popular alternative to Pandas - Polars

It's written in Rust

I have adapted their code in my tiny utility to extract audio files from parquet files in parallel using rayon

All we need is blazingly fast programs

#rust

138 views09:52

doing something

https://github.com/glotlabs/gdrive #rust

So useful, uploading 110 GB to gdrive

https://github.com/glotlabs/gdrive

#rust

127 viewsedited 20:23

doing something

https://iree.dev/guides/ml-frameworks/onnx/#samples

Learned that we can upgrade ONNX version using https://github.com/onnx/onnx

#ml #optimization

118 views21:08

doing something

https://youtu.be/_-jBclcGiMY?feature=shared

#rust

YouTube

Parsing text data with SIMD in Rust / Аналіз текстових даних з SIMD в Rust (Maksym Vorobiov)

Speech 1

UARust Conference 2024🦀

Topic: “Parsing text data with SIMD in Rust”
Speaker: Maksym Vorobiov
Language: ukrainian 🇺🇦
Subtitles: english 🇬🇧, ukrainian 🇺🇦
Presentation: https://github.com/dunnock/rust-parsing-with-simd/blob/master/slides/slides.pdf…

121 views09:07

doing something

Created a simple tool to check NodeJS packages for vulnerabilities, you just add a YAML file to your CI

https://github.com/egorsmkv/npm-audit-to-report

#go

GitHub

GitHub - egorsmkv/npm-audit-to-report: A simple Go program that converts `security-audit.json` to `security-audit.md` so you can…

A simple Go program that converts `security-audit.json` to `security-audit.md` so you can use it in CI pipeline. - egorsmkv/npm-audit-to-report

112 views10:58

doing something

Also, yesterday I have been working on this tool - https://github.com/egorsmkv/extract-audio

Now, processing a large audio dataset using it, total size is 744 GB

#rust

GitHub

GitHub - crs-org/extract-audio: Extract audio files from a parquet or arrow file generated by Hugging Face `datasets` library.

Extract audio files from a parquet or arrow file generated by Hugging Face `datasets` library. - crs-org/extract-audio

390 views12:42

doing something

https://huggingface.co/docs/transformers/en/serialization

#ai #optimization

huggingface.co

Exporting to production · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

114 views14:23

doing something

https://iree.dev/guides/ml-frameworks/onnx/#samples Learned that we can upgrade ONNX version using https://github.com/onnx/onnx #ml #optimization

113 views14:35

doing something

Photo

Successfully converted w2v2 model to this format

On the photo you can see logits from the NN

#ai #ml

👍1

111 viewsedited 15:00

doing something

Successfully converted w2v2 model to this format On the photo you can see logits from the NN #ai #ml

Also, tested it on CUDA, sent it 1000 tensors to inference

111 views16:05

doing something

Converted my w2v2 model to ONNX:

- https://huggingface.co/Yehor/w2v-bert-uk-v2.1-onnx

- https://huggingface.co/Yehor/w2v-bert-uk-v2.1-onnx-gpu

#ai #speech

114 views16:21

doing something

https://github.com/SamKG/repocat

Useful

#rust

GitHub

GitHub - SamKG/repocat

Contribute to SamKG/repocat development by creating an account on GitHub.

114 views16:36

doing something

https://github.com/rust-embedded-community/tinyrlibc

#rust

GitHub

GitHub - rust-embedded-community/tinyrlibc: Tiny C library written in Rust

Tiny C library written in Rust. Contribute to rust-embedded-community/tinyrlibc development by creating an account on GitHub.

117 views16:54

doing something

Also, tested it on CUDA, sent it 1000 tensors to inference

https://github.com/gmmyung/eerie

#rust

GitHub

GitHub - gmmyung/eerie: Rustic bindings to the IREE Compiler/Runtime

Rustic bindings to the IREE Compiler/Runtime. Contribute to gmmyung/eerie development by creating an account on GitHub.

127 views17:26

doing something

#python

119 views22:28

doing something

https://github.com/rust-embedded-community/tinyrlibc #rust

https://github.com/sunfishcode/c-ward

#rust

GitHub

GitHub - sunfishcode/c-ward: An implementation of libc written in Rust

An implementation of libc written in Rust. Contribute to sunfishcode/c-ward development by creating an account on GitHub.

116 views10:55

doing something

Data Hoarding in Action:

- 7.4k hours of audio;
- 10.7 million of utterances;
- 2 GPU - 3090;
- batch size d0 = 60, d1 = 3400;
- sphn (rust-based library to load audio to numpy ndarray);

- RTF: ~0.0044 (data should be done in ~16 hours).

#python #rust

🔥1

351 viewsedited 13:54

doing something

https://docs.rs/env_logger/latest/env_logger/

#rust

109 views15:38

doing something

https://github.com/webonnx/wonnx

#rust #optimization

GitHub

GitHub - webonnx/wonnx: A WebGPU-accelerated ONNX inference run-time written 100% in Rust, ready for native and the web

A WebGPU-accelerated ONNX inference run-time written 100% in Rust, ready for native and the web - webonnx/wonnx

113 views15:57

doing something