Forwarded from Yehor Smoliakov
Please open Telegram to view this post
VIEW IN TELEGRAM
Yehor Smoliakov
Blazingly fast 🦀
https://github.com/egorsmkv/kulyk-rust with CUDA backend for GGML model (qunatized int8)
GitHub
GitHub - egorsmkv/kulyk-rust: Rust-based translator that utilizes the KULYK models to do machine translation for English<->Ukrainian
Rust-based translator that utilizes the KULYK models to do machine translation for English<->Ukrainian - egorsmkv/kulyk-rust
🔥2
doing something
https://github.com/egorsmkv/kulyk-rust with CUDA backend for GGML model (qunatized int8)
now in docker images; do just pull it and run:
docker run -p 3000:3000 --rm ghcr.io/egorsmkv/kulyk-rust:latest
Also:
#vllm
Valid options: ['FLASH_ATTN', 'TRITON_ATTN', 'XFORMERS', 'ROCM_FLASH', 'ROCM_AITER_MLA', 'ROCM_AITER_FA', 'TORCH_SDPA', 'FLASHINFER', 'FLASHINFER_MLA', 'TRITON_MLA', 'CUTLASS_MLA', 'FLASHMLA', 'FLASH_ATTN_MLA', 'PALLAS', 'IPEX', 'DUAL_CHUNK_FLASH_ATTN', 'DIFFERENTIAL_FLASH_ATTN', 'NO_ATTENTION', 'FLEX_ATTENTION', 'TREE_ATTN', 'ROCM_ATTN']
#vllm