Yehor Smoliakov
Blazingly fast 🦀
https://github.com/egorsmkv/kulyk-rust with CUDA backend for GGML model (qunatized int8)
GitHub
GitHub - egorsmkv/kulyk-rust: Rust-based translator that utilizes the KULYK models to do machine translation for English<->Ukrainian
Rust-based translator that utilizes the KULYK models to do machine translation for English<->Ukrainian - egorsmkv/kulyk-rust
🔥2
doing something
https://github.com/egorsmkv/kulyk-rust with CUDA backend for GGML model (qunatized int8)
now in docker images; do just pull it and run:
docker run -p 3000:3000 --rm ghcr.io/egorsmkv/kulyk-rust:latest
Also:
#vllm
Valid options: ['FLASH_ATTN', 'TRITON_ATTN', 'XFORMERS', 'ROCM_FLASH', 'ROCM_AITER_MLA', 'ROCM_AITER_FA', 'TORCH_SDPA', 'FLASHINFER', 'FLASHINFER_MLA', 'TRITON_MLA', 'CUTLASS_MLA', 'FLASHMLA', 'FLASH_ATTN_MLA', 'PALLAS', 'IPEX', 'DUAL_CHUNK_FLASH_ATTN', 'DIFFERENTIAL_FLASH_ATTN', 'NO_ATTENTION', 'FLEX_ATTENTION', 'TREE_ATTN', 'ROCM_ATTN']
#vllm
Hacker News
Nvidia DGX Spark: great hardware, early days for the ecosystem Article, Comments
www.lmsys.org
NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference
Thanks to NVIDIA’s early access program, we are thrilled to get our hands on the NVIDIA DGX™ Spark. It’s quite an unconventional system, as NVIDIA rarely releases compact, all-in-one machines that bri...