Explore "Pretraining LLMs," a short course developed with upstageai.

The course covers pretraining from scratch, continuing pretraining on custom data, and how using smaller open-source models can reduce costs.

Take the course for free: https://hubs.la/Q02YFKyx0

https://t.me/DataScienceT ✅

⭐️ Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinement

RAG-Diffusion now supports FLUX.1 Redux!

🔥 Ready to take control? Customize your region-based images with our training-free solution and achieve powerful, precise results!

🔗 Code: https://github.com/NJU-PCALab/RAG-Diffusion

https://t.me/DataScienceT ✅

🌟 INTELLECT-1: Release of the first decentralized learning model.

PRIME Intellect has published INTELLECT-1 ( Instruct + Base ), the first 10 billion parameter language model collaboratively trained in 50 days by 30 participants worldwide.

PRIME Intellect used its own PRIME platform, designed to address the main problems of decentralized learning: network unreliability and dynamic management of computing nodes.

The platform utilized a network of 112 H100 GPUs across 3 continents and achieved a compute utilization rate of 96% under optimal conditions.

The training corpus consisted of 1 trillion public dataset tokens with the following percentage distribution: 55% fineweb-edu, 10% fineweb, 20% Stack V1, 10% dclm-baseline, 5% open-web-math.

▶️ Technical specifications:

🟢 Parameters: 10B;
🟢 Layers: 42;
🟢 Attention Heads: 32;
🟢 Hidden Size: 4096;
🟢 Context Length: 8192;
🟢 Vocabulary Size: 128256.

INTELLECT-1 achieved 37.5% accuracy on the MMLU test and 72.26% on HellaSwag, and outperformed several other open-source models on WinoGrande with a score of 65.82%.

While these figures lag slightly behind today's popular models, the results of the experiment are a critical step toward democratizing AI development and preventing the consolidation of AI capabilities within a few organizations.

▶️ GGUF quantized versions of INTELLECT-1_Instruct in 3-bit (5.46 GB) to 8-bit (10.9 GB) bit depths from the LM Studio community.

▶️ Example of inference on Transformers:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.set_default_device("cuda")
model = AutoModelForCausalLM.from_pretrained("PrimeIntellect/INTELLECT-1")
tokenizer = AutoTokenizer.from_pretrained("PrimeIntellect/INTELLECT-1")

input_text = "%prompt%"
input_ids = tokenizer.encode(input_text, return_tensors="pt")
output_ids = model.generate(input_ids, max_length=50, num_return_sequences=1)
output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)

print(output_text)

📌 Licensing: Apache 2.0 License.


🟡 Article
🟡 HF Model Kit
🟡 Set of GGUF versions
🟡 Technical report
🟡 Demo
🖥 GitHub

https://t.me/DataScienceT ✅