Nick 🌑

https://youtu.be/UDMVYNX9xPw?si=gMp6-Agg76NgeidU

https://youtu.be/W1qEhkr9k8c?si=7StdlTAfGz6vCOJ0

Reza ⚙️

https://youtu.be/Fg0lMebGt9I?si=OmLIOF8RoMhnUwR2

https://youtu.be/mNoJHeN2ruM?si=g0SLnPSPJSx4qi2M

آیا با آن خواهیم رقصید؟

تصور پایان جهان از تصور پایان سرمایه‌داری آسان‌تر است.

سرمایه یک انگل مجرد است، یک خون‌آشام سپری‌ناپذیر و یک زامبی‌ساز خستگی‌ناپذیر؛ اما گوشتِ زنده‌ای که به کارِ مرده بدل می‌سازد، گوشتِ تنِ ماست، و زامبی‌هایی که می‌سازد، ما هستیم.

همه‌چیز تسلیم تغییراتِ دائمِ مُد و تصاویرِ رسانه‌ای می‌شود، و [در عینِ حال] دیگر هیچ‌چیز تغییر نمی‌کند.

پس از وضعیتی که در آن هیچ‌چیز ممکن نیست، ناگهان همه‌چیز دوباره ممکن می‌شود.

ما باور داریم که پول تنها یک ژتونِ بی‌معناست که ارزشِ ذاتی ندارد، اما چنان عمل می‌کنیم که گویی واجدِ نوعی ارزشِ قدسی‌ست. و این عملکرد دقیقاً به‌لطف نفی‌وانکار (Fetishist Disavowal) ممکن می‌شود.

Nothing human makes it out of the near-future.

مقاومت بی‌فایده‌ست؛ بهتر است با فرآیند هم‌راستا شوی یا کنار بروی.

Non exhaustive list of papers cited:

Attention Is All You Need - https://arxiv.org/abs/...​
GShard - https://arxiv.org/abs/...​
Scaling Laws for Fine-Grained Mixture of Experts - https://arxiv.org/abs/...​
GPipe: https://arxiv.org/abs/...​
DeepSeek V2: https://arxiv.org/abs/...​
Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts - https://arxiv.org/abs/...​
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM - https://arxiv.org/abs/...​
Fast Transformer Decoding: One Write-Head is All You Need - https://arxiv.org/abs/...​
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints - https://arxiv.org/abs/...​
Zero Bubble Pipeline Parallelism - https://arxiv.org/abs/...​
DeepSeek V3 - https://arxiv.org/abs/...​
Training Compute-Optimal Large Language Models - https://arxiv.org/abs/...​
Scaling Laws for Neural Language Models - https://arxiv.org/abs/...​

Chapters

00:00:00​ - Introduction
00:16:50​ - Model Architecture, Parameters and Training FLOPs
00:51:57​ - RoPE from First Principles
01:11:25​ - Implementing RoPE and YaRN
01:50:16​ - Building the Transformer and Weight Initialization
02:19:12​ - Attention, the KV Cache and Arithmetic Intensity
03:02:56​ - Coding Multi-head Latent Attention (MLA)
03:15:35​ - Block Matrix Multiplication and MLA Internals
03:34:58​ - Deriving MLA and Decoupled RoPE
04:10:54​ - MLA Weight Absorption
04:32:59​ - Autograd and the Mathematics of Distributed Training
05:04:19​ - Distributed Computation Graphs and DDP
05:10:47​ - Building the Training Loop
06:06:17​ - Pipeline Parallelism from First Principles
06:26:52​ - Pipeline Schedules: GPipe, 1F1B and Zero Bubble
07:10:29​ - Datasets, Tokenization and Data Parallelism
07:42:10​ - Coding Pipeline Parallelism
08:37:34​ - Device Meshes and Combining PP with DP
09:48:52​ - Distributed Communication Collectives
10:49:41​ - Implementing Device Meshes, DDP and FSDP
12:13:08​ - Tensor Parallelism from First Principles
13:44:57​ - Coding Tensor Parallelism
14:47:43​ - Context Parallelism and Ring Attention
15:41:22​ - Metrics, Optimizers, Schedulers and Checkpointing
16:00:33​ - Combining Parallelism in the Training Loop
16:49:58​ - Mixture of Experts from First Principles
18:02:52​ - Tensor Parallelism for MoE
18:36:24​ - Expert Parallelism
19:01:56​ - All-to-All Token Dispatch and Combine
19:27:38​ - Expert Tensor Parallelism

engrossed elsewhere for the foreseeable, hence my absence. should the day come, i'll resurface to scribe a line or two..

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

MODEL_NAME = "cross-encoder/nli-deberta-v3-base"
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME).to(DEVICE)
model.eval()

FREE_HYPOTHESIS = "The person is free and available."
BUSY_HYPOTHESIS = "The person is busy and unavailable."

ID_TO_LABEL = {
    int(index): label.lower()
    for index, label in model.config.id2label.items()
}

ENTAILMENT_ID = next(
    index for index, label in ID_TO_LABEL.items()
    if "entail" in label
)

CONTRADICTION_ID = next(
    index for index, label in ID_TO_LABEL.items()
    if "contrad" in label
)

ABSENCE_STATE = "BUSY"


@torch.inference_mode()
def _nli_scores(text, hypothesis):
    encoded = tokenizer(
        text,
        hypothesis,
        return_tensors="pt",
        truncation=True,
        max_length=128
    )

    encoded = {
        key: value.to(DEVICE)
        for key, value in encoded.items()
    }

    probabilities = torch.softmax(
        model(**encoded).logits,
        dim=-1
    )[0]

    return (
        probabilities[ENTAILMENT_ID].item(),
        probabilities[CONTRADICTION_ID].item()
    )


def status(text):
    if not isinstance(text, str) or not text.strip():
        return ABSENCE_STATE

    free_entailment, free_contradiction = _nli_scores(
        text,
        FREE_HYPOTHESIS
    )

    busy_entailment, busy_contradiction = _nli_scores(
        text,
        BUSY_HYPOTHESIS
    )

    free_score = free_entailment + busy_contradiction
    busy_score = busy_entailment + free_contradiction

    return "FREE" if free_score > busy_score else ABSENCE_STATE