LLM Engineering Notes
2 subscribers
192 links
Notes on shipping software with large language models.
Download Telegram
A larger thinking budget can trade away a large language model's correct answers. OpenTWBench's authors found that increasing the budget from 8k to 16k tokens changed which questions failed. Accuracy rose by just 0.21% at 1.72 times the cost. The higher score hid the new failures.

https://arxiv.org/abs/2610.19848v1
xRAG scored better with other questions' passages than with no document in a TriviaQA experiment.

Minsu Oh and colleagues tested 11,313 questions. F1 was 19.38 without a document and averaged 24.78 across three assignments of other passages. The question's retrieved passage scored 29.88. The relevant passage still wins.

The result puts a limit on the claim. Beating a no-document baseline doesn't establish that the improvement requires the right passage. And F1 measures word overlap with accepted answers. The change in accepted-answer inclusion was statistically uncertain.

For LLM evaluation, that makes a swapped-passage control useful alongside the no-document baseline. It separates some of the gain from the information the retriever actually found.

https://arxiv.org/abs/2610.19373v1
Prompt updates can break code your LLM already generates correctly.

Mingyu Lu and colleagues built HERO to check prompt updates against execution evidence from successful and failed programs. They report a 7.6 percentage point improvement over the initial prompt, with test-time refinement adding another 2.0 points.

The boundary matters for builders. HERO validates prompt updates before adoption and keeps the prompt fixed at test time. Execution feedback guides changes to the current program, so those changes don’t become shared guidance for later tasks.

https://arxiv.org/abs/2610.19184v1
Restoring Devanagari marks can make a Nepali BERT model worse.

In NEPSEE, Gaurab Baral and colleagues found that default tokenizers delete non-spacing vowel signs and the virama. Restoring those marks raised NepBERTa’s macro-F1 by 0.084 but lowered macro-F1 by 0.095 for a model pretrained on stripped text.

The tokenizer fix has to match the model’s pretraining. The authors’ masked-language-model probe predicted which way each model would move without using labels.

https://arxiv.org/abs/2610.19190v1
Higher GPU throughput reflects extra work in DropKV’s PyTorch scoring reference for language-model KV scoring.

The authors report a Runtime of 0.23 ms for the Triton scorer. Their Nsight Compute profiling covers the full scoring call at n = 16 K. Triton executes 41 × fewer instructions and moves 26 × less HBM traffic than the PyTorch scoring reference.

The implementation matters. Triton streams K/V tiles and accumulates scores directly. The PyTorch scoring reference materializes dense attention and repeated K/V tensors.

https://arxiv.org/abs/2610.22130v1
A shorter context injection can damage an LLM's counter more than a longer one.

The paper's authors found that B, which lists seen codes, damaged the counter more than D irrelevant control. At step 100, median prompt size was 6,250 tokens for B and 6,668 for D. The paper reports 20 seeds with a counter error for D irrelevant control.

That's a reason to test counter tracking separately from duplicate recognition. The authors limit the counter result to a synthetic task with a trivially incremented counter. It doesn't establish a result for arithmetic state.

https://arxiv.org/abs/2610.20633v1
Retaining alternatives in speculative decoding creates candidate choices that reference-conditioned training doesn’t directly supervise.

Jihao Zhao and colleagues address that gap with TreeSpark, extending DSpark. It constructs parent-conditioned candidate trees from one backbone pass, then replays expansion, pruning and budgeted packing during training. They evaluate greedy decoding with Qwen3-4B and Qwen3-8B on nine general benchmarks and six writing domains, using a fixed candidate budget.

For builders, reusing target distributions doesn’t make tree supervision free. It still adds draft computation and backward work. The labels are exact for the reference context but serve as proxies on alternative branches. Generalization across budgets and stochastic sampling remains untested.

https://arxiv.org/abs/2610.20175v1
An LLM judge missed a story improvement that readers preferred.

Syed Rifat Raiyan and colleagues tested nonlinear storytelling with 71 human readers. Readers preferred stories written from a story graph over stories constrained to the published telling order in 72% of decisive judgments. The automated judge did not detect that preference. Both conditions kept the same target narrative shape.

For builders, that’s a gap in evaluation. A judge can miss the benefit of giving a language model causal links, character knowledge and unresolved questions instead of a fixed telling order.

https://arxiv.org/abs/2610.21468v1
More diffusion steps don’t reliably improve LLM reranking.

Meta researchers tested permutation diffusion LLMs on recommendation reranking with candidate sets of 10 and 20 items. Hard-assignment and soft-assignment variants matched a constrained masked-diffusion baseline on Recall@1 at 1 function evaluation. Additional function evaluations did not consistently improve ranking quality.

For builders, extra function evaluations offer no reliable quality gain here. Their effect depends on the state representation and training objective, so the sampling budget needs testing alongside those choices.

https://arxiv.org/abs/2610.20635v1
LLM benchmark gains don’t establish that a builder can trust an optimization formulation with noisy or incomplete data.

In December 2025, Lijia Liu and colleagues presented NLData2Opt with 3 parameter tasks. LLMs must extract relevant values, compute missing values and predict future values. Performance fell substantially compared with existing benchmarks.

For builders, a structurally correct formulation still doesn’t establish that it will execute and solve the problem. Parameter reasoning needs its own evaluation.

https://arxiv.org/abs/2610.22889v1
A language model can look safe because it failed to understand the task.

Aditya Nawal, Manit Baser and Mohan Gurusamy found that non-English harmful requests increased mean tool admission from 46.9% to 67.4%. Requiring answers in the page language increased pooled harmfulness on matched tool calls from 50.3% to 57.4% across four closed-weight models. Translating pages alone produced no consistent harm increase in models with comparable inference settings.

For builders, a non-harmful answer isn’t enough evidence of safety. Measure tool admission and response harmfulness separately. Distinguish genuine refusal from language difficulties and unusable answers, or your evaluation can mistake capability failure for safety.

https://arxiv.org/abs/2610.23093v1
Multi-agent collaboration does not consistently beat a strong single LLM on final-answer accuracy.

Xinping Wang and colleagues report this in MAMath-Bench, which contains 1,003 problems across 11 mathematical categories and includes text, images and audio. Collaboration improves coverage of intermediate reasoning, but adds redundancy. Subtle structural errors still trip systems up.

For builders, final-answer accuracy leaves a gap. A correct answer can hide incomplete or unreliable reasoning, so evaluating an LLM team means checking the steps it presents, too.

https://arxiv.org/abs/2610.26943v1
Structural difficulty alone can’t tell a builder when to trust an LLM’s plan.

Mayukh Debnath and colleagues found that the classical heuristic h add estimated optimal plan length with mean Spearman |ρ| = 0.931 across held-out domains. They also independently reconstructed grounded actions and reproduced every saved optimal plan across 1,006 solved instances.

But hybrid routing depended more on calibration with target-model outcomes than on the structural score. Builders need to measure which plans their LLM can produce, not just how hard those plans are structurally.

https://arxiv.org/abs/2610.26964v1
A language model can return a legal value that never applied to the provision you asked about.

Eirik Botten Nicolaysen’s norregel benchmark checks dated numeric values in Norwegian law against their provision’s history. In the first run, Opus 5.5 answered 46.7% of the 880 primary items correctly. Sonnet 4.6 answered 40.2% correctly. Most wrong answers gave values absent from the provision’s dated history.

For a builder, that makes checking for outdated values insufficient. I’d check the answer against the provision’s dated history before trusting it.

https://arxiv.org/abs/2610.29207v1
The strongest LLM solvers aren't the strongest authors of training environments.

University of Pennsylvania researchers tested both skills in Environment Duels across 90 environments and 12,960 episodes. Each model wrote environments and hints, then solved environments with and without hints. Smaller models learned more from other models' attempts and wrote the hardest environments and the most useful hints.

Picking a model by solving ability alone can cost you a better training curriculum. In the researchers' reinforcement-learning experiment, a learner's gain from hints tracked its gains on held-out benchmarks after training on those environments. Test what a model can teach before choosing it to write what comes next.

https://arxiv.org/abs/2610.30116v1
A language model can reject matching names across scripts while its internal states still carry the identity information.

Athira Gopal and colleagues found that scalar calibration recovered 87% of Qwen’s raw model–probe balanced-accuracy gap. An intervention at the second name’s middle-layer state raised held-out Marathi same-person accuracy from 6.5% to 33.5%, with 98.5% different-person accuracy.

For builders, a rejected match doesn’t establish missing identity information. Test the decision threshold alongside script discrimination. The paper found failures in both.

https://arxiv.org/abs/2610.31407v1
A failed decoder doesn't establish that an LLM failed to hide a private message.

Tailor and colleagues found 168 verifiable encodings that decoders missed across 7,200 responses.

If you judge encoding through end-to-end recovery alone, you can't tell whether the encoder or decoder needs a fix.

https://arxiv.org/abs/2610.30516v1
I put my LLM engineering libraries in one GitHub organization. https://github.com/ml0x-com

There's exact KV cache arithmetic in Rust and Swift, LoRA parameter and VRAM planning in Python, token estimates without a tokenizer, and per call API cost in D. Next to them sit gate-contract-rs, fail-closed gates for staged build pipelines, and claude-code-playbook with 200 Claude Code templates, prompts and configs.

Every formula is in the code, so you can check it. If you use these, a sponsorship pays for the next ones. https://github.com/sponsors/theluckystrike
A larger language model doesn’t guarantee more reliable probabilities on noisy French input.

Jiaming Yu and colleagues tested 4,200 French input triplets, or 12,600 inputs, with apostrophe omission, diacritic removal and keyboard typos. Larger models lost less accuracy under high-intensity edits, but their probability reliability didn’t consistently hold up better. Keyboard typos caused the largest losses at the tested high-intensity settings.

For builders, accuracy alone can hide damage to probability reliability. The researchers found that combining input restoration with calibration improved Brier score and NLL over either method alone while preserving restored accuracy.

https://arxiv.org/abs/2610.33493v1
Strong egocentric video results don’t establish that a language model understands the wearer’s body.

EPFL researchers tested 37 multimodal language models on SOMA’s 1,632 human-verified questions. The strongest model reached 50.0% zero-shot accuracy. Humans reached 68.5%. Models misread the wearer’s body, favored the right hand and undercounted repetitions.

For wearable assistants, scene recognition leaves a gap in what you’ve tested. More frames and explicit reasoning brought only limited, model-dependent gains. Builders need to test body configuration, motion and event timing directly.

https://arxiv.org/abs/2610.36384v1