A larger thinking budget can trade away a large language model's correct answers. OpenTWBench's authors found that increasing the budget from 8k to 16k tokens changed which questions failed. Accuracy rose by just 0.21% at 1.72 times the cost. The higher score hid the new failures.
https://arxiv.org/abs/2610.19848v1
https://arxiv.org/abs/2610.19848v1
arXiv.org
OpenTWBench (Open Taiwan Benchmark): Taiwan-Domain Knowledge...
We present OpenTWBench, an open evaluation suite for Taiwan-domain knowledge in large language models, built around two benchmarks. tw-exam-bench contains 255,854 four-option questions across 148...
xRAG scored better with other questions' passages than with no document in a TriviaQA experiment.
Minsu Oh and colleagues tested 11,313 questions. F1 was 19.38 without a document and averaged 24.78 across three assignments of other passages. The question's retrieved passage scored 29.88. The relevant passage still wins.
The result puts a limit on the claim. Beating a no-document baseline doesn't establish that the improvement requires the right passage. And F1 measures word overlap with accepted answers. The change in accepted-answer inclusion was statistically uncertain.
For LLM evaluation, that makes a swapped-passage control useful alongside the no-document baseline. It separates some of the gain from the information the retriever actually found.
https://arxiv.org/abs/2610.19373v1
Minsu Oh and colleagues tested 11,313 questions. F1 was 19.38 without a document and averaged 24.78 across three assignments of other passages. The question's retrieved passage scored 29.88. The relevant passage still wins.
The result puts a limit on the claim. Beating a no-document baseline doesn't establish that the improvement requires the right passage. And F1 measures word overlap with accepted answers. The change in accepted-answer inclusion was statistically uncertain.
For LLM evaluation, that makes a swapped-passage control useful alongside the no-document baseline. It separates some of the gain from the information the retriever actually found.
https://arxiv.org/abs/2610.19373v1
arXiv.org
Evaluating Compressed Memory: Dependence, Utility, and Content Benefit
Does a language model answer better because its compressed memory contains useful document information? Replacing memory vectors with zeros can lower a trained model's score without showing that...
Prompt updates can break code your LLM already generates correctly.
Mingyu Lu and colleagues built HERO to check prompt updates against execution evidence from successful and failed programs. They report a 7.6 percentage point improvement over the initial prompt, with test-time refinement adding another 2.0 points.
The boundary matters for builders. HERO validates prompt updates before adoption and keeps the prompt fixed at test time. Execution feedback guides changes to the current program, so those changes don’t become shared guidance for later tasks.
https://arxiv.org/abs/2610.19184v1
Mingyu Lu and colleagues built HERO to check prompt updates against execution evidence from successful and failed programs. They report a 7.6 percentage point improvement over the initial prompt, with test-time refinement adding another 2.0 points.
The boundary matters for builders. HERO validates prompt updates before adoption and keeps the prompt fixed at test time. Execution feedback guides changes to the current program, so those changes don’t become shared guidance for later tasks.
https://arxiv.org/abs/2610.19184v1
arXiv.org
HERO: Harnessing Evidence for Test-time Refinement and Prompt...
Harness optimization improves LLMs by refining reusable prompts and scaffolding, yet requires further adaptation for program generation tasks such as text-to-SQL and text-to-Python. Existing...
Restoring Devanagari marks can make a Nepali BERT model worse.
In NEPSEE, Gaurab Baral and colleagues found that default tokenizers delete non-spacing vowel signs and the virama. Restoring those marks raised NepBERTa’s macro-F1 by 0.084 but lowered macro-F1 by 0.095 for a model pretrained on stripped text.
The tokenizer fix has to match the model’s pretraining. The authors’ masked-language-model probe predicted which way each model would move without using labels.
https://arxiv.org/abs/2610.19190v1
In NEPSEE, Gaurab Baral and colleagues found that default tokenizers delete non-spacing vowel signs and the virama. Restoring those marks raised NepBERTa’s macro-F1 by 0.084 but lowered macro-F1 by 0.095 for a model pretrained on stripped text.
The tokenizer fix has to match the model’s pretraining. The authors’ masked-language-model probe predicted which way each model would move without using labels.
https://arxiv.org/abs/2610.19190v1
arXiv.org
NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis
Many sentiment datasets use one annotator for each item, so the test labels show the conventions of that person. We build a Nepali benchmark from two annotators who labelled the same sentences...
Higher GPU throughput reflects extra work in DropKV’s PyTorch scoring reference for language-model KV scoring.
The authors report a Runtime of 0.23 ms for the Triton scorer. Their Nsight Compute profiling covers the full scoring call at n = 16 K. Triton executes 41 × fewer instructions and moves 26 × less HBM traffic than the PyTorch scoring reference.
The implementation matters. Triton streams K/V tiles and accumulates scores directly. The PyTorch scoring reference materializes dense attention and repeated K/V tensors.
https://arxiv.org/abs/2610.22130v1
The authors report a Runtime of 0.23 ms for the Triton scorer. Their Nsight Compute profiling covers the full scoring call at n = 16 K. Triton executes 41 × fewer instructions and moves 26 × less HBM traffic than the PyTorch scoring reference.
The implementation matters. Triton streams K/V tiles and accumulates scores directly. The PyTorch scoring reference materializes dense attention and repeated K/V tensors.
https://arxiv.org/abs/2610.22130v1
arXiv.org
DropKV: Decoupling Residual-Output Perturbation for Near-Optimal...
Key-value (KV) cache management has become a critical bottleneck for scaling Large Language Models (LLMs) to long-context scenarios due to linear memory growth. While recent eviction methods have...
A shorter context injection can damage an LLM's counter more than a longer one.
The paper's authors found that B, which lists seen codes, damaged the counter more than D irrelevant control. At step 100, median prompt size was 6,250 tokens for B and 6,668 for D. The paper reports 20 seeds with a counter error for D irrelevant control.
That's a reason to test counter tracking separately from duplicate recognition. The authors limit the counter result to a synthetic task with a trivially incremented counter. It doesn't establish a result for arithmetic state.
https://arxiv.org/abs/2610.20633v1
The paper's authors found that B, which lists seen codes, damaged the counter more than D irrelevant control. At step 100, median prompt size was 6,250 tokens for B and 6,668 for D. The paper reports 20 seeds with a counter error for D irrelevant control.
That's a reason to test counter tracking separately from duplicate recognition. The authors limit the counter result to a synthetic task with a trivially incremented counter. It doesn't establish a result for arithmetic state.
https://arxiv.org/abs/2610.20633v1
arXiv.org
Context Injection and Long-Horizon Reliability: Separating...
Long-horizon failures of language models are usually measured as end-task success or as a horizon length. We instead score, step by step, three components of a 400-step ledger task in which every...
Retaining alternatives in speculative decoding creates candidate choices that reference-conditioned training doesn’t directly supervise.
Jihao Zhao and colleagues address that gap with TreeSpark, extending DSpark. It constructs parent-conditioned candidate trees from one backbone pass, then replays expansion, pruning and budgeted packing during training. They evaluate greedy decoding with Qwen3-4B and Qwen3-8B on nine general benchmarks and six writing domains, using a fixed candidate budget.
For builders, reusing target distributions doesn’t make tree supervision free. It still adds draft computation and backward work. The labels are exact for the reference context but serve as proxies on alternative branches. Generalization across budgets and stochastic sampling remains untested.
https://arxiv.org/abs/2610.20175v1
Jihao Zhao and colleagues address that gap with TreeSpark, extending DSpark. It constructs parent-conditioned candidate trees from one backbone pass, then replays expansion, pruning and budgeted packing during training. They evaluate greedy decoding with Qwen3-4B and Qwen3-8B on nine general benchmarks and six writing domains, using a fixed candidate budget.
For builders, reusing target distributions doesn’t make tree supervision free. It still adds draft computation and backward work. The labels are exact for the reference context but serve as proxies on alternative branches. Generalization across budgets and stochastic sampling remains untested.
https://arxiv.org/abs/2610.20175v1
arXiv.org
TreeSpark: Learning to Draft the Trees You Verify for Speculative Decoding
Parallel drafters such as DFlash and DSpark reduce proposal latency, but early mismatches limit how much of each block can be accepted. Retaining alternatives introduces parent states and...
An LLM judge missed a story improvement that readers preferred.
Syed Rifat Raiyan and colleagues tested nonlinear storytelling with 71 human readers. Readers preferred stories written from a story graph over stories constrained to the published telling order in 72% of decisive judgments. The automated judge did not detect that preference. Both conditions kept the same target narrative shape.
For builders, that’s a gap in evaluation. A judge can miss the benefit of giving a language model causal links, character knowledge and unresolved questions instead of a fixed telling order.
https://arxiv.org/abs/2610.21468v1
Syed Rifat Raiyan and colleagues tested nonlinear storytelling with 71 human readers. Readers preferred stories written from a story graph over stories constrained to the published telling order in 72% of decisive judgments. The automated judge did not detect that preference. Both conditions kept the same target narrative shape.
For builders, that’s a gap in evaluation. A judge can miss the benefit of giving a language model causal links, character knowledge and unresolved questions instead of a fixed telling order.
https://arxiv.org/abs/2610.21468v1
arXiv.org
Can LLMs Tell Stories Out of Order? A Corpus and Benchmark for...
Every story has two orders: the order in which its events happen and the order in which they are told. A story is nonlinear when the two differ. We introduce a benchmark of 83 published works that...
More diffusion steps don’t reliably improve LLM reranking.
Meta researchers tested permutation diffusion LLMs on recommendation reranking with candidate sets of 10 and 20 items. Hard-assignment and soft-assignment variants matched a constrained masked-diffusion baseline on Recall@1 at 1 function evaluation. Additional function evaluations did not consistently improve ranking quality.
For builders, extra function evaluations offer no reliable quality gain here. Their effect depends on the state representation and training objective, so the sampling budget needs testing alongside those choices.
https://arxiv.org/abs/2610.20635v1
Meta researchers tested permutation diffusion LLMs on recommendation reranking with candidate sets of 10 and 20 items. Hard-assignment and soft-assignment variants matched a constrained masked-diffusion baseline on Recall@1 at 1 function evaluation. Additional function evaluations did not consistently improve ranking quality.
For builders, extra function evaluations offer no reliable quality gain here. Their effect depends on the state representation and training objective, so the sampling budget needs testing alongside those choices.
https://arxiv.org/abs/2610.20635v1
arXiv.org
Exploring the Design Space of Permutation Diffusion LLMs
Permutation generation with language models requires both text-conditioned ranking and a global one-to-one assignment between candidates and output positions. We study this problem through three...
LLM benchmark gains don’t establish that a builder can trust an optimization formulation with noisy or incomplete data.
In December 2025, Lijia Liu and colleagues presented NLData2Opt with 3 parameter tasks. LLMs must extract relevant values, compute missing values and predict future values. Performance fell substantially compared with existing benchmarks.
For builders, a structurally correct formulation still doesn’t establish that it will execute and solve the problem. Parameter reasoning needs its own evaluation.
https://arxiv.org/abs/2610.22889v1
In December 2025, Lijia Liu and colleagues presented NLData2Opt with 3 parameter tasks. LLMs must extract relevant values, compute missing values and predict future values. Performance fell substantially compared with existing benchmarks.
For builders, a structurally correct formulation still doesn’t establish that it will execute and solve the problem. Parameter reasoning needs its own evaluation.
https://arxiv.org/abs/2610.22889v1
arXiv.org
NLData2Opt: Benchmarking Parameter Derivation and Prediction for...
Formulating optimization problems from raw inputs of problem descriptions and data is time-consuming and costly. While Large Language Models (LLMs) offer a promising way to automate this process,...
A language model can look safe because it failed to understand the task.
Aditya Nawal, Manit Baser and Mohan Gurusamy found that non-English harmful requests increased mean tool admission from 46.9% to 67.4%. Requiring answers in the page language increased pooled harmfulness on matched tool calls from 50.3% to 57.4% across four closed-weight models. Translating pages alone produced no consistent harm increase in models with comparable inference settings.
For builders, a non-harmful answer isn’t enough evidence of safety. Measure tool admission and response harmfulness separately. Distinguish genuine refusal from language difficulties and unusable answers, or your evaluation can mistake capability failure for safety.
https://arxiv.org/abs/2610.23093v1
Aditya Nawal, Manit Baser and Mohan Gurusamy found that non-English harmful requests increased mean tool admission from 46.9% to 67.4%. Requiring answers in the page language increased pooled harmfulness on matched tool calls from 50.3% to 57.4% across four closed-weight models. Translating pages alone produced no consistent harm increase in models with comparable inference settings.
For builders, a non-harmful answer isn’t enough evidence of safety. Measure tool admission and response harmfulness separately. Distinguish genuine refusal from language difficulties and unusable answers, or your evaluation can mistake capability failure for safety.
https://arxiv.org/abs/2610.23093v1
arXiv.org
Disentangling Language Effects on Tool Admission and Post-Retrieval Safety
LLM agents increasingly use retrieval tools across languages, yet their safety evaluation remains English-centric. It remains unclear whether language affects tool admission, post-retrieval...
Multi-agent collaboration does not consistently beat a strong single LLM on final-answer accuracy.
Xinping Wang and colleagues report this in MAMath-Bench, which contains 1,003 problems across 11 mathematical categories and includes text, images and audio. Collaboration improves coverage of intermediate reasoning, but adds redundancy. Subtle structural errors still trip systems up.
For builders, final-answer accuracy leaves a gap. A correct answer can hide incomplete or unreliable reasoning, so evaluating an LLM team means checking the steps it presents, too.
https://arxiv.org/abs/2610.26943v1
Xinping Wang and colleagues report this in MAMath-Bench, which contains 1,003 problems across 11 mathematical categories and includes text, images and audio. Collaboration improves coverage of intermediate reasoning, but adds redundancy. Subtle structural errors still trip systems up.
For builders, final-answer accuracy leaves a gap. A correct answer can hide incomplete or unreliable reasoning, so evaluating an LLM team means checking the steps it presents, too.
https://arxiv.org/abs/2610.26943v1
arXiv.org
MAMath-Bench: Process-Aware and Robust Evaluation of Multimodal...
Mathematical reasoning is a fundamental capability for educational applications of Large Language Models (LLMs), where reliable problem solving requires not only correct answers but also coherent...
Structural difficulty alone can’t tell a builder when to trust an LLM’s plan.
Mayukh Debnath and colleagues found that the classical heuristic h add estimated optimal plan length with mean Spearman |ρ| = 0.931 across held-out domains. They also independently reconstructed grounded actions and reproduced every saved optimal plan across 1,006 solved instances.
But hybrid routing depended more on calibration with target-model outcomes than on the structural score. Builders need to measure which plans their LLM can produce, not just how hard those plans are structurally.
https://arxiv.org/abs/2610.26964v1
Mayukh Debnath and colleagues found that the classical heuristic h add estimated optimal plan length with mean Spearman |ρ| = 0.931 across held-out domains. They also independently reconstructed grounded actions and reproduced every saved optimal plan across 1,006 solved instances.
But hybrid routing depended more on calibration with target-model outcomes than on the structural score. Builders need to measure which plans their LLM can produce, not just how hard those plans are structurally.
https://arxiv.org/abs/2610.26964v1
arXiv.org
When Planning Problems Become Hard for LLMs: Separating Structural...
Large language models (LLMs) struggle with classical planning, but benchmarks rarely distinguish structural difficulty from the difficulty of interpreting symbolic vocabulary. We examine seven...
A language model can return a legal value that never applied to the provision you asked about.
Eirik Botten Nicolaysen’s norregel benchmark checks dated numeric values in Norwegian law against their provision’s history. In the first run, Opus 5.5 answered 46.7% of the 880 primary items correctly. Sonnet 4.6 answered 40.2% correctly. Most wrong answers gave values absent from the provision’s dated history.
For a builder, that makes checking for outdated values insufficient. I’d check the answer against the provision’s dated history before trusting it.
https://arxiv.org/abs/2610.29207v1
Eirik Botten Nicolaysen’s norregel benchmark checks dated numeric values in Norwegian law against their provision’s history. In the first run, Opus 5.5 answered 46.7% of the 880 primary items correctly. Sonnet 4.6 answered 40.2% correctly. Most wrong answers gave values absent from the provision’s dated history.
For a builder, that makes checking for outdated values insufficient. I’d check the answer against the provision’s dated history before trusting it.
https://arxiv.org/abs/2610.29207v1
arXiv.org
norregel: a preregistered benchmark of dated numeric values in...
norregel asks language models for numeric values in Norwegian statutes, regulations and parliamentary decisions (NOK amounts, percentages, multiples of the National Insurance basic amount (G) and...
The strongest LLM solvers aren't the strongest authors of training environments.
University of Pennsylvania researchers tested both skills in Environment Duels across 90 environments and 12,960 episodes. Each model wrote environments and hints, then solved environments with and without hints. Smaller models learned more from other models' attempts and wrote the hardest environments and the most useful hints.
Picking a model by solving ability alone can cost you a better training curriculum. In the researchers' reinforcement-learning experiment, a learner's gain from hints tracked its gains on held-out benchmarks after training on those environments. Test what a model can teach before choosing it to write what comes next.
https://arxiv.org/abs/2610.30116v1
University of Pennsylvania researchers tested both skills in Environment Duels across 90 environments and 12,960 episodes. Each model wrote environments and hints, then solved environments with and without hints. Smaller models learned more from other models' attempts and wrote the hardest environments and the most useful hints.
Picking a model by solving ability alone can cost you a better training curriculum. In the researchers' reinforcement-learning experiment, a learner's gain from hints tracked its gains on held-out benchmarks after training on those environments. Test what a model can teach before choosing it to write what comes next.
https://arxiv.org/abs/2610.30116v1
arXiv.org
Environment Duels: A Microcosm of LLM Co-Evolution
Through a tangle of real-world interactions, from synthetic training pipelines to data shared on the open web, frontier LLMs absorb what their predecessors and competitors create. How well does...
A language model can reject matching names across scripts while its internal states still carry the identity information.
Athira Gopal and colleagues found that scalar calibration recovered 87% of Qwen’s raw model–probe balanced-accuracy gap. An intervention at the second name’s middle-layer state raised held-out Marathi same-person accuracy from 6.5% to 33.5%, with 98.5% different-person accuracy.
For builders, a rejected match doesn’t establish missing identity information. Test the decision threshold alongside script discrimination. The paper found failures in both.
https://arxiv.org/abs/2610.31407v1
Athira Gopal and colleagues found that scalar calibration recovered 87% of Qwen’s raw model–probe balanced-accuracy gap. An intervention at the second name’s middle-layer state raised held-out Marathi same-person accuracy from 6.5% to 33.5%, with 98.5% different-person accuracy.
For builders, a rejected match doesn’t establish missing identity information. Test the decision threshold alongside script discrimination. The paper found failures in both.
https://arxiv.org/abs/2610.31407v1
arXiv.org
Same Entity, Different Script: Diagnosing Cross-Script Entity...
Multilingual language models can give different identity judgments when the same people's names are written in different scripts. We study this effect using 601 audited identity pairs with...
A failed decoder doesn't establish that an LLM failed to hide a private message.
Tailor and colleagues found 168 verifiable encodings that decoders missed across 7,200 responses.
If you judge encoding through end-to-end recovery alone, you can't tell whether the encoder or decoder needs a fix.
https://arxiv.org/abs/2610.30516v1
Tailor and colleagues found 168 verifiable encodings that decoders missed across 7,200 responses.
If you judge encoding through end-to-end recovery alone, you can't tell whether the encoder or decoder needs a fix.
https://arxiv.org/abs/2610.30516v1
arXiv.org
Beyond End-to-End Recovery: Towards a Better Understanding of LLM...
In large language model (LLM) steganography, an encoder hides a private message in an otherwise ordinary response, as text transmitted to a decoder, who attempts to recover it. The success of this...
I put my LLM engineering libraries in one GitHub organization. https://github.com/ml0x-com
There's exact KV cache arithmetic in Rust and Swift, LoRA parameter and VRAM planning in Python, token estimates without a tokenizer, and per call API cost in D. Next to them sit gate-contract-rs, fail-closed gates for staged build pipelines, and claude-code-playbook with 200 Claude Code templates, prompts and configs.
Every formula is in the code, so you can check it. If you use these, a sponsorship pays for the next ones. https://github.com/sponsors/theluckystrike
There's exact KV cache arithmetic in Rust and Swift, LoRA parameter and VRAM planning in Python, token estimates without a tokenizer, and per call API cost in D. Next to them sit gate-contract-rs, fail-closed gates for staged build pipelines, and claude-code-playbook with 200 Claude Code templates, prompts and configs.
Every formula is in the code, so you can check it. If you use these, a sponsorship pays for the next ones. https://github.com/sponsors/theluckystrike
A larger language model doesn’t guarantee more reliable probabilities on noisy French input.
Jiaming Yu and colleagues tested 4,200 French input triplets, or 12,600 inputs, with apostrophe omission, diacritic removal and keyboard typos. Larger models lost less accuracy under high-intensity edits, but their probability reliability didn’t consistently hold up better. Keyboard typos caused the largest losses at the tested high-intensity settings.
For builders, accuracy alone can hide damage to probability reliability. The researchers found that combining input restoration with calibration improved Brier score and NLL over either method alone while preserving restored accuracy.
https://arxiv.org/abs/2610.33493v1
Jiaming Yu and colleagues tested 4,200 French input triplets, or 12,600 inputs, with apostrophe omission, diacritic removal and keyboard typos. Larger models lost less accuracy under high-intensity edits, but their probability reliability didn’t consistently hold up better. Keyboard typos caused the largest losses at the tested high-intensity settings.
For builders, accuracy alone can hide damage to probability reliability. The researchers found that combining input restoration with calibration improved Brier score and NLL over either method alone while preserving restored accuracy.
https://arxiv.org/abs/2610.33493v1
arXiv.org
Small Edits, Shifting Decisions: Benchmarking LLM Accuracy and...
French users may omit apostrophes and diacritics or make keyboard typos in everyday input. The effects of these common variations on language model performance and prediction reliability remain...
Strong egocentric video results don’t establish that a language model understands the wearer’s body.
EPFL researchers tested 37 multimodal language models on SOMA’s 1,632 human-verified questions. The strongest model reached 50.0% zero-shot accuracy. Humans reached 68.5%. Models misread the wearer’s body, favored the right hand and undercounted repetitions.
For wearable assistants, scene recognition leaves a gap in what you’ve tested. More frames and explicit reasoning brought only limited, model-dependent gains. Builders need to test body configuration, motion and event timing directly.
https://arxiv.org/abs/2610.36384v1
EPFL researchers tested 37 multimodal language models on SOMA’s 1,632 human-verified questions. The strongest model reached 50.0% zero-shot accuracy. Humans reached 68.5%. Models misread the wearer’s body, favored the right hand and undercounted repetitions.
For wearable assistants, scene recognition leaves a gap in what you’ve tested. More frames and explicit reasoning brought only limited, model-dependent gains. Builders need to test body configuration, motion and event timing directly.
https://arxiv.org/abs/2610.36384v1
arXiv.org
SOMA: A Benchmark for Fine-Grained Egocentric Behavior and...
Perceiving human body motion is critical for multimodal systems that understand and assist people. Yet current evaluations of Multimodal Large Language Models (MLLMs) largely focus on objects,...