TokenTown –> Learn how LLM's work in a SimCity
TokenTown A language model laid out as a city, one token at a time About & accuracy Hide panel Pass prefill Layer 1 / 6 KV cache 0 tokens Generated 0 / 8 Prefill: every prompt token rides through together, in parallel. +−⤢ Details Press Run to start The city is idle. Type a prompt below and watch a single token make the round trip. reading stop: press Space to hold it here Residual stream idle 12 numbers standing in for the 4,096+ a real model carries. Blue is negative, warm is positive. Attention softmax over the KV cache Runs at the Attention Plaza. Next token logits → softmax Computed at the Vocabulary Stadium. Context 4 tokens Output the city of tokens| Districts click to fly there Tokenizer Docks Embedding Foundry Positional Beacon Pre-Norm Gate Attention Plaza KV Cache Warehouse Residual Bridge Feed-Forward Mill Lay
TokenTown A language model laid out as a city, one token at a time About & accuracy Hide panel Pass prefill Layer 1 / 6 KV cache 0 tokens Generated 0 / 8 Prefill: every prompt token rides through together, in parallel. +−⤢ Details Press Run to start The city is idle. Type a prompt below and watch a single token make the round trip. reading stop: press Space to hold it here Residual stream idle 12 numbers standing in for the 4,096+ a real model carries. Blue is negative, warm is positive. Attention softmax over the KV cache Runs at the Attention Plaza. Next token logits → softmax Computed at the Vocabulary Stadium. Context 4 tokens Output the city of tokens| Districts click to fly there Tokenizer Docks Embedding Foundry Positional Beacon Pre-Norm Gate Attention Plaza KV Cache Warehouse Residual Bridge Feed-Forward Mill Lay
er Counter Arch Final Norm Vocabulary Stadium The Sampler Output Plaza Feedback Highway Prompt Run ❚❚⇥⟲ ⚙ Speed 1.00× Layers 6 Temperature 0.80 Top-p 0.90 - [x] Follow - [x] Labels × What this is TokenTown is an isometric city where every district is one stage of a transformer language model. A convoy carries a hidden state along the roads: it is cut into tokens at the docks, cast into a vector at the foundry, stamped with its position, then driven around the layer ring (attention, residual, feed-forward, residual) once per layer, before the stadium turns it into a probability distribution and the sampler picks one token. That token drives back up the feedback highway and the whole city runs again. How much of it is real Genuinely computed, live, in your browser: the tokenizer split; the embedding lookup; sinusoidal positional encoding; LayerNorm; multi-head scaled dot-product attention with causal masking over a real growing KV cache; the residual adds; a GELU feed-forward; and temperature / top-p sampling. The bars on the truck are the actual vector. The beams over the warehouse are the actual softmax weights. Prefill really does process every prompt token at once while decode really does process only one. Scaled down: 12 dimensions instead of thousands, 2 attention heads instead of dozens, 2–12 layers instead of 80, and a vocabulary of a few hundred words instead of 100k+. Deliberately faked: the weights are random, and nothing here was trained, so a random-weight model would emit noise. To keep the output readable, the final logits blend the real hidden-state projection with a bigram prior built from a small fixed corpus. The attention scores are also sharpened, and given a small first-token ("sink") and recency bias, so the map looks like the patterns trained models actually produce. Treat the text this city writes as scenery; treat the mechanism as the lesson. Pacing The first time the convoy reaches a district it stops long enough to read that district's explanation, between 9 and 26 seconds depending on how much there is to say. A progress bar under the panel text shows how long the stop has left. Once every district has been explained the city runs at a watchable pace instead of a readable one, and the repeated layers fast-forward because they are the same road with different weights. Space holds any stop indefinitely, S steps one stage at a time, and the Speed slider scales everything, including the reading stops, from 0.4× to 8×. Reset (⟲) replays the slow tour from the beginning; Run keeps what you have already read. Controls * Space play / pause · S advance one stage · R reset and replay the tour · F follow camera · L labels * Drag to pan, scroll to zoom, click any district for its explanation. * The view starts close on the convoy and rides along with it. The ⤢ button on the left (or a double-click on the map) pulls back to the whole city; turning off Follow lets you wander on your own. Inspired by the idea behind PGSimCity (a city-shaped model of PostgreSQL); all code, art and copy here are original.
英伟达与微软牵头成立开放 AI 安全联盟，OpenAI、谷歌等未加入
英伟达于周一宣布，与微软、SpaceX、IBM 等科技公司共同组建“开放安全 AI 联盟”（Open Secure AI Alliance），旨在协作开发并共享开源 AI 安全工具。该联盟认为，面对日益强大的前沿模型可能带来的攻击威胁，只有采用开放的工具才能实现有效防御。此举是对业界不断升温的 AI 安全担忧的直接回应。值得注意的是，OpenAI、谷歌和 Anthropic 等头部 AI 公司并未加入这一联盟，使得该倡议在开放合作的同时也凸显了行业内部在安全路径上的分歧。 #英伟达 #微软 #AI安全 #开源 #联盟 #SpaceX #IBM #科技
英伟达于周一宣布，与微软、SpaceX、IBM 等科技公司共同组建“开放安全 AI 联盟”（Open Secure AI Alliance），旨在协作开发并共享开源 AI 安全工具。该联盟认为，面对日益强大的前沿模型可能带来的攻击威胁，只有采用开放的工具才能实现有效防御。此举是对业界不断升温的 AI 安全担忧的直接回应。值得注意的是，OpenAI、谷歌和 Anthropic 等头部 AI 公司并未加入这一联盟，使得该倡议在开放合作的同时也凸显了行业内部在安全路径上的分歧。 #英伟达 #微软 #AI安全 #开源 #联盟 #SpaceX #IBM #科技
开发者成功在16GB MacBook上复现BM25、稠密检索与SPLADE模型
检索增强生成（RAG）系统开发中，团队常对比BM25、稠密检索和SPLADE等基线方法，但很少有人真正在标准数据集上复现并验证其评分。一名开发者使用仅16GB内存的MacBook，借助Castorini研究组的Anserini和Pyserini开源工具包，在真实文档集合上完整复现了这三种检索模型。过程中遭遇了内存溢出、原生库崩溃以及模型API权限问题，但最终所有输出分数均与官方预期值完全吻合。实验表明，即使是消费级硬件也能运行这些基线，为RAG系统构建者提供了可复现的参考点：BM25的稀疏向量关键词评分无需训练数据，稠密检索通过神经网络生成语义嵌入，而SPLADE则结合两者优势。在优化提示词或更换大模型之前，应首先验证检索质量。 #检索模型 #BM25 #稠密检索 #SPLADE #RAG #开源工具 #机器学习 #实验复现
检索增强生成（RAG）系统开发中，团队常对比BM25、稠密检索和SPLADE等基线方法，但很少有人真正在标准数据集上复现并验证其评分。一名开发者使用仅16GB内存的MacBook，借助Castorini研究组的Anserini和Pyserini开源工具包，在真实文档集合上完整复现了这三种检索模型。过程中遭遇了内存溢出、原生库崩溃以及模型API权限问题，但最终所有输出分数均与官方预期值完全吻合。实验表明，即使是消费级硬件也能运行这些基线，为RAG系统构建者提供了可复现的参考点：BM25的稀疏向量关键词评分无需训练数据，稠密检索通过神经网络生成语义嵌入，而SPLADE则结合两者优势。在优化提示词或更换大模型之前，应首先验证检索质量。 #检索模型 #BM25 #稠密检索 #SPLADE #RAG #开源工具 #机器学习 #实验复现
思科Outshift提出“互联网认知”
思科旗下Outshift部门提出一种新型水平架构，旨在解决多智能体系统之间缺乏协调的根本问题。该架构包含两层：底层“互联网代理”（Internet of Agents）提供发现、身份验证和跨域消息传递的标准化协议，其开源项目AGNTCY已纳入Linux基金会；上层“互联网认知”（Internet of Cognition）作为语义层，使不同领域的智能体能够共享意图、上下文和推理，从而像人类团队一样自主协同解决从未见过的问题。Outshift高级副总裁Vijoy Pandey指出，传统AI聚焦于垂直扩展（更大模型、更多数据），但要让智能体跨系统、跨公司协作，必须转向水平扩展。当前多智能体系统在软件工程、药物发现等领域表现不佳，失败率高达41%–87%，其根源在于架构而非提示工程。这一新架构有望打破瓶颈，为医疗、科研等复杂场景提供真正的分布式决策能力，迈出通向人工超级智能的关键一步。 #人工智能 #多智能体 #超级智能 #思科 #Outshift #互联网认知 #开源 #AI #技术前沿
思科旗下Outshift部门提出一种新型水平架构，旨在解决多智能体系统之间缺乏协调的根本问题。该架构包含两层：底层“互联网代理”（Internet of Agents）提供发现、身份验证和跨域消息传递的标准化协议，其开源项目AGNTCY已纳入Linux基金会；上层“互联网认知”（Internet of Cognition）作为语义层，使不同领域的智能体能够共享意图、上下文和推理，从而像人类团队一样自主协同解决从未见过的问题。Outshift高级副总裁Vijoy Pandey指出，传统AI聚焦于垂直扩展（更大模型、更多数据），但要让智能体跨系统、跨公司协作，必须转向水平扩展。当前多智能体系统在软件工程、药物发现等领域表现不佳，失败率高达41%–87%，其根源在于架构而非提示工程。这一新架构有望打破瓶颈，为医疗、科研等复杂场景提供真正的分布式决策能力，迈出通向人工超级智能的关键一步。 #人工智能 #多智能体 #超级智能 #思科 #Outshift #互联网认知 #开源 #AI #技术前沿