From Byte to Token: Why Karpathy's View of LLMs Transforms the Way We Think About Computing
LLMs are not merely "better chatbots." They act like a new computational layer, with tokens as the data format and the context window as working memory.
If you still treat LLMs like an app, you are missing the actual shift. Andrej Karpathy describes LLMs as a new kind of computer programmed in natural language. This is less science fiction than a pragmatic model for understanding what is currently happening.
Karpathy is no bystander in this debate. He was a Research Scientist and Founding Member at OpenAI and later led the teams at Tesla responsible for computer vision, data labeling, training, and deployment for Autopilot. Today, he is building AI-native education with Eureka Labs and continues to publish practical and technical content on LLMs.
His core point is simple. We are getting a new computing layer, where a language model acts as the central processing unit and "programs" take the form of prompts or instructions. Karpathy also calls this "Software 3.0," because you program a model in English instead of Python or C++.
The metaphor of "LLM as CPU" is less about technical mechanics and more about utility. Classical CPUs are deterministic; they execute precisely defined instructions. An LLM is probabilistic; it produces tokens based on probabilities derived from training and context. This explains why two identical requests do not necessarily yield identical answers.
The second part of the analogy is the data format. While classical systems process bytes, LLMs operate on tokens—text fragments produced by tokenizers. Tokenization is a distinct step in the LLM pipeline, and many "strange" model behaviors are directly related to it. Wikipedia also summarizes tokenization as a foundational preliminary step required for text to be processed as numbers in the first place.
And then comes memory. An LLM only "sees" what is present within the context window. This context window functions somewhat like a temporary workspace, holding the relevant history, documents, or tool outputs during generation. At the same time, it is not a true RAM analogy with arbitrary reading and writing, but rather a limited visual cone reconstructed with every response.
From this perspective emerges the idea of an "LLM OS." This describes a system in which core functions such as interaction, task management, and system control are driven heavily by an LLM, with primary operation handled via natural language. This is not a new Linux distribution. It is a new way to "orchestrate" workflows using language, tools, and context.
Distinguishing between concept and product is essential. For instance, there is the open-source project Eclipse LMOS, which orchestrates agents in enterprise environments and even offers a structured, versionable description of agent behavior with ADL (Agent Definition Language). That represents a concrete stack for agentic AI. Karpathy's "LLM as an OS" mindset is more the mental model into which such platforms neatly fit.
| Classical Computer Science | LLM Paradigm | What This Means in Practice |
| Bytes as the basic unit | Tokens as the basic unit | Text is "compiled" via tokenizers. Prompt design and token budgets suddenly become relevant. |
| CPU executes deterministic instructions | Model generates text probabilistically | You need verification, testing, and guardrails, because "sounds good" does not equal "is true." |
| RAM as working memory | Context window as temporary workspace | Good systems actively manage context. RAG, memory strategies, and tool outputs belong in system design. |
| OS coordinates programs | LLM OS coordinates tasks, tools, and agents | Natural language orchestration becomes the UX layer. Agent frameworks move closer to the "operating system." |
The catch with this new freedom is a new class of errors. LLMs can produce plausible-sounding yet false statements, and they inherit bias from training data. That is precisely why Karpathy emphasizes the "generation-verification" principle in multiple contexts: generate first, verify next, and keep humans meaningfully in the loop.
This New Computing Is Not Magic. But It Is Different
Karpathy's perspective is valuable because it does not attempt to explain LLMs as a mere "feature," but rather as a new computational layer. Tokens instead of bytes, context windows instead of classical RAM logic, and probabilistic outputs instead of deterministic execution.