Designing LLM Systems PART 1 Foundations Chapter 1 AI Systems Architecture The Real Currency of AI
What this covers
#Mathematics #AppliedMath#EngineeringMath #MathEducation #LinearAlgebra #Calculus #ComplexNumbers #MathematicalPhysics #STEM #LearnMath #FourierTransform #FFT #SignalProcessing #DigitalSignalProcessing #DSP #FrequencyDomain #TimeDomain #FrequencyAnalysis #Harmonics #WaveAnalysis
#Python #NumPy #SciPy #DataScience #MachineLearning #Coding #PythonProgramming #AI #Algorithm #OpenSource #QuantTrading #AlgorithmicTrading #QuantFinance #FinancialEngineering #MarketCycles #CycleAnalysis #StockMarket #TradingStrategy #TimeSeries #MarketAnalysis #LearnFourier #MathForEveryone #ScienceEducation #Engineering #LearnEngineering #EducationalContent #TechEducation #ExplainedSimply #STEMLearning #Knowledge def system_pipeline(query): # Step 1: Tokenization happens implicitly tokens = tokenize(query)
# Step 2: Routing intent = router(tokens)
# Step 3: Planning plan = planner(tokens)
# Step 4: Retrieval context = retrieve(tokens)
# Step 5: LLM generation response = llm(tokens, context)
return response
Source description (no synthesized summary yet).
AI systems operate as token-based economies where intelligence emerges not from word understanding but from sophisticated probabilistic prediction and efficient token management across retrieval, routing, and generation stages.
- Tokens are the fundamental currency AI processes—not words or sentences, but discrete mathematical units that all system operations cost and consume
- The core LLM task is predicting the next token in a sequence given prior tokens, repeated iteratively to generate complete answers
- System design becomes an optimization problem: allocating finite token budgets (context windows) across retrieval, instructions, background facts, and generation to minimize hallucination and maximize output quality
This asset isn't compiled yet
You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.
AI systems do not process language the way humans do—they break sentences into discrete tokens rather than reading sentences fluidly, fundamentally changing how they parse meaning.
“the real language these powerful models speak isn't English or Spanish or any human language for that matter... the AI doesn't see a sentence. It sees a list of four separate tokens.”
The core function of a large language model is to predict the probability of the next token given a sequence of prior tokens, and it performs this prediction iteratively (one token at a time) to generate complete sentences, paragraphs, and answers.
“the LLM has what is a surprisingly simple job. Now, don't let the math here scare you. This formula just says, given a sequence of tokens, what is the probability of the next token? That's it. That's the core of what these gigantic models are doing. Its entire multi-billion-dollar job is to look at the stream of tokens it's been given, your question, the facts from the retriever, and then predict one by one the most likely next token that should come after. It just does this over and over to write a sentence, then a paragraph, and then your full answer.”
Text is tokenized into discrete units, each token is assigned a numeric ID from a dictionary, and that ID is converted into a complex vector (embedding) that represents the token's meaning in mathematical form.
“your text gets chopped up into tokens. Then, each unique token gets a number, like an ID from a giant dictionary. And finally, that ID is turned into a really complex set of numbers called a vector. This vector, which you'll often hear called an embedding, is what the machine actually works with. It's the token's meaning captured in pure math.”
Tokens are not always full words—words can be broken into multiple tokens (e.g., 'unbelievable' becomes 'un', 'believe', 'able'), allowing the model to understand linguistic building blocks like prefixes, suffixes, and roots.
“A token isn't always a full word. Check out the word unbelievable. The AI might see this as three separate tokens. Un, believe, and able. Now, this is absolutely crucial because it lets the model understand the building blocks of words. You know, the prefixes, the suffixes, the roots, which gives it a much, much deeper grasp on how language actually works.”
Within a context window, every single token counts toward a total token budget for a single interaction, including the user's question, system instructions, background information retrieval, and the generated answer—all competing for the same limited allocation.
“inside that context window, you've got a token budget. For any one interaction, every single token, and I mean every single one, counts towards your spending limit. And this isn't just your question, it's everything the AI needs to do its job. And look at how fast that budget gets spent. Your question? That's just a small slice of the pie. Then you have the system's hidden instructions, the background info it has to pull up to answer you, and of course the answer it generates. It all costs tokens, and every part of the system is spending from the same limited budget.”
When a conversation exceeds the context window limit, the AI begins forgetting earlier parts of the conversation, which is why long chats can go off the rails or become incoherent.
“If your conversation gets too long and goes over this limit, the AI starts forgetting what you talked about at the beginning. This is exactly why a long chat can sometimes go completely off the rails. The AI's wallet is full, and it literally can't remember what you said 10 minutes ago.”
Hallucination can be mitigated by allocating token budget to retrieval of real factual tokens and injecting them into the context window, providing the model with higher-quality 'money' to spend rather than forcing it to guess.
“So, what's the solution? Don't let it guess. Spend some of your token budget on retrieving real factual tokens and inject them into the context window. You're basically giving it better, more valuable money to spend.”
The retriever component searches a database for tokens mathematically similar to those in the user's query (by converting them to vectors) in order to inject relevant factual information into the token budget.
“a really key player here is called the retriever. Its job is basically to go shopping for information. It takes the tokens from your question, turns them into those mathematical vectors we talked about, and then it searches a huge database for other tokens that are mathematically similar. It's all about finding the most relevant facts to add to the token budget.”
An AI query passes through multiple processing stages (tokenization, routing, planning, information retrieval, generation) in a multi-stage assembly line, and each stage consumes tokens from the shared budget.
“It's really like a multi-stage assembly line. Your query gets tokenized, it gets routed to the right place, a plan is made, information is retrieved, and only then, after all that, does the main LLM step in to predict the final answer. Every single one of these steps runs on tokens.”
A token is the smallest unit of information an AI can process, analogous to coins and bills in a currency system, and every AI operation and thought has a token cost.
“A token is the smallest little piece of information an AI can deal with. The best way to think about it is like this. Tokens are the coins and the bills of the AI world. Every single operation, every thought, every answer, it all costs a certain number of tokens.”
An AI system has a context window (token limit) representing its short-term memory capacity, and this limit is the absolute maximum number of tokens the model can process for any single task, fundamentally constraining its capabilities.
“The number one rule is the context window. You can think of this as the size of the AI's short-term memory. Or, to stick with our analogy, it's the size of its wallet. It's the absolute maximum number of tokens the model can look at for any single task.”
Intelligence in AI systems is not about understanding words in the human sense, but rather about sophisticated management and allocation of tokens across system components and stages.
“Intelligence in these systems isn't really about understanding words in a human sense. It's about the incredibly sophisticated management of tokens.”
Improving AI systems fundamentally comes down to spending tokens more wisely and efficiently, rather than adding more compute or parameters.
“if everything is about tokens, then building better AI really just comes down to one thing, spending those tokens more wisely. It's all about becoming more efficient with your token spending.”
Effective AI system design requires treating the system as a financial planner for tokens: minimizing waste through concise prompts, maximizing signal by including only the most valuable information, and structuring tokens clearly so the AI knows what to attend to.
“This leads to a whole new way of designing AI systems. You have to be like a savvy financial planner, but for your tokens. You minimize waste by writing really concise prompts. You maximize the signal by only including the most valuable information. And you structure your tokens clearly, so the AI knows exactly what it should be paying attention to.”
Mastering the token economy is the real secret to building progressively smarter and more capable AI systems.
“Mastering this token economy, well, that's the real secret to building smarter and smarter AI.”
Older AI models have smaller context windows (e.g., 8,000 tokens, comparable to a small coin purse), while newer, more powerful models have much larger context windows (e.g., 128,000+ tokens, comparable to a bank vault), enabling processing of entire books.
“An older model might only have an 8,000 token window, which is kind of like a small coin purse. But the newer, more powerful models, they can have context windows of 128,000 tokens, or even more. That's not a wallet, that's a huge bank vault, which lets them process entire books all at once.”
AI hallucination occurs because the model runs out of factual tokens in the budget and is forced to guess at the next token, producing plausible-sounding but false information.
“Take hallucination, right? When an AI just makes stuff up, why does it do that? Well, it's because it ran out of factual tokens and had to guess the next one.”
Tokens are the universal 'currency' of AI systems because every component (retrieval, transformation, generation) reads, transforms, and generates them, making the entire system one integrated economy of transactions.
“tokens are the currency of LLM systems. Every single component reads them, transforms them, and generates them. The whole system is just one big economy, and every action is a transaction.”