Karpathy's LLM Wiki Goes Further Than Everyone Realised

The three-layer pattern ensures that as raw data grows, deterministic scripts can extract needed information from the schema because the structure tells the agent exactly where to find specific data (like a client's audit or Remotion templates) without semantic search.

“there's a there's a you can almost write deterministic scripts to be able to extract the information that you need because it's all the schema is telling us, "Hey, look, if you want to go and find this client's audit or this video, and you want to, you know, figure find where all the templates for Remotion are, for example, that's all done in the Claude MD file, and that navigation is set up.”

The core objective of the three-layer architecture is controlled context injection into LLM conversations, which prevents the AI from indexing all raw data and bloating the context window when searching for information.

“the general idea here is that we're telling the agent exactly where to go to to be able to find information so that when your knowledge base starts to stack up, it doesn't bloat your context window massively just to try and locate the information that the agent needs to locate cuz at the end of the day, that is the objective of this is controlled context injection into a conversation.”

Claude is building agent memory functionality into its products, and Adam's team uses separate running memory markdown files that Claude updates, which is more effective and lightweight than RAG because it avoids the overhead of maintaining vector databases.

“Claude's actually building I [9:17] I think they're building this into their products cuz I've been seeing a lot about updating memory, stuff like that. So, we we have a separate running memory markdown file that Claude will update, but it looks like this might even start to be built into the Claude product itself as well. But in essence, this is much more effective than rag because rag runs based on semantic search, but it's it's also overkill really to have to set up a vector database and maintain that vector database.”

An AI first business system or AI operating system is fundamentally a way of building different departments in a business using Claude code and organized agent plugins.

“what is an AI first business system or an AI operating system? It's getting thrown around as. Essentially, what it is is it's a way of building different departments in your business using Claude code.”

The Karpathy LLM Wiki framework involves a three-level structure: raw data ingestion, summarization and structuring of data, and schema creation that tells the AI where data is stored and how to access it, enabling knowledge base growth without loss of maintainability.

“there's a three-level structure there. We're going to talk about the raw digestion of data, then the summarization and structuring of that data, uh and then the the schema creation so that the AI knows where that data is, how it's stored, and and how to access it in order for us to maintain and structure that information as as the knowledge base grows and grows and grows and the complexity of our systems grow.”

The audit JSON and related Wiki data are stored in a central knowledge base alongside Claude MD files and skill.md files that tell the agent what plugins, skills, data, and connectors exist in the system, enabling agent awareness of available tools.

“We're also storing all that key information that we're then using later down the line... and all comes from this audit data JSON, this central knowledge base, or this Wiki on exactly what what they have. And what we have on top of that is a Claude MD file. We also have the skill.md file in there. This is actually what's telling the agent what's in the Wiki and what's in the plugin as well.”

The audit extractor agent pulls all connected data and stores it in a raw folder structure (e.g., meetings/transcripts), then extracts that data into JSON format to create the Wiki layer, which captures who completes tasks, duration, frequency, tools used, pain points, and proposed optimizations.

“it pulls all that information and stores it in a in a folder that looks like this. It's just meetings/meetings/transcripts or whatever... And then what we do from there is we extract all this data, and we put it in this JSON format so we can organize the information really clearly... we're trying to uncover three things, right? We're trying to to be basically figure out, all right, who is completing a task? Who's responsible? How long does it take, and how often are they doing it? And what tools are they using?”

Understanding the three-layer structure and how to organize a repository folder is really important when building larger production systems, and this determines whether agents can efficiently route to information or must do expensive discovery.

“understanding this structure and understanding how to actually organize a repository folder is [snorts] really, really important when you're building larger production systems and I'll I'll show you through the video editor in course.”

The core approach is just data organization using proven formatting like YAML and JSON to store information in a controlled way that enables scripts to filter information and extract exactly what's needed, preventing unnecessary context bloat from being injected into conversations.

“It's it's all about just data organization and using proven formatting like YAML and JSON to be able to store the information that we And because it's done in a controlled way, we can easily use scripts to filter information and extract exactly what we need even if the that file starts to get very, very large as long as it's it's correctly architected so that we can filter it down and and remove any unneeded or unwanted context from being injected into a conversation, that is all we need.”

In the video editor, the storyboard is a JSON structure that specifies for each clip: which motion graphic to show, the transcript text, zoom effects, etc., functioning as a Wiki layer that Remotion then reads and renders.

“From there, we're then compiling a storyboard. Now, the storyboard is is a JSON structure again that's saying, "Hey, for the first 4 seconds, show this motion graphic. This is the transcript. These should be the zoom effects, etc." See how in essence, that's actually a Wiki? It's telling the AI, "Hey, look, this is exactly how the whole thing's going to be constructed." And then Remotion [clears throat] then reads that file.”

In the video workflow, agents know to look in specific folders (video ingest) for raw files, process them through transcription, audio analysis, and storyboard generation, place outputs in labeled folders (motion graphics), enabling accurate multi-step processing without requiring agents to discover steps.

“It knows to look to it in this exact folder for any raw video files. It'll process them and put them in the raw folder of the video editor. Then, it will follow the workflow to actually proceed through every step on the transcription, run the audio analysis, build the storyboard. Then, it will put the motion graphics in the correct folder. And see how it's all nicely organized in here? So, that it's labeled clearly so the AI knows exactly which folders to go to to extract the information that it needs to proceed to the next step.”

The video editor pipeline at APG Software uses Remotion and a three-layer structure similar to the audit system: raw video ingestion, storyboard JSON generation, and Remotion skill files that define pacing rules and templates for long-form, short-form, sales letters, and advertisements.

“Another example is our video editor pipeline... we have This is editing the video that you're watching right now. Again, we have multiple agents uh and we have multiple skills. So, we it's done by Remotion, so we have a Remotion skill, but then we have separate skills for long-form, short-form, video sales letters, advertisements, etc.”

At APG Software, Adam has built seven different plugins using AI that drop project delivery time by 40-80% while enabling tasks like AI-based video editing, presentation building, and full day-to-day process automation.

“we've built seven different plugins that are genuinely dropping the time it takes for us to deliver projects for clients by anywhere from 40 to 80% whilst also helping me do things like edit the exact video that you're doing entirely with AI, build the presentation that you're watching, and everything else that is involved in my day-to-day process.”

The audit extraction workflow connects directly to meeting transcripts (via Fathom.ai), client communications (via Twilio), and emails (via Gmail) to pull all contextual information into a single raw data layer before structuring it.

“we we connect it directly to our meeting transcript and our meeting note-taker, which is fathom.ai, which is completely free. And we also connect it to uh Twilio to get every message and call that we make to a client, and we connect to Gmail so we can pull every email.”

Without proper folder organization and clear labeling, agents perform extra digging to locate information, pulling unnecessary extra data into the context window, which reduces accuracy of responses.

“If you don't have this level of organization, what happens is the AI has to do its digging itself and it's running all these extra commands and pulling in all this extra information that's bloating your context window uh when and that's going to reduce the accuracy of your response in [11:52] in short in short.”

At APG Software, the business audit process—which previously took 200+ hours—now takes 20-40 hours (an 80% reduction) because they implemented an AI first business system that automates the heavy lifting of data collection and organization.

“That process used to take, I swear to God, 200 hours or more, which is now dropped down to something we can do in 20 to 40 hours. AI is So, that's an 80% decrease.”

At APG Software's video plugin, the code plugin folder structure corresponds directly to the video production folder structure, with raw video ingestion, processing, and output stages organized so agents know exactly which folders to access for each step.

“So, if we look at our video plugin, this actually corresponds directly to what we had in code in our code code plugin, so they're directly connected. But, this is exactly where we're actually building these agents. And if we go down to an actual content [10:51] so our projects, so if I come down to content and I look at our projects for example and I go to any video, let's just do this one. We can see we have our video ingest folder.”