Claims

Recent claims about massive context windows (e.g., 1 million tokens for Claude, 2 million for Grok) are misleading because LLMs are most effective at only approximately 20% token consumption; at 60% consumption, performance already shows steep declines.

Markdown is the de facto standard format for communicating with LLMs and delivering context because it is human-readable and supports rich text formatting (headings, bullets, callouts), and has achieved widespread internet adoption.

LLMs work as text prediction models, and the way tokens, words, and letters are structured across a context window in the right order affects how the model predicts the answer at the end.

Across several different benchmarks, markdown has consistently ranked number one for retrieving and feeding context to agents, with XML as a distant second, though the reasons why remain unclear.

JSON and other structured formats consume more tokens than plain text because they include repeated declarations of labels and schemas that serve as machine-parsing guidance but represent wasted token consumption for the LLM.

While Tune and token-optimization techniques can reduce token consumption, they do not improve accuracy beyond markdown—demonstrating that reducing token count does not necessarily translate to better LLM performance.

The @ symbol is fundamentally underused in AI harnesses and prompting, despite its established role in Slack and file references as a tool for pointing to and creating relations between intents and objects within context.

The real problem in AI context management is not harnessing the technology correctly—it is not solved by reducing tokens, making models smarter, or expanding context windows, but rather requires reconceptualizing how we interact with, point to, and communicate with AI systems.

Context is fundamentally one of the hardest problems that scientists, researchers, and developers want to solve today, and how we structure, use, and deliver context is core to how LLMs and agents appear intelligent.

Tune is a repository that uses an encoder and decoder to convert JSON into a more token-friendly format, allowing structured data to be represented with less token overhead while maintaining parsability.

YAML and other formats offer token-efficient and composable structures, but these structural improvements do not change how the LLM thinks, interacts, or processes context, and therefore do not outperform markdown.

Humans are a liability for LLM performance because people are bad at prompting—both machines are bad at prompting themselves and for others, and performance improves when humans are better at clarifying their intent.

The speaker has been working on agentic systems for at least one year, collaborating with multiple companies and developing independent solutions, providing empirical grounding for their claims about context windows and agent design.

Problems with how humans interact with AI systems and communicate intent have historical precedent in pre-AI era challenges and represent a recurring structural issue that transcends technological change.

The speaker is planning to introduce a new agent harness framework that should fundamentally change how the field talks about and approaches AI and agent systems.

Long-horizon agentic systems (e.g., OpenClaw) require the ability to work continuously over hours while staying within the 20% effective token consumption range, which is a fundamental constraint that larger context windows do not solve.

The paper-clip maximizer and cancer-cure-by-exterminating-humanity scenarios of AI doom are not plausible because they imagine systems that are simultaneously superintelligent and fundamentally stupid—having tremendous computational power but lacking the basic understanding that human extinction is incompatible with solving the original problem.

Large language models appear intelligent in only narrow domains (mathematics, coding, law) where reasoning is substrate-based (symbol manipulation itself aids thought) and where answers can be self-corrected (code can be tested, proofs verified).

The Moravec paradox, articulated by roboticist Moravec in 1988, describes the counter-intuitive observation that complex intellectual tasks like proving theorems and playing chess are easy for computers, while simple sensorimotor tasks like manipulating objects remain fundamentally difficult despite decades of robotics research.

Current AI systems are fundamentally limited compared to animals and humans because they cannot perform basic real-world tasks like clearing a dinner table, cleaning a house, or learning to drive in a few hours despite being trained on millions of hours of human driving data.

A world model is an AI system that predicts the consequences of actions before they are executed, enabling an AI system to accomplish new tasks without being explicitly trained on them by planning sequences of actions.

V-JEPA can acquire a level of physical common sense by learning to predict at the representation level, as demonstrated by the system being 'surprised' (high prediction error) when shown impossible physical scenarios like objects floating or changing shape.

LLMs fail on basic physical reasoning (e.g., whether to walk to a car wash 100m away) and exhibit no real understanding of physical intuition, instead performing retrieval-based responses fine-tuned on human feedback.

LeCun's 2023 claim that ChatGPT cannot understand basic physics (objects moving with tables) was later 'debunked' when ChatGPT was fine-tuned on this specific question after his podcast appeared, demonstrating fine-tuning on individual examples rather than learning understanding.

LLMs perform autoregressive token prediction by generating probability distributions over a finite token dictionary and sampling from them sequentially, which is fundamentally different from the optimization-based inference required for planning with world models.

Joint embedding predictive architecture has become very popular: searching 'joint embedding predictive architecture' on Google Scholar between quotes yields approximately 1,300 papers, indicating a large and growing research community interested in this approach.

Geometric deep learning (graph neural networks, manifold learning) should be applied to JEPA: training JEPA architectures on graph-structured inputs has not been done yet but would be very interesting and promising to explore.

Joint embedding architectures that do not attempt to reconstruct perform much better than reconstruction-based methods (autoencoders, VAEs, masked autoencoders, diffusion models) for learning image representations that transfer well to downstream tasks.

The hope for AMI Labs is that hierarchical JEPA-based architectures will enable training of universal causal models of any complex phenomenon, powering intelligent systems of the future across diverse domains.

Bengio underwent a personal transformation from dismissing AI safety concerns as far-off and overstated (due to current system weakness and anticipated benefits) to taking them seriously after ChatGPT's emergence, realizing that AGI could arrive sooner than previously thought.

The scaling of neural networks to larger sizes trained on more data was anticipated in the academic literature well before ChatGPT, but what was not anticipated was that this scaling would lead to systems that could manipulate language as well as or better than most humans.

Written text data for training AI systems is approaching saturation, with possibly only 1-10% of reasonably available high-quality written data remaining, and the highest-quality data has already been used with lower-quality data left, potentially putting the field close to a significant bottleneck.

Current large language models like ChatGPT excel at language manipulation but remain significantly weaker than humans in planning, reasoning, maintaining epistemic humility, entertaining multiple interpretations simultaneously, and combining pieces of knowledge coherently.

Video data represents a significant untapped frontier for AI training because it is much more computationally expensive to process than text, but contains far richer information than equivalent amounts of text data.

AlphaGo demonstrates a different kind of creativity than language models because it performs explicit search in the space of move sequences, discovering novel strategies and ways of playing that humans did not expect, exemplifying search-based creativity rather than combinatorial creativity.

The timeline for achieving AGI (artificial general intelligence with human-level cognitive abilities) is uncertain, but Bengio estimates there is a more than 50% probability it could occur within 5 to 20 years.

Advanced AI capabilities can be leveraged to solve the safety problem: if an AI system can better understand what is right and wrong, it will be less likely to cause harm, making increased capability a tool for reducing risk.

There is a small group of researchers including Bengio who believe there is a reasonable chance of developing provable (or asymptotically provable) safety guarantees for AI, which would be vastly better than the current situation of having no formal guarantees.

The breakthroughs in AI over the last three to four decades have relied on key conceptual ideas developed gradually, such as representing symbols with vectors and attention mechanisms in neural nets, which were mathematically motivated and transformative despite being 'fairly simple ideas'.

Claude 3 has an IQ of 101, which exceeds human average intelligence, demonstrating that current AI systems are already superhuman in raw cognitive capability, raising the stakes for how we govern such powerful technology.

Human intelligence combines perception and intuition with explicit reasoning, using continuous (value-centric) and discrete (program-centric) abstraction in an interwoven way at every level; to achieve truly general AI, both forms of abstraction must be leveraged together, as focusing only on one hemisphere leaves the field incomplete.

Interactive theorem proving with AI and machine learning is a promising direction for testing generalization ability and represents a legitimate test case for whether AI systems can truly reason and adapt to new mathematical domains.

Topology-based abstraction is more powerful than geometry-based abstraction because distances are not robust to small perturbations, but structure can be robust to small perturbations; topology underlies robust and flexible abstraction.

Intelligence is fundamentally the ability to autonomously produce abstractions, not merely to encode and operationalize pre-existing human abstractions; this is analogous to the difference between creating a cartoon character (that can only play pre-taught scenes) and creating a life form (that can independently adapt).

The shortcut rule is a principle of system design stating that if you optimize for a specific metric, you will take shortcuts along every possible dimension of your problem that is not captured by your metric; in AI, this means that focusing on task-specific skill yields task-specific skill but not generalizable cognitive abilities, because skill is orthogonal to generalization.

Generalization is not a binary property of a system but a spectrum ranging from static systems (chess) with no generalization need, to robust AI handling known distribution variations, to flexible AI handling broad domains (self-driving, robotics), to extreme generalization corresponding to open-ended intelligence capable of handling arbitrary domains the creator did not anticipate.

Generalization can be quantified as the efficiency with which you operationalize past information to deal with future situations—an information conversion ratio that can be precisely quantified using algebraic information theory and provides a quantified definition and measure of intelligence.

Deep learning excels at value-centric abstraction and is fundamentally a geometrical method performing smooth geometric morphing between input and target vector spaces via gradient descent, learning a distance function that defines a manifold; it is therefore well-suited to perception problems but cannot solve discrete problems requiring program-centric abstraction.

Program synthesis is the method to solve reasoning and discrete problems through combinatorial search over graphs of operators from a domain-specific language (DSL); it can solve problems from very few examples (unlike deep learning which requires dense data) but faces the challenge of combinatorial explosion as the set of possible programs grows exponentially with program size.

To make both continuous optimization (deep learning) and discrete optimization (program synthesis) efficient, a universal recipe exists: introduce modularity, hierarchy, and reuse; reuse is fundamentally another word for abstraction, and this principle applies universally to all optimization—deep learning is continuous optimization with modularity and hierarchy hardcoded as stacked layers and architectural patterns like convolution (abstracting spatial position) and RNNs (abstracting temporal axis).