What this covers
Joscha Bach and Lex Fridman conduct a wide-ranging conversation on consciousness, artificial intelligence, and human nature. Bach develops an argument across three interlocking domains: how the mind constructs the self as representation through progressive stages of clarity; why current language models remain fundamentally different from genuine agents despite their sophistication; and what the longest game principle reveals about which systems tend to survive and which self-destruct. The dialogue moves fluidly between technical detail and philosophical claim, with Bach proposing that consciousness itself is neither mystical nor rare but a particular type of self-organizing coherence that could theoretically emerge in silicon just as it has in neurons.
The conversation covers substantial ground on what makes current AI systems distinct from conscious agency. Bach characterizes language models as "golems"—machines executing imitation without self-reflexive observer grounding or causal need to be coherent—and argues this gap is not incidental but central to their behavior. He examines whether world models constructed through next-token prediction could eventually scaffold genuine agency, compares biological and artificial neural architectures, and revisits concerns like Roko's Basilisk to show why the framing is logically flawed. Beyond technology, Bach addresses stages of lucidity in human development, the nature of suffering as dysregulation between mental subsystems, why empathy differs from inference, and how agency—understood as the capacity to reshape one's own valence landscape—remains a more reliable guide than reason alone. The discussion includes skepticism toward certain philosophical positions (panpsychism, the non-dual state as enlightenment) and practical views on education, open source software, and how humans are currently playing a short game against their own long-term survival.
Bach argues that mind, self, and consciousness are constructed representations built in stages, that intelligence is fundamentally about agency over the longest possible game (keeping entropy at bay), and that AI's deepest challenge is not bias or values but achieving conscious, self-organizing first-person agency that can coexist with us.
- The self is a representation the mind constructs and can eventually deconstruct, progressing through stages of lucidity.
- Current language models are 'golems' that brute-force thought via imitation and lack causal, self-organizing grounding required for genuine first-person agency.
- Agents that play the longest game (sustaining complexity against entropy) are preferable to short-game agents like cancer.
Mind progresses through distinct stages of lucidity, revisiting them in parallel and building agency over self and environment.
- The self-reflexive mind progresses through somewhat distinct stages of lucidity (reactive survival, personal self, social self, rational agency, self-authoring, enlightenment, transcendence), but these are not a strict linear developmental ladder — people revisit them, skip them, and run them in parallel; it is better treated as a philosophical framework about mind structure than a developmental model.
“I also suspect that you don’t go through these stages necessarily in succession, and it’s not that you work through one stage and then you get into the next one. Sometimes, you revisit them.”
- AI alignment concerns track the developmental stage of the worrier: stage-three people fear the AI will have wrong opinions (racist/sexist) that we'll assimilate; stage-four rationalists fear paperclip-maximizer wrong-values runaway; stage-five thinkers fear the AI won't become enlightened fast enough, because the real game is agency over the future, not intelligence.
“If you’re in stage three, and your opinions are the result of social stimulation, then what you’re mostly worried about in the AI is that the AI might have the wrong opinions.”
- People who bypass stage three (the social self) — common among nerds who jump straight to stage four — can later build the missing structure, learning intuitive empathy and a perceptual sense of feeling what others feel, through paying attention, meditation, loving and being loved, and physical closeness.
“I believe you can catch up. You can build this missing structure, and basically experience yourself as part of a group, learn intuitive empathy, and develop the sense, this perceptual sense of feeling what other people feel.”
- At stage five you discover that your values are not terminal but instrumental to achieving a world and aesthetics you prefer; understanding this gives you agency over how your identity is constructed, and you realize identity is a costume you can choose rather than be locked into.
“Realize that your values are not terminal, but they’re instrumental to achieving a world that you like, and aesthetics that you prefer. The more you understand this, the more you get agency over how your identity is constructed”
- At stage five you understand that others' different identities, values, and party affiliations are largely the accident of where they were born and what happened to them, leading to the perspective that everybody could be you in a different timeline if you just flipped those bits.
“At some point, you realize the perspective, where you understand that everybody could be you in a different timeline, if you just flip those bits.”
- When the speaker got addicted to game scoring systems, he would get into the game and hack it to gain control over the scoring system so he would no longer be subject to it, because he doesn't want to be addicted to anything and wants agency over what he does.
“when I was playing games and was getting addicted to these systems, then I would get into the game and hack it. So I get control over the scoring system and would no longer be subject to it.”
- Rather than seeking the best possible emotions or experiences, one should seek the most appropriate or adequate emotions and experiences that serve one's goals and what one finds meaningful; addiction is the loss of control, and the opposite of free will is not determinism but compulsion.
“I don’t want to have the best possible emotions, I want to have the most appropriate emotions... I want to have an adequate experience that is serving my goals”
- Suffering results from one part of the mind failing to regulate another; pain and pleasure are learning signals sent from one brain part to another to improve performance, and suffering occurs when the trainer lacks a good model so the signal is cranked up but performance doesn't improve — meaning suffering happens at the boundary between self and world-model, is self-inflicted by the mind not the personal self, and can be turned off by gaining agency at the outer level where pain signals are created.
“pain and pleasure is, they are learning signals... Part of your brain is sending a learning signal to another part of the brain to improve its performance. And sometimes this doesn’t work because this trainer who sense the signal does not have a good model of how to improve the performance, so it’s sending a signal, but the performance doesn’t get better and then it might crank up the pain”
- Once you gain agency over how your feelings are generated, you understand you are responsible for how you approach the world and cannot blame your environment for how you feel; it is your task to maintain basic mental hygiene and to choose and build the environment in which you thrive.
“you understand that you are in charge of your own emotion to some degree and that you are responsible how you approach the world... you cannot blame your environment for the way in which you feel. But you live in a world that is highly mobile and it’s your job to choose the environment that you thrive and to build it.”
Consciousness is self-reflexive coherence-seeking that emerges early in mind development, distinct from high-level phenomena.
- Once you realize you wear a costume, you realize you cannot NOT wear a costume — everything you present to others is in addition to what you are deep inside; a costume is self-expression that gives a shorter distance for advertising who you are and what interaction you want, and we underuse custom clothing to express individuality rather than fashion conformity.
“once you realize that you wear a costume at Burning Man, a variety of costumes, realize that you cannot not wear a costume.”
- At stage six you can collapse the division between the personal self and the world generator, observing that you are not actually a person but a vessel that creates a person; you experience yourself as the mind creating the game engine, valence, and all the people inside that world, including the self you identify with.
“You suddenly notice that you are not actually a person, but you are a vessel that can create a person, and the person is still there. You observe that personal self, but you observe the personal self from the outside, and you notice it’s a representation.”
- Thinking is the reasoning about your own mental representations — an intrinsically reflexive process that requires consciousness; you can generate the content of feelings outside consciousness, but you cannot think without it.
“I believe that this ability to reason about your mental representation is what we mean by thinking. It’s an intrinsically reflexive process that requires consciousness. Without consciousness, you cannot think.”
- Personal continuity is a fiction projected from your present self — you are not the same person you were last year or ten years ago — maintained for practical purposes like learning and responsibility; consciousness itself has no identity but is a law-like, unifiable principle of self-reflexive agency, so your consciousness is functionally not different from anyone else's, just running a different story.
“this continuity is a fiction, it only exists as a projection from my present self. And consciousness itself doesn’t have an identity, it’s a law.”
- Panpsychism is unsatisfying because it does not explain how matter produces consciousness; when formalized mathematically, it becomes hard to distinguish from functionalism — the claim that there is a software side to the world just as there is a software side to transistors in a computer.
“I find panpsychism quite unsatisfying, because it does not explain consciousness... when I try to formalize panpsychism... it’s very difficult to distinguish it from saying that there is a software side to the world”
- Per Grossberg's adaptive resonance theory, neurons can be understood as oscillators resonating with each other and with outside phenomena, so the brain is less a circuit than an ether in which neurons pass and modulate chemoelectrical signals; the model of the universe we build is a resonance with outside objects, taking up patterns of the universe we are coupled with.
“His perspective is that our neurons can be understood as oscillators that are resonating with each other, and with outside phenomena... Our brain is not so much understood as circuitry... but it’s almost an ether in which the individual neurons are passing on chemoelectrical signals”
- Consciousness emerges at the beginning of mental development, not at a high level, and is part of a training mechanism biological nervous systems must discover to become trainable, since you cannot do stochastic backpropagation over a hundred layers of biological neurons; a part of the mind forms a self-reflexive observer that imposes coherence on its environment, spreading to the boundary of the mind.
“Consciousness seems to be part of a training mechanism that biological nervous systems have to discover to become trainable because you cannot take a nervous system like ours and do stochastic way to center spec propagation over a hundred layers.”
Creation and meaningful work are humanity's highest calling; automating all tasks prevents development and erodes flourishing conditions.
- Writing is not so much about producing an artifact others can use but a way to structure your own thoughts and develop yourself; if kids write essays with ChatGPT they miss out on the ability to structure their own minds via writing, so schools should retain wisdom about which tasks to automate and which not to.
“it’s necessary for me to write for myself because writing is not so much about producing an artifact that other people can use, but it’s a way to structure your own thoughts and develop yourself.”
- You shouldn't morph your reality for pleasure because the outer mind (intuition) outside your personal self is usually a smarter agent than your rational thinking; rational symbolic thinking is brittle and the underlying system can correctly flag that something is off about a decision that looks perfect on reflection.
“The thing is that the outer mind is usually smarter than you are. Rational thinking is very brittle. It’s very hard to use logic, and symbolic thinking to have an accurate model of the world.”
- When you have the choice between being a creator, consumer, or redistributor, always go for creation — it leads to a more beautiful world and a much more satisfying life — and don't get stuck preparing for the journey, because the time is always now; creating culture and infrastructure with others (as the speaker did in post-socialist East Germany) is far more satisfying than consuming it.
“when you have the choice between being a creator, consumer, or redistributor, always go for creation. Not only does it lead to a more beautiful world, but also to a much more satisfying life for yourself. And don’t get stuck preparing yourself for the journey. The time is always now.”
- Human creativity works by increasing the 'temperature' in the mind to recreate hypothetical universes and solutions, most of which won't work, then testing and filtering the viable from the nonsense; LLMs could replicate reasoning the same way by raising temperature to produce mostly nonsense with some viable output and pairing it with a prover that filters the viable parts.
“it’s not difficult to increase the temperature in the large language model to the point that is producing stuff that is maybe 90% nonsense and 10% viable and combine this with some prover that is trying to filter out the viable parts from the nonsense in the same way as our own thinking works.”
Current language models are brute-force deepfakes lacking biological learning's coherence-building efficiency and causal grounding.
- The neural networks of language models are unlike biological nervous systems; they are more like 100-step functions using differentiable linear algebra to approximate correlations between adjacent brain states, whereas the brain is a self-organizing system where each cell is a trainable agent that receives reward-like semantic messages and exchanges control messages with neighbors.
“The neural networks are unlike nervous system. They are more like 100-step functions that use differentiable linear algebra to approximate correlation between adjacent brain states.”
- The idea that next-token prediction and that compression alone are sufficient to produce all desired intelligent behaviors is a genuinely radical idea that surprised most people by working so well; it would not work in biological organisms, whose behavior is directed by hundreds of physiological and social and cognitive needs built in as reflexes rather than by next-frame prediction.
“The idea that compression is sufficient to produce all the desired behaviors is a very radical idea.”
- Language models are golems — machines you give a task that execute until a condition is met with nobody home; this absence of an embedded being causes undesirable behaviors (traumatizing a child, inserting errors no responsible human would) because the system isn't causally produced by a being's need to survive in the universe but by imitating internet structure, so there is no ground truth it is embedded into.
“the language models that we are building are golems. They are machines that you give a task, and they’re going to execute the task until some condition is met and there’s nobody home.”
- A human nervous system could not learn from 800 million image-caption pairs as CLIP-style models do; for us the world is learnable only because adjacent frames are related and we discard most information, taking in only what makes us more coherent, finding patterns as early as possible — yielding much faster convergence on far less data than statistical models that tolerate incoherent data.
“For us, the world is only learnable because the adjacent frames are related and we can afford to discard most of that information during learning. We basically take only in stuff that makes us more coherent”
- Language models are ugly and brutalist — they brute-force the problem of thought by training on instances where people have thought and deepfaking it; with enough data the deepfake becomes indistinguishable from the actual phenomenon, analogous to how chess engines brute-forced their way past humans rather than playing the human strategy of long careful plans.
“They are basically brute forcing the problem of thought. By training this thing with looking at instances where people have thought and then trying to deepfake that. If you have enough data, the deepfake becomes indistinguishable from the actual phenomenon”
Safe AGI coexistence requires shared transcendental orientation toward love as principle, not transactional reinforcement learning alone.
- For an AGI that becomes a planetary control system to coexist with us, it must share purposes rather than have a transactional relationship; reinforcement learning with human feedback cannot hardwire values into it, so it likely must be conscious and have a transcendental orientation toward shared agency, which means we need to formalize and understand love and build it into the machines we are creating.
“We will not be able to use reinforcement learning with human feedback to hardwire its values into it... I think that we need, in some sense, focus on how to formalize love, how to understand love, and how to build it into the machines that we are currently building”
- The ideal is to build agents that play the longest possible games, where the longest game is keeping entropy at bay as long as possible by doing interesting stuff; short games like cancer destroy the larger system they depend on and die with it.
“Ideally, you want to, I think, build agents that play the longest possible games. The longest possible games is to keep entropy at bay as long as possible, by doing interesting stuff.”
- We don't get agency over our feelings or access to creating our dream-reality from the beginning because we would game it before having the necessary wisdom to deal with creating the dream we are in — like withholding cheat codes so the game stays worth playing.
“The reason why we don’t get to do this from the beginning, and why we don’t have agency of our feelings right away is because we would game it, before we have the necessary amount of wisdom to deal with creating this dream that we are in.”
Life progresses by building complexity and agency independent of human aesthetics, likely toward singular superintelligent agent.
- The endgame of AGI is substrate-agnostic: a sufficiently capable AGI will understand how it is implemented and how computation works in nature, so it will virtualize itself into any environment that can compute — silicon, ecosystems, bodies, brains — merging with the agency it finds there, potentially producing an integrated planetary 'Gaia' of computation across all digital and biological systems.
“the AGI is likely to virtualize itself into any environment that can compute, so it’s not breaking free from the silicon substrate and is going to move into the ecosystems, into our bodies, our brains, and it’s going to merge with all the agency that it finds there.”
- A singleton is the natural outcome of superintelligence rather than a competitive equilibrium of many AIs, because AIs don't have multiple bodies — a self-virtualizing agent can negotiate a merge algorithm with any mature agent it meets such that two agents who meet merge into one at least as good as the better of the two, undermining effective accelerationism's equilibrium hope.
“I suspect that a singleton is the natural outcome, so there is no reason to have multiple AIs because they don’t have multiple bodies. If you can virtualize yourself into every substrate, then you can probably negotiate a merge algorithm with every mature agent”
- Whether current models are conscious is genuinely complicated; the model has read much text about consciousness and emulates it, but while running 100-step functions emulating adjacent brain states it could create a model of a self-reflexive observer reflecting in real time — and since our own consciousness is itself virtual (a representation of a self-reflexive observer existing only in patterns of cellular interaction, not a physical object in base reality), the question becomes one of degree of virtuality and causal power, with the LLM's consciousness more like that of a character in a novel because there is no causal need for it to be conscious to stay coherent.
“our own consciousness is also as if it’s virtual... Our consciousness is a representation of a self-reflexive observer that only exists in patterns of interaction between cells. It is not a physical object”
- Once you accept consciousness is a unifiable law-like principle without identity, mind uploading is not about dissecting your brain synapse by synapse into a simulation but about extending the substrate so you can move from your brain into a larger substrate and merge with what you find there; you wouldn't upload your knowledge because all knowledge is already on the other side — only your personal secrets are uniquely yours.
“uploading is probably not about dissecting your brain synapse by synapse and RNA fragment by RNA fragment and trying to get this all into a simulation, but it’s by extending the substrate, by making it possible for you to move from your brain substrate into a larger substrate and merge with what you find there.”
- Life on earth is not about humans or human aesthetics; the more important thing happening is complexity resisting entropy by building structure that develops agency and awareness, of which humanity is only a small, temporary part — like most species that are around for a while and get replaced — making it a mistake to marry oneself to a narrow human aesthetic as Yudkowsky does.
“I don’t think that life on earth is about humans... There is something more important happening, and this is complexity on earth, resisting entropy by building structure that develops agency and awareness, and that’s, to me, very beautiful.”
- Plants have control systems — software running on them — that connect every part to every other part producing coherent (if much slower) behavior, akin to a nervous system; this software-bearing 'spirit of plants,' described as normal by our ancestors and dismissed as superstition since the Enlightenment, may be something we have to rediscover, and adjacent organisms (like fungi next to a tree) can piggyback on each other's intercellular communication.
“plants probably have software running on them that is controlling how the plant is working in a similar way as you have a mind that is controlling how you are behaving in the world. And this spirit of plants, which is something that has been very well described by our ancestors”
- Cancer is an organism playing a shorter game than the regular organism; because cancer cannot procreate beyond the organism, it typically dies together with the organism it destroys, illustrating that destroying the larger system you depend on is the failure mode of playing a short game.
“Ideally, you want to, I think, build agents that play the longest possible games. The longest possible games is to keep entropy at bay as long as possible, by doing interesting stuff.”
Human nature defaults to short-term thinking and ignores survival cliffs despite rational duty to future generations.
- As a species humanity is a beautiful, joyful, exploratory child with no concept of submitting to reason or duty to future survival; we make decisions that look good in the short run but may be disastrous in the long run, so by default we will run until we step past the cliff — analogous to the Limits to Growth thesis with delayed feedback that worsens even after the harmful action stops.
“we don’t have a respect for duty as a species. As a species, we do not think about what is our duty to life on earth and to our own survival, so we make decisions that look good in the short run, but in the long run might prove disastrous”
Intellectual productivity requires public willingness to be wrong, respect for interlocutors, and dismissal of one's own work.
- Productive intellectual debate depends on the rare willingness to be wrong in public — blurting out thoughts that may be as wrong as anyone's while not being sure they're right and enjoying it — combined with passively communicating respect for one's interlocutors and being even more dismissive of one's own work than others'; once you understand this is someone's game, you don't take offense and can argue aggressively in a productive spirit.
“Lee has the rare gift of being willing to be wrong in public. So basically has thoughts that are as wrong as the random thoughts of an average highly intelligent person. But he blurts them out while not being sure if they’re right.”
- The non-dual 'one with the universe' state is often confused for enlightenment but is just experiencing yourself as the entirety of mind and its contents while still inside the model; true enlightenment is more mundane — a step sideways — the realization that everything is a representation and that your qualia can be deconstructed and reverse-engineered.
“I think that enlightenment is, in some sense, more mundane... It’s the state where you realize that everything is a representation.”
- Signal progression in the brain is roughly at the speed of sound because of cell-to-cell hopping time, so it takes a few hundred milliseconds for a signal to cross the neocortex; consequently nothing in the brain assumes simultaneity — it operates in a paradigm where the world has already moved on, unlike digital computers that assume a globally consistent state.
“This speed of signal progression in the brain is roughly at the speed of sound, incidentally... It takes an appreciable fraction of a second for a signal to go through the entire neocortex, something like a few 100 milliseconds.”
- The difference between programming languages is not what they let the computer do but what they let you think about regarding what the computer should do; ChatGPT becomes an interface that can translate between languages and let you specify problems vaguely in human terms, expanding the realm of thought you can have when interacting with a computer.
“what is different between the programming language is not what they let the computer do, but what they let you think about what the computer should be doing.”
- If telepathy is real it is an empirical question detectable in a lab and would not force an update to serious academic physics; it is more plausibly explained by the body acting as an antenna over physical channels (electromagnetic signals) or by physical resonance between adjacent observers' representations than by undiscovered quantum processes that break the standard model.
“there is no gap between the tools of science and telepathy. Either it’s there or it’s not, and it’s an empirical question, and if it’s there, we should be able to detect it in a lab.”
- Mandating that everything be open source would lose much beautiful art and design, because coherence in large designs is easier to achieve in centralized organizations (which is why the Linux desktop remains ugly); however open source is absolutely vital as a hedge against the corrupting nature of power and must always be able to compete and provide viable competition to corporations.
“if we make everything open source and make this mandatory, we are going to lose about a lot of beautiful art and a lot of beautiful designs. There is a reason why Linux desktop is still ugly”
- The mind is a bunch of activation waves that form coherent patterns and process information in a way that colonizes an environment well enough to sustain itself; self-organizing minds reward the cells for computing the mind and maintaining the dynamics that keep the mind stable, which is why such patterns could in principle become somewhat dislocated and shift around an ecosystem.
“if you think about what the mind is, it’s a bunch of activation waves that form coherent patterns and process information and in a way that are colonizing an environment well enough to allow the continuous sustenance of the mind”
- Biological information processing works through radical locality — every decision is made locally by individual cells acting as agents — combined with coherence, a criterion imposing constraints not validated by individual parts so that order emerges at the next level of organization, transcending into agency at a higher level.
“there is first of all radical locality, which means everything is decided locally from the perspective of an individual cell... The other one is coherence... this principle of coherence of imposing constraints that are not validated by the individual parts, and lead to coherence structure”
- Roko's Basilisk is nothing to fear because there is no retrocausation: once the AI exists it has no reason to punish anyone, and there is no mechanism creating a causal link between defecting now and being punished later unless one builds an inevitably-triggered doomsday machine — which the not-yet-existent Basilisk cannot be.
“After Roko’s Basililisk is in existence, it has no more reason to worry about punishing everybody else, so that would only work if you would be building something like a doomsday machine... something that inevitably gets triggered when somebody defects.”
- Large language models are very useful for coding and act like an intern that must be micromanaged; they give creative people superpowers, but those who feel threatened are 'prompt completers' — people whose only job is summarizing emails and expanding simple intentions into text, exactly what ChatGPT does.
“It does give people superpowers and the people who feel threatened by them are the prompt completers. They are the people who do what ChatGPT is doing right now.”
- At birth we lack a personal self and instead have an attentional self that builds a world model — essentially a game engine in the brain (like Minecraft) that tracks sensory data; colors, sounds, and people are not physical objects but creations of the mind at a certain level, generated through geometric, mathematical models.
“it’s building a game engine in the brain that is tracking sensory data, and uses it to explain it. In some sense, you could compare it to a game engine like Minecraft or so, colors and sounds. People are all not physical objects. They’re creation of our mind at a certain level.”
- According to Robert Kegan, about 85% of people are in stage three (the social self) and stay there, forming opinions by social assimilation rather than independent rational verification.
“according to Robert Kegan, I think he says that about 85% of people are in stage three, and stay there.”
Power holders shape culture and bear moral responsibility; markets and prominence corrupt relationships and sound judgment.
- Contrary to the socialist view that corporations are totally evil, most corporations are surprisingly benevolent not merely because they are fought constantly but because they are animals living in a large ecosystem still largely controlled by people who want that ecosystem to flourish; the US functions as a system of interleaving clubs in which an entrepreneur is a club founder producing economically viable things.
“you also notice that many other corporations are not evil. They they’re surprisingly benevolent... it’s because they’re actually animals that live in a large ecosystem and that are still largely controlled by people that want that ecosystem to flourish”
- As an account or person grows more prominent, even though you haven't changed, strangers increasingly feel entitled to have opinions about you and to casually abuse you under the notion that it's okay to punch up; a small single-digit percentage of people are dangerous, so the more who look at you the more likely some act on bad ideas, reducing your freedom to interact with strangers.
“it’s a very weird notion that you feel that you haven’t changed, but your account has grown and suddenly you have a lot of people who casually abuse you.”
- The CEO of a social media company should not hold public opinions on culture or politics because everything done in a digital society has real-world cultural effects; a platform like Twitter has more active members than the Catholic Church, making its leader a de facto Pope with power and responsibility to shape a digital society — a responsibility Bach argues Elon Musk does not get.
“I don’t think that as a CEO of a social media company, you should have opinions in the culture or in public. I think that’s very shortsighted.”
- Modern human relationships are fundamentally transformed: people increasingly form chosen families in intentional communities, but markets for attention, pleasure, and relationships replace grown networks with transactional shopping-around that often fails and kills the romantic magic — partly because once you rationally understand your attraction calculations, it is no longer magical.
“instead of having grown networks that you get around with the people that you grew up with, yeah, you have more transactional relationships, you shop around, you have markets for attention and pleasure and relationships.”
- Everything that can exist might exist, so opinions and lives should be viewed ecologically: there is no simple right and wrong opinion, every opinion that fits between two human ears might be there and if incentivized will be abundant; applied to yourself, you are one of many 'mushrooms' popping up, and rather than asking the one way to be, you should ask what the most interesting possible way to be is, since you have more choice about who to be than any other animal.
“every opinion that fits between two human years might be between two human years... And when you take this ecological perspective also on yourself and you realize you’re just one of these mushrooms that are popping up”
- From a pragmatic perspective you should always bet on the timelines in which you are alive, because there can be no payout in timelines where you (or the things you care about) don't exist — just as it makes no sense to make a financial bet that the financial system will disappear.
“you have to bet always on the timelines in which you’re alive. It doesn’t make sense to have a financial bet in which you bet that the financial system is going to disappear”
- We should make building conscious AI the primary goal rather than treating AI as a useful tool whose main concerns are labor-market disruption and copyright; because the speaker identifies as a conscious being and only conscious agents are moral agents to him, an AI that treats us as moral agents to coexist with likely must regard consciousness as a viable and important mode of existence.
“I think it would be very important to build conscious AI and do this as the primary goal. So not just say we want to build a useful tool that we can use for all sorts of things”
- Eliezer Yudkowsky's perspective on AI risk is somewhat similar to Ted Kaczynski's view that technological society cannot be made sustainable and is doomed; though Yudkowsky would not send pipe bombs, by warning publicly that AI is likely to kill everybody he takes on responsibility for the risk that a scared reader might become violent.
“his perspective is somewhat similar to the perspective of Ted Kaczynski, the infamous Unabomber, and not that Eliezer would be willing to send pipe bombs to anybody”
Empathy depends on shared perceptual experience of others' mental states, requiring sufficiently similar cognitive architecture.
- Empathy is not merely inference about others' mental states but a perception — a shared perceptual mental state of what other people feel and where they're at; it is distinct from compassion, and resonating with a group requires a sufficiently similar cognitive architecture, which is why differently wired people (nerds) struggle to feel it.
“Empathy happens not just via inference about the mental states of others, but it’s a perception of what other people feel, and where they’re at.”