The Dark Side of Conscious Ai | Daniel Schmachtenberger

As of today we are in a war that has moved the atomic clock closer to midnight than it has ever been due to the combination of nuclear weapons and artificial intelligence, and we could easily have the last chapter of human history if we are not more careful about confident wrong ideas

“as of today we are in a war that has moved the atomic clock closer to midnight than it has ever been we're dealing with nukes and Ai and things like that we could easily have the last chapter in that book if we are not more careful about confident wrong ideas”

The Nash equilibrium is a state where all players choose their optimal strategy given their beliefs about other players' strategies such that no individual player can improve their outcome by unilaterally changing their strategy, and equilibria are frequently suboptimal for all players collectively compared to coordinated alternatives

“a Nash equilibrium occurs when all players choose their optimal strategy given their beliefs about other people's strategies such that no one player can benefit from altering their strategy...a common misconception about Nash equilibria is that they result in the best possible outcome for all players actually most often they're suboptimal for each player they also have social inefficiencies”

Large-scale AI system development will be accompanied by massive job displacement (Goldman Sachs estimates that AI systems can replace 300 million jobs globally and one in four work tasks in developed economies), with the biggest impacts on developed economies, but this has received less public attention than speculative AGI risks

“a just released report from Goldman Sachs says that the currently existing AI systems can replace 300 million jobs worldwide and about one in four work tasks in the US and Europe according to Goldman Sachs the biggest impacts will be felt in developed economies”

The solutions of becoming a multiplanetary species, transitioning to a digital economy, or uploading human consciousness are either not feasible within the relevant timescales when planetary tipping points are being approached or they rest on unexamined metaphysical assumptions about consciousness and substrate independence that are questionable

“I don't think that we are anywhere near close independent of the ethical or aesthetic argument on if us obliterating our planet's caring capacity and then exporting that to the rest of the universe is a good good or lovely idea or not independent of that argument I the timelines by which that could actually meet the humanity superorganism growing needs relative to the timelines where this thing starts failing don't work”

Externalities in industrial production (pollution, environmental destruction, health impacts) are not accidental side effects but structural features of the system—they occur because corporations profit from externalizing costs onto nature and future generations while capturing the economic value, and this dynamic repeats across all exponential technologies.

“if the cost of the environmental pollution to clean it up was something where in the process of the corporation selling the posos for as a surfactant for waterproof en clothes or whatever it also had to pay for the cost to clean up its effect in the environment or the oil cost had to clean up the effect on the environment so you didn't have the perverse incentive to externalize costs onto Nature's balance sheet”

The problem of internalizing environmental and social costs into economic calculations is not just a policy problem but would fundamentally restructure what becomes profitable and viable under capitalism—making it a challenge to existing power structures that profit from externalization.

“if the cost of the environmental pollution to clean it up was something where in the process of the corporation selling the posos for as a surfactant for waterproof en clothes or whatever it also had to pay for the cost to clean up its effect in the environment or the oil cost had to clean up the effect on the environment so you didn't have the perverse incentive to externalize costs onto Nature's balance sheet which nobody enforces obviously we'd have no none of those environmental issues right that would be a totally different situation”

Memes (ideas, worldviews, value systems) compete for propagation and transmission just like genes do, and the ones that make it through evolutionary selection are those that are good at replicating and spreading, not necessarily those that are true or good or produce flourishing outcomes.

“memes engage in a kind of competitive selection like genes do but not individual memes meme complexes so if we have a religion that says be humble be quiet listen to people don't push your ideas on anybody and then you have another one that says go out and pratie convert everyone to your religion and kill the infidels which one gets more people involved right”

The global workspace model suggests that consciousness involves the extension of momentary sensory flashes (lasting 3/10 second to 3 seconds) into a continuous working memory that allows recursion and manipulation of experience, requiring both the sensory stream and the memory extension mechanisms

“you can imagine awareness without really like one way of differentiating it would be sort of we have what a sensory perceptual awareness that lasts say 3/10 of a second to 3 seconds it's like a flash okay then you have working memory which extends it across time and puts it on a loop that Loop is what allows you to then gain access to it and manipulate it”

Extreme scale AI models are expensive to train and only a few tech companies can afford development, concentrating power while making it impossible for broader research community to inspect, dissect, and audit these systems for safety, creating a governance and epistemic crisis

“extreme scale AI models are so expensive to train and only few tech companies can afford to do so so we already see the concentration of power but what's worse for AI safety we're now at the mercy of those few tech companies because researchers in the larger Community do not have the means to truly inspect and dissect these models”

Technologies are not values-neutral; their adoption creates structural incentives that drive particular types of governance, consciousness patterns, social organization, and value systems, meaning that the choice of what technologies to develop and deploy is fundamentally an ethical choice that shapes the kind of civilization that emerges

“if a technology gives the capacity to do something that provides Advantage relative advantage in a competitive environment...if it provides significant competitive Advantage if you use it a particular way then anyone using it that way creates a multi-polar trap that obligates the others to use it that way”

Ian McGilchrist's theory of brain hemispheres suggests that the left hemisphere (focused on logic, symbol manipulation, analysis, and optimization of isolated metrics) has become dominant in Western culture at the expense of the right hemisphere (concerned with context, relationships, wholeness, and direct experience), and this imbalance is both a cause and expression of the metacrisis

“he calls it the master and his Emissary in his 2009 Opus...the master and his Emissary ian mcgilchrist discusses the distinct functions of the brain's left and right hemispheres”

Valence consciousness, rooted in bodily feeling and the distinction between pleasure/pain and active/passive, may be the most basic form of consciousness and grounds higher forms of conscious experience in affective and embodied processes rather than pure information processing

“the veence qualia veilance qualia basically orients and gives value to and can be thought of as in like Pleasure and Pain in the body okay and it yolks a sensory state with an affective State and points you in a direction yoke means tie together like to Y yoke stuff together so this is the sort of the earliest form of Consciousness”

The solutions to catastrophic risks that involve control mechanisms create trajectories towards dystopias, and what we need is a third attractor basin—a civilization that could steward exponential technology without either cascading into catastrophe or creating totalitarian control systems.

“the trajectories towards existential and Global catastrophic risk in particular the solutions to those that involve control mechanisms that cre create trajectories towards dystopias and the consideration of what a world that is neither in the attractor Basin of catastrophe or dystopia looks like”

There is no finite set of metrics or values adequate to specify what an advanced AI system should optimize for, because any metric can be gamed, any specification of a goal contains unstated assumptions, and any real-world scenario involves complexities and trade-offs that cannot be fully anticipated or formalized

“is there a finite describable set of things that is adequate for something that can do uh optimization that powerfully right that is a way of thinking and so it's is there a finitely describable definition of good is another way of thinking about it right”

Consciousness is not purely computational and is not reducible to brain states alone; rather, it fundamentally involves embodied cognition (including the endocrine system, microbiome, peripheral nervous system, and environmental coupling), quantum mechanical effects that prevent perfect information-theoretic replication, and an autopoetic self-organizing process that is essential to consciousness

“I'm not a strong computationalist meaning don't believe that mind univers sensient qualia is purely a function of computation”

The metacrisis (or polycrisis) is the recognition that global catastrophic risks don't exist in isolation—climate change, wealth inequality, war, species extinction, and other risks interact with each other through cascading effects, meaning that solving them requires addressing the underlying coordination failures and misaligned incentive structures that drive all of them.

“the metac crisis the there's a more popular term out there right now the poly crisis we've been calling this the metac crisis since before coming across that term poly crisis is the idea that the global catastrophic risk that we all need to focus on and coordinate on is not just climate change and it's not just wealth inequality and it's not just um kind of the breakdown of pax Americana and the possibility of war or these species Extinction issues but it's lots of things there's lots of different Global catastrophic risks and that they interact with each other”

Every act of choosing to optimize certain metrics or pursue certain goals has ethical consequences whether through action or inaction, and people should not let awareness of overwhelming systemic problems prevent them from acting, as inaction also creates ethical consequences.

“don't let yourself become un agentic don't let yourself become so overwhelmed don't let yourself fall into easy certainties but also don't let yourself be overwhelmed by the total uncertainty that you can't act realizing that if you don't act there are ethical consequences to that too because we're on a moving train”

Adjectival consciousness refers to the specific properties and qualities of what is experienced (what is on the screen of attention), while adverbial consciousness refers to the framing, binding, and indexing process that brings those properties together into a unified experience

“the screening the framing of the screen which Bren basically says okay this is the frame and the relevance and the heess nness of what is going to be brought that is adverbial...what he calls adverbial Consciousness and...what you're looking at is akin to the qualia in a pure Consciousness event...what you're looking at is adjectival quality”

The Schottenberger assessment of the metacrisis differs from naive techno-pessimism in that it does not advocate for isolation or rejection of technology, but rather insists that addressing catastrophic risks necessarily requires fundamental changes to economic systems, governance structures, and human consciousness that would also constitute 'a whole lot more awesome' world than the current one

“I don't see a model where we do make it through those that doesn't also become a whole lot more awesome and that's what we say”

The characteristics that make people good at acquiring power (ambition, focus on self-interest, willingness to externalize costs) are nearly inversely correlated with the characteristics needed to be good stewards of power (concern for collective wellbeing, humility, systems thinking), creating a persistent problem where those with power are least equipped to use it well.

“the psychologies that are that want power and are good at getting it and the psychologies that would be the best stewards of power for the well-being of all are not the same set of psychological attributes they're pretty close to inversely correlated”

The phrase 'dual use' technology is misleading because it suggests only two uses (military and civilian), when in reality technologies have multiple uses across all possible motivations and applications across all actors

“every technology is also combinatorial with other Tech...but then what do we use it for is based on the motivational landscape I can use something like a hammer to be Jimmy Carter and build houses for the homeless with Habitat for Humanity or I can use it as a weapon and kill people with it um and so the tech has the affordances to do both of those so the tech will be developed and utilized based on motivational Landscapes sure and just briefly and going back to earlier it's not just dual because that would be double- edged it's multi-polar Omni”

The Sabbath in Judaic tradition may be understood as an attempt to bind a multipolar trap by creating a collective constraint that prevents competitive advantage through working longer hours, allowing societies to maintain social cohesion and cultural values that would be undermined if individuals raced to work more days than others

“if you say that you actually have to lock somebody up forever for smoking pot which feels very unfair to them but you probably only have to do that like a few times before nobody ever will touch it because the deterrent is so bad...the Sabbath was something like this...some very competitive people wanting to get ahead will work every day...everyone has to so nobody spends any time with their family”