Authors of "THE AI CON" discussing the problematic hype about AI & what those working tech should do

When evaluating whether to use LLMs for a task, the critical questions are: what is the specific task you're trying to automate, what input is available, how will users be using the output, and how will users have the information they need to use the output in an appropriate and reliable way?

“what my job is is building language technology and understanding how language works. And so my advice to somebody who is asking questions like what you're asking is to say, well, what's the specific task that I'm trying to automate? What is the input that I have available? How will the user be using the output? and how do I build something so that the user can that the user has the information that they need to use the output in an appropriate and reliable way.”

People interpret synthetic text from LLMs as meaningful because humans are exceptionally good at making sense of language by imagining what a person was trying to communicate, but this human meaning-making capacity is applied to text that was generated mechanically without intent or understanding.

“The problem is that as people we are so good at making sense of language and we are so used to doing that by imagining what the person who said the words was trying to tell us that as soon as we see the synthetic text we make sense of it.”

Knowledge and accountability in science depend on understanding the positionality and responsibility of people who did the work; when work is attributed to LLMs or generated using opaque processes, the chain of accountability breaks down and interpretation becomes untethered from authorial intention.

“The fundamental issue there is who's accountable for it. It's important to understand the positionality of the people who have done the work. That's one thing. And it's important that any anything that happens in language we interpret based on who we think actually has responsibility and accountability for it.”

Facial emotion detection systems are fundamentally flawed because facial expressions can be caused by many things unrelated to actual emotional state; people can be pressured to show expressions they don't feel; neurodivergent people may not make their face show what they feel in interpretable ways; and systems show differentially poor performance across different skin tones.

“So something that purports to measure emotions or ironically I've got my cat purring close to the microphone. So that so you know you definitely hear the purr but to to bring that into the conversation we believe that a purr is a sign of contentment in cats and we interpret it that way. And similarly we take a smile as a sign of joy or friendliness or something. But there's many different reasons that somebody might show something on their face that's at odds with how they're feeling. They might feel pressure to show something. People who are not neurotypical might actually have a very difficult time making their face show what they're feeling inside in a way that's interpretable by people around them. And so you can very quickly end up with these systems getting weaponized against people.”

Language modeling as a technology is old (going back to Claude Shannon in the 1940s) and genuinely useful in narrow applications like automatic transcription and machine translation, where it serves as one component to rank which interpretation of input is most likely given training data—but current large LLMs are trained on undisclosed data that amplifies rather than mitigates bias.

“language modeling as a technology is actually pretty old. It goes back to the work of Claude Shannon in the 1940s. And it's an important component of systems like automatic transcription and machine translation where you have some component of the system that says here's the words that that input signal, so words in another language or audio might have been. And then you use the language model to say well of those various possibilities which one of these looks most likely given the training data. That is actually quite useful. And the current transformer-based approaches allow us to have a much finer model of the distribution of the word form and text and so we get increases in performance in those technologies. My concern with the LLMs that everyone thinks about, so Gemini, Chad GPT, Claude, all these things is that they are built on absolutely enormous, poorly curated and undisclosed training sets.”

Detecting agitation in automatic speech systems is unnecessary; instead, systems should detect when audio signals differ from training data distribution and take appropriate actions—this avoids the false task of emotion detection while achieving practical robustness.

“Another example, turns out that automatic speech systems perform worse when people get agitated because we speak differently. And so we're different from the training data. And so you could imagine wanting a system that can detect when that's happened. And the nice thing is you actually don't have to detect agitation. You can just detect, oh, this audio signal is different to the range of things that were in the training data and so therefore take these actions. Right? So I think basically stepping away from this thing is humanlike and stepping away from this thing is set up to detect emotions positions you to actually make more effective automation.”

Machines trained on existing training data cannot replace scientific breakthroughs achieved through diverse perspectives and methodologies; this is supported by historians and philosophers of science who understand that scientific progress does not work through pattern prediction on existing data.

“There's no way that these machines that take existing training data and produce things that look, you know, predicting new things as if that is going to replace those point of views, those scientific breakthroughs. Ask any historian of science or any kind of philosopher of science. This is not how this work gets done.”

There is a fundamental misunderstanding of science in claims that LLMs will do science, imagining that anything making progress in LLMs is the only science, when in fact science across multiple fields uses diverse mechanisms and is funded through NIH and NSF grants which are being gutted at the federal level.

“I think there's a fundamental misunderstanding what science is. And in the artifact we reviewed last time on the podcast, the AI 2027 one, there was a kind of a notion of that that there was kind of science that would just be done by the LLMs, which is kind of a misunderstanding of what science is. First, it kind of imagines that anything that makes progress in LLM is the only science which is patently absurd, right? I mean, there is so much more that is being done in every different kind of field. And this is done with a lot of different mechanisms, right? It's with people who are coming from diverse backgrounds. It's people who are on these NIH grants, these NSF grants, agencies which are being absolutely gutted at the federal government in the US right now and we're losing out just on a generation of trainees in this.”

Physiognomy-based AI mirrors historical physiognomy and phrenology that claimed to determine criminality and personality from facial features, with clear racial and ability-based implications rooted in pseudoscience.

“Yeah. I mean, just remarks the advice to this person, I would I'd say don't do this. I mean, there's it's kind of mimics a lot of prior technologies that attempted to show emotions. I mean, there's a great paper by Luke Stark and Jean Tutton in a law journal which they called this physio physionic AI. And they kind of any assessment of kind of internal states that are in the assessments. It it really harks back to this early vision of physionometry in which there are kind of internal states whether it's criminality or personality determined by individuals faces and what they look like. As Emily mentioned, very clear kind of racial implications, ones that are based on ability.”

Science is not done when work remains private; the peer review process is a crucial part of science as conversation in which papers are collectively examined to determine which meet standards of scientific rigor.

“I like to say that science is a conversation. And you could do really interesting work and get some really interesting results, but if you've just kept it to yourself, you haven't done science.”

AI hype consists of multiple elements including comparing machines to humans (a pattern dating to 1956 when Marvin Minsky and John McCarthy claimed they could build circuits mimicking human thinking) and claiming that wide swaths of human tasks can be automated perfectly with no pain points.

“It looks like it's comparing machines to people. That's one move in the hype book and that's been around as long as AI AI quote unquote AI has been around. So 1956 when Marvin Minsky and John McCarthy were developing these technologies or they wanted to have a workshop around quote unquote thinking machines, Minsky is quoted to say we could probably build the circuit that is so good that it could mimic what humans do kind of in their everyday thinking.”

Physiognomist AI systems that attempt to detect internal states (emotions, criminality, personality) from facial appearance replicate historical pseudosciences of phrenology and physiognomy, which have clear racial and ability-based harm histories.

“Yeah. I mean, just remarks the advice to this person, I would I'd say don't do this. I mean, there's it's kind of mimics a lot of prior technologies that attempted to show emotions. I mean, there's a great paper by Luke Stark and Jean Tutton in a law journal which they called this physio physionic AI. And they kind of any assessment of kind of internal states that are in the assessments. It it really harks back to this early vision of physionometry in which there are kind of internal states whether it's criminality or personality determined by individuals faces and what they look like. As Emily mentioned, very clear kind of racial implications, ones that are based on ability.”

Science is fundamentally a conversation that requires peer review—not perfect, but a critical form of collective gatekeeping that ensures work meets standards of rigor; many 'paper-shaped objects' from tech companies do not meet this bar.

“I like to say that science is a conversation. And you could do really interesting work and get some really interesting results, but if you've just kept it to yourself, you haven't done science. But a really important part of that conversation is the peer review process. And peer review is not perfect. There's all kinds of issues in that system. But the idea that we collectively look at something and we look at it critically and do some it's a good kind of gatekeeping I think and it's not gatekeeping in the sense that we're saying who can and can't publish but which papers do and do not meet the bar of scientific rigor. And a lot of these things that we've taken to calling papershaped objects absolutely do not meet the bar of scientific rigor.”

When journalists like Kevin Ruse write credulous articles about AI rights and whether robots feel pain, they are doing bad journalism because they are not holding power to account, understanding the incentive structure behind AI narratives, or examining who benefits and who gets harmed.

“is not doing the job of journalism to be a journalist is to you know I think I would hold not to rely so much on access to these large institutions but to hold power to account to understand what's the incentive of of these narratives who's behind it who benefits and who gets harmed in the process.”

The appropriate use of AI in customer service is routing calls to human representatives based on transparent criteria (like an old phone menu that's annoyingly limited but clear about what it can/can't do) rather than attempting to fully automate customer service or infer emotional states.

“So I think stepping away from this is an automated customer service agent to this is an automated system that we use to help the customer connect to the right resource sort of making it a narrower use case and making it very transparent to the user about what it actually can and can't do is fine. Right? If you think about the press or say one phone menus those are a little bit annoying but you know what it's going to do. If you've gone through five options and none of them match what you're looking for, then you know that that menu can't do the thing you're looking for and you hope for the press zero to talk to a person directly kind of a thing, right?”

AI hype consists of multiple rhetorical moves that have recurred since the inception of AI (comparing machines to humans, promising wide-scale automation with no pain points), with elements like AI agents, AI tutors, and AI secretaries that anthropomorphize machines into human roles.

“So AI hype is a thing that exists where there is a technology and it does this thing notably word prediction next to word prediction very well or yeah not that well in some cases... It looks like it's comparing machines to people. That's one move in the hype book and that's been around as long as AI AI quote unquote AI has been around. So 1956 when Marvin Minsky and John McCarthy were developing these technologies or they wanted to have a workshop around quote unquote thinking machines.”

The claim that you could replace federal workers with years and decades of institutional knowledge (like Cobol at the Social Security Administration) by translating it into Java or an LLM is hype—it fundamentally misunderstands what institutional knowledge is and how it works.

“And we're seeing this kind of at the large scale in our federal government in which many people have been issued a reduction in workforce or reduction in force orders. And there's this claim that you're going to replace these workers that have so much institutional law knowledge that's hype. The idea idea that you could translate years and decades of cobalt at the Social Security Administration in the US government and translate it into Java, that is that is hype.”

World Coin (Sam Altman's biometric data project) is targeting people in the majority world (Kenya, South Africa) where extractive colonial conditions create incentive structures that encourage people to surrender biometric information for financial incentive—this is exploitation of postcolonial economic vulnerability, not innovation.

“With regard to the World Coin, I mean, you have to also see where World Coin was operating. They had a lot of effort to do this. In Kenya and South Africa, there were places in the majority world in which there was incentives being offered to give something of a likeness or to scan a retina to give this biometric information. So there's cases in which the incentives are pulled against people. We also saw this with regards to crypto in which maintaining crypto was seen as kind of a more secure investment in places in which fiat currency had been at more of a risk. That's not an issue. That's not some innovation in crypto. That's taking advantage of an extractive colonial economy in which inflation has come out of control. So there's hype around it. But I mean like you can hype better than pointing to that and saying like this is where this is offering real economic independence for people in the majority world. But no, that's taking advantage of postc colonial situations in the majority world.”

AI boosters and AI doomers represent two sides of the same coin—both assuming that LLMs will spontaneously combust into consciousness and either save humanity or destroy it, which is not based in reality, and framing these as the extremes of a valid discourse range narrows actual debate to this false binary.

“They are two sides of the same coin. You've got the boosters who say any moment now this is going to be AGI and it's going to give us fully automated luxury communism and it's going to solve all our problems. It's going to solve climate change. It's going to sol it's going to cure cancer. It's going to lead to this age of super abundance. And then you got the doomers who are saying it's going to kill us all. it's going to turn into the paperclip maximizer or whatever and decide that humans aren't important because it's got its goal and so on. And in the media, those two are often set up as the sort of extremes of the range that you can be in. So you're either fully for it because it's going to be great or you're fully against it cuz it's going to kill us all. And that is the full range of discourse. But from where we sit, it's actually all the same story. It's all this idea that these systems that are in particular the synthetic text extruding machines are somehow going to combust into consciousness and be awesome or combust into consciousness and be terrible. And none of that is based in reality.”

To responsibly build with or deploy AI, people must ask basic questions: What are the inputs and outputs? What data was it trained on? Is that data actually helpful? How is it evaluated? How will it actually be useful for your work? These practical questions differ fundamentally from abstract hype about capabilities.

“There's a lot of very basic questions and we offer this at the end of the book. What are the inputs and outputs of such a system?...What's the data that is trained on? Is that data going to be helpful? When people say it's something that's trained on the quote unquote whole internet, no it's not like what is the whole internet? There's a great study that came out of AI2 in which Jesse Dodge and some of his co-authors were looking into one of these data sets...How is it evaluated? Typically these companies are not releasing any evaluation metrics...people who are actually building with these things need to really evaluate what's actually behind the screen, what's going in, what's coming out, what's in the data, how is it evaluated, how's it going, how's it going to be useful for your actual work.”

Chat GPT was not 'bad on arrival' like the enshittification argument might suggest; instead, it was useful initially, but as capital demands limitless growth, it becomes subject to enshittification—a trajectory predicted by the theory applied to LLMs.

“The argument in the book is not that chat GPT is and should have that it's bad on arrival. Right? I mean this is a thing that's that not producing any kind of useful outputs. The kind of initification argument if it were to work is that maybe this is not only useful at one's job kind of for writing code or maybe kind of in initial playing around but then it becomes this thing in which employers rely on so heavily it becomes required in your work and we're actually seeing that”

Enshittification happens when Chat GPT moves from optional to mandatory at work despite inability to validate outputs—workers cannot take time to quality-check because they'd be left behind, creating a forced choice between using unreliable tools or being deemed unproductive.

“If you took the careful time to actually validate this, you'd be left in the dust, right?”

Emotional AI (systems that measure emotions from facial movements) cannot be validated as worthwhile because faces show emotions for many reasons (cultural pressure, neurodivergence, stress) unrelated to internal emotional states, and such systems are not equally effective across skin tones, making them likely to be weaponized against marginalized people.

“So that's a use case that I have a really hard time validating as a worthwhile one. I think the questions that I would ask is first of all, what's your intended use case, but how else might this be used and how is it going to be weaponized against people?...we take a smile as a sign of joy or friendliness or something. But there's many different reasons that somebody might show something on their face that's at odds with how they're feeling. They might feel pressure to show something. People who are not neurotypical might actually have a very difficult time making their face show what they're feeling inside in a way that's interpretable by people around them. And so you can very quickly end up with these systems getting weaponized against people. There's also the fact that the systems are not equally effective in handling faces across different skin tones. And so you can imagine you could very quickly get differentially bad outputs for people with darker skin.”

Resistance to AI hype is possible because the narrative of inevitability and technological determinism has been so dominant that people underestimate how much agency exists; recognizing economic headwinds and failed ROI opens space for resistance.

“I think a lot of it is because the narrative has been so dominated that resistance is feudal but we're not in a borg situation bringing back to Star Trek resistance is very much possible both kinds of in terms of banning of workers and also see where the economic headwinds are going where the promise and the return has to be huge to actually return on the investment.”

Large Language Models like ChatGPT, Gemini, and Claude are built on absolutely enormous, poorly curated and undisclosed training sets, which means we don't really know what's in them and we don't have the ability to reason about the kinds of biases that we would want to mitigate for particular use cases.

“My concern with the LLMs that everyone thinks about so Gemini, Chad GPT, Claude, all these things is that they are built on absolutely enormous, poorly curated and undisclosed training sets. And so we don't really know what's in there. we don't have the ability to start reasoning about the kinds of biases that we would want to mitigate for particular use cases.”

The asymmetry in valuing outcomes of AI systems is that concrete financial outputs (cryptocurrency value) have price tags while intangible human values like privacy, biometric autonomy, and person-to-person connection cannot be easily quantified in dollar terms—this creates an epistemological impasse where quantified harms appear less real than quantified gains.

“The whole world coin and retina scan thing is we live in a world where some things are overtly valued, right? So you can change cryptocurrency into real money at a specific value at a specific time and something like privacy and the impact of giving up personal biometric information is more difficult to value because it doesn't have a dollar sign on it. And similarly, sort of the the chance to be human together with other people, that sort of person-toperson connection and the opportunity to band together, as Alice was talking about, extremely valuable, but not going to have a price on it in the same way. And I think that's part of the impass. And I would never seed the ground that the ones with the dollar signs are more real because they're specific numbers.”

LLMs being used to write peer reviews creates a mockery of the peer review system because the point is not the words on the page but the thinking that went into the words—if an LLM generates the review, no thinking has occurred.

“And yes, there are issues with the peerreview process getting sort of snowed under because now it's very easy to create paper-shaped objects and people sometimes do throw those into peer review and there already is it's very hard to get the volunteer labor lined up to do the reviewing because it is volunteer labor in most systems. And then sometimes you get people who have bought into the hype who think that it's okay to put the paper into one of these services like chat GPT and take what comes back out as a review and then send that into peer review system which is just a mockery right what the yes peer review is one of those things that happens in text but again the point is not the words on the page it's the thinking that went into the words on the page now if we zoom out”

There is no presupposition of a path from language modeling to reasoning; reasoning requires building a model of the world with rules and patterns for inferring conclusions, which is fundamentally different from predicting which word is likely to come next.

“There's a presupposition in your question that there's a path from language modeling to reasoning. And I don't think there is.”

OpenAI's Whisper automatic transcription system, based on large language modeling, sometimes outputs random strings that were not in the audio input at all because the architecture decouples input from output, allowing the language model's text fabrication to dominate.

“folks have shown, and I'm sorry I could look up the citation for your show notes, that Whisper, for example, which comes from OpenAI, which is a a automatic transcription system based on large language modeling technology, will sometimes output just random strings that weren't in the audio input at all. And that's because the architecture has changed from really being tightly coupled input to output to effectively letting the language model do its text fabrication thing.”

The privileging of LLM-based approaches as 'science' fundamentally misunderstands what science is and ignores the actual diversity of scientific work done across fields with different mechanisms and done by people from diverse backgrounds supported by NIH and NSF grants—agencies currently being gutted, losing a generation of trainees.

“I'd also say there is a privileging of a certain discipline in doing this work, right? I think there's a fundamental misunderstanding what science is. And in the artifact we reviewed last time on the podcast, the AI 2027 one, there was a kind of a notion of that that there was kind of science that would just be done by the LLMs, which is kind of a misunderstanding of what science is. First, it kind of imagines that anything that makes progress in LLM is the only science which is patently absurd...There is so much more that is being done in every different kind of field. And this is done with a lot of different mechanisms, right? It's with people who are coming from diverse backgrounds. It's people who are on these NIH grants, these NSF grants, agencies which are being absolutely gutted at the federal government in the US right now and we're losing out just on a generation of trainees in this.”

People anthropomorphize LLMs by granting them a mind behind their words and imagining mutual meaning-making is happening, but this is dangerous because it leads to conversation about whether robots feel pain or have rights instead of the critical conversations about how these technologies are displacing work, devaluing human labor, and attacking existing institutions.

“The danger is to grant, as Emily had suggested, this notion that there is a mind behind the words and that there is any kind of mutual meaning making that's happening between these word prediction machines, right? And so we make meaning with people in different ways... The useful conversations are the ways in which these technologies are supplanting work are being used as a means to devalue the work of people to attack existing institutions.”

The useful use cases for LLMs are where the goal is modeling the distribution of word forms in text, such as automatic transcription and machine translation, where language modeling serves as a component that identifies likely word sequences from ambiguous inputs—these are relatively narrow use cases.

“So they have to be use cases where what you're interested in is a good match for modeling the distribution of word forms in text. And language modeling as a technology is actually pretty old. It goes back to the work of Claude Shannon in the 1940s. And it's an important component of systems like automatic transcription and machine translation where you have some component of the system that says here's the words that that input signal, so words in another language or audio might have been.”

Automatic transcription systems can perform worse when people are agitated because humans speak differently when distressed, making the audio signal different from training data—this can be addressed by detecting distribution shift (audio different from training) rather than trying to measure internal emotional states.

“Another example, turns out that automatic speech systems perform worse when people get agitated because we speak differently. And so we're different from the training data. And so you could imagine wanting a system that can detect when that's happened. And the nice thing is you actually don't have to detect agitation. You can just detect, oh, this audio signal is different to the range of things that were in the training data and so therefore take these actions.”

Natural language interfaces require extra work to maintain transparency because natural language is flexible and looks more capable than it is; designers must work hard to prevent users from overestimating system capability.

“if we're doing something that is a natural language interface, we have to work extra hard to maintain that transparency because natural language as we use it is very flexible. And so it's going to look more flexible than it is. So if you basically empower the user to know what it is that they can expect to get out of the system, then you can also set it up so that the system isn't expected to be able to reason about, let's say, you know, systems that can't reason.”

Studies show that code written by LLMs tends to be low quality, requires extensive oversight, has been introducing more bugs into code, and in the case of GitHub Copilot (per a MITRE study), has reintroduced existing security vulnerabilities—making security a major reason companies have been cold or hesitant in adopting LLMs.

“I think there had been another study that that said that in the cases in which there was code writing that had been done, it tended to be that the output was pretty low quality or there had to be a lot more oversight on this. And there's been other research that shown that it's been introducing more bugs into code writing. And there was an older study with co-pilot done by the independent research agency MITER that had said that about there had been the reintroduction of a lot of existing security vulnerabilities and so security has been a huge issue. A lot of the cases in which companies haven't adopted or have become cold in adopting LLM has been because of security issues of cyber security issues.”

Academic papers published as preprints by tech companies (Microsoft, Anthropic, OpenAI) on their own websites bypass peer review entirely, violating the original purpose of preprints and undermining the gatekeeping function of peer review that ensures scientific rigor.

“So, first of all, they're not published in the same sense, right? Many of these papers are just called preprints and thrown up on archive or not even that and they're just on company websites. And preprints originally, the idea was the publishing process is slow. This paper is finished. It's passed peer review and I want it to be accessible to people and it's going to take, you know, months or whatever for it to show up in the conference or the journal. So, fine. But now it's like, oh, I peer review. Who's that? Right? Never heard of her.”

The major hypemen behind AI are billionaires and tech executives (Sam Altman, Dario Amodei) and their funders (Andreessen Horowitz, Sequoia Capital, SoftBank) as well as corporate leaders (Satya Nadella, Sundar Pichai, Brett Smith), plus ideological architects like Eliezer Yudkowsky and Nick Bostrom who promote transhumanism and longtermism, plus credulous journalists with cultural capital (Kevin Roose, Casey Newton).

“There's a lot of different hype man. It's to say suggested is necessarily the culpability of people like you and I who are small-time content creators or academics is probably not the case. It's more the people who are in control of these large companies, your Sam Alman's, your Dario Amadees and also their funders. So Andre Horowitz, Sequoia Capital, Softbank and then the people who are funding those at big tech companies, your Sajia Nadellas, your Sundai, your Brett Smiths. I mean those are people who are major players just in terms of the people who are institutionally your your your major hypeman. You also have a lot of people who are the architects of very very disturbing ideologies like the the testal bundle the transhumanists transhumanist extropinists singulararians cos cosmicisms your people like Eliza Yudski people who are these AI doomers people who think that if we optimize things to a certain degree we're going to have perfect morality your Nick Bowstrms the long term the people take this long termist view”

Writers in the 2023 Writers Guild of America strike opposed LLM tools in the writer room both because they could be handed scripts to rewrite at lower rates (rewrite rate vs original writing rate) and because they signaled that writers were disposable—AI tools served as a threat-capacity to devalue worker labor.

“This was a major theme in the writers guild of America strike in which the Hollywood writers who are not living glamorous jobs I mean these are folks who are piecing together multiple writing gigs to stay afloat and in the negotiations in addition to being a major contention around residuals from suing services these workers were also opposing this kind of notion of the AI tools being in the writer room it supposed as possibly it's in the writer room. You might have to do rewriting of a script that might be produced or the output of a synthetic text model. But writers also knew that in that they had the potential of being handed a script and having to rewrite it and being paid a rewriting rate which is much lower than the writing rate, the cost of producing original content.”

Large language models like Gemini, ChatGPT, and Claude are built on absolutely enormous, poorly curated and undisclosed training sets, making it impossible to reason about the kinds of biases that should be mitigated for particular use cases.

“My concern with the LLMs that everyone thinks about so Gemini, Chad GPT, Claude, all these things is that they are built on absolutely enormous, poorly curated and undisclosed training sets. And so we don't really know what's in there. we don't have the ability to start reasoning about the kinds of biases that we would want to mitigate for particular use cases.”

Tech workers can engage in collective action and solidarity across industries to resist AI automation—they are workers, and the same labor organizing strategies that have worked across other sectors apply to tech as well.

“Yeah, resistance is always an option and tech workers are workers, right? And they can come together in solidarity amongst themselves and across industries.”

Being a software engineer involves much more than writing code—it requires design thinking, testing, security consideration, and collaboration—and automating code output does not automate software engineering.

“So being a software engineer is more than just writing code for example. And even if you can automate in a janky way the output of code that sometimes helps you that doesn't mean you've automated software engineering.”

The book 'The AI Con' argues that while LLMs have some narrow legitimate use cases (transcription, translation), the hype mountain vastly exceeds actual capability, and the positive use cases 'wouldn't even be a foothill' compared to the exaggerated claims.

“So the collection of use cases if you put them next to mountain wouldn't even be a foothill right but there is there is a collection of positive use cases for this kind of technology.”

People often interpret convictions with confidence even when they're not reasoning at all—we are so good at making sense of language that when we see synthetic LLM text, we make sense of it through our own meaning-making, attributing intelligence to the system when we should attribute it to ourselves.

“And the problem is that as people we are so good at making sense of language and we are so used to doing that by imagining what the person who said the words was trying to tell us that as soon as we see the synthetic text we make sense of it.”

Work, especially care work like nursing, teaching, and eldercare, is intensely human-to-human and cannot be meaningfully automated by LLMs because it fundamentally requires human relationships, comfort, and presence—not just knowledge dispensing.

“I mean I think it also takes a it's a very narrow idea of what work is. If you think about all of the kinds of work in the world, think about especially care work. So caring for the elderly, caring for children, being a teacher, working in a hospital, all of that is so like intensely human to human.”

The job of journalism is to hold power to account, understand the incentives behind narratives, and examine who benefits and who gets harmed—not to rely on access to large institutions or amplify their claims without criticism.

“It's not doing the job of journalism to be a journalist is to you know I think I would hold not to rely so much on access to these large institutions but to hold power to account to understand what's the incentive of of these narratives who's behind it who benefits and who gets harmed in the process.”

Understanding the positionality and accountability of the people who have done the work is fundamental to how we interpret language and claims about responsibility—this is why anonymously generated text (from LLMs) undermines knowledge accountability.

“The fundamental issue there is who's accountable for it. It's important to understand the positionality of the people who have done the work. That's one thing. And it's important that any anything that happens in language we interpret based on who we think actually has responsibility and accountability for it.”

Focusing on automation rather than augmentation in AI deployment represents a choice in how to design labor relationships, not a technological inevitability; designers can choose to support worker capability instead of replacement.

“stepping away from this thing is humanlike and stepping away from this thing is set up to detect emotions positions you to actually make more effective automation”

Studies have shown that code generation LLMs produce low-quality output, require high levels of oversight, and have been found to introduce more bugs into code, with security vulnerabilities being a major reason companies have not adopted or have become cold to adopting LLMs.

“I think there had been another study that that said that in the cases in which there was code writing that had been done, it tended to be that the output was pretty low quality or there had to be a lot more oversight on this. And there's been other research that shown that it's been introducing more bugs into code writing.”