Google Translate translated 'the legislator accidentally left a copy of the important bill he was writing in the taxi' with 'bill' as 'facture' (an invoice, like a plumber might give) rather than as a legislative bill, showing a failure to understand context, and translation errors have real effects when the US uses Google Translate to translate documents from refugees, resulting in disqualification.

A transformer network is a type of neural network architecture used by large language models that includes an embedding layer (turning words into patterns of numbers), attention layers (computing interactions among words), and traditional neural networks, arranged in multiple layers (GPT-3 has about 100 transformer blocks pipelined together).

GPT (Generative Pre-trained Transformer) models work by predicting the probability of the next word: for every word in their vocabulary (about 50,000 words), they compute the probability that word should be output next, and the system selects the word based on these probabilities.

Large language models are trained on enormous amounts of text data from the internet, Wikipedia, Reddit, digitized books, and computer code—ChatGPT was trained on approximately 500 billion words, whereas a typical human child encounters approximately 100 million words by age 10, making ChatGPT's training data about 5,000 times larger than a child's linguistic exposure.

Ilya Sutskever, co-founder of OpenAI, called GPT-4 'the most complex software object ever made,' suggesting it is more intricate than other complex software objects in the world, though this claim's validity is uncertain.

A 2020 paper on 'emergent abilities of large language models' documents that LLMs exhibit abilities they were never explicitly trained to have, including passing business school tests, law school exams, medical licensing exams, writing poetry, translating languages, and solving math problems.

Some researchers claim that 'scale is all you need'—that scaling up current language models without changing the architecture will eventually lead to general intelligence, by simply converting everything to tokens and predicting the next token.

Chris Manning, head of Stanford's AI lab, said there is 'a sense of optimism that we're seeing the emergence of these systems that have a degree of general intelligence,' but this claim is contested within the AI field.

Alison Gopnik, a prominent cognitive psychologist, argued that the terms 'intelligence' and 'agency' are wrong categories for thinking about LLMs, which are better understood as large databases or libraries with a natural language interface.

Yann LeCun, head of AI at Meta, and Jake Browning argued that 'a system trained on language alone will never approximate human intelligence even if trained from now until the heat death of the universe,' suggesting that language-only training has fundamental limitations.

Mitchell wrote an article called 'How Do We Know How Smart AI Systems Are' highlighting that tests of AI capabilities often don't reveal limitations because: (1) test questions may have been in the training data, (2) the vast training data makes it unclear what was memorized vs. learned, and (3) systems may use narrow, non-transferable procedures rather than genuine understanding.

The 1956 Dartmouth Workshop proposal stated an attempt will be made to find out how to make machines use language, form abstractions and concepts, solve kinds of problems reserved for humans, and improve themselves, with the belief that a significant advance can be made if a carefully selected group of scientists work on it together for a summer.

Frank Rosenblatt's Mark 1 perceptron neural network in 1957 led the Navy to fund neural network research, and the New York Times reported in 1958 that the Navy revealed 'the embryo of an electronic computer today that expects will be able to walk talk see write reproduce itself and be conscious of its existence.'

Claude Shannon, the inventor of information theory, confidently expected that within 10-15 years from 1961 we would have robots of science fiction fame, and Herbert Simon (Nobel Prize winner) believed that within 20 years from 1965 we would have machines that could do any work that a man could do.

After earning a PhD in AI in 1990, Mitchell was advised not to use the term 'AI' on her job applications because job prospects for AI researchers were poor due to the AI winter.

With the rise of the internet and the worldwide web, huge datasets like ImageNet (1.5 million human-labeled images) became available, enabling machine learning systems trained on fast hardware to learn to recognize objects in images, leading to the Deep Learning Revolution around 2010.

A 2017 paper titled 'Neural Networks Are Easily Fooled' showed that a deep neural network confident it recognized a school bus with 100% confidence was 99% sure it was a garbage truck when the object was photoshopped into a weird pose, or 100% sure it was a punching bag in another pose, demonstrating lack of robustness.

Hans Moravec proposed Moravec's Paradox in the 1980s: it is comparatively easy to make computers exhibit adult-level performance on intelligence tests or games like checkers and chess, but difficult or impossible to give them the skills of a one-year-old in perception and mobility, and Mitchell would add common sense to this observation.

When asked 'How many states in the US have names beginning with the letter K?', ChatGPT confidently answered 'four: Kansas, Kentucky, Kansas, and Kentucky' (listing the same states twice), failing a simple counting task that requires understanding.

Mitchell's research group tested abstract reasoning puzzles where humans are shown three demonstrations of a grid transformation and asked to identify which test grid follows the same pattern; humans achieved about 90% accuracy but GPT-4 only achieved about 33% accuracy on 480 such puzzles, suggesting GPT-4 lacks basic abstract reasoning abilities.