Insights for Making Better Decisions | Gary Klein | Knowledge Project 144

Groups as a heuristic make terrible decisions; individuals should make decisions while groups provide input, which increases both quality and accountability, since with group decisions no one is responsible when wrong and everyone claims responsibility when right.

“I have problems with group decisions. I think groups as a rule like a heuristic make terrible decisions. Individuals should make decisions. Groups should provide input. Um but there should be one sort of person making the decision.”

Cognitive flexibility theory posits that expertise can be developed by preventing people from locking into routines and standard approaches, giving them vicarious experiences that can't be handled by usual routines to force them out of comfort zones into adaptive mindsets.

“cognitive flexibility theory is the the notion of trying to help people achieve expertise by um preventing them from locking in to routines and standard ways of doing things so that they can become more naturally adaptive. And um you can give them vicarious experiences that can't be handled by the usual routines just to force them out of their comfort zone”

Gary Klein disagrees with the dominant heuristics-and-biases research community's focus on debiasing, arguing that attempts to debias people have largely failed and that cognitive biases are actually reflections of useful heuristics that only look like biases in hindsight.

“I have to be careful about this. And just caveat what I'm about to say that I'm an outlier. That the whole issue of decision and judgment biases is a major line of research... However, I don't agree. I don't like the idea of decision biases. I know my sense of the literature is that there's been little if any success in debiasing people. And I think that's a good thing.”

There are three distinct pathways to insight: connection pathways where ideas are combined without contradicting existing beliefs (e.g., Darwin connecting population growth to evolution); contradiction pathways where anomalies trigger investigation without requiring belief change (e.g., the police officer noticing ashes flicked in a new BMW); and correction pathways where flawed beliefs must be unlearned through curiosity about anomalies.

“One pathway is a connection pathway where we put things together... We're not contradicting anything... Darwin, you know, knowing that there was uh change in species, but what's driving the change? And then he reads Malthus's book about uh uh population growth and competition for scarce resources.”

People drive in low-power mode (automaticity with minimal attention), and anomalies switch them to high-power mode (alert and engaged), but at work people encounter anomalies and ignore them or explain them away rather than investigating, missing the cue that their mental model is misaligned with reality.

“we're not consciously paying attention to the road, but we notice anomalies. We notice something that is unexpected. And in in a vehicle, that that's our cue to pay attention. Right. As a driver of a vehicle, for whatever reason, we get encoded with this after years of driving. It's like, 'Okay, now I I'm switched on. Maybe my heart races a little bit, but like I react. I get out of my trance, and my body awakens.'”

Availability bias, representativeness bias, and anchoring bias are actually beneficial heuristics: availability uses relevant experience to guide decisions, representativeness uses similarity to recognize patterns, and anchoring provides a starting point that expertise allows proper adjustment from.

“availability bias. Of course, that's going to be helpful if I want to draw on my experience in order to know how to work in a situation... representativeness bias. Anchoring bias. I'm going to anchor on a number... generally I've got to start somewhere. And so, I draw on my experience to see what's this like and I'll I'll anchor on that and I'll adjust from that because my initial impression if I have if I have a reasonable amount of expertise is going to be accurate.”

Organizations can increase insight adoption by creating appeal mechanisms so that if someone believes an idea has been rejected prematurely, they can request re-examination by a different part of the organization, rather than allowing single gatekeepers to kill promising ideas.

“they could give you or they could give me an option for a review if I think the idea has been rejected prematurely. I can I really appeal I don't want to make trouble, but there there could be somebody there could be uh uh some organ- some part of the organization that could re-examine re-examine the idea.”

Decision journals—written records of decisions including the decision itself, goals, prime information used, what's being noticed, what's driving the decision, and affected stakeholders—create a record of thinking at the time, allowing retrospective learning about missed cues and unconsidered goals.

“I would like to know when the employee is making a decision, um what is the decision? What are the goals that the employee wants to achieve?... what are the prime pieces of information the employee is using to make the decision?... Who are the other people or or teams that are going to be affected by this decision?... in retrospect, you know, we we can see what were there Were there Were there cues that I should have been paying attention to that I wasn't?”

Learning from others' abstractions (learning by teaching) differs from learning through direct experience: learning abstractions is like exact pattern matching (searching with quotes), while learning through experience is associative pattern matching (fuzzy search), and reflection on experience enables the fuzzy learning to work.

“When you learn a concept from somebody else, you're you're learning their abstraction. So, you're putting an abstraction into use as your action. And if you take the time to reflect, well, reflection is how we learn those edge cases... in the first case, when you're learning from somebody else, I think you have like pattern matching, but it's exact pattern matching. It's like searching Google with quotes. Whereas in the second case, you're doing pattern matching, but it's associative pattern matching.”

Organizations can provide vicarious learning experiences by encapsulating crises that more experienced workers have faced into stories or scenarios, giving less-experienced workers learning surface area without requiring them to live through all the actual crises.

“You can take some of these um crises that worker that the second worker had and encapsulate them in story or a scenario, and you can present them to worker A just to to give them vicarious experiences to to get them to move forward.”

The pre-mortem is a structured 20-minute exercise conducted at project kickoff where the team assumes the project has failed catastrophically, spends 2 minutes individually writing reasons for failure, then shares non-redundantly around the room, and finally spends 2 minutes identifying what each person can do to reduce risk.

“How a pre-mortem works. Um we didn't need one. I just sort of invented it... I'm looking in an imaginal imaginary crystal ball and it's now 6 months from now. Uh and this project has gone off the rails in a major way. It's been a disaster... What What happened is the crystal ball is showing failure... everybody around the table, they have a pad of paper in front of you and a pen. Take the next 2 minutes and write down all the reasons why this project failed.”

Experienced leaders often reject innovations because they remember past failures with similar ideas, but the context may have changed—technology may have matured, world situation may have shifted—making ideas that previously failed now viable, yet experienced leaders' scar tissue from past failures blocks adoption.

“people with with experience have lots of scar tissue of things that that got tried before and failed... People trying it and failing and then somebody else trying it and failing until eventually it succeeds. But until you get to the point where it succeed, you're going to have people in the experienced people, the senior people in the organization, what are they going to say? They're going to say, 'You know, we tried that and it didn't work.'”

A decision scorecard method of evaluating employees—reviewing major decisions made over the past year, assessing which worked and which didn't, examining the quality of thinking at the time (not retrospectively), and learning from both successes and failures—is less stressful and more educational than traditional objective-based evaluations that become stale.

“Instead of evaluating their have you know, I meet my objectives... use a a decision scorecard. I sit down with the employee... Let's go back in the previous year. What were some of the major decisions you made?... which ones worked and which ones didn't? Now, some decisions may have been good decisions that didn't work through no fault of the employee. And some may have been bad decisions that worked because the employee got lucky.”

Pre-mortems work by reversing the usual incentive structure of meetings: in typical kickoff meetings at the end, no one wants to voice problems after hours of planning discussion, but in pre-mortems, demonstrating competence is redefined as surfacing high-quality problems others haven't considered, creating a competition for insights.

“there's a competition cuz I want to come up with good problems that nobody else has come up with. And so, people are in this competition to surface problems that in an ordinary meeting nobody would ever say.”

Klein challenges bias researchers to accept elective surgery that would lesion the ventromedial prefrontal cortex to eliminate emotions from decision-making, which he would fund; no bias researchers have accepted the challenge, suggesting even they don't believe emotions should be eliminated from decisions.

“So, you will no longer be troubled by emotions when you make decisions. And my challenge is to any of the judgment decision researchers who worry about emotions crowding our decisions, would you have the surgery done? I have yet to find a backer. And I've said I will pay for the surgery myself.”

Project review meetings can be enriched by asking 'Has anything happened in the last 3 months that surprised you?' because surprises in complex situations indicate that mental models need updating; absence of surprises in a wicked problem is concerning rather than reassuring.

“I can also ask you in that review meeting, 'Shane, in the last 3 months since the our last meeting, has anything happened that surprised you?' And if we're dealing with a complex situation, with a wicked problem, that's different from something that's sort of like painting a room where everything is predictable. So I say, 'Has anything happened in the last 3 months that has surprised you?' And if you say, 'No, no, nothing nothing has surprised me.' You don't have to worry. That's when you have to worry.”

The 'Good Strangers Project' (funded by DARPA) studied military personnel and police officers skilled at de-escalation, finding that the key to gaining voluntary compliance is trust-building rather than intimidation, with the goal of leaving interactions with greater trust than you entered.

“the nickname for this project was funded by DARPA, Defense Advanced Research Projects Agency. The nickname was the Good Strangers Project. And we studied military uh personnel, and we studied police officers who were known to be good at diffusing situations, at creating an environment that was a benign rather than a high-conflict environment.”

A large, strong police officer learned through experience that while he nearly always won physical confrontations, he was accumulating damage; he observed a colleague who used verbal de-escalation techniques to gain compliance, and shifted to building a repertoire of 'golden words' to obtain voluntary compliance instead.

“I remember one one one police officer that we interviewed... He was a large police officer... he said, 'I joined the police because I liked the action... I noticed as my career was going on, if I would sometimes get in fights with people, and I almost always won the fight. But I always There was damage. I was accumulating damage. And then I looked at one of my colleagues. He could get people to do things for him and just by the way he carried himself... what's he doing? What What's his technique? And he said, ever since then I've been trying to come up with golden words'”

Cognitive interviews—deep questioning about how someone sized up a situation, what they noticed, and what inferences they drew—provide better insight into decision-making than surface-level explanations like 'I thought the market was due for a correction.'

“we need to go beyond a simple surface explanation of why people did things. And we want what what all my colleagues and I talk about as cognitive interviews. To get beyond the surface explanation to find out how were you sizing this situation up? What were you noticing? And you know, what what inferences were you drawing?”

People stagnate at intermediate skill levels rather than progressing to mastery because they stop unlearning—they must discard conventions and beliefs they've adopted that either are wrong or have limited applicability as conditions change.

“people get up to a certain level of performance, and then they start to stagnate. And they start to plateau. And people who who break through and move to the next level um are the ones who have to who engage in unlearning. Who realize there are certain uh conventions they've bought into or beliefs they hold that are either wrong or are limited and and don't apply as broadly as as they imagine.”

An effective story must have mystery (something unexpected happens), the storyteller must be someone who naturally tells stories (some people's brains aren't wired to recall or construct stories), and critically, a good story results in insight—people experience a realization they didn't have at the beginning.

“What makes for an effective story? Um first of all, it's uh there's got to be some sort of mystery about something happened that wasn't expected. And and then people listening to the story want to know how it how it uh wound up. Second is the storyteller uh should be somebody who likes to tell stories.”

Shadow-boxing is a scenario-based training method where learners work through challenging scenarios, making decisions and providing reasoning at decision points, compare their rankings to expert rankings, and examine expert reasoning to see the world through expert eyes.

“Shadow-boxing works. It's primarily a scenario-based approach. I create a tough, challenging scenario. And we we go through the scenario... I stop it part of the way through and I say, 'Here's decision point one, Shane. At this point, here's four different options. There are things you can do, courses of action. Rank order which you would do from top to bottom, and then write down your reasons.'”

Fixation is caused by explaining away anomalies and holding onto the original wrong impression until it becomes too late to correct course; the way to break fixation is to notice anomalies, not to discourage quick reactions, because quick pattern-based reactions are usually right but must be revisited if anomalies emerge.

“we quickly make sense of it. We we use the patterns that we've built up. And uh usually works, but it doesn't always work... some people say we should discourage people from coming up with immediate reactions. But that's ridiculous... So instead you want us to come up with a a quick reaction, but if something if we're wrong, there's going to be an anomaly, and we want to be able to to revisit it.”

The mindset shift from 'I want people to obey me through toughness' to 'I want to earn their trust' is an important re-framing in high-stakes interpersonal decision-making, showing how internal goals reshape behavior patterns.

“In terms of mindsets, that's such an important mindset shift. His previous mindset was, I want to get people to do what I tell them, and so I've got to be tough, and I've got to be threatening. And his new mindset was, how can I get them to do what I want, and um and have faith in me.”

In a petrochemical plant study, two promoted workers were run through decision-making exercises: the confident one who impressed everyone actually lacked understanding of critical plant dynamics, while the tentative one who worried colleagues actually had deep causal knowledge; the tentative one was cleared for panel operations while the confident one was sent back for another year of training.

“One of the people was somebody everybody felt good about. He just carried himself with great confidence... And the other one was was sort of uh tentative about things... ran them through the decision-making exercises. What did they find? The first person didn't really get it. He didn't really understand critical dynamics of the plant. And the second person, for all of his uh caveats, really did get it.”

Individual-level strategies to maximize insight include being deliberately insight-focused rather than defensively recoiling from unexpected information, celebrating one's own small insights throughout the day, attuning oneself to insights others have, and cultivating curiosity by wondering 'what does that mean?' for a few seconds rather than recoiling.

“You can be deliberate... you can say, 'I want to have more insights.' Which means taking an insight stance rather than a a stance that recoils at something new and unexpected and jarring. You can become curious. You can start to celebrate your own insights.”

The training exercise's consequences backfired—after revealing that one worker was unsafe, no one wanted to do further decision-making training because the training was now associated with evaluative consequences, turning a learning tool into a threat.

“after they did that, nobody wanted to engage with the decision-making exercise. They realized if you don't do well, you could have consequences. And And they weren't intending the consequences to be to be used to to to evaluate people. But because they used it in that way, nobody wanted to do any more of that training.”

To evaluate whether someone has expertise, you can examine their mental model by running them through a scenario and assessing the richness of their reasoning—which shows whether they understand causal relationships and critical variables.

“you run them through a scenario. And um see what choices they make, find out what their rationale is, find out the reasons that they're picking one thing over another, and use that to determine if if you think that their mental model is rich enough.”

Pre-mortems create a culture of candor in teams where people become comfortable expressing problems without fear of criticism, and they create a sense of trust that concerns can be raised without repercussions.

“it also creates a culture of candor in the in the team where people are used have are starting to get used to expressing problems rather than covering them up. And it creates a sense of trust that I can say something and I'm not going to get criticized for it.”

Consensus decisions in high-stakes situations create pressure for conformity and can lead to risky outcomes; in The Martian crew example, the consensus to rescue Matt Damon's character involved enormous risk that could have endangered everyone, unlike private voting which would have allowed dissent.

“the decision is do we go back or not?... they said let's come up with let's see if we can come to a a consensus about what to do. And I don't like the idea of consensus decisions and I don't like the idea of a consensus decision in this kind of a dangerous environment. Because as the people on the spaceship went around there was enormous pressure on everybody to go along with the with the the the consensus which was we should go back”

A single decision-maker can acknowledge other people's problem definitions while ultimately being responsible for defining the problem for the group, allowing the leader to synthesize multiple perspectives while maintaining clarity and direction.

“if you have a sole decision maker instead of a group you can acknowledge that the responsibility of that decision maker maybe to listen to other people's definitions of the problem but ultimately they're responsible for defining the problem for the group and for themselves.”

Consensus decisions tend to be risk-averse and suppress bold or innovative ideas because they require everyone to agree, which incentivizes settling on the safest common ground rather than allowing individuals with new insights to advance novel ideas.

“With the consensus is to try to do um the safe thing that's comfortable that everybody can can live with rather than to do something that's bold and innovative. So, I have I have problems with consensus decisions.”

A worker with a turbulent year (many problems, failures, adaptations, diagnoses) develops far richer and more detailed expertise than a worker with a smooth year, because surface area of learning correlates with the diversity and density of challenges encountered.

“Who is the one that you would like to have working with you after that first year? Worker A or worker B?... I would rather I I don't know how you're going to answer, so rather than Oh, I would choose person B because the surface area of their learning is so much richer and detailed. Exactly. So worker the second worker with a turbulent year has learned much more.”

Knowledge shields—techniques for explaining away inconvenient data and anomalies—are used by all people and even scientists, who resist information that challenges their theories, demonstrating the universal tendency to hold onto one's existing story about the world.

“researchers who have investigated this find that scientists do the same thing. They have something that's called knowledge shields, the way all of us do, that that a a a variety of techniques for explaining away inconvenient data and anomalies.”

Organizations block insights by design because insights are disorganizing and disruptive, and managers prioritize smooth operations and predictability over innovation, even though they claim to want insights and innovation.

“What blocks insight is that organizations say they want insights and they want innovation and they really don't... insights are disorganizing... most organizations actually inhibit insights. When when when people bring up new ideas, they they tend to stifle them.”

In group decision-making, having each member individually and privately write out the problem they think they're solving, then comparing, reveals significant variance in problem definition; this variance must be acknowledged before jumping to solve the problem.

“When you go to a decision to make or go to a meeting to make a decision have everybody in the room write out the problem on a piece of paper that they think they're solving with this decision and then compare how different and how much variance there is in those problem statements.”

A mental model has four components: how something works (mechanisms), what its limitations are (boundary conditions), how to work around those limitations (workarounds), and how to anticipate where others might get confused or make mistakes.

“I thought that a mental model was a set of beliefs about how something works... Here Here's how it works... But we spent a a week at a petrochemical plant... And we found that there was another part of the mental model, um not just how does it work, but what are its limitations, what are its boundary conditions, where can it go wrong.”

The zone of indifference is the range where two decision options have nearly balanced advantages and disadvantages, making the choice extremely difficult psychologically even though the outcome is similar regardless of which option you pick—recognition that you're in this zone enables efficient decision-making.

“if I've got two choices, a terrible option and a wonderful option, quick, which one do you pick? Okay, that's not a hard decision. As we move them closer together, and so now the the terrible option has some positive features and the great option has some negative consequences, and we get to this point where it's really hard because uh the strengths and advantages of of one versus the other are almost totally balanced. And we get to this point, these are the hardest decisions people ever wrestle with. And the paradox is if the advantages and disadvantages of the two options are almost perfectly balanced, it doesn't matter which one we pick.”