Embracing Uncertainty: The Cognitive Dimension with Gary Klein, PhD | Ep 17

Gary Klein, PhD is an internationally renowned expert in decision-making, creativity, and leadership. He is a research psychologist, most famous for pioneering the field of Naturalistic Decision Making (NDM), along with his groundbreaking recognition primed decision (RPD) model, which has been used to revolutionize the way the Marines and Army train their officers. Dr. Klein has written several books, articles, and chapters about these topics, such as Sources of Power: How People Make Decisions, The Power of Intuition: How to Use Your Gut Feelings to Make Better Decisions at Work, and his most recent book, Snapshots of the Mind. He is the founder and CEO of Klein Associates, a consulting firm that specializes in decision-making, creative problem solving, and leadership development. Dr. Klein is a much sought-after keynote speaker and workshop facilitator, and has also developed the widely-used Learning-Focused Leadership model. His work has been honored with awards from the American Psychological Association, the Society for Industrial and Organizational Psychology, and the International Society for Performance Improvement.

Be sure to use the Chapters Feature on Apple and Spotify to quickly browse and navigate to segments of this episode.

Gary Klein, PhD
Gary Klein on LinkedIn
Snapshots of the Mind
Linking Expertise and Naturalistic Decision Making
Red Team Handbook

AGLX Confidence in Complexity short commercial 

Want to develop your organization’s capacity for free and independent action (Organic Success)? Learn more and follow us at:
https://www.aglx.com/
https://www.youtube.com/@AGLXConsulting
https://www.linkedin.com/company/aglx-consulting-llc/
https://www.linkedin.com/in/briandrivera
https://www.linkedin.com/in/markjmcgrath1
https://www.linkedin.com/in/stevemccrone
https://flowguides.org/
https://www.getflowtrained.com/

Recent podcasts where you’ll also find Mark and Ponch:

Eddy Network Podcast Ep 56 – with Ed Brenegar
The School of War Ep 84 – with Aaron MacLean
Spatial Web AI Podcast – with Denise Holt
OODAcast Ep 113 – with Bob Gourley
No Fallen Heroes – with Whiz Buckley
Salience – with Ian Snape, PhD
Connecting the Dots – with Skip Steward
The F-14 Tomcast – with Crunch and Bio
Economic...

Transcript

Transcripts are machine generated and are NOT edited for grammar or spelling.

00:00:00:00 - 00:00:03:18
Brian "Ponch" Rivera 1
All right. Welcome to No Way Out. Dr. Klein, how are you today, sir?

00:00:04:07 - 00:00:08:11
Gary Klein, PhD
I'm fine. Thank you for inviting me for this conversation. I appreciate the time.

00:00:08:14 - 00:00:28:04
Brian "Ponch" Rivera 1
No, thank you. And in fact, I want to start off with a big thank you. I think I want to spend a minute here first to kind of paint a picture of who you are relative to saving lives in the military, if you don't mind. So I'm going to I'm going to start there. All right. Back in 2019, you and I met in Whistler, Canada, through Snowden.

00:00:28:20 - 00:00:59:03
Brian "Ponch" Rivera 1
I got to spend a few days with you and Dave and a few other brilliant folks. You and I sat down and we talked about recognition, prime decision making. Some overlap with the OODA loop as far as counterfactuals, mental simulations, things like that. We had a fantastic conversation. We also had dinner a few times. We talked about what it's like to lead in a complex environment and more importantly, how do you develop leadership skills or leaders to do that, to work in uncertainty.

00:00:59:15 - 00:01:21:17
Brian "Ponch" Rivera 1
During the conversation, you pointed out a few things that I did not know, and I want to make sure the world kind of knows this. Number one, you mentioned I think I think the context was I was talking about a recent program I was working on with the U.S. Navy following the 2017 mishaps at sea. And during that time, I got to engage with quite a few folks.

00:01:21:17 - 00:01:27:22
Brian "Ponch" Rivera 1
And one of them was Dr. David Woods. And do you remember that you have a connection with Dr. Woods? Correct.

00:01:28:11 - 00:01:30:05
Gary Klein, PhD
Yes. I've known him for a few decades.

00:01:30:08 - 00:01:54:17
Brian "Ponch" Rivera 1
Yeah, a few decades. So for those listeners out there who aren't familiar with Dr. David Woods, the the DevOps community development operations is built heavily on his resilience, engineering, thinking and strong connection to complexity, theory, human factors and safety. So that was fantastic to learn from you that you had a connection there. And then you start to ask me, not a strange question, but a question I just couldn't wrap my head around.

00:01:54:17 - 00:02:14:20
Brian "Ponch" Rivera 1
And that was Ponch. Do you know John Schmidt? I thought for a minute I'm like, Well, I know of a John Schmidt, but I don't know John Schmidt. And you kept talking and we kept talking. And I paused and I said, Wait a minute. Are you talking about John Schmidt, the author of M.S. Deep One War Fighting? And he said, Yeah, yeah, of course, that's that's John.

00:02:14:20 - 00:02:21:07
Brian "Ponch" Rivera 1
And would you mind sharing with our listeners how, you know, John Schmidt?

00:02:21:07 - 00:02:50:16
Gary Klein, PhD
All right. So, gosh, it would have been in about 20 years ago. John was was in the Marines. And he read an article ago that I wrote for the For Military Review about decision making, about my recommendation, prime decision model. And John really resonated through it. He said this is a model that captures what I think is actually happening.

00:02:51:01 - 00:03:21:16
Gary Klein, PhD
So John got excited and he called me and put me in touch with General Van Riper, who was then a two star, eventually became a three star, and the drone went further. He led an invasion of them as our next naturalistic decision making conference. We were invaded by the Marines. This was in Dayton, Ohio, and there was a contingent of seven Marines that we hadn't really been expecting.

00:03:22:04 - 00:04:00:03
Gary Klein, PhD
And it was wonderful. And they really changed the whole nature of the conference. John ran a tactical decision game. General Graham referred to the group of bad decision making. And so John and I have been friends and colleagues ever since. And John is now a member of my Shadow Box Company. He's his about half time, three quarters time working with us on a variety of projects involving military, law enforcement, firefighting, things like that.

00:04:00:10 - 00:04:28:05
Brian "Ponch" Rivera 1
It's just an amazing connection. And of course, you're connected to more folks like Dave Snowden. I know you've done work with him on Anthro simulation answer complexity. But at the at the heart of everything and I want to go back to this a little bit deeper. One of the reasons I'll argue that a lot of my brothers and sisters are still alive in fighter aviation and surface warfare is because a lot of work you did following a an awful, tragic tragedy near Iran.

00:04:28:06 - 00:04:42:19
Brian "Ponch" Rivera 1
Right. We had a U.S. sends I believe it was mistakenly shot down an Airbus. And the Navy has to come in to take a look at what is known or what was known as tactical decision making under stress. Can you walk us through a little bit behind that?

00:04:43:21 - 00:05:21:11
Gary Klein, PhD
Right. So we had the very first conference on naturalistic decision making in Dayton, Ohio, in 1989. And there was a contingent from the Navy, because this has happened right after they had been sent to shoot down, which I think was in 1988. And this should never have happened. And it sends was an Egis class cruiser with tremendously advanced how could it confuse an Airbus, a commercial airliner with an Iranian fighter plane?

00:05:22:01 - 00:06:05:04
Gary Klein, PhD
Yet it did and it shot it down. And the Navy was trying to understand how that happened and how and what they could do about it. And the Navy started a ten year program of research, quote, admits tactical decision making on distress to try to understand that. And we became part of that tag team. I had a chance to conduct a few interviews with Captain Will Rogers, who was the the captain on the descent, to try to get underneath his decision making process and see what happened, and then to try to develop some ideas for training, decision making and for designing systems to support decision making.

00:06:06:00 - 00:06:14:16
Gary Klein, PhD
So that was an unfortunate disaster, a terrible disaster that led to this program of research.

00:06:15:13 - 00:06:42:10
Brian "Ponch" Rivera 1
Yeah, thank you for that. The Naval Aviation picked up on this after the program went away, and we included a lot of lessons into what is now known as crew resource management and I believe you worked with Dr. Sallis on, on some of that. Eduardo Salas But you know, a lot of your research, from what I read, is connected to high risk environments, including aviation firefighting, first responders, cognitive task analysis, things like that.

00:06:42:10 - 00:07:06:23
Brian "Ponch" Rivera 1
And of course, our red teaming school that went away a few years ago, a year ago, borrows a lot from your work as well. But let's get down to the big thank you. Why does this matter? Your work has helped us learn how to make decisions in complex environments, right? We have programs that help us do that. And unlike a physical object such as an ejection seat, we can't put our fingers finger on that.

00:07:06:23 - 00:07:27:02
Brian "Ponch" Rivera 1
Say, that's why I'm alive today. Right. But the reason many of us are thriving in these complex environments, in these high, reliable, high reliability environments, is because the work that you and others around you that we mentioned have done. So I just wanted you to take a moment to thank you for all that awesome work that many people don't know about.

00:07:27:10 - 00:07:34:05
Brian "Ponch" Rivera 1
And I want to just say thanks on behalf of my brothers and sisters that are all alive because of you. So thank thanks very much.

00:07:35:07 - 00:07:50:20
Gary Klein, PhD
Thank you for all of the heroic work that you and your brothers and sisters and the military perform on a regular basis that never gets appropriately appreciated. So, Frank, thank you. Back to you.

00:07:50:20 - 00:08:10:23
Brian "Ponch" Rivera 1
Punch card. Thanks again. So there's there's one story I want to talk about before we dove into the cognitive dimension. And I think this story I read after I was an instructor in the F-14, and that is a story about an F-4 pilot who was transitioning to an A-6 and having a challenging time landing on an aircraft carrier.

00:08:11:03 - 00:08:12:11
Brian "Ponch" Rivera 1
Do you recall that story?

00:08:12:19 - 00:08:14:10
Gary Klein, PhD
Yes.

00:08:14:10 - 00:08:36:08
Brian "Ponch" Rivera 1
Yeah, it's so it reminded me of when we were transitioning pilots from the P-3 community or the helicopter community, we really had to discover a way to skimming. I had to figure out how to do this in a naturalistic setting. And that was how do we get them to see things from a different perspective so they can make a connection and ultimately come up with an answer, right?

00:08:36:08 - 00:09:05:14
Brian "Ponch" Rivera 1
Because having an instructor telling them that they're doing something wrong or how to do something right all the time doesn't always work. And I believe that story about the 4 to 8 six pilot was the landing signal officer had to sit down with him and just had the pilot use his thumb. I think he put his thumb out in front of him to figure out where the sight picture was and see the sight picture and say, hey, look, that's your old sight picture from the F for your new sight pictures a little bit to the right.

00:09:05:14 - 00:09:11:17
Brian "Ponch" Rivera 1
And he discovered that through that. I don't know if I did that story any justice, but you have anything else to add to that?

00:09:12:01 - 00:09:35:21
Gary Klein, PhD
Yeah. You captured the essence, the gist of it. But I like the story so much. So let me fill it out a little bit for the pilot was a friend of mine, Doug Harrington Jr and he was a very good naval pilot and became an instructor pilot and he was flying Air Force and then he would fly here for the training.

00:09:35:21 - 00:10:02:21
Gary Klein, PhD
He he would be flying with the student and but, but, you know, he would be sitting behind the student. And that's the way the configuration of the F-4 worked. And he did an outstanding job. And I always consider him to transition to an 86. And they figure you show him the layout, he understands the controls and it seemed like a piece of cake.

00:10:02:21 - 00:10:33:12
Gary Klein, PhD
So but he had to do six carrier landings in order to qualify. And I think he had to do another two at night. So he takes off and he thinks everything is going to be fine circles around. Now he's got to do the carrier landing. He's got his nose perfectly positioned, everything is fine. And the landing signal officer waves him off and they want him perfectly lined up.

00:10:33:12 - 00:11:01:01
Gary Klein, PhD
What's going on here is he is the hazing me. So he goes around again, lines up again that it perfectly positioned nose on the center line and then said no officer tells them come right, come right. That's what he had told them before. And Herrington knew that he was perfectly lined up. So he was ignoring that because he knows what, he knows how to fly and he gets waved off again.

00:11:01:21 - 00:11:27:19
Gary Klein, PhD
So he's got to go around and try again. And he figures this landing signal officer has more patience than I have fuel, so maybe I need to listen to him. So he comes right just a little bit and he will come right further. And eventually he's able to land. But it's a ragged landing. And in fact, none of his landings that day were were particularly good.

00:11:28:09 - 00:11:57:18
Gary Klein, PhD
And if he's told we're not going to actually do any landings tonight because he did so poorly on the landings during the day, we're going to repeat it tomorrow. If you don't do better tomorrow, that's it. That's the end of your flying career in the Navy. Doug is in shock and his friends come up to him and they they are for him really useful advice like Doug, you really got to bear down tomorrow or you know, Doug, you got to pay attention to it.

00:11:57:18 - 00:12:23:03
Gary Klein, PhD
We're telling him things that were of no value whatsoever and we're just driving his anxiety level even higher. So that night he's getting ready to go to sleep, hoping that he'll wake up from this nightmare and there's a knock on the door and Doug says, go away from anybody else. But it's the landing signal officer who's not in charge of training but really wanted to help Doug.

00:12:23:13 - 00:12:48:18
Gary Klein, PhD
So he eventually persuaded them to let him in and and and he said, you know, let's let's try to figure out what went wrong today. And Doug said, I had it perfectly lined up. And you kept telling me, come right in, waving me off and landing. Seven Officer said, okay, here, try your landing on on eight six. What was the last point you were with your mom?

00:12:48:19 - 00:13:15:15
Gary Klein, PhD
Five Well, we qualify. They said Doug said that for the landing signal officer thought a little bit and said now an eighth and therefore you're lined up in a line with the nose of the plane. Everything's in a straight line, Doug said. Yeah, and a six. You're side by side with a copilot, so you're not lined up with the central line of the plane.

00:13:16:01 - 00:13:39:04
Gary Klein, PhD
Doug said, Tell me what? What is it, a foot and a half? What kind of a difference is that going to make? And that's when the landing the signal officer said, let's try this. Hold out your son who's when I hold out your thumb. And that's going to be the nose of the airplane, you know, line it up where that door, you know.

00:13:40:01 - 00:14:07:17
Gary Klein, PhD
Okay, you got two lined up. Fine. Now move your head over a foot. And how do you have to pull your thumb over to to make it line up with the door and Doug said, Oh, my gosh, what an idiot I am. And hadn't thought about the parallax. I hadn't thought about the parallax effect. He hadn't realized what a scrum attracted wife.

00:14:08:07 - 00:14:45:12
Gary Klein, PhD
And once he had that in mind, that totally changed his mental model. And the next day he had no trouble qualifying. I love this story because when an external officer wasn't in charge of training him, he was also bright enough, sensitive enough, and alert enough to ask him what was he transitioning from so he could diagnose why Doug was having this problem and was also creative enough to come up with a simple exercise that Doug could use, that the learning officer didn't have to tell him anything.

00:14:45:12 - 00:14:53:23
Gary Klein, PhD
He could read Doug Discovery for himself them to discover and make more of a difference. So that's why I think it's such a powerful example.

00:14:54:15 - 00:15:21:21
Brian "Ponch" Rivera 1
Yeah. And I got to live that as an instructor. And when we think about, you know, winning in a complex or an environment or in combat or conflict or winning in business, it really comes down to people, ideas and things in that order. And in fact, it's the mind that will ultimately win. Right. And that's what we're talking about is the cognitive dimension with you today.

00:15:22:16 - 00:15:39:18
Brian "Ponch" Rivera 1
Basically, we're not machines, right? We are very different than ants and data processors and things like that. So we have a different way to look. We we need a different way to look at how humans make decisions. But you I don't know if you coined the term cognitive advanced dimension, but you use it in your new book, Snapshots of the Mind.

00:15:40:12 - 00:15:44:20
Brian "Ponch" Rivera 1
Can you tell me a little bit more about this dimension of the mind?

00:15:44:20 - 00:16:17:06
Gary Klein, PhD
Sure. So a lot of our training assumes that complex path is broken down into procedures that people just have to follow. Those procedures and procedures are important. I'm not criticizing procedure, but they're not enough. It doesn't the procedures don't capture what we've learned. Our expertize, what sometimes is called tacit knowledge, which means the kinds of patterns we can recognize, the kinds of perceptual discriminations we make.

00:16:17:11 - 00:16:45:08
Gary Klein, PhD
Our mental models here with Doug was an example of somebody who had a flawed mental model, and it's not boiling down to for him following procedure, he needed to adapt his previous mental model and replace it and replace parts of it with a much more effective one. So the cognitive dimension is largely invisible. It's not a set of steps to follow.

00:16:45:18 - 00:17:02:23
Gary Klein, PhD
It's what you have to know in order to follow those steps effectively, to know when to depart from those steps, when to modify those steps. And it's really at the basis of the expertize that people need to develop.

00:17:02:23 - 00:17:27:19
Brian "Ponch" Rivera 1
So you brought a mental model and there might be a connection here to these mental steps, these cognitive steps that people have to take. I've been it's it's my understanding that we can take lessons from high performing teams, like how do they work or high performing leaders or decision makers. And we just kind of borrow their their steps, you know, if they're thinking through something and we can adapt it for them.

00:17:28:05 - 00:17:36:15
Brian "Ponch" Rivera 1
Is that aligned to your the idea of a mental model, borrowing that from someplace else and bringing it over to another domain or another aspect?

00:17:37:07 - 00:18:11:20
Gary Klein, PhD
That certainly is a part of it. And I can give you an example of that. This is an example from about over a century ago, around 1900 British warships were firing cannons and the accuracy rate was pretty miserable. And one captain, Percy Scott, I think, was his right. He later became an admiral and he just didn't tolerate mediocre performance.

00:18:12:09 - 00:18:40:17
Gary Klein, PhD
So he was having his crew do target practice. And you're shaking. It's hardly how inaccurate they were. But he noticed one of the gun crews was doing much better than the other. So he went over and he watched them for a while to try to see what they were doing. That was different. And he noticed that the chief gunner, which I have to do, is you have to figure out how much charge to have.

00:18:40:17 - 00:19:03:00
Gary Klein, PhD
That depends on how far away the target is. But the ship is rolling. If you fire or it's too far down, it's going to hit the water too far up, it's going to go over. And so they would fix the elevation within hours of the ship rolled. You try to wait until the cannon was exactly at that right point.

00:19:03:00 - 00:19:27:09
Gary Klein, PhD
Then you would try to get the shot off. And that was just an instant. And, you know, if something went wrong, you would miss it. And this one crew was doing such a good job. You noticed that the chief gunner unconsciously was moving the elevation. He didn't leave it fixed. And so as the ship was rolling, he would start to raise the elevation.

00:19:27:17 - 00:19:52:10
Gary Klein, PhD
And as the ship was rolling back, you would lower. And that gave him a bigger window. So to pull the trigger rather than this instantaneous demand that they had and Scott said, I could teach that to the other gun crew and they call it continuing now. They call it continuous aim firing. And he started from the other gun crew.

00:19:52:17 - 00:20:14:10
Gary Klein, PhD
The accuracy increased tremendous rate. So that's an example of transferring a mental model to other teams. And this was an unconscious mental model was that Gunner didn't know what he was doing. He just had learned that this you know, he just had learned it without even thinking about it. That's an example of tacit knowledge.

00:20:15:18 - 00:20:24:22
Commercial 
You are listening to No Way Out, sponsored by adults. Now let's get back to building your confidence in complexity.

00:20:24:22 - 00:20:28:20
Brian "Ponch" Rivera 1
And sometimes experts don't know how to describe why they're experts. Right?

00:20:29:11 - 00:20:39:08
Gary Klein, PhD
Right, exactly. Yeah. But it happens more and more because what makes experts so good is that they've internalized all of these lessons.

00:20:39:17 - 00:20:39:20
Brian "Ponch" Rivera 1
Yeah.

00:20:40:01 - 00:20:57:00
Gary Klein, PhD
And they think everybody sees it. They don't even, they're not even aware of what they do and it's pass it knowledge, which means that it's very difficult for them to articulate to others what they're doing. Yeah, I'm just say watch me and you'll figure it out.

00:20:57:11 - 00:21:23:03
Brian "Ponch" Rivera 1
I think this is a very important point to emphasize for transitioning veterans who have gone through human factors training, have learned lessons from mis operational risk management, threaten our management. A lot of the work that I or psychologists have done for the military, but because it's built into our culture, we just don't think anything of it. When we go to an organization, we don't take the way we used to plan and identify the threats early.

00:21:23:03 - 00:21:48:09
Brian "Ponch" Rivera 1
We don't, you know, map out things in an organization. And I'm talking about veterans that transitioned over. But what's interesting is a lot of organizations now are trying to learn how do we make decisions in conditions of uncertainty? And, you know, again, a lot of lessons in my psychology are being transferred into that space to underneath team science and distributed leadership and teaching complexity theory.

00:21:48:18 - 00:22:07:03
Brian "Ponch" Rivera 1
But when we talk about like a basketball team or a a firefighting squad, I think that's really called can we take lessons from those domains and apply them to like software development in decision making? Is that possible?

00:22:07:21 - 00:22:43:19
Gary Klein, PhD
I think this is one of the things that Eduardo was done with his team's approach, where he's watched different kinds of teams, extracted important lessons that generalize from one domain to another. So I think there are general strategies that teams can use. But let me give you another example. John Schmitt had been presenting tactical decision games during the pandemic, the height of the pandemic, to a small group of young, young officers.

00:22:43:19 - 00:23:07:03
Gary Klein, PhD
A lot of things he noticed was they had been taught in the school solution and how to make decisions and they had all kinds of ways of calculating things. But in a tactical decision games, John wouldn't give them all the information that they wanted to have, and they would say, But how do I make a decision if I don't have all of these data?

00:23:07:12 - 00:23:34:17
Gary Klein, PhD
And John said, in a real world situation, you're not going to have all these data. You're going to have to make a decision when there's a fair amount of uncertainty. And it took them a while to adapt to that idea that it wasn't just about calculating, it was about speculative thinking and imagining it. And, and, and and being Brazilian and, and being able to improvise.

00:23:35:01 - 00:24:02:14
Gary Klein, PhD
But eventually they got there. So he was able to get them to that point. And that was a change in their mindset, where instead of thinking, my decision making is about following the steps that I learned in school to my decision making is going to boil down to my ability to size up situations and make reasonable assumptions. And I decide what might happen.

00:24:03:05 - 00:24:11:07
Brian "Ponch" Rivera 1
So on triggers these happen to be a great way for adult learning or providing experiential learning activities. Correct?

00:24:11:07 - 00:24:11:16
Gary Klein, PhD
Right.

00:24:12:04 - 00:24:27:03
Brian "Ponch" Rivera 1
Okay. So the way I understand we need to train leaders in decision making is we have to give them some type of experience. And going back to the A64 story, we have to let them see things for themselves and make decisions and make the connections to their their day to day work. Is that what tags kind of do?

00:24:27:20 - 00:25:00:20
Gary Klein, PhD
Right, with a good facilitator and George Schmidt is is one of them is one of the best facilitators that I've seen. I've also seen mediocre facilitators who know what the steps are and they're trying to follow the steps of the TG without thinking about who are the people that I'm that I'm working with and how can I give them a good mental workout and how can I read read their reactions to know where they're confused and to try to work with them.

00:25:01:22 - 00:25:20:11
Brian "Ponch" Rivera 1
So this actually brings up a great point right now and I don't know if I can share it moment, but when we were in Whistler, you share with me a card that hopefully I can share right now. I'm going to bring it up here so you and I can view it. And I believe you said it's in your wallet and it connects back to decision making games.

00:25:21:11 - 00:25:31:02
Brian "Ponch" Rivera 1
But here's a card. And when you're doing these tags, I believe you can use this card, which has it's called situational awareness calibration questions.

00:25:31:14 - 00:25:31:22
Gary Klein, PhD
Yeah.

00:25:32:11 - 00:25:43:09
Brian "Ponch" Rivera 1
And I'll read the first one. What is the immediate goal of your squad? And when we're doing a tag, sometimes we can hit pause and have people answer these questions. Is that the intent behind this card?

00:25:44:04 - 00:26:12:20
Gary Klein, PhD
Yeah, let me tell you how the card got. That card that developed it was developed by a colleague, a former colleague of mine, Dave Klinger. We were working he was working with nuclear power plants. And they were there were a very good nuclear power plant, but they weren't handling emergency drills very well. And the Nuclear Regulatory Commission would come by periodically putting through an emergency drill.

00:26:13:02 - 00:26:40:17
Gary Klein, PhD
And they weren't doing well. And they needed to improve. They needed to improve rapidly before the next visit from the Nuclear Regulatory Committee Commission. Otherwise, they were going to have all kinds of penalties. Additional training was going to cost them maybe $1,000,000 a year if they didn't shape up. So they brought Doug Harrington Jr and with with with Dave Krieger and our group and they diagnosed that one of the problems was there.

00:26:40:17 - 00:27:14:10
Gary Klein, PhD
It was a lack of common ground. So they've developed this card to help create a better common ground from the crisis manager and the team who would ask these questions and have each of the team members write out their answer during a drill that they had, and he would collect all their answers. And then at the end of your question, in one interrupted drill, at the end of the drill, he would show them how many different ways they answered the same question.

00:27:14:10 - 00:27:45:19
Gary Klein, PhD
So he didn't have to say, You guys aren't calibrated. That's sort of, you know, shaking his finger at them. They could see for themselves how poorly calibrated they were. So this was a card aimed at helping them figure out, especially the crisis manager figuring out figure out how to create better common ground. Now, the strange part of the story, this is just a training made about six months after they put on the training and worked with the nuclear and nuclear group.

00:27:46:00 - 00:28:13:00
Gary Klein, PhD
They had a small upset. It wasn't this big a three mile island, but it was a small little upset that they had to react to. And it took them about 12 hours to get it under control. Halfway through the upset, 6 hours into it, the crisis manager turned to his team and says, okay, time out. We're all under the gun here, but I don't want us to make any any dangerous mistakes.

00:28:13:09 - 00:28:35:05
Gary Klein, PhD
You all have that card, that laminated card that we got from Dave Krueger and his Associates group. Take out that card. I'm going to tell you. Think about how you're going to answer each of these questions on the card that you just showed punch. And we're not we don't have the time to write it out. Just think about how you answer it.

00:28:35:05 - 00:28:54:18
Gary Klein, PhD
I'm going to tell you the way I would answer them. And you could see if you agree and if you think I'm wrong, let me know. But he was essentially doing a come up, a calibration, a mid-course correction during an actual crisis. So we never designed it as an operational tool. That's how he used it.

00:28:54:23 - 00:29:12:10
Brian "Ponch" Rivera 1
Yeah. To me, it's a fantastic way to understand if we have a shared mental model. I think I'm using that term correctly of what we're trying to do as a group or organization and within a TTG or a training environment. It allows us to see it in this in the safety, in that environment. But we can also apply it in our day to day work.

00:29:12:17 - 00:29:29:00
Brian "Ponch" Rivera 1
And when we do use the card or a card like it or sense make to to help people see the differences in their perspectives, it's absolutely shocking to them. And again, it connects back to the basics to worry about. They have to see it for themselves. We can't tell them they're all seeing things differently. They have to see it.

00:29:29:16 - 00:29:45:20
Brian "Ponch" Rivera 1
But back to TD's and training right now, there's, you know, a lot of digital capabilities out there. We prefer what we call high touch, low tech approaches. Is that similar to what you're using in in training?

00:29:46:16 - 00:30:07:12
Gary Klein, PhD
Right, for the techniques that we tend to use most of the time now is a technique called shot. And shadowboxing is a way of having people see the world through the eyes of experts without the experts being there. It's a scenario based approach, and you go through a scenario and the scenario is designed in advance, can be challenging.

00:30:08:13 - 00:30:37:10
Gary Klein, PhD
And then at a certain point the scenario will have a decision point and it might be here for courses of action rank order, which one you're most likely to write to pursue and then write down your reason scenario continues. The other decision point Here are three goals rank order which one has the highest value to you and write down your reason and then you go back to the scenario.

00:30:37:10 - 00:31:05:08
Gary Klein, PhD
It continues. It might be another decision point. Here's five course five types of information. Which are you going to pursue as the most important to you and rank order them that way and write down what your reasons are. So you go through this usually about 4 to 6 decision points like that. Now we've also had a small group of experts, maybe three or four, not usually, not more than five, go through the same scenario.

00:31:05:13 - 00:31:32:00
Gary Klein, PhD
They've done their ranking, they've written down everything. So when I do my ranking, I can immediately see how do the experts rank them? And then I also have what I wrote down as my rationale. I get to see what their rationale of the experts were. I could see what influences they're making. I could see what concerned that. I can see what reactions they had.

00:31:32:08 - 00:31:56:15
Gary Klein, PhD
We're all looking at the same scenario, but I get to see the scenario through their eyes as well as through my own. And this could be done with paper and pencil. We have software versions of it and it's we've done a variety of studies showing that going through these exercises in just a half day, people align with the experts.

00:31:56:15 - 00:31:59:10
Gary Klein, PhD
It increases by about 25%.

00:32:00:17 - 00:32:09:17
Brian "Ponch" Rivera 1
Now that expertize is about the the subject matter expert are they subject matters in a particular field or is it about how we make decisions? I'm a little confused on that.

00:32:10:02 - 00:32:11:23
Gary Klein, PhD
It's the experts in North York.

00:32:12:07 - 00:32:17:16
Brian "Ponch" Rivera 1
Okay. Okay. All right. All right. So we like firefighters. I believe you're using it with firefighters as.

00:32:17:16 - 00:32:33:09
Gary Klein, PhD
A correct to use it with firefighters. Yes. And as a matter of fact, that's where it originated with a friend of mine, Neil Hynes from New York Fire Department. He was he was the one who would developed this approach in the aftermath of the 911 disaster.

00:32:33:19 - 00:32:50:23
Brian "Ponch" Rivera 1
Yeah. Yeah. Now so now there are two types of problems in the world that we just can't solve for, right? And at the same time, we still have to develop decision making capabilities in that space. And I think it's called a wicked problem. Is that correct?

00:32:51:06 - 00:32:51:16
Gary Klein, PhD
Right.

00:32:52:03 - 00:33:05:02
Brian "Ponch" Rivera 1
Right. You have some insights on what a wicked problem is and what we need to do to make better decisions. And by the way, a lot of the problems I believe that we're experiencing globally today are their wicked problems, correct?

00:33:05:02 - 00:33:35:14
Gary Klein, PhD
Yes, they are. Yes. And these are really challenging, extremely vexing. A wicked problem is one where there's no right answer. It's not something you can calculate the solution to because there's so many variables. They interact in so many different ways. The context keeps changing. There's different stakeholders with different motivations and different goals that have to be negotiated. So you can find here the right answer.

00:33:36:20 - 00:34:03:03
Gary Klein, PhD
So you see how pervasive wicked problems are, and in social problems that we deal with. So working with John Schmidt and with another colleague of ours, Sean Murphy, we got together and said, what can we say that's useful about having wicked problems? It's not going to be a recipe. There's not going to be a set of procedures. Here are the steps to follow to solve wicked problem.

00:34:03:03 - 00:34:35:14
Gary Klein, PhD
Because there's no solution. There's no solution that everybody would agree with or even one person would agree with from time to time, too. And we we did it using a naturalistic study we collected over 30 examples of people wrestling with wicked problems to try to see what did they do, what worked, what didn't work. And what were some of the mistaken ideas people had about about wicked problems?

00:34:36:01 - 00:35:09:23
Gary Klein, PhD
And then what can we do to help people show you handle wicked problems more effectively? We came up with a variety of tools that we are hoping to to pilot it very soon so we could. Problems are very different from the usual decision making training that people are given. They involve mental flexibility. They involve an ability to take other people's perspective.

00:35:09:23 - 00:35:37:17
Gary Klein, PhD
They involve an ability to shift our mental model about what's going on. And that gives them a tool. One of the tools we have is a clarity frame to show people your understanding of this wicked problem at the beginning. Here are you who you think are the stakeholders. Here are the goals that you've identified. Here are the kind of barriers that your that you're up against.

00:35:38:03 - 00:36:03:01
Gary Klein, PhD
Here's what you think are the big challenges in a situation of things like that. And then after they work and wrestle with the wicked problem for while they do another clarity frame and then after a while they do another one they get to see. Even though they're still wrestling, it's still hard. They look back at their original one and they think, How is ISO done?

00:36:03:20 - 00:36:08:18
Gary Klein, PhD
Yeah, look at it. Look at how shallow my understanding was then compared to what is now.

00:36:10:06 - 00:36:34:13
Brian "Ponch" Rivera 1
So on a wicked problem, we need to have multiple perspectives which requires the flow of information to move freely around a group of people. Correct? Is that right? Okay. The moment we shut that down, we lose. What do we lose? When? When don't get the best or when people can't be themselves and bring the best answer to work when they just feel I guess there's fear in the organization.

00:36:34:13 - 00:36:39:10
Brian "Ponch" Rivera 1
They don't feel comfortable in sharing their perspective. That's a problem when we're trying to solve wicked problems, is it not?

00:36:40:04 - 00:37:03:12
Gary Klein, PhD
Right. So one of the mistaken mindsets that people have of the wicked problem is I'm going to find the solution and then I'm going to reveal it to everybody and everybody will jump on the bandwagon. That doesn't happen. There's always going to be some people, maybe most of the people who are frustrated. There's a sense now, I don't want to have to make trade offs.

00:37:03:12 - 00:37:30:12
Gary Klein, PhD
I want to find a solution that everybody can resonate to. Well, too bad when you're dealing with multiple stakeholders, each with their own goals, you're going to have to make tradeoffs. You're another mindset problem that people have is I know how things are working out. And so I need other people to buy into my mindset. That's not going to get you very far.

00:37:30:20 - 00:37:52:22
Gary Klein, PhD
You need to do engage in perspective taking to see why are other people resisting what is. It's not just what they want but how are they viewing this situation where it's different from what I was expecting, what can I learn from it? And it's a very difficult for a lot of people, including a lot of leaders, to take other people's perspective.

00:37:53:23 - 00:38:24:15
Brian "Ponch" Rivera 1
So this is interesting because when I was in active duty military, going from the tactical, operational, strategic level warfare, those lessons that we had at the tactical level are very powerful. But as years passed and now I'm at the operational or strategic level, I don't have that tactical knowledge anymore. I'm dangerous, right. So as a leader, what we try to coach leaders is you want to have those people that had that tactical knowledge of what's going on, the subject matter experts be part of the planning process or be part of this decision making process.

00:38:25:16 - 00:38:30:00
Brian "Ponch" Rivera 1
Are we often in that thinking or can you comment on that? Is that how do you like to that view?

00:38:30:23 - 00:39:14:16
Gary Klein, PhD
Right. So I think you're spot on, except that you're not emphasizing how difficult that shift is, but you've been able to make that many people, many people are anchored in at the tactical level because that's where they were so successful. That's why they got promoted. And so they're they're still trying to adjust the thinking of the people who are working underneath them, rather than shifting to a leadership role of managing precious resources and valuable resources and getting the most out of other people.

00:39:14:22 - 00:39:42:14
Gary Klein, PhD
And in the baseball quote that I remember, somebody a manager said, My job is to manager is to get paid for other people's home run. And that's right. That's the shift that's hard for a lot of players who become managers and aren't really thinking that they've got to get the most out of their players rather than making it all about themselves and reflecting on themselves.

00:39:43:06 - 00:39:52:14
Brian "Ponch" Rivera 1
Can we help train people and leaders with tags and chatterbox? Does that help develop that type of I'm going to use the word mindset.

00:39:52:14 - 00:40:17:21
Gary Klein, PhD
I think we can. I think it's possible there are some people who are never going to make that mindset transition. So I don't want to claim that we can get everybody there, but I think everybody can be moved in a better direction if they get the right kinds of t gene scenario based shadow box experiences, it can at least move them in the right direction.

00:40:18:03 - 00:40:26:12
Gary Klein, PhD
And for some people the light will go on and they'll know they'll be able to make the transition. You describe to the operational and strategic level.

00:40:27:21 - 00:40:57:18
Brian "Ponch" Rivera 1
Yeah, I believe so. So we we have a lot of ideas. There are a lot of things we can talk about. We can go in many, many directions. You know, a lot of people use your pre mortem techniques. I just want to hit on that for a moment in the training in a training scenario. So when we're trying to help people or teams develop a mental model of how to think through complex problems, maybe even wicked problems, leverage, cognitive diversity, we want to mitigate I hate to use the word bias as neurotics.

00:40:57:18 - 00:41:18:09
Brian "Ponch" Rivera 1
I know we don't want to go down that path, but we're trying to leverage the cognitive diversity of the group the best we can. We also want to see from their perspective on why we're going to fail potentially, or why this project may fail. Can you just comment on your pre mortem technique in injecting inside of a TG or experience monitoring activity?

00:41:19:03 - 00:41:43:19
Gary Klein, PhD
Sure. So I developed the pre mortem technique over 30 years ago. I never developed the technique. It was just something that we get inside my company and most of our projects went well, but occasionally one of them would not and we would do a mortem to see where we would. We'd go wrong. And I thought, We're learning from that.

00:41:43:19 - 00:42:02:09
Gary Klein, PhD
Too bad we didn't have this at the very beginning of the project. Then I wondered, why don't we do it at the beginning of the project? So now we do it at the beginning of projects as we're getting started now, many times when there is a kick off meeting or preparation for a new project, people will do a critique.

00:42:03:20 - 00:42:30:01
Gary Klein, PhD
They'll say, Here's the plan. Does anybody see any problem? And they'll think that that covers them, that that gives them the gives everybody the opportunity to speak out. But the brutal fact is, hardly anybody raises any problem for a couple of reasons. One, there's a lot of enthusiasm for the plan, so you don't want to spoil that kind of harmony.

00:42:31:01 - 00:43:01:05
Gary Klein, PhD
Another is you're not thinking about what can go wrong. You're ready to get started. And and this is slowing you down. And so there's just a variety of of social harmony issues. And focus issues. People don't give you the answers that they need. So the pre-modern technique is a way of changing people's mindsets. And the way it works is, say, here's the plan.

00:43:01:05 - 00:43:36:10
Gary Klein, PhD
We will be briefed on the plan. Now, if I'm leading the session, I'll say I'm looking in a crystal ball. And now the crystal ball shows clearly this plan has failed, it's fallen apart, is now six months later, maybe a year later, whatever is the right time frame. And this plan has been a fiasco. And when we look past each other in the hall, we avoid eye contact because it's just been such a painful disaster.

00:43:37:07 - 00:44:12:11
Gary Klein, PhD
So we know the plan is fail, right? Everybody around the table, take 2 minutes, each of you individually 2 minutes and write down all the reasons why this plan has failed. You know, it's failed. We don't know why you're going to write down all the reasons you can think up and people start writing like crazy. I only give them 2 minutes because I want to keep the energy level up and then when 2 minutes is up, I say, okay, hands up the time to see what we've learned.

00:44:12:11 - 00:44:33:08
Gary Klein, PhD
And I go around the room, what, you know, what's at the top of your list? And I'm right there on her whiteboard. What's the top of your address that hasn't been covered? We'll write that and we'll just compile what the people in here and the group have come up with. And it's amazing the kinds of things that people articulate that they would never say.

00:44:33:08 - 00:44:59:04
Gary Klein, PhD
If you're just ask them, anybody see any problems? Nobody would because of pressure, because you've just been working together to build this plan. So there's a pressure to encourage you believe that the plan is going to work and you don't want any naysayers with the pre mortem the way you show your smart is by coming up with really insightful reasons why this plan fail.

00:44:59:09 - 00:45:21:11
Gary Klein, PhD
Now, once we've always done it inside the company, but once had a big project, an Air Force project, and I said, Let's do it with a larger team. We had our Air Force sponsor, we had other contractors, and we had like a three day kickoff meeting. And I said, let's do a pre mortem at the end because I think it'll help.

00:45:21:19 - 00:45:48:19
Gary Klein, PhD
And my Air Force sponsor, who is to ten colonel said absolutely no way. That's yeah I described how the method worked and he said that is such a downer. That is such that's going to drain all the enthusiasm. No way. And I said, trust me on this. And reluctantly we went ahead and did the pre mortem and were going around the room.

00:45:48:19 - 00:46:15:18
Gary Klein, PhD
We were building a system that was going to be for targeting and it was going to be put out in the field. This was right after Desert Storm and it was going to be fielded and we thought we had the technology you needed. We just had to put it over into the computers they had in field. And we come to this young Captain Jensen word for three days, but it was his turn.

00:46:16:22 - 00:46:58:12
Gary Klein, PhD
And he looked and he said, top of my list is the kind of algorithm you're talking about runs on a supercomputer, and it takes about 48 hours to get an answer. They don't have that kind of equipment in the field. They have a very limited laptop. They don't. The technique we're talking about isn't going to work. And it was quiet in the room as everybody realized he was exactly right and somebody else up and said, I have a back of the envelope technique that I use as a shortcut that generally gets me pretty close.

00:46:59:05 - 00:47:29:16
Gary Klein, PhD
And we said, okay, maybe we're not dead yet. And it became a very effective project that was still going on ten years after we finished. That's how a lot works. It gets people to voice their their genuine concerns. They're not it's not like an exercise where somebody is told, you know, your job is to come up with criticisms and nobody knows they're legitimate or not.

00:47:31:04 - 00:47:34:03
Gary Klein, PhD
These are genuine concerns people would have. That's how it.

00:47:34:03 - 00:47:54:00
Brian "Ponch" Rivera 1
Works. Yeah. So there's an aspect of this that I love and that's using the pre mortem to break the artificial harmony that's in an organization or a team and to increase those insights. Right. And that's that's why we use it quite effectively. But like you pointed out, not all leaders and team members want to do it because it's you know, they didn't grow up with this.

00:47:54:00 - 00:48:15:12
Brian "Ponch" Rivera 1
It's not familiar to them. But there's another aspect of this type of thinking and that when it comes to improving performance, I believe it helps reduce errors. Right. And you write about that years ago and even in your new book, Improving Performance depends on reducing errors, but it also depends on increasing insights. That's what we're trying to do with the pre mortem is increase insights.

00:48:16:11 - 00:48:19:19
Brian "Ponch" Rivera 1
Can the same be true about decreasing errors through a pre mortem?

00:48:21:07 - 00:48:44:19
Gary Klein, PhD
Right. So so one of the issues that I have is about how to improve performance. And to think that I'm one way of looking at is there's two things you want to do. If something you want to reduce, you want to reduce errors or reduce mistakes, there's another thing you want to do. You want to increase insights and expertize because you don't want to just reduce mistakes.

00:48:44:19 - 00:49:08:11
Gary Klein, PhD
You don't want to go home at the end of the day and say, I had a great day. I didn't make any mistakes. Well, if that's your goal, why did you even go to work? You want to accomplish things. But most organizations only know about the reducing errors. They have all kinds of techniques for reducing errors. You not have any, hardly any do any work on increasing insights.

00:49:08:11 - 00:49:38:19
Gary Klein, PhD
And this is I know this was true in some work we did with the intelligence community. A pre mortem, I think can do both. A pre mortem is helping people flag potential mistakes before you make it because at the end of a pre mortem, we look at the list of possible problems that might have arisen and we say, how can we avoid or prevent some of these one, what changes can we make in the plan?

00:49:39:00 - 00:50:05:22
Gary Klein, PhD
What changes can each of us make? So I think it's a way of cutting down the number of mistakes that you make by improving the plan. It also increases the chance for insights. Because I may write down my my problems with the pre mortem and I think this can be obvious to everybody. But as we go around the room, I'm hearing things from my colleagues that I had never even considered.

00:50:05:22 - 00:50:38:13
Gary Klein, PhD
So we're learning from each other. There's a collective wisdom that's being grown in the team, and we're also increasing the sophistication of our mental model as we listen to the sorts of things that people have identified. And it changes the culture of the team, right? Of course, now people have gone through this exercise where they have surfaced in doubts and public, which is a very difficult and most teams.

00:50:38:13 - 00:50:43:06
Gary Klein, PhD
So there's now a different culture of candor than existed before.

00:50:43:15 - 00:51:05:22
Brian "Ponch" Rivera 1
Now it's absolutely fantastic as a coach to see teams adopt this and take it on as their own. And really, you know, the more coaching capability we develop in an organization and we see that return when we see the work that, you know, like a pre mortem being shared with us down the road somewhere else. So we do know that it's been picked up quite a bit within the organizations that we coach.

00:51:06:01 - 00:51:23:11
Brian "Ponch" Rivera 1
I want to shift over to your new book, Snap Snapshots of the Mind, and there's something in there that my kids just absolutely loved. And that was the tower of Hanoi, the classic puzzle there. To be honest, I probably enjoyed it more than they did, but it was super fun. Can you walk us through that?

00:51:23:23 - 00:51:54:13
Gary Klein, PhD
Sure. So I wrote this book as a collection of essays that I've been preparing on a blog that I have essays going back almost a decade, and I have many essays, and I picked the ones I thought would be the most exciting for readers. And then I collected them, and then I wrote an introduction to the book. And then for each of the sections I wrote an introduction and reflections at the end of the section.

00:51:54:13 - 00:52:34:19
Gary Klein, PhD
So basically the book is, is my attempt to get people to adopt a sensitivity to the cognitive dimension so that they get out of a procedural mindset to a more sophisticated mindset about the expertize, the tacit knowledge that often goes unnoticed, especially from from the experts themselves. So there's nine different sections that involve training and they involve techniques were collecting information for building insights, things like that.

00:52:34:20 - 00:52:59:07
Gary Klein, PhD
Now the Tower of Hanoi essay which was fun to write the tower I no I don't know if you're the people tuning in to to this podcast are familiar with it if they're not so much the better, then you need to buy my new book and turn to that chapter. But it's a simple problem. It's been around for about a century.

00:52:59:07 - 00:53:30:04
Gary Klein, PhD
You've got three rods and you have a number of discs that fit over one of them and descending order of side and you've got to move them from the first rod to the third one. And the only rule is you can have a larger one go over a smaller one. So it's kind of a complex task. And as the more your greater number of you have greater number of discs, the harder it gets.

00:53:30:04 - 00:53:56:13
Gary Klein, PhD
This has been around for a century. Psychologists have studied it for at least half a century. So this is exactly the kind of toy problem I never wanted to study in my career. I wanted to study people like, you can't. I want to study firefighters. I want to study emergency departments, physicians. Those are the people that that I'm excited about, not about for you laboratory problem.

00:53:57:02 - 00:54:34:12
Gary Klein, PhD
But nevertheless, we did a study. We did it. I'm ashamed to tell you that because of greed, we had a project and we were a subcontractor and there was money left over and we hadn't whether we had finished our tasks and we they still were supposed to give us some money, about 25,000. And they said under contract monitor and said, okay, well I'd like you to do is cognitive interviews around the Tower of Hanoi problem and I feel like there's no in the world to do that.

00:54:35:10 - 00:54:58:14
Gary Klein, PhD
But I also didn't want to wave goodbye to 25,000. So I well, we had just made a new hire, a guy, a very sweet guy had emailed me and I said, Andy, you're going to do this project, you're going to do those interviews. This is how you're going to learn to do cognitive interviews. And I had tried to foist this project on all kinds of people in my company.

00:54:58:20 - 00:55:18:01
Gary Klein, PhD
They all turned it down. But Andy was so new, he didn't know that history. So he came out of my office feeling great. He thought, Wow, this, I'm going to get to do this project. And he told it a couple of our colleagues about it, and they shook their heads. And so Gary found this pigeon, of course he's.

00:55:18:01 - 00:55:18:20
Brian "Ponch" Rivera 1
Going to do it. Yeah.

00:55:19:17 - 00:55:57:01
Gary Klein, PhD
And there's only one way to do a tower and not a problem. And you're not going to learn anything. Well, he interviewed seven people. No, two people did it the same way. And we wound up learning things about the tower and the problem that no one had ever come up with before. And we were able to make discovery, even with something as simple as this laboratory project that demonstrates the power of cognitive interviews that you're able to make, these discoveries that are domains that people thought were solved a long time ago.

00:55:57:01 - 00:55:57:06
Gary Klein, PhD
Yeah.

00:55:57:16 - 00:56:22:12
Brian "Ponch" Rivera 1
Well, there's many reasons. Buy your book, including that your new book, you do talk about heuristics and biases, which we'll leave for the readers to go our listeners to go out and read that in the book you talk about fixation errors, confirmation bias a little bit. Maybe we can get on that. But one thing that's missing, and I didn't see it was a Six Sigma.

00:56:22:12 - 00:56:48:02
Gary Klein, PhD
So I have enough enemies in the world that I don't. You can make noise all the time. I'm not a fan of Six Sigma. What I like about Six Sigma. I have this this diagram about improving performance and there's a down arrow, what you want to reduce, which is errors. I mentioned that just a few minutes ago, and that's Arrow, which is increasing insights and increasing expertize.

00:56:49:00 - 00:57:21:23
Gary Klein, PhD
And Six Sigma is all about the down arrow. And that turns your creative, vital organization into one that's obsessed with avoiding errors and doesn't really care about what the consequences are, even if it's squelching insights and discoveries and new products. I once had Chief Innovation officer of a company tell me that his company had bought in on Six Sigma and techniques like that, and they had been very innovative at the beginning.

00:57:21:23 - 00:57:51:02
Gary Klein, PhD
That's how they got to be so successful. But since they've adopted that technique in a decade or two, since they've had like 50 proposals come out about new kinds of ventures they could engage in, and they turned down every single one that had changed from a creative company into one that was frightened of any possibility of making a mistake.

00:57:52:17 - 00:58:11:08
Brian "Ponch" Rivera 1
Yep. And we see Lean Six Sigma in a lot of organizations and we do know it has bounded applicability and using data. Snowden can have a framework more on the ordered side or clear domain. So there's that aspect of it. I want to kind of wrap it up. I know we're coming close to our time together today, but anything else you want to cover?

00:58:11:08 - 00:58:22:14
Brian "Ponch" Rivera 1
Anything you want to ask about, anything around our ecosystem. Your ecosystem want to make more enemies, anything like that, no.

00:58:23:01 - 00:58:53:13
Gary Klein, PhD
I don't need more enemies, but I do. I the reason I wrote the book and I do podcasts like this and I am trying to get people sensitized to the cognitive dimension and, and to recognize it when they encounter it in their, in their everyday life and to appreciate the expertize around them, to appreciate the insights that they come up with.

00:58:53:19 - 00:59:23:15
Gary Klein, PhD
Often we don't realize the insights that we have. When we make a mistake, we take ourselves. Why was life so stupid? But when we have an insight, we don't. We usually fail to appreciate it and savor it. I also want to invite listeners and viewers are to plug in to the naturalistic decision making association, which was set up about a year ago.

00:59:24:01 - 00:59:56:11
Gary Klein, PhD
And it puts on our webinars, we put on another conference on naturalistic decision making back in October. John Schmidt ran that and that's, I think the 16th that we've had since 1989. So we put them on every two or three years. And so anybody who wants to get more attuned to the cognitive, those are some of the things you can take or you can follow on follow me on on LinkedIn or Twitter.

00:59:57:06 - 01:00:03:10
Brian "Ponch" Rivera 1
Fantastic. I do have one last question. Michael Lewis, as he commented on anything about your new book?

01:00:05:08 - 01:00:22:22
Gary Klein, PhD
I have. He has. I haven't seen it. I did a podcast with Michael Lewis a few years ago, and I've I've met him. We had lunch together a few times, but I haven't seen any any comments on my new book. But that's a good idea. Maybe I should reach out to him.

01:00:23:00 - 01:00:41:08
Brian "Ponch" Rivera 1
That's another reason why our listeners need to go read your book to understand the context of, that question Right. So, yes, Gary Klein, it's been a pleasure. It's been an honor. It's been just a fantastic conversation. I wish we could have four more hours of your time and then another week after that. But I do know that we're coming up on our time today.

01:00:42:00 - 01:00:46:11
Brian "Ponch" Rivera 1
Thank you very much for being on. No Way Out and we'll connect with you soon.

01:00:46:11 - 01:00:48:10
Gary Klein, PhD
I hope so. Take your punch.