A Morbid Sense of Pride
Once you’ve decided to adopt a new view of safety, and want to help an organization change, how do you know if you are changing anything? How do you know if you are successful? Especially considering the total organization overhaul you hope to achieve. It can be useful to identify some markers somewhere between nothing, and total success. What are some markers that demonstrate we are on our way?
A recent incident, which involved a system I am partially accountable/responsible for, left me with a morbid sense of pride. After the initial response and immediate aftermath. I took a step back and looked at how I and my organization responded. While I would not say my organization is operating with a new view mentality, the incident was a high water mark in terms of progress. It’s easy to find faults in any system. So, most importantly, seeing the markers of progress is the most important outcome.
Important context, this incident was bad enough that it was triaged to a high level of severity. With high levels of severity come high levels of scrutiny. While it can be easier to try new things when the pressure is lower, arguably if you can’t use new tools in your most high-pressure situations they may not be valuable.
That being said. I was proud of this incident for a few reasons:
- We were supported and surrounded by people taking practical, responsible, and novel action.
- We are creating demand for learning reviews and generating valuable insights in the process.
- I was able to stay calm.
In incident response and investigation these are not things we can take for granted. In some cases we built them, but in some cases we just got lucky. In all cases their presence is progress.
Practical, Responsible, and Novel
If your goal is to change the cultural defaults and adopt a new view of incidents and safety, high-severity events can be a major test. All things being equal, under those circumstances, your organization will revert to the mean. It will do whatever the cultural defaults are. If you have a well-worn process, it will do that. If you don’t have a process people will still probably have questions. People’s internal sense of justice will want answers. If your goal is to try something new, some effort will have to be made to create a different outcome.
Therefore I was surprised that there was a demand for a novel approach and this came from my leadership. I’ve been pushing for new learning-based reviews. After this incident happened, they suggested we try this new thing. (Queue panic!) Sure…, but, umm…, like… what… if we… fail? (Implicitly if we try this once and it doesn’t work we will never be allowed to do this again). To their credit, the response was so what, lets’s start learning. This is a meaty incident, let’s learn from it. Which was just the right push I needed, to dive in.
Within an hour or two of starting my investigation, I found a number of fascinating insights that I knew folks would be delighted by. Excited by this small win, I was curious about my panic. I realized that my panic came from my unstated goals. I want to help build a robust organization. There is no straight line, in some cases, I can’t even see the next step. So, in some ways, any step is the only step. IE we do one thing and the next days, the change is complete. Which is clearly absurd. Whatever robust looks like in the future will be very different from where we are. But, healthy change doesn’t happen overnight. My implicit goals made it difficult to see small practical steps we can take forward.
This was another marker of progress. Letting an individual, take some time to learn more about your organization through an incident investigation is all by itself valuable.
Another example of progress was our responsible and practical approach. While we did want to try something new, we also couldn’t throw out the old approach to incidents. So, we found ways to run both in parallel. We contributed our insights to the official incident process, had the regular conversations, and worked on responsible remediation items. This side-by-side process felt okay. It’s probably ridiculous to completely change the traditional process all at once. This slow integration from old to new felt like the most responsible and practical approach. Again, a huge marker of progress.
Demand for Learning
It’s easy to forget that we don’t do learning reviews to change the world, we do them to learn! The goal isn’t overnight organizational change, the goal is to stoke a hunger for insight. We want “standing room only” incident reviews. We want people to go holy shit I didn’t know that! I can’t believe that! If people find value in the learning review itself we can build momentum around it. And maybe over time, it might lead to organizational changes where appropriate.
While we create demand for insight, we are also finding ways to put those insights into action. After the incident was over, we dug into what happened and figured out how this incident happened to evade every protection we had already put in place. We are now able to quickly incorporate this new learning into our system and make things more robust. The ability of the team to do this was built on the fact that for the last year or so we’ve had a number of people who could focus on it consistently. That focus means we didn’t have to context switch from one project back into this project. On top of that, it means that we are really getting into the meat of the work. We are entering the territory of the novel. We are starting to learn things we didn’t know that we needed to know about.
The Right Attitude
If you’ve had the opportunity to be involved with incident response, you might be familiar with the feeling of panic. I panic. I’m a panicker. My default narrative is that failure is bad and we are all going to die. This kind of narrative is not helpful in the middle of an incident. Given that it’s a limbic response, there is a physical reaction. It can cause tunnel vision (blood is pulled to your vital organs and away from your eyes shrinking your field of vision) and it prevents fine motor movements (because the blood is protecting your vital organs it’s not in the small muscles near your fingers which you probably need to type out commands on the command line). Both reactions reduce your ability to respond. In situations where my life isn’t immanently in danger, these reactions are heightened because of my overriding narrative (failure is only bad). Fortunately, there is a better way.
Many folks I know just white knuckle their way through this feeling. We soldier on, strap on our armor, etc. We deal with it. In an ideal world though, our narrative would be a failure is interesting it is a signal from the system and now I get to investigate it! Which has the effect of allowing our brain the best chance at being open to signals from anywhere, and we have increased flexibility. Which in the middle of a novel incident can be valuable.
Unlike organizational issues, there was work I had to do to switch my narrative from we are all going to die to hmm, this is interesting… Over a number of years, I learned more about the limits of control, a lot about systems, and how psychological safety impacts teams. Lately, I became a water polo coach where I live and breathe this stuff. Little kids have the same fears of failure that adults do, they are just way more honest about it, which allows us to work through it openly.
After years of trying to switch my narrative, it’s starting to take hold. I still have lots of distance to go but in the spirit of taking the win. This was a great week!
All of this has left me with a morbid sense of pride in my team, my job, and myself. Yes, something terrible happened, but that bad thing will be used to good effect. Maybe it will allow fewer bad things to happen in the future!