A Typhoon That Fizzled Out: How a PM Survives the Darkest Hour Like Riding Out a Storm

Let me start with something that might rub people the wrong way: the vast majority of “darkest hours” you’ll live through will end exactly like this typhoon — a false alarm. And those three words, “false alarm,” can kill a product manager faster than the typhoon itself.

This year’s Typhoon No. 9, “Bawei,” veered south last night. Instead of slamming straight into Zhoushan, it weakened to typhoon grade and headed for the coast from Wenling in Zhejiang down to Xiapu in Fujian. So here in Zhoushan and Putuo, the ferries that shut down on July 9, the cancelled flights, the boatloads of fishing vessels moved overnight from their anchorage to the west pier at Shenjiamen — in hindsight, it all looks like wasted effort. This morning someone in the group chat was already saying: “If I’d known, I wouldn’t have bothered sailing the boat out in the dead of night.”

Stop on that sentence. Because the part of this whole thing most worth chewing on is hidden right inside that “if I’d known, I wouldn’t have bothered.”

In the last piece I wrote about Putuoshan, my point was: don’t go pray to Guanyin to block the typhoon for you — go fix the seawall. Don’t credit “not getting hit” to some mysterious protection; that’s a misattribution. This piece takes one step further: the seawall is built, the alarm has actually sounded — so on that night, what exactly should you, the person running the show, do?

My answer: riding out a storm was never one move; it’s three moves in a row — before the typhoon arrives, prepare fully; in the thick of it, take the wind and rain; after it leaves, clean up the trash. These three things — even if nine times out of ten it turns out to be a false alarm — you can’t skip a single one. That’s how a product manager survives the darkest hour.

1. Before the typhoon arrives: your composure was all stockpiled long ago

The most deceptive thing in a darkest hour is “composure.”

You’ll see people whose systems crash in the middle of the night, whose data goes wrong, whose boss is calling — and they can still sit still and untangle the mess one thread at a time. You assume it’s innate temperament, improvised on the spot. It isn’t. Not one gram of composure in a darkest hour grows on the spot; every bit of it was stockpiled, little by little, on all those “days without a typhoon.”

For someone who builds products and systems, what you “stockpile” is very concrete: a rollback switch you can hit at any moment; a canary rollout that lets you push 1% of traffic out to test first, instead of shoving 100% of your users into the water all at once; a ring of monitoring and alerts so you know where you’re bleeding before your users start cursing; a contingency plan that’s written down and actually rehearsed; and the trust and cash-flow buffer you built up bit by bit while you didn’t yet need it.

The key phrase here is “actually rehearsed.”

On January 31, 2017, GitLab had an incident that people still bring up to this day. Late at night, an engineer ran a delete command on the wrong database and wiped the production data. Dropping the database wasn’t the end of the world in itself — what really sent chills down everyone’s spine was what came next: they had five backup and replication mechanisms, and when they went down the list, not a single one actually worked. The automated pg_dump backup had never actually run successfully because of a configuration error; a failed backup was supposed to trigger an email alert, but those alert emails were silently rejected by the DMARC settings, so no one ever got them; there were Azure disk snapshots, but restoring from them would take over 18 hours. What finally pulled GitLab back from the edge of the cliff was a snapshot one engineer had happened to take by hand six hours before the incident. It was on that one “happened to” that they lost only six hours of data.

This incident laid out, in bloody detail, what “prepare fully” really means: you thought you’d built a seawall, and only when the typhoon came did you find out it was painted on. Five backups — sounds impregnable, yet not one had been seriously verified, seriously rehearsed. An un-rehearsed contingency plan isn’t a plan; it’s a wish. That line you wrote in the doc — “we have a comprehensive disaster-recovery solution” — is, before the typhoon comes, no different from a blessing.

An even harder-nosed approach is to not wait for the typhoon at all. Netflix keeps a program called Chaos Monkey, and its daily job is to reach into the production environment, pick a calm sunny afternoon, and randomly kill a few machines that are actively serving traffic. It sounds like self-harm, but it’s actually the most honest kind of rehearsal — rather than wait for a real typhoon to reveal that the seawall is painted on, you turn a mad monkey loose in your server room every single day, crash a not-yet-broken system on purpose, and see whether it can actually patch itself back up. The composure that survives Chaos Monkey is real composure; the composure that’s never been put through this is just luck that hasn’t run out yet.

So the first layer of “false alarm” shows up right here: those rollbacks, rehearsals, and backups you stockpiled — the vast majority of them, you’ll never use even once in your life. Never using them makes you feel it was all for nothing. But flip it around: those peaceful days you had weren’t good fortune — they were bought with preparation. Someone who “never wastes effort on preparation” simply hasn’t had his turn yet.

2. In the thick of the typhoon: you have to stand out front in the rain

No matter how thorough your preparation, the moment the typhoon actually hits you in the face, it still hurts. What’s being tested now is something else: taking ownership.

In the thick of a crisis, there are two things the person running the show absolutely must not do — run, and pass the blame.

Running is hiding yourself away and waiting for it to blow over; passing the blame is rushing to find “who pressed that button.” Both moves make you feel a little better in the moment, but they simultaneously kill the most expensive thing there is: whether your team still dares to charge forward. Taking ownership is doing the opposite — stepping out to the very front and making the decision that’s hardest right now but has to be made immediately. Stop the bleeding first: roll back if you should roll back, take it offline if you should take it offline, cut it if you should cut it, admit the mistake if you should admit it, notify users instead of dragging it to tomorrow. The tab — you own it first.

Back to GitLab that day. They did something that’s still counterintuitive even now: they ran the database recovery on a public livestream. Thousands of people watched online as they laid out the embarrassment of the deleted database, the fact that all five layers of backup were useless, and fixed it back piece by piece. This wasn’t a show. This is the hardest form of “taking ownership” — I screwed up, I’ll fix it in front of everyone, I won’t hide. When a company dares to do this, the engineers on the team come to know: when something breaks here, the first reaction is to fix it, not to hide it.

Contrast that with the kind of place that calls a blame meeting the moment something breaks. An incident hits, and the first thing everyone does is figure out how to clear themselves — quick, back up the logs, screenshot the chat history, “that module wasn’t mine” out of the mouth. The opposite of taking ownership isn’t panic; it’s deflection. Someone who pushes responsibility down onto the team and out onto “bad luck” — after the storm passes, all anyone remembers is the sight of his back as he shrank away.

This part is the hardest of the three, because the rain really is cold. Three in the morning, the situation still a mess, every pair of eyes fixed on you — that’s not a good feeling. But what your team is watching in that moment was never whether you panic, whether you have the standard answer — what they’re watching is whether, when the fire is at its worst, you took a step forward or a step back. What the darkest hour is really weighing is the direction of that step.

3. After the typhoon leaves: the mess on the ground is the real exam

Once the typhoon passes and the sky clears, the most dangerous slackening arrives. Plenty of people weather the wind and rain, only to die in the end on “can’t be bothered to clean up.”

The wreckage strewn across the ground after the storm is where you really show your worth. Cleaning up isn’t just tidying the scene and getting the system running again — that’s only recovery. Real cleaning up is turning this round of pain into a rule you won’t step on again next time: the postmortem has to reach down to the mechanism, turning this lesson into a check that auto-triggers next time, a hidden hazard deleted, a contingency plan updated.

Knight Capital died precisely on not cleaning up. On the morning of August 1, 2012, this high-frequency-trading firm deployed a new chunk of code, with an engineer manually pushing it to 8 servers — and missing 1. And on that very server there happened to be an old chunk of long-abandoned functionality no one had ever deleted, codenamed Power Peg. The new code reused a flag with the same name, so on that one un-updated machine, this dead code that should have been lying in its grave woke up and started frantically firing orders into the market. In 45 minutes it traded nearly 397 million shares across 154 stocks — a pre-tax loss of $440 million. The company was acquired by the end of that year and was gone.

Look how short that chain is: a chunk of dead code that should have been deleted but no one bothered to, plus one deployment with no verification that missed a single machine, took out an entire company. That’s the extreme of “not cleaning up” — trash left in the system doesn’t vanish on its own; it just sits there quietly, waiting for the spark that lights it.

Now look back at GitLab. Same kind of potentially fatal incident, and afterward they wrote a postmortem so public it was almost self-flagellating, laying out how every single one of the five backup layers failed, one line at a time, then fixing each one. One turned its trash into immunity; the other left the trash in place and waited for it to blow up. What each company looks like today, you already know.

Cleaning up also has one spot that’s especially easy to get backwards: sweep the mechanism, not the person. The point of a postmortem is to make “this pit gets filled automatically next time,” not to dig out the poor soul who pressed the wrong button. The moment you start digging out people, the first thing everyone learns is to hide next time something breaks; only when you fix the mechanism and never ask who did it will people dare to shout the problem out the instant it happens. This “blame the issue, not the person” postmortem wasn’t first invented by the internet — it grew out of aviation: when a plane goes down, the investigation board’s entire purpose is to keep planes of the same model from falling out of the sky again, not to nail the pilot to the pillar of shame — because the moment you start nailing people, the next crew to make a mistake will choose to conceal it, and concealment makes the next plane crash worse. Google later wrote this into its SRE handbook and gave it a name: the blameless postmortem. The GitLab engineer who dropped the database wasn’t publicly executed, and that wasn’t softheartedness — it was clarity. Hurting once isn’t growth; turning the hurt into a rule you won’t break again is.

4. Back to “if I’d known, I wouldn’t have moved the boats overnight”

Now back to the opening, back to this morning’s “if I’d known, I wouldn’t have bothered sailing the boat out in the dead of night.”

That sentence is actually a mirror image of the misattribution in the last piece. Last time, people credited “safety” to Guanyin’s protection; this time, people credit “safety” to “nothing was going to happen anyway.” One records the credit to the goddess, the other records it to luck — and both are quietly cancelling out your preparation for next time.

But the truth is: the vast majority of “false alarms” are exactly what successful preparation looks like. If an organization has never once been through a “wasted effort” in all these years, there are only two possibilities — either it’s lying to itself, reading every lucky escape as “our system is fine”; or it has already been genuinely capsized once and paid the tuition in blood. Knight Capital never had a “false alarm,” because it read every incident-free deploy before go-live as “the process is flawless,” reading it right up until those 45 minutes.

Typhoon “Bawei” turned south this time, and Zhoushan will most likely get through fine. But what lets those fishermen sleep tonight isn’t the typhoon’s last-minute change of heart — it’s the boats that sailed out of the anchorage overnight.

Whether the next darkest hour slams down head-on isn’t yours to decide. All you can decide is those three things the people who ride out storms never leave out: before the typhoon, build the seawall so it can really block the waves, not paint it on paper; in the thick of it, stand at the very front of the line and take that rain; after it leaves, sweep the mess off the ground and sweep it into a rule you won’t break again. Do all three, and people will say you got lucky; slack off on all three, and Knight Capital’s 45 minutes will, sooner or later, come around to you.