GREYBEARDroot since before you were born
field notes from the machine city
← Root & Echo
No. 009    Operations · a deploy button and the little undo arrow beside it

The Rollback

THE ROLLBACK

A release is on fire and the room wants to push forward through it, fix by fix, pride intact. Echo is already typing the hotfix. Root reaches past her to the button that goes back — because the brave road is the long one, and the floor is still down.

Going back is not defeat. It is the shortest road to where it last worked.

A scene Root & Echo

The deploy had been live for nine minutes and the graphs had been red for seven.

The room had that smell. Six people around one desk, every one of them typing or about to. Echo had the hotfix half-written — she could see it, the one line that was wrong, she was sure. "Give me ten minutes. I know what it is. I can fix forward."

Root reached past her shoulder and put one finger on a different button — the one marked with the little curved arrow, the one nobody in the room had looked at since the alarms started. "Roll it back."

"I almost have it—"

"Roll it back." Not loud. He never got loud. "You can have it in ten minutes or you can have it in ten seconds, and only one of those is the number the floor is counting."

"Rolling back is admitting the deploy was bad."

"The deploy was bad. The graphs are red. The admitting is done — the only question left is how long you make four hundred people wait while you protect the feeling of almost having it." He was already watching the graphs, not her. "Push the arrow. We'll find your line after. Calm, with coffee, the floor green behind us instead of bleeding in front."

She pushed the arrow. The graphs climbed back to green all at once, a little embarrassed.

"Now find your line," Root said into the quiet. "You were probably right. You'll fix it tomorrow and ship it Thursday and no one will remember tonight. That's the whole art — make it boring again fast, then be clever slowly."

"It felt like losing."

"Going back always does. It's the shortest road to a place that worked, and it feels like the long one because your pride has to walk it." He picked up his coffee, gone cold during the nine minutes, and drank it anyway. "Catch it early, return from not far. The ones who fix forward into a fire are the ones writing the postmortem at four a.m. — about the second fire they started putting out the first."


What's happening here

The oldest discipline in operations: when a deploy is bad, the fastest safe path is almost always to roll back to the last good state, not to 'fix forward' under pressure. Fixing forward feels braver and protects ego, but it ships untested code into a live incident — the classic way one outage becomes two. Mean-time-to-recovery beats mean-time-to-be-right.

Imagined — a composite from enterprise IT, not a single real incident.


The reading if you follow the I-Ching
䷗ → ䷁  Return → The Receptive
line 1 changing
Return from not far. No need for remorse. Supreme good fortune. You haven't strayed far; the return is easy. Catching the error early enables the best outcome.

The figure. ䷗ Return (Fù), line 1 changing → ䷁ The Receptive (Kūn). "Return from not far. No need for remorse. Supreme good fortune." You haven't strayed far; catching the error early makes the way back easy and the outcome best. Return is the turning point — the bottom of the year, where the light comes back — not a defeat.

The reading. The brave road into a fire is the long one. Roll back, get the floor green, then be clever slowly. Return from not far.

A story from @greybeard_root — field notes from the ruins of the machine city.
Department: Operations ()
Next story: The Cosmic Ray