How We Accidentally Declared War on Our Own Database
It all started with a simple task: "Hey, let’s check if two databases have the same data over time!" Sounds harmless, right? Oh, how young and foolish we were.
Our team did some research (aka Googling) and found a potential solution. That’s when our senior teammate our so-called MongoDB expert stepped in. We called him that because he had slightly more experience than the rest of us.
Anyway, he found The Command a magical piece of code that would generate the hash for the collection
db.runCommand({ dbHash: 1, collections: [] }).md5
He was thrilled! Excited! Practically bouncing in his chair! His master plan? Run this command on both databases on each collection, compare the hashed, and if they didn’t match boom! Data inconsistency detected!
First, he tested it on a small database. It worked. Beautiful. Perfect. No issues at all. So, naturally, he did what any overconfident engineer would do: "Let’s try this on the live database with 200 million records!" This, my friends, was the moment it all went horribly, terribly wrong.
The Moment of Doom
At first, everything seemed fine. He ran the command. He waited. And waited. And waited.
Then he got impatient and thought, Huh, maybe I should stop this command? So he tried to kill it.
Plot twist: The command refused to die.
Seconds later, all our database connections spiked to 2,900%. Every single query stopped working. The app crashed. Users were probably screaming at their screens.
At this point, chaos erupted. Alarms were ringing (in our heads). People were yelling. Our team was frantically Googling like hackers in a bad Hollywood movie.
But no matter what we tried, nothing worked. But to reduce the connection in mongodb we were stoped some not crucial service and we try to keep the connection as low as possible
Our one innocent looking command had completely paralyzed the entire database. It was like throwing one small rock into a river and somehow causing a tsunami.
The Horrifying Truth
Then, finally, our MongoDB expert (who, at this point, had aged 10 years in stress) decided to read the official documentation. And there, hidden like a villain’s evil plot, was this small but deadly sentence:
The dbHash command locks the database and doesn’t allow any changes until it finishes.
Translation: "HAHA, YOU CAN’T DO ANYTHING UNTIL I’M DONE. GOOD LUCK, LOSERS!"
At this point, we had two choices:
- Sit there and hope for a miracle.
- Find a way to forcefully wake up our poor, frozen database.
Luckily, our DevOps teammate had an idea. "Let’s scale up the database. That’ll force a restart and kill the stuck command!"
We nodded furiously and watched as he logged into MongoDB Atlas, cranked up the database size like turning up the heat on a broken stove, and hit the magic button.
The Resurrection
For the next five minutes, we sat there in complete silence, watching the screen, praying to the database gods. Then BAM! The database restarted! The stuck command was gone! The app was alive again! We cheered. We cried. We swore to never, ever, EVER run random commands on a live database again.
Lesson Learned:
Always read the instructions before trying something new. And maybe just maybe don’t test experimental ideas on a live system unless you enjoy heart attacks.
