Production 404 - Taking down Production by Accident

13 Feb 2026

Once I was doing what I do best and exploring a product and I decided to see what happens when I have a lot of concurrent users running on the system. So I started performing this test in a staging (pre-production) environment. Then as I was doing this test I started to notice the screen in the room that displayed the status of the servers in production all going red one by one signalling that they were going off line disrupting customer service. Everyone panicked saying the production system was down and they were not sure why. I started to feel in my gut that my activities might have caused that and went slightly red in the face. The engineering director looked at me and said "what are you doing right now?". I was open and explained what I was doing. They said "You do know the staging and production databases are linked right?". I honestly had no idea!! I felt so embarrassed! Luckily the system recovered very quickly, but a lesson learnt to always understand the architecture before you go testing.

Do you have any similar stories to tell? Let me know in the comments
Melissa Fisher profile image
Melissa Fisher
Quality and Test Leader
She/Her
Open To
Work
Teach
Speak
Mentor
Podcasting
Review Conference Proposals
Write
CV Reviews
Attending MoTaCon 🤝
Chapter Lead
Ambassador
Simon Tomes
🙈 Oof, I can only imagine how awkward that was. Yet I'm hopeful your lesson also leads to folks considering how staging and production environments are set up. As you say, at the very least, for us to find out how they might be connected.

Jesper Ottosen
years ago I was part in shipping a backend system update to 80% of the country's internet users, who then couldn't go online. Same story as above, it was first picked up by customer service. and the luckily just a DB fix. but still...

Dan Alloun
You really made me smile ... even if indeed one can feel very bad when that happens. Understanding the system environment is indeed important and often forgotten or misled. Thank you for sharing!

Sign in to comment
Explore MoT
QFireside Chat: The AI Eval Mindset image
Moving from Vibe Checks to Automated Quality Gates
MoT Software Testing Essentials Certificate image
Boost your career in software testing with the MoT Software Testing Essentials Certificate. Learn essential skills, from basic testing techniques to advanced risk analysis, crafted by industry experts.
Into The Motaverse image
Into the MoTaverse is a podcast by Ministry of Testing, hosted by Rosie Sherry, exploring the people, insights, and systems shaping quality in modern software teams.
Subscribe to our newsletter