Production 404 - Taking down Production by Accident
13 Feb 2026
Once I was doing what I do best and exploring a product and I decided to see what happens when I have a lot of concurrent users running on the system. So I started performing this test in a staging (pre-production) environment. Then as I was doing this test I started to notice the screen in the room that displayed the status of the servers in production all going red one by one signalling that they were going off line disrupting customer service. Everyone panicked saying the production system was down and they were not sure why. I started to feel in my gut that my activities might have caused that and went slightly red in the face. The engineering director looked at me and said "what are you doing right now?". I was open and explained what I was doing. They said "You do know the staging and production databases are linked right?". I honestly had no idea!! I felt so embarrassed! Luckily the system recovered very quickly, but a lesson learnt to always understand the architecture before you go testing.
Do you have any similar stories to tell? Let me know in the comments
Do you have any similar stories to tell? Let me know in the comments
Melissa Fisher
Senior Test Engineer
She/Her
Hi, I'm Melissa, a software testing professional I enjoy all topics on software testing. I have a special interest in brain science and linking up how people think or feel about products they use. Feel free to get in contact.
Simon Tomes
🙈 Oof, I can only imagine how awkward that was.
Yet I'm hopeful your lesson also leads to folks considering how staging and production environments are set up. As you say, at the very least, for us to find out how they might be connected.
Jesper Ottosen
years ago I was part in shipping a backend system update to 80% of the country's internet users, who then couldn't go online. Same story as above, it was first picked up by customer service. and the luckily just a DB fix. but still...
Dan Alloun
You really made me smile ... even if indeed one can feel very bad when that happens.
Understanding the system environment is indeed important and often forgotten or misled. Thank you for sharing!
Sign in
to comment
Jira Issue Connect brings live, real-time Jira data directly into TestRail, eliminating tab-switching and stale fields.
Explore MoT
Thu, 1 Oct
Previously known as TestBash, MoTaCon is the new name for our annual conference. It's where quality people gather.
Boost your career in software testing with the MoT Software Testing Essentials Certificate. Learn essential skills, from basic testing techniques to advanced risk analysis, crafted by industry experts.
Into the MoTaverse is a podcast by Ministry of Testing, hosted by Rosie Sherry, exploring the people, insights, and systems shaping quality in modern software teams.