Once I was doing what I do best and exploring a product and I decided to see what happens when I have a lot of concurrent users running on the system. So I started performing this test in a staging (pre-production) environment. Then as I was doing this test I started to notice the screen in the room that displayed the status of the servers in production all going red one by one signalling that they were going off line disrupting customer service. Everyone panicked saying the production system was down and they were not sure why. I started to feel in my gut that my activities might have caused that and went slightly red in the face. The engineering director looked at me and said "what are you doing right now?". I was open and explained what I was doing. They said "You do know the staging and production databases are linked right?". I honestly had no idea!! I felt so embarrassed! Luckily the system recovered very quickly, but a lesson learnt to always understand the architecture before you go testing.
Do you have any similar stories to tell? Let me know in the comments
Do you have any similar stories to tell? Let me know in the comments
Melissa Fisher
Test Engineer
She/Her
Ministry of Testing Ambassador for 2026 - Feel free to get in touch!
Simon Tomes
🙈 Oof, I can only imagine how awkward that was.
Yet I'm hopeful your lesson also leads to folks considering how staging and production environments are set up. As you say, at the very least, for us to find out how they might be connected.
Jesper Ottosen
years ago I was part in shipping a backend system update to 80% of the country's internet users, who then couldn't go online. Same story as above, it was first picked up by customer service. and the luckily just a DB fix. but still...
Dan Alloun
You really made me smile ... even if indeed one can feel very bad when that happens.
Understanding the system environment is indeed important and often forgotten or misled. Thank you for sharing!
Sign in
to comment
Manage your entire QA lifecycle in one place. Sync Jira, automate scripts, and use AI to accelerate your testing.
Explore MoT
Boost your career in software testing with the MoT Software Testing Essentials Certificate. Learn essential skills, from basic testing techniques to advanced risk analysis, crafted by industry experts.
Debrief the week in Quality via a community radio show hosted by Simon Tomes and members of the community
