In October I was cleaning up a reliability dashboard and found that 9.6% of all production requests came from the platform's own synthetic monitors. On one endpoint it was 62%.
Nobody had set out to load-test production. The monitors were there to tell the team whether the product worked. Somewhere along the way they had become a large part of what the product was doing.
The setup looked modest. Nine browser synthetic tests, each running every 15 minutes, each walking through a real page the way a user would.
A browser test does not load one URL. It loads the whole page, the same as a person would. On these pages that meant about 330 calls per run to one endpoint that hands out image URLs, because the pages showed a lot of images.
330 calls, times nine tests, every 15 minutes, all day and all night. Real users come and go. The monitors never stop. Over a full day they came out at almost a tenth of everything.
The bigger problem was not the volume. It was what the volume did to the view of performance.
The image endpoint showed a median latency of 108 ms. That looked slow for what it did, and it had been flagged as a performance problem. I had looked at it earlier and agreed it was slow, and a code change went in to make it faster.
When I split the traffic by source, real users were getting a median of 9 ms. The 108 ms belonged to the synthetic runs, and they made up most of the samples. The median was describing the monitors, not the people.
So there was a ticket, a code change and a diagnosis, mine included, all aimed at making the monitors happier. I had to go back and correct what I had said about that endpoint. That was not fun, but it was better than letting the next person build on it.
I paused all nine browser tests on production. The endpoint's p99 went from 185 ms to 63 ms. A big part of that was simply no longer mixing two very different kinds of traffic in the same number.
I also moved the reliability dashboard from synthetic-backed SLOs to request-based ones that exclude synthetic traffic, so the headline numbers describe what customers get.
This was the second strike against them in a few weeks. Earlier, during a real production outage, the synthetic checks barely moved. The homepage check still read 99.97% for that week. The outage was real for users, but it barely showed up in the checks that were supposed to catch it.
So the monitors were both too loud and too quiet. They added a tenth of the traffic and distorted one endpoint's latency, and they did not see an outage that customers felt.
I am not against synthetic monitoring. It answers one question well: can someone, from outside, complete this flow right now. That is useful at 3 a.m. when there is no real traffic to measure. It is also useful as a smoke test after a deploy.
What synthetics should not do is stand in for users in your metrics. They run a fixed path on a schedule. Real users take paths you did not script, at times you did not pick, from networks you do not control. If you blend the two, the blend describes neither.
Tag synthetic traffic at the source. Most synthetic tools can add a header or a user agent string. Make sure your load balancer logs, APM and request metrics can filter on it.
Exclude synthetic traffic from SLOs and latency reports by default. Keep a separate view for the monitors themselves.
Count what a browser test really costs. One page load can mean hundreds of backend calls. Multiply by tests, regions and frequency before you turn them on, and compare that with your real traffic, especially at night.
Point heavy browser tests at a staging environment when you can, and keep a small number of light checks on production.
And when an endpoint looks slow, split its traffic by source before anyone writes code. I would have saved a ticket and a correction if I had done that first.
If your dashboards and your customers disagree about how the product is doing, I can help you find out which one is right. Get in touch.
