I have the same problem, the master servers for most games generally show very few servers, and show high latency for them, but if I connect to one of them, the latency is fine.
So what I have to do is use game-monitor.com or a similar service for my servers, and then favorite the ones that I like and connect directly to them. And I think in my case is an ISP issue since it's been happening no matter how much I tweak my configurations, and it's been going on for several PCs / routers.