It is all relative to the type of internet you have. If you have cable and neighbors are moving in...etc then yeah, you might experience slow down because you all share the allocated bandwidth per section. If you're using DSL and it's getting slower...it could be to line degradation..weather issues even.
It may even be your router, it could be the time of day...your speeds are almost negligibly different anyway.
If you start seeing some major drops or issues, easiest thing to do is to call your ISP.