Stuff you can do to make internet REALLY go faster.

[cooper]

I'm sure we've all seen (and been annoyed by) the ads that spit out garbage like "Make your internet faster" which for the most part is a pipe dream anyways. So the thing I'm looking into right now is ways in which you really CAN improve the speed of your internet connection.

I'm aiming for the smart tricks to pull that improve efficiency rather than hacking the firmware of the modem your ISP provides. If you want to discuss that, please make a separate topic.

Given the above, the total amount of data that can pass through your internet connection has a certain limit. What tricks can you pull to improve the responsiveness of websites?

Run your own caching DNS server

Ever go to a website and notice in the status bar it says it's resolving images.slow-ass.server.org and it takes AGES before the browser gets past that? That's time lost while trying to resolve a DNS name. If you run a caching DNS server locally it will send your DNS query on to your ISP's server when it's a not yet cached domain name, sending you back the response when it gets it. If however you asked for that domain name previously, it will send the cached copy instead and do so considerably faster.

For this to work you need to tell your OS of choice to use the local, caching DNS server. Only that DNS server needs to know the IP address of your ISP's DNS servers. On Linux you can use Bind for this but I'm sure there are many others out there, including some that run on Windows.

Run an ad blocker

I'm pretty sure most of you people are already doing this. But if you don't it's really something to consider. Those ads take time to download, and tend to be taken from rather overload servers. It really sucks when you want to browse a website and you can't see the article you want yet because some stupid ad didn't finish downloading yet. There are various tools that can help you with this. In Mozilla/Firefox there are plugins that will take care of this quite nicely. Alternatively you can run a program called Privoxy, which is a proxy server that not only blocks ads, but completely filters them out of the HTML it sends on. It can also sanitize the javascript on a site prior to passing it onto your browser, further protecting your machine. Unfortunately Privoxy needs the full webpage before it can sanitize it, so it tends to be rather slow. Probably a better idea would be to...

Run a caching proxy server

Normally your browser is supposed to cache all those images you see on webpages so that it can display them instantly. My personal experience with this though is that most browsers SUCK ASS at this. Maybe it's because they constantly bombard the server with requests to see if the file in question has been changed or what not. A caching proxy server such as Squid will work MIRACLES here. Be sure to turn the browser cache off as it will have essentially become useless and will only be taking up space.

Note that ad blocking SHOULD be a task performed by a proxy server, and there are helper programs out there for squid that will make this a breeze, and be pretty much transparant to the user.

Traffic shaping and advanced traffic control

This is really cool stuff. The theory is that you're doing a TON of stuff using the internet. Chatting, gaming, email, bittorrent, browsing, what have you. Some of those things are highly interactive and require quick response, such as chatting, gaming and browsing. Other things aren't that interactive, and if it takes a few extra seconds before something happens there, it's not a big deal. Bittorrent comes to mind here, and, to a lesser extent, email.

The theory is that you want to prioritise certain types of traffic. There's only so much data that can travel through your connection, and you don't want to wait for a webpage because you're bittorrenting the new Hak.5 DVD. So, using some advanced routing rules you can tell your system to prioritise, say, http traffic over bittorrent. This should be smooth and dynamic to the end user. Regular web traffic just goes on as if nothing happened, squeezing out bittorrent a bit as it's transmitting and receiving stuff, and once it's done the total amount of bandwidth jumps back to your bittorrent.

In the case of cable modems and particularly DSL lines, it gets even better. ISPs added enormous buffers to the routers they give you because having big buffers (queues, really) reduces TCP/IP packet loss and thus improves throughput. Unfortunately that means that if you send something, whilst some heavy bittorrent action is going on, it may take a while for your packets to enter the top of the queue before they get sent. You presumably gained throughput at the cost of responsiveness. Using traffic shaping you can feed data to your DSL router at pretty much the exact same rate as it's able to transmit it, nullifying that queue and improving responsiveness of the connection.

While very cool, this is a very advanced solution, and so far I've only seen it possible with Linux machines. Here's the howto that gives you all the gory details: http://www.tldp.org/HOWTO/Adv-Routing-HOWT...ntro.linux.html

Any other cool techniques out there that people use?

[stingwray]

Good little article.

They do all make a big improvement as well, and if you have some time, a spare box running 247 then it can all be done for free as well. What more could you ask for?

[cooper]

Another thing that might be worth having is a local mail server. This however requires you to either have your own domain (or one you got using DynDNS or whatever. It's what the DNS MX record is for) or to have an ISP that allows auto-forwarding of your email.

Most people who fetch their email using either POP3 or IMAP automatically check for new mail every 5 minutes or so. More if they feel important, less if they don't really care. BUT that checking for new email will yield no new messages 90% of the time for all but the most popular people: You're once again wasting bandwidth!

By running your own mail server the external mail server will forward all email intended for you directly to your mail server. Using email post-processors like procmail, typically used to filter out spam and such, you can play a sound whenever an email arrives (and since you've got the full email at your disposal within procmail, you can play different tunes for emails from different people). At that point you can go to or start your mail client, fetch the mail (and since the server is local, you could leave the client running and check every minute without hurting your total bandwidth) and basically take it from there.

An additional benefit would be that you don't need to have your memory hogging mail client up and running 24/7, freeing up those precious resources that can be put to a better use.

The only problematic thing here is that you'll need to configure your mailserver for this. Depending on your mailserver this may be a daunting task *cough*sendmail*cough*. Just make sure it's set to only _receive_ email. It's normally best to just use your ISP's mailserver for your outbound email (there was some talk a while back about AOL delaying or outright denying your mails simply because the sending server isn't the mail server of an ISP they know). And the last thing you want is your mailserver (not to mention your precious bandwidth) to be used as a spam relay.

I'm still thinking if it would be interesting to have a kind of man-in-the-middle caching proxy that allows the caching of data from HTTPS websites. The question really becomes if you trust this type of proxy with your data...