Pfsense & Squid Proxy

[DaBeach]

Currently I am running pfSense with the Squid proxy system setup. If I understand this proxy system correct it caches every web page and or data such as pictures, files, etc on to the router system cache so that if another LAN user requests the same data and it had not changed on the website since it was cached the LAN user would receive the cache data saving bandwidth on the WAN.

My question is where exactly are these files stored. I can find the Squid cache directories (and there are many) however I cannot recognize the files. Would I not find HTML, .ZIP, .JPG etc files in the cache if I looked into the directory.

What am I missing?

[Infiltrator]

There are 3 basic directories that squid uses for storing information.

The first is called \var\squid\cache (here the cached websites are stored)

and the second \var\squid\logs (here squid logs, all transactions like, cache, store and access)

And the most important one \var\squid\etc\ (in this directory squid.config main configuration file is stored)

Now another important thing to consider is that, squid only caches static web pages, meaning that dynamic web sites will be very difficult to cache.

Edited June 27, 2010 by Infiltrator

[DaBeach]

There are 3 basic directories that squid uses for storing information.

The first is called \var\squid\cache (here the cached websites are stored)

and the second \var\squid\logs (here squid logs, all transactions like, cache, store and access)

And the most important one \var\squid\etc\ (in this directory squid.config main configuration file is stored)

Now another important thing to consider is that, squid only caches static web pages, meaning that dynamic web sites will be very difficult to cache.

Ok.

I checked the cache directory and found many sub directories from 00 through 09, I checked a few say 00 and found again many sub directories 00 through ff, then the 00 finding 2.94kb document as filename 00000000. Would I not find in these directories the actual files downloaded? If say most websites visited are dynamic sites I would expect not to find html but what about jpg or zip or exe etc...?

Additionally I was browsing the logs and did like the ability to view the access.log which showed what seems to be all http with links to pics downloaded etc including access to dynamic page addresses. However I am disappointed that I would have to export this out into say a database to analyze what IP did what. I had tried a squid add on package that was supposed to view the log and format it up however, it is not currently setup to work with pfSense snapshots.

Would there be another way to say access the router box with putty and grab out the access.log file by command line and have another program open and format it at the click of an icon?

I am not sure if say Wireshark could do this I have not yet tried.

And by the way thanks for your help and input on my posts.

[Infiltrator]

Additionally I was browsing the logs and did like the ability to view the access.log which showed what seems to be all http with links to pics downloaded etc including access to dynamic page addresses. However I am disappointed that I would have to export this out into say a database to analyze what IP did what. I had tried a squid add on package that was supposed to view the log and format it up however, it is not currently setup to work with pfSense snapshots.

There several freeware tools for analyzing squid logs.

http://www.squid-cache.org/Scripts/

I personally use a very simple batch script that I created to search for accessed URLs based on ip address or username, though it may not represent the information in a nice layout, it works well for my needs.

Here is my batch script.

access.bat

Find /I "192.168.1.11" access.log > accessed_URL.txt

As you can see I can replace the ip address within the quotes for the username that I want to generate a report for and then run the batch file, it may take sometime for the accessed_url text file to be generated, depending on how large the log file is, but in the end you have the info you need.

Edited June 27, 2010 by Infiltrator

[DaBeach]

There several freeware tools for analyzing squid logs.

http://www.squid-cache.org/Scripts/

I personally use a very simple batch script that I created to search for accessed URLs based on ip address or username, though it may not represent the information in a nice layout, it works well for my needs.

Here is my batch script.

access.bat

Find /I "192.168.1.11" access.log > accessed_URL.txt

As you can see I can replace the ip address within the quotes for the username that I want to generate a report for and then run the batch file, it may take sometime for the accessed_url text file to be generated, depending on how large the log file is, but in the end you have the info you need.

Ok, I like this.

Here is a modifed windows batch file that will process "access.log" in the same directory that the batch is run in but uses a menu to select either a particular ip or all ip's into seperate files.

@ECHO OFF

:start

CLS

ECHO 1. 172.16.10.102

ECHO 2. 172.16.10.104

ECHO 3. 172.16.10.162

ECHO 4. PARSE ALL OF THE ABOVE

ECHO.

ECHO 5. EXIT

ECHO.

set choice=

set /p choice=

if not '%choice%'=='' set choice=%choice:~0,1%

if '%choice%'=='1' goto 102

if '%choice%'=='2' goto 104

if '%choice%'=='3' goto 162

if '%choice%'=='4' goto ALL

if '%choice%'=='5' goto EXIT

ECHO.

ECHO "%choice%" INVALID Choice, Please try again

ECHO.

ECHO.

ECHO.

GOTO START

:102

ECHO PARSING 102

Find /I "172.16.10.102" access.log > ACCESS_LOG_102.txt

start notepad.exe ACCESS_LOG_102.txt

GOTO EXIT

:104

ECHO PARSING 104

Find /I "172.16.10.104" access.log > ACCESS_LOG_104.txt

start notepad.exe ACCESS_LOG_104.txt

GOTO EXIT

:162

ECHO PARSING 162

Find /I "172.16.10.162" access.log > ACCESS_LOG_162.txt

start notepad.exe ACCESS_LOG_162.txt

GOTO EXIT

:ALL

ECHO PARSING ALL

Find /I "172.16.10.102" access.log > ACCESS_LOG_102.txt

Find /I "172.16.10.104" access.log > ACCESS_LOG_104.txt

Find /I "172.16.10.162" access.log > ACCESS_LOG_162.txt

start notepad.exe ACCESS_LOG_102.txt

start notepad.exe ACCESS_LOG_104.txt

start notepad.exe ACCESS_LOG_162.txt

CLS

:EXIT

Now, is there a way to run a windows batch file that would access the routers squid access.log and either parse it and place the data on the windows machine or copy the router file to the windows machine to process it? With pfSense I would have to go into the router and via the WWW GUI run the edit file option and rummage through the folder until I found the access file, I would then have to copy and past and save it to the windows machine. I would think there would be an automated way to do this. Any suggestions?

Edited June 27, 2010 by DaBeach

[DaBeach]

You know, I think I will try to figure a way to have the windows machine grab the file and import it into MS Access as I would be able to already have reports setup for each IP in questions and I could format it to my hearts content. Do you know if the access.log file is saved as a tab delimited text file?

[Infiltrator]

You know, I think I will try to figure a way to have the windows machine grab the file and import it into MS Access as I would be able to already have reports setup for each IP in questions and I could format it to my hearts content. Do you know if the access.log file is saved as a tab delimited text file?

I haven't actually imported this file into any MS database, so I am not 100% sure whether it is tab delimited or not.

Edited June 27, 2010 by Infiltrator