[pfSense] Regarding Web Filtering

Adam Thompson athompso at athompso.net
Sat Feb 16 23:26:37 EST 2013

Squid in transparent mode is typically detected by sites like that through inspection of headers.  Squid behaves like a “good” proxy server by inserting things like X-Forwarded-By: headers into the HTTP request.

UTM devices (e.g. FortiNet’s FortiGate line) deliberately act like they’re in “stealth” mode, and avoid leaking that information to the outside world.  They only proxy the content long enough to determine whether or not it violates the current security policy; once a PASS action has been decided upon (e.g., the URL is not blocked, no viruses are detected, etc.) they revert to a cut-through mode.  This is more sophisticated than Squid currently supports, because they are trying to do something quite different… but they are using essentially the same technique of intercepting requests on port 80 (or intercepting everything that “looks like” HTTP, which is quite evil IMHO and breaks many protocols) without letting the client know.

Yes, there are ways to detect that you’re behind a firewall like that, but they are non-trivial.  You can make squid behave (mostly) like that, too, if you really want to.  Remember that doing so typically violates privacy and/or wiretap laws, which is one reason (among many) that Squid does advertise its presence.

If you have control over both endpoints but not the firewall in the middle, you can always(?) detect the presence of an intercepting firewall, because no vendors utterly replicate the TCP stack behaviour of the two endpoints perfectly.  You could, in fact, inject deliberate anomalies into the TCP headers and watch to see if they get scrubbed out on either side…


1)      They don’t (AFAIK) run Squid, but they do intercept traffic.  I’ve worked with firewalls for over two decades now, and have had to bug-fix vendor beta code more than once.  I’m not guessing, I’m *telling* you they *do* run proxies.  Not every single one, but most.

2)      Other than the two methods I described at first, how *else* could it work?  Magic?  If anyone else knows of an alternate content inspection scheme, I’d very much want to hear about it.  (Yes, there’s WCCP et al., but that’s not in-line.)  It can be argued that a generic TCP proxy (à la TIS/Gauntlet) is a distinct technique, but I categorize it as a type of traffic interception.


There was some talk of implementing the second option (inspection of the TCP stream) using hashes about a decade, i.e. generalizing the virus-detection mechanism to apply to blocked content as well, but AFAIK no-one has implemented a workable example as of yet.

Also, the “watch-everything and abort-if-bad” approach is functionally very similar to current IPS-based UTMs.


Check out  <http://community.spiceworks.com/topic/201156-history-and-evolution-of-firewalls-part-2> http://community.spiceworks.com/topic/201156-history-and-evolution-of-firewalls-part-2 for some background on how firewalls operate.


-Adam Thompson

 <mailto:athompso at athompso.net> athompso at athompso.net


[Same problem, new software.  If anyone knows how to bottom-post in Outlook 2013, please let me know.  Or how to switch back to text mode without losing all the reply markings/indentation.]



From: Joy [mailto:pj.netfilter at gmail.com] 
Sent: Saturday, February 16, 2013 9:29 PM
To: athompso at athompso.net; pfSense support and discussion
Subject: Re: [pfSense] Regarding Web Filtering


No using squid in transparent mode is caught by sites like http://whatismyipaddress.com and others when you open these sites from inside and even user can know the same by issuing few windows command.


I am only willing to know the concept what actually they do to filter.


On Sun, Feb 17, 2013 at 4:47 AM, Adam Thompson <athompso at athompso.net <mailto:athompso at athompso.net> > wrote:

> > Question:- How commercial UTM like sonicwall and others filters
> > website and content without a proxy?
> In short - they don't. They proxy things, just without a separate
> proxy package like our Squid.

Also, they work much like a transparent proxy, so the user is unaware of their existence (normally).
Some of them "watch" the TCP stream and issue an RSET when they see something "bad", others do behave just like a transparent proxy.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pfsense.org/pipermail/list/attachments/20130216/aaa49e0b/attachment-0001.html>

More information about the List mailing list