info@squid-cache.orgYou can download the FAQ as HTML, PDF, compressed Postscript, plain text, linuxdoc SGML source or as a compressed tar of HTML.
Squid is a high-performance proxy caching server for web clients, supporting FTP, gopher, and HTTP data objects. Unlike traditional caching software, Squid handles all requests in a single, non-blocking, I/O-driven process.
Squid keeps meta data and especially hot objects cached in RAM, caches DNS lookups, supports non-blocking DNS lookups, and implements negative caching of failed requests.
Squid supports SSL, extensive access controls, and full request logging. By using the lightweight Internet Cache Protocol, Squid caches can be arranged in a hierarchy or mesh for additional bandwidth savings.
Squid consists of a main server program squid, a Domain Name System lookup program dnsserver, some optional programs for rewriting requests and performing authentication, and some management and client tools. When squid starts up, it spawns a configurable number of dnsserver processes, each of which can perform a single, blocking Domain Name System (DNS) lookup. This reduces the amount of time the cache waits for DNS lookups.
Squid is derived from the ARPA-funded Harvest project.
Internet object caching is a way to store requested Internet objects (i.e., data available via the HTTP, FTP, and gopher protocols) on a system closer to the requesting site than to the source. Web browsers can then use the local Squid cache as a proxy HTTP server, reducing access time as well as bandwidth consumption.
Harris' Lament says, ``All the good ones are taken."
We needed to distinguish this new version from the Harvest cache software. Squid was the code name for initial development, and it stuck.
Squid is updated often; please see the Squid home page for the most recent versions.
Squid is the result of efforts by numerous individuals from the Internet community. Duane Wessels of the National Laboratory for Applied Network Research (funded by the National Science Foundation) leads code development. Please see the CONTRIBUTORS file for a list of our excellent contributors.
You can download Squid via FTP from the primary FTP site or one of the many worldwide mirror sites.
Many sushi bars also have Squid.
The software is designed to operate on any modern Unix system, and is known to work on at least the following platforms:
For more specific information, please see platforms.html. If you encounter any platform-specific problems, please let us know by registering a entry in our bug database.
Recent versions of Squid will compile and run on Windows/NT with the Cygwin / Mingw packages.
Guido Serassio maintains the native NT port of Squid and is actively working on having the needed changes integrated into the standard Squid distribution. Partially based on earlier NT port by Romeo Anghelache.
LogiSense has ported Squid to Windows NT and sells a supported version. You can also download the source from their FTP site. Thanks to LogiSense for making the code available as required by the GPL terms.
We also have a few other mailing lists which are not strictly Squid-related.
All of our mailing lists have ``-subscribe'' and ``-unsubscribe'' addresses that you must use for subscribe and unsubscribe requests. To unsubscribe from the squid-users list, you send a message to squid-users-unsubscribe@squid-cache.org.
As of version 2.5, Squid can terminate SSL connections. This is perhaps only useful in a surrogate (http accelerator) configuration. You must run configure with --enable-ssl. See https_port in squid.conf for more information.
Squid also supports these encrypted protocols by ``tunelling'' traffic between clients and servers. In this case, Squid can relay the encrypted bits between a client and a server.
Normally, when your browser comes across an https URL, it does one of two things:
The CONNECT method is a way to tunnel any kind of connection through an HTTP proxy. The proxy doesn't understand or interpret the contents. It just passes bytes back and forth between the client and server. For the gory details on tunnelling and the CONNECT method, please see RFC 2817 and Tunneling TCP based protocols through Web proxy servers (expired).
Squid is copyrighted by the University of California San Diego. Squid uses some code developed by others.
Squid is Free Software.
Squid is licensed under the terms of the GNU General Public License.
We think so. Squid uses the Unix time format for all internal time representations. Potential problem areas are in printing and parsing other time representations. We have made the following fixes in to address the year 2000:
Year-2000 fixes were applied to the following Squid versions:
Patches:
Squid-2.2 and earlier versions have a New Year bug. This is not strictly a Year-2000 bug; it would happen on the first day of any year.
Yep. Please see the commercial support page.
The following people have made contributions to this document:
Please send corrections, updates, and comments to: squid-faq@squid-cache.org.
This document is copyrighted (2000) by Duane Wessels.
This document was written in SGML and converted with the SGML-Tools package.
Most current version of this document can always be found at http://www.squid-cache.org/Doc/FAQ/ in HTML, Plain Text, Postscript and SGML formats.
It is easier for us if you send us text which is close to "correct" SGML. The SQUID FAQ currently uses the LINUXDOC DTD. Its probably easiest to follow examples in the this file. Here are the basics:
Use the <url> tag for links, instead of HTML <A HREF ...>
<url url="http://www.squid-cache.org" name="Squid Home Page">
Use <em> for emphasis, config options, and pathnames:
<em>usr/local/squid/etc/squid.conf</em> <em/cache_peer/
Here is how you do lists:
<itemize> <item>foo <item>bar </itemize>
Use <verb>, just like HTML's <PRE> to show unformatted text.
You must download a source archive file of the form squid-x.y.z-src.tar.gz (eg, squid-1.1.6-src.tar.gz) from the Squid home page, or. the Squid FTP site. Context diffs are available for upgrading to new versions. These can be applied with the patch program (available from the GNU FTP site).
For Squid-1.0 and Squid-1.1 versions, you can just type make from the top-level directory after unpacking the source files. For example:
% tar xzf squid-1.1.21-src.tar.gz % cd squid-1.1.21 % make
For Squid-2 you must run the configure script yourself before running make:
% tar xzf squid-2.0.RELEASE-src.tar.gz % cd squid-2.0.RELEASE % ./configure % make
To compile Squid, you will need an ANSI C compiler. Almost all modern Unix systems come with pre-installed compilers which work just fine. The old SunOS compilers do not have support for ANSI C, and the Sun compiler for Solaris is a product which must be purchased separately.
If you are uncertain about your system's C compiler, The GNU C compiler is available at the GNU FTP site. In addition to gcc, you may also want or need to install the binutils package.
You will need Perl installed on your system.
The developers do not have the resources to make pre-compiled binaries available. Instead, we invest effort into making the source code very portable. Some people have made binary packages available. Please see our Platforms Page.
The SGI Freeware site has pre-compiled packages for SGI IRIX.
Squid binaries for FreeBSD on Alpha and Intel.
Squid binaries for NetBSD on everything
Gurkan Sengun has some Sparc/Solaris packages available.
You need the patch program. You should probably duplicate the
entire directory structure before applying the patch. For example, if
you are upgrading from squid-1.1.10 to 1.1.11, you would run
these commands:
cd squid-2.5.STABLE3 mkdir ../squid-2.5.STABLE4 find . -depth -print | cpio -pdv ../squid-1.1.11 cd ../squid-1.1.11 patch -p1 < /tmp/squid-2.5.STABLE3-STABLE4.diff or alternatively cp -rl squid-2.5.STABLE3 squid-2.5.STABLE4 cd squid-2.5.STABLE4 zcat /tmp/squid-2.5.STABLE3-STABLE4.diff.gz | patch -p1After the patch has been applied, you must rebuild Squid from the very beginning, i.e.:
make distclean ./configure ... make make install
If your patch program seems to complain or refuses to work,
you should get a more recent version, from the
GNU FTP site, for example.
The configure script can take numerous options. The most
useful is --prefix to install it in a different directory.
The default installation directory is /usr/local/squid/. To
change the default, you could do:
% cd squid-x.y.z % ./configure --prefix=/some/other/directory/squid
Type
% ./configure --helpto see all available options. You will need to specify some of these options to enable or disable certain features. Some options which are used often include:
--prefix=PREFIX install architecture-independent files in PREFIX
[/usr/local/squid]
--enable-dlmalloc[=LIB] Compile & use the malloc package by Doug Lea
--enable-gnuregex Compile GNUregex
--enable-splaytree Use SPLAY trees to store ACL lists
--enable-xmalloc-debug Do some simple malloc debugging
--enable-xmalloc-debug-trace
Detailed trace of memory allocations
--enable-xmalloc-statistics
Show malloc statistics in status page
--enable-carp Enable CARP support
--enable-async-io Do ASYNC disk I/O using threads
--enable-icmp Enable ICMP pinging
--enable-delay-pools Enable delay pools to limit bandwith usage
--enable-mem-gen-trace Do trace of memory stuff
--enable-useragent-log Enable logging of User-Agent header
--enable-kill-parent-hack
Kill parent on shutdown
--enable-snmp Enable SNMP monitoring
--enable-cachemgr-hostname[=hostname]
Make cachemgr.cgi default to this host
--enable-arp-acl Enable use of ARP ACL lists (ether address)
--enable-htpc Enable HTCP protocol
--enable-forw-via-db Enable Forw/Via database
--enable-cache-digests Use Cache Digests
see http://www.squid-cache.org/Doc/FAQ/FAQ-16.html
--enable-err-language=lang
Select language for Error pages (see errors dir)
by Kevin Sartorelli and Andreas Doering.
Probably you've recently installed bind 8.x. There is a mismatch between the header files and DNS library that Squid has found. There are a couple of things you can try.
First, try adding -lbind to XTRA_LIBS in src/Makefile.
If -lresolv is already there, remove it.
If that doesn't seem to work, edit your arpa/inet.h file and comment out the following:
#define inet_addr __inet_addr #define inet_aton __inet_aton #define inet_lnaof __inet_lnaof #define inet_makeaddr __inet_makeaddr #define inet_neta __inet_neta #define inet_netof __inet_netof #define inet_network __inet_network #define inet_net_ntop __inet_net_ntop #define inet_net_pton __inet_net_pton #define inet_ntoa __inet_ntoa #define inet_pton __inet_pton #define inet_ntop __inet_ntop #define inet_nsap_addr __inet_nsap_addr #define inet_nsap_ntoa __inet_nsap_ntoa
If you have source for BIND, you can modify it as indicated in the diff below. It causes the global variable _dns_ttl_ to be set with the TTL of the most recent lookup. Then, when you compile Squid, the configure script will look for the _dns_ttl_ symbol in libresolv.a. If found, dnsserver will return the TTL value for every lookup.
This hack was contributed by Endre Balint Nagy.
diff -ru bind-4.9.4-orig/res/gethnamaddr.c bind-4.9.4/res/gethnamaddr.c
--- bind-4.9.4-orig/res/gethnamaddr.c Mon Aug 5 02:31:35 1996
+++ bind-4.9.4/res/gethnamaddr.c Tue Aug 27 15:33:11 1996
@@ -133,6 +133,7 @@
} align;
extern int h_errno;
+int _dns_ttl_;
#ifdef DEBUG
static void
@@ -223,6 +224,7 @@
host.h_addr_list = h_addr_ptrs;
haveanswer = 0;
had_error = 0;
+ _dns_ttl_ = -1;
while (ancount-- > 0 && cp < eom && !had_error) {
n = dn_expand(answer->buf, eom, cp, bp, buflen);
if ((n < 0) || !(*name_ok)(bp)) {
@@ -232,8 +234,11 @@
cp += n; /* name */
type = _getshort(cp);
cp += INT16SZ; /* type */
- class = _getshort(cp);
- cp += INT16SZ + INT32SZ; /* class, TTL */
+ class = _getshort(cp);
+ cp += INT16SZ; /* class */
+ if (qtype == T_A && type == T_A)
+ _dns_ttl_ = _getlong(cp);
+ cp += INT32SZ; /* TTL */
n = _getshort(cp);
cp += INT16SZ; /* len */
if (class != C_IN) {
And here is a patch for BIND-8:
*** src/lib/irs/dns_ho.c.orig Tue May 26 21:55:51 1998
--- src/lib/irs/dns_ho.c Tue May 26 21:59:57 1998
***************
*** 87,92 ****
--- 87,93 ----
#endif
extern int h_errno;
+ int _dns_ttl_;
/* Definitions. */
***************
*** 395,400 ****
--- 396,402 ----
pvt->host.h_addr_list = pvt->h_addr_ptrs;
haveanswer = 0;
had_error = 0;
+ _dns_ttl_ = -1;
while (ancount-- > 0 && cp < eom && !had_error) {
n = dn_expand(ansbuf, eom, cp, bp, buflen);
if ((n < 0) || !(*name_ok)(bp)) {
***************
*** 404,411 ****
cp += n; /* name */
type = ns_get16(cp);
cp += INT16SZ; /* type */
! class = ns_get16(cp);
! cp += INT16SZ + INT32SZ; /* class, TTL */
n = ns_get16(cp);
cp += INT16SZ; /* len */
if (class != C_IN) {
--- 406,416 ----
cp += n; /* name */
type = ns_get16(cp);
cp += INT16SZ; /* type */
! class = _getshort(cp);
! cp += INT16SZ; /* class */
! if (qtype == T_A && type == T_A)
! _dns_ttl_ = _getlong(cp);
! cp += INT32SZ; /* TTL */
n = ns_get16(cp);
cp += INT16SZ; /* len */
if (class != C_IN) {
cache_cf.c: In function `parseConfigFile': cache_cf.c:1353: yacc stack overflow before `token' ...
You may need to upgrade your gcc installation to a more recent version. Check your gcc version with
gcc -vIf it is earlier than 2.7.2, you might consider upgrading.
The following error occurs on Solaris systems using gcc when the Solaris C compiler is not installed:
/usr/bin/rm -f libmiscutil.a /usr/bin/false r libmiscutil.a rfc1123.o rfc1738.o util.o ... make[1]: *** [libmiscutil.a] Error 255 make[1]: Leaving directory `/tmp/squid-1.1.11/lib' make: *** [all] Error 1Note on the second line the /usr/bin/false. This is supposed to be a path to the ar program. If configure cannot find ar on your system, then it substitues false.
To fix this you either need to:
Please check the page of platforms on which Squid is known to compile. Your problem might be listed there together with a solution. If it isn't listed there, mail us what you are trying, your Squid version, and the problems you encounter.
Warnings are usually not a big concern, and can be common with software designed to operate on multiple platforms. If you feel like fixing compile-time warnings, please do so and send us the patches.
by Doug Nazar
In order in compile squid, you need to have a reasonable facsimile of a Unix system installed. This includes bash, make, sed, emx, various file utilities and a few more. I've setup a TVFS drive that matches a Unix file system but this probably isn't strictly necessary.
I made a few modifications to the pristine EMX 0.9d install.
You will need to run scripts/convert.configure.to.os2 (in the Squid source distribution) to modify the configure script so that it can search for the various programs.
Next, you need to set a few environment variables (see EMX docs for meaning):
export EMXOPT="-h256 -c" export LDFLAGS="-Zexe -Zbin -s"
Now you are ready to configure squid:
./configure
Compile everything:
make
and finally, install:
make install
This will by default, install into /usr/local/squid. If you wish to install somewhere else, see the --prefix option for configure.
Now, don't forget to set EMXOPT before running squid each time. I recommend using the -Y and -N options.
There are no hard-and-fast rules. The most important resource for Squid is physical memory. Your processor does not need to be ultra-fast. Your disk system will be the major bottleneck, so fast disks are important for high-volume caches. Do not use IDE disks if you can help it.
In late 1998, if you are buying a new machine for a cache, I would recommend the following configuration:
Also, see Squid Sizing for Intel Platforms by Martin Hamilton This is a very nice page summarizing system configurations people are using for large Squid caches.
After compiling Squid, you can install it with this simple command:
% make installIf you have enabled the ICMP features then you will also want to type
% su # make install-pinger
After installing, you will want to edit and customize the squid.conf file. By default, this file is located at /usr/local/squid/etc/squid.conf.
Also, a QUICKSTART guide has been included with the source distribution. Please see the directory where you unpacked the source archive.
The squid.conf file defines the configuration for squid. the configuration includes (but not limited to) HTTP port number, the ICP request port number, incoming and outgoing requests, information about firewall access, and various timeout information.
Yes, after you make install, a sample squid.conf file will
exist in the ``etc" directory under the Squid installation directory.
The sample squid.conf file contains comments explaining each option.
First you need to make your Squid configuration. The Squid configuration can be found in /usr/local/squid/etc/squid.conf and by default includes documentation on all directives.
In the Suqid distribution there is a small QUICKSTART guide indicating which directives you need to look closer at and why. At a absolute minimum you need to change the http_access configuration to allow access from your clients.
To verify your configuration file you can use the -k parse option
% /usr/local/squid/sbin/squid -k parseIf this outputs any errors then these are syntax errors or other fatal misconfigurations and needs to be corrected before you continue. If it is silent and immediately gives back the command promt then your squid.conf is syntactically correct and could be understood by Squid.
After you've finished editing the configuration file, you can start Squid for the first time. The procedure depends a little bit on which version you are using.
First, you must create the swap directories. Do this by running Squid with the -z option:
% /usr/local/squid/sbin/squid -z
NOTE: If you run Squid as root then you may need to first create /usr/local/squid/var/logs and your cache_dir directories and assign ownership of these to the cache_effective_user configured in your squid.conf.
Once the creation of the cache directories completes, you can start Squid and try it out. Probably the best thing to do is run it from your terminal and watch the debugging output. Use this command:
% /usr/local/squid/sbin/squid -NCd1If everything is working okay, you will see the line:
Ready to serve requests.If you want to run squid in the background, as a daemon process, just leave off all options:
% /usr/local/squid/sbin/squid
NOTE: depending on which http_port you select you may need to start squid as root (http_port <1024).
NOTE: In Squid-2.4 and earlier Squid was installed in bin by default, not sbin.
Squid-2 has a restart feature built in. This greatly simplifies starting Squid and means that you don't need to use RunCache or inittab. At the minimum, you only need to enter the pathname to the Squid executable. For example:
/usr/local/squid/sbin/squid
Squid will automatically background itself and then spawn a child process. In your syslog messages file, you should see something like this:
Sep 23 23:55:58 kitty squid[14616]: Squid Parent: child process 14617 startedThat means that process ID 14563 is the parent process which monitors the child process (pid 14617). The child process is the one that does all of the work. The parent process just waits for the child process to exit. If the child process exits unexpectedly, the parent will automatically start another child process. In that case, syslog shows:
Sep 23 23:56:02 kitty squid[14616]: Squid Parent: child process 14617 exited with status 1 Sep 23 23:56:05 kitty squid[14616]: Squid Parent: child process 14619 started
If there is some problem, and Squid can not start, the parent process will give up after a while. Your syslog will show:
Sep 23 23:56:12 kitty squid[14616]: Exiting due to repeated, frequent failuresWhen this happens you should check your syslog messages and cache.log file for error messages.
When you look at a process (ps command) listing, you'll see two squid processes:
24353 ?? Ss 0:00.00 /usr/local/squid/bin/squid 24354 ?? R 0:03.39 (squid) (squid)The first is the parent process, and the child process is the one called ``(squid)''. Note that if you accidentally kill the parent process, the child process will not notice.
If you want to run Squid from your termainal and prevent it from backgrounding and spawning a child process, use the -N command line option.
/usr/local/squid/bin/squid -N
On systems which have an /etc/inittab file (Digital Unix, Solaris, IRIX, HP-UX, Linux), you can add a line like this:
sq:3:respawn:/usr/local/squid/sbin/squid.sh < /dev/null >> /tmp/squid.log 2>&1We recommend using a squid.sh shell script, but you could instead call Squid directly with the -N option and other options you may require. A sameple squid.sh script is shown below:
#!/bin/sh
C=/usr/local/squid
PATH=/usr/bin:$C/bin
TZ=PST8PDT
export PATH TZ
# User to notify on restarts
notify="root"
# Squid command line options
opts=""
cd $C
umask 022
sleep 10
while [ -f /var/run/nosquid ]; do
sleep 1
done
/usr/bin/tail -20 $C/logs/cache.log \
| Mail -s "Squid restart on `hostname` at `date`" $notify
exec bin/squid -N $opts
On BSD-ish systems, you will need to start Squid from the ``rc'' files, usually /etc/rc.local. For example:
if [ -f /usr/local/squid/sbin/squid ]; then
echo -n ' Squid'
/usr/local/squid/sbin/squid
fi
Squid ships with a init.d type startup script in contrib/squid.rc which works on most init.d type systems. Or you can write your own using any normal init.d script found in your system as template and add the start/stop fragments shown below.
Start:
/usr/local/squid/sbin/squid
Stop:
/usr/local/squid/sbin/squid -k shutdown
n=120
while /usr/local/squid/sbin/squid -k check && [ $n -gt 120 ]; do
sleep 1
echo -n .
n=`expr $n - 1`
done
You can use the squidclient program:
% squidclient http://www.netscape.com/ > test
There are other command-line HTTP client programs available as well. Two that you may find useful are wget and echoping.
Another way is to use Squid itself to see if it can signal a running Squid process:
% squid -k checkAnd then check the shell's exit status variable.
Also, check the log files, most importantly the access.log and cache.log files.
These are the command line options for Squid-2:
Specify an alternate port number for incoming HTTP requests. Useful for testing a configuration file on a non-standard port.
Debugging level for ``stderr'' messages. If you use this option, then debugging messages up to the specified level will also be written to stderr.
Specify an alternate squid.conf file instead of the pathname compiled into the executable.
Prints the usage and help message.
Sends a HUP signal, which causes Squid to re-read its configuration files.
Sends an USR1 signal, which causes Squid to rotate its log files. Note, if logfile_rotate is set to zero, Squid still closes and re-opens all log files.
Sends a TERM signal, which causes Squid to wait briefly for current connections to finish and then exit. The amount of time to wait is specified with shutdown_lifetime.
Sends an INT signal, which causes Squid to shutdown immediately, without waiting for current connections.
Sends a KILL signal, which causes the Squid process to exit immediately, without closing any connections or log files. Use this only as a last resort.
Sends an USR2 signal, which causes Squid to generate full debugging messages until the next USR2 signal is recieved. Obviously very useful for debugging problems.
Sends a ``ZERO'' signal to the Squid process. This simply checks whether or not the process is actually running.
Send debugging (level 0 only) message to syslog.
Specify an alternate port number for ICP messages. Useful for testing a configuration file on a non-standard port.
Prints the Squid version.
Creates disk swap directories. You must use this option when installing Squid for the first time, or when you add or modify the cache_dir configuration.
Do not make initial DNS tests. Normally, Squid looks up some well-known DNS hostnames to ensure that your DNS name resolution service is working properly.
If the swap.state logs are clean, then the cache is rebuilt in the ``foreground'' before any requests are served. This will decrease the time required to rebuild the cache, but HTTP requests will not be satisified during this time.
Do not automatically become a background daemon process.
Do not set the SO_REUSEADDR option on sockets.
Enable virtual host support for the httpd-accelerator mode. This is identical to writing httpd_accel_host virtual in the config file.
Enable full debugging while parsing the config file.
Return ICP_OP_MISS_NOFETCH instead of ICP_OP_MISS while the swap.state file is being read. If your cache has mostly child caches which use ICP, this will allow your cache to rebuild faster.
Squid is a single process application and can not make use of SMP. If you want to make Squid benefit from a SMP system you will need to run multiple instances of Squid and find a way to distribute your users on the different Squid instances just as if you had multiple Squid boxes.
Having two CPUs is indeed nice for running other CPU intensive tasks on the same server as the proxy, such as if you have a lot of logs and need to run various statistics collections during peak hours.
The authentication and group helpers barely use any CPU and does not benefit from dual-CPU configuration.
RAID1 is fine, and so are separate drives.
RAID0 (striping) with Squid only gives you the drawback that if you lose one of the drives the whole stripe set is lost. There is no benefit in performance as Squid already distributes the load on the drives quite nicely.
Squid is the worst case application for RAID5, whether hardware or software, and will absolutely kill the performance of a RAID5. Once the cache has been filled Squid uses a lot of small random writes which the worst case workload for RAID5, effectively reducing write speed to only little more than that of one single drive.
Generally seek time is what you want to optimize for Squid, or more precisely the total amount of seeks/s your system can sustain. Choosing the right RAID solution generally decreases the amount of seeks/s your system can sustain significantly.
To place your cache in a hierarchy, use the cache_peer
directive in squid.conf to specify the parent and sibling
nodes.
For example, the following squid.conf file on
childcache.example.com configures its cache to retrieve
data from one parent cache and two sibling caches:
# squid.conf - On the host: childcache.example.com
#
# Format is: hostname type http_port udp_port
#
cache_peer parentcache.example.com parent 3128 3130
cache_peer childcache2.example.com sibling 3128 3130
cache_peer childcache3.example.com sibling 3128 3130
The cache_peer_domain directive allows you to specify that
certain caches siblings or parents for certain domains:
# squid.conf - On the host: sv.cache.nlanr.net
#
# Format is: hostname type http_port udp_port
#
cache_peer electraglide.geog.unsw.edu.au parent 3128 3130
cache_peer cache1.nzgate.net.nz parent 3128 3130
cache_peer pb.cache.nlanr.net parent 3128 3130
cache_peer it.cache.nlanr.net parent 3128 3130
cache_peer sd.cache.nlanr.net parent 3128 3130
cache_peer uc.cache.nlanr.net sibling 3128 3130
cache_peer bo.cache.nlanr.net sibling 3128 3130
cache_peer_domain electraglide.geog.unsw.edu.au .au
cache_peer_domain cache1.nzgate.net.nz .au .aq .fj .nz
cache_peer_domain pb.cache.nlanr.net .uk .de .fr .no .se .it
cache_peer_domain it.cache.nlanr.net .uk .de .fr .no .se .it
cache_peer_domain sd.cache.nlanr.net .mx .za .mu .zm
The configuration above indicates that the cache will use
pb.cache.nlanr.net and it.cache.nlanr.net
for domains uk, de, fr, no, se and it, sd.cache.nlanr.net
for domains mx, za, mu and zm, and cache1.nzgate.net.nz
for domains au, aq, fj, and nz.
We have a simple set of guidelines for joining the NLANR cache hierarchy.
The NLANR hierarchy can provide you with an initial source for parent or sibling caches. Joining the NLANR global cache system will frequently improve the performance of your caching service.
Just enable these options in your squid.conf and you'll be registered:
cache_announce 24 announce_to sd.cache.nlanr.net:3131
NOTE: announcing your cache is not the same thing as joining the NLANR cache hierarchy. You can join the NLANR cache hierarchy without registering, and you can register without joining the NLANR cache hierarchy.
Visit the NLANR cache registration database to discover other caches near you. Keep in mind that just because a cache is registered in the database does not mean they are willing to be your parent/sibling/child. But it can't hurt to ask...
This entry has been moved to a different section.
Note: The information here is current for version 2.2.
If you are behind a firewall then you can't make direct connections to the outside world, so you must use a parent cache. Squid doesn't use ICP queries for a request if it's behind a firewall or if there is only one parent.
You can use the never_direct access list in
squid.conf to specify which requests must be forwarded to
your parent cache outside the firewall, and the always_direct access list
to specify which requests must not be forwarded. For example, if Squid
must connect directly to all servers that end with mydomain.com, but
must use the parent for all others, you would write:
acl INSIDE dstdomain .mydomain.com always_direct allow INSIDE never_direct allow all
You could also specify internal servers by IP address
acl INSIDE_IP dst 1.2.3.0/24 always_direct allow INSIDE_IP never_direct allow allNote, however that when you use IP addresses, Squid must perform a DNS lookup to convert URL hostnames to an address. Your internal DNS servers may not be able to lookup external domains.
If you use never_direct and you have multiple parent caches, then you probably will want to mark one of them as a default choice in case Squid can't decide which one to use. That is done with the default keyword on a cache_peer line. For example:
cache_peer xyz.mydomain.com parent 3128 0 default
Note: The information here is current for version 2.2.
First, you need to give Squid a parent cache. Second, you need to tell Squid it can not connect directly to origin servers. This is done with three configuration file lines:
cache_peer parentcache.foo.com parent 3128 0 no-query default acl all src 0.0.0.0/0.0.0.0 never_direct allow allNote, with this configuration, if the parent cache fails or becomes unreachable, then every request will result in an error message.
In case you want to be able to use direct connections when all the parents go down you should use a different approach:
cache_peer parentcache.foo.com parent 3128 0 no-query prefer_direct offThe default behaviour of Squid in the absence of positive ICP, HTCP, etc replies is to connect to the origin server instead of using parents. The prefer_direct off directive tells Squid to try parents first.
The dnsserver processes are used by squid because the gethostbyname(3) library routines used to
convert web sites names to their internet addresses
blocks until the function returns (i.e., the process that calls
it has to wait for a reply). Since there is only one squid
process, everyone who uses the cache would have to wait each
time the routine was called. This is why the dnsserver is
a separate process, so that these processes can block,
without causing blocking in squid.
It's very important that there are enough dnsserver processes to cope with every access you will need, otherwise squid will stop occasionally. A good rule of thumb is to make sure you have at least the maximum number of dnsservers squid has ever needed on your system, and probably add two to be on the safe side. In other words, if you have only ever seen at most three dnsserver processes in use, make at least five. Remember that a dnsserver is small and, if unused, will be swapped out.
First, find out if you have enough dnsserver processes running by looking at the Cachemanager dns output. Ideally, you should see that the first dnsserver handles a lot of requests, the second one less than the first, etc. The last dnsserver should have serviced relatively few requests. If there is not an obvious decreasing trend, then you need to increase the number of dns_children in the configuration file. If the last dnsserver has zero requests, then you definately have enough.
Another factor which affects the dnsserver service time is the proximity of your DNS resolver. Normally we do not recommend running Squid and named on the same host. Instead you should try use a DNS resolver (named) on a different host, but on the same LAN. If your DNS traffic must pass through one or more routers, this could be causing unnecessary delays.
Before you run the configure script, simply set the CACHE_HTTP_PORT environment variable.
setenv CACHE_HTTP_PORT 8080 ./configure make make install
With Squid-1.1 it is NOT possible. Each cache_dir is assumed to be the same size. The cache_swap setting defines the size of all cache_dir's taken together. If you have N cache_dir's then each one will hold cache_swap ÷ N Megabytes.
Most people have a disk partition dedicated to the Squid cache. You don't want to use the entire partition size. You have to leave some extra room. Currently, Squid is not very tolerant of running out of disk space.
Lets say you have a 9GB disk. Remember that disk manufacturers lie about the space available. A so-called 9GB disk usually results in about 8.5GB of raw, usable space. First, put a filesystem on it, and mount it. Then check the ``available space'' with your df program. Note that you lose some disk space to filesystem overheads, like superblocks, inodes, and directory entries. Also note that Unix normally keeps 10% free for itself. So with a 9GB disk, you're probably down to about 8GB after formatting.
Next, I suggest taking off another 10% or so for Squid overheads, and a "safe buffer." Squid normally puts its swap.state files in each cache directory. These grow in size until you rotate the logs, or restart squid. Also note that Squid performs better when there is more free space. So if performance is important to you, then take off even more space. Typically, for a 9GB disk, I recommend a cache_dir setting of 6000 to 7500 Megabytes:
cache_dir ... 7000 16 256
Its better to start out conservative. After the cache becomes full, look at the disk usage. If you think there is plenty of unused space, then increase the cache_dir setting a little.
If you're getting ``disk full'' write errors, then you definately need to decrease your cache size.
With Squid-1.1, yes, you will lose your cache. This is because version 1.1 uses a simplistic algorithm to distribute files between cache directories.
With Squid-2, you will not lose your existing cache. You can add and delete cache_dir's without affecting any of the others.
Several people on both the fwtk-users and the squid-users mailing asked about using Squid in combination with http-gw from the TIS toolkit. The most elegant way in my opinion is to run an internal Squid caching proxyserver which handles client requests and let this server forward it's requests to the http-gw running on the firewall. Cache hits won't need to be handled by the firewall.
In this example Squid runs on the same server as the http-gw, Squid uses 8000 and http-gw uses 8080 (web). The local domain is home.nl.
Either run http-gw as a daemon from the /etc/rc.d/rc.local (Linux Slackware):
exec /usr/local/fwtk/http-gw -daemon 8080or run it from inetd like this:
web stream tcp nowait.100 root /usr/local/fwtk/http-gw http-gwI increased the watermark to 100 because a lot of people run into problems with the default value.
Make sure you have at least the following line in /usr/local/etc/netperm-table:
http-gw: hosts 127.0.0.1You could add the IP-address of your own workstation to this rule and make sure the http-gw by itself works, like:
http-gw: hosts 127.0.0.1 10.0.0.1
The following settings are important:
http_port 8000 icp_port 0 cache_peer localhost.home.nl parent 8080 0 default acl HOME dstdomain .home.nl alwayws_direct allow HOME never_direct allow allThis tells Squid to use the parent for all domains other than home.nl. Below, access.log entries show what happens if you do a reload on the Squid-homepage:
872739961.631 1566 10.0.0.21 ERR_CLIENT_ABORT/304 83 GET http://www.squid-cache.org/ - DEFAULT_PARENT/localhost.home.nl - 872739962.976 1266 10.0.0.21 TCP_CLIENT_REFRESH/304 88 GET http://www.nlanr.net/Images/cache_now.gif - DEFAULT_PARENT/localhost.home.nl - 872739963.007 1299 10.0.0.21 ERR_CLIENT_ABORT/304 83 GET http://www.squid-cache.org/Icons/squidnow.gif - DEFAULT_PARENT/localhost.home.nl - 872739963.061 1354 10.0.0.21 TCP_CLIENT_REFRESH/304 83 GET http://www.squid-cache.org/Icons/Squidlogo2.gif - DEFAULT_PARENT/localhost.home.nl
http-gw entries in syslog:
Aug 28 02:46:00 memo http-gw[2052]: permit host=localhost/127.0.0.1 use of gateway (V2.0beta) Aug 28 02:46:00 memo http-gw[2052]: log host=localhost/127.0.0.1 protocol=HTTP cmd=dir dest=www.squid-cache.org path=/ Aug 28 02:46:01 memo http-gw[2052]: exit host=localhost/127.0.0.1 cmds=1 in=0 out=0 user=unauth duration=1 Aug 28 02:46:01 memo http-gw[2053]: permit host=localhost/127.0.0.1 use of gateway (V2.0beta) Aug 28 02:46:01 memo http-gw[2053]: log host=localhost/127.0.0.1 protocol=HTTP cmd=get dest=www.squid-cache.org path=/Icons/Squidlogo2.gif Aug 28 02:46:01 memo http-gw[2054]: permit host=localhost/127.0.0.1 use of gateway (V2.0beta) Aug 28 02:46:01 memo http-gw[2054]: log host=localhost/127.0.0.1 protocol=HTTP cmd=get dest=www.squid-cache.org path=/Icons/squidnow.gif Aug 28 02:46:01 memo http-gw[2055]: permit host=localhost/127.0.0.1 use of gateway (V2.0beta) Aug 28 02:46:01 memo http-gw[2055]: log host=localhost/127.0.0.1 protocol=HTTP cmd=get dest=www.nlanr.net path=/Images/cache_now.gif Aug 28 02:46:02 memo http-gw[2055]: exit host=localhost/127.0.0.1 cmds=1 in=0 out=0 user=unauth duration=1 Aug 28 02:46:03 memo http-gw[2053]: exit host=localhost/127.0.0.1 cmds=1 in=0 out=0 user=unauth duration=2 Aug 28 02:46:04 memo http-gw[2054]: exit host=localhost/127.0.0.1 cmds=1 in=0 out=0 user=unauth duration=3
To summarize:
Advantages:
Disadvantages:
-- Rodney van den Oever
When a proxy-cache is used, a server does not see the connection coming from the originating client. Many people like to implement access controls based on the client address. To accommodate these people, Squid adds its own request header called "X-Forwarded-For" which looks like this:
X-Forwarded-For: 128.138.243.150, unknown, 192.52.106.30Entries are always IP addresses, or the word unknown if the address could not be determined or if it has been disabled with the forwarded_for configuration option.
We must note that access controls based on this header are extremely weak and simple to fake. Anyone may hand-enter a request with any IP address whatsoever. This is perhaps the reason why client IP addresses have been omitted from the HTTP/1.1 specification.
Because of the weakness of this header, support for access controls based on X-Forwarder-For is not yet available in any officially released version of squid. However, unofficial patches are available from the follow_xff Squid development project and may be integrated into later versions of Squid once a suitable trust model have been developed.
Yes it can, however the way of doing it has changed from earlier versions of squid. As of squid-2.2 a more customisable method has been introduced. Please follow the instructions for the version of squid that you are using. As a default, no anonymizing is done.
If you choose to use the anonymizer you might wish to investigate the forwarded_for option to prevent the client address being disclosed. Failure to turn off the forwarded_for option will reduce the effectiveness of the anonymizer. Finally if you filter the User-Agent header using the fake_user_agent option can prevent some user problems as some sites require the User-Agent header.
With the introduction of squid 2.2 the anonoymizer has become more customisable. It now allows specification of exactly which headers will be allowed to pass. This is further extended in Squid-2.5 to allow headers to be anonymized conditionally.
For details see the documentation of the http_header_access and header_replace directives in squid.conf.default.
References: Anonymous WWW
Sure, just use the always_direct access list.
For example, if you want Squid to connect directly to hotmail.com servers, you can use these lines in your config file:
acl hotmail dstdomain .hotmail.com always_direct allow hotmail
Sure, there are few things you can do.
You can use the no_cache access list to make Squid never cache any response:
acl all src 0/0 no_cache deny all
With Squid-2.4 and later you can use the ``null'' storage module to avoid having a cache directory:
cache_dir null /tmp
Note: a null cache_dir does not disable caching, but it does save you from creating a cache structure if you have disabled caching with no_cache.
Note: the directory (e.g., /tmp) must exist so that squid can chdir to it, unless you also use the coredump_dir option.
To configure Squid for the ``null'' storage module, specify it on the configure command line:
./configure --enable-storeio=ufs,null ...
You can set the global reply_body_max_size parameter. This option controls the largest HTTP message body that will be sent to a cache client for one request.
If the HTTP response coming from the server has a Content-length
header, then Squid compares the content-length value to the
reply_body_max_size value. If the content-length is larger,
the server connection is closed and the user receives an error
message from Squid.
Some responses don't have Content-length
headers. In this case, Squid counts how many bytes are written
to the client. Once the limit is reached, the client's connection
is simply closed.
Note that ``creative'' user-agents will still be able to download really large files through the cache using HTTP/1.1 range requests.
Most web browsers available today support proxying and are easily configured to use a Squid server as a proxy. Some browsers support advanced features such as lists of domains or URL patterns that shouldn't be fetched through the proxy, or JavaScript automatic proxy configuration.
Select Network Preferences from the Options menu. On the Proxies page, click the radio button next to Manual Proxy Configuration and then click on the View button. For each protocol that your Squid server supports (by default, HTTP, FTP, and gopher) enter the Squid server's hostname or IP address and put the HTTP port number for the Squid server (by default, 3128) in the Port column. For any protocols that your Squid does not support, leave the fields blank.
Here is a screen shot of the Netscape Navigator manual proxy configuration screen.
Netscape Navigator's proxy configuration can be automated with JavaScript (for Navigator versions 2.0 or higher). Select Network Preferences from the Options menu. On the Proxies page, click the radio button next to Automatic Proxy Configuration and then fill in the URL for your JavaScript proxy configuration file in the text box. The box is too small, but the text will scroll to the right as you go.
Here is a screen shot of the Netscape Navigator automatic proxy configuration screen.
You may also wish to consult Netscape's documentation for the Navigator JavaScript proxy configuration
Here is a sample auto configuration JavaScript from Oskar Pearson:
//We (www.is.co.za) run a central cache for our customers that they
//access through a firewall - thus if they want to connect to their intranet
//system (or anything in their domain at all) they have to connect
//directly - hence all the "fiddling" to see if they are trying to connect
//to their local domain.
//Replace each occurrence of company.com with your domain name
//and if you have some kind of intranet system, make sure
//that you put it's name in place of "internal" below.
//We also assume that your cache is called "cache.company.com", and
//that it runs on port 8080. Change it down at the bottom.
//(C) Oskar Pearson and the Internet Solution (http://www.is.co.za)
function FindProxyForURL(url, host)
{
//If they have only specified a hostname, go directly.
if (isPlainHostName(host))
return "DIRECT";
//These connect directly if the machine they are trying to
//connect to starts with "intranet" - ie http://intranet
//Connect directly if it is intranet.*
//If you have another machine that you want them to
//access directly, replace "internal*" with that
//machine's name
if (shExpMatch( host, "intranet*")||
shExpMatch(host, "internal*"))
return "DIRECT";
//Connect directly to our domains (NB for Important News)
if (dnsDomainIs( host,"company.com")||
//If you have another domain that you wish to connect to
//directly, put it in here
dnsDomainIs(host,"sistercompany.com"))
return "DIRECT";
//So the error message "no such host" will appear through the
//normal Netscape box - less support queries :)
if (!isResolvable(host))
return "DIRECT";
//We only cache http, ftp and gopher
if (url.substring(0, 5) == "http:" ||
url.substring(0, 4) == "ftp:"||
url.substring(0, 7) == "gopher:")
//Change the ":8080" to the port that your cache
//runs on, and "cache.company.com" to the machine that
//you run the cache on
return "PROXY cache.company.com:8080; DIRECT";
//We don't cache WAIS
if (url.substring(0, 5) == "wais:")
return "DIRECT";
else
return "DIRECT";
}
For Mosaic and Lynx, you can set environment variables before starting the application. For example (assuming csh or tcsh):
% setenv http_proxy http://mycache.example.com:3128/ % setenv gopher_proxy http://mycache.example.com:3128/ % setenv ftp_proxy http://mycache.example.com:3128/
For Lynx you can also edit the lynx.cfg file to configure proxy usage. This has the added benefit of causing all Lynx users on a system to access the proxy without making environment variable changes for each user. For example:
http_proxy:http://mycache.example.com:3128/
ftp_proxy:http://mycache.example.com:3128/
gopher_proxy:http://mycache.example.com:3128/
There's one nasty side-effect to using auto-proxy scripts: if you start the web browser it will try and load the auto-proxy-script.
If your script isn't available either because the web server hosting the script is down or your workstation can't reach the web server (e.g. because you're working off-line with your notebook and just want to read a previously saved HTML-file) you'll get different errors depending on the browser you use.
The Netscape browser will just return an error after a timeout (after that it tries to find the site 'www.proxy.com' if the script you use is called 'proxy.pac').
The Microsoft Internet Explorer on the other hand won't even start, no window displays, only after about 1 minute it'll display a window asking you to go on with/without proxy configuration.
The point is that your workstations always need to locate the proxy-script. I created some extra redundancy by hosting the script on two web servers (actually Apache web servers on the proxy servers themselves) and adding the following records to my primary nameserver:
proxy CNAME proxy1
CNAME proxy2
The clients just refer to 'http://proxy/proxy.pac'. This script looks like this:
function FindProxyForURL(url,host)
{
// Hostname without domainname or host within our own domain?
// Try them directly:
// http://www.domain.com actually lives before the firewall, so
// make an exception:
if ((isPlainHostName(host)||dnsDomainIs( host,".domain.com")) &&
!localHostOrDomainIs(host, "www.domain.com"))
return "DIRECT";
// First try proxy1 then proxy2. One server mostly caches '.com'
// to make sure both servers are not
// caching the same data in the normal situation. The other
// server caches the other domains normally.
// If one of 'm is down the client will try the other server.
else if (shExpMatch(host, "*.com"))
return "PROXY proxy1.domain.com:8080; PROXY proxy2.domain.com:8081; DIRECT";
return "PROXY proxy2.domain.com:8081; PROXY proxy1.domain.com:8080; DIRECT";
}
I made sure every client domain has the appropriate 'proxy' entry. The clients are automatically configured with two nameservers using DHCP.
-- Rodney van den Oever
The Sharp Super Proxy Script page contains a lot of good information about hash-based proxy auto-configuration scripts. With these you can distribute the load between a number of caching proxies.
Select Options from the View menu. Click on the Connection tab. Tick the Connect through Proxy Server option and hit the Proxy Settings button. For each protocol that your Squid server supports (by default, HTTP, FTP, and gopher) enter the Squid server's hostname or IP address and put the HTTP port number for the Squid server (by default, 3128) in the Port column. For any protocols that your Squid does not support, leave the fields blank.
Here is a screen shot of the Internet Explorer proxy configuration screen.
Microsoft is also starting to support Netscape-style JavaScript automated proxy configuration. As of now, only MSIE version 3.0a for Windows 3.1 and Windows NT 3.51 supports this feature (i.e., as of version 3.01 build 1225 for Windows 95 and NT 4.0, the feature was not included).
If you have a version of MSIE that does have this feature, elect Options from the View menu. Click on the Advanced tab. In the lower left-hand corner, click on the Automatic Configuration button. Fill in the URL for your JavaScript file in the dialog box it presents you. Then exit MSIE and restart it for the changes to take effect. MSIE will reload the JavaScript file every time it starts.
Netmanage WebSurfer supports manual proxy configuration and exclusion lists for hosts or domains that should not be fetched via proxy (this information is current as of WebSurfer 5.0). Select Preferences from the Settings menu. Click on the Proxies tab. Select the Use Proxy options for HTTP, FTP, and gopher. For each protocol that enter the Squid server's hostname or IP address and put the HTTP port number for the Squid server (by default, 3128) in the Port boxes. For any protocols that your Squid does not support, leave the fields blank.
Take a look at this screen shot if the instructions confused you.
On the same configuration window, you'll find a button to bring up the exclusion list dialog box, which will let you enter some hosts or domains that you don't want fetched via proxy. It should be self-explanatory, but you might look at this screen shot just for fun anyway.
Select Proxy Servers... from the Preferences menu. Check each protocol that your Squid server supports (by default, HTTP, FTP, and Gopher) and enter the Squid server's address as hostname:port (e.g. mycache.example.com:3128 or 123.45.67.89:3128). Click on Okay to accept the setup.
Notes:
-- Hume Smith
Insert your username in the host part of the URL, for example:
ftp://joecool@ftp.foo.org/Squid should then prompt you for your account password. Alternatively, you can specify both your username and password in the URL itself:
ftp://joecool:secret@ftp.foo.org/However, we certainly do not recommend this, as it could be very easy for someone to see or grab your password.
You may like to start by reading the Expired Internet-Draft that describes WPAD.
After reading the 8 steps below, if you don't understand any of the terms or methods mentioned, you probably shouldn't be doing this. Implementing wpad requires you to fully understand:
This is not a recommendation for any product or version. As far as I know IE5 is the only browser out now implementing wpad. I think wpad is an excellent feature that will return several hours of life per month. Hopefully, all browser clients will implement it as well. But it will take years for all the older browsers to fade away though.
I have only focused on the domain name method, to the exclusion of the DHCP method. I think the dns method might be easier for most people. I don't currently, and may never, fully understand wpad and IE5, but this method worked for me. It may work for you.
But if you'd rather just have a go ...
Andrei Ivanov notes that you should be able to use an HTTP redirect if you want to store the wpad.dat file somewhere else. You can probably even redirect wpad.dat to proxy.pac:
Redirect /wpad.dat http://racoon.riga.lv/proxy.pac
http://www.your.domain.name/wpad.dat should bring up
the script text in your browser window.
application/x-ns-proxy-autoconfig datAnd then restart your web server, for new mime type to work.
http://www.your.domain.name/wpad.dat Test that
that all works as per your script and network. There's no point
continuing until this works ...
wpad.your.domain.name resolves to the host above where
you have a functioning auto config script running. You should
now be able to use http://wpad.your.domain.name/wpad.dat
as the Auto Config Script location in step 5 above.
Anybody using these steps to install and test, please feel free to make notes, corrections or additions for improvements, and post back to the squid list...
There are probably many more tricks and tips which hopefully will be detailed here in the future. Things like wpad.dat files being served from the proxy server themselves, maybe with a round robin dns setup for the WPAD host.
You can also use DHCP to configure browsers for WPAD. This technique allows you to set any URL as the PAC URL. For ISC DHCPD, enter a line like this in your dhcpd.conf file:
option wpad code 252 = text; option wpad "http://www.example.com/proxy.pac";
Replace the hostname with the name or address of your own server.
Ilja Pavkovic notes that the DHCP mode does not work reliably with every version of Internet Explorer. The DNS name method to find wpad.dat is more reliable.
Another user adds that IE 6.01 seems to strip the last character from the URL. By adding a trailing newline, he is able to make it work with both IE 5.0 and 6.0:<
option wpad "http://www.example.com/proxy.pac\n";
There was a bug in the 5.0x releases of Internet Explorer in which IE cropped any trailing slash off an FTP URL. The URL showed up correctly in the browser's ``Address:'' field, however squid logs show that the trailing slash was being taken off.
An example of where this impacted squid if you had a setup where squid would go direct for FTP directory listings but forward a request to a parent for FTP file transfers. This was useful if your upstream proxy was an older version of Squid or another vendors software which displayed directory listings with broken icons and you wanted your own local version of squid to generate proper FTP directory listings instead. The workaround for this is to add a double slash to any directory listing in which the slash was important, or else upgrade to IE 5.5. (Or use Netscape)
When using authentication with Internet Explorer 6 SP1, you may encounter issues when you first launch Internet Explorer. The problem will show itself when you first authenticate, you will receive a "Page Cannot Be Displayed" error. However, if you click refresh, the page will be correctly displayed.
This only happens immediately after you authenticate.
This is not a Squid error or bug. Microsoft broke the Basic Authentication when they put out IE6 SP1.
There is a knowledgebase article ( KB 331906) regarding this issue, which contains a link to a downloadable "hot fix." They do warn that this code is not "regression tested" but so far there have not been any reports of this breaking anything else. The problematic file is wininet.dll. Please note that this hotfix is included in the latest security update.
Lloyd Parkes notes that the article references another article, KB 312176. He says that you must not have the registry entry that KB 312176 encourages users to add to their registry.
According to Joao Coutinho, this simple solution also corrects the problem:
Another possible workaround to these problems is to make the ERR_CACHE_ACCESS_DENIED larger than 1460 bytes. This should trigger IE to handle the authentication in a slightly different manner.
The logs are a valuable source of information about Squid workloads and performance. The logs record not only access information, but also system configuration errors and resource consumption (eg, memory, disk space). There are several log file maintained by Squid. Some have to be explicitely activated during compile time, others can safely be deactivated during run-time.
There are a few basic points common to all log files. The time stamps logged into the log files are usually UTC seconds unless stated otherwise. The initial time stamp usually contains a millisecond extension.
If you run your Squid from the RunCache script, a file squid.out contains the Squid startup times, and also all fatal errors, e.g. as produced by an assert() failure. If you are not using RunCache, you will not see such a file.
The cache.log file contains the debug and error messages that Squid generates. If you start your Squid using the default RunCache script, or start it with the -s command line option, a copy of certain messages will go into your syslog facilities. It is a matter of personal preferences to use a separate file for the squid log data.
From the area of automatic log file analysis, the cache.log file does not have much to offer. You will usually look into this file for automated error reports, when programming Squid, testing new features, or searching for reasons of a perceived misbehaviour, etc.
The user agent log file is only maintained, if
From the user agent log file you are able to find out about distributation of browsers of your clients. Using this option in conjunction with a loaded production squid might not be the best of all ideas.
The store.log file covers the objects currently kept on disk or removed ones. As a kind of transaction log it is ususally used for debugging purposes. A definitive statement, whether an object resides on your disks is only possible after analysing the complete log file. The release (deletion) of an object may be logged at a later time than the swap out (save to disk).
The store.log file may be of interest to log file analysis which looks into the objects on your disks and the time they spend there, or how many times a hot object was accessed. The latter may be covered by another log file, too. With knowledge of the cache_dir configuration option, this log file allows for a URL to filename mapping without recursing your cache disks. However, the Squid developers recommend to treat store.log primarily as a debug file, and so should you, unless you know what you are doing.
The print format for a store log entry (one line) consists of eleven space-separated columns, compare with the storeLog() function in file src/store_log.c:
"%9d.%03d %-7s %02d %08X %4d %9d %9d %9d %s %d/%d %s %s\n"
The timestamp when the line was logged in UTC with a millisecond fraction.
The action the object was sumitted to, compare with src/store_log.c:
The cache_dir number this object was stored into, starting at 0 for your first cache_dir line.
The file number for the object storage file. Please note that the path to this file is calculated according to your cache_dir configuration.
A file number of FFFFFFFF denominates "memory only" objects. Any action code for such a file number refers to an object which existed only in memory, not on disk. For instance, if a RELEASE code was logged with file number FFFFFFFF, the object existed only in memory, and was released from memory.
The HTTP reply status code.
The value of the HTTP "Date: " reply header.
The value of the HTTP "Last-Modified: " reply header.
The value of the HTTP "Expires: " reply header.
The HTTP "Content-Type" major value, or "unknown" if it cannot be determined.
This column consists of two slash separated fields:
If the advertised (or expected) length is missing, it will be set to zero. If the advertised length is not zero, but not equal to the real length, the object will be realeased from the cache.
The request method for the object, e.g. GET.
The key to the object, usually the URL.
The timestamp format for the columns Date to Expires are all expressed in UTC seconds. The actual values are parsed from the HTTP reply headers. An unparsable header is represented by a value of -1, and a missing header is represented by a value of -2.
The column key usually contains just the URL of the object. Some objects though will never become public. Thus the key is said to include a unique integer number and the request method in addition to the URL.
This logfile exists for Squid-1.0 only. The format is
[date] URL peerstatus peerhost
Most log file analysis program are based on the entries in access.log. Currently, there are two file formats possible for the log file, depending on your configuration for the emulate_httpd_log option. By default, Squid will log in its native log file format. If the above option is enabled, Squid will log in the common log file format as defined by the CERN web daemon.
The common log file format contains other information than the native log file, and less. The native format contains more information for the admin interested in cache evaluation.
The Common Logfile Format is used by numerous HTTP servers. This format consists of the following seven fields:
remotehost rfc931 authuser [date] "method URL" status bytes
It is parsable by a variety of tools. The common format contains different information than the native log file format. The HTTP version is logged, which is not logged in native log file format.
The native format is different for different major versions of Squid. For Squid-1.0 it is:
time elapsed remotehost code/status/peerstatus bytes method URL
For Squid-1.1, the information from the hierarchy.log was moved into access.log. The format is:
time elapsed remotehost code/status bytes method URL rfc931 peerstatus/peerhost type
For Squid-2 the columns stay the same, though the content within may change a little.
The native log file format logs more and different information than the common log file format: the request duration, some timeout information, the next upstream server address, and the content type.
There exist tools, which convert one file format into the other. Please mind that even though the log formats share most information, both formats contain information which is not part of the other format, and thus this part of the information is lost when converting. Especially converting back and forth is not possible without loss.
squid2common.pl is a conversion utility, which converts any of the squid log file formats into the old CERN proxy style output. There exist tools to analyse, evaluate and graph results from that format.
It is recommended though to use Squid's native log format due to its greater amount of information made available for later analysis. The print format line for native access.log entries looks like this:
"%9d.%03d %6d %s %s/%03d %d %s %s %s %s%s/%s %s"
Therefore, an access.log entry usually consists of (at least) 10 columns separated by one ore more spaces:
A Unix timestamp as UTC seconds with a millisecond resolution. You can convert Unix timestamps into something more human readable using this short perl script:
#! /usr/bin/perl -p s/^\d+\.\d+/localtime $&/e;
The elapsed time considers how many milliseconds the transaction busied the cache. It differs in interpretation between TCP and UDP:
Please note that the entries are logged after the reply finished being sent, not during the lifetime of the transaction.
The IP address of the requesting instance, the client IP address. The client_netmask configuration option can distort the clients for data protection reasons, but it makes analysis more difficult. Often it is better to use one of the log file anonymizers.
Also, the log_fqdn configuration option may log the fully qualified domain name of the client instead of the dotted quad. The use of that option is discouraged due to its performance impact.
This column is made up of two entries separated by a slash. This column encodes the transaction result:
Several codes from older versions are no longer available, were renamed, or split. Especially the ERR_ codes do not seem to appear in the log file any more. Also refer to section Squid result codes for details on the codes no longer available in Squid-2.
The NOVM versions and Squid-2 also rely on the Unix buffer cache, thus you will see less TCP_MEM_HITs than with a Squid-1. Basically, the NOVM feature relies on read() to obtain an object, but due to the kernel buffer cache, no disk activity is needed. Only small objects (below 8KByte) are kept in Squid's part of main memory.
The size is the amount of data delivered to the client. Mind that this does not constitute the net object size, as headers are also counted. Also, failed requests may deliver an error page, the size of which is also logged here.
The request method to obtain an object. Please refer to section request-methods for available methods. If you turned off log_icp_queries in your configuration, you will not see (and thus unable to analyse) ICP exchanges. The PURGE method is only available, if you have an ACL for ``method purge'' enabled in your configuration file.
This column contains the URL requested. Please note that the log file may contain whitespaces for the URI. The default configuration for uri_whitespace denies whitespaces, though.
The eigth column may contain the ident lookups for the requesting client. Since ident lookups have performance impact, the default configuration turns ident_loookups off. If turned off, or no ident information is available, a ``-'' will be logged.
The hierarchy information consists of three items:
The content type of the object as seen in the HTTP reply header. Please note that ICP exchanges usually don't have any content type, and thus are logged ``-''. Also, some weird replies have content types ``:'' or even empty ones.
There may be two more columns in the access.log, if the (debug) option log_mime_headers is enabled In this case, the HTTP request headers are logged between a ``['' and a ``]'', and the HTTP reply headers are also logged between ``['' and ``]''. All control characters like CR and LF are URL-escaped, but spaces are not escaped! Parsers should watch out for this.
The TCP_ codes refer to requests on the HTTP port (usually 3128). The UDP_ codes refer to requests on the ICP port (usually 3130). If ICP logging was disabled using the log_icp_queries option, no ICP replies will be logged.
The following result codes were taken from a Squid-2, compare with the log_tags struct in src/access_log.c:
A valid copy of the requested object was in the cache.
The requested object was not in the cache.
The requested object was cached but STALE. The IMS query for the object resulted in "304 not modified".
The requested object was cached but STALE. The IMS query failed and the stale object was delivered.
The requested object was cached but STALE. The IMS query returned the new content.
The client issued a "no-cache" pragma, or some analogous cache control command along with the request. Thus, the cache has to refetch the object.
The client issued an IMS request for an object which was in the cache and fresh.
The object was believed to be in the cache, but could not be accessed.
Request for a negatively cached object, e.g. "404 not found", for which the cache believes to know that it is inaccessible. Also refer to the explainations for negative_ttl in your squid.conf file.
A valid copy of the requested object was in the cache and it was in memory, thus avoiding disk accesses.
Access was denied for this request.
The requested object was retrieved from the cache during offline mode. The offline mode never validates any object, see offline_mode in squid.conf file.
A valid copy of the requested object was in the cache.
The requested object is not in this cache.
Access was denied for this request.
An invalid request was received.
During "-Y" startup, or during frequent failures, a cache in hit only mode will return either UDP_HIT or this code. Neighbours will thus only fetch hits.
Seen with errors and cachemgr requests.
The following codes are no longer available in Squid-2:
Errors are now contained in the status code.
See: TCP_CLIENT_REFRESH_MISS.
See: TCP_SWAPFAIL_MISS.
Deleted, TCP_IMS_HIT used instead.
Hit objects are no longer available.
See: UDP_MISS_NOFETCH.
These are taken from RFC 2616 and verified for Squid. Squid-2 uses almost all codes except 307 (Temporary Redirect), 416 (Request Range Not Satisfiable), and 417 (Expectation Failed). Extra codes include 0 for a result code being unavailable, and 600 to signal an invalid header, a proxy error. Also, some definitions were added as for RFC 2518 (WebDAV). Yes, there are really two entries for status code 424, compare with http_status in src/enums.h:
000 Used mostly with UDP traffic. 100 Continue 101 Switching Protocols *102 Processing 200 OK 201 Created 202 Accepted 203 Non-Authoritative Information 204 No Content 205 Reset Content 206 Partial Content *207 Multi Status 300 Multiple Choices 301 Moved Permanently 302 Moved Temporarily 303 See Other 304 Not Modified 305 Use Proxy [307 Temporary Redirect] 400 Bad Request 401 Unauthorized 402 Payment Required 403 Forbidden 404 Not Found 405 Method Not Allowed 406 Not Acceptable 407 Proxy Authentication Required 408 Request Timeout 409 Conflict 410 Gone 411 Length Required 412 Precondition Failed 413 Request Entity Too Large 414 Request URI Too Large 415 Unsupported Media Type [416 Request Range Not Satisfiable] [417 Expectation Failed] *424 Locked *424 Failed Dependency *433 Unprocessable Entity 500 Internal Server Error 501 Not Implemented 502 Bad Gateway 503 Service Unavailable 504 Gateway Timeout 505 HTTP Version Not Supported *507 Insufficient Storage 600 Squid header parsing error
Squid recognizes several request methods as defined in RFC 2616. Newer versions of Squid (2.2.STABLE5 and above) also recognize RFC 2518 ``HTTP Extensions for Distributed Authoring -- WEBDAV'' extensions.
method defined cachabil. meaning --------- ---------- ---------- ------------------------------------------- GET HTTP/0.9 possibly object retrieval and simple searches. HEAD HTTP/1.0 possibly metadata retrieval. POST HTTP/1.0 CC or Exp. submit data (to a program). PUT HTTP/1.1 never upload data (e.g. to a file). DELETE HTTP/1.1 never remove resource (e.g. file). TRACE HTTP/1.1 never appl. layer trace of request route. OPTIONS HTTP/1.1 never request available comm. options. CONNECT HTTP/1.1r3 never tunnel SSL connection. ICP_QUERY Squid never used for ICP based exchanges. PURGE Squid never remove object from cache. PROPFIND rfc2518 ? retrieve properties of an object. PROPATCH rfc2518 ? change properties of an object. MKCOL rfc2518 never create a new collection. COPY rfc2518 never create a duplicate of src in dst. MOVE rfc2518 never atomically move src to dst. LOCK rfc2518 never lock an object against modifications. UNLOCK rfc2518 never unlock an object.
The following hierarchy codes are used with Squid-2:
For TCP HIT, TCP failures, cachemgr requests and all UDP requests, there is no hierarchy information.
The object was fetched from the origin server.
The object was fetched from a sibling cache which replied with UDP_HIT.
The object was requested from a parent cache which replied with UDP_HIT.
No ICP queries were sent. This parent was chosen because it was marked ``default'' in the config file.
The object was requested from the only parent appropriate for the given URL.
The object was fetched from the first parent in the list of parents.
The object was fetched from the origin server, because no parents existed for the given URL.
The object was fetched from the parent with the fastest (possibly weighted) round trip time.
This parent was chosen, because it included the the lowest RTT measurement to the origin server. See also the closests-only peer configuration option.
The parent selection was based on our own RTT measurements.
Our own RTT measurements returned a shorter time than any parent.
The object could not be requested because of a firewall configuration, see also never_direct and related material, and no parents were available.
The origin site was chosen, because the source ping arrived fastest.
No ICP replies were received from any parent. The parent was chosen, because it was marked for round robin in the config file and had the lowest usage count.
The peer was chosen, because the cache digest predicted a hit. This option was later replaced in order to distinguish between parents and siblings.
The parent was chosen, because the cache digest predicted a hit.
The sibling was chosen, because the cache digest predicted a hit.
This output seems to be unused?
The peer was selected by CARP.
part of src/peer_select.c:hier_strings[].
part of src/peer_select.c:hier_strings[].
Almost any of these may be preceded by 'TIMEOUT_' if the two-second (default) timeout occurs waiting for all ICP replies to arrive from neighbors, see also the icp_query_timeout configuration option.
The following hierarchy codes were removed from Squid-2:
code meaning -------------------- ------------------------------------------------- PARENT_UDP_HIT_OBJ hit objects are not longer available. SIBLING_UDP_HIT_OBJ hit objects are not longer available. SSL_PARENT_MISS SSL can now be handled by squid. FIREWALL_IP_DIRECT No special logging for hosts inside the firewall. LOCAL_IP_DIRECT No special logging for local networks.
This file has a rather unfortunate name. It also is often called the swap log. It is a record of every cache object written to disk. It is read when Squid starts up to ``reload'' the cache. If you remove this file when squid is NOT running, you will effectively wipe out your cache contents. If you remove this file while squid IS running, you can easily recreate it. The safest way is to simply shutdown the running process:
% squid -k shutdownThis will disrupt service, but at least you will have your swap log back. Alternatively, you can tell squid to rotate its log files. This also causes a clean swap log to be written.
% squid -k rotate
For Squid-1.1, there are six fields:
In Squid-2, the swap log file is now called swap.state. This is a binary file that includes MD5 checksums, and StoreEntry fields. Please see the Programmers Guide for information on the contents and format of that file.
If you remove swap.state while Squid is running, simply send Squid the signal to rotate its log files:
% squid -k rotateAlternatively, you can tell Squid to shutdown and it will rewrite this file before it exits.
If you remove the swap.state while Squid is not running, you will not lose your entire cache. In this case, Squid will scan all of the cache directories and read each swap file to rebuild the cache. This can take a very long time, so you'll have to be patient.
By default the swap.state file is stored in the top-level of each cache_dir. You can move the logs to a different location with the cache_swap_log option.
You should never delete access.log, store.log, cache.log, or swap.state while Squid is running. With Unix, you can delete a file when a process has the file opened. However, the filesystem space is not reclaimed until the process closes the file.
If you accidentally delete swap.state while Squid is running, you can recover it by following the instructions in the previous questions. If you delete the others while Squid is running, you can not recover them.
The correct way to maintain your log files is with Squid's ``rotate'' feature. You should rotate your log files at least once per day. The current log files are closed and then renamed with numeric extensions (.0, .1, etc). If you want to, you can write your own scripts to archive or remove the old log files. If not, Squid will only keep up to logfile_rotate versions of each log file. The logfile rotation procedure also writes a clean swap.state file, but it does not leave numbered versions of the old files.
If you set logfile_rotate to 0, Squid simply closes and then re-opens the logs. This allows third-party logfile management systems, such as newsyslog, to maintain the log files.
To rotate Squid's logs, simple use this command:
squid -k rotateFor example, use this cron entry to rotate the logs at midnight:
0 0 * * * /usr/local/squid/bin/squid -k rotate
For Squid 2.4:
To disable access.log:
cache_access_log /dev/null
To disable store.log:
cache_store_log none
To disable cache.log:
cache_log /dev/null
For Squid 2.5:
To disable access.log:
cache_access_log none
To disable store.log:
cache_store_log none
To disable cache.log:
cache_log /dev/null
Note : It is a bad idea to disable the cache.log because this file contains many important status and debugging messages. However, if you really want to, you can.
Warning : If /dev/null is specified to any of the above log files, logfile rotate must also be set to 0 or else risk Squid rotating away /dev/null making it a plain log file.
Tip : Instead of disabling the log files, it is advisable to use a smaller value for logfile_rotate and properly rotating Squid's log files in your cron. That way, your log files are more controllable and self-maintained by your system.
You need to rotate your log files with a cron job. For example:
0 0 * * * /usr/local/squid/bin/squid -k rotate
If you set logfile_rotate to 0, Squid simply closes and then re-opens the logs. This allows third-party logfile management systems, such as newsyslog, to maintain the log files.
The preferred log file for analysis is the access.log file in native format. For long term evaluations, the log file should be obtained at regular intervals. Squid offers an easy to use API for rotating log files, in order that they may be moved (or removed) without disturbing the cache operations in progress. The procedures were described above.
Depending on the disk space allocated for log file storage, it is recommended to set up a cron job which rotates the log files every 24, 12, or 8 hour. You will need to set your logfile_rotate to a sufficiently large number. During a time of some idleness, you can safely transfer the log files to your analysis host in one burst.
Before transport, the log files can be compressed during off-peak time. On the analysis host, the log file are concatinated into one file, so one file for 24 hours is the yield. Also note that with log_icp_queries enabled, you might have around 1 GB of uncompressed log information per day and busy cache. Look into you cache manager info page to make an educated guess on the size of your log files.
The EU project DESIRE developed some some basic rules to obey when handling and processing log files:
This message means that the requested object was in ``Delete Behind'' mode and the user aborted the transfer. An object will go into ``Delete Behind'' mode if
This means that a timeout occurred while the object was being transferred. Most likely the retrieval of this object was very slow (or it stalled before finishing) and the user aborted the request. However, depending on your settings for quick_abort, Squid may have continued to try retrieving the object. Squid imposes a maximum amount of time on all open sockets, so after some amount of time the stalled request was aborted and logged win an ERR_LIFETIME_EXP message.
I've been asked to retrieve an object which was accidentally destroyed at the source for recovery. So, how do I figure out where the things are so I can copy them out and strip off the headers?
The following method applies only to the Squid-1.1 versions:
Use grep to find the named object (Url) in the cache/log file. The first field in this file is an integer file number.
Then, find the file fileno-to-pathname.pl from the ``scripts'' directory of the Squid source distribution. The usage is
perl fileno-to-pathname.pl [-c squid.conf]file numbers are read on stdin, and pathnames are printed on stdout.
Sort of. You can use store.log to find out if a particular response was cached.
Cached responses are logged with the SWAPOUT tag. Uncached responses are logged with the RELEASE tag.
However, your analysis must also consider that when a cached response is removed from the cache (for example due to cache replacement) it is also logged in store.log with the RELEASE tag. To differentiate these two, you can look at the filenumber (3rd) field. When an uncachable response is released, the filenumber is FFFFFFFF (-1). Any other filenumber indicates a cached response was released.
The Squid distribution includes a CGI utility called cachemgr.cgi which can be used to view squid statistics with a web browser. This document has a section devoted to cachemgr.cgi usage which you should consult for more information.
sort -r -n +4 -5 access.log | awk '{print $5, $7}' | head -25
Note: The information here is current for version 2.2 and later.
First of all, you must stop Squid of course. You can use the command:
% squid -k shutdown
The fastest way to restart with an entirely clean cache is to over write the swap.state files for each cache_dir in your config file. Note, you can not just remove the swap.state file, or truncate it to zero size. Instead, you should put just one byte of garbage there. For example:
% echo "" > /cache1/swap.stateRepeat that for every cache_dir, then restart Squid. Be sure to leave the swap.state file with the same owner and permissions that it had before!
Another way, which takes longer, is to have squid recreate all the cache_dir directories. But first you must move the existing directories out of the way. For example, you can try this:
% cd /cache1 % mkdir JUNK % mv ?? swap.state* JUNK % rm -rf JUNK &Repeat this for your other cache_dir's, then tell Squid to create new directories:
% squid -z
by Rodney van den Oever, and James R Grinter
The RealPlayer (and RealPlayer Plus) manual states:
Use HTTP Only Select this option if you are behind a firewall and cannot receive data through TCP. All data will be streamed through HTTP. Note: You may not be able to receive some content if you select this option.
Again, from the documentation:
RealPlayer 4.0 identifies itself to the firewall when making a request for content to a RealServer. The following string is attached to any URL that the Player requests using HTTP GET: /SmpDsBhgRl Thus, to identify an HTTP GET request from the RealPlayer, look for: http://[^/]+/SmpDsBhgRl The Player can also be identified by the mime type in a POST to the RealServer. The RealPlayer POST has the following mime type: "application/x-pncmd"
Note that the first request is a POST, and the second has a '?' in the URL, so standard Squid configurations would treat it as non-cachable. It also looks rather ``magic.''
HTTP is an alternative delivery mechanism introduced with version 3 players, and it allows a reasonable approximation to ``streaming'' data - that is playing it as you receive it.
It isn't available in the general case: only if someone has made the realaudio file available via an HTTP server, or they're using a version 4 server, they've switched it on, and you're using a version 4 client. If someone has made the file available via their HTTP server, then it'll be cachable. Otherwise, it won't be (as far as we can tell.)
The more common RealAudio link connects via their own pnm: method and is transferred using their proprietary protocol (via TCP or UDP) and not using HTTP. It can't be cached nor proxied by Squid, and requires something such as the simple proxy that Progressive Networks themselves have made available, if you're in a firewall/no direct route situation. Their product does not cache (and I don't know of any software available that does.)
Some confusion arises because there is also a configuration option to use an
HTTP proxy (such as Squid) with the Realaudio/RealVideo players. This is
because the players can fetch the ``.ram'' file that contains the pnm:
reference for the audio/video stream. They fetch that .ram file from an HTTP
server, using HTTP.
Squid does not allow you to purge objects unless it is configured with access controls in squid.conf. First you must add something like
acl PURGE method PURGE acl localhost src 127.0.0.1 http_access allow PURGE localhost http_access deny PURGEThe above only allows purge requests which come from the local host and denies all other purge requests.
To purge an object, you can use the squidclient program:
squidclient -m PURGE http://www.miscreant.com/If the purge was successful, you will see a ``200 OK'' response:
HTTP/1.0 200 OK Date: Thu, 17 Jul 1997 16:03:32 GMT Server: Squid/1.1.14If the object was not found in the cache, you will see a ``404 Not Found'' response:
HTTP/1.0 404 Not Found Date: Thu, 17 Jul 1997 16:03:22 GMT Server: Squid/1.1.14
As of version 1.1.9, Squid is able to