IT Dribble

For tricky squid ACL troubleshooting situations, it is helpful to be able to see which access control entries a request matches and does not match. This information can be discovered easily using squid’s debugging facility.

Step 1: RTFM

check debug sections avaliable: http://wiki.squid-cache.org/KnowledgeBase/DebugSections

In this case, we can see that squid’s ACLs are managed by section 28.

Step 2: Make squid more chatty

Given the ACL section, we can tell squid to log more information about ACL traversal. We feed him the section (28) and the log level (3, or similar) in squid.conf (near the top usually).

Code:

debug_options 28,3

… and we tell the daemon to re-read the configuration:

service squid reload

Step 3: Test and evaluate

Now check the logs:

tail -f /var/log/squid/cache.log

Note: realistically you probably don’t want to tail the logs, you are best to try your failing web site then open the log with ‘less’ etc and do a search for you website.

In this example my blocklist had downloaded some unfiltered characters and ended up with a zero (”0”) on a line by itself. (Why they ended up there is a different conversation)

2015/01/07 15:51:42.237| ACL::checklistMatches: checking 'zeus_block_list'
2015/01/07 15:51:42.237| aclRegexData::match: checking 'mt0.google.com'
2015/01/07 15:51:42.237| aclRegexData::match: looking for '24b5'
2015/01/07 15:51:42.237| aclRegexData::match: looking for '0'
2015/01/07 15:51:42.238| aclRegexData::match: match '0' found in 'mt0.google.com'
2015/01/07 15:51:42.238| ACL::ChecklistMatches: result for 'zeus_block_list' is 1

Removing the zero from the

zeus_block_list

and reloading squid resolved the issue.

Note that true evaluations are represented by 1, while false evaluations are represented by 0.

Step 4: Post-troubleshooting cleanup

It is important to disable the debug_options when you are finished troubleshooting. They produce a copious amount of logging, and they can generally be a (disk space) liability when you aren’t using them.

To reverse the changes, simply comment out the debug_options line above, and reload squid.

Thanks to FreeBSD forums for the walk-through

Via the webconsole, click: Event, then read all the events, while taking note of them and actioning them as required.
Finally click: Clear All Events

<?php
$everything = get_defined_vars();
ksort($everything);
echo '<pre>';
print_r($everything);
echo '</pre>';
?>

Well actually it’s not in the foreground but to a log file, but if you tail / follow the logfile you could pretend it was the foreground :-)

mysqld_safe --log-error=/var/log/mysql.err

And perhaps you have imported a database for forensic investigation and you don’t know the database password, you can just skip the authentication:

mysqld_safe --skip-grant-tables --log-error=/var/log/mysql.err

Using Splunk is great! It makes my IT life so much easier but occasionaly there is a use case to only run something once a week. While Splunk will allow this, it won’t allow you to ‘distribute’ running of the script across 7 days. For example you manage over 1000 servers and you require that the script is run by approx 1/7th of the servers each day, how do you do this easily without creating different server classes or whatever…

The solution is my python header script. Basically what is does is assign each server a number between 0 and 99, then splits up each day of the week into multiples of 14, eg Monday is 0-13 (inclusive), then checks if it’s own ‘number’ matches today’s numbers and runs the script if true.

#!/bin/env python
## Needed for weekday selection
import zlib
import datetime
import socket

## Weekday selection 
now = datetime.datetime.now()
today = now.weekday()
weekdayChooser = [0,14,28,42,56,70,84,100]

hostname = socket.gethostname()
hash = str(zlib.crc32(hostname))
# grab it from the back because sometimes negative values are given
dayToRun = int(hash[-2:])

## And finally check if its our day to run and runCode if it is...
if dayToRun in range(weekdayChooser[today],weekdayChooser[today +1]):
    runCode()