Squid HTTPS interception and filtering without client certificates

I had a requirement to filter (all) web traffic on a few servers. This is typically easy with Squid and using it’s transparent proxy function. Where it gets difficult is filtering domains for HTTPS traffic.
I don’t want to SSL intercept the traffic, I don’t want to install CA certificates on the clients, I only want to filter the URLs based on a whitelist to which it can access. This is how it is done:

yum install squid
# I used squid 3.5.20

/usr/lib64/squid/ssl_crtd -c -s /var/lib/ssl_db
chown -R squid.squid /var/lib/ssl_db

mkdir /etc/squid/ssl_cert/
chown -R squid.squid /etc/squid/ssl_cert/
cd /etc/squid/ssl_cert
openssl req -new -newkey rsa:1024 -days 1365 -nodes -x509 -keyout myca.pem -out myca.pem

echo "www.google.com" > /etc/squid/whitelist
chmod 640 /etc/squid/whitelist
chown root:squid /etc/squid/whitelist


acl localnet src	# RFC1918 possible internal network
acl localnet src	# RFC1918 possible internal network
acl localnet src	# RFC1918 possible internal network
acl localnet src	# RFC1918 possible internal network
acl localnet src fc00::/7       # RFC 4193 local private network range
acl localnet src fe80::/10      # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http

http_access deny !Safe_ports

http_access deny CONNECT !SSL_ports

http_access allow localhost manager
http_access deny manager

acl step1 at_step SslBump1
acl whitelist_ssl ssl::server_name "/etc/squid/whitelist"
acl whitelist dstdomain "/etc/squid/whitelist"
acl port_80 port 80
acl http proto http

ssl_bump peek step1
ssl_bump splice whitelist_ssl
ssl_bump terminate all !whitelist_ssl

http_access deny http port_80 localnet !whitelist
http_access allow localnet
http_access deny all

https_port 3127 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/myca.pem key=/etc/squid/ssl_cert/myca.pem
http_port 3128 transparent

coredump_dir /var/spool/squid

refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320

# Test it with:

iptables -m owner --uid-owner cm -t nat -A OUTPUT -p tcp --dport 80 -j DNAT --to
iptables -m owner --uid-owner cm -t nat -A OUTPUT -p tcp --dport 443 -j DNAT --to

# Closing notes and thoughts

Around this section here:
http_access deny http port_80 localnet !whitelist
http_access allow localnet
http_access deny all

It looks a bit funny because we ‘allow localnet’ which typically allows our clients open access. However assessing:

ssl_bump terminate all !whitelist_ssl
http_access deny http port_80 localnet !whitelist

rules first, you see that we filter out all sites other than the whitelist with an explicit ‘deny’ or ssl ‘terminate’.

Also trying to use a proxy-aware application with the above configuration will not work because the proxy is configured in transparent / intercept mode ONLY. This is likely due to not having a normal http_port directive, this is good for me as it’s minimizing the abuse avenues.

Also for a final, final step, you need to configure your edge (or local) firewall to do destination NAT’ing back to the two Squid ports.

Block network traffic based on UID / User and GID / Group

I just found out that you can apply different IPTables rules based on UID and GID.

Just check that your kernel / iptables supports the module:

iptables -m owner --help

Which should output near the bottom like:

owner match options:
[!] --uid-owner userid[-userid]      Match local UID
[!] --gid-owner groupid[-groupid]    Match local GID
[!] --socket-exists                  Match if socket exists

Then make a rule as required. Eg. User ‘cm’ gets their web traffic transparently proxied via Squid.

iptables -m owner --uid-owner cm -t nat -A OUTPUT -i eth0 -p tcp --dport 80 -j DNAT --to

Pretty cool!

Fast development of Grok / Logstash extractions and fields

I had the fun times of trying to write grok rules in a particular way along with a complicated pipeline. I got tried of pushing the rules and restarting logstash, there had to be a better way!

This is want I ended up doing on my development system:

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.1.rpm
yum localinstall logstash-6.3.1.rpm 

Create your pipeline in: /etc/logstash/conf.d/

Create the following example files:


2018-07-16T01:53:28.716258+00:00 acme-host1 sshd[12522]: Disconnected from port 37972


input {
    file {
	path => [ "/tmp/input.txt" ]
	start_position => beginning
	type => "test"
	add_field => { "sourcetype" => "test" }
	sincedb_path => "/dev/null"


filter {
    if [type] == "test" {
        grok {
            match => { "message" => "%{TIMESTAMP_ISO8601} %{SYSLOGHOST:logsource} %{SYSLOGPROG}?: %{GREEDYDATA:message}" }
            overwrite => [ "message" ]
            add_tag => [ "p25vls" ]
        date {
            locale => "en"
            match => [ "timestamp", "MMM dd HH:mm:ss", "MMM  d HH:mm:ss"  ]
            timezone => "UTC"


output {
    stdout { codec => rubydebug }


/usr/share/logstash/bin/logstash -r -f /etc/logstash/conf.d/

Give it a minute, because well Java

Now in a second window, modify you pipeline (or file 25-filter.conf etc), save it.

You should see Logstash reprocess the data from ‘/tmp/input.txt’

Happy iterational development :-)