IT Dribble

In this day and age load balancers are pretty common, whether it’s a service like Incapsula, Amazon ELB or anything else that proxies web traffic. One of the annoyances of proxies is that source address gets rewritten and if you want to know where your visitors are coming from then having the original source IP address helps.
Within Apache or most proxies for that matter they insert headers and the header we would like to take advantage of is “X-Forwarder-For”.

Now you could rewrite the Apache configuration to use the X-Forwarder-For header in place of the originating server’s (load balancer) IP address, the benefit of this is that Splunk will be happy with the builtin extractions. The downside is you should never trust headers as they can be manipulated client side and forged.

Here are two example from the vhost configuration file:

<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName www.cammckenzie.com
        ErrorLog logs/ssl_error_log
        LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i""

The above example is from a default configuration

Now if you wanted to replace the load balancer’s IP address with the X-Forwarder-For IP addresses you “could” use:

<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName www.cammckenzie.com
        ErrorLog logs/ssl_error_log
        LogFormat "%{X-Forwarded-For}i %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i""

The above example dropped the load balancer’s IP address altogether and replaced it with the X-Forwarder-For address. This example is ultimately bad due to: dropping the true originating server altogether and now the Splunk inbuilt extractions will be broken if there is more that one address in the X-Forwarder-For list.

Perhaps what is a better idea is:

<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName www.cammckenzie.com
        ErrorLog logs/ssl_error_log
        LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" srcip="%{X-Forwarded-For}i""

This example is perhaps the best solution because you keep the original information available and intact, you haven’t broken the Splunk extractions and you have helped your Splunk install (a little bit) by creating a key value pair for “srcip”.

And just to show off and get really complicated you could perform a bit of filtering and if you know that all requests will be X-Forwarded-For and tired of your load balancer health checks spamming your logs you could perform something like the following:

<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName www.cammckenzie.com
        ErrorLog logs/ssl_error_log
        LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" ELBCheck
        LogFormat "%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i" srcip="%{X-Forwarded-For}i"" XForwarderFor
        SetEnvIf X-Forwarded-For "^.*..*..*..*" forwarded
        CustomLog logs/access_log XForwarderFor env=forwarded
        CustomLog logs/elb_health_check_access_log ELBCheck env=!forwarded

Now this example basically puts the Amazon’s elastic load balancer’s health check in one file and genuine requests in another!

Instead of just documenting the IPTables configuration file eg: /etc/sysconfig/iptables with comments (#’s) you can also input comments as part of the ruleset itself. So when you perform iptables -L -v -n you get the following output:

root@server070:[~]: iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  64M 4727M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
    5   474 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0
 202K   27M ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
   16   880 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
 137M   38G ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           udp dpt:514 /* Syslog traffic */
   28  1664 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:514 /* Syslog traffic */
41067 2050K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:9997 /* Universal Forwarder traffic */
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:8089 /* Splunk SSL traffic */
   47  2564 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:8000 /* Splunk web interface */
14135 1313K LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 5/min burst 5 LOG flags 0 level 7 prefix `iptables denied: '
 218K   21M REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

All that you need to do use the following example in your configuration file:

root@server070:[~]: cat /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p udp -m udp --dport 514 -m comment --comment "Syslog traffic" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 514 -m comment --comment "Syslog traffic" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 9997 -m comment --comment "Universal Forwarder traffic" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8089 -m comment --comment "Splunk SSL traffic" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8000 -m comment --comment "Splunk web interface" -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Happy commenting!

If you do a lot of scripting then this snippet maybe useful. I use this snippet to change the default password for Splunk Universal Forwarder installations, from the default to random. You don’t actually need to record the password for the forwarder - and it’s easy enough to reset if you do.

@echo off
set pass=
set s=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890
set m=0
:loop
set /a n=%random% %% 62
call set pass=%pass%%%s:~%n%,1%%
set /a m=m+1
if not %m%==32 goto loop:
echo %pass%

Finally someone has created an RPM for Fedora 23 so that I can use mutt-sidebar!

Details are here: https://copr.fedorap … /cam34/mutt-sidebar/

Monitoring our net streams for SMTP traffic, I have a particular Splunk alert configured to alert when a spam campaign in underway. It’s quite interesting to see what tricks spammers try and how long some campaigns last.

Something that has caught my attention is the increased use of unsolicited “invoice” and “order” emails which get sent with .doc documents attached. Reviewing the documents nearly all of them contain malicious VB scripting usually set to auto run, which downloads and tries to execute binaries.

Now I have created a number of spam assassin rules to block these types of emails (invoices with .doc files attached) but it’s a cat and mouse game, and you can’t be too generic as you may block legitimate email. What I can block ruthlessly is documents and spreadsheets which contain autorun VB scripting.

Here is how to do it:

Create the following file on your amavisd server at the location “/usr/local/bin/detectvba.pl”

#!/usr/bin/perl -w
# technion@lolware.net
# Detects vba macros containing blacklisted strings.
# https://github.com/technion/maia_mailguard/blob/master/scripts/detectvba.pl
# Mods by www.cammckenzie.com
#
# Suggested amavisd/maiad.conf config:
# ['Detect-VBA',
#   '/usr/local/bin/detectvba.pl', "{}",
#      [0], qr/INFECTED/, qr/\bINFECTED (.+)\b/m ],
#
use strict;

my $sigtool = '/usr/bin/sigtool'; #Clamav sigtool path

if ($#ARGV != 0) {
    print "Please supply directory to scan\n";
    exit 0;
}

#Sanity check directory
my $dir = $ARGV[0];
if ($dir !~ /^[a-z0-9A-Z\/-]+$/) {
    print "Invalid directory passed\n";
    exit 0;
}

opendir DIR, $dir or die "Cannot open dir $dir: $!";
my @files = readdir DIR;

foreach my $file (@files) {
    next if $file =~ /^\.$/;
    next if $file =~ /^\.\.$/;
    my $scan = `$sigtool --vba="$dir/$file"`;
    if ($scan =~ /autoopen/i ) {
        print "Scanning $file: INFECTED VBA\n";
        exit 1;
    } else {
        print "Scanning $file: OK\n";
    }
}

closedir DIR;

exit 0;

Then in amavisd.conf (/etc/amavisd/amavisd.conf on CentOS) modify the section “@av_scanners” and insert as a primary scanner the following stanza:

['Detect-VBA',
   '/usr/loca/bin/detectvba.pl', "{}",
      [0], qr/INFECTED/, qr/\bINFECTED (.+)\b/m ],

Then after that just restart your amavisd service and you should see, it finds Detect-VBA as a primary scanner, test it by sending yourself a malicious macro document lol.

Download the raw howto here, so that special characters are intact as FlatPress has a habit of removing them….https://www.cammcken … assin-or-amavisd.txt