List comparison and list manipulation in Ansible

I keep saying time and time again that Ansible is not a programming language, it’s similar to one, it can do some programming things but ultimately it’s messy and I hate it BUT I can make it do some strange things.
List manipulation being one of those.

In this example I have two directories that I want to compare, directory one (/tmp/1) and directory two (/tmp/2). Directory one is the Source, that I want directory two to look like.

The use case is I want to sync /tmp/1 to /tmp/2 but you only want to remove the files in that are no longer /tmp/1, then you can sync (copy/template) the /tmp/1 directory knowing that nothing exists /tmp/2 that shouldn’t be there.

The ansible code is this with debug statements:

- hosts: local
  become: false
  tasks:

    - name: find 1
      find: path=/tmp/1
      register: one
    - debug: msg="{{ one }}"

    - name: find 2
      find: path=/tmp/2
      register: two

    - debug: msg="{{ item.path }}"
      with_items:
        - "{{ two.files }}"

    - set_fact:
        one_list: []
        two_list: []
        new_list: []

    - name: append
      set_fact: one_list="{{ one_list }} + [ '{{ item.path | basename }}' ]"
      with_items:
        - "{{ one.files }}"

    - name: append
      set_fact: two_list="{{ two_list }} + [ '{{ item.path | basename }}' ]"
      with_items:
        - "{{ two.files }}"

    - debug: msg="{{ one_list }}"
    - debug: msg="{{ two_list }}"

    - set_fact: new_list="{{ two_list | difference(one_list) }}"
    - debug: msg="{{ new_list }}"

The final result is new_list is a list (array) that contains what needs to be removed from /tmp/2 to bring it in line with /tmp/1

Docker and IPtables Firewall Merger

The problem: Modifying firewall rules on a host that runs Docker or Rancher (cattle) causes the docker-bridges and rancher NAT rules to be blown away, causing all your containers networking to break.

The solution: Modify /etc/sysconfig/iptables as normal and instead of running iptables-restore /etc/sysconfig/iptables run as root: dockerFirewallMerge.py

I’d appreciate some constructive feedback! https://github.com/c … /DockerFirewallMerge

Using OpenSCAP to scan and harden your servers

Determine which profile you want to use: oscap info /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml replacing the –profile line as required.

Perform a scan:

  oscap xccdf eval --report report.html 
      --profile xccdf_org.ssgproject.content_profile_CS2 
       /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml 

Apply a remediation:

  oscap xccdf eval --remediate --report report.html 
      --profile xccdf_org.ssgproject.content_profile_CS2 
       /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml 

Backup Windows Server with TrueCrypt / VeraCrypt

TrueCrypt is considered dead these days, but back when it was trusted this is a script I wrote to backup Windows Server (2008 and above) with TrueCrypt. It uses a loopback VHD (loopback file/drive) on a removable USB harddrive that it RAW formatted with TrueCrypt. That TrueCrypt volume then contains a large VHD file to the size of your backup volume.

How to configure it:

  1. Save the bat file on your server
  2. Format a (USB?) drive as a RAW TrueCrypt volume
  3. Mount the TrueCrypt partition
  4. Create a VHD volume with the filename: z:Backups.vhd within the TrueCrypt volume
  5. Mount the VHD volume as Z Drive
  6. Configure Windows Server Backup to use the Z Drive as a backup destination
  7. Unmount the VHD Volume
  8. Unmount the TrueCrypt volume
  9. Create the directories: C:backupscripts

How to use it:
With the above completed:

  1. Configure a scheduled task to run 10 minutes before your nightly backup job to run the backup script (Mount-truecrypt.bat)
  2. Eg. If your backup is to run at 11pm, configure the script to run at 10:50pm.
  3. Then after your backup finishes execute another scheduled task to UNmount the VHD and TrueCrypt volume (UNmount-truecrypt.bat)

Your done!

P.S. I’m not actually sure if this works with VeraCrypt but I think they have the same command line flags…. :-P

:: Truecrypt backup script written by Campbell McKenzie - www.cammckenzie.com 

:: =================================
:: ==  START Mount-truecrypt.bat  ==
:: =================================

:: "Auto" mount the RAW TrueCrypt disk as drive Z: password 1234

eventcreate /L Application /T INFORMATION /SO Backup /ID 666 /D "Attempting to mount backup disks..."

CD C:\Program Files\TrueCrypt
TrueCrypt.exe /auto devices /q /lZ /p 1234
if '%errorlevel%' EQU '0' (
    eventcreate /L Application /T SUCCESS /SO Backup /ID 666 /D "RAW Disk mounted on Z:\ "
    goto MountVHD
) else ( goto ERROR-TC )

:MountVHD
:: Perform directory listing for Truecrypt Bugs
dir z:\ > nul
:: Create the scriptlet
cd C:\backup\scripts

echo sel vdisk file="Z:\Backups.vhd" >mount.diskpart
echo attach vdisk >> mount.diskpart
echo select partition 1 >> mount.diskpart
echo assign letter=X >> mount.diskpart

:: Run the cmdlet
diskpart /s mount.diskpart
if '%errorlevel%' EQU '0' (
    eventcreate /L Application /T SUCCESS /SO Backup /ID 666 /D "Loopback VHD Disk mounted on X:\ - Mount Completed"
    goto TidyUp
) else ( goto ERROR-DP )

:TidyUp 
del /q mount.diskpart
EXIT 0 

:ERROR-TC
eventcreate /L Application /T ERROR /SO Backup /ID 666 /D "TrueCrypt Mount Failed..."
EXIT 1

:ERROR-DP
eventcreate /L Application /T ERROR /SO Backup /ID 666 /D "Loopback VHD Disk mount Failed..."
EXIT 1

:: REF: http://nicj.net/mounting-vhds-in-windows-7-from-a-command-line-script/
:: ===============================
:: ==  END Mount-truecrypt.bat  ==
:: ===============================
:: ==================================
:: ==  START UNmount-truecrypt.bat ==
:: ==================================

:: UnmountVHD.cmd
eventcreate /L Application /T INFORMATION /SO Backup /ID 667 /D "Attempting to unmount backup disks..."

cd C:\backup\scripts

echo sel vdisk file="z:\Backups.vhd" >unmount.diskpart
echo detach vdisk >>unmount.diskpart
:: Run the cmdlet
diskpart /s unmount.diskpart
if '%errorlevel%' EQU '0' (
    eventcreate /L Application /T SUCCESS /SO Backup /ID 667 /D "Loopback VHD Disk unmounted successfully..."
    goto UnmountTrueCrypt
) else ( goto ERROR-DP )

:: Unmount the RAW disk
:UnmountTrueCrypt
"C:\Program Files\TrueCrypt\TrueCrypt.exe" /d /q /s
if '%errorlevel%' EQU '0' (
    eventcreate /L Application /T SUCCESS /SO Backup /667 /D "RAW Disk unmounted successfully - Unmount Completed"
    goto TidyUp
) else ( goto ERROR-TC )

:TidyUp 
del /q unmount.diskpart
EXIT 0

:ERROR-TC
eventcreate /L Application /T ERROR /SO Backup /ID 666 /D "TrueCrypt Unmount Failed..."
EXIT 1

:ERROR-DP
eventcreate /L Application /T ERROR /SO Backup /ID 666 /D "Loopback VHD Disk Unmount Failed..."
EXIT 1

:: ==================================
:: ==  END UNmount-truecrypt.bat ==
:: ==================================

Splunk - run script once a week across mulitple servers

Using Splunk is great! It makes my IT life so much easier but occasionaly there is a use case to only run something once a week. While Splunk will allow this, it won’t allow you to ‘distribute’ running of the script across 7 days. For example you manage over 1000 servers and you require that the script is run by approx 1/7th of the servers each day, how do you do this easily without creating different server classes or whatever…

The solution is my python header script. Basically what is does is assign each server a number between 0 and 99, then splits up each day of the week into multiples of 14, eg Monday is 0-13 (inclusive), then checks if it’s own ‘number’ matches today’s numbers and runs the script if true.

#!/bin/env python
## Needed for weekday selection
import zlib
import datetime
import socket

## Weekday selection 
now = datetime.datetime.now()
today = now.weekday()
weekdayChooser = [0,14,28,42,56,70,84,100]

hostname = socket.gethostname()
hash = str(zlib.crc32(hostname))
# grab it from the back because sometimes negative values are given
dayToRun = int(hash[-2:])

## And finally check if its our day to run and runCode if it is...
if dayToRun in range(weekdayChooser[today],weekdayChooser[today +1]):
    runCode()