List comparison and list manipulation in Ansible
I keep saying time and time again that Ansible is not a programming language, it’s similar to one, it can do some programming things but ultimately it’s messy and I hate it BUT I can make it do some strange things.
List manipulation being one of those.
In this example I have two directories that I want to compare, directory one (/tmp/1) and directory two (/tmp/2). Directory one is the Source, that I want directory two to look like.
The use case is I want to sync /tmp/1 to /tmp/2 but you only want to remove the files in that are no longer /tmp/1, then you can sync (copy/template) the /tmp/1 directory knowing that nothing exists /tmp/2 that shouldn’t be there.
The ansible code is this with debug statements:
- hosts: local become: false tasks: - name: find 1 find: path=/tmp/1 register: one - debug: msg="{{ one }}" - name: find 2 find: path=/tmp/2 register: two - debug: msg="{{ item.path }}" with_items: - "{{ two.files }}" - set_fact: one_list: [] two_list: [] new_list: [] - name: append set_fact: one_list="{{ one_list }} + [ '{{ item.path | basename }}' ]" with_items: - "{{ one.files }}" - name: append set_fact: two_list="{{ two_list }} + [ '{{ item.path | basename }}' ]" with_items: - "{{ two.files }}" - debug: msg="{{ one_list }}" - debug: msg="{{ two_list }}" - set_fact: new_list="{{ two_list | difference(one_list) }}" - debug: msg="{{ new_list }}"
The final result is new_list is a list (array) that contains what needs to be removed from /tmp/2 to bring it in line with /tmp/1
Docker and IPtables Firewall Merger
The problem: Modifying firewall rules on a host that runs Docker or Rancher (cattle) causes the docker-bridges and rancher NAT rules to be blown away, causing all your containers networking to break.
The solution: Modify /etc/sysconfig/iptables as normal and instead of running iptables-restore /etc/sysconfig/iptables run as root: dockerFirewallMerge.py
I’d appreciate some constructive feedback! https://github.com/c … /DockerFirewallMerge
Using OpenSCAP to scan and harden your servers
Determine which profile you want to use: oscap info /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml replacing the –profile line as required.
Perform a scan:
oscap xccdf eval --report report.html --profile xccdf_org.ssgproject.content_profile_CS2 /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml
Apply a remediation:
oscap xccdf eval --remediate --report report.html --profile xccdf_org.ssgproject.content_profile_CS2 /usr/share/xml/scap/ssg/content/ssg-rhel7-ds.xml
Backup Windows Server with TrueCrypt / VeraCrypt
TrueCrypt is considered dead these days, but back when it was trusted this is a script I wrote to backup Windows Server (2008 and above) with TrueCrypt. It uses a loopback VHD (loopback file/drive) on a removable USB harddrive that it RAW formatted with TrueCrypt. That TrueCrypt volume then contains a large VHD file to the size of your backup volume.
How to configure it:
- Save the bat file on your server
- Format a (USB?) drive as a RAW TrueCrypt volume
- Mount the TrueCrypt partition
- Create a VHD volume with the filename: z:Backups.vhd within the TrueCrypt volume
- Mount the VHD volume as Z Drive
- Configure Windows Server Backup to use the Z Drive as a backup destination
- Unmount the VHD Volume
- Unmount the TrueCrypt volume
- Create the directories: C:backupscripts
How to use it:
With the above completed:
- Configure a scheduled task to run 10 minutes before your nightly backup job to run the backup script (Mount-truecrypt.bat)
- Eg. If your backup is to run at 11pm, configure the script to run at 10:50pm.
- Then after your backup finishes execute another scheduled task to UNmount the VHD and TrueCrypt volume (UNmount-truecrypt.bat)
Your done!
P.S. I’m not actually sure if this works with VeraCrypt but I think they have the same command line flags…. :-P
:: Truecrypt backup script written by Campbell McKenzie - www.cammckenzie.com :: ================================= :: == START Mount-truecrypt.bat == :: ================================= :: "Auto" mount the RAW TrueCrypt disk as drive Z: password 1234 eventcreate /L Application /T INFORMATION /SO Backup /ID 666 /D "Attempting to mount backup disks..." CD C:\Program Files\TrueCrypt TrueCrypt.exe /auto devices /q /lZ /p 1234 if '%errorlevel%' EQU '0' ( eventcreate /L Application /T SUCCESS /SO Backup /ID 666 /D "RAW Disk mounted on Z:\ " goto MountVHD ) else ( goto ERROR-TC ) :MountVHD :: Perform directory listing for Truecrypt Bugs dir z:\ > nul :: Create the scriptlet cd C:\backup\scripts echo sel vdisk file="Z:\Backups.vhd" >mount.diskpart echo attach vdisk >> mount.diskpart echo select partition 1 >> mount.diskpart echo assign letter=X >> mount.diskpart :: Run the cmdlet diskpart /s mount.diskpart if '%errorlevel%' EQU '0' ( eventcreate /L Application /T SUCCESS /SO Backup /ID 666 /D "Loopback VHD Disk mounted on X:\ - Mount Completed" goto TidyUp ) else ( goto ERROR-DP ) :TidyUp del /q mount.diskpart EXIT 0 :ERROR-TC eventcreate /L Application /T ERROR /SO Backup /ID 666 /D "TrueCrypt Mount Failed..." EXIT 1 :ERROR-DP eventcreate /L Application /T ERROR /SO Backup /ID 666 /D "Loopback VHD Disk mount Failed..." EXIT 1 :: REF: http://nicj.net/mounting-vhds-in-windows-7-from-a-command-line-script/ :: =============================== :: == END Mount-truecrypt.bat == :: ===============================
:: ================================== :: == START UNmount-truecrypt.bat == :: ================================== :: UnmountVHD.cmd eventcreate /L Application /T INFORMATION /SO Backup /ID 667 /D "Attempting to unmount backup disks..." cd C:\backup\scripts echo sel vdisk file="z:\Backups.vhd" >unmount.diskpart echo detach vdisk >>unmount.diskpart :: Run the cmdlet diskpart /s unmount.diskpart if '%errorlevel%' EQU '0' ( eventcreate /L Application /T SUCCESS /SO Backup /ID 667 /D "Loopback VHD Disk unmounted successfully..." goto UnmountTrueCrypt ) else ( goto ERROR-DP ) :: Unmount the RAW disk :UnmountTrueCrypt "C:\Program Files\TrueCrypt\TrueCrypt.exe" /d /q /s if '%errorlevel%' EQU '0' ( eventcreate /L Application /T SUCCESS /SO Backup /667 /D "RAW Disk unmounted successfully - Unmount Completed" goto TidyUp ) else ( goto ERROR-TC ) :TidyUp del /q unmount.diskpart EXIT 0 :ERROR-TC eventcreate /L Application /T ERROR /SO Backup /ID 666 /D "TrueCrypt Unmount Failed..." EXIT 1 :ERROR-DP eventcreate /L Application /T ERROR /SO Backup /ID 666 /D "Loopback VHD Disk Unmount Failed..." EXIT 1 :: ================================== :: == END UNmount-truecrypt.bat == :: ==================================
Splunk - run script once a week across mulitple servers
Using Splunk is great! It makes my IT life so much easier but occasionaly there is a use case to only run something once a week. While Splunk will allow this, it won’t allow you to ‘distribute’ running of the script across 7 days. For example you manage over 1000 servers and you require that the script is run by approx 1/7th of the servers each day, how do you do this easily without creating different server classes or whatever…
The solution is my python header script. Basically what is does is assign each server a number between 0 and 99, then splits up each day of the week into multiples of 14, eg Monday is 0-13 (inclusive), then checks if it’s own ‘number’ matches today’s numbers and runs the script if true.
#!/bin/env python ## Needed for weekday selection import zlib import datetime import socket ## Weekday selection now = datetime.datetime.now() today = now.weekday() weekdayChooser = [0,14,28,42,56,70,84,100] hostname = socket.gethostname() hash = str(zlib.crc32(hostname)) # grab it from the back because sometimes negative values are given dayToRun = int(hash[-2:]) ## And finally check if its our day to run and runCode if it is... if dayToRun in range(weekdayChooser[today],weekdayChooser[today +1]): runCode()