IT Dribble

I had the fun times of trying to write grok rules in a particular way along with a complicated pipeline. I got tried of pushing the rules and restarting logstash, there had to be a better way!

This is want I ended up doing on my development system:

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.1.rpm
yum localinstall logstash-6.3.1.rpm 

Create your pipeline in: /etc/logstash/conf.d/

Create the following example files:

/tmp/input.txt:

2018-07-16T01:53:28.716258+00:00 acme-host1 sshd[12522]: Disconnected from 8.8.8.8 port 37972

000-file-in.conf:

input {
    file {
	path => [ "/tmp/input.txt" ]
	start_position => beginning
	type => "test"
	add_field => { "sourcetype" => "test" }
	sincedb_path => "/dev/null"
    }
}

25-filter.conf:

filter {
    if [type] == "test" {
        grok {
            match => { "message" => "%{TIMESTAMP_ISO8601} %{SYSLOGHOST:logsource} %{SYSLOGPROG}?: %{GREEDYDATA:message}" }
            overwrite => [ "message" ]
            add_tag => [ "p25vls" ]
        }
    
        date {
            locale => "en"
            match => [ "timestamp", "MMM dd HH:mm:ss", "MMM  d HH:mm:ss"  ]
            timezone => "UTC"
        }
    }
}

999-output.conf:

output {
    stdout { codec => rubydebug }
}

Run:

/usr/share/logstash/bin/logstash -r -f /etc/logstash/conf.d/

Give it a minute, because well Java

Now in a second window, modify you pipeline (or file 25-filter.conf etc), save it.

You should see Logstash reprocess the data from ‘/tmp/input.txt’

Happy iterational development :-)