Date: Sun, 29 Sep 2013 14:04:55 GMT From: "Regis A. Despres" <regis.despres@gmail.com> To: FreeBSD-gnats-submit@freebsd.org Subject: ports/182490: Endless loop using discogrok Message-ID: <201309291404.r8TE4tZa033367@fbsd-test.home.in.indolore.net> Resent-Message-ID: <201309291720.r8THK0P0088221@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 182490
>Category: ports
>Synopsis: Endless loop using discogrok
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-ports-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Sun Sep 29 17:20:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator: Regis A. Despres
>Release: FreeBSD 9.1-RELEASE amd64
>Organization:
>Environment:
System:
FreeBSD fbsd-test 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
>Description:
Unable to use discogrok command due to endless loop in program
>How-To-Repeat:
The command below never ends.
With --verbose it shows a loop loadind patterns
head -n1 /var/log/messages | discogrok --verbose --patterns /usr/local/share/grok/patterns/base
[33329] [patterns] [grok_patterns_import_from_file:58] Importing pattern file: '/usr/local/share/grok/patterns/base'
[33329] [patterns] [grok_patterns_import_from_string:98] Importing patterns from string
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'USERNAME' => '[a-zA-Z0-9_-]+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'USER' => '%{USERNAME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'INT' => '(?:[+-]?(?:[0-9]+))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE10NUM' => '(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'NUMBER' => '(?:%{BASE10NUM})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE16NUM' => '(?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE16FLOAT' => '\b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'POSINT' => '\b(?:[0-9]+)\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WORD' => '\b\w+\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'NOTSPACE' => '\S+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATA' => '.*?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'GREEDYDATA' => '.*'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'QUOTEDSTRING' => '(?:(?<!\\)(?:"(?:\\.|[^\\"])*"|(?:'(?:\\.|[^\\'])*')|(?:`(?:\\.|[^\\`])*`)))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MAC' => '(?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'CISCOMAC' => '(?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WINDOWSMAC' => '(?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'COMMONMAC' => '(?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'IP' => '(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOSTNAME' => '\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOST' => '%{HOSTNAME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'IPORHOST' => '(?:%{HOSTNAME}|%{IP})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOSTPORT' => '(?:%{IPORHOST=~/\./}:%{POSINT})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'PATH' => '(?:%{UNIXPATH}|%{WINPATH})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'UNIXPATH' => '(?<![\w\\/])(?:/(?:[\w_%!$@:.,-]+|\\.)*)+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'LINUXTTY' => '(?:/dev/pts/%{POSINT})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BSDTTY' => '(?:/dev/tty[pq][a-z0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TTY' => '(?:%{BSDTTY}|%{LINUXTTY})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WINPATH' => '(?:[A-Za-z]+:|\\)(?:\\[^\\?*]*)+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPROTO' => '[A-Za-z]+(\+[A-Za-z+]+)?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIHOST' => '%{IPORHOST}(?::%{POSINT:port})?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPATH' => '(?:/[A-Za-z0-9$.+!*'(),~:#%_-]*)+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPARAM' => '\?[A-Za-z0-9$.+!*'(),~#%&/=:;_-]*'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPATHPARAM' => '%{URIPATH}(?:%{URIPARAM})?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URI' => '%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MONTH' => '\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MONTHNUM' => '(?:0?[1-9]|1[0-2])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MONTHDAY' => '(?:3[01]|[1-2]?[0-9]|0?[1-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DAY' => '(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'YEAR' => '[0-9]+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOUR' => '(?:2[0123]|[01][0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MINUTE' => '(?:[0-5][0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SECOND' => '(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TIME' => '(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATE_US' => '%{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATE_EU' => '%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'ISO8601_TIMEZONE' => '(?:Z|[+-]%{HOUR}(?::?%{MINUTE}))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'ISO8601_SECOND' => '(?:%{SECOND}|60)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TIMESTAMP_ISO8601' => '%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATE' => '%{DATE_US}|%{DATE_EU}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATESTAMP' => '%{DATE}[- ]%{TIME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TZ' => '(?:[PMCE][SD]T)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATESTAMP_RFC822' => '%{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATESTAMP_OTHER' => '%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGTIMESTAMP' => '%{MONTH} +%{MONTHDAY} %{TIME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'PROG' => '(?:[\w._/-]+)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGPROG' => '%{PROG:program}(?:\[%{POSINT:pid}\])?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGHOST' => '%{IPORHOST}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGFACILITY' => '<%{POSINT:facility}.%{POSINT:priority}>'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HTTPDATE' => '%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT:ZONE}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'QS' => '%{QUOTEDSTRING}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGBASE' => '%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'COMBINEDAPACHELOG' => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}'
[33329] [patterns] [grok_patterns_import_from_file:58] Importing pattern file: '/usr/local/share/grok/patterns/base'
[33329] [patterns] [grok_patterns_import_from_string:98] Importing patterns from string
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'USERNAME' => '[a-zA-Z0-9_-]+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'USER' => '%{USERNAME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'INT' => '(?:[+-]?(?:[0-9]+))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE10NUM' => '(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'NUMBER' => '(?:%{BASE10NUM})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE16NUM' => '(?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE16FLOAT' => '\b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'POSINT' => '\b(?:[0-9]+)\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WORD' => '\b\w+\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'NOTSPACE' => '\S+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATA' => '.*?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'GREEDYDATA' => '.*'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'QUOTEDSTRING' => '(?:(?<!\\)(?:"(?:\\.|[^\\"])*"|(?:'(?:\\.|[^\\'])*')|(?:`(?:\\.|[^\\`])*`)))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MAC' => '(?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'CISCOMAC' => '(?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WINDOWSMAC' => '(?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'COMMONMAC' => '(?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'IP' => '(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOSTNAME' => '\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOST' => '%{HOSTNAME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'IPORHOST' => '(?:%{HOSTNAME}|%{IP})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOSTPORT' => '(?:%{IPORHOST=~/\./}:%{POSINT})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'PATH' => '(?:%{UNIXPATH}|%{WINPATH})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'UNIXPATH' => '(?<![\w\\/])(?:/(?:[\w_%!$@:.,-]+|\\.)*)+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'LINUXTTY' => '(?:/dev/pts/%{POSINT})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BSDTTY' => '(?:/dev/tty[pq][a-z0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TTY' => '(?:%{BSDTTY}|%{LINUXTTY})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WINPATH' => '(?:[A-Za-z]+:|\\)(?:\\[^\\?*]*)+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPROTO' => '[A-Za-z]+(\+[A-Za-z+]+)?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIHOST' => '%{IPORHOST}(?::%{POSINT:port})?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPATH' => '(?:/[A-Za-z0-9$.+!*'(),~:#%_-]*)+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPARAM' => '\?[A-Za-z0-9$.+!*'(),~#%&/=:;_-]*'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPATHPARAM' => '%{URIPATH}(?:%{URIPARAM})?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URI' => '%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MONTH' => '\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MONTHNUM' => '(?:0?[1-9]|1[0-2])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MONTHDAY' => '(?:3[01]|[1-2]?[0-9]|0?[1-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DAY' => '(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'YEAR' => '[0-9]+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOUR' => '(?:2[0123]|[01][0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MINUTE' => '(?:[0-5][0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SECOND' => '(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TIME' => '(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATE_US' => '%{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATE_EU' => '%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'ISO8601_TIMEZONE' => '(?:Z|[+-]%{HOUR}(?::?%{MINUTE}))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'ISO8601_SECOND' => '(?:%{SECOND}|60)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TIMESTAMP_ISO8601' => '%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATE' => '%{DATE_US}|%{DATE_EU}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATESTAMP' => '%{DATE}[- ]%{TIME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TZ' => '(?:[PMCE][SD]T)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATESTAMP_RFC822' => '%{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATESTAMP_OTHER' => '%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGTIMESTAMP' => '%{MONTH} +%{MONTHDAY} %{TIME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'PROG' => '(?:[\w._/-]+)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGPROG' => '%{PROG:program}(?:\[%{POSINT:pid}\])?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGHOST' => '%{IPORHOST}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGFACILITY' => '<%{POSINT:facility}.%{POSINT:priority}>'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HTTPDATE' => '%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT:ZONE}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'QS' => '%{QUOTEDSTRING}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGBASE' => '%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'COMBINEDAPACHELOG' => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}'
[33329] [patterns] [grok_patterns_import_from_file:58] Importing pattern file: '/usr/local/share/grok/patterns/base'
[33329] [patterns] [grok_patterns_import_from_string:98] Importing patterns from string
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'USERNAME' => '[a-zA-Z0-9_-]+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'USER' => '%{USERNAME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'INT' => '(?:[+-]?(?:[0-9]+))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE10NUM' => '(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'NUMBER' => '(?:%{BASE10NUM})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE16NUM' => '(?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE16FLOAT' => '\b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'POSINT' => '\b(?:[0-9]+)\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WORD' => '\b\w+\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'NOTSPACE' => '\S+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATA' => '.*?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'GREEDYDATA' => '.*'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'QUOTEDSTRING' => '(?:(?<!\\)(?:"(?:\\.|[^\\"])*"|(?:'(?:\\.|[^\\'])*')|(?:`(?:\\.|[^\\`])*`)))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MAC' => '(?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'CISCOMAC' => '(?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WINDOWSMAC' => '(?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'COMMONMAC' => '(?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'IP' => '(?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOSTNAME' => '\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOST' => '%{HOSTNAME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'IPORHOST' => '(?:%{HOSTNAME}|%{IP})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOSTPORT' => '(?:%{IPORHOST=~/\./}:%{POSINT})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'PATH' => '(?:%{UNIXPATH}|%{WINPATH})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'UNIXPATH' => '(?<![\w\\/])(?:/(?:[\w_%!$@:.,-]+|\\.)*)+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'LINUXTTY' => '(?:/dev/pts/%{POSINT})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BSDTTY' => '(?:/dev/tty[pq][a-z0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TTY' => '(?:%{BSDTTY}|%{LINUXTTY})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WINPATH' => '(?:[A-Za-z]+:|\\)(?:\\[^\\?*]*)+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPROTO' => '[A-Za-z]+(\+[A-Za-z+]+)?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIHOST' => '%{IPORHOST}(?::%{POSINT:port})?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPATH' => '(?:/[A-Za-z0-9$.+!*'(),~:#%_-]*)+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPARAM' => '\?[A-Za-z0-9$.+!*'(),~#%&/=:;_-]*'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URIPATHPARAM' => '%{URIPATH}(?:%{URIPARAM})?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'URI' => '%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MONTH' => '\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MONTHNUM' => '(?:0?[1-9]|1[0-2])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MONTHDAY' => '(?:3[01]|[1-2]?[0-9]|0?[1-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DAY' => '(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'YEAR' => '[0-9]+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HOUR' => '(?:2[0123]|[01][0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'MINUTE' => '(?:[0-5][0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SECOND' => '(?:(?:[0-5][0-9]|60)(?:[.,][0-9]+)?)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TIME' => '(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATE_US' => '%{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATE_EU' => '%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'ISO8601_TIMEZONE' => '(?:Z|[+-]%{HOUR}(?::?%{MINUTE}))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'ISO8601_SECOND' => '(?:%{SECOND}|60)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TIMESTAMP_ISO8601' => '%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATE' => '%{DATE_US}|%{DATE_EU}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATESTAMP' => '%{DATE}[- ]%{TIME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'TZ' => '(?:[PMCE][SD]T)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATESTAMP_RFC822' => '%{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATESTAMP_OTHER' => '%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGTIMESTAMP' => '%{MONTH} +%{MONTHDAY} %{TIME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'PROG' => '(?:[\w._/-]+)'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGPROG' => '%{PROG:program}(?:\[%{POSINT:pid}\])?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGHOST' => '%{IPORHOST}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGFACILITY' => '<%{POSINT:facility}.%{POSINT:priority}>'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'HTTPDATE' => '%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT:ZONE}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'QS' => '%{QUOTEDSTRING}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'SYSLOGBASE' => '%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'COMBINEDAPACHELOG' => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}'
[33329] [patterns] [grok_patterns_import_from_file:58] Importing pattern file: '/usr/local/share/grok/patterns/base'
[33329] [patterns] [grok_patterns_import_from_string:98] Importing patterns from string
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'USERNAME' => '[a-zA-Z0-9_-]+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'USER' => '%{USERNAME}'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'INT' => '(?:[+-]?(?:[0-9]+))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE10NUM' => '(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'NUMBER' => '(?:%{BASE10NUM})'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE16NUM' => '(?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+))'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'BASE16FLOAT' => '\b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'POSINT' => '\b(?:[0-9]+)\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'WORD' => '\b\w+\b'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'NOTSPACE' => '\S+'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'DATA' => '.*?'
[33329] [patterns] [grok_pattern_add:29] Adding new pattern 'GREEDYDATA' => '.*'
[33329] [patterns] ^C
>Fix:
Modify some code (source : https://github.com/jordansissel/grok/commit/f1858bfa347bc79ad9aa6f3425edd3c19ffecf42)
Makefile :
@@ -1,6 +1,6 @@
PACKAGE=grok
-PLATFORM=$(shell (uname -o || uname -s) | tr -d "/" 2> /dev/null)
+PLATFORM=$(shell (uname -s || uname -o) | tr -d "/" 2> /dev/null)
FLEX?=flex
FORCE_FLEX?=0
discover_main.c :
@@ -33,7 +33,7 @@ int main(int argc, char **argv) {
grok_init(&grok);
int pattern_count = 0;
- while ((opt = getopt_long_only(argc, argv, "hp:v", options, &optind)) != -1) {
+ while ((opt = getopt_long_only(argc, argv, "hp:v", options, NULL)) != -1) {
switch (opt) {
case 'h':
usage();
grok_discover.c :
@@ -187,6 +187,9 @@ void grok_discover(const grok_discover_t *gdt, /*grok_t *dest_grok, */
if (first_match_endpos > 0) {
offset += first_match_endpos;
}
+ else {
+ offset += 1;
+ }
} else { /* We found a match, replace it in the pattern */
grok_log(gdt, LOG_DISCOVER, "%d: Matched %s on '%.*s'",
rounds, best_match.grok->pattern,
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201309291404.r8TE4tZa033367>