Vista - PowerShell script to search text logs with many regular expressions

Hello:

Below is a PowerShell script which takes a text file containing regular
expression patterns (signatures.txt) to search a textual log file, or,
actually, to search a text file of any type, just so long as each entry to
be searched in the textual file/log does not span more than one line. Each
line from the log is compared against every regex in the patterns file. The
patterns file must contain a description of what each regex indicates if a
match to that pattern is found. When the search completes, a summary report
is shown with the count of matches to each pattern. The output is an array
of objects with three properties on which you can filter: Count,
Description, and the regex Pattern itself.


Sample use:
search-textlog.ps1 iis.log signatures.txt


Sample output when piped through format-table:

Count Description
----- -----------
74 Attempts to use a backslash in a request (folder traversal).
15 Attempts to access the /printers folder (reconnaissance).
13 Attempts to access CMD.EXE (command execution).
12 Attempts to access \Winnt or \Windows (command execution).
12 Attempts to access /scripts or /cgi-bin (reconnaissance).
10 Attempts to access the IIS Administration web site.
9 Attempts to access Certificate Server web pages (reconnaissance).
6 Attempts to send many repeating characters (buffer overflow).
6 Attempts to use %u Unicode encoding (IDS evasion).
5 Attempts by the Code Red Worm (buffer overflow).


Or if you use the -ShowMatchedLines switch with the script, the summary will
not be shown, but every line from the log which matched at least one pattern
will be outputted (and the line will be output only once, no matter how many
additional patterns it might have also matched).

Sample use:
search-textlog.ps1 iis.log signatures.txt -showmatchedlines

In real life you would have a different patterns file for each type of log
you wanted to search, e.g., syslog, web, ftp, smtp, firewall, etc. You will
look for different things in different types of files/logs, hence, you'll
have different regular expressions for each. The format of each line in the
patterns file must be "<regexpattern> tab(s) <description>". Blank lines
and lines which begin with hashmarks (#) or semicolons ( are ignored. If
you are going to use the -ShowMatchedLines switch a lot, you can optimize
your searches by putting the patterns which will match the most entries near
the top of the patterns file.

The full script is below and a couple sample files are attached to this
posting (iis.log and signatures.txt from the examples above), but if you
want to see some screenshots too (it's all in the public domain):

https://blogs.sans.org/windows-secur...r-expressions/

Cheers,
J.



######################################################################################
# Script: Search-TextLog.ps1
# Date: 27.May.2007
# Version: 1.0
# Author: Jason Fossen security
# Purpose: Will search every line of a textual log file against every regex
# pattern provided in a second file, producing a summary of matches
# found, or, if -ShowMatchedLines is specified, only the log lines
# which matched at least one regex with no summary report.
# LEGAL: PUBLIC DOMAIN. SCRIPT PROVIDED "AS IS" WITH NO WARRANTIES
# OR GUARANTEES OF
# ANY KIND, INCLUDING BUT NOT LIMITED TO MERCHANTABILITY AND/OR
# FITNESS FOR A PARTICULAR PURPOSE. ALL RISKS OF DAMAGE REMAINS WITH
# THE USER, EVEN IF THE AUTHOR, SUPPLIER OR DISTRIBUTOR HAS BEEN ADVISED OF
# THE POSSIBILITY OF ANY SUCH DAMAGE.
######################################################################################

param ($logfile, $patternsfile, [Switch] $ShowMatchedLines)

# Load file with the regex patterns, but ignore blank lines.
$patterns = ( get-content $patternsfile | where-object {$_.length -ne 0} )


# From each line in $patterns, extract the regex pattern and its
description, add these
# back as synthetic properties to each line, plus a counter of matches
initialized to zero.

foreach ($line in $patterns)
{
if ( $line -match "(?<pattern>^[^\t]+)\t+(?<description>.+$)" )
{
add-member -membertype NoteProperty -name Pattern -value
$matches.pattern -input $line | out-null
add-member -membertype NoteProperty -name Description -value
$matches.description -input $line | out-null
add-member -membertype NoteProperty -name Count -value
-input $line | out-null
}
}

# Remove lines which could not be parsed correctly (they will not have Count
property).
# If you have comments lines, don't include any tabs in those lines so
they'll be ignored.
$patterns = ( $patterns | where-object {$_.count -ne $null } )


# Use Switch to process each line of logfile, one line at a time, comparing
each line against
# all the patterns, incrementing the counter of matches to each pattern.

switch -file $logfile {
{$_ -match '^\;|^\#'} { continue } # Ignore lines that begin with
comment characters: ; #

{$_.length -eq 0 } { continue } # Ignore blank lines in log file.

{ $true } {
foreach ($line in $patterns) {
if ($_ -match $line.pattern) {
if ($ShowMatchedLines) { $_ ; break } #Break out of
foreach, one match good enough.
$line.count++
}
}
}
}


# Emit count of patterns which matched at least one line.

if (-not $ShowMatchedLines)
{
$patterns | where-object { $_.count -gt 0 } |
select-object Count,Description,Pattern | sort-object count -desc
}

####END#####################################################################

Mmm since this is the year of data parsing
and charting (and yes Regex is also included)
then FYI

Remember, Windows PowerShell
is not a data parser, but is a
Windows-based automation tool,
meant to pass "data"
from tool to tool, to tool,
until output (or whatever)!

Mmm automate tools and there really are
lots of Windows-based tools available,
for Windows PowerShell to automateI

So for here perhaps (like so many IT Pros)
automate a data parser tool within powershell.exe!

Perhaps Microsoft's IIS's (local or remote)
data parser, Log Parser!

"Log Parser" - Bing
http://www.bing.com/search?q=%22Log+Parser%22

And surely after all these years,
any PowerShell user can help
one automate Log Parser too!

As always enjoy the automation
of tools within the Windows-based,
..NET aware, WPF accessible,
admin's automation tool,
powershell.exe!

Brilliant! Flowering Weeds, you rock! =))
--
WBR, Vadims Podans
MVP: PowerShell
PowerShell blog - www.sysadmins.lv

"Flowering Weeds" <no@xxxxxx> rakstija zinojuma
"news:#9nZb2o9JHA.1492@xxxxxx"...