Parse Log files into a strongly type Object

Fellow humans,

I'm trying to process the following data from a log file into a
strongly typed object for further analysis:-

05-Nov-06 12:00:06.287 ( -1 4360 4356)
dual_flsrv.exe:dual_flsrv.c,v:267: connection accepted from client
SVRPCSA address 158.89.5.16
05-Nov-06 12:00:06.287 ( -1 4360 8220)
dual_flsrv.exe:rcv_u_fl.c,v:165: serving client SVRPCSA address
158.89.5.16

This is the script that I'm using to parse the data into an object:-

switch -file c:\svrlog.txt -regex {

"^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
{
$obj = New-Object -TypeName System.Management.Automation.PSObject
$prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
$matches[1]
$obj.PSObject.Properties.Add($prop)
$prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
$matches[2]
$obj.PSObject.Properties.Add($prop)
$prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
$matches[3]
$obj.PSObject.Properties.Add($prop)
$prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
$matches[4]
$obj.PSObject.Properties.Add($prop)
$obj
}

}


The question is, how can I collect the output of the above script into
a collection of an object for further analysis. e.g say sort on code
and list.

say like $x = switch -file c:\svrlog.txt -regex { .........} (which
doesn't work.)

Any idea's

Many thanks in advance

"Karlos" wrote:
> The question is, how can I collect the output of the above script into
> a collection of an object for further analysis. e.g say sort on code
> and list.
> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
> doesn't work.)
I have modify the script just a bit so instead of using "-file" switch for
the "switch",
the script will get content and try to match each line in a file
(I haven't tried to change the script more PowerShell like though)

# get the content of log file and match each line one by one
# each matched data will be stored in "$result"
$result = (gc .\log.txt) | % {
switch -regex ($_)
{

"^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)" {
$obj = New-Object -TypeName System.Management.Automation.PSObject
$prop = New-Object System.Management.Automation.PSNoteProperty
'Date', $matches[1]
$obj.PSObject.Properties.Add($prop)
$prop = New-Object System.Management.Automation.PSNoteProperty
'Time', $matches[2]
$obj.PSObject.Properties.Add($prop)
$prop = New-Object System.Management.Automation.PSNoteProperty
'Code', $matches[3]
$obj.PSObject.Properties.Add($prop)
$prop = New-Object System.Management.Automation.PSNoteProperty
'Desc', $matches[4]
$obj.PSObject.Properties.Add($prop)

$obj
}
}}
$result

# the result will display the following
# and the type of $result is "System.Object[]"
Date Time Code
Desc
---- ---- ----
----
05-Nov-06 12:00:06.287 ( -1
4360 4356)
05-Nov-06 12:00:06.287 ( -1
4360 8220)

"Karlos" <kbeigan@gmail.com> wrote in message
news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
> switch -file c:\svrlog.txt -regex {
>
> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
> {
> $obj = New-Object -TypeName System.Management.Automation.PSObject
> $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
> $matches[1]
> $obj.PSObject.Properties.Add($prop)
> $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
> $matches[2]
> $obj.PSObject.Properties.Add($prop)
> $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
> $matches[3]
> $obj.PSObject.Properties.Add($prop)
> $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
> $matches[4]
> $obj.PSObject.Properties.Add($prop)
> $obj
> }
>
> }
>
>
> The question is, how can I collect the output of the above script into
> a collection of an object for further analysis. e.g say sort on code
> and list.
>
> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
> doesn't work.)
>
> Any idea's

Try this:

$x = $(switch .... )

Apparently after the "=" of an assignment statement you can use a command
directly but if you use a language keyword like switch it has to appear in a
subexpression - $().

BTW you could simplify the above:

$obj = new-object psobject
add-member -inp $obj -MemberType NoteProperty Time $matches[2]
....

--
Keith

You might find this helpful:
http://www.leeholmes.com/blog/parset...Vengeance.aspx


[C:\temp]
PS:10 > $parseExpression =
"^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"

[C:\temp]
PS:11 > $propertyName = "Date","Time","Code","Description"

[C:\temp]
PS:12 > gc log.txt | Parse-TextObject -ParseExpression
$parseExpression -PropertyName $propertyName

Date Time Code
Description
---- ---- ----
-----------
05-Nov-06 12:00:06.287 ( -1 4360 4356)
dual_flsrv.exe:dual_flsrv....
05-Nov-06 12:00:06.287 ( -1 4360 8220)
dual_flsrv.exe:rcv_u_fl.c,...

--
Lee Holmes [MSFT]
Windows PowerShell Development
Microsoft Corporation
This posting is provided "AS IS" with no warranties, and confers no rights.

"Keith Hill [MVP]" <r_keith_hill@no.spam.thank.u.hotmail.com> wrote in
message news:uY1pCKaNHHA.4000@TK2MSFTNGP06.phx.gbl...
> "Karlos" <kbeigan@gmail.com> wrote in message
> news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
>> switch -file c:\svrlog.txt -regex {
>>
>> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
>> {
>> $obj = New-Object -TypeName System.Management.Automation.PSObject
>> $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
>> $matches[1]
>> $obj.PSObject.Properties.Add($prop)
>> $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
>> $matches[2]
>> $obj.PSObject.Properties.Add($prop)
>> $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
>> $matches[3]
>> $obj.PSObject.Properties.Add($prop)
>> $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
>> $matches[4]
>> $obj.PSObject.Properties.Add($prop)
>> $obj
>> }
>>
>> }
>>
>>
>> The question is, how can I collect the output of the above script into
>> a collection of an object for further analysis. e.g say sort on code
>> and list.
>>
>> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
>> doesn't work.)
>>
>> Any idea's
>
> Try this:
>
> $x = $(switch .... )
>
> Apparently after the "=" of an assignment statement you can use a command
> directly but if you use a language keyword like switch it has to appear in
> a subexpression - $().
>
> BTW you could simplify the above:
>
> $obj = new-object psobject
> add-member -inp $obj -MemberType NoteProperty Time $matches[2]
> ...
>
> --
> Keith
>
>

Lee,

Your function / cmdlet looks really brilliant, however I'm still getting my
head around Powershell, and I'm unsure about how to implement
parse-textobject.

I've read your article and copied the entire contents of the text box from
your blog :

----- Start of textfile: Parse-TextObject.ps1
param([string] $parseExpression, [string[]] $propertyName, [type[]]
$propertyType, [string] $delimiter, [switch] $unitTest)

################################################################################################
##
## Parse-TextObject.ps1 -- Parse a simple string into a custom MshObject.
##
....
....
....

Assert "return-Property2 * 2 should be '56785678'"
$(($return.Property2 * 2) -eq "56785678")
}

Main $input $parseExpression $propertyType $propertyName $delimiter $unitTest
----- end of textfile: Parse-TextObject.ps1

I copyied this file to the c:\Program Files\Windows PowerShell\v1.0\

and then ran this command:

$parExp =
"^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
$ObjDef = @("Time","Date","Code","Desc")
$res = gc c:\svr.txt | parse-textobject -ParseExpression:$pasExp
-ObjectDefinition:$ObjDef

---------- svr.txt -------------
26-Dec-06 08:30:46.772 mooring(Mooring.cpp):1124: still alive counter = 6002
26-Dec-06 08:30:46.912 ( -1 4416 22636)
hscopcserv.exe:hscopcservobject.cpp,v:1930: OPC Read error for item 87 with
pointno 59839 and paramno 32755
26-Dec-06 08:30:47.115 GDAerror 0xa8194d14 0x800003e8: Device is off-net
26-Dec-06 08:30:47.115 ( -1 4416 22636)
hscopcserv.exe:hscopcservobject.cpp,v:1930: OPC Read error for item 88 with
pointno 59839 and paramno 32756
26-Dec-06 08:30:47.162 GDAerror 0xa8194d14 0x800003e8: Device is off-net
--------------------------------

The powershell shell reported this :

The term 'parse-textobject' is not recognized as a cmdlet, function,
operable program, or script file. Verify the term
and try again.
At line:1 char:40
+ $res = gc c:\svr.txt | parse-textobject <<<< -ParseExpression $pasExp
-ObjectDefinition $ObjDef

Any ideas?


"Lee Holmes [MSFT]" wrote:

> You might find this helpful:
> http://www.leeholmes.com/blog/parset...Vengeance.aspx
>
>
> [C:\temp]
> PS:10 > $parseExpression =
> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
>
> [C:\temp]
> PS:11 > $propertyName = "Date","Time","Code","Description"
>
> [C:\temp]
> PS:12 > gc log.txt | Parse-TextObject -ParseExpression
> $parseExpression -PropertyName $propertyName
>
> Date Time Code
> Description
> ---- ---- ----
> -----------
> 05-Nov-06 12:00:06.287 ( -1 4360 4356)
> dual_flsrv.exe:dual_flsrv....
> 05-Nov-06 12:00:06.287 ( -1 4360 8220)
> dual_flsrv.exe:rcv_u_fl.c,...
>
> --
> Lee Holmes [MSFT]
> Windows PowerShell Development
> Microsoft Corporation
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
> "Keith Hill [MVP]" <r_keith_hill@no.spam.thank.u.hotmail.com> wrote in
> message news:uY1pCKaNHHA.4000@TK2MSFTNGP06.phx.gbl...
> > "Karlos" <kbeigan@gmail.com> wrote in message
> > news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
> >> switch -file c:\svrlog.txt -regex {
> >>
> >> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
> >> {
> >> $obj = New-Object -TypeName System.Management.Automation.PSObject
> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
> >> $matches[1]
> >> $obj.PSObject.Properties.Add($prop)
> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
> >> $matches[2]
> >> $obj.PSObject.Properties.Add($prop)
> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
> >> $matches[3]
> >> $obj.PSObject.Properties.Add($prop)
> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
> >> $matches[4]
> >> $obj.PSObject.Properties.Add($prop)
> >> $obj
> >> }
> >>
> >> }
> >>
> >>
> >> The question is, how can I collect the output of the above script into
> >> a collection of an object for further analysis. e.g say sort on code
> >> and list.
> >>
> >> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
> >> doesn't work.)
> >>
> >> Any idea's
> >
> > Try this:
> >
> > $x = $(switch .... )
> >
> > Apparently after the "=" of an assignment statement you can use a command
> > directly but if you use a language keyword like switch it has to appear in
> > a subexpression - $().
> >
> > BTW you could simplify the above:
> >
> > $obj = new-object psobject
> > add-member -inp $obj -MemberType NoteProperty Time $matches[2]
> > ...
> >
> > --
> > Keith
> >
> >
>
>
>

Lee, Many thanks for the excellent response which is now working (my lame
error).

great reusable script,

Cheers


"Lee Holmes [MSFT]" wrote:

> You might find this helpful:
> http://www.leeholmes.com/blog/parset...Vengeance.aspx
>
>
> [C:\temp]
> PS:10 > $parseExpression =
> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
>
> [C:\temp]
> PS:11 > $propertyName = "Date","Time","Code","Description"
>
> [C:\temp]
> PS:12 > gc log.txt | Parse-TextObject -ParseExpression
> $parseExpression -PropertyName $propertyName
>
> Date Time Code
> Description
> ---- ---- ----
> -----------
> 05-Nov-06 12:00:06.287 ( -1 4360 4356)
> dual_flsrv.exe:dual_flsrv....
> 05-Nov-06 12:00:06.287 ( -1 4360 8220)
> dual_flsrv.exe:rcv_u_fl.c,...
>
> --
> Lee Holmes [MSFT]
> Windows PowerShell Development
> Microsoft Corporation
> This posting is provided "AS IS" with no warranties, and confers no rights.
>
> "Keith Hill [MVP]" <r_keith_hill@no.spam.thank.u.hotmail.com> wrote in
> message news:uY1pCKaNHHA.4000@TK2MSFTNGP06.phx.gbl...
> > "Karlos" <kbeigan@gmail.com> wrote in message
> > news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
> >> switch -file c:\svrlog.txt -regex {
> >>
> >> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
> >> {
> >> $obj = New-Object -TypeName System.Management.Automation.PSObject
> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
> >> $matches[1]
> >> $obj.PSObject.Properties.Add($prop)
> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
> >> $matches[2]
> >> $obj.PSObject.Properties.Add($prop)
> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
> >> $matches[3]
> >> $obj.PSObject.Properties.Add($prop)
> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
> >> $matches[4]
> >> $obj.PSObject.Properties.Add($prop)
> >> $obj
> >> }
> >>
> >> }
> >>
> >>
> >> The question is, how can I collect the output of the above script into
> >> a collection of an object for further analysis. e.g say sort on code
> >> and list.
> >>
> >> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
> >> doesn't work.)
> >>
> >> Any idea's
> >
> > Try this:
> >
> > $x = $(switch .... )
> >
> > Apparently after the "=" of an assignment statement you can use a command
> > directly but if you use a language keyword like switch it has to appear in
> > a subexpression - $().
> >
> > BTW you could simplify the above:
> >
> > $obj = new-object psobject
> > add-member -inp $obj -MemberType NoteProperty Time $matches[2]
> > ...
> >
> > --
> > Keith
> >
> >
>
>
>

My pleasure!

--
Lee Holmes [MSFT]
Windows PowerShell Development
Microsoft Corporation
This posting is provided "AS IS" with no warranties, and confers no rights.

"Kbgo" <Kbgo@discussions.microsoft.com> wrote in message
news:19D31DE7-5D28-4DFA-85E3-C0DAC4D8A24E@microsoft.com...
>
> Lee, Many thanks for the excellent response which is now working (my lame
> error).
>
> great reusable script,
>
> Cheers
>
>
> "Lee Holmes [MSFT]" wrote:
>
>> You might find this helpful:
>> http://www.leeholmes.com/blog/parset...Vengeance.aspx
>>
>>
>> [C:\temp]
>> PS:10 > $parseExpression =
>> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
>>
>> [C:\temp]
>> PS:11 > $propertyName = "Date","Time","Code","Description"
>>
>> [C:\temp]
>> PS:12 > gc log.txt | Parse-TextObject -ParseExpression
>> $parseExpression -PropertyName $propertyName
>>
>> Date Time Code
>> Description
>> ---- ---- ----
>> -----------
>> 05-Nov-06 12:00:06.287 ( -1 4360
>> 4356)
>> dual_flsrv.exe:dual_flsrv....
>> 05-Nov-06 12:00:06.287 ( -1 4360
>> 8220)
>> dual_flsrv.exe:rcv_u_fl.c,...
>>
>> --
>> Lee Holmes [MSFT]
>> Windows PowerShell Development
>> Microsoft Corporation
>> This posting is provided "AS IS" with no warranties, and confers no
>> rights.
>>
>> "Keith Hill [MVP]" <r_keith_hill@no.spam.thank.u.hotmail.com> wrote in
>> message news:uY1pCKaNHHA.4000@TK2MSFTNGP06.phx.gbl...
>> > "Karlos" <kbeigan@gmail.com> wrote in message
>> > news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
>> >> switch -file c:\svrlog.txt -regex {
>> >>
>> >> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
>> >> {
>> >> $obj = New-Object -TypeName System.Management.Automation.PSObject
>> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
>> >> $matches[1]
>> >> $obj.PSObject.Properties.Add($prop)
>> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
>> >> $matches[2]
>> >> $obj.PSObject.Properties.Add($prop)
>> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
>> >> $matches[3]
>> >> $obj.PSObject.Properties.Add($prop)
>> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
>> >> $matches[4]
>> >> $obj.PSObject.Properties.Add($prop)
>> >> $obj
>> >> }
>> >>
>> >> }
>> >>
>> >>
>> >> The question is, how can I collect the output of the above script into
>> >> a collection of an object for further analysis. e.g say sort on code
>> >> and list.
>> >>
>> >> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
>> >> doesn't work.)
>> >>
>> >> Any idea's
>> >
>> > Try this:
>> >
>> > $x = $(switch .... )
>> >
>> > Apparently after the "=" of an assignment statement you can use a
>> > command
>> > directly but if you use a language keyword like switch it has to appear
>> > in
>> > a subexpression - $().
>> >
>> > BTW you could simplify the above:
>> >
>> > $obj = new-object psobject
>> > add-member -inp $obj -MemberType NoteProperty Time $matches[2]
>> > ...
>> >
>> > --
>> > Keith
>> >
>> >
>>
>>
>>