Windows Vista Forums

Parse Log files into a strongly type Object
  1. #1


    Karlos Guest

    Parse Log files into a strongly type Object

    Fellow humans,

    I'm trying to process the following data from a log file into a
    strongly typed object for further analysis:-

    05-Nov-06 12:00:06.287 ( -1 4360 4356)
    dual_flsrv.exe:dual_flsrv.c,v:267: connection accepted from client
    SVRPCSA address 158.89.5.16
    05-Nov-06 12:00:06.287 ( -1 4360 8220)
    dual_flsrv.exe:rcv_u_fl.c,v:165: serving client SVRPCSA address
    158.89.5.16

    This is the script that I'm using to parse the data into an object:-

    switch -file c:\svrlog.txt -regex {

    "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    {
    $obj = New-Object -TypeName System.Management.Automation.PSObject
    $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
    $matches[1]
    $obj.PSObject.Properties.Add($prop)
    $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
    $matches[2]
    $obj.PSObject.Properties.Add($prop)
    $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
    $matches[3]
    $obj.PSObject.Properties.Add($prop)
    $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
    $matches[4]
    $obj.PSObject.Properties.Add($prop)
    $obj
    }

    }


    The question is, how can I collect the output of the above script into
    a collection of an object for further analysis. e.g say sort on code
    and list.

    say like $x = switch -file c:\svrlog.txt -regex { .........} (which
    doesn't work.)

    Any idea's

    Many thanks in advance




      My System SpecsSystem Spec

  2. #2


    Sung M Kim Guest

    RE: Parse Log files into a strongly type Object

    "Karlos" wrote:
    > The question is, how can I collect the output of the above script into
    > a collection of an object for further analysis. e.g say sort on code
    > and list.
    > say like $x = switch -file c:\svrlog.txt -regex { .........} (which
    > doesn't work.)

    I have modify the script just a bit so instead of using "-file" switch for
    the "switch",
    the script will get content and try to match each line in a file
    (I haven't tried to change the script more PowerShell like though)

    # get the content of log file and match each line one by one
    # each matched data will be stored in "$result"
    $result = (gc .\log.txt) | % {
    switch -regex ($_)
    {

    "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)" {
    $obj = New-Object -TypeName System.Management.Automation.PSObject
    $prop = New-Object System.Management.Automation.PSNoteProperty
    'Date', $matches[1]
    $obj.PSObject.Properties.Add($prop)
    $prop = New-Object System.Management.Automation.PSNoteProperty
    'Time', $matches[2]
    $obj.PSObject.Properties.Add($prop)
    $prop = New-Object System.Management.Automation.PSNoteProperty
    'Code', $matches[3]
    $obj.PSObject.Properties.Add($prop)
    $prop = New-Object System.Management.Automation.PSNoteProperty
    'Desc', $matches[4]
    $obj.PSObject.Properties.Add($prop)

    $obj
    }
    }}
    $result

    # the result will display the following
    # and the type of $result is "System.Object[]"
    Date Time Code
    Desc
    ---- ---- ----
    ----
    05-Nov-06 12:00:06.287 ( -1
    4360 4356)
    05-Nov-06 12:00:06.287 ( -1
    4360 8220)


      My System SpecsSystem Spec

  3. #3


    Keith Hill [MVP] Guest

    Re: Parse Log files into a strongly type Object

    "Karlos" <kbeigan@gmail.com> wrote in message
    news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
    > switch -file c:\svrlog.txt -regex {
    >
    > "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    > {
    > $obj = New-Object -TypeName System.Management.Automation.PSObject
    > $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
    > $matches[1]
    > $obj.PSObject.Properties.Add($prop)
    > $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
    > $matches[2]
    > $obj.PSObject.Properties.Add($prop)
    > $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
    > $matches[3]
    > $obj.PSObject.Properties.Add($prop)
    > $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
    > $matches[4]
    > $obj.PSObject.Properties.Add($prop)
    > $obj
    > }
    >
    > }
    >
    >
    > The question is, how can I collect the output of the above script into
    > a collection of an object for further analysis. e.g say sort on code
    > and list.
    >
    > say like $x = switch -file c:\svrlog.txt -regex { .........} (which
    > doesn't work.)
    >
    > Any idea's


    Try this:

    $x = $(switch .... )

    Apparently after the "=" of an assignment statement you can use a command
    directly but if you use a language keyword like switch it has to appear in a
    subexpression - $().

    BTW you could simplify the above:

    $obj = new-object psobject
    add-member -inp $obj -MemberType NoteProperty Time $matches[2]
    ....

    --
    Keith



      My System SpecsSystem Spec

  4. #4


    Lee Holmes [MSFT] Guest

    Re: Parse Log files into a strongly type Object

    You might find this helpful:
    http://www.leeholmes.com/blog/parset...Vengeance.aspx


    [C:\temp]
    PS:10 > $parseExpression =
    "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"

    [C:\temp]
    PS:11 > $propertyName = "Date","Time","Code","Description"

    [C:\temp]
    PS:12 > gc log.txt | Parse-TextObject -ParseExpression
    $parseExpression -PropertyName $propertyName

    Date Time Code
    Description
    ---- ---- ----
    -----------
    05-Nov-06 12:00:06.287 ( -1 4360 4356)
    dual_flsrv.exe:dual_flsrv....
    05-Nov-06 12:00:06.287 ( -1 4360 8220)
    dual_flsrv.exe:rcv_u_fl.c,...

    --
    Lee Holmes [MSFT]
    Windows PowerShell Development
    Microsoft Corporation
    This posting is provided "AS IS" with no warranties, and confers no rights.

    "Keith Hill [MVP]" <r_keith_hill@no.spam.thank.u.hotmail.com> wrote in
    message news:uY1pCKaNHHA.4000@TK2MSFTNGP06.phx.gbl...
    > "Karlos" <kbeigan@gmail.com> wrote in message
    > news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
    >> switch -file c:\svrlog.txt -regex {
    >>
    >> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    >> {
    >> $obj = New-Object -TypeName System.Management.Automation.PSObject
    >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
    >> $matches[1]
    >> $obj.PSObject.Properties.Add($prop)
    >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
    >> $matches[2]
    >> $obj.PSObject.Properties.Add($prop)
    >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
    >> $matches[3]
    >> $obj.PSObject.Properties.Add($prop)
    >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
    >> $matches[4]
    >> $obj.PSObject.Properties.Add($prop)
    >> $obj
    >> }
    >>
    >> }
    >>
    >>
    >> The question is, how can I collect the output of the above script into
    >> a collection of an object for further analysis. e.g say sort on code
    >> and list.
    >>
    >> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
    >> doesn't work.)
    >>
    >> Any idea's

    >
    > Try this:
    >
    > $x = $(switch .... )
    >
    > Apparently after the "=" of an assignment statement you can use a command
    > directly but if you use a language keyword like switch it has to appear in
    > a subexpression - $().
    >
    > BTW you could simplify the above:
    >
    > $obj = new-object psobject
    > add-member -inp $obj -MemberType NoteProperty Time $matches[2]
    > ...
    >
    > --
    > Keith
    >
    >




      My System SpecsSystem Spec

  5. #5


    Kbgo Guest

    Re: Parse Log files into a strongly type Object

    Lee,

    Your function / cmdlet looks really brilliant, however I'm still getting my
    head around Powershell, and I'm unsure about how to implement
    parse-textobject.

    I've read your article and copied the entire contents of the text box from
    your blog :

    ----- Start of textfile: Parse-TextObject.ps1
    param([string] $parseExpression, [string[]] $propertyName, [type[]]
    $propertyType, [string] $delimiter, [switch] $unitTest)

    ################################################################################################
    ##
    ## Parse-TextObject.ps1 -- Parse a simple string into a custom MshObject.
    ##
    ....
    ....
    ....

    Assert "return-Property2 * 2 should be '56785678'"
    $(($return.Property2 * 2) -eq "56785678")
    }

    Main $input $parseExpression $propertyType $propertyName $delimiter $unitTest
    ----- end of textfile: Parse-TextObject.ps1

    I copyied this file to the c:\Program Files\Windows PowerShell\v1.0\

    and then ran this command:

    $parExp =
    "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    $ObjDef = @("Time","Date","Code","Desc")
    $res = gc c:\svr.txt | parse-textobject -ParseExpression:$pasExp
    -ObjectDefinition:$ObjDef

    ---------- svr.txt -------------
    26-Dec-06 08:30:46.772 mooring(Mooring.cpp):1124: still alive counter = 6002
    26-Dec-06 08:30:46.912 ( -1 4416 22636)
    hscopcserv.exe:hscopcservobject.cpp,v:1930: OPC Read error for item 87 with
    pointno 59839 and paramno 32755
    26-Dec-06 08:30:47.115 GDAerror 0xa8194d14 0x800003e8: Device is off-net
    26-Dec-06 08:30:47.115 ( -1 4416 22636)
    hscopcserv.exe:hscopcservobject.cpp,v:1930: OPC Read error for item 88 with
    pointno 59839 and paramno 32756
    26-Dec-06 08:30:47.162 GDAerror 0xa8194d14 0x800003e8: Device is off-net
    --------------------------------

    The powershell shell reported this :

    The term 'parse-textobject' is not recognized as a cmdlet, function,
    operable program, or script file. Verify the term
    and try again.
    At line:1 char:40
    + $res = gc c:\svr.txt | parse-textobject <<<< -ParseExpression $pasExp
    -ObjectDefinition $ObjDef

    Any ideas?


    "Lee Holmes [MSFT]" wrote:

    > You might find this helpful:
    > http://www.leeholmes.com/blog/parset...Vengeance.aspx
    >
    >
    > [C:\temp]
    > PS:10 > $parseExpression =
    > "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    >
    > [C:\temp]
    > PS:11 > $propertyName = "Date","Time","Code","Description"
    >
    > [C:\temp]
    > PS:12 > gc log.txt | Parse-TextObject -ParseExpression
    > $parseExpression -PropertyName $propertyName
    >
    > Date Time Code
    > Description
    > ---- ---- ----
    > -----------
    > 05-Nov-06 12:00:06.287 ( -1 4360 4356)
    > dual_flsrv.exe:dual_flsrv....
    > 05-Nov-06 12:00:06.287 ( -1 4360 8220)
    > dual_flsrv.exe:rcv_u_fl.c,...
    >
    > --
    > Lee Holmes [MSFT]
    > Windows PowerShell Development
    > Microsoft Corporation
    > This posting is provided "AS IS" with no warranties, and confers no rights.
    >
    > "Keith Hill [MVP]" <r_keith_hill@no.spam.thank.u.hotmail.com> wrote in
    > message news:uY1pCKaNHHA.4000@TK2MSFTNGP06.phx.gbl...
    > > "Karlos" <kbeigan@gmail.com> wrote in message
    > > news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
    > >> switch -file c:\svrlog.txt -regex {
    > >>
    > >> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    > >> {
    > >> $obj = New-Object -TypeName System.Management.Automation.PSObject
    > >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
    > >> $matches[1]
    > >> $obj.PSObject.Properties.Add($prop)
    > >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
    > >> $matches[2]
    > >> $obj.PSObject.Properties.Add($prop)
    > >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
    > >> $matches[3]
    > >> $obj.PSObject.Properties.Add($prop)
    > >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
    > >> $matches[4]
    > >> $obj.PSObject.Properties.Add($prop)
    > >> $obj
    > >> }
    > >>
    > >> }
    > >>
    > >>
    > >> The question is, how can I collect the output of the above script into
    > >> a collection of an object for further analysis. e.g say sort on code
    > >> and list.
    > >>
    > >> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
    > >> doesn't work.)
    > >>
    > >> Any idea's

    > >
    > > Try this:
    > >
    > > $x = $(switch .... )
    > >
    > > Apparently after the "=" of an assignment statement you can use a command
    > > directly but if you use a language keyword like switch it has to appear in
    > > a subexpression - $().
    > >
    > > BTW you could simplify the above:
    > >
    > > $obj = new-object psobject
    > > add-member -inp $obj -MemberType NoteProperty Time $matches[2]
    > > ...
    > >
    > > --
    > > Keith
    > >
    > >

    >
    >
    >


      My System SpecsSystem Spec

  6. #6


    Kbgo Guest

    Re: Parse Log files into a strongly type Object


    Lee, Many thanks for the excellent response which is now working (my lame
    error).

    great reusable script,

    Cheers


    "Lee Holmes [MSFT]" wrote:

    > You might find this helpful:
    > http://www.leeholmes.com/blog/parset...Vengeance.aspx
    >
    >
    > [C:\temp]
    > PS:10 > $parseExpression =
    > "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    >
    > [C:\temp]
    > PS:11 > $propertyName = "Date","Time","Code","Description"
    >
    > [C:\temp]
    > PS:12 > gc log.txt | Parse-TextObject -ParseExpression
    > $parseExpression -PropertyName $propertyName
    >
    > Date Time Code
    > Description
    > ---- ---- ----
    > -----------
    > 05-Nov-06 12:00:06.287 ( -1 4360 4356)
    > dual_flsrv.exe:dual_flsrv....
    > 05-Nov-06 12:00:06.287 ( -1 4360 8220)
    > dual_flsrv.exe:rcv_u_fl.c,...
    >
    > --
    > Lee Holmes [MSFT]
    > Windows PowerShell Development
    > Microsoft Corporation
    > This posting is provided "AS IS" with no warranties, and confers no rights.
    >
    > "Keith Hill [MVP]" <r_keith_hill@no.spam.thank.u.hotmail.com> wrote in
    > message news:uY1pCKaNHHA.4000@TK2MSFTNGP06.phx.gbl...
    > > "Karlos" <kbeigan@gmail.com> wrote in message
    > > news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
    > >> switch -file c:\svrlog.txt -regex {
    > >>
    > >> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    > >> {
    > >> $obj = New-Object -TypeName System.Management.Automation.PSObject
    > >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
    > >> $matches[1]
    > >> $obj.PSObject.Properties.Add($prop)
    > >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
    > >> $matches[2]
    > >> $obj.PSObject.Properties.Add($prop)
    > >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
    > >> $matches[3]
    > >> $obj.PSObject.Properties.Add($prop)
    > >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
    > >> $matches[4]
    > >> $obj.PSObject.Properties.Add($prop)
    > >> $obj
    > >> }
    > >>
    > >> }
    > >>
    > >>
    > >> The question is, how can I collect the output of the above script into
    > >> a collection of an object for further analysis. e.g say sort on code
    > >> and list.
    > >>
    > >> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
    > >> doesn't work.)
    > >>
    > >> Any idea's

    > >
    > > Try this:
    > >
    > > $x = $(switch .... )
    > >
    > > Apparently after the "=" of an assignment statement you can use a command
    > > directly but if you use a language keyword like switch it has to appear in
    > > a subexpression - $().
    > >
    > > BTW you could simplify the above:
    > >
    > > $obj = new-object psobject
    > > add-member -inp $obj -MemberType NoteProperty Time $matches[2]
    > > ...
    > >
    > > --
    > > Keith
    > >
    > >

    >
    >
    >


      My System SpecsSystem Spec

  7. #7


    Lee Holmes [MSFT] Guest

    Re: Parse Log files into a strongly type Object

    My pleasure!

    --
    Lee Holmes [MSFT]
    Windows PowerShell Development
    Microsoft Corporation
    This posting is provided "AS IS" with no warranties, and confers no rights.

    "Kbgo" <Kbgo@discussions.microsoft.com> wrote in message
    news:19D31DE7-5D28-4DFA-85E3-C0DAC4D8A24E@microsoft.com...
    >
    > Lee, Many thanks for the excellent response which is now working (my lame
    > error).
    >
    > great reusable script,
    >
    > Cheers
    >
    >
    > "Lee Holmes [MSFT]" wrote:
    >
    >> You might find this helpful:
    >> http://www.leeholmes.com/blog/parset...Vengeance.aspx
    >>
    >>
    >> [C:\temp]
    >> PS:10 > $parseExpression =
    >> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    >>
    >> [C:\temp]
    >> PS:11 > $propertyName = "Date","Time","Code","Description"
    >>
    >> [C:\temp]
    >> PS:12 > gc log.txt | Parse-TextObject -ParseExpression
    >> $parseExpression -PropertyName $propertyName
    >>
    >> Date Time Code
    >> Description
    >> ---- ---- ----
    >> -----------
    >> 05-Nov-06 12:00:06.287 ( -1 4360
    >> 4356)
    >> dual_flsrv.exe:dual_flsrv....
    >> 05-Nov-06 12:00:06.287 ( -1 4360
    >> 8220)
    >> dual_flsrv.exe:rcv_u_fl.c,...
    >>
    >> --
    >> Lee Holmes [MSFT]
    >> Windows PowerShell Development
    >> Microsoft Corporation
    >> This posting is provided "AS IS" with no warranties, and confers no
    >> rights.
    >>
    >> "Keith Hill [MVP]" <r_keith_hill@no.spam.thank.u.hotmail.com> wrote in
    >> message news:uY1pCKaNHHA.4000@TK2MSFTNGP06.phx.gbl...
    >> > "Karlos" <kbeigan@gmail.com> wrote in message
    >> > news:1168531851.839808.105750@o58g2000hsb.googlegroups.com...
    >> >> switch -file c:\svrlog.txt -regex {
    >> >>
    >> >> "^\s*(\d\d-\w\w\w-\d\d)\s*(\d\d:\d\d:\d\d\.\d\d\d)\s*(.*[0-9]\))\s*(.*)"
    >> >> {
    >> >> $obj = New-Object -TypeName System.Management.Automation.PSObject
    >> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Date',
    >> >> $matches[1]
    >> >> $obj.PSObject.Properties.Add($prop)
    >> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Time',
    >> >> $matches[2]
    >> >> $obj.PSObject.Properties.Add($prop)
    >> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Code',
    >> >> $matches[3]
    >> >> $obj.PSObject.Properties.Add($prop)
    >> >> $prop = New-Object System.Management.Automation.PSNoteProperty 'Desc',
    >> >> $matches[4]
    >> >> $obj.PSObject.Properties.Add($prop)
    >> >> $obj
    >> >> }
    >> >>
    >> >> }
    >> >>
    >> >>
    >> >> The question is, how can I collect the output of the above script into
    >> >> a collection of an object for further analysis. e.g say sort on code
    >> >> and list.
    >> >>
    >> >> say like $x = switch -file c:\svrlog.txt -regex { .........} (which
    >> >> doesn't work.)
    >> >>
    >> >> Any idea's
    >> >
    >> > Try this:
    >> >
    >> > $x = $(switch .... )
    >> >
    >> > Apparently after the "=" of an assignment statement you can use a
    >> > command
    >> > directly but if you use a language keyword like switch it has to appear
    >> > in
    >> > a subexpression - $().
    >> >
    >> > BTW you could simplify the above:
    >> >
    >> > $obj = new-object psobject
    >> > add-member -inp $obj -MemberType NoteProperty Time $matches[2]
    >> > ...
    >> >
    >> > --
    >> > Keith
    >> >
    >> >

    >>
    >>
    >>




      My System SpecsSystem Spec

Parse Log files into a strongly type Object problems?

Similar Threads
Thread Thread Starter Forum Replies Last Post
Casting and iteration in object of type object eliassal .NET General 2 08 Jan 2010
parse files phoenix VB Script 3 23 Oct 2008
Re: Unable to cast COM object of type 'ADODB.RecordsetClass' to class type 'System.Object[]' Michel Posseth [MCP] .NET General 2 27 Mar 2008
Parse XML files from Powershell? Duncan Smith PowerShell 8 21 Mar 2007
Parse Log files into a strongly type Object Karlos PowerShell 1 11 Jan 2007