Vista - return only certain lines from large text file

I am trying to put together a scipt that will only return certain lines
based on content in that line to an output file. The file we have is one
that records every logon on our domain and outputs lines in the following
format.

8/7/2006 10:10:29 AM Username Computername

The total file is 700+MB with over 5 Million lines in it. So importing it to
Excel and sorting is not an option.

What I want to do first is be able to put in either a Username, or
Computername and output only those lines to another file.

A second thing I would like to do is to slit this large file into many files
based on date range, Username or Computername.

Any help would be apprecited. I am new to Powershell. Thanks.

Rob

Personally for that amount of data I would put it in a SQL table so I could
index it. Scanning through that much data is not going to be fast. Assuming
that it has to be accessed as a flat file can you confirm the record
structure. Is it fixed field as you show?
--
Richard Siddaway

Please note that all scripts are supplied "as is" and with no warranty


"Microsoft News Groups" wrote:

> I am trying to put together a scipt that will only return certain lines
> based on content in that line to an output file. The file we have is one
> that records every logon on our domain and outputs lines in the following
> format.
>
> 8/7/2006 10:10:29 AM Username Computername
>
> The total file is 700+MB with over 5 Million lines in it. So importing it to
> Excel and sorting is not an option.
>
> What I want to do first is be able to put in either a Username, or
> Computername and output only those lines to another file.
>
> A second thing I would like to do is to slit this large file into many files
> based on date range, Username or Computername.
>
> Any help would be apprecited. I am new to Powershell. Thanks.
>
> Rob
>
>
>

I'd start with something like:

gc <input file spec>|where-object {$_ -like "*<computername/username>*} |
out-file "<output file spec>"

"Microsoft News Groups" wrote:

> I am trying to put together a scipt that will only return certain lines
> based on content in that line to an output file. The file we have is one
> that records every logon on our domain and outputs lines in the following
> format.
>
> 8/7/2006 10:10:29 AM Username Computername
>
> The total file is 700+MB with over 5 Million lines in it. So importing it to
> Excel and sorting is not an option.
>
> What I want to do first is be able to put in either a Username, or
> Computername and output only those lines to another file.
>
> A second thing I would like to do is to slit this large file into many files
> based on date range, Username or Computername.
>
> Any help would be apprecited. I am new to Powershell. Thanks.
>
> Rob
>
>
>

Yes the fields are fixed as is shows below. That was a cut and paste from
some sample output. In the future, the files will not be as large. We are
still considering moving this to a SQL DB, but thought this would be a good
exercise for powershell to both get particular data from the file and split
it up in to multiple files based on particular criteria that we define.


"RichS" <RichS@discussions.microsoft.com> wrote in message
news:36C2C7A1-F68A-4A6D-B116-F0513ADC6218@microsoft.com...
> Personally for that amount of data I would put it in a SQL table so I
> could
> index it. Scanning through that much data is not going to be fast.
> Assuming
> that it has to be accessed as a flat file can you confirm the record
> structure. Is it fixed field as you show?
> --
> Richard Siddaway
>
> Please note that all scripts are supplied "as is" and with no warranty
>
>
> "Microsoft News Groups" wrote:
>
>> I am trying to put together a scipt that will only return certain lines
>> based on content in that line to an output file. The file we have is one
>> that records every logon on our domain and outputs lines in the following
>> format.
>>
>> 8/7/2006 10:10:29 AM Username
>> Computername
>>
>> The total file is 700+MB with over 5 Million lines in it. So importing it
>> to
>> Excel and sorting is not an option.
>>
>> What I want to do first is be able to put in either a Username, or
>> Computername and output only those lines to another file.
>>
>> A second thing I would like to do is to slit this large file into many
>> files
>> based on date range, Username or Computername.
>>
>> Any help would be apprecited. I am new to Powershell. Thanks.
>>
>> Rob
>>
>>
>>

"Rob Campbell" <RobCampbell@discussions.microsoft.com> wrote in message
news:EA954EF5-849D-43E3-B371-033CE6CF2BF9@microsoft.com...
> I'd start with something like:
>
> gc <input file spec>|where-object {$_ -like "*<computername/username>*} |
> out-file "<output file spec>"

If you read a large file, you may want to play with the -readcount parameter
of get-content. This parameter affects the number of lines sent to the
pipeline at a time (default is 1). See the performance of various values on
a file that contains 12440 lines of text:

PS> measure-command {gc acl.csv}
....
TotalMilliseconds : 1309,3295

PS> measure-command {gc acl.csv -readcount 100}
....
TotalMilliseconds : 54,8195

PS> measure-command {gc acl.csv -readcount 1000}
....
TotalMilliseconds : 47,587

Note that the value passed to the pipeline will vary with the parameter
(from 1 to x lines of text), therefore you might have to adapt the logic of
your script down the pipeline.

Jacques

Just to clarify, readcount will cat multiple lines together and send them
down the pipeline as a single object?

"Jacques Barathon [MS]" wrote:

> "Rob Campbell" <RobCampbell@discussions.microsoft.com> wrote in message
> news:EA954EF5-849D-43E3-B371-033CE6CF2BF9@microsoft.com...
> > I'd start with something like:
> >
> > gc <input file spec>|where-object {$_ -like "*<computername/username>*} |
> > out-file "<output file spec>"
>
> If you read a large file, you may want to play with the -readcount parameter
> of get-content. This parameter affects the number of lines sent to the
> pipeline at a time (default is 1). See the performance of various values on
> a file that contains 12440 lines of text:
>
> PS> measure-command {gc acl.csv}
> ....
> TotalMilliseconds : 1309,3295
>
> PS> measure-command {gc acl.csv -readcount 100}
> ....
> TotalMilliseconds : 54,8195
>
> PS> measure-command {gc acl.csv -readcount 1000}
> ....
> TotalMilliseconds : 47,587
>
> Note that the value passed to the pipeline will vary with the parameter
> (from 1 to x lines of text), therefore you might have to adapt the logic of
> your script down the pipeline.
>
> Jacques
>
>

"Rob Campbell" <RobCampbell@discussions.microsoft.com> wrote in message
news:824F7E38-BDEE-485D-A2DF-AAAC9A83A688@microsoft.com...
> Just to clarify, readcount will cat multiple lines together and send them
> down the pipeline as a single object?

Yep.

Jacques

Thanks. That's good stuff.

"Jacques Barathon [MS]" wrote:

> "Rob Campbell" <RobCampbell@discussions.microsoft.com> wrote in message
> news:824F7E38-BDEE-485D-A2DF-AAAC9A83A688@microsoft.com...
> > Just to clarify, readcount will cat multiple lines together and send them
> > down the pipeline as a single object?
>
> Yep.
>
> Jacques
>
>