reading a big file line by line to be "out of memory" safe

I have a powershell script that needs to read a big file and I want to make
sure not to be "out of memory" so I thought three ways to read the file line
by line:

(1) $f = get-content bigfile.txt; $f | foreach { ...}
(2) get-content bigfile.txt | foreach {...}
(3) get-content bigfile.txt | foreach-object {...}
(4) $f = openfile(bigfile); while ($line = readline(f$)) do {...};
closefile($f)

(Yes, I know that (4) is syntactically incorrect, I show it just to make my
point)

I'm definitively sure that (1) will read all the file at one chunk.
But how about the (2) and (3)?
The (4) is the way traditional programming languages do it and it reads line
by line.
Is there any way to implement (4) in PowerShell?

Any help would be appreciated. Thanks.

Regards,
Freddy Chavez.

You can always measures the time it takes to run script blocks and cmdlets
or script files
by using measure-command

Type "help Measure-Command -examples" to get some examples.

As for the performance, I'd choose option 3. It will pipe line by line instead
of reading
the whole file and than start to pipe (option 1,2)


Shay
http://scriptolog.blogspot.com



> I have a powershell script that needs to read a big file and I want to
> make sure not to be "out of memory" so I thought three ways to read
> the file line by line:
>
> (1) $f = get-content bigfile.txt; $f | foreach { ...}
> (2) get-content bigfile.txt | foreach {...}
> (3) get-content bigfile.txt | foreach-object {...}
> (4) $f = openfile(bigfile); while ($line = readline(f$)) do {...};
> closefile($f)
> (Yes, I know that (4) is syntactically incorrect, I show it just to
> make my point)
>
> I'm definitively sure that (1) will read all the file at one chunk.
> But how about the (2) and (3)?
> The (4) is the way traditional programming languages do it and it
> reads line
> by line.
> Is there any way to implement (4) in PowerShell?
> Any help would be appreciated. Thanks.
>
> Regards,
> Freddy Chavez.

"freddy chavez" <freddychavez22@hotmail.com> wrote in message
news:%23atheAE1HHA.4652@TK2MSFTNGP05.phx.gbl...
>I have a powershell script that needs to read a big file and I want to make
>sure not to be "out of memory" so I thought three ways to read the file
>line by line:
>
> (1) $f = get-content bigfile.txt; $f | foreach { ...}
> (2) get-content bigfile.txt | foreach {...}
> (3) get-content bigfile.txt | foreach-object {...}
> (4) $f = openfile(bigfile); while ($line = readline(f$)) do {...};
> closefile($f)
....
> I'm definitively sure that (1) will read all the file at one chunk.
> But how about the (2) and (3)?

(2) and (3) are identical: in (2) foreach is an alias of foreach-object.

Performance and memory-wise I would recommend that option: get-content will
read your file one line at a time and will pass the line to your foreach
block. Depending on your actual script the performance may vary but you will
likely optimise the usage of your system's memory.

> The (4) is the way traditional programming languages do it and it reads
> line by line.
> Is there any way to implement (4) in PowerShell?

You can with direct access to .NET classes:

$f = [System.IO.File]::OpenText(bigfile.txt)
while (! $f.EndOfStream) {$line = $f.ReadLine(); ...}
$f.Close()

It is another technique that should save your memory, although not super
PowerShellish :-)

Jacques

"Jacques Barathon [MS]" wrote:

> You can with direct access to .NET classes:
>
> $f = [System.IO.File]::OpenText(bigfile.txt)
> while (! $f.EndOfStream) {$line = $f.ReadLine(); ...}
> $f.Close()
>
> It is another technique that should save your memory, although not super
> PowerShellish :-)

Depends on how you define "PowerShellish". ;-)
As I see it, task oriented cmdlets are just *ONE* aspect of PowerShell.
Direct access to everything that .NET offers is just *ANOTHER* very
important aspect. The same goes for native COM and WMI access. If we had only
one of these, PoSh would probably not deserve the name PowerShell. Using
..NET, COM, native apps etc. usually has big advantages in performance, so in
my opinion "PowerShellish" is using the optimal combination of all these
different possiblities to solve a specific problem.

--
greetings
dreeschkind

Another very powerful way to do this is using PowerShell's switch statement
which has a -file parameter that you can use to process big files line by
line.
Type the following to get more information about the syntax:
PS> help about_switch
Bruce Payette's book "PowerShell in Action" also has some very good examples
how to use this feature.

--
greetings
dreeschkind

"freddy chavez" wrote:

> I have a powershell script that needs to read a big file and I want to make
> sure not to be "out of memory" so I thought three ways to read the file line
> by line:
>
> (1) $f = get-content bigfile.txt; $f | foreach { ...}
> (2) get-content bigfile.txt | foreach {...}
> (3) get-content bigfile.txt | foreach-object {...}
> (4) $f = openfile(bigfile); while ($line = readline(f$)) do {...};
> closefile($f)
>
> (Yes, I know that (4) is syntactically incorrect, I show it just to make my
> point)
>
> I'm definitively sure that (1) will read all the file at one chunk.
> But how about the (2) and (3)?
> The (4) is the way traditional programming languages do it and it reads line
> by line.
> Is there any way to implement (4) in PowerShell?
>
> Any help would be appreciated. Thanks.
>
> Regards,
> Freddy Chavez.
>
>
>

Glad to see you got the book. Have you checked the type verification in the
Setter scriptblock of a scriptProperty close to the end of § 8.2.3 on p.229?

--
Kiron

Actually, I'm on page 230 right now.
Why do you ask?

--
greetings
dreeschkind

"Kiron" wrote:

> Glad to see you got the book. Have you checked the type verification in the
> Setter scriptblock of a scriptProperty close to the end of § 8.2.3 on p.229?
>
> --
> Kiron
>
>

Just thought you might have an idea about the subject, since your apparent
interest in it on your reply to klumsy@xtra.co.nz's post "Strongly typed
scriptproperties" of 6/29/2007.

--
Kiron

LOL, now you got me! :-)

I completely forgot this thread.
For my mind this was like centuries ago!

But yeah, I'm still interested in a lot of stuff.

--
greetings
dreeschkind

"Kiron" wrote:

> Just thought you might have an idea about the subject, since your apparent
> interest in it on your reply to klumsy@xtra.co.nz's post "Strongly typed
> scriptproperties" of 6/29/2007.
>
> --
> Kiron
>