![]() |
![]() | ![]() | ![]() | ![]() | ![]() | ![]() | ![]() |
| Welcome to Windows Vista Forums. Our forum is dedicated to helping you find solutions with any problems, errors or issues you are experiencing with Windows Vista. The Vista forum also covers news and updates and has an extensive Windows Vista tutorial section that covers a wide range of tips and tricks. |
| |||||||
![]() |
| |
| | #1 (permalink) |
| | reading a big file line by line to be "out of memory" safe I have a powershell script that needs to read a big file and I want to make sure not to be "out of memory" so I thought three ways to read the file line by line: (1) $f = get-content bigfile.txt; $f | foreach { ...} (2) get-content bigfile.txt | foreach {...} (3) get-content bigfile.txt | foreach-object {...} (4) $f = openfile(bigfile); while ($line = readline(f$)) do {...}; closefile($f) (Yes, I know that (4) is syntactically incorrect, I show it just to make my point) I'm definitively sure that (1) will read all the file at one chunk. But how about the (2) and (3)? The (4) is the way traditional programming languages do it and it reads line by line. Is there any way to implement (4) in PowerShell? Any help would be appreciated. Thanks. Regards, Freddy Chavez. |
My System Specs![]() |
| | #2 (permalink) |
| | Re: reading a big file line by line to be "out of memory" safe You can always measures the time it takes to run script blocks and cmdlets or script files by using measure-command Type "help Measure-Command -examples" to get some examples. As for the performance, I'd choose option 3. It will pipe line by line instead of reading the whole file and than start to pipe (option 1,2) Shay http://scriptolog.blogspot.com > I have a powershell script that needs to read a big file and I want to > make sure not to be "out of memory" so I thought three ways to read > the file line by line: > > (1) $f = get-content bigfile.txt; $f | foreach { ...} > (2) get-content bigfile.txt | foreach {...} > (3) get-content bigfile.txt | foreach-object {...} > (4) $f = openfile(bigfile); while ($line = readline(f$)) do {...}; > closefile($f) > (Yes, I know that (4) is syntactically incorrect, I show it just to > make my point) > > I'm definitively sure that (1) will read all the file at one chunk. > But how about the (2) and (3)? > The (4) is the way traditional programming languages do it and it > reads line > by line. > Is there any way to implement (4) in PowerShell? > Any help would be appreciated. Thanks. > > Regards, > Freddy Chavez. |
My System Specs![]() |
| | #3 (permalink) |
| | Re: reading a big file line by line to be "out of memory" safe "freddy chavez" <freddychavez22@hotmail.com> wrote in message news:%23atheAE1HHA.4652@TK2MSFTNGP05.phx.gbl... >I have a powershell script that needs to read a big file and I want to make >sure not to be "out of memory" so I thought three ways to read the file >line by line: > > (1) $f = get-content bigfile.txt; $f | foreach { ...} > (2) get-content bigfile.txt | foreach {...} > (3) get-content bigfile.txt | foreach-object {...} > (4) $f = openfile(bigfile); while ($line = readline(f$)) do {...}; > closefile($f) .... > I'm definitively sure that (1) will read all the file at one chunk. > But how about the (2) and (3)? (2) and (3) are identical: in (2) foreach is an alias of foreach-object. Performance and memory-wise I would recommend that option: get-content will read your file one line at a time and will pass the line to your foreach block. Depending on your actual script the performance may vary but you will likely optimise the usage of your system's memory. > The (4) is the way traditional programming languages do it and it reads > line by line. > Is there any way to implement (4) in PowerShell? You can with direct access to .NET classes: $f = [System.IO.File]::OpenText(bigfile.txt) while (! $f.EndOfStream) {$line = $f.ReadLine(); ...} $f.Close() It is another technique that should save your memory, although not super PowerShellish :-) Jacques |
My System Specs![]() |
| | #4 (permalink) |
| | Re: reading a big file line by line to be "out of memory" safe "Jacques Barathon [MS]" wrote: > You can with direct access to .NET classes: > > $f = [System.IO.File]::OpenText(bigfile.txt) > while (! $f.EndOfStream) {$line = $f.ReadLine(); ...} > $f.Close() > > It is another technique that should save your memory, although not super > PowerShellish :-) Depends on how you define "PowerShellish". ;-) As I see it, task oriented cmdlets are just *ONE* aspect of PowerShell. Direct access to everything that .NET offers is just *ANOTHER* very important aspect. The same goes for native COM and WMI access. If we had only one of these, PoSh would probably not deserve the name PowerShell. Using ..NET, COM, native apps etc. usually has big advantages in performance, so in my opinion "PowerShellish" is using the optimal combination of all these different possiblities to solve a specific problem. -- greetings dreeschkind |
My System Specs![]() |
| | #5 (permalink) |
| | RE: reading a big file line by line to be "out of memory" safe Another very powerful way to do this is using PowerShell's switch statement which has a -file parameter that you can use to process big files line by line. Type the following to get more information about the syntax: PS> help about_switch Bruce Payette's book "PowerShell in Action" also has some very good examples how to use this feature. -- greetings dreeschkind "freddy chavez" wrote: > I have a powershell script that needs to read a big file and I want to make > sure not to be "out of memory" so I thought three ways to read the file line > by line: > > (1) $f = get-content bigfile.txt; $f | foreach { ...} > (2) get-content bigfile.txt | foreach {...} > (3) get-content bigfile.txt | foreach-object {...} > (4) $f = openfile(bigfile); while ($line = readline(f$)) do {...}; > closefile($f) > > (Yes, I know that (4) is syntactically incorrect, I show it just to make my > point) > > I'm definitively sure that (1) will read all the file at one chunk. > But how about the (2) and (3)? > The (4) is the way traditional programming languages do it and it reads line > by line. > Is there any way to implement (4) in PowerShell? > > Any help would be appreciated. Thanks. > > Regards, > Freddy Chavez. > > > |
My System Specs![]() |
| | #6 (permalink) |
| | Re: reading a big file line by line to be "out of memory" safe Glad to see you got the book. Have you checked the type verification in the Setter scriptblock of a scriptProperty close to the end of § 8.2.3 on p.229? -- Kiron |
My System Specs![]() |
| | #7 (permalink) |
| | Re: reading a big file line by line to be "out of memory" safe Actually, I'm on page 230 right now. ![]() Why do you ask? -- greetings dreeschkind "Kiron" wrote: > Glad to see you got the book. Have you checked the type verification in the > Setter scriptblock of a scriptProperty close to the end of § 8.2.3 on p.229? > > -- > Kiron > > |
My System Specs![]() |
| | #8 (permalink) |
| | Re: reading a big file line by line to be "out of memory" safe Just thought you might have an idea about the subject, since your apparent interest in it on your reply to klumsy@xtra.co.nz's post "Strongly typed scriptproperties" of 6/29/2007. -- Kiron |
My System Specs![]() |
| | #9 (permalink) |
| | Re: reading a big file line by line to be "out of memory" safe LOL, now you got me! :-) I completely forgot this thread. For my mind this was like centuries ago! But yeah, I'm still interested in a lot of stuff. -- greetings dreeschkind "Kiron" wrote: > Just thought you might have an idea about the subject, since your apparent > interest in it on your reply to klumsy@xtra.co.nz's post "Strongly typed > scriptproperties" of 6/29/2007. > > -- > Kiron > |
My System Specs![]() |
![]() |
| Thread Tools | |
| |
Similar Threads | ||||
| Thread | Forum | |||
| reading last line of file | VB Script | |||
| Displaying Sender's "FROM" line in Junk Mail Folder | Live Mail | |||
| Reading a text file and retaining line feeds | PowerShell | |||
| New powershell Analyzer with Suspendshell and Single Line "Immediate Mode" | PowerShell | |||
| "<SPACE> next page; <CR> next line; Q quit" not cleared in console | PowerShell | |||