Vista - How can I avoid an out of memory exception?

I have some Internet log files that run 8-9GB each. I am trying to convert
the format to one that my reporting software will accept. This code works on
smaller files but when I run it against the big ones I get an out of memory
exception.

filter convert
{
if ($_.linenumber -gt 4)
{
$line = $_.line -replace " \(.*\)","-"
$line = $line.split()
$time = ($line[0] +" "+ $line[1])
$t = [datetime] $time
$newtime = "`"[" + $t.tolocaltime().toString("dd/MMM/yyyy:HH:MM:ss") + "
-0" + $hour + "00]`""
$newtime +" `""+ $line[23] +"`" "+ $line[3] +" "+ $line[9]+ "://" +
$line[10] + $line[12] +" "+ $line[20] +" "+ $line[21] +" "+ $line[27] +" "+
$line[6]
}
If ($_.linenumber -eq 4) {"#Fields: localtime s-computername c-ip c-uri
sc-filter-result cs-categories cs-user sc-bytes"}
If ($_.linenumber -le 3) {$_.line}
}

$files = dir
measure-command {foreach ($file in $files) {get-content $file -readCount 1
-include *.txt | convert | set-content "$file.log"}}

This throws and out of memory exception. I have also tried the following...

$files = dir
measure-command {foreach ($file in $files) {select-string $file.fullname
-pattern "." | convert | set-content "$file.log"}}

With the same result. I think what is happening is that select-string and
get-content are building a collection which contains each line in the file
which is using all of my system memory.

How can I read in one line at a time from a file and pass it along the
pipeline without creating a collection?

Thanks,
Jonathan

JW wrote:
> With the same result. I think what is happening is that select-string and
> get-content are building a collection which contains each line in the file
> which is using all of my system memory.
That would be pretty lame, but you may be right. :-/

> How can I read in one line at a time from a file and pass it along the
> pipeline without creating a collection?
I think your best bet is to use .NET. Unfortunately, I don't think
there's a built-in line enumerator.

But I think you could create one and compile it into your utilities DLL
(say "JW.Utilities.dll") which you load in your profile:

[Reflection.Assembly]::LoadWithPartialName("JW.Utilities")

Assuming it's in the GAC... or:

[Reflection.Assembly]::LoadFrom("c:/myutils/JW.Utilities.dll")

If it's not.

Then you can use it like so:

PS> new-object JW.LineReader c:/foo.log | convert | set-content bar.log


Here's a bit of (untested) code to implement a simple line enumerator.
I'm assuming that PowerShell can use any IEnumerator object. If not, you
may need to implement IEnumerable<string> as well.


public class LineEnumerator : IEnumerator<string>
{
public LineEnumerator(string path)
: this(new StreamReader(path)) { }

public LineEnumerator(Stream stream)
: this(new StreamReader(stream)) { }

public LineEnumerator(TextReader reader)
{
if(reader == null) throw new ArgumentNullException();
this.reader = reader;
}

readonly TextReader reader;

#region IEnumerator Members
public string Current
{
get
{
if(currentLine == null) throw new InvalidOperationException();
return currentLine;
}
}

public void Dispose()
{
reader.Dispose();
}

object System.Collections.IEnumerator.Current
{
get { return Current; }
}

public bool MoveNext()
{
currentLine = reader.ReadLine();
return currentLine != null;
}

public void Reset()
{
throw new NotSupportedException();
}
#endregion

string currentLine;
}

"Adam Milazzo" wrote:

> [Reflection.Assembly]::LoadFrom("c:/myutils/JW.Utilities.dll")
<snip>
> PS> new-object JW.LineReader c:/foo.log | convert | set-content bar.log
<snip>

Using .NET directly from PowerShell I think you should avoid forward slashes
and use backslashes instead.

--
greetings
dreeschkind

JW,

Might want to try using .NET framework as think Get-Content reads the entire
file into memory... obviously with an 8gb file that presents a problem.
Here's a very simple example...

$filepath = "c:\whatever.txt"
$read = [system.io.file]::OpenText($filepath)
while ($read.Peek() -ne -1) {
write-host $read.readline()
}

If you do a $read | gm, you'll notice that [system.io.file]::OpenText
returns a System.IO.StreamReader type. If you look at the properties and
methods of this type, you'll notice other options for reading the file.

Should get you started.

Gaurhoth


"JW" <JW@discussions.microsoft.com> wrote in message
news:85655ABC-8765-4214-B7E2-51F3067A38CF@microsoft.com...
>I have some Internet log files that run 8-9GB each. I am trying to convert
> the format to one that my reporting software will accept. This code works
> on
> smaller files but when I run it against the big ones I get an out of
> memory
> exception.
>
> filter convert
> {
> if ($_.linenumber -gt 4)
> {
> $line = $_.line -replace " \(.*\)","-"
> $line = $line.split()
> $time = ($line[0] +" "+ $line[1])
> $t = [datetime] $time
> $newtime = "`"[" + $t.tolocaltime().toString("dd/MMM/yyyy:HH:MM:ss") + "
> -0" + $hour + "00]`""
> $newtime +" `""+ $line[23] +"`" "+ $line[3] +" "+ $line[9]+ "://" +
> $line[10] + $line[12] +" "+ $line[20] +" "+ $line[21] +" "+ $line[27] +"
> "+
> $line[6]
> }
> If ($_.linenumber -eq 4) {"#Fields: localtime s-computername c-ip c-uri
> sc-filter-result cs-categories cs-user sc-bytes"}
> If ($_.linenumber -le 3) {$_.line}
> }
>
> $files = dir
> measure-command {foreach ($file in $files) {get-content $file -readCount 1
> -include *.txt | convert | set-content "$file.log"}}
>
> This throws and out of memory exception. I have also tried the
> following...
>
> $files = dir
> measure-command {foreach ($file in $files) {select-string $file.fullname
> -pattern "." | convert | set-content "$file.log"}}
>
> With the same result. I think what is happening is that select-string and
> get-content are building a collection which contains each line in the file
> which is using all of my system memory.
>
> How can I read in one line at a time from a file and pass it along the
> pipeline without creating a collection?
>
> Thanks,
> Jonathan

> I have some Internet log files
> that run 8-9GB each. I am trying to
> convert the format to one that my
> reporting software will accept.

Perhaps Microsoft's Log Parser
http://www.microsoft.com/technet/scr...r/default.mspx

this test:

filter testfilter {
write-Host "test $_"
$_
}

1..10 | % {"hello $_" } > c:\outtest.txt
gc c:\outtest.txt | % {write-Host "before: $_";$_ } | testfilter | %
{write-Host "after: $_";$_}
DIR C:\OUTTEST.TXT | select-String "hello" | % {$_.line} | %
{write-Host "before: $_";$_ } | testfilter | % {write-Host "after:
$_";$_}
gc c:\outtest.txt | % {write-Host "before: $_";$_ } | % {write-Host
"after: $_";$_}

shows that neither select-string nor getcontent are blocking the
pipeline... i'll have to look at your filter to see more.. maybe
another way to test is to rewrite your filter as a function with -begin
-process and -end sections..

Actually the problem appears to be in set-content. We're investigating. For
now, it you use out-file instead of set-content, you shouldn't get the
exception.

-bruce

--
Bruce Payette [MSFT]
Windows PowerShell Technical Lead
Microsoft Corporation
This posting is provided "AS IS" with no warranties, and confers no rights.



<klumsy@xtra.co.nz> wrote in message
news:1163534623.889212.180670@h48g2000cwc.googlegroups.com...
> this test:
>
> filter testfilter {
> write-Host "test $_"
> $_
> }
>
> 1..10 | % {"hello $_" } > c:\outtest.txt
> gc c:\outtest.txt | % {write-Host "before: $_";$_ } | testfilter | %
> {write-Host "after: $_";$_}
> DIR C:\OUTTEST.TXT | select-String "hello" | % {$_.line} | %
> {write-Host "before: $_";$_ } | testfilter | % {write-Host "after:
> $_";$_}
> gc c:\outtest.txt | % {write-Host "before: $_";$_ } | % {write-Host
> "after: $_";$_}
>
> shows that neither select-string nor getcontent are blocking the
> pipeline... i'll have to look at your filter to see more.. maybe
> another way to test is to rewrite your filter as a function with -begin
> -process and -end sections..
>