Vista - Select-String problem with not English ASCII text

Perhaps, I have got a disappointing problem: if I use Select-String with
ASCII text files with not English text then not English text is disappeared
(replaced with spaces?) in the result strings.

In my case text files contain Russian text. Other console applications
(including .NET) output Russian text from those files just fine. Well,
PowerShell Get-Content works fine, too.

So, is it an issue of Select-String, or am I missing something?

--
Thanks,
Roman

Roman,

The following is my understanding but I'm no expert in this stuff.

ASCII covers up to U+0255, I think.

Russian characters begin at U+0402 or thereabouts.

So I don't see how an "ASCII" file can contain Cyrillic characters.

So, is it possible that you "lost" the Cyrillic characters (they were
replaced by spaces or some other character) when the ASCII files were
created? i.e. before you used select-string on the files?

Or are the files not ASCII files?

Andrew Watt MVP


On Tue, 16 Jan 2007 07:34:00 -0800, Roman Kuzmin
<RomanKuzmin@discussions.microsoft.com> wrote:

>Perhaps, I have got a disappointing problem: if I use Select-String with
>ASCII text files with not English text then not English text is disappeared
>(replaced with spaces?) in the result strings.
>
>In my case text files contain Russian text. Other console applications
>(including .NET) output Russian text from those files just fine. Well,
>PowerShell Get-Content works fine, too.
>
>So, is it an issue of Select-String, or am I missing something?
>
>--
>Thanks,
>Roman

> So I don't see how an "ASCII" file can contain Cyrillic characters.
Don't you think that Russian text exists only as Unicode? Of course, it is
not true (thanks, God). Well, just believe me, Russian text exists in ASCII
files and Get-Content perfectly proves it. Unfortunately, I guess,
Select-String is not so accurate as Get-Content.

--
Thanks,
Roman

Roman -- does

Get-Content cyrillic.txt | select-string <whatever>

work?

I assume the problem you're running into is:

Select-String <whatever> cyrillic.txt

--
Lee Holmes [MSFT]
Windows PowerShell Development
Microsoft Corporation
This posting is provided "AS IS" with no warranties, and confers no rights.


"Roman Kuzmin" <z@z.z> wrote in message
news:us5C6vZOHHA.780@TK2MSFTNGP03.phx.gbl...
>> So I don't see how an "ASCII" file can contain Cyrillic characters.
> Don't you think that Russian text exists only as Unicode? Of course, it is
> not true (thanks, God). Well, just believe me, Russian text exists in
> ASCII files and Get-Content perfectly proves it. Unfortunately, I guess,
> Select-String is not so accurate as Get-Content.
>
> --
> Thanks,
> Roman
>
>

Hi Roman,

Any chance you could give us the commands you are running and if its not too
large the ASCII file?

Thanks
Marcel

"Roman Kuzmin" <z@z.z> wrote in message
news:us5C6vZOHHA.780@TK2MSFTNGP03.phx.gbl...
>> So I don't see how an "ASCII" file can contain Cyrillic characters.
> Don't you think that Russian text exists only as Unicode? Of course, it is
> not true (thanks, God). Well, just believe me, Russian text exists in
> ASCII files and Get-Content perfectly proves it. Unfortunately, I guess,
> Select-String is not so accurate as Get-Content.
>
> --
> Thanks,
> Roman
>
>

> Any chance you could give us the commands you are running and if its not
> too
> large the ASCII file?

Get-Content cyrillic.txt | select-string far
- works like a charm, but result is just strings, not what I want.

Select-String far cyrillic.txt
- does not work fine: i.e. cyrillic text in strings is mangled

Please, find attached cyrillic.txt

--
Thanks,
Roman

Just an observation:

[System.IO.File]::ReadAllLines('cyrillic.txt')
- has the same problems as: Select-String far cyrillic.txt

[System.IO.File]::ReadAllLines('cyrillic.txt',
[System.Text.Encoding]:efault)
- works fine, cyrillic text is as it should be

--
Thanks,
Roman

"Roman Kuzmin" <z@z.z> wrote in message
news:us5C6vZOHHA.780@TK2MSFTNGP03.phx.gbl...
>> So I don't see how an "ASCII" file can contain Cyrillic characters.
> Don't you think that Russian text exists only as Unicode? Of course, it is
> not true (thanks, God). Well, just believe me, Russian text exists in
> ASCII files and Get-Content perfectly proves it. Unfortunately, I guess,
> Select-String is not so accurate as Get-Content.

ASCII really only defines the code points from 0 to 127 (decimal) or U+0000
to U+007F. For single byte character sets, Windows uses ANSI code pages to
define the code points between U+0080 and U+00FF. Localized versions of
Windows would most likely use a code page that defines the native code
points for the locale in the upper half of the range of code points in a
byte.

By the time Select-String sees the text it has been converted to Unicode
because .NET strings are strictly Unicode. It does seem like a limitation
that Select-String's Path oriented parameter set doesn't let you indicate
the encoding to use when reading the file. However if the file is either
Unicode or UTF-8, it should just work. If you open the text file in
Notepad, go to save as, what does the "Encoding" drop down say? If it is
ANSI, try changing to UTF-8, save the file out and do Select-String on the
UTF-8 encoded file.

--
Keith

Keith,

Thank you for your nice explanation, it looks like I did not use term ASCII
precisely. Meantime it looks like Select-String really has lack of encoding
flexibility and moreover its default behavior is rather disappointing
(Get-Content's default behavior is just fine).

--
Thanks,
Roman

"Roman Kuzmin" <z@z.z> wrote in message
news:OL4zd8aOHHA.1248@TK2MSFTNGP02.phx.gbl...
> Keith,
>
> Thank you for your nice explanation, it looks like I did not use term
> ASCII precisely. Meantime it looks like Select-String really has lack of
> encoding flexibility and moreover its default behavior is rather
> disappointing (Get-Content's default behavior is just fine).

Roman, just out of curiousity, if you save the file out from notepad using
Unicode (or UTF-8) does Select-String work on that file? I would expect
that to work.

--
Keith