When non-ASCII creeps into script

What to do, what to do...

When non-ASCII characters creep into a script, what's the best way to see
them and remove them? Without re-typing the whole text.

Grasping at straws doesn't seem to work:
gc psa1.ps1|out-file -encoding ascii psa2.ps1

"Marco Shaw" wrote:

> What to do, what to do...
>
> When non-ASCII characters creep into a script, what's the best way to see
> them and remove them? Without re-typing the whole text.

Hm, why not just use Unicode?
You could try an editor like Scite that can set the encoding.

--
greetings
dreeschkind

Marco,

I'm wondering if this non-ASCII idea is a red herring.

Both the line feed and carriage return are in the ASCII character set.
As I understood things you were getting a message about end of line.

Andrew Watt MVP

On Wed, 15 Nov 2006 07:59:36 -0400, "Marco Shaw" <marco@Znbnet.nb.ca>
wrote:

>What to do, what to do...
>
>When non-ASCII characters creep into a script, what's the best way to see
>them and remove them? Without re-typing the whole text.
>
>Grasping at straws doesn't seem to work:
>gc psa1.ps1|out-file -encoding ascii psa2.ps1

Powershell engine can handle unicode completely. However powershell.exe
cannot display unicode characters in the console screen. (limitations
of the underlying console subsystem)

> Powershell engine can handle unicode completely. However powershell.exe
> cannot display unicode characters in the console screen. (limitations
> of the underlying console subsystem)

Not sure to understand what you say (due to my poor english) but for me
that depends of the console's font (its code page and unicode
supports).
You can change this in console's "Properties"/"Font";.
By default it's Raster fonts but if as me you have "Lucida Console"
font it has a better unicode support.

I post a *draft* script I'm working on to test in reply to this
message, chartable.ps1.

try ie:
.\chartable 850 (258..365)

If you haven't a console's font that support your code page and/or
unicode needs, Out-Image could be an "alternative" (big quotes here).

Regards,

--
Jean - JMST
Belgium

> I post a *draft* script I'm working on to test in reply to this message

#---8<---CharTable.ps1---DRAFT---Jean-JMST-Belgium---

# -codepage
# defines code page (default is current code page)
# -range
# defines range of chars (default is 0..255)
# -ucat (facultative)
# a unicode category to highlight
#
# ie :
# .\chartable 850 (0..127) "UppercaseLetter"

param(
[int]$codepage,
[array]$range=0..255,
$ucat
)

filter out{$_}

function get-codepage{
return [int]((chcp).Split(':')[1].Trim())
}

function is-unicodecategory([string]$c){
return [enum]::GetValues([globalization.unicodecategory]) `
-contains $c
}

$eunprint="`0","`a","`b","`f","`n","`r","`t","`v"`
," ",[string][char]160
$uunprint='`0','`a','`b','`f','`n','`r','`t','`v'`
,'SPC','NBS'
for($i=0;$i -lt $eunprint.length;$i++){
$h+=@{$eunprint[$i]=$uunprint[$i]}
}
if($codepage){
$ocp=get-codepage
[void](chcp $codepage)
}
if(is-unicodecategory($ucat))
{
[string]$add=[enum]::`
GetValues([globalization.unicodecategory])`
|where{$_ -eq $ucat}
}
write-host `
($add+(get-codepage).ToString().PadLeft(88-$add.Length))`n `
-BackgroundColor 'black' `
-ForegroundColor 'green' `
-NoNewLine

for($i=0;$i -lt $range.Length;$i++)
{
write-host $range[$i].ToString("0000") `
-BackgroundColor 'white' `
-ForegroundColor 'darkcyan' `
-NoNewLine
write-host ' ' `
-BackgroundColor 'gray' `
-NoNewLine
if($eunprint -contains [char]($range[$i])){
write-host ($h[[string]([char]($range[$i]))]).PadLeft(3) `
-BackgroundColor 'darkcyan' `
-ForegroundColor 'blue' `
-NoNewLine
}
elseif(is-unicodecategory($ucat)){
if(
[Globalization.CharUnicodeInfo]::`
GetUnicodeCategory(([string]([char]($range[$i]))|out)) `
-eq $ucat
){
write-host ([string]([char]($range[$i]))).PadLeft(3) `
-BackgroundColor 'darkcyan' `
-ForegroundColor 'green' `
-NoNewLine
}
else{
write-host ([string]([char]($range[$i]))).PadLeft(3) `
-BackgroundColor 'darkcyan' `
-ForegroundColor 'white' `
-NoNewLine
}

}
write-host ' ' `
-NoNewLine
if(($i+1) % 10 -eq 0){''}
}
''
[void](chcp $ocp)
#---8<---CharTable.ps1---DRAFT---Jean-JMST-Belgium---

#Regards,

--
Jean - JMST
Belgium

> post a *draft* script

Sorry ... to work "as is" you need to set -ucat parameter.

Regards,

--
Jean - JMST
Belgium

The Format-Hex script might be helpful --
http://www.leeholmes.com/blog/HexDumperInMonad.aspx.

It gives the ASCII codes of the characters, which may help clarify the
problem.

--
Lee Holmes [MSFT]
Windows PowerShell Development
Microsoft Corporation
This posting is provided "AS IS" with no warranties, and confers no rights.

"Andrew Watt [MVP]" <SVGDeveloper@aol.com> wrote in message
news:s25ml2din23clqqo4jlhbqudu97gk9pfjm@4ax.com...
> Marco,
>
> I'm wondering if this non-ASCII idea is a red herring.
>
> Both the line feed and carriage return are in the ASCII character set.
> As I understood things you were getting a message about end of line.
>
> Andrew Watt MVP
>
> On Wed, 15 Nov 2006 07:59:36 -0400, "Marco Shaw" <marco@Znbnet.nb.ca>
> wrote:
>
>>What to do, what to do...
>>
>>When non-ASCII characters creep into a script, what's the best way to see
>>them and remove them? Without re-typing the whole text.
>>
>>Grasping at straws doesn't seem to work:
>>gc psa1.ps1|out-file -encoding ascii psa2.ps1

i strongly disagree from my experience and understanding of the console
subsystem. at least when it comes to utf16 and asian font/code pages...

i.e with lucinda console.. my japanese and chinese work fine in
notepad!, but not on the console (when set with that font).. yhowever
in powershell analyzer they work fine.. I know with code pages that
MultiByte CharacterSystems work fine if you have an os set up that way
(i.e cmd.exe and powershell.exe will show MBCS ( MultiByte Character
System: in contrast to unicode) fine on a japanese system)

I so wish that you were right though..
also unicode IME's don't work at the console either. i.e i can't type
in chinese or japanese at the console.

> i strongly disagree from my experience and understanding of the console
> subsystem. at least when it comes to utf16 and asian font/code pages...

I undestand you better now :-)

Regards,

--
Jean - JMST
Belgium