can i preprocess text for ssis?

as background, i have a conditional split transform in front of several
derived columns, the condition is based on what characters appear at position
60 in each line. this was needed because the state changed the file format,
but the files are all stored in a shared way, we cant move or change them.
there are 440 files of several hundred thousand lines, so inspection is not
a possibiltiy, the spec was fine for spelling out the 'switch' characters for
most of the files, but we'd like to do kind of a select distinct to show each
character at position sixty for each line in each file, so we are sure that
the conditional split transform has accounted for all the switch characters,
because we suspect that there are other characters in that position that we
need to trap, (they dont error, but they dont find a home in the distribution
either) but dont want to sort though all the text files to find the unknown
character(s) at position sixty.

the other thing is related, but on different text files from the unix side
of the house, to be able to insert a CRLF at position 1201 in a file that has
no line feeds, it just one giant line, so we can chop it into 1200 byte size
lines each. i am sick of them telling me to open it in wordpad to fix it, and
that gets old fast too.
i have no idea how to go about doing these two jobs, but they would really
move the rock.
thanks very much
drew

Does this work for you for your 2nd question?

#Get the contents of the file
$string=get-content test.txt
#Loop here, start with $i=0, and increment by 4.
#You'd do perhaps $jump=1200
#And stop and the end.
$jump=4
for($i=0;$i -lt $string.length;$i=$i+$jump)
{
#Get the substring, after moving up
#to the current $i position, and getting
#the next $jump characters.
$cur=$string.substring($i,$jump)
#Echo out
$cur
#Or write to a file, but uncomment
#$cur|add-content -path test1.txt
}


--
Microsoft MVP - Windows PowerShell
http://www.microsoft.com/mvp

PowerGadgets MVP
http://www.powergadgets.com/mvp

Blog:
http://marcoshaw.blogspot.com

Hal Rottenberg wrote: And pipe to set-content to create a new file.

Get-Content unixfile.txt | % { $_.Insert(1200, "`n") } |
set-content dosfile.txt



--
Hal Rottenberg <hal@xxxxxx>
Author, TechProsaic (http://halr9000.com)
Webmaster, Psi (http://psi-im.org)
Co-host, PowerScripting Podcast (http://powerscripting.net)

Hi
In working through this, (after making the changes to 1200 and add-content)
i get an error that says [System.Object[]] does not contain a method called
substring at 14:26.
What is peculiar is that it worked the first time as you wrote it ($jump = 4
writing to the console) but this error materialzed when I set $jump=1200 and
wrote it to a file.

thanks very much for bearing with me, using it is the only way to learn it!
drew


"Marco Shaw [MVP]" wrote:

drew wrote: Does the file contain sensitive data? Maybe you could send me a copy?

marco DOT shaw AT gmail DOT com

Marco Shaw [MVP] wrote: The thing Hal and I didn't think of is that your original file, is a
single line.

get-content always seems to produce an array, by default. So if you
have a single line, get-content is giving you an array object. The
array object doesn't have a insert or substring method.

You have to specifically cast to a string:

[string]$content=get-content my_one_liner_file

Then, you have the string methods available for $content.


--
Microsoft MVP - Windows PowerShell
http://www.microsoft.com/mvp

PowerGadgets MVP
http://www.powergadgets.com/mvp

Blog:
http://marcoshaw.blogspot.com