Hosting Powershell in app -- Question about Pipeline

Hi all,

I have a question regarding embedded Powershell within an
application.

I'm currently working on a rules engine which, as one of its rule
types, allows the user to use a Powershell script as a rule.

These rules are used in the context of applying flexible business
rules to the processing of large sets of data from a database, and get
executed at least once for each database row, for databases with row
counts of around 20 - 40 million rows. There may be multiple
Powershell rules per row (depending on the rule set).

So, it is in my interest to optimize the embedded Powershell runtime
as much as possible.

Currently, I do that by using a single instance of a Runspace, and
create a new Pipeline for each rule/script, Invoke it, and check it's
exit code for the rule's "result" which is then used by the calling
rule engine.

I would like to reuse the same Pipeline object, in order to reduce the
creation overhead associated with newing up that object for every rule
times every row of database I need to process... However after once
being invoked, there seems to be no way to reset a Pipeline object to
it's initial state (PipelineState.NotStarted).

Ideally, I would just change the script text and call invoke multiple
times on the same Pipeline object... But there doesn't seem to be any
way to do that.

All of this is in the goal of improving speed of the script evaluation
when called many times repeatedly. My current implementation executes
a simple script (a few lines doing basic comparison logic) in about 1
millisecond... Unfortunately, this is extremely slow compared to the
other rules types in the engine, which execute at roughly .001
milliseconds per rule (average performance). So, the Powershell rules,
at the moment are about 1000 times slower than the normal rules... I
don't expect to get that number to match the normal rules, but I would
like to improve it somewhat. I feel that if I didn't need to new up
the Pipeline objects I might be able to reduce the execution time by a
factor of 10.

Is there anyone out there who knows how I could do this, and if not,
do you have any ideas about how I could get around this issue?

Thanks,
Troy

I asked this of others...

Don Jones (MVP):

"You can reuse a runspace, but you're supposed to create a new pipeline
for each new command you want to run - just as the console
PowerShell.exe app does it. There's not that much overhead involved in
spinning up a pipeline; it's getting the shell itself loaded and
running initially that's the bear. PowerShell unfortunately isn't
speed-optimized at this point."

Karl Prosser (MVP):

"Each time you run something its a new pipeline.. what i would recommend
is batching things, but modifying the script...
basically take 100 or more objects at a time, have it the input object
to the pipeline, and have your cmdlet/function work asa pipeline,
running the rule and processing it.. this the code is only parsed one,
and its not jumping in and out of powershell.. but the prefiormance gain
will not to be too significant.... the question are you querying the
database for each item.. that is one sort of latency point.. depending
on the architecture there are many different things you can do to help
it be better, sometimes that might be using multiple runspaces to run
stuff in parralell, othertimes not.

but powershell really is the slowest kid on the block preformancewise"


--
Microsoft MVP - Windows PowerShell
http://www.microsoft.com/mvp

PowerGadgets MVP
http://www.powergadgets.com/mvp

Blog:
http://marcoshaw.blogspot.com

On Jan 10, 1:29*pm, "Marco Shaw [MVP]" <marco.shaw@_NO_SPAM_gmail.com>
wrote: Thanks alot for the quick response. This is kind of what I thought the
answer would be... I'll see if I can find some other ways of speeding
it up.

Well, since it's already there, I left the support for Powershell
scripting in, but I added support for C#, VB.NET, and JScript .NET as
well. For those I just use a CodeDom.CompilerProvider and generate
assemblies in memory from the user supplied source code.

That way, if someone really wants to use Powershell code, they can,
but it will be slow... If they want a faster script option they can
use one of the above three languages... A reasonable compromise, I
guess.

Thanks,
Troy