Suggestion: New special variable for pipeline enumeration index

A while back I posted a suggestion on to MS Connect regarding the
possible addition of a special variable to numerically signify how far
through an iteration of input items a pipeline current is.

https://connect.microsoft.com/feedba...8135&SiteID=99

Has any thought been given to implementing this?


In summary I'm lobbying for the addition a new variable (such as $%),
similar to $_, that will would be an integer value holding a count of
the number of items that a pipeline has processed.

It would be really useful to have such a simple means of determining
how far though a pipeline you are.

Michael Lewis

I'm not sure I see the use for this in the grand scheme of things ...
since you don't know how many there will be, it's of limited use,
right? But anyway, any cmdlet or script/function that wants to know
how many items it has processed in the current pipeline can already
count them on it's own.

On Apr 9, 10:23*am, MichaelL <goo...@xxxxxx> wrote: I can't see the page you refer to but I also don't see how knowing
the number of records already processed (which your function can
count itself pretty easily) can tell you how far along you are.
Knowing how many are left would require that you somehow know how
many objects will ultimately be processed. Such information is, in
most cases I can think of, simply not available. It's like asking
how many words I will type today. Moreover, this 'count' could be
different at every step in the pipeline. Maintenance of all these
counts represents an amount of overhead I'd just as soon not have
every application I write saddled with. Especially when I can't
think of even one practical use much less enough uses to consider
that it might possibly be justifiable.

In certain circumstances I can see that this would\could be useful but the
overhead of maintaining it I think is too high. It would also mean that each
step in the pipeline had to complete its processing before passing data along
otherwise subsequent cmdlets wouldn't have an accurate count. This could be
a severe performance hit
--
Richard Siddaway
All scripts are supplied "as is" and with no warranty
PowerShell MVP
Blog: http://richardsiddaway.spaces.live.com/
PowerShell User Group: http://www.get-psuguk.org.uk


"RickB" wrote:

On Apr 9, 4:54*pm, RickB <rbiel...@xxxxxx> wrote: You're misreading me.
Clearly, knowing how many records are yet to come in the future is not
something that can be know - but that's not what I was suggesting.

Rather I was suggesting that a count be kept of how many records have
been processed thus far, through the current pipeline step.

This would be exactly analogous the the NR variable in awk (another
record processing language).

Granted, if you've not much file or log processing in powershell then
I agree the usefulness of this feature may not be obvious to you.

And as to the overhead of maintaining this count - it's a single
integer which the pipeline could increment every time it calls
ProcessRecord. By any measure that cant be considered too expensive
to maintain given all the other processing that goes on in a pipeline.

On Apr 12, 7:46*pm, RichS [MVP] <RichS...@xxxxxx>
wrote: Sorry Rich, I didn't explain my suggestion adequately.

A single integer, incremented once each time that ProcessRecord is
called, would not be a performance hit of any significance at all.

I guess the obvious implemetation would be to make this integer a
member of the Cmdlet class (although implementing it in the same may
as $_ would also be appropraite).

And much like $_, this proposed variable would only exist in the
context of the current pipeline step. It would not imply that any
other processing needs to complete as you suggested. And since it
only exists in the context of the current command, it wouldn't get
passed on to subsequent cmdlets, so there's no overhead there either.


To illustrate... if this counter were accessible as $%, then....

dir | % { $%, Name }
would emit...
0
bin
1
cygwin
2
dell
3
Dev
4

dir | ? { [int](%_/2) -eq ($%/2) }
would emit every alternate record

(These examples are meant to show behaviour, not usefullness)

Michael

Correction to that first example...

dir | % { $%, Name }
should read,
dir | % { $%, $_.Name }

On Apr 15, 3:36*am, MichaelL <goo...@xxxxxx> wrote: But even here, there must be a place to store it, meaning the integer
must be a property on some object associated with every single step in
every single pipeline of every user everywhere, just because you
don't
feel like adding a counter to the step where you happen to need
counts.

You could even create a step in your pipeline that attaches a
noteproperty
to each object with it's sequence. Then you would have it everywhere
without even having to maintain counters everywhere.

get-sourcedata|add-counter|all-my|other-steps|?{<criteria}|add-counter|
more-steps....

You could even sort your objects and know the original order unless
(as in my example) you updated the counter values by piping them thru
add-counter a second time.

Personally I'm in the process of developing a script that, under some
circumstances, might have a great many steps in progress at any given
time and I don't see anywhere in my system where such information
might
be an advantage to 'have at my fingertips'.

I know your examples were "meant to show behaviour, not usefullness"
but seeing as how I may need to incur the memory, performance, and
code
bloat of another property on every pipeline, I don't think it's
terribly out of line to ask for some real usefullness to the general
population of PS users.

Rick already said this, but I'm going to say it again, differently.

There can't be a global variable that represents the "current"
pipeline index, it would have to be "attached" to the pipeline
objects. Let me illustrate in my traditional manner... (and please,
stick with me, even though this is going to look a mess).

# Create and increment $i and write it out
1..6 | &{BEGIN{$global:i = 0}PROCESS{
$global:i++;
$_
}} | &{PROCESS{ # Print out $i
Write-Host ("{0:0.0} $global:i" -f $_) -fore yellow; $_
}} | &{PROCESS{ # This cmdlet outputs 2x
$_ -.5; $_
}} | &{PROCESS{ # Print out $i
Write-Host ("{0:0.0} $global:i" -f $_) -fore green; $_
}} | &{BEGIN{$x=@()} PROCESS{
$x+=$_;
if($x.Count -gt 5){ # stalls the pipeline
$x;
$x=@();
}
}} | &{PROCESS{ # Print out $i
Write-Host ("{0:0.0} $global:i" -f $_) -fore cyan;
}}


If you run that, you'll see that $i increments and prints correctly on
the first (Yellow) Write-Host, but on the second (Green) one, an extra
pipeline element has been introduced, so the $i counter is wrong (and
getting wronger by the minute) ... and on the last (Cyan) one, you'll
only see $i at the multiples of 3.

Even attaching a counter to the pipeline items as they're created
wouldn't work (I left this out of the example, but there could have
been a cmdlet like WHERE-OBJECT which absorbed some of the items) --
PowerShell would have to create and increment a SEPARATE COUNTER for
EACH ELEMENT of the pipeline (that is, for each cmdlet, function,
etc), and in order for it to work properly, it would have pass that
variable to you somehow (maybe by attaching it to the pipeline input
objects?), but it would have to track it itself, and pass it
separately to each pipeline element, because there's no guarantee the
counters would stay correct when they come out the other end of the
element...

So, like I said before ... any cmdlet or script/function that wants to
know how many items it has processed in the current pipeline can
already count them on it's own -- having PowerShell count them would
be complicated, resource intensive, and would basically slow the
pipeline down without providing any additional value.

On Apr 15, 10:56 pm, RickB <rbiel...@xxxxxx> wrote:
Simple - store the integer as a member of the Cmdlet class. The memory
(and cpu) overhead would be minuscule.

And you're focusing on the wrong thing - you're considering this as
though it were a counter of objects. My suggestion for $% would be a
counter of operations on objects, a logical supplement to $_. And
just like $_ it is transient and provides transient information while
processing the pipeline - but this information can then be used
wherever beneficial. Your arguments against it would equally apply to
$_ itself!
Granted that annotating the objects themselves, as you have suggested,
is trivial enough, but doing so addresses a somewhat different use
case. It's also a very expensive operation.

What I'm suggesting is massively cheaper, in terms of both cpu and
memory, (vanishingly so in fact), is trivial to implement, is highly
generic and provides a feature that is directly analogous to one of
the most commonly used in other text processing languages. (such as
'NR' in awk) It's usefulness in such languages is proven and
incontrovertible.

What I'm suggesting isn't going to change the world because, and, yes,
the effect can be simulated in other ways, but at either much greater
cost or significant inconvenience. However improving the feature-set
of a product such as powershell involves comparing the cost of a
feature to the possible usefulness. I'm suggesting provide easily
accessible information about the activity of the pipeline which
typically relates to a "real-world" meaning of the entities that are
being processes, and does so with virtually no cost.

All your arguments have been centered on misunderstandings that have
led you to be inappropriately concerned about a performance hit that
doesn't exist.

And arguing, as you have done, that a feature has no usefulness
because you don't do the kinds of operations where it would be useful
is like saying no one could possibly have a need for Excel because all
you do is write letters.

Michael