working with pipline

Working with Pipelines

 A Pipeline Example

The pipeline is an old concept developed in the UNIX command language.  It is a series of commands connected by pipeline operators “|”. Each pipeline operator sends the results of the preceding command to the next command.  The result is a very powerful command chain or "pipeline" that is comprised of a series of simple commands. where the output of one command becomes the input for the next, like stages on an assembly line.  As the data passes through the pipeline it is refined at each stage.  In a pipeline, the commands are processed from left to right in the order that they appear. The processing is handled as a single operation and output is displayed as it is generated. For example:

Type:  get-process taskmgr | Stop-Process

Parameter binding is done either by “by value”, which means that the piped object must be of the same type as the parameter object or be able to be converted to it, or “by PropertyName, which means the piped object must have the same name as the parameter of the input command.

To find out which parameters are able to be used in a pipeline, type the following:

Error Checking the Pipeline

Type: get-help stop-process –parameter *

Figure 2: Finding Cmdlet Parameters

We see that the Name parameter accepts pipeline input, but only as a propertyname which means that the get-process command must find the string name of the process and pass that to the name parameter of the stop-process.  We can see the pipeline process in real time using the trace-command.  This is an excellent command to debug the pipeline process when errors occur.

Type:  trace-command -name parameterbinding -expression {get-process taskmgr | `
stop-process} -pshost -filepath debugTaskmgr.txt

Note: the line continuation character “`” when you have long lines of text.

                Figure 3: Using the Trace-Command to Discover Pipeline Errors

Open the debugTaskmgr file.  We can see that the binding failed because the name parameter “taskmgr” could not be found by the get-process cmdlet so the Stop-Process name parameter had nothing to bind to and could not execute.  The reason of course is that we didn’t open the application Task Manager before running the command. Open Task Manager and run the

 Get-Process taskmgr | Stop-Process pipeline again.

Notice that each command in the pipeline is executed separately; the get-process command searches the running processes and finds the name and ID of the taskmgr process and passes this information to

Figure 4: Get-Process Taskmgr Output

the stop-process command. The stop-process binds the input name to its property name and then uses the associated ID number to close the program.

By combining cmdlets together using the pipe operator, we can create powerful compound commands that process the data as it travels through the pipeline.  A graphical example is below.  Suppose there was a cmdlet called get shapes.  This command returns a collection of object shapes installed on the local system. 

|

Get-Shapes    |

|OUT-FILE OrangeShapes.txt

SORT-Object

WHERE-OBJECT { $_ .Color -eq  “orange”}  |

Figure 4:Graphical Example of Pipelining

The output of this command becomes the input for the Where-Object cmdlet which filters the collection based on the expression contained in the script blocks.  In this case, it selects only the objects which are equal to orange in color.  Notice a new variable “$_”; this is an automatic variable which is created by PowerShell to hold each item in the pipeline.  Each item in the collection is processed one at a time; then the output is piped to the Sort-Object cmdlet which arranges the orange objects in ascending order.  This sorted list is then sent to the Out-file cmdlet which takes the input and formats it into a table contained in a file.  You can see that when working with pipes the order of the commands is important, for example you can’t  Sort-Object | Get-Sharps, or Out-File| Sort Object.

A Practical Pipeline Example

A typical network job is finding specific processes running on a local system.  The Get-Process cmdlet returns a collection of running processes.  Suppose you wanted to find all the Internet explore processes and output the information to a file.

Type: Get-Process
Type: Get-Process | Where-Object { $_.Name –eq “iexplore”} | Sort-Object | Out-File IEProcess.txt

Note: an alternate method to use the “`” escape character as an end of line marker, is to press the enter key after a command and PowerShell will display an additional line prompt”>>” to complete the command.

The next command Sort-Object filters the collection looking only for iexplore processes.  Sort-Object takes the entire filtered collection and sorts it ascending by name, which is the default, since we did not add any parameters.  The Sort-object command is called a sequential mode pipeline because sorting cannot begin until the entire collection has been received, working on each item one at a time.  This mode is slow and hogs memory because results are returned only after all commands finish their work and the pipeline has to store the entire results of each command.  Normally, however, the pipeline works on each item-by-item in what is called streaming mode.  In this mode, every single result is passed directly onto the subsequent command. This mode is quick and saves memory because results are output while the pipeline commands are still performing their tasks; the pipeline only stores a single item at a time in the $_ variable.  The final result is piped to a file called IEProcess.txt

 We need to be careful when pipelining specially on commands like Sort-object   because if the collection is large, it can take long time to complete.  PowerShell will look like it is frozen and you could run out of memory and crash your system.  Commands like Get-ChildItem C:\ -recurse | Sort-Object  --- Would likely crash your system-- Do Not execute this command!

Using the Out-Host –Paging CmdLet to Save System Resources

The Out-Host -Paging command is a useful pipeline element whenever you want to display a lengthy output slowly. It is especially useful if the operation is very CPU-intensive because processing is transferred to the Out-Host cmdlet page by page, and the cmdlet that precedes it in the pipeline halt their operation until the next page of output is available. You can see this if you use the Windows Task Manager to monitor the CPU and memory consumed by Windows PowerShell. For example, run the following commands:

Type: taskmgr  . Click on the Performance tab to monitor CPU usage.

Type: Get-ChildItem C:\windows –recurse    Record the usage

Type: Get-ChildItem c:\windows –recurse | Out-Host –paging  Record the usage.

                Figure 5: Comparing CPU Usage Using Out-Host -Paging Cmdlet

The Out-Host –paging command provides a significant saving in system resources, but requires user interaction during the pipeline to process each page.

Modifying Pipeline Execution: Whatif Confirm and Suspend

PowerShell provides some common parameters to modify the execution of the pipeline.

Common Parameters[1]
Parameter	Meaning
-whatif	Tells the cmdlet not to execute; instead it will tell you what would happen if the cmdlet were to actually run.
-confirm	Tells the cmdlet to prompt prior to executing the command. .
-verbose  .	Instructs the cmdlet to provide a higher level of detail than a cmdlet not using the verbose parameter.
-debug	- Instructs the cmdlet to provide debugging information.
-erroraction	- Instructs the cmdlet to perform a certain action when an error occurs. Allowable actions are: continue, stop, SilentlyContinue, and inquire.
-errorvariable	Instructs the cmdlet to use a specific variable to hold error information. This is in addition to the standard $error variable.
-outvariable	Instructs the cmdlet to use a specific variable to hold the output information.
-outbuffer	Instructs the cmdlet to hold a certain number of objects prior to calling the next cmdlet in the pipeline

For example, suppose we want to stop all notepad processes on the local system.  To illustrate open two instances of notepad:

Type: notepad
Type: notepad

Type: Get-Process |Where-Object { $_.Name –eq “Notepad”}

Figure 6: Output of the Notepad pipe

This confirms two sessions are currently active. Now suppose we type a command like the following:

Type:   Get-Process | Where-Object { $_.Name –like “n*”} | Stop-Process –whatif

Figure 7: Using the Whatif Common Parameter

Your system display will be different than the above screenshot.  The –whatif parameter gives you the opportunity to model the command prior to execution showing you what will happen.  Now you can modify the command to get the desired result.

The –confirm parameter is similar to the whatif, but prompts you for a yes selection prior to execution.

Type:  Get-Process | Where-Object { $_.Name –like “n*”} | Stop-Process –Confirm

Figure 8:Using the Confirm Common Parameter

 Adding the confirm parameter also provides a “quick and dirty” menu for user preference. You can also use the –suspend parameter which suspends the execution of the command and opens a nested command prompt so you can execute another command.  When that command is finished, you type exit to return to the suspended command.  For example:

The use of whatif, confirm and suspend parameters are essential  tools for administrators. Using them to model commands before execution can save hours of work each year.

Type: Get-Process | Where-Object { $_.Name –like “n*”} | Stop-Process –Confirm

Type: press s for suspend -- notice the prompt now looks like this “>>”. This means a child shell has been launched inside the parent shell.

Type: mspaint

Type: Get-Process | Where-Object { $_.Name –eq “ms*”} | Stop-Process -confirm

Type:  8   (enter key)

Type: exit – this returns to the parent shell and the suspended command

Type: 8   (enter key) twice to close the 2 notepad sessions.

Using the Tee-Object and Pass Through Parameters

So far the pipeline is one linear action, however, there are times when an administrator needs to access the items travelling through the pipeline.  The most common example is logging.  Suppose you are administering a network and you want to get a listing of all the bad processes which will be stopped.   If you wait to the end of the pipeline, it’s too late to create a log file.  The bad processes don’t exist anymore.  You need to create the log file in the middle of the pipeline.  This is a job for the Tee-Object; it makes the pipeline go in two directions at once.

Type: mspaint

Type:  Get-Process mspaint| Tee-Object  -file kill.log |Stop-Process

Type: Get-Content kill.log

An alternate method of saving the bad process to a log file is to use the passthru parameter with the Stop-Process cmdlet.  This is a linear process, but the passthru parameter allows the system prompting to pass through the pipeline which can be captured to a file at the end of the pipeline.  Without the passthru parameter or the Tee-Object cmdlet there would be nothing to log after the Stop-Process command finished.

Type:mspaint

Type: Get-Process mspaint| Stop-Process –passthru > kill.log

Measuring and Comparing Objects

A common scenario for a network administrator is to provide the number, maximum and minimum size of files and comparing the differences between two objects.  The first procedure is best performed with the Measure-Object cmdlet.  For simple statistics, this is the best choice.  It is much easier to use than a foreach loop.  The compare-Object cmdlet uses one object as a reference to find differences in the other object.  This is an excellent tool to compare scripts when one is working and the other is not.

The Measure-Object cmdlet calculates three types of measurements on objects. It can count objects and calculate the minimum, maximum, sum, and average of the numeric values. For text objects, it can count and calculate the number of lines, words, and characters.

Make sure you are in your documents folder.

Type: Get-ChildItem  –recurse |Measure-Object –property length –min –max –average –sum

The command looks at the length property of all the files in the documents folder and subfolders and calculates the minimum length, maximum length, average and sum of all files.

Type Get-Content echoname.cmd | Measure-Object –character –line –word

All scripters will eventually have to compare two objects or collections to find the differences, things added or deleted.  The Compare-Object cmdlet knows how to loop over the objects or collections, without having to write an iterative control structure.  One object is declared a “reference set” which Compare-Object uses to identify the “difference set”. Let’s compare the two scripts we wrote in week 2 echoname.cmd and echoname.ps1.

Type: Compare-Object  -referenceobject $(get-content .\echoname.cmd)8

 –differenceobject $(get-content .\echoname.ps1) –includeequal