Friday, June 28, 2013

How to throttle workflow activites in Powershell V4

With the release of Powershell V4, there is a new interesting feature concerning the use of Workflows: the ThrottleLimit property. Until now, there was a fixed hard limit to 5 concurrent activities when a workflow was started with the -parallel switch. This limitation was largely discussed by Jeff Wouters on his blog a few months ago. Jeff also took the time to develop a cool workaround.

Since the throttling of threads has been now built-in inside Workflows, I decided to give it a try.

I wrote a small Throttle-Me workflow which accepts an integer parameter and passes it to 'foreach -parallel'. Inside the foreach block I added two commands. The first one shows the value of a counter and the second one the current time in the form hh:mm:ss. This kind of output makes it easy to understand the way the ThrottleLimit property works because the current time is showed in groups the size of the ThrottleLimit parameter.

Let's start to have a look at the code (it's just an easy example wrote for this purpose, and, to be honest, I re-used part of the code which I saw on a recent Mike's post)
workflow Throttle-Me {

[cmdletbinding()]
    param(
        [int]$ThrottleLimit = 5
        )

foreach -parallel -throttlelimit $ThrottleLimit ($n in 1..5){
   "Working on $n"
   "{0:hh}:{0:mm}:{0:ss}" -f (Get-Date)
   }

}
If we call the Workflow with the limit set to 1 here's what we get:
throttle-me -throttlelimit 1

Working on 1
11:10:21
Working on 2
11:10:21
Working on 3
11:10:21
Working on 4
11:10:21
Working on 5
11:10:21
As you can see, one thread is executed at the time and each group of results (counter + date) is returned before the following thread starts.

If we change the limit value to 2 we get:
throttle-me -throttlelimit 2

Working on 2
Working on 1
11:11:38
11:11:38
Working on 3
Working on 4
11:11:38
11:11:38
Working on 5
11:11:38
Here two threads are started in parallel and the output contains two groups of results. The only exception is the last thread which comes alone for evident reason.

Same with the Throttlelimit set to four:
throttle-me -throttlelimit 4

Working on 4
Working on 3
Working on 2
Working on 1
11:12:16
11:12:16
11:12:16
11:12:16
Working on 5
11:12:16
Or to five:
throttle-me -throttlelimit 5

Working on 5
Working on 4
Working on 3
Working on 2
Working on 1
11:12:47
11:12:47
11:12:47
11:12:47
11:12:47
Now you aks what is the limit of concurrent runnable threads, and, while the real answer depens on your hardware configuration, the theoretical answer is given by the type of value the ThrottleLimit property accepts: Int32.

The largest possible value of Int32 is 2147483647, so if we run it with an higher value we get:
throttle-me -throttlelimit  2147483648

throttle-me : Cannot process argument transformation on parameter 'throttlelimit'. Cannot convert value
"2147483648" to type "System.Int32". Error: "Value was either too large or too small for an Int32."
At line:16 char:29
+ throttle-me -throttlelimit  2147483648
+                             ~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [throttle-me], ParameterBindingArgumentTransformationExceptio
   n
    + FullyQualifiedErrorId : ParameterArgumentTransformationError,throttle-me

In any case take the time to measure the load of, say, 10 threads and see how your hardware reacts. Know that the reaction will depend on what you asked your workflow to do inside the foreach block.

That's all for the ThrotteLimit property. I will come back on Workflows in Powershell v4 in a future post.

8 comments:

  1. Great! Well explained and easy to understand.
    Thanks

    ReplyDelete
  2. Even if I change the throttlelimit to 1000 it still limits to 5 at a time.

    workflow Test-ping
    { 
    param(

    [parameter(mandatory=$true)]
    [int]$ThrottleLimit

    )

    $servers = Get-Content -Path servers.txt
    foreach -parallel -throttlelimit $ThrottleLimit ($server in $servers){Test-Connection -ComputerName $server -Count 1}

    }

    ReplyDelete
    Replies
    1. Hi Raghav,

      that's interesting, I tested your code and it works like a charm....

      Can you confirm that you are using Powershell 4.0? Do you mind comparing the output of $psversiontable with mine:

      PS C:\Windows\debug> $PSVersionTable

      Name Value
      ---- -----
      PSVersion 4.0
      WSManStackVersion 3.0
      SerializationVersion 1.1.0.1
      CLRVersion 4.0.30319.33440
      BuildVersion 6.3.9600.16384
      PSCompatibleVersions {1.0, 2.0, 3.0, 4.0}
      PSRemotingProtocolVersion 2.2

      Carlo

      Delete
    2. PS C:\> $PSVersionTable

      Name Value
      ---- -----
      PSVersion 4.0
      WSManStackVersion 3.0
      SerializationVersion 1.1.0.1
      CLRVersion 4.0.30319.18052
      BuildVersion 6.3.9421.0
      PSCompatibleVersions {1.0, 2.0, 3.0, 4.0}
      PSRemotingProtocolVersion 2.2


      I'm running this on 2k8 r2 sp1.

      Delete
    3. Ok, I suppose that it works for me because I ran it on Windows 2012. Probably you have Powershell 4.0 which has the throttlelimit switch, but for some reasons Windows 2K8 behind the scenes does not leverage it. Remember that workflows aren’t actually executed by PowerShell, but by Windows Workflow Foundation (WF), which is a piece of .NET.

      Delete
  3. That Jeff Wouters blog link doesn't work anymore as the site is bust. I found a copy on the wayback machine though; https://web.archive.org/web/20131002095108/http://jeffwouters.nl/index.php/2012/11/powershell-workflow-foreach-parallel-limited-to-5-parallel-threads/

    ReplyDelete

Related Posts Plugin for WordPress, Blogger...