Monday, February 20, 2017

Ramp up your PowerShell knowledge in 2017 with these books

For once I am going to write a blog post which is not focused on a technical subject, and for a reason. As far as I have been able to observe, there is still a great amount of IT administrators who aren't using Windows PowerShell as their main tool for server administration even do Microsoft has been pushing it with all of its strength. Sometimes the reason is that IT guys simply lack of time. Sometimes they can't find motivation from the surrounding environment, because procedures and tools are already in place. Sometimes they are scared by the extent of the change that comes from switching away from a GUI-based server administration.

TIME TO GET A GOOD BOOK

What I think, is that often the best starting point is a good book to take on the challenge, like the one of learning PowerShell: a good book helps you to move your first steps with the language, and then to get a solid understanding of its capabilities. It will teach you how it works, how it binds to the operating system and how you can best benefit from its usage. And it will keep your motivation high by giving real world examples that you can start using straight away.


Now, if you look, say, on Amazon, you'll find tons of PowerShell books: 662 items on 56 pages, and if you're a novice, you might very well get stuck here because you don't know the authors and you can't make a choice of what book is the best for you.

That's why I have decided to write a post to mention the books that in 2017 will be a must for the modern system administrator. Most of them are still work-in-progress, but will be achieved in 2017 and will definitively be worth their price not only because they will be focused on the most recent PowerShell version (5/5.1), but also because their authors are well-renowned book writers and conference speakers that know not just how to teach a technical subject, but also how to stimulate you interest in the language. What's more, some of them, through their initiatives, have been the real engine behind the widespread of PowerShell as the vital skill to have in 2017: I am thinking for instance to Bruce Payette (principal author of the language) and Don Jones (founder of PowerShell.org) just to name two.

Let's now have a look at this list of books.

WINDOWS POWERSHELL IN ACTION

With 14 out of 19 chapters already available and planned to be published in April 2017, the must-have book in 2017 is the one by Bruce Payette and fellow MVP Richard Siddaway titled Windows PowerShell in Action, Third Edition (ISBN 9781633430297 - $59 for the printed version) which Manning makes available through the MEAP program. This is program that allows buyers to have access to the book chapters as soon as they are ready, so that their content is not locked up by a long writing and publishing process.


This book, which in its second edition already boasted 983 pages, has the advantage of being the most complete book around on this subject. This third edition covers massively some pretty hot topics such as PowerShell Classes, Workflows and Desired State Configuration (aka DSC) and should therefore be used by the system administrator who already uses PowerShell and who wants to build a rock-solid knowledge of the language. Furthermore, being Bruce Payette a founder of the language, you will get an insight of many design decisions that the PowerShell Team has had to make.

Here's a sample screenshot from the free sample of this book, just to show you the level of detail in its first pages:


It's also worth noticing that Manning did the smart move of providing a forum where you can give feedback on the content on the book as it's being written. You can access it here.

THE POWERSHELL SCRIPTING AND TOOLMAKING BOOK

The second must-have book is for sure the one by Don Jones and Jeffery Hicks titled The PowerShell Scripting and Toolmaking Book. We already know Don Jones for being the co-founder of PowerShell.org and the author of the blockbuster Learn PowerShell in a month of lunches. Jeffery Hicks is also a well-known author of PowerShell books (whose listing you can find on his blog).


As you can see when you follow the link above, they have chosen LeanPub to publish their book. LeanPub is a platform that allows technical authors to ship chapters in a Agile-manner, like you would on a blog, similarly to the MEAP program we presented above.

For the moment, their book is 80% complete and they set a target selling price at $60, but you can freely decide to pay it anything between $40 and $120. The price could seem a bit high, especially if compared to the book by Bruce Payette and Richard Siddaway, but you can be assured of two things: the first one is that Don Jones and Jeffery Hicks are excellent authors that know how to teach a subject. The second one, is that their books comes as a 'forever edition', meaning that all the updates that the authors will make to the content in the future are included in that price.

Concerning the content, the book is going to provide you with in-depth information on how to build advanced functions that include professional-grade parameter management, error handling, and built-in help, like real cmdlets. Other hot topics in this book are Unit Testing with Pester, Source Control, PowerShell ClassesPowerShell Script Analyzer and Just Enough Administration (aka JEA), just to mention a few. It will also teach you other interesting things such as how to publish to the PowerShell Gallery, as you can see from the screenshot taken from the free sample:


Same as with MEAP, LeanPub gives you the possibility to give feedback to Don Jones and Jeffery Hicks on the content of the book through a specific web page which you can find here.

POWERSHELL 101 - THE NO-NONSENSE BEGINNER'S GUIDE TO POWERSHELL

The third book I want to talk about and which will be published in 2017, is the one by Mike F Robbins. Mike is a former PowerShell MVP and now a Cloud and DataCenter Management MVP. He has already co-authored some books like PowerShell Deep Dives and Windows PowerShell TFM 4th Edition. Not to mention that is the winner of the advanced category in the 2013 PowerShell Scripting Games. So he knows his subject as you can see if you follow his blog.

The title of the book is PowerShell 101 – The No-Nonsense Beginner’s Guide to PowerShell and is published through LeanPub, just like the book by Don Jones and Jeffery Hicks. For the moment the first two chapters have been published and a third one is almost ready, so I am fully confident Mike will be able to complete it by the end of 2017 with highly valuable content.


Concerning the content, my understanding is that this book is aimed at Windows administrators that want to enter the PowerShell arena but with a focus on real world scenarios: this is a key point that will make the learning process smoother for those moving their first steps away from the GUI. Here's a screenshot from the free sample of the book:

Notwithstanding the fact that this book starts with real simple examples, like the one above, doesn't mean that you won't find a lot of very good hints on how to improve your scripting skills: Mike has a reputation for being able to write real complex functions that are extremely easy to reuse and, believe me, you won't be deceived by this book that has the added values of being sold at a very low price, $11.99.

Like for the others LeanPub books, you can find the feedback page for Mike's book at this link.

If you want to embrace PowerShell and develop your automation skills, these are the books to step up your game and add value to your career in 2017. Just choose one and skyrocket your performance.

Tuesday, February 14, 2017

Using the PoshRSJob module with a PowerShell function to perform a WinRM check up

As I explained in the previous post, I have written a function to test whether the WinRM service on a remote computer is able to accept connections and effectively execute commands. It all started with the finding that in very large environments with mixed OS versions, you can only be assured of the proper functioning of WinRM by trying to execute an actual command on a remote host. Only if the execution of that command succeeds you can safely state that the whole PowerShell remoting subsystem is correctly configured end-to-end.

The WinRM host process in action on the target server

Now you probably know that you have a couple of cmdlets that can be used to test this (I am thinking to Test-WSMan and Connect-WSMan), and that you can use Invoke-Command to run a block of code on a remote computer.

What I wanted to achieve here goes a bit further. I wanted a PowerShell function capable of testing all the possible configurations in a large environment with a high execution speed.

This involves testing things as the different authentication mechanisms such as Kerberos or NTLM, as well as testing against closed TCP ports, if any.

And this involves including some kind of parallelization.

Now to make a long story short, I have split the tests I perform in a function that I called Test-PSRemoting (whose full code you can find on my GitHub) in five blocks, where each block is accessed in the function through a Switch parameter.

Just to be clear, a Switch parameter is a parameter whose value is False by default and gets set to True when it is included. Here's a neat example I wrote to show you how this kind of parameter works:

function Get-SwitchValue ([switch]$switch1, [switch]$switch2)
{
 "Switch1 is $switch1."
 "Switch2 is $switch2."   
}

Get-SwitchValue -switch2

Switch1 is False.
Switch2 is True.
So as I said, there are five regions, which are only accessed if the corresponding Switch parameter is set to True.

The first Switch, named $Ping, is for the region of code where a ping is sent to the target server using the System.Net.NetworkInformation.Ping class:


The second Switch, named $Resolve, is used to query the DNS and return the IP address of the target server. This is accomplished with Resolve-DnsName with a query type set to A, so that only the IPv4 address is returned:


The third Switch, which I named $TCPCheck, is called when you want to check that the TCP ports used by the WinRM service are open on the destination server. As you might know, there are two ports for WinRM:
  • TCP port 5985 is for HTTP traffic and is used when you don't need to authenticate the target server because you can rely on Kerberos and on the Active Directory to authenticate it for you
  • TCP port 5986 is for HTTPS traffic and is used when you can't rely on an Active Directory  domain to authenticate the target server (like when it is in a Workgroup) and therefore you require that that target server identity is confirmed by a certificate issued by a trusted CA
Now, recent PowerShell versions have a native cmdlet for checking open TCP ports which is called Test-NetConnection. This cmdlet is designed in a way that it can be used to check for the standard WinRM port 5985 in a quick manner:
Test-NetConnection srv01 -CommonTCPPort WinRM
The issue is that this cmdlet seems always trying to ping the remote server before issuing the TCP connection. Since I haven't been able to determine whether using it with the InformationLevel parameter set to Quiet suppresses the ping, I have decided to fail back to use Windows Sockets through the System.Net.Sockets class provided in .NET framework. This has the important advantage of letting me use AsyncWaitHandle to handle timeouts shorter than the one of Test-NetConnection when used against a server which is unresponsive:
Measure-Command {([System.Net.Sockets.TcpClient]::new().BeginConnect('nosrv',5985,$null,$null)).AsyncWaitHandle.WaitOne(100,$False)}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 125
Ticks             : 1256004
TotalDays         : 1.45370833333333E-06
TotalHours        : 3.4889E-05
TotalMinutes      : 0.00209334
TotalSeconds      : 0.1256004
TotalMilliseconds : 125.6004

Measure-Command {Test-NetConnection nosrv -port 5985}
WARNING: Name resolution of nosrv failed -- Status: No such host is known

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 2
Milliseconds      : 306
Ticks             : 23065325
TotalDays         : 2.66959780092593E-05
TotalHours        : 0.000640703472222222
TotalMinutes      : 0.0384422083333333
TotalSeconds      : 2.3065325
TotalMilliseconds : 2306.5325
As you can see, the Test-NetConnection method is twenty times slower, and like I have said, speed is one of my main requirement for the function I am writing.

The next region, $Dcom, is where I check if DCOM can be used to retrieve the operating system of the target server as well as the name of the Active Directory Domain it belongs to. Actually this is kind of an optional part of my function: there's no link between WinRM and DCOM but it can always be interesting to know if you can switch back to DCOM/RPC to query the WMI provider on the remote host. Here's how I use the New-CimSessionOption to force my request to go over the DCOM protocol:
$SessionOp = New-CimSessionOption –Protocol DCOM
Also, always having that idea of making my function go fast, and to be robust in case the remote WMI provider is for some reasons broken, I use the New-CimSession cmdlet with a OperationTimeOut parameter set to 1 seconds. Here's the block of code for the DCOM check:



Now the final block of code. This is the part where I perform the following tests:
  • Test-WSMan with a challenge-response scheme named Negotiate, that allows me to authenticate the account I am using with Kerberos and to switch back to NTLM in case it fails
  • Test-WSMan with Negotiate on port 80, which is the old TCP port used for WinRM on Windows 2003 servers (and I have still a few of them in the place I am using this function)
  • Invoke-Command with Negotiate: in this case, since the cmdlet doesn't have a Timeout parameter, I run it in a background job which I discard after two seconds
  • Test-WSMan with Kerberos authentication
  • Test-WSMan with Kerberos authentication on port 80 for servers trunning Windows 2003 as base operating system
  • Invoke-Command with Kerberos authentication
As you can easily understand, the test with Invoke-Command is the most important part of the function since it effectively tries to retrieve the list of running services on the target server over WSMan:


Now that was for the Test-PSRemoting function.

Concerning the parallelization of the execution, I have gone down a few roads: first of all I have tried to build a quick and dirty RunspacePool but soon discovered that their implementation is so developerish that it goes well beyond what a system administrator should be able to know and understand. In the end I have decided to choose the easy path and reuse a module written and maintained by fellow MVP Boe Prox which adds a layer of abstraction to the runspaces below and makes it easy to use for the classic system administrator. The name of the module is PoshRSJob and you can find it here.

To install this module just run:

Install-Module -Name PoshRSJob
Version 1.7.3.5 has the following cmdlets:

Get-Command -Module PoshRSJob

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Function        Get-RSJob                                          1.7.3.5    PoshRSJob
Function        Receive-RSJob                                      1.7.3.5    PoshRSJob
Function        Remove-RSJob                                       1.7.3.5    PoshRSJob
Function        Start-RSJob                                        1.7.3.5    PoshRSJob
Function        Stop-RSJob                                         1.7.3.5    PoshRSJob
Function        Wait-RSJob                                         1.7.3.5    PoshRSJob
These cmdlets can be used in a oneliner fashion, just by piping Start-RSJob into Wait-RSJob and then into Receive-RSJob:

Start-RSJob -InputObject ('1/1/2017','2/2/2017') -ScriptBlock { Get-Date $_ } | Wait-RSJob | Receive-RSJob

Sunday, January 1, 2017 12:00:00 AM
Thursday, February 2, 2017 12:00:00 AM
Before you use my function, it is important to understand how this module accesses variables which should be available in the runspace. This is done through $Using, like in the example below, where I add one day to a given date:

$One = 1
Start-RSJob -InputObject ('1/1/2017','2/2/2017') -ScriptBlock { (Get-Date $_).AddDays($Using:One) } | Wait-RSJob | Receive-RSJob

Monday, January 2, 2017 12:00:00 AM
Friday, February 3, 2017 12:00:00 AM
It is also important to force Start-RSJob to evaluate any function you want to use in your parallel execution. This is done through the FunctionsToLoad parameter, which in my case I use to load the Test-PSRemoting function.

The last hint about this module is that you should make an heavy use of the Verbose parameter to follow whatever is happening in your runspaces and also to add a nice and useful progress bar via the ShowProgress switch inside the Wait-Job cmdlet.

So let's see now a few examples of how I use the PoshRSJob module to make the execution of my Test-PSremoting function lightning fast. In the first example I retrieve all the Windows Server names from the Active Directory, and I check whether they ping and can be reached via Invoke-Command, to return only those that actually passed this last test and save their data in a CSV file:

$Report = Start-RSJob -Throttle 20 -Verbose -InputObject ((Get-ADComputer -server dc01 -filter {(name -notlike 'win7*') -AND (OperatingSystem -Like "*Server*")} -searchbase "OU=SRV,DC=Domain,DC=Com").name) -FunctionsToLoad Test-PSRemoting -ScriptBlock {Test-PSRemoting $_ -Ping -Kerberos -Credential $using:cred -Verbose} | Wait-RSJob -Verbose -ShowProgress | Receive-RSJob -Verbose

$Report | ? Remoting_Kerberos -eq 'ok' | convertto-csv -Delimiter ',' -NoTypeInformation | Out-File C:\WinRM-after-gpo.csv
In the second example I use an ADSISearcher to query an old Windows 2003 Domain to retrieve all the Windows Servers and then I try all the different blocks of code (ping, name resolution, TCP check, NTLM, Kerberos) to return only the servers that actually responded to ping in a table:
$Servers = ((New-Object -typename ADSISearcher -ArgumentList @([ADSI]"LDAP://domain.com/dc=domain,dc=com","(&(&(sAMAccountType=805306369)(objectCategory=computer)(operatingSystem=*Server*)))")).FindAll()).properties.name

$Report = Start-RSJob -Throttle 20 -Verbose -InputObject $Servers -FunctionsToLoad Test-PSRemoting -ScriptBlock {Test-PSRemoting -Ping -Resolve -TCPCheck -DCOM -Negotiate -Kerberos $_ -Credential $using:cred -Verbose} | Wait-RSJob -Verbose -ShowProgress | Receive-RSJob -Verbose

$Report | ? ping -eq 'ok' | format-table * -AutoSize
For sure you can imagine here any kind of grouping of your results, with Group-Object, or you could print the results in a dynamic table with Out-GridView. What you do will depend on your needs.

Kudos to Boe for the PoshRSJob module. If you have any question on the function I wrote, or if you want to improve it, feel free to get in touch with me.

Tuesday, February 7, 2017

From Test-Connection to a one line long Filter to ping via PowerShell with a timeout

Having had to deal with mass activation of WinRM in large environments, I have been asked to provide a report of the servers actually answering to remote requests. Sure enough I wrote a PowerShell function for this task, which I am going to present in a future post. Today I just want to take the time to do a bit of a back-to-the-basics post to explain the first part of the function that does a pretty common job: pinging servers to see if they are alive.

PITFALLS OF TEST-CONNECTION

You must be thinking that this is easily accomplished with Test-Connection, but the problem of this cmdlet is that it can be super slow in large environments, where you probably have servers that happen to be offline, servers that don't exist anymore but are still referenced in the Active Directory or in the DNS, servers that block ICMP and servers which are just temporarily unresponsive.

The reason for not to use Test-Connection is that, believe it or not, Test-Connection does not support a -TimeOut parameter so each ping by default respects the timeout of 1000ms that was designed with networks of 20-30 years ago in mind. So for each offline computer you have by default to wait for four times 1000ms and pinging many servers can take a long time.

A little side note here: do not confuse the -TimeToLive or TTL parameter that comes with Test-Connection with the Timeout parameter we hope to have: a TTL is the maximum number of hops it can take to get from one host to another, not the time to wait for a reply. It is used to make certain a packet doesn't live forever on your network when it's lost.  The PowerShell help and a lot of other resources out there are confusing those and that's why in IPv6 TimeToLive has been renamed to Hop Limit:


So, even if Test-Connection does not have a -TimeOut parameter, there are a couple of ways to make it run a bit faster. Let's have a lot at those ways of pinging and see how and why they make the classical error handling a bit complex.

The first option to speed it up is to use it with the -Quiet switch set to True and the -Count parameter set to 1, like in the following example:

Test-Connection $Computer -Quiet -Count 1

Now, for the error handling part, you could think that including that command in a Try{}Catch{} block would work. You're wrong: using the -Quiet parameter forces Test-Connection to suppress the errors and just return a Boolean value, which is $True if any of the four pings succeeds, otherwise it is $False.

So you have to enclose the command in a conditional statement to return its outcome:

if(Test-Connection $Computer -Quiet -Count 1) { 'OK' } else { 'NOK' }

Now if you run Test-Connection without -Quiet, the cmdlet returns a Win32_PingStatus object when the query is succesful:

Test-Connection $Computer -Count 1 | Get-Member

   TypeName: System.Management.ManagementObject#root\cimv2\Win32_PingStatus

Name                           MemberType     Definition
----                           ----------     ----------
PSComputerName                 AliasProperty  PSComputerName = __SERVER
Address                        Property       string Address {get;set;}
BufferSize                     Property       uint32 BufferSize {get;set;}
NoFragmentation                Property       bool NoFragmentation {get;set;}
PrimaryAddressResolutionStatus Property       uint32 PrimaryAddressResolutionStatus
ProtocolAddress                Property       string ProtocolAddress {get;set;}

On the contrary, if the query is unsuccessful, it doesn't return anything at all and you have to use a standard Try{}Catch{} block in order to retrieve the result:

try { Test-Connection $Computer -Count 1 -ErrorAction Stop; 'OK' } catch { 'NOK' }

To sum up, sending just one echo request reduces the execution time but, still, if I can't shorten the timeout, each query will last way too long for the inventory of many servers.

Let's now have a look then at something that manages very low timeouts (down to 5 milliseconds or less).

THE WIN32_PINGSTATUS CLASS

The first possibility is through Get-WmiObject Win32_PingStatus, which supports a timeout: as you can read in the documentation:


Get-WmiObject Win32_PingStatus -filter "address='$computer' and timeout=5"

The issue with this class is that Win32_PingStatus -f "Address='Serverdoesnotexist'" doesn't lead to an error, so you have to rely on the StatusCode property to see if the ping succeeded or not:

if ((Get-WmiObject Win32_PingStatus -filter "address='$computer' and timeout=5").StatusCode -eq 0) {
    'OK'
}
else {
    'NOK'
}

THE SYSTEM.NET.NETWORKINFORMATION.PING CLASS

The other possibility is to rely on the System.Net.NetworkInformation.Ping class.

$ping = new-object System.Net.NetworkInformation.Ping 
$ping.Send($computer)

In this case, while you have to mess with the PingOptions class to add such things as a TTL or to set a DontFragment flag, you can directly use the (IPAddress^, Int32) overload to define the maximum number of milliseconds to wait for the ping to succeeded:

$ping.Send($computer,5)

As you can see I have set the TimeOut in seconds to a very low value if compared to the standard 1000ms of the classical ping command: setting the value so low will make your ping sweep go much faster while keeping a very good precision since, on modern network, servers that are alive will send an echo reply at the speed of light, even if there are a bunch of routers in between.

The difficulty with using this class is that there are three possible cases that require different error handling methods:

Case n. 1: you try to ping a server that is temporarily offline

$ping.send($offline).status

This won't return an error but will return a PingReply object:


This object will have the status set to TimedOut, so to catch this you have to use an If statement:

if($ping.send($offline).Status -eq 'Success'){'OK'}else{'NOK'}

Case n. 2: you try to ping a server that doesn't exist anymore (meaning no DNS record)

$ping.send($not_existing)

This will return an exception:

Exception calling "Send" with "1" argument(s): "An exception occurred during a Ping request."

that you can grab with Try{}Catch{}:

try {$ping.send($not_existing)}catch{'NOK'}

Case n. 3: you try to ping a server that is up and running

$ping.send($ok).Status

Since this case will return Success, using an If statement is the only option:

if($ping.send($ok).Status -eq 'Success'){'OK'}else{'NOK'}

At the end of the day, it's clear that in order to get all the possible errors, you have to enclose that command in both a Try{}Catch{} block and in a If{}() block:

try {
    $ping = new-object System.Net.NetworkInformation.Ping
    if ($ping.send($computer,5)) {
      'NOK'
    }
    else {
      'OK'
    }
  } catch {
    'NOK'
  }

SPEED COMPARISON

Now let's do a few measures to compare all the methods presented above and see what is actually the fastest way to ping in PowerShell when you try to reach a server that is temporarily offline:

$sb = { Test-Connection $offline -Quiet -Count 1 }
(Measure-Command $sb).TotalSeconds

$sb = { Get-WmiObject Win32_PingStatus -filter "address='$offline' and timeout=5" }
(Measure-Command $sb).TotalSeconds

$sb = { $ping = new-object System.Net.NetworkInformation.Ping,$ping.send($offline,5) }
(Measure-Command $sb).TotalSeconds

The first Test-Connection will last four seconds, while Win32_PingStatus will last 0,3 seconds and System.Net.NetworkInformation.Ping will last 0,1 seconds.

Once we know that System.Net.NetworkInformation.Ping is the faster method, let's build something that allows us to easy reuse it.

BUILDING A FILTER

My idea is just to use something old but still useful in some cases: a PowerShell Filter:

filter Invoke-FastPing {(New-Object System.Net.NetworkInformation.Ping).send($_,5)}

Here's three examples of how you can quickly take advantage of this Filter:

Example 1: ping two or more hosts and suppresses errors for unresponsive servers:
'srv1','srv2' | Invoke-FastPing 2>0

Example 2: ping sweep a whole subnet:
1..254| foreach { "192.168.1.$_"| invoke-fastping } | Where-Object status -eq 'success' | Format-Table * -AutoSize

Example3: ping al the servers in your Active Directory and present the results in a dynamic table:
(Get-ADComputer -filter {OperatingSystem -Like "*Server*"} -searchbase "OU=Servers,DC=MyCompany,DC=Com").dnshostname | Invoke-FastPing | Out-GridView

As you can see, just one line of code and you have a small tool to ping remote hosts with a small timeout. Great.

FUTURE IMPROVEMENTS

To end this post, let me tell you that there is an open issue (by J. Snover himself) on the necessity to add a timeout to Test-Connection:

https://github.com/PowerShell/PowerShell/issues/2478

Hope this is implemented soon.

Stay tuned for more PowerShell.
Related Posts Plugin for WordPress, Blogger...