Friday, November 28, 2014

First look at ConvertFrom-String in Powershell v5

I've been playing with ConvertFrom-String for some days now and must admit that this cmdlet is to me one of the best improvement that came with WMF5.0 and Powershell v5.
 
For the moment there are not so many resources to learn from, but those that exist (especially the ones by fellow MVP François-Xavier Cart) are very well written. Let me mention them, so that this gives you a starting point for getting a good grasp on the mechanism behind it:
In addition to this posts there is a magic interface by Doug Finke on GitHub that allows you to quickly test your ConvertFrom-String template against any set of data. So, before you proceed, I definitively suggest you copy Doug's interface on your test bed to speed up your learning curve.

Ok, now that you have checked those resources and that you know that ConvertFrom-String is aimed at adding structure to unstructured string content, let's quickly get to the basic syntax.

ConvertFrom-String supports two ParameterSets:

ConvertFrom-String [-Delimiter ] [-PropertyNames ] -InputObject  []
and

ConvertFrom-String [-TemplateContent ] [-TemplateFile ] -InputObject  []
The cmdlet's default parameter set is FromDelimiter, which splits any line of text on whitespaces, so, while this does nothing

PS C:\> '12345' | ConvertFrom-String
this works and generates a certain amount of properties:

PS C:\> '1 2 3 4 5' | ConvertFrom-String

P1 : 1
P2 : 2
P3 : 3
P4 : 4
P5 : 5
The properties are dinamically casted to a type:

PS C:\> '1 2 3 4 5' | ConvertFrom-String | Select-Object -ExpandProperty P1 | Get-Member


   TypeName: System.Byte

PS C:\> 'a b c d e f g' | ConvertFrom-String | Select-Object -ExpandProperty P1 | Get-Member


   TypeName: System.Char
But I don't want to spend more time speaking of this, since Doug and François-Xavier did a great job of analyzing the basics of this cmdlet in their blog posts. I just suggest you get yourself familiar with the way the -TemplateFile parameter works against a txt file and the way the -TemplateContent parameter works againsts a Here-String before you proceed.

I want to try to build a concrete example of how to use ConvertFrom-String to build a structured object starting from the output of the legacy command Tracert.exe (guys, I know there is the 'Test-NetConnection -TraceRoute' option, but for the sake of this test I want to base my work on a legacy command that is not supposed to produce anything other then raw text).

Tracert prints a text which looks like this:

Tracing route to happysysadm.com [167.4.251.13] over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  HOST1 [164.129.210.252] 
  2     1 ms     1 ms     1 ms  host2.contoso.com [164.129.250.168] 
  3    11 ms    12 ms    12 ms  host3.subdomain.abc.com [10.230.15.194] 
  4    15 ms    15 ms    12 ms  10.230.14.9 
  5     9 ms    16 ms    16 ms  10.230.14.46 
  7    40 ms    40 ms    41 ms  10.75.200.1 
  8    38 ms    39 ms    39 ms  10.75.200.33 
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12    38 ms    39 ms    39 ms  167.4.251.13


Trace complete
From this printed output I want to extract the name of each device I cross on my route to the target and its IP address.

As you can understand, our ConvertFrom-String template based example, which is what we want to use here to get those messy data sorted out, needs to take care of different facts which I am going to oversimplify here:
  • the device name can come with or without a DNS suffix
  • the device name can be lowercase or uppercase or a mix of both
  • the device name can be missing!!!
  • the IP address can come inside square brackets or alone
  • there can be lines with a 'Request timed out' text instead of a pair Devicename/IPaddress
  • there are additional lines of text at the beginning and at the end of the output ('Tracing route...' and 'Trace complete...')
I copy paste the output of Tracert (I suggest you use the clip command: Tracert happysysadm.com | clip) inside Doug's ConvertFrom-String Buddy in both the Data and the Template textboxes

ConvertFrom-String Buddy interface by fellow MVP Doug Finke

then start modifying the template box. I want to create a HostInfo sequence on each line, so, following the well-known syntax
{[optional-typecast]namesequence-spec:example-value}
I modify the first line of the template as follow:

1     1 ms    <1 ms    <1 ms  {HostInfo*:{Computername:HOST1} [{IPAddress:164.129.210.252}]} 
and I get the following result:

HostInfo                                                                                            
--------                                                                                            
{@{ExtentText=HOST1 [164.129.210.252]; Computername=HOST1; IPAddress=164.129.210.252}}              
{@{ExtentText=host2.contoso.com [164.129.250.168]; Computername=host2; IPAddress=164.129.250.168}}  
{@{ExtentText=host3.subdomain.abc.com [10.230.15.194]; Computername=host3; IPAddress=10.230.15.194}}
Ok, looks like ConvertFrom-String is learning from my example as expected. Great, but I want the parse engine to take also FQDNs, so I modify the second line

2     1 ms     1 ms     1 ms  {HostInfo*:{Computername:host2.contoso.com} [{IPAddress:164.129.250.168}]} 
and this modification allow me to get the whole FQDN for the second and third record:

HostInfo                                                                                                              
--------                                                                                                              
{@{ExtentText=HOST1 [164.129.210.252]; Computername=HOST1; IPAddress=164.129.210.252}}                                
{@{ExtentText=host2.contoso.com [164.129.250.168]; Computername=host2.contoso.com; IPAddress=164.129.250.168}}        
{@{ExtentText=host3.subdomain.abc.com [10.230.15.194]; Computername=host3.subdomain.abc.com; IPAddress=10.230.15.194}}
This was the easy part.
 
Now I have to get a sequence even when the device name is missing. I must admit I had a hard time getting beyond this and was able to solve only thanks to the help of Gustavo (which you can reach at psdmfb @ microsoft com).

The key here is to tell the parse engine that the ComputerName property is optional on each line of the example, then to add a negative example for ComputerName on the third example so that Powershell does not incorrectly identify an IP as a Computername when this property is missing.

To make it simple, here's the syntax to use:

{!Computername?:10.230.15.194}
The first part of the template then becomes (notice the question marks):

  1     1 ms    <1 ms    <1 ms  {HostInfo*:{Computername?:HOST1} [{IPAddress:164.129.210.252}]} 
  2     1 ms     1 ms     1 ms  {HostInfo*:{Computername?:host2.contoso.com} [{IPAddress:164.129.250.168}]} 
  4    15 ms    15 ms    12 ms  {HostInfo*: {IPAddress:{!Computername?:10.230.14.9}}} 
and returns:

HostInfo                                                                                                              
--------                                                                                                              
{@{ExtentText=HOST1 [164.129.210.252]; Computername=HOST1; IPAddress=164.129.210.252}}
{@{ExtentText=host2.contoso.com [164.129.250.168]; Computername=host2.contoso.com; IPAddress=164.129.250.168}}
{@{ExtentText=host3.subdomain.abc.com [10.230.15.194]; Computername=host3.subdomain.abc.com; IPAddress=10.230.15.194}}
{@{ExtentText=10.230.14.9; IPAddress=10.230.14.9}}
{@{ExtentText=10.230.14.46; IPAddress=10.230.14.46}}
{@{ExtentText=10.75.200.1; IPAddress=10.75.200.1}}
{@{ExtentText=10.75.200.33; IPAddress=10.75.200.33}}
{@{ExtentText=Request timed out.; Computername=Request}}
{@{ExtentText=Request timed out.; Computername=Request}}
{@{ExtentText=Request timed out.; Computername=Request}}
{@{ExtentText=167.4.251.13; IPAddress=167.4.251.13}}
Woah! I got what I want plus some undesired text: {@{ExtentText=Request timed out.; Computername=Request}}

If from the Template I remove the line with

9     *        *        *     Request timed out.
I get an error:

ConvertFrom-String appears to be having trouble parsing your data using the template you’ve provided.
So I have to make my HostInfo examples more expressive so the parser knows what kind of data I am after. Nothing easier. I just have to add an example which tells ConvertFrom-String that the IP address can come alone:

14    10 ms    10 ms     7 ms  {HostInfo*:{IPAddress:10.230.15.193}} 
and, there you are, we have a working template which output the following result:

HostInfo                                                                                                              
--------                                                                                                              
{@{ExtentText=HOST1 [164.129.210.252]; Computername=HOST1; IPAddress=164.129.210.252}}
{@{ExtentText=host2.contoso.com [164.129.250.168]; Computername=host2.contoso.com; IPAddress=164.129.250.168}}
{@{ExtentText=host3.subdomain.abc.com [10.230.15.194]; Computername=host3.subdomain.abc.com; IPAddress=10.230.15.194}}
{@{ExtentText=10.230.14.9; IPAddress=10.230.14.9}}
{@{ExtentText=10.230.14.46; IPAddress=10.230.14.46}}
{@{ExtentText=10.75.200.1; IPAddress=10.75.200.1}}
{@{ExtentText=10.75.200.33; IPAddress=10.75.200.33}}
{@{ExtentText=167.4.251.13; IPAddress=167.4.251.13}}
What's brilliant here is that Doug's ConvertFrom-String Buddy generate all the required code for you to copy paste it into your ISE and keep up working on the resulting object. What else?

On top of this you could add any possible action, like in the following example:

$targetData = @'
Tracing route to happysysadm.com [167.4.251.13] over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  HOST1 [164.129.210.252] 
  2     1 ms     1 ms     1 ms  host2.contoso.com [164.129.250.168] 
  3    11 ms    12 ms    12 ms  host3.subdomain.abc.com [10.230.15.194] 
  4    15 ms    15 ms    12 ms  10.230.14.9 
  5     9 ms    16 ms    16 ms  10.230.14.46 
  7    40 ms    40 ms    41 ms  10.75.200.1 
  8    38 ms    39 ms    39 ms  10.75.200.33 
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11     *        *        *     Request timed out.
 12    38 ms    39 ms    39 ms  167.4.251.13


Trace complete
'@

$TemplateContent = @'
Tracing route to happysysadm.com [167.4.251.13] over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  {HostInfo*:{Computername?:HOST1} [{IPAddress:164.129.210.252}]} 
  2     1 ms     1 ms     1 ms  {HostInfo*:{Computername?:host2.contoso.com} [{IPAddress:164.129.250.168}]} 
  4    15 ms    15 ms    12 ms  {HostInfo*: {IPAddress:{!Computername?:10.230.14.9}}} 
  9     *        *        *     Request timed out.
  14    10 ms    10 ms     7 ms  {HostInfo*:{IPAddress:10.230.15.193}} 

Trace complete
'@

$targetData |
    ConvertFrom-String -TemplateContent $TemplateContent |
    Select-Object -ExpandProperty HostInfo |
    Select-Object IPAddress | % {
                                "Your code to run against $($_.IPAddress) goes here"
                                }
If you want to get in the deep of ConvertFrom-String, focus on the use of the -Debug switch:

$targetData |
    ConvertFrom-String -TemplateContent $TemplateContent -Debug

DEBUG: Property: HostInfo
Program: ESSL((Contains(Number([0-9]+(\,[0-9]{3})*(\.[0-9]+)?), Dot(\.), Number([0-9]+(\,[0-9]{3})*(\.[0-9]+)?), 1)): 1, 2, ...: Number([0-9]+(\,[0-9]{3})*(\.[0-9]+)?)
, Dynamic Token(\ ms\ \ )(\ ms\ \ )...ε, 3 + ε...ε, 0)
-------------------------------------------------
Property: Computername
Program: ESSL((EndsWith(Dot(\.), Number([0-9]+(\,[0-9]{3})*(\.[0-9]+)?), Right Bracket(\]))): 0, 1, ...: ε...ε, 1 + ε...WhiteSpace(( )+), Left Bracket(\[), Number([0-9
]+(\,[0-9]{3})*(\.[0-9]+)?), 1)
-------------------------------------------------
Property: IPAddress
Program: ESSL((Contains(Number([0-9]+(\,[0-9]{3})*(\.[0-9]+)?), Dot(\.), Number([0-9]+(\,[0-9]{3})*(\.[0-9]+)?), 1)): 0, 1, ...: ε...Number([0-9]+(\,[0-9]{3})*(\.[0-9]
+)?), Dot(\.), Number([0-9]+(\,[0-9]{3})*(\.[0-9]+)?), 1 + Number([0-9]+(\,[0-9]{3})*(\.[0-9]+)?), Dot(\.), Number([0-9]+(\,[0-9]{3})*(\.[0-9]+)?)...ε, 1)
-------------------------------------------------
As you can understand using this debugging option, and to make a long story short, ConvertFrom-String is an intelligent tool that writes regex for you! Let's make a simple example to better grasp the concept:

$targetData = @'
a
'@

$TemplateContent = @'
{Letter*:a}
'@

$targetData | ConvertFrom-String -TemplateContent $TemplateContent -Debug

DEBUG: Property: Letter
Program: ESSL((EndsWith(all lower((?<![\p{Lu}\p{Ll}])(\p{Ll})+))): 0, 1, ...: ε...ε, 1 + ε...ε, 0)
The regex above is trying to match the given text against some Unicode categories. Without going too deep in this vast subject, what you must know is that each Unicode char matches one or often more than one category. An example of category is \p{L} or \p{Letter} which represents all the possible letters whatever the language. Another exampe of category is \p{N} or \p{Number} that contains any numeric char out there.
 
In the simple example above the ConvertFrom-String engine tries to match lowercase (\p{Ll}) and uppercase (\p{Lu}) letters. Got it?
 
That's all for the moment on ConvertFrom-String. The last piece of information I can give is that ConvertFrom-String has an alias: cfs.
 
If you want to learn more, know that there is a brilliant video on text parsing by fellow MVP Tobias Weltner (minutes 24 to 32 are specific to ConvertFrom-String) In its video, Tobias shows also the power of ISESteroids console, so it it worth checking it out.
 
Tobias Weltner on ConvertFrom-String with ISESteroids
 
If you liked this post, be kind, share!

1 comment:

Related Posts Plugin for WordPress, Blogger...