Wednesday, November 30, 2016

A PowerShell function to monitor physical disk activity per storage bay during sync activities

These days I have been migrating data on a old Windows 2003 server from an old HP XP128 storage array to a newer one, a HP 3PAR. Both those fiber channel SAN were mounted and managed on the server through Veritas Enterprise Administrator (VEA) version 5.1. At first I started with a Robocopy to migrate data, ACLs, and all the rest from the old volume to the new one, but soon discovered that there could be better ways to move huge amount of data (I am talking here of several million sensitive files).

One of the main advantages of using Robocopy is that you have fine grained control over your sync. The downside is that, after the sync, you have to stop the old volume and move all your pointers to the new volume, which has a big impact on the automation systems relying on those files to keep their 24/7 activity.

I decided then to change plans and build a mirror on VEA between the old storage array and the new one.

The only problem with such a old version of VEA is that you don't have access to such a basic information as the fact that the mirror sync is completed. The interface just shows you that you have successfully built your mirror but hides the information about the actual data sync taking place behind the curtains.

That's the moment the manager came in and asked for a way to keep an eye the sync. And that's the moment I replied: I can do that for you with PowerShell, sir.

I knew fairly well that though there's no PowerShell on a Windows 2003 server unless you have taken the time to install it, I could access its performance counters from a distant, more recent workstation trough Get-Counter:

What I wanted was to give the manager a script he could run himself that showed the activity for the disks involved in the sync. So I knew that I had to rely on cmdlets I am not used to put in my functions, such as Clear-Host or Write-Host.
But, as for anything else, there are times you have to make exceptions. And Write-Host can have its use sometimes.
In the end I came up with a function that given a set of physical disks on a source server and on a target server, monitors the disk activity in terms of read and written bytes per seconds and in case those are not null, set the font color to green, so that they're highlighted.

The names of the disks can be found in the Perfmon GUI itself as well as in VEA:


Their names can be given as input to the functions as a pattern for a regular expression. In my case this gave:

-SourceDiskPattern '\(2\)|\(14\)'
and
-DestinationDiskPattern '\(8\)|\(9\)|\(10\)|\(20\)|\(21\)|\(22\)' 
because I am trying to match those hard disk numbers.

I have also added to the function a couple of parameters to show the processor activity and the disk queue, since these counters can always be of use when tracing a workload:


In the end, here's the output expected by the manager during the sync, with the green lines highlighting the disks where the data are read or written:


I rely on Clear-Host to refresh the screen so that the manager can only see the current workload. This can be a bad practice as Invoke-ScriptAnalyzer will tell you, but in my case this is exactly the cmdlet I needed.


Here's the code for the Get-DiskStat function, which by the way you can find on Github:

<#
.Synopsis
   Monitors physical disk activity per bay during sync activities
.DESCRIPTION
   Monitors physical disk activity during a sync and highlights the disks that are active reading or writing bytes and the bay they belong to
.EXAMPLE
   Get-DiskStat -SourceComputer srv1 -SourceStorageBayName 'HPE 3PAR' -SourceDiskPattern '\(2\)|\(14\)' -DestinationComputer srv2 -DestinationStorageBayName 'HP XP128' -DestinationDiskPattern '\(7\)|\(10\)' -Refresh -Frequency 2 -Repeat 10
.EXAMPLE
   Get-DiskStat -SourceComputer srv1 -SourceStorageBayName 'HPE 3PAR' -SourceDiskPattern '\(2\)|\(14\)' -DestinationComputer srv2 -DestinationStorageBayName 'HP XP128' -DestinationDiskPattern '\(7\)|\(10\)' -Refresh -ShowCpu
.EXAMPLE
   Get-DiskStat -SourceComputer srv1 -SourceStorageBayName 'HPE 3PAR' -SourceDiskPattern '\(2\)|\(14\)' -DestinationComputer srv2 -DestinationStorageBayName 'HP XP128' -DestinationDiskPattern '\(7\)|\(10\)' -Refresh -ShowCpu -ShowQueue
.EXAMPLE
   Get-DiskStat -SourceComputer srv1 -SourceStorageBayName 'HPE 3PAR' -SourceDiskPattern '\(2\)|\(14\)' -DestinationComputer srv2 -DestinationStorageBayName 'HP XP128' -DestinationDiskPattern '\(7\)|\(10\)' -Refresh -ShowCpu -ShowQueue -Credential (Get-Credential)
.EXAMPLE
   Get-DiskStat -sc srv1 -sbn 'HPE 3PAR' -sdp '\(2\)|\(14\)' -dc srv2 -dbn 'HP XP128' -ddp '\(7\)|\(10\)' -R -C -Q -Cred (Get-Credential) -F 1 -rep 1000
.AUTHOR
   Carlo MANCINI
#>
function Get-DiskStat
{
    Param
    (
        # Source computer for the sync
        [Parameter(Mandatory=$true,Position=0)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("sc")]
        $SourceComputer,

        # Source bay name for the sync
        [Parameter(Mandatory=$true,Position=1)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("sbn")]
        $SourceStorageBayName,

        # Source disk pattern for the sync
        [Parameter(Mandatory=$true,Position=2)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("sdp")]
        $SourceDiskPattern,

        # Destination computer for the sync
        [Parameter(Mandatory=$true,Position=3)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("dc")]
        $DestinationComputer,

        # Destination bay name for the sync
        [Parameter(Mandatory=$true,Position=4)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("dbn")]
        $DestinationStorageBayName,

        # Destination disk pattern for the sync
        [Parameter(Mandatory=$true,Position=5)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("ddp")]
        $DestinationDiskPattern,

        # Clear the screen between each execution
        [Parameter(Position=6)][Alias("r")][Switch]$Refresh,

        # Show Active and Idle CPU counters
        [Parameter(Position=7)][Alias("c")][Switch]$ShowCpu,

        # Show disk queue for selected disks
        [Parameter(Position=8)][Alias("q")][Switch]$ShowQueue,

        # Specifies a user account that has permission to perform this action
        [Parameter(Mandatory=$false,Position=9)]
        [System.Management.Automation.PSCredential]
        [System.Management.Automation.Credential()]
        $Credential = [System.Management.Automation.PSCredential]::Empty,

        # Frequency of the polling in seconds
        [Parameter(Position=10)]
        [Alias("f")]
        $Frequency = 10,

        # Total number of polling to perform
        [Parameter(Position=11)]
        [Alias("rep")]
        $Repeat = 10

    )
    
    Try { 
    
        Test-Connection $SourceComputer,$DestinationComputer -Count 1 -ErrorAction Stop | Out-Null
        
        }
    
    Catch {
    
        Throw "At least one of the target servers is not reachable. Exiting."
        
        }

    $CounterList = '\PhysicalDisk(*)\Disk Read Bytes/sec','\PhysicalDisk(*)\Disk Write Bytes/sec','\PhysicalDisk(*)\Current Disk Queue Length','\Processor(_Total)\% Idle Time','\Processor(_Total)\% Processor Time'

    1..$Repeat | % {

        $SourceCounterValue = (Get-Counter $CounterList -ComputerName $SourceComputer).countersamples

        if($DestinationComputer -eq $SourceComputer) {

            $DestinationCounterValue = $SourceCounterValue

            $SameHost = $True

            }

        else {
        
            $DestinationCounterValue = (Get-Counter $CounterList -ComputerName $DestinationComputer).countersamples

            }

        if($Refresh) {Clear-Host}

        if($ShowCpu) {

                    "$SourceComputer CPU Activity & Idle"
                    
                    $SourceCounterValue | ? {$_.path -match 'processor'} | % {
    
                            Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10)
                            
                            }

                    if(!$SameHost) {
                    
                        "$DestinationComputer CPU Activity & Idle"
                    
                        $DestinationCounterValue | ? {$_.path -match 'processor'} | % {
    
                            Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10)
                            
                            }
                        }

                    }

        if($ShowQueue) {

            "$SourceStorageBayName Storage Bay Disk Queue on $SourceComputer"

            $SourceCounterValue | ? {($_.path -match $SourceDiskPattern) -and ($_.path -match 'queue')} | % {
    
                    if($_.cookedvalue -gt 0) {
                    
                        Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $_.cookedvalue.tostring().padright(10) -ForegroundColor Green
                            
                        }
    
                    else {
                    
                        Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $_.cookedvalue.tostring().padright(10) -ForegroundColor White
                        
                        }
                    
                    }

            "$DestinationStorageBayName Storage Bay Disk Queue on $DestinationComputer"

            $DestinationCounterValue | ? {($_.path -match $DestinationDiskPattern) -and ($_.path -match 'queue')} | % {
    
                    if($_.cookedvalue -gt 0) {
                    
                        Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $_.cookedvalue.tostring().padright(10) -ForegroundColor Green
                            
                        }
    
                    else {
                    
                        Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $_.cookedvalue.tostring().padright(10) -ForegroundColor White
                        
                        }
                    
                    }

            }


        "$SourceStorageBayName Read stats on $SourceComputer"

        $SourceCounterValue | ? {($_.path -match $SourceDiskPattern) -and ($_.path -match 'read')} | % {
    
            if($_.cookedvalue -gt 0) {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor Green
                
                }
    
            else {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor White
                
                }
            
            }

        "$SourceStorageBayName Write stats on $SourceComputer"

        $SourceCounterValue | ? {($_.path -match $SourceDiskPattern) -and ($_.path -match 'write')} | % {
    
            if($_.cookedvalue -gt 0) {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor Green
                
                }
    
            else {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor White
                
                }
            
            }

        "$DestinationStorageBayName Read stats on $DestinationComputer"

        $DestinationCounterValue | ? {($_.path -match $DestinationDiskPattern) -and ($_.path -match 'read')} | % {
    
            if($_.cookedvalue -gt 0) {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor Green
                
                }
    
            else {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor White
                
                }
            
            }

        "$DestinationStorageBayName Write stats on $DestinationComputer"

        $DestinationCounterValue | ? {($_.path -match $DestinationDiskPattern) -and ($_.path -match 'write')} | % {
    
            if($_.cookedvalue -gt 0) {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor Green
                
                }
    
            else {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor White
                
                }
            
            }

        Start-Sleep -Seconds $frequency

        }    

}
PowerShell, once again the tool for the job.

Friday, November 25, 2016

On the road to Overlay networking on Docker for Windows

The container networking stack has gone through many rapid improvements on Windows Server 2016, and it's nice to see that some new features are coming out on a regular basis: Docker's release pace is fast, and though they have had a few missteps, most of the discovered bug are promptly addressed.

In this post I want to talk you about the implementation of multi-host networking on Docker for Windows.

On Linux this is supported since Kernel version 3.16 but on Windows, Containers are a recent feature and Overlay networking is likely going to be released pretty soon.

So, let's have a look at what this is and how it works.

As you have learned from my previous posts, the Docker engine communicates with the underlying Host Network Service (HNS) through a Libnetwork plugin. This plugin implements the Docker Container Network Model (CNM) which is composed of three main components:
  • A Sandbox, where the network configuration (IP address, mac address, routes and DNS entries) of the container is stored
  • An Endpoint linking the container Sandbox to a Network: this is a vNIC in the case of a Windows Container or a vmNIC in case of a Hyper-V container
  • A Network, which is a group of Endpoints belonging to different containers that can communicate directly
Behind each Network a built-in Driver performs the actual work of providing the required connectivity and isolation.

There are four possible driver packages inside Libnetwork:
  • null
  • bridge
  • overlay
  • remote
No network interface is attached to a container which is started with the Null driver:
docker run -it --network none microsoft/nanoserver powershell
Get-NetAdapter in this case returns nothing. And upon inspection this container will show no network:

In the second case, when you use the Bridge driver, the container won’t have a public IP but will be assigned a private address from the 20-bit private range defined by RFC 1918:

172.16.0.0 - 172.31.255.255 (172.16/12 prefix)

Get-Netadapter will show the virtual Ethernet adapter:
Get-NetAdapter

Name                      InterfaceDescription                    ifIndex
----                      --------------------                    -------
vEthernet (Container N... Hyper-V Virtual Ethernet Adapter #2          19
and Get-NetIpAddress will show the private IP address:
Get-NetIPAddress | Format-Table

ifIndex IPAddress                                       PrefixLength PrefixOrigin
------- ---------                                       ------------ ------------
19      fe80::29aa:cc8a:43f2:ae0f%19                              64 WellKnown   
18      ::1                                                      128 WellKnown   
19      172.31.2.5                                                20 Manual      
18      127.0.0.1                                                  8 WellKnown   
If I inspect this container, I can see the JSON describing the network specifications:
docker container inspect 4a44649f2b8d

Now just a couple of weeks ago (on version v1.13.0-rc1), Docker has implemented the third Driver (read Swarm-mode overlay networking support for windows), which basically means that your Windows running containers will be able to communicate even if they are residing on different hosts.

Actually this is a bit more complicated than that, because Overlay networking has been implemented in the Docker engine but not yet in the HNS service of Windows. So if you try to build a multi-host network you will get the following error message:
docker network create -d overlay --subnet 10.1.1.0/24 multihost
Error response from daemon: HNS failed with error : Catastrophic failure
Same output if you try the PowerShell version:
New-ContainerNet -Driver overlay -Name MultiHost
New-ContainerNet : Docker API responded with status code=InternalServerError, response={"message":"HNS failed witherror : Catastrophic failure "}
At line:1 char:1
+ New-ContainerNet -Driver overlay -Name MultiHost
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  + CategoryInfo          : NotSpecified: (:) [New-ContainerNet], DockerApiException
  + FullyQualifiedErrorId : Docker Client Exception,Docker.PowerShell.Cmdlets.NewContainerNet
Once the required Windows binaries to build a Overlay network will be released, it will be interesting to see if Microsoft is going to embed in a Nano Server the required Key-Value store which has to be accessible to all the Containers belonging to the same Overlay network to be discoverable.

For the moment the most used key-value store is the one provided by Consul, but it is only Linux based so you won’t be able to run it on Windows:
docker run -p 8500:8500 -d consul --name consul
Unable to find image 'consul:latest' locally
latest: Pulling from library/consul
C:\Program Files\Docker\docker.exe: image operating system "linux" cannot be used on this platform.
See 'C:\Program Files\Docker\docker.exe run --help'.
All the same, Overlay networking is going to be soon available for Docker containers on Windows. The first step has been done. Now it is up to Microsoft to do the next move. Stay tuned for more on the subject.

Wednesday, November 16, 2016

Building a Docker container for the Image2Docker tool

I have been playing a bit with Image2Docker with the intention to see how far I could go into containerizing existing workloads. To date, this PowerShell-based module by fellow MVP and Docker Captain Trevor Sullivan mounts a vhdx or wim Windows Image and tries to discover running artifacts, such as IIS, SQL or Apache, and generates a Dockerfile for a container hosting these services.

Now this is still experimental, and the list of accepted artifacts is still short, but I couldn't retain myself from trying to build a Docker container for the job.

Here's how I tackled this, knowing that as most of us, I am moving my first steps into this new feature of Windows 2016.

First of all I built the following Dockerfile in Visual Studio Code:


Basically I am issuing five statements:
  1. pull the microsoft/nanoserver image. Actually I could have used the microsoft/windowsservercore image as well but that would have taken longer
  2. state that I am the mantainer of the repository
  3. install the package manager called Nuget
  4. install the actual Image2Docker module (version 1.5 at the time of writing)
  5. set the ConvertTo-Dockerfile cmdlet as entry point for this container, so that I can pass the .vhdx or .wim image path straight into this dedicated container on execution
Then the next steps to publish to the Docker Hub are:

docker build .  -t happysysadm/image2docker:latest -t happysysadm/image2docker:v0.1

In the step above I am issuing the build command from the folder containing the Dockerfile file, and I am setting two tags for the same image: latest and v0.1.


Then I logged in to the Hub:

docker login -u user -p password

And pushed the container into my public registry:

docker push happysysadm/image2docker

At this moment this repo becomes visible on the web:


Once I got your container up in the Hub, I cleaned up my local image:

docker rmi -f happysysadm/image2docker:v0.1

and pulled it again:

docker pull happysysadm/image2docker

Every time I have gone through an update of my Dockerfile, I had to do a rebuild and increment the version tag:

docker build .  -t happysysadm/image2docker:latest -t happysysadm/image2docker:v0.2

In the step above the latest tag is passed to v0.2 and the previous image retains only the tag v0.1.

Now this container is public and you can just do:

docker run happysysadm/image2docker sample.vhdx

and get the Dockerfile for your Windows image created for you. Let me know how it goes and remember that this project is open source so everybody's contribution is accepted.

Monday, November 14, 2016

Step up container management with PowerShell for Docker

I remember that one of the first reasons I started using Windows PowerShell is that it uses objects to represent the data, which is great when you are interacting with a object-oriented Windows ecosystem. Now that some historical borders have been crossed between Linux and Windows, and that preexisting tools have been translated to the Microsoft's OS, we, as PowerShell guys, could face a bit of a throwback in the way we use the shell.

Just have a look at Docker.

Invented in 2013 by a French guy named Solomon Hykes, this open source project aimed at automating the deployment of Linux containers has been quickly adopted by Microsoft for their last operating system and can today be run on both Windows 10 and Windows 2016.

The main drawback of adopting such a tool, is that it comes with a command line which is obsolete if you look at it in PowerShell terms: it only produces strings, which are hardly reusable, unless you feed them to ConvertFrom-String:

docker images | ConvertFrom-String -Delimiter "\s{2,}" | Format-Table

P1                          P2     P3           P4          P5
--                          --     --           --          --
REPOSITORY                  TAG    IMAGE ID     CREATED     SIZE
microsoft/iis               latest 211fecef1e6b 5 days ago  9.48 GB
microsoft/sample-dotnet     latest c14528829a37 2 weeks ago 911 MB
microsoft/windowsservercore latest 93a9c37b36d0 7 weeks ago 8.68 GB
microsoft/nanoserver        latest e14bc0ecea12 7 weeks ago 810 MB

Now, tough ConvertFrom-String is a extremely powerful cmdlet released with PowerShell 5.0 (check my blog post on the subject), it take some time to feel easy with its syntax. In the previous example for instance I am outputting the list of the images I have pulled from the Docker Hub onto my system. The text that comes through the pipeline once I run 'docker images' has to be split whenever I find at least 2 empty spaces. To achieve that I have to use the Delimiter parameter and match a whitespace \s at least two times {2,}.

Needless to say, knowing regular expressions becomes a must.

Happily enough we have an alternative to this. Since Docker comes with a nice API, there is open source project for a module exposing PowerShell cmdlets to manage Docker images, containers and networks. Tough still in development, I heartedly suggest you start using it to maintain consistency with your existing environment.

You can find it here:




The installation is straightforward.

Register-PSRepository -Name DockerPS-Dev -SourceLocation https://ci.appveyor.com/nuget/docker-powershell-dev

Install-Module Docker -Repository DockerPS-Dev -Scope CurrentUser

Here's the list of cmdlets that come with it:

Get-Command -Module Docker -CommandType Cmdlet

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Cmdlet          Add-ContainerImageTag                              0.1.0.111  Docker
Cmdlet          ConvertTo-ContainerImage                           0.1.0.111  Docker
Cmdlet          Copy-ContainerFile                                 0.1.0.111  Docker
Cmdlet          Enter-ContainerSession                             0.1.0.111  Docker
Cmdlet          Export-ContainerImage                              0.1.0.111  Docker
Cmdlet          Get-Container                                      0.1.0.111  Docker
Cmdlet          Get-ContainerDetail                                0.1.0.111  Docker
Cmdlet          Get-ContainerImage                                 0.1.0.111  Docker
Cmdlet          Get-ContainerNet                                   0.1.0.111  Docker
Cmdlet          Get-ContainerNetDetail                             0.1.0.111  Docker
Cmdlet          Import-ContainerImage                              0.1.0.111  Docker
Cmdlet          Invoke-ContainerImage                              0.1.0.111  Docker
Cmdlet          New-Container                                      0.1.0.111  Docker
Cmdlet          New-ContainerImage                                 0.1.0.111  Docker
Cmdlet          New-ContainerNet                                   0.1.0.111  Docker
Cmdlet          Remove-Container                                   0.1.0.111  Docker
Cmdlet          Remove-ContainerImage                              0.1.0.111  Docker
Cmdlet          Remove-ContainerNet                                0.1.0.111  Docker
Cmdlet          Request-ContainerImage                             0.1.0.111  Docker
Cmdlet          Start-Container                                    0.1.0.111  Docker
Cmdlet          Start-ContainerProcess                             0.1.0.111  Docker
Cmdlet          Stop-Container                                     0.1.0.111  Docker
Cmdlet          Submit-ContainerImage                              0.1.0.111  Docker
Cmdlet          Wait-Container                                     0.1.0.111  Docker
This module also exposes a bunch of aliases, though I don't recommend their use since they seem confusing to me and don't add anything in terms of command line agility:

Get-Command -Module Docker -CommandType Alias | Format-Table Name,ResolvedCommandName

Name                 ResolvedCommandName
----                 -------------------
Attach-Container     Enter-ContainerSession
Build-ContainerImage New-ContainerImage
Commit-Container     ConvertTo-ContainerImage
Exec-Container       Start-ContainerProcess
Load-ContainerImage  Import-ContainerImage
Pull-ContainerImage  Request-ContainerImage
Push-ContainerImage  Submit-ContainerImage
Run-ContainerImage   Invoke-ContainerImage
Save-ContainerImage  Export-ContainerImage
Tag-ContainerImage   Add-ContainerImageTag

So, docker images becomes:

Get-ContainerImage

RepoTags                              ID                   Created                Size(MB)
--------                              --                   -------                --------
microsoft/sample-dotnet:latest        sha256:c14528829a... 25/10/2016 13:55:28    869,05
microsoft/windowsservercore:latest    sha256:93a9c37b36... 22/09/2016 10:51:07    8 273,19
microsoft/nanoserver:latest           sha256:e14bc0ecea... 22/09/2016 09:39:30    772,81

and the returned object is a heavily reusable ImagesListResponse object:

Get-ContainerImage | Get-Member

   TypeName: Docker.DotNet.Models.ImagesListResponse

Name        MemberType Definition
----        ---------- ----------
Equals      Method     bool Equals(System.Object obj)
GetHashCode Method     int GetHashCode()
GetType     Method     type GetType()
ToString    Method     string ToString()
Created     Property   datetime Created {get;set;}
ID          Property   string ID {get;set;}
Labels      Property   System.Collections.Generic.IDictionary[string,string] Labels...
ParentID    Property   string ParentID {get;set;}
RepoDigests Property   System.Collections.Generic.IList[string] RepoDigests {get;set;}
RepoTags    Property   System.Collections.Generic.IList[string] RepoTags {get;set;}
Size        Property   long Size {get;set;}
VirtualSize Property   long VirtualSize {get;set;}

Same model for the list of existing containers:

Get-Container

ID                   Image           Command              Created                Status
--                   -----           -------              -------                ------
43a05b618697033eb... microsoft/na... c:\windows\system... 14/11/2016 09:44:19    Exited...
005b51dbe002324f8... microsoft/na... --name nanoserver1   14/11/2016 09:44:04    Created
e8b31c61d5f42b271... microsoft/na... --name nanoserver1   14/11/2016 09:42:12    Created
547b7dbd3b1473127... microsoft/sa... dotnet dotnetbot.dll 06/11/2016 16:11:07    Exited...
Get-Container | Get-Member

   TypeName: Docker.DotNet.Models.ContainerListResponse

Name            MemberType Definition
----            ---------- ----------
Equals          Method     bool Equals(System.Object obj)
GetHashCode     Method     int GetHashCode()
GetType         Method     type GetType()
ToString        Method     string ToString()
Command         Property   string Command {get;set;}
Created         Property   datetime Created {get;set;}
ID              Property   string ID {get;set;}
Image           Property   string Image {get;set;}
ImageID         Property   string ImageID {get;set;}
Labels          Property   System.Collections.Generic.IDictionary[string,string] Labels...
Mounts          Property   System.Collections.Generic.IList[Docker.DotNet.Models.MountP...
Names           Property   System.Collections.Generic.IList[string] Names {get;set;}
NetworkSettings Property   Docker.DotNet.Models.SummaryNetworkSettings NetworkSettings...
Ports           Property   System.Collections.Generic.IList[Docker.DotNet.Models.Port]...
SizeRootFs      Property   long SizeRootFs {get;set;}
SizeRw          Property   long SizeRw {get;set;}
State           Property   string State {get;set;}
Status          Property   string Status {get;set;}

Now that you have this module, you have two ways to run a container. Either by using:

docker run -it microsoft\nanoserver powershell

or by using Invoke-ContainerImage (aliased as Run-ContainerImage):

Invoke-ContainerImage -ImageIdOrName microsoft/nanoserver:latest -Command powershell -Input -Terminal

which, at its best, can be shortened to:

Run-ContainerImage microsoft/nanoserver:latest powershell -In -T
None of the PowerShell syntaxes are as short as the 'legacy' one, but again, the produced object is what makes them worthy using.

I hope you have enjoyed this first post on the PowerShell module for the Docker Engine, which brings close integration between those that not so long ago were distant worlds. Stay tuned for more.

Friday, November 4, 2016

Announcing the winner of the PowerShell Oneliner Contest 2016

I am excited to announce the winner of the second PowerShell Oneliner Contest. But before I do it, let me tell you one thing. This year, I received over ninety submissions from wannabe PowerShell Monks from all over the world. Some solutions stood out as the most striking and imaginative entries. Some others were not successful in achieving what I asked, but showed a lot of effort in learning and initiative. Everybody seemed to understand that the aim of such a contest is not just to push PowerShell to its limit and beyond, by bending the command line to your will. It's a matter of generating knowledge and sharing it for others to learn from. Building code that can benefit the whole community is of paramount importance here.

THE WINNER IS...

So thanks to all of the entrants and, without further ado, let's have a look at the winning solution, by Sam Seitz, with 65 chars:

([char[]](71..89)|?{!(gdr $_)2>0}|sort{[guid]::newguid()})[0]+':'
#Posted by Sam Seitz to Happy SysAdm at October 25, 2016 at 8:02 AM

I got in touch with Sam so he could share a bit about himself and his thought process for the script.

Two years into his IT career, Sam is a 25-year old systems engineer for Network Technologies, Inc., an MSP in Olathe, KS. He spends his days perpetually amazed that his employer pays him to "play with computers" (as his father would say). Outside of work, his incredible wife and their pair of regal beagles keep him happier than a man has any right to be.

Seeing this challenge made me realize two things: 1) off the top of my head I know, maybe, five default aliases and 2) I should really start using more aliases. I'm normally extremely verbose in my scripting, so this proved to be a unique challenge. To keep it as short as possible, I used a few interesting techniques, which I'll break down step-by-step:
([char[]](71..89)
To generate the array of letters from g-y, I took advantage of the fact that the 71 through 89 is g through y in the ASCII table. When cast as a [char], 71 is g, 72 is h, etc.
?{!(gdr $_)2>0}
I then filtered out occupied drive letters by using Where-Object (?), the alias for the "-not" operator (!), and Get-Drive (gdr). 2>0 redirects the inevitable error output to null. (If you don't mind seeing each error Get-Drive throws when it's used with a non-existant drive, the 2>0 could be removed as it isn't necessary for the success of the one-liner. But who likes all those ugly red errors on their screen? Terrorists, that's who.)
sort{[guid]::newguid()}
In order to ensure the result was random, I used Sort-Object (sort) on the array of letters and told it to sort by the new GUID created using the .Net method [guid]::newguid().
[0]+':'
Finally, I selected the first result [0] from the array of randomly sorted available drive letters in the output and threw a colon on the end (+':').

Thanks to Sam for sharing his deep knowledge of PowerShell with us. For those interested, I created a Gist with a list of working solutions I got, sorted by line length.

COMMON ERRORS

Now in the following section I will explain why I could not accept some entries.

The most common error by far was the use of Random as an alias of Get-Random. Though I understand the extreme difficulty of generating a random number without using Get-Random, I couldn't accept oneliners using Random as an alias for the simple reason that it is not one. It just works as an alias because the PowerShell interpreter prepends by default the verb 'Get-' to nouns if it can't find another match.

Get-Alias | ? {$_.Definition -match "Get-Random"}

Trace-Command can confirm that interpreter behavior:

Trace-Command -Name CommandDiscovery -PSHost -Expression { random }

...
DEBUG: CommandDiscovery Information: 0 : The command [random] was not found, trying again with get- prepended
DEBUG: CommandDiscovery Information: 0 : Looking up command: get-random
DEBUG: CommandDiscovery Information: 0 : Cmdlet found: Get-Random  Microsoft.PowerShell.Commands.GetRandomCommand

MY SOLUTIONS

Now a word about my solutions to the contest. I wrote four of them. Since they are pretty short, I am pleased to share them with you.

In the first solution I was actually able to get a random GUID to sort on by fetching it from the internet.

There are for sure many websites exposing an engine for GUID generation (https://www.uuidgenerator.net/ or https://www.guidgen.com/ for instance) but in our case we want the shortest URL possible, and I was lucky enough to find a website named guid.it. There's a funky cmdlet for getting stuff from the web and it is Invoke-RestMethod. It has an alias which is irm. Now the cool thing of Internet is that nowadays many websites have adopted a JSON API to let consumers retrieve and manipulate their content using HTTP requests. And guid.it is one of them. Luck, again. So, doing the alike of:

Invoke-WebRequest http://www.guid.it/json -UseBasicParsing | ConvertFrom-Json

can be achieved in a simpler manner with:

Invoke-RestMethod http://www.guid.it/json

which can be shortened to:

irm guid.it/json

or even shorter:

irm guid.it/api

And there I have my random guid. Internet for the IT pro, I daresay.

For the rest my first solution matches Sam's one:
([char[]](71..89)|?{!(gdr $_)2>0}|sort{irm guid.it/api})[0]+':'

In my second solution, I leverage the .NET Framework's System.Random class, but instead of using [random]::New().Next() I went for a shorter ([random]@{}).Next():
do{$x=[char]([random]@{}).Next(71,89)}while(gdr($x)2>0)$x+':'

My third solution relies on use of my favorite cmdlets, Select-String (aliased as sls) in conjunction with the [guid] type accelerator, which called the NewGuid static method:
''+(ls function:[g-y]:|sls(gdr)-n|sort{[guid]::NewGuid()})[0]

In my fourth and last solution I mixed both my third and first solution, and so I was able to go down to 59 chars. It's a bit slower than the others because of the action of fetching GUIDs from the Internet, but for the purpose of the contest this is the shortest solution I was able to come up with:
''+(ls function:[g-y]:|sls(gdr)-n|sort{irm guid.it/api})[0]

I have created a Gist with my solutions, which you can find here.

UNIT TESTING WITH PESTER

Now a word about how the testing of the posted oneliners went.

Since I was rapidly flowed by plenty of tricky oneliners, I was a bit scared by having to check all of them manually for respect of contest rules. Fortunately I had already worked a bit with Pester to define test cases on other projects, so I just had adapt what I knew to the contest I had just started.

Then, and it was sheer luck, I got contacted by Jakub who proposed a complete solution to test those oneliners.

I am glad to say that what Jakub came up with is just brilliant. So, who better than him to explain his approach. Take it away Jakub.

I always loved one liners. "Make the code as short as possible" is such a simple, yet so challenging restriction. Such restriction does not exist in our day to day work, we care about readability, understandability and performance, but rarely about the length of our code. Putting this restriction in place and removing any other, turns our usual focus on its head. For once we get to write code that is so unreadable and uses so many quirks of the language that we will need to explain it at least twice (if not three times). Finally we can put all the side notes we read in books to work and use all the features, that we thought were bugs when we saw them for the first time, to force the language syntax to it's limits, and then watch in awe when others produce a solutions twice as short as ours.

For this reason I had to take part in the Oneliner contest of 2016 hosted by Carlo on his blog. Once I read the requirements I thought to myself: Well that's more that one requirement, what a nice opportunity to take this on another level and write some tests as well. And so I approached the whole problem in a kata-like way, which means not only taking my time to think about the problem itself, but also taking time to reason about the tests and the process of writing tests. Now since I know I have no way of winning the contest, especially after seeing how creative were people last year, I will at least walk you through my thought process.

First I read the requirements just to make sure they are quantifiable, what I mean by that is that I can measure if the requirement was met. A quantifiable requirement is for example "contains no semi-colon", a non-quantifiable requirement (at least not easily) would be "the code looks nice".

Once I made sure I will be able to write some tests for all of the requirements I proceeded to categorize the requirements and realized that they can be split to wwo categories: stylistic, and functional. Where stylistic is how the code should look like, and functional is how the code should behave.

I started with the functional part of the tests as they seemed much simpler to implement.

### Test 1 - Outputs single string
The first decision I made, was to store my one liner as a script block. This enabled me to reuse the same script block in all the test cases, and it also enabled me to change my one liner very easily.

The first test checks that the output of the oneliner is a single string. Pester has a built-in assertion `BeOfType` which was my first choice, but then I realized that piping the output through pipeline would expand the array that I might get, and I wouldn't be able to check if I got just a single item or whole array or items. So I went oldschool and used the `-is` operator.

It "outputs single string" {
    (&$oneliner) -is [string] | Should be $True
}

### Test 2 - Outputs one letter followed by colon
Next requirement forces me to match the text and specifies that it should be a letter followed by colon. Any text matching is easy with the `Match` assertion which uses regular expressions. The only thing I had to watch out for was matching the start and end of the string, to make sure that no sorrounding characters are matched.

It "Outputs single letter followed by colon" {
    &$oneliner | Should Match "^[a-z]\:$"
}

I decided to match the whole alphabet in this test to limit mixing the requirements. I find it being a good practice to specify requirements in one place without unnecessarily resticting other unrelated tests.

### Test 3 - Should exclude drives A-F and Z
Yet another requirement forces me to exclude some of the drive letters. I decided to use test cases to have a single test for each excluded letter and specified a list of test cases. This feature of Pester generates a single test per testcase and also modifies the name of the test to reflect the actual value of `$DriveLetter` for extra readability. The scriptblock then contains parameter I named $DriveLetter, which I use to write the assertion.

It "Should not output drive letter " -TestCases `
    @{DriveLetter = "a:"},
    @{DriveLetter = "b:"},
    @{DriveLetter = "c:"},
    @{DriveLetter = "d:"},
    @{DriveLetter = "e:"},
    @{DriveLetter = "f:"},
    @{DriveLetter = "z:"}{
    param ($DriveLetter)
        &$oneliner | Should Not Be $DriveLetter
}

### Test 4 - Drive should not be used
This test could not be easier. I am used the `Exist` assertion which I know uses `Test-Path` internally. Nothing else was needed here.

It "Resulting drive should not exist" {
    &$oneLiner | Should Not Exist
}

### Test 5 - Drive should be random
This test I found interesting because randomness is something to avoid in tests as much as possible. Randomness can make test fail from time to time and that unexpected failures lower the trust in we have in tests. But well in this case I'll be using the tests locally so I decided to take the simplest route and run the code twice and then compare the results. If the results are not the same the output is probably "random". This is far from perfect, but in this simple case I can validate by running the test multiple times. In a real production environment I'd run the code more than twice and compare the results.

It "Should be random" {
    &$oneLiner | Should Not Be (&$oneLiner)
}

Another interesting thing about this test is that I did not notice the randomness requirement at first and posted my solution without it, which automatically makes my solution incorrect :)

### Test 6 - Code should be error free
This test seemed straight forward because any terminating error (exception) in a Pester test makes the test fail. The difficult part was capturing non-terminating errors as well. I had to set the error action preference to `Stop` and also pipe to `Not Throw` to make the test behave correctly. That's something to be improved in the next version of Pester.

It "Should be error-free" {
    $errorActionPreference = 'Stop'
    $oneLiner | Should Not Throw
}

That was it for the functional tests. All of them were pretty easy to write, and there was not much to figure out. Next up were te the stylistic tests, which were a bit more challenging as I first needed to write some helper functions to avoid any ifs and for loops in the body of my tests.

### Test 6 - All cmdlets must have an alias
This test was the most challenging test to write. There are two things that I needed to figure out. First I needed a way to parse the code and find all the commands. For that I knew I could use the AST, but I had to write and test the code to find all the commands. The other thing was checking if all the found commands have aliases. First I started with the tests for AST parsing and then I implemented the function:

Describe "Get-ScriptBlockCommand" { 
    It "Finds basic cmdlet" {
        Get-ScriptBlockCommand { Get-Date } | Should Be "Get-Date"
    }
    It "Finds basic alias" {
        Get-ScriptBlockCommand { gci } | Should Be "gci"
    }
    It "Finds multiple commands alias" {
        $actual = Get-ScriptBlockCommand { ps; get-process } 
        $actual[0] | Should Be 'ps'
        $actual[1] | Should Be 'get-process'
    }
    It "Ignores keywords" {
        Get-ScriptBlockCommand { if ($true) {} } | Should BeNullOrEmpty
    }
    It "Ignores other tokens" {
        Get-ScriptBlockCommand { $a = 10 ; $false } | Should BeNullOrEmpty
    }
}

function Get-ScriptBlockCommand ($ScriptBlock) {
     $tokens = [System.Management.Automation.PSParser]::Tokenize($ScriptBlock,[ref]$null)
     $tokens | where { $_.Type -eq 'Command' } | select -expand content
}

Then I followed with looking up aliases and testing the every command has at least one:

Describe "Test-Alias" {
    It "Finds alias for basic cmdlet" {
        Test-Alias Get-ChildItem | Should Be $True
        Test-Alias Test-Path | Should Be $False        
    }

    It "Finds alias when given alias " {
        Test-Alias gci | Should Be $True
        Test-Alias ps | Should Be $True 
    }

    It "Returns true when all commands have aliases" {
        Test-Alias ("gci", "ps", "get-childItem") | Should Be $True
    }

    It "Returns false when any of the commands does not have an alias" {
        Test-Alias ("Test-path", "ps", "get-childItem") | Should Be $false
    }
}

function Test-Alias ([string[]] $Name) {
    end {
        $aliases = Get-Alias
        foreach ($n in $name) {
            if ($null -eq ($aliases | Where {$_.Name -eq $n -or $_.Definition -eq $n}))
            {
                return $false
            }
        }
        $true
    }
}

Then I could finally proceed to writing the main test:

It "All used cmdlets have an alias" {
    $commands = Get-ScriptBlockCommand $oneliner
    Test-Alias $commands | Should Be $True
}

### Test 7 - Code must not contain semicolon
And finally I finished with another primitive test checking that semicolon is nowhere to be found in my oneliner. The one liner is also not executed this time. Rather we implicitly convert it to string and pass it to the `Match` assertion.

It "contains no semicolons" {
    $oneliner | Should Not Match "\;"
}

And that was it for my testing. I hope you enjoyed the competition and congratulation to the winners!!!
@nohwnd

Thanks again to all the competitors, to Mike F Robbins for the original function, to Sam Seitz for his brilliant solution and to Jakub Jares for showing us the way to functional testing. And remember, it was all about learning.

Monday, October 24, 2016

PowerShell Oneliner Contest 2016

A lot of time has passed since I have organized a PowerShell oneliner contest. So when I saw the post by fellow MVP and scripting champion Mike F Robbins on a PowerShell Function to Determine Available Drive Letters, I thought that it could be fun to organize a contest to see who can manage to write the shortest possible oneliner that achieves the same result as Mike's function.


As you can see reading his blogpost, the function accepts parameters such as -Random, to return one or more available drive letters at random, or -All, to return all the available drive letters. It also allows you to exclude some letters from the match (A, B, C, D, E, F and Z) by means of a -ExcludeDriveLetter parameter.

Now, for this specific contest, what I want to get in a comment to this post is:
  • a oneliner (meaning in particular no semi-colon) that
  • returns one and only one random available drive letter on the system where it runs
  • with the exception of A-F and Z
  • whose object type is a System.String (I'll check this Get-Member)
  • and whose formatting is, say, G: or h: (case doesn't matter, we are on Windows)
For sure
  • aliases are mandatory, meaning that you can't use a cmdlet unless it has an alias
  • backticks are accepted for readability
  • you can use every PowerShell version, including 5.1, just state in the comment what version you tested it with
  • should you find a shorter oneliner to solve a task you are allowed to post additional comments (just remember to sign your comments so that I know who's who and so that I can get in touch with the winner)
A few more rules:
  • Entries (comments) will not be made public until after the submission deadline.
  • The first person to produce the shortest working solutions to the task wins.
  • The winner will be announced on Friday, November 4th on this blog.
  • I'll be the only judge.
If you want to spread the word about this PowerShell contest, feel free to twit about it. You can use the hashtags #poshcontest2016 and #powershell so that other competitors can share their thoughts (not the solutions of course!).

UPDATE Nov 4 2016
We have a winner! Check it here.

Thursday, October 20, 2016

How to query the Docker repo and find out the latest Master Build with PowerShell

PowerShell can be used to interact with the web and I have therefore decided to use it to stay tuned with the Docker project which I am currently pretty much interested into (check my previous series on Docker on Windows 2016). Docker is an open source project, and when you say open source nowadays you have to think Github, which is basically a hosting service for open source software projects with features for version control, issue tracking, commit history and all the rest.

Now Github has a REST API that can be consumed in PowerShell through the use of the Invoke-RestMethod cmdlet (aliased as irm in your oneliners). The JSON formatted answer is converted by Invoke-RestMethod to a custom object.

Under Github, URLs are in the form api.github.com/user/repos, so for the Docker project we have query api.github.com/repos/Docker/Docker

A simple request against this URL will return all the basic information on the project:
Selecting the properties I could be interested into is easily achieved:
irm api.github.com/repos/Docker/Docker | ft forks,open_issues,watchers

forks open_issues watchers
----- ----------- --------
10626        1929    36095
Once we know that this API can be consumed with PowerShell, we could very well think of retrieving all the published releases. The URL syntax is found in the result on the previous query:

Here's how I can get the first release returned by the API and see how the information is structured:
(irm api.github.com/repos/Docker/Docker/releases)[0]
A lot of information here, including a body containing the full description of the changes that come with the currently listed version.

As I said, we are only interested in getting the published releases of Docker, so let's use a sieve and keep just four key properties: the name of the package, its ID, author, creation date and publication date.

A short oneliner will do:
irm api.github.com/repos/Docker/Docker/releases |

        sort id |

        ft name,id,*at,@{n='author';e={$_.author.login}} -auto
Let's apply a couple of best practices and put the resulting object in a variable, for reusability, as well as add a bit of formatting for the dates:
$u = irm api.github.com/repos/Docker/Docker/releases

$u |
 sort id |
 
 ft  -auto name,id,

  @{n='author';e={$_.author.login}},

  @{n='creationdate';e={get-date $_.created_at}},

  @{n='publicationdate';e={get-date $_.published_at}}
Basically, with
Get-Date '2016-10-11T23:35:27Z'
I am using Get-Date to convert the timestamps, which are expressed in UTC, to my local time zone. It's the Z in the end of the date (which is a special UTC designator) to tell me that the timestamp is expressed in Coordinated Universal Time.

These two lines of code above return:
name             id author     creationdate        publicationdate    
----             -- ------     ------------        ---------------    
v1.10.1-rc1 2590708 tiborvass  10/02/2016 23:12:24 11/02/2016 00:09:31
v1.10.1     2598018 tiborvass  11/02/2016 22:14:44 11/02/2016 22:16:24
v1.10.2-rc1 2652399 tiborvass  20/02/2016 08:00:24 20/02/2016 08:23:08
v1.10.2     2666504 tiborvass  22/02/2016 23:57:57 23/02/2016 00:05:08
v1.10.3-rc1 2777835 tiborvass  09/03/2016 18:11:06 09/03/2016 18:14:08
v1.10.3-rc2 2780060 tiborvass  09/03/2016 22:58:41 09/03/2016 23:02:32
v1.10.3     2788494 tiborvass  10/03/2016 23:01:03 10/03/2016 23:07:27
v1.11.0-rc1 2875983 tiborvass  23/03/2016 21:15:31 23/03/2016 21:20:29
v1.11.0-rc2 2890861 tiborvass  25/03/2016 22:28:59 25/03/2016 22:31:10
v1.11.0-rc3 2937939 tiborvass  02/04/2016 01:59:32 02/04/2016 02:01:26
v1.11.0-rc4 2968912 tiborvass  07/04/2016 04:28:01 07/04/2016 04:56:10
v1.11.0-rc5 2998258 tiborvass  12/04/2016 01:33:25 12/04/2016 01:34:20
v1.11.0     3014278 tiborvass  13/04/2016 21:56:07 14/04/2016 00:10:23
v1.11.1-rc1 3097597 mlaventure 26/04/2016 10:01:02 26/04/2016 10:05:57
v1.11.1     3105125 mlaventure 27/04/2016 03:51:45 27/04/2016 03:59:03
v1.11.2-rc1 3327300 mlaventure 28/05/2016 21:44:50 28/05/2016 23:47:25
v1.11.2     3354503 tiborvass  02/06/2016 02:59:52 02/06/2016 03:08:13
v1.12.0-rc1 3447699 tiborvass  15/06/2016 10:39:54 15/06/2016 10:55:14
v1.12.0-rc2 3471944 tiborvass  17/06/2016 23:39:11 18/06/2016 00:49:34
v1.12.0-rc3 3573896 tiborvass  02/07/2016 05:26:36 02/07/2016 05:30:18
v1.12.0-rc4 3644623 tiborvass  13/07/2016 07:27:26 13/07/2016 07:25:48
v1.12.0-rc5 3744904 tiborvass  26/07/2016 22:48:09 26/07/2016 22:48:18
v1.12.0     3766135 tiborvass  29/07/2016 02:06:45 29/07/2016 02:07:31
v1.12.1-rc1 3879305 tiborvass  13/08/2016 01:25:24 13/08/2016 01:28:06
v1.12.1-rc2 3909470 tiborvass  17/08/2016 19:50:45 17/08/2016 19:53:00
v1.12.1     3919520 tiborvass  18/08/2016 20:14:05 18/08/2016 20:19:55
v1.12.2-rc1 4246481 vieux      27/09/2016 22:37:47 28/09/2016 02:05:20
v1.12.2-rc2 4304701 vieux      04/10/2016 07:37:23 05/10/2016 01:41:11
v1.12.2-rc3 4336430 vieux      06/10/2016 23:27:01 07/10/2016 21:15:16
v1.12.2     4364345 vieux      11/10/2016 07:23:52 12/10/2016 01:35:27
I am immediately surprised to see that the last (and more recent) release of Docker is the 1.12.2. I have been playing with Docker under Windows 2016 enough to know that there is a 1.13 version under development. So why can't I see it here?

Well, the answer is simple. Github doesn't show you the Master Build of Docker. For those that are encountering problems with Docker on Windows 2016, and for those that love to always have the last version no matter what, master.dockerproject.org is the place to look for:



Unfortunately there's no RestAPI for this site, and since it returns a table in old style HTML code, Invoke-RestMethod has no use here.

Happily enough there's a nice fail-back solution: using Invoke-WebRequest in conjunction with a nice script developed by Lee Holmes that does the job of extracting tables from web pages.


Save the code in a file named get-webrequesttable.ps1 so that you can reuse it, and feed it with the output of Invoke-WebRequest:
$uri = 'https://master.dockerproject.org/'

$r = iwr $uri

$o = .\get-webrequesttable.ps1' $r -TableNumber 0 

$o | Get-Member


   TypeName: System.Management.Automation.PSCustomObject

Name          MemberType   Definition                                   
----          ----------   ----------                                   
Equals        Method       bool Equals(System.Object obj)               
GetHashCode   Method       int GetHashCode()                            
GetType       Method       type GetType()                               
ToString      Method       string ToString()                            
Name          NoteProperty string Name=commit                           
Size          NoteProperty string Size=40 B                             
Uploaded Date NoteProperty string Uploaded Date=2016-10-20T07:12:53.000Z
Lee's script has found three columns and built an object with three properties: name, size and date of the upload.

With Format-Table we can produce a readable object:
$o | ft name,size,@{n='uploadeddate';e={get-date $_.'uploaded date'}} -auto

Name                                             Size     uploadeddate       
----                                             ----     ------------       
commit                                           40 B     20/10/2016 09:12:53
darwin/amd64/docker                              10.35 MB 20/10/2016 09:12:53
darwin/amd64/docker-1.11.0-dev                   10.44 MB 14/04/2016 22:14:23
darwin/amd64/docker-1.11.0-dev.md5               52 B     14/04/2016 22:14:24
darwin/amd64/docker-1.11.0-dev.sha256            84 B     14/04/2016 22:14:24
darwin/amd64/docker-1.11.0-dev.tgz               3.176 MB 14/04/2016 22:14:45
darwin/amd64/docker-1.11.0-dev.tgz.md5           56 B     14/04/2016 22:14:46
darwin/amd64/docker-1.11.0-dev.tgz.sha256        88 B     14/04/2016 22:14:46
darwin/amd64/docker-1.12.0-dev                   13.77 MB 29/07/2016 19:01:15
darwin/amd64/docker-1.12.0-dev.md5               52 B     29/07/2016 19:01:15
Since I am interested just in the versions of Docker for Windows, I can add a bit of filtering:
$o |? name -Match windows | ft name,size,@{n='uploadeddate';e={get-date $_.'uploaded date'}} -auto

Name                                             Size     uploadeddate       
----                                             ----     ------------       
windows/386/docker-1.11.0-dev.exe                9.456 MB 14/04/2016 22:14:38
windows/386/docker-1.11.0-dev.exe.md5            56 B     14/04/2016 22:14:39
windows/386/docker-1.11.0-dev.exe.sha256         88 B     14/04/2016 22:14:39
windows/386/docker-1.11.0-dev.tgz                3.089 MB 01/04/2016 01:28:35
windows/386/docker-1.11.0-dev.tgz.md5            56 B     01/04/2016 01:28:35
windows/386/docker-1.11.0-dev.tgz.sha256         88 B     01/04/2016 01:28:35
windows/386/docker-1.11.0-dev.zip                3.092 MB 14/04/2016 22:14:50
windows/386/docker-1.11.0-dev.zip.md5            56 B     14/04/2016 22:14:51
windows/386/docker-1.11.0-dev.zip.sha256         88 B     14/04/2016 22:14:51
windows/386/docker-1.12.0-dev.exe                12.3 MB  29/07/2016 19:01:36
windows/386/docker-1.12.0-dev.exe.md5            56 B     29/07/2016 19:01:37
windows/386/docker-1.12.0-dev.exe.sha256         88 B     29/07/2016 19:01:37
windows/386/docker-1.12.0-dev.zip                4.041 MB 29/07/2016 19:01:50
windows/386/docker-1.12.0-dev.zip.md5            56 B     29/07/2016 19:01:50
windows/386/docker-1.12.0-dev.zip.sha256         88 B     29/07/2016 19:01:50
windows/386/docker-1.13.0-dev.exe                10.55 MB 20/10/2016 09:13:18
windows/386/docker-1.13.0-dev.exe.md5            56 B     20/10/2016 09:13:19
windows/386/docker-1.13.0-dev.exe.sha256         88 B     20/10/2016 09:13:20
windows/386/docker-1.13.0-dev.zip                3.698 MB 20/10/2016 09:14:14
windows/386/docker-1.13.0-dev.zip.md5            56 B     20/10/2016 09:14:14
windows/386/docker-1.13.0-dev.zip.sha256         88 B     20/10/2016 09:14:14
windows/386/docker.exe                           10.55 MB 20/10/2016 09:13:20
windows/amd64/docker-1.11.0-dev.exe              30.46 MB 14/04/2016 22:14:40
windows/amd64/docker-1.11.0-dev.exe.md5          56 B     14/04/2016 22:14:43
windows/amd64/docker-1.11.0-dev.exe.sha256       88 B     14/04/2016 22:14:43
windows/amd64/docker-1.11.0-dev.tgz              8.567 MB 01/04/2016 01:28:35
windows/amd64/docker-1.11.0-dev.tgz.md5          56 B     01/04/2016 01:28:35
windows/amd64/docker-1.11.0-dev.tgz.sha256       88 B     01/04/2016 01:28:36
windows/amd64/docker-1.11.0-dev.zip              8.591 MB 14/04/2016 22:14:51
windows/amd64/docker-1.11.0-dev.zip.md5          56 B     14/04/2016 22:14:52
windows/amd64/docker-1.11.0-dev.zip.sha256       88 B     14/04/2016 22:14:52
windows/amd64/docker-1.12.0-dev.exe              15.19 MB 29/07/2016 19:01:38
windows/amd64/docker-1.12.0-dev.exe.md5          56 B     29/07/2016 19:01:39
windows/amd64/docker-1.12.0-dev.exe.sha256       88 B     29/07/2016 19:01:39
windows/amd64/docker-1.12.0-dev.zip              16.37 MB 29/07/2016 19:01:51
windows/amd64/docker-1.12.0-dev.zip.md5          56 B     29/07/2016 19:01:51
windows/amd64/docker-1.12.0-dev.zip.sha256       88 B     29/07/2016 19:01:52
windows/amd64/docker-1.13.0-dev.exe              11.67 MB 20/10/2016 09:13:20
windows/amd64/docker-1.13.0-dev.exe.md5          56 B     20/10/2016 09:13:21
windows/amd64/docker-1.13.0-dev.exe.sha256       88 B     20/10/2016 09:13:21
windows/amd64/docker-1.13.0-dev.zip              14.65 MB 20/10/2016 09:14:15
windows/amd64/docker-1.13.0-dev.zip.md5          56 B     20/10/2016 09:14:17
windows/amd64/docker-1.13.0-dev.zip.sha256       88 B     20/10/2016 09:14:17
windows/amd64/docker-proxy-1.12.0-dev.exe        2.936 MB 29/07/2016 19:01:39
windows/amd64/docker-proxy-1.12.0-dev.exe.md5    62 B     29/07/2016 19:01:39
windows/amd64/docker-proxy-1.12.0-dev.exe.sha256 94 B     29/07/2016 19:01:39
windows/amd64/docker-proxy-1.13.0-dev.exe        1.875 MB 20/10/2016 09:13:21
windows/amd64/docker-proxy-1.13.0-dev.exe.md5    62 B     20/10/2016 09:13:22
windows/amd64/docker-proxy-1.13.0-dev.exe.sha256 94 B     20/10/2016 09:13:22
windows/amd64/docker-proxy.exe                   1.875 MB 20/10/2016 09:13:22
windows/amd64/docker.exe                         11.67 MB 20/10/2016 09:13:22
windows/amd64/dockerd-1.12.0-dev.exe             40.28 MB 29/07/2016 19:01:40
windows/amd64/dockerd-1.12.0-dev.exe.md5         57 B     29/07/2016 19:01:42
windows/amd64/dockerd-1.12.0-dev.exe.sha256      89 B     29/07/2016 19:01:42
windows/amd64/dockerd-1.13.0-dev.exe             32.42 MB 20/10/2016 09:13:23
windows/amd64/dockerd-1.13.0-dev.exe.md5         57 B     20/10/2016 09:13:25
windows/amd64/dockerd-1.13.0-dev.exe.sha256      89 B     20/10/2016 09:13:25
windows/amd64/dockerd.exe                        32.42 MB 20/10/2016 09:13:25
Excluding all that's not a zip archive is achieved with Regex:
$o |? name -Match ^windows.*?zip$ | ft name,size,@{n='uploadeddate';e={get-date $_.'uploaded date'}} -auto

Name                                Size     uploadeddate       
----                                ----     ------------       
windows/386/docker-1.11.0-dev.zip   3.092 MB 14/04/2016 22:14:50
windows/386/docker-1.12.0-dev.zip   4.041 MB 29/07/2016 19:01:50
windows/386/docker-1.13.0-dev.zip   3.698 MB 20/10/2016 09:14:14
windows/amd64/docker-1.11.0-dev.zip 8.591 MB 14/04/2016 22:14:51
windows/amd64/docker-1.12.0-dev.zip 16.37 MB 29/07/2016 19:01:51
windows/amd64/docker-1.13.0-dev.zip 14.65 MB 20/10/2016 09:14:15
So cool. We have the list of the latest master builds for each release. Now, getting only the current master build is just one step away:
($o |? name -Match ^windows.*?zip$)[-1] | ft name,size,@{n='uploadeddate';e={get-date $_.'uploaded date'}} -auto

Name                                Size     uploadeddate       
----                                ----     ------------       
windows/amd64/docker-1.13.0-dev.zip 14.65 MB 20/10/2016 09:14:15
Hey, this is newer than the one I currently have, so let me download it:
$l = ($o |? name -Match ^windows.*?zip$)[-1].Name

iwr "$uri$l" -OutFile "$env:TEMP\docker.zip" -UseBasicParsing
I am out testing it. If you have any question on the code or on the aliases that I have used, do not hesitate to ask.

Wednesday, October 19, 2016

First steps with Microsoft Containers - part 3

A lot of things could not work out of the box when you move your first steps with Docker (which we have learned to install in this previous post) under Windows 2016. This is a new project, the partnership between Microsoft and Docker is definitively recent, and while the community is contributing greatly to it, it's always good to know where the logs are when you can't get things to work, like in the following screenshot:




A post by a technical write at Microsoft showed how to use Get-EventLog to retrieve events generated by the Docker engine:
Get-EventLog -LogName Application -Source Docker -After (Get-Date).AddMinutes(-1000) |
        Sort-Object Time
Now, Get-EventLog is sort of legacy cmdlet that is supported by older versions of Windows  PowerShell.

Get-WinEvent is the cmdlet you want to be using, since it is definitively faster.




Get-WinEvent -FilterHashtable @{
        logname='application'
        providername='docker'
        starttime=((Get-Date).AddMinutes(-1000))
        }

   ProviderName: docker

TimeCreated               Id LevelDisplayName Message                                                            
-----------               -- ---------------- -------                                                            
18/10/2016 14:14:44       1  Error            Handler for GET /v1.24/images...
18/10/2016 14:14:26       1  Information      API listen on //./pipe/docker...
18/10/2016 14:14:26       11 Information      Docker daemon [version=1.12.2...
18/10/2016 14:14:26       1  Information      Daemon has completed initiali...
18/10/2016 14:14:26       1  Information      Loading containers: done.
18/10/2016 14:14:26       1  Information      Loading containers: start.
18/10/2016 14:14:26       1  Information      Graph migration to content-ad...
18/10/2016 14:14:25       1  Information      [graphdriver] using prior sto...
18/10/2016 14:14:25       1  Information      Windows default isolation mod...
The difference in performance is easily demonstrated:
Measure-Command {Get-EventLog -LogName Application -Source Docker}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 527
Ticks             : 5274839
TotalDays         : 6,10513773148148E-06
TotalHours        : 0,000146523305555556
TotalMinutes      : 0,00879139833333333
TotalSeconds      : 0,5274839
TotalMilliseconds : 527,4839

Measure-Command {Get-WinEvent -FilterHashtable @{
        logname='application'
        providername='docker'}
        }

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 80
Ticks             : 801087
TotalDays         : 9,27184027777778E-07
TotalHours        : 2,22524166666667E-05
TotalMinutes      : 0,001335145
TotalSeconds      : 0,0801087
TotalMilliseconds : 80,1087

Stay tuned for more Docker tips.

Tuesday, October 18, 2016

First steps with Microsoft Containers - part 2

As I said in the previous post, Docker is required in order to work with Windows containers. The source can be downloaded and installed in a couple simple steps:
Invoke-WebRequest "https://get.docker.com/builds/Windows/x86_64/docker-latest.zip" -OutFile "$env:TEMP\docker-latest.zip" -UseBasicParsing

Expand-Archive $env:TEMP\docker-latest.zip -DestinationPath $env:ProgramFiles

[Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:\Program Files\Docker", [EnvironmentVariableTarget]::Machine)
Closed and reopen PowerShell, then:
& $env:ProgramFiles\docker\dockerd.exe --register-service

Start-Service docker
There you are.



EXPLORING DOCKER

Once you have installed Docker on Windows, you can explore the possible parameters:

Usage: docker [OPTIONS] COMMAND [arg...]
       docker [ --help | -v | --version ]

A self-sufficient runtime for containers.

Options:

  --config=%USERPROFILE%\.docker              Location of client config files
  -D, --debug                                 Enable debug mode
  -H, --host=[]                               Daemon socket(s) to connect to
  -h, --help                                  Print usage
  -l, --log-level=info                        Set the logging level
  --tls                                       Use TLS; implied by --tlsverify
  --tlscacert=%USERPROFILE%\.docker\ca.pem    Trust certs signed only by this CA
  --tlscert=%USERPROFILE%\.docker\cert.pem    Path to TLS certificate file
  --tlskey=%USERPROFILE%\.docker\key.pem      Path to TLS key file
  --tlsverify                                 Use TLS and verify the remote
  -v, --version                               Print version information and quit

Commands:
    attach    Attach to a running container
    build     Build an image from a Dockerfile
    commit    Create a new image from a container's changes
    cp        Copy files/folders between a container and the local filesystem
    create    Create a new container
    diff      Inspect changes on a container's filesystem
    events    Get real time events from the server
    exec      Run a command in a running container
    export    Export a container's filesystem as a tar archive
    history   Show the history of an image
    images    List images
    import    Import the contents from a tarball to create a filesystem image
    info      Display system-wide information
    inspect   Return low-level information on a container, image or task
    kill      Kill one or more running containers
    load      Load an image from a tar archive or STDIN
    login     Log in to a Docker registry.
    logout    Log out from a Docker registry.
    logs      Fetch the logs of a container
    network   Manage Docker networks
    node      Manage Docker Swarm nodes
    pause     Pause all processes within one or more containers
    port      List port mappings or a specific mapping for the container
    ps        List containers
    pull      Pull an image or a repository from a registry
    push      Push an image or a repository to a registry
    rename    Rename a container
    restart   Restart a container
    rm        Remove one or more containers
    rmi       Remove one or more images
    run       Run a command in a new container
    save      Save one or more images to a tar archive (streamed to STDOUT by default)
    search    Search the Docker Hub for images
    service   Manage Docker services
    start     Start one or more stopped containers
    stats     Display a live stream of container(s) resource usage statistics
    stop      Stop one or more running containers
    swarm     Manage Docker Swarm
    tag       Tag an image into a repository
    top       Display the running processes of a container
    unpause   Unpause all processes within one or more containers
    update    Update configuration of one or more containers
    version   Show the Docker version information
    volume    Manage Docker volumes
    wait      Block until a container stops, then print its exit code

Run 'docker COMMAND --help' for more information on a command.

Have a look at each parameter, they are easily understood. The key commands to start with are 'pull' and 'run'. The first one is used to pull an image from the Docker public registry, which is basically a repo of all the images found on the Docker Hub (https://hub.docker.com/):




The second one is used to bring up the Container, and is often used in conjunction with the -it switches, where 'i' makes it interactive and 't' opens a pseudo terminal. You can also directly start a Container with run without pulling it from the registry first: the Docker client will do it for you.

GETTING AROUND PROXY ISSUES

If, like me, you are behind a corporate proxy, and aren't able to connect directly to the internet, you have two alternatives: either downloading the two classical Microsoft Containers from https://aka.ms/tp5/6b/docker/nanoserver or https://aka.ms/tp5/6b/docker/windowsservercore or use the Start-BitsTransfer cmdlet, which is used to replace Install-ContainerOSImage (this has been removed, as I said in the previous post).

Start-BitsTransfer https://aka.ms/tp5/6b/docker/nanoserver -Destination nanoserver_10.0.14300.1030_4.tar.gz

docker load -i .\nanoserver_10.0.14300.1030_4.tar.gz
Once you get the message 'Loaded image: microsoft/nanoserver:10.0.14300.1030' you are good to use the NanoServer image:
docker images
REPOSITORY             TAG                 IMAGE ID            CREATED             SIZE
microsoft/nanoserver   10.0.14300.1030     3a703c6e97a2        4 months ago        969.8 MB
The tag here is important, because, in case of multiple images, you can refer to it by the tag.

HOW TO RUN A CONTAINER

Nothing simpler:

docker run -it microsoft/nanoserver:10.0.14300.1030 powershell

TWO COMMON MISTAKES

Now, a common beginner mistake, if you try to run the NanoServer Container on a full Windows 2016 Server, you get:

docker run -it microsoft/nanoserver:10.0.14300.1030 powershell
C:\Program Files\Docker\docker.exe: Error response from daemon: container 379f187cf37ff65df4ffbc4cc2dc98441a7e932443f155596f17f4e066c1585c encountered an error during CreateContainer failed in Win32: The operating system of the container does not match the operating system of the host.
(0xc0370101) extra info:{"SystemType":"Container","Name":"379f187cf37ff65df4ffbc4cc2dc98441a7e932443f155596f17f4e066c1585c","Owner":"docker","IsDummy":false,"VolumePath":"\\\\?\\Volume{4d073ac2-946711e6ae0a005056836799}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\379f187cf37ff65df4ffbc4cc2dc98441a7e932443f155596f17f4e066c1585c","Layers"[{"ID":"115df3f3089b5f51ba901cf50b70e57a","Path":"C:\\ProgramData\\docker\\windowsfilter\\15d854d2cf0ddc892b511b920988221ff68e683df8c71c858d72fe151117b027"}],"HostName":"379f187cf37f","MappedDirectories":[],"SandboxPath":"","HvPartition":false,"EndpointList":["5f95dbcb-2bcc-44e8-a13d-2a91b97ec0ac"],"HvRuntime":null,"Servicing":false}.
As you remember from the previous post, containers are operating system level virtualization, so Nano Server core can only run as a Hyper-V container if the Container host is a full Windows 2016, or as a Windows Container, if the Container host is a NanoServer. Simple matching of kernels.

docker run -it --isolation hyperv microsoft/nanoserver:10.0.14300.1030 powershell
The second common mistake is that, if you don't have the Hyper-V feature installed and try to install a Hyper-V Container, you'll get the following error:

docker run -it --isolation hyperv microsoft/nanoserver:10.0.14300.1030 powershell
C:\Program Files\Docker\docker.exe: Error response from daemon: container cd93f27118a2e80964e4162b8d107a39fd78b7c0de3e2f
6e5377c4f998118c36 encountered an error during CreateContainer failed in Win32: No hypervisor is present on this system.
(0xc0351000) extra info: {"SystemType":"Container","Name":"cd93f27118a2e80964e4162b8d107a39fd78b7c0de3e2f6e5377c4f998118c36","Owner":"docker","IsDummy":false,"VolumePath":"","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\cd93f27118a2e80964e4162b8d107a39fd78b7c0de3e2f6e5377c4f998118c36","Layers":[{"ID":"115df3f3-089b-5f51ba901cf50b70e57a","Path":"C:\\ProgramData\\docker\\windowsfilter\\15d854d2cf0ddc892b511b920988221ff68e683df8c71c858d72fe151117b027"}],"HostName":"cd93f27118a2","MappedDirectories"[],"SandboxPath":"C:\\ProgramData\\docker\\windowsfilter","HvPartition":true,"EndpointList"["977f140941694fb786ff4abb20316975"],"HvRuntime"{"ImagePath":"C:\\ProgramData\\docker\\windowsfilter\\15d854d2cf0ddc892b511b920988221ff68e683df8c71c858d72fe151117b027\\UtilityVM"},"Servicing":false}.
Installing Hyper-V will solve this in a breeze.

OPEN ISSUE WITH THE HOST NETWORK SERVICE

Docker relies on the microsoft/hccshim package to call the Host Compute Service (vmcompute.dll) to run Windows Containers. When the service starts (dockerd.exe) a network setup occurs and a vmswitch is configured. This vmswitch is managed by the Host Network Service (HNS) subsytem which is in charge of the IPAM role.

In my labs I have often encoutered a bug that prevent the Docker service from starting because the HNS subsytem is unable to do all the network setup:

dockerd
time="2016-10-18T02:21:35.312426300+02:00" level=info msg="Windows default isolation mode: process"
time="2016-10-18T02:21:35.339425700+02:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2016-10-18T02:21:35.339425700+02:00" level=info msg="Loading containers: start."
time="2016-10-18T02:21:35.393424100+02:00" level=error msg="Resolver Setup/Start failed for container none, \"json: cann ot unmarshal array into Go value of type hcsshim.HNSNetwork\""
Error starting daemon: Error initializing network controller: Error creating default network: HNS failed with error : Unspecified error
It looks like this issue should be solved installing a cumulative update for Windows 10 Version 1607 and Windows Server 2016: October 11, 2016 (aka 9D).

In my case installing this patch, as well all the others in the standard channel, did not solve my issue:

Get-Hotfix

Source         Description      HotFixID      InstalledBy          InstalledOn
------         -----------      --------      -----------          -----------
SRV1           Update           KB3176936     NT AUTHORITY\SYSTEM  18/10/2016 00:00:00
SRV1           Update           KB3192137     NT AUTHORITY\SYSTEM  12/09/2016 00:00:00
SRV1           Update           KB3199209     NT AUTHORITY\SYSTEM  18/10/2016 00:00:00
SRV1           Security Update  KB3194798     NT AUTHORITY\SYSTEM  18/10/2016 00:00:00
I will post un update as soon as I find out more, but in the mean time you can check the open issue on GitHub.
More on Docker in the next post.

Related Posts Plugin for WordPress, Blogger...