Wednesday, November 30, 2016

A PowerShell function to monitor physical disk activity per storage bay during sync activities

These days I have been migrating data on a old Windows 2003 server from an old HP XP128 storage array to a newer one, a HP 3PAR. Both those fiber channel SAN were mounted and managed on the server through Veritas Enterprise Administrator (VEA) version 5.1. At first I started with a Robocopy to migrate data, ACLs, and all the rest from the old volume to the new one, but soon discovered that there could be better ways to move huge amount of data (I am talking here of several million sensitive files).

One of the main advantages of using Robocopy is that you have fine grained control over your sync. The downside is that, after the sync, you have to stop the old volume and move all your pointers to the new volume, which has a big impact on the automation systems relying on those files to keep their 24/7 activity.

I decided then to change plans and build a mirror on VEA between the old storage array and the new one.

The only problem with such a old version of VEA is that you don't have access to such a basic information as the fact that the mirror sync is completed. The interface just shows you that you have successfully built your mirror but hides the information about the actual data sync taking place behind the curtains.

That's the moment the manager came in and asked for a way to keep an eye the sync. And that's the moment I replied: I can do that for you with PowerShell, sir.

I knew fairly well that though there's no PowerShell on a Windows 2003 server unless you have taken the time to install it, I could access its performance counters from a distant, more recent workstation trough Get-Counter:

What I wanted was to give the manager a script he could run himself that showed the activity for the disks involved in the sync. So I knew that I had to rely on cmdlets I am not used to put in my functions, such as Clear-Host or Write-Host.
But, as for anything else, there are times you have to make exceptions. And Write-Host can have its use sometimes.
In the end I came up with a function that given a set of physical disks on a source server and on a target server, monitors the disk activity in terms of read and written bytes per seconds and in case those are not null, set the font color to green, so that they're highlighted.

The names of the disks can be found in the Perfmon GUI itself as well as in VEA:


Their names can be given as input to the functions as a pattern for a regular expression. In my case this gave:

-SourceDiskPattern '\(2\)|\(14\)'
and
-DestinationDiskPattern '\(8\)|\(9\)|\(10\)|\(20\)|\(21\)|\(22\)' 
because I am trying to match those hard disk numbers.

I have also added to the function a couple of parameters to show the processor activity and the disk queue, since these counters can always be of use when tracing a workload:


In the end, here's the output expected by the manager during the sync, with the green lines highlighting the disks where the data are read or written:


I rely on Clear-Host to refresh the screen so that the manager can only see the current workload. This can be a bad practice as Invoke-ScriptAnalyzer will tell you, but in my case this is exactly the cmdlet I needed.


Here's the code for the Get-DiskStat function, which by the way you can find on Github:

<#
.Synopsis
   Monitors physical disk activity per bay during sync activities
.DESCRIPTION
   Monitors physical disk activity during a sync and highlights the disks that are active reading or writing bytes and the bay they belong to
.EXAMPLE
   Get-DiskStat -SourceComputer srv1 -SourceStorageBayName 'HPE 3PAR' -SourceDiskPattern '\(2\)|\(14\)' -DestinationComputer srv2 -DestinationStorageBayName 'HP XP128' -DestinationDiskPattern '\(7\)|\(10\)' -Refresh -Frequency 2 -Repeat 10
.EXAMPLE
   Get-DiskStat -SourceComputer srv1 -SourceStorageBayName 'HPE 3PAR' -SourceDiskPattern '\(2\)|\(14\)' -DestinationComputer srv2 -DestinationStorageBayName 'HP XP128' -DestinationDiskPattern '\(7\)|\(10\)' -Refresh -ShowCpu
.EXAMPLE
   Get-DiskStat -SourceComputer srv1 -SourceStorageBayName 'HPE 3PAR' -SourceDiskPattern '\(2\)|\(14\)' -DestinationComputer srv2 -DestinationStorageBayName 'HP XP128' -DestinationDiskPattern '\(7\)|\(10\)' -Refresh -ShowCpu -ShowQueue
.EXAMPLE
   Get-DiskStat -SourceComputer srv1 -SourceStorageBayName 'HPE 3PAR' -SourceDiskPattern '\(2\)|\(14\)' -DestinationComputer srv2 -DestinationStorageBayName 'HP XP128' -DestinationDiskPattern '\(7\)|\(10\)' -Refresh -ShowCpu -ShowQueue -Credential (Get-Credential)
.EXAMPLE
   Get-DiskStat -sc srv1 -sbn 'HPE 3PAR' -sdp '\(2\)|\(14\)' -dc srv2 -dbn 'HP XP128' -ddp '\(7\)|\(10\)' -R -C -Q -Cred (Get-Credential) -F 1 -rep 1000
.AUTHOR
   Carlo MANCINI
#>
function Get-DiskStat
{
    Param
    (
        # Source computer for the sync
        [Parameter(Mandatory=$true,Position=0)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("sc")]
        $SourceComputer,

        # Source bay name for the sync
        [Parameter(Mandatory=$true,Position=1)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("sbn")]
        $SourceStorageBayName,

        # Source disk pattern for the sync
        [Parameter(Mandatory=$true,Position=2)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("sdp")]
        $SourceDiskPattern,

        # Destination computer for the sync
        [Parameter(Mandatory=$true,Position=3)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("dc")]
        $DestinationComputer,

        # Destination bay name for the sync
        [Parameter(Mandatory=$true,Position=4)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("dbn")]
        $DestinationStorageBayName,

        # Destination disk pattern for the sync
        [Parameter(Mandatory=$true,Position=5)]
        [ValidateNotNull()]
        [ValidateNotNullOrEmpty()]
        [Alias("ddp")]
        $DestinationDiskPattern,

        # Clear the screen between each execution
        [Parameter(Position=6)][Alias("r")][Switch]$Refresh,

        # Show Active and Idle CPU counters
        [Parameter(Position=7)][Alias("c")][Switch]$ShowCpu,

        # Show disk queue for selected disks
        [Parameter(Position=8)][Alias("q")][Switch]$ShowQueue,

        # Specifies a user account that has permission to perform this action
        [Parameter(Mandatory=$false,Position=9)]
        [System.Management.Automation.PSCredential]
        [System.Management.Automation.Credential()]
        $Credential = [System.Management.Automation.PSCredential]::Empty,

        # Frequency of the polling in seconds
        [Parameter(Position=10)]
        [Alias("f")]
        $Frequency = 10,

        # Total number of polling to perform
        [Parameter(Position=11)]
        [Alias("rep")]
        $Repeat = 10

    )
    
    Try { 
    
        Test-Connection $SourceComputer,$DestinationComputer -Count 1 -ErrorAction Stop | Out-Null
        
        }
    
    Catch {
    
        Throw "At least one of the target servers is not reachable. Exiting."
        
        }

    $CounterList = '\PhysicalDisk(*)\Disk Read Bytes/sec','\PhysicalDisk(*)\Disk Write Bytes/sec','\PhysicalDisk(*)\Current Disk Queue Length','\Processor(_Total)\% Idle Time','\Processor(_Total)\% Processor Time'

    1..$Repeat | % {

        $SourceCounterValue = (Get-Counter $CounterList -ComputerName $SourceComputer).countersamples

        if($DestinationComputer -eq $SourceComputer) {

            $DestinationCounterValue = $SourceCounterValue

            $SameHost = $True

            }

        else {
        
            $DestinationCounterValue = (Get-Counter $CounterList -ComputerName $DestinationComputer).countersamples

            }

        if($Refresh) {Clear-Host}

        if($ShowCpu) {

                    "$SourceComputer CPU Activity & Idle"
                    
                    $SourceCounterValue | ? {$_.path -match 'processor'} | % {
    
                            Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10)
                            
                            }

                    if(!$SameHost) {
                    
                        "$DestinationComputer CPU Activity & Idle"
                    
                        $DestinationCounterValue | ? {$_.path -match 'processor'} | % {
    
                            Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10)
                            
                            }
                        }

                    }

        if($ShowQueue) {

            "$SourceStorageBayName Storage Bay Disk Queue on $SourceComputer"

            $SourceCounterValue | ? {($_.path -match $SourceDiskPattern) -and ($_.path -match 'queue')} | % {
    
                    if($_.cookedvalue -gt 0) {
                    
                        Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $_.cookedvalue.tostring().padright(10) -ForegroundColor Green
                            
                        }
    
                    else {
                    
                        Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $_.cookedvalue.tostring().padright(10) -ForegroundColor White
                        
                        }
                    
                    }

            "$DestinationStorageBayName Storage Bay Disk Queue on $DestinationComputer"

            $DestinationCounterValue | ? {($_.path -match $DestinationDiskPattern) -and ($_.path -match 'queue')} | % {
    
                    if($_.cookedvalue -gt 0) {
                    
                        Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $_.cookedvalue.tostring().padright(10) -ForegroundColor Green
                            
                        }
    
                    else {
                    
                        Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $_.cookedvalue.tostring().padright(10) -ForegroundColor White
                        
                        }
                    
                    }

            }


        "$SourceStorageBayName Read stats on $SourceComputer"

        $SourceCounterValue | ? {($_.path -match $SourceDiskPattern) -and ($_.path -match 'read')} | % {
    
            if($_.cookedvalue -gt 0) {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor Green
                
                }
    
            else {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor White
                
                }
            
            }

        "$SourceStorageBayName Write stats on $SourceComputer"

        $SourceCounterValue | ? {($_.path -match $SourceDiskPattern) -and ($_.path -match 'write')} | % {
    
            if($_.cookedvalue -gt 0) {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor Green
                
                }
    
            else {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor White
                
                }
            
            }

        "$DestinationStorageBayName Read stats on $DestinationComputer"

        $DestinationCounterValue | ? {($_.path -match $DestinationDiskPattern) -and ($_.path -match 'read')} | % {
    
            if($_.cookedvalue -gt 0) {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor Green
                
                }
    
            else {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor White
                
                }
            
            }

        "$DestinationStorageBayName Write stats on $DestinationComputer"

        $DestinationCounterValue | ? {($_.path -match $DestinationDiskPattern) -and ($_.path -match 'write')} | % {
    
            if($_.cookedvalue -gt 0) {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor Green
                
                }
    
            else {
            
                Write-Host $_.path.padright(65)`t $_.InstanceName.padright(5)`t $([math]::round($_.cookedvalue)).tostring().padright(10) -ForegroundColor White
                
                }
            
            }

        Start-Sleep -Seconds $frequency

        }    

}
PowerShell, once again the tool for the job.

No comments:

Post a Comment

Related Posts Plugin for WordPress, Blogger...