Thursday, October 20, 2016

How to query the Docker repo and find out the latest Master Build with PowerShell

PowerShell can be used to interact with the web and I have therefore decided to use it to stay tuned with the Docker project which I am currently pretty much interested into (check my previous series on Docker on Windows 2016). Docker is an open source project, and when you say open source nowadays you have to think Github, which is basically a hosting service for open source software projects with features for version control, issue tracking, commit history and all the rest.

Now Github has a REST API that can be consumed in PowerShell through the use of the Invoke-RestMethod cmdlet (aliased as irm in your oneliners). The JSON formatted answer is converted by Invoke-RestMethod to a custom object.

Under Github, URLs are in the form api.github.com/user/repos, so for the Docker project we have query api.github.com/repos/Docker/Docker

A simple request against this URL will return all the basic information on the project:
Selecting the properties I could be interested into is easily achieved:
irm api.github.com/repos/Docker/Docker | ft forks,open_issues,watchers

forks open_issues watchers
----- ----------- --------
10626        1929    36095
Once we know that this API can be consumed with PowerShell, we could very well think of retrieving all the published releases. The URL syntax is found in the result on the previous query:

Here's how I can get the first release returned by the API and see how the information is structured:
(irm api.github.com/repos/Docker/Docker/releases)[0]
A lot of information here, including a body containing the full description of the changes that come with the currently listed version.

As I said, we are only interested in getting the published releases of Docker, so let's use a sieve and keep just four key properties: the name of the package, its ID, author, creation date and publication date.

A short oneliner will do:
irm api.github.com/repos/Docker/Docker/releases |

        sort id |

        ft name,id,*at,@{n='author';e={$_.author.login}} -auto
Let's apply a couple of best practices and put the resulting object in a variable, for reusability, as well as add a bit of formatting for the dates:
$u = irm api.github.com/repos/Docker/Docker/releases

$u |
 sort id |
 
 ft  -auto name,id,

  @{n='author';e={$_.author.login}},

  @{n='creationdate';e={get-date $_.created_at}},

  @{n='publicationdate';e={get-date $_.published_at}}
Basically, with
Get-Date '2016-10-11T23:35:27Z'
I am using Get-Date to convert the timestamps, which are expressed in UTC, to my local time zone. It's the Z in the end of the date (which is a special UTC designator) to tell me that the timestamp is expressed in Coordinated Universal Time.

These two lines of code above return:
name             id author     creationdate        publicationdate    
----             -- ------     ------------        ---------------    
v1.10.1-rc1 2590708 tiborvass  10/02/2016 23:12:24 11/02/2016 00:09:31
v1.10.1     2598018 tiborvass  11/02/2016 22:14:44 11/02/2016 22:16:24
v1.10.2-rc1 2652399 tiborvass  20/02/2016 08:00:24 20/02/2016 08:23:08
v1.10.2     2666504 tiborvass  22/02/2016 23:57:57 23/02/2016 00:05:08
v1.10.3-rc1 2777835 tiborvass  09/03/2016 18:11:06 09/03/2016 18:14:08
v1.10.3-rc2 2780060 tiborvass  09/03/2016 22:58:41 09/03/2016 23:02:32
v1.10.3     2788494 tiborvass  10/03/2016 23:01:03 10/03/2016 23:07:27
v1.11.0-rc1 2875983 tiborvass  23/03/2016 21:15:31 23/03/2016 21:20:29
v1.11.0-rc2 2890861 tiborvass  25/03/2016 22:28:59 25/03/2016 22:31:10
v1.11.0-rc3 2937939 tiborvass  02/04/2016 01:59:32 02/04/2016 02:01:26
v1.11.0-rc4 2968912 tiborvass  07/04/2016 04:28:01 07/04/2016 04:56:10
v1.11.0-rc5 2998258 tiborvass  12/04/2016 01:33:25 12/04/2016 01:34:20
v1.11.0     3014278 tiborvass  13/04/2016 21:56:07 14/04/2016 00:10:23
v1.11.1-rc1 3097597 mlaventure 26/04/2016 10:01:02 26/04/2016 10:05:57
v1.11.1     3105125 mlaventure 27/04/2016 03:51:45 27/04/2016 03:59:03
v1.11.2-rc1 3327300 mlaventure 28/05/2016 21:44:50 28/05/2016 23:47:25
v1.11.2     3354503 tiborvass  02/06/2016 02:59:52 02/06/2016 03:08:13
v1.12.0-rc1 3447699 tiborvass  15/06/2016 10:39:54 15/06/2016 10:55:14
v1.12.0-rc2 3471944 tiborvass  17/06/2016 23:39:11 18/06/2016 00:49:34
v1.12.0-rc3 3573896 tiborvass  02/07/2016 05:26:36 02/07/2016 05:30:18
v1.12.0-rc4 3644623 tiborvass  13/07/2016 07:27:26 13/07/2016 07:25:48
v1.12.0-rc5 3744904 tiborvass  26/07/2016 22:48:09 26/07/2016 22:48:18
v1.12.0     3766135 tiborvass  29/07/2016 02:06:45 29/07/2016 02:07:31
v1.12.1-rc1 3879305 tiborvass  13/08/2016 01:25:24 13/08/2016 01:28:06
v1.12.1-rc2 3909470 tiborvass  17/08/2016 19:50:45 17/08/2016 19:53:00
v1.12.1     3919520 tiborvass  18/08/2016 20:14:05 18/08/2016 20:19:55
v1.12.2-rc1 4246481 vieux      27/09/2016 22:37:47 28/09/2016 02:05:20
v1.12.2-rc2 4304701 vieux      04/10/2016 07:37:23 05/10/2016 01:41:11
v1.12.2-rc3 4336430 vieux      06/10/2016 23:27:01 07/10/2016 21:15:16
v1.12.2     4364345 vieux      11/10/2016 07:23:52 12/10/2016 01:35:27
I am immediately surprised to see that the last (and more recent) release of Docker is the 1.12.2. I have been playing with Docker under Windows 2016 enough to know that there is a 1.13 version under development. So why can't I see it here?

Well, the answer is simple. Github doesn't show you the Master Build of Docker. For those that are encountering problems with Docker on Windows 2016, and for those that love to always have the last version no matter what, master.dockerproject.org is the place to look for:



Unfortunately there's no RestAPI for this site, and since it returns a table in old style HTML code, Invoke-RestMethod has no use here.

Happily enough there's a nice fail-back solution: using Invoke-WebRequest in conjunction with a nice script developed by Lee Holmes that does the job of extracting tables from web pages.


Save the code in a file named get-webrequesttable.ps1 so that you can reuse it, and feed it with the output of Invoke-WebRequest:
$uri = 'https://master.dockerproject.org/'

$r = iwr $uri

$o = .\get-webrequesttable.ps1' $r -TableNumber 0 

$o | Get-Member


   TypeName: System.Management.Automation.PSCustomObject

Name          MemberType   Definition                                   
----          ----------   ----------                                   
Equals        Method       bool Equals(System.Object obj)               
GetHashCode   Method       int GetHashCode()                            
GetType       Method       type GetType()                               
ToString      Method       string ToString()                            
Name          NoteProperty string Name=commit                           
Size          NoteProperty string Size=40 B                             
Uploaded Date NoteProperty string Uploaded Date=2016-10-20T07:12:53.000Z
Lee's script has found three columns and built an object with three properties: name, size and date of the upload.

With Format-Table we can produce a readable object:
$o | ft name,size,@{n='uploadeddate';e={get-date $_.'uploaded date'}} -auto

Name                                             Size     uploadeddate       
----                                             ----     ------------       
commit                                           40 B     20/10/2016 09:12:53
darwin/amd64/docker                              10.35 MB 20/10/2016 09:12:53
darwin/amd64/docker-1.11.0-dev                   10.44 MB 14/04/2016 22:14:23
darwin/amd64/docker-1.11.0-dev.md5               52 B     14/04/2016 22:14:24
darwin/amd64/docker-1.11.0-dev.sha256            84 B     14/04/2016 22:14:24
darwin/amd64/docker-1.11.0-dev.tgz               3.176 MB 14/04/2016 22:14:45
darwin/amd64/docker-1.11.0-dev.tgz.md5           56 B     14/04/2016 22:14:46
darwin/amd64/docker-1.11.0-dev.tgz.sha256        88 B     14/04/2016 22:14:46
darwin/amd64/docker-1.12.0-dev                   13.77 MB 29/07/2016 19:01:15
darwin/amd64/docker-1.12.0-dev.md5               52 B     29/07/2016 19:01:15
Since I am interested just in the versions of Docker for Windows, I can add a bit of filtering:
$o |? name -Match windows | ft name,size,@{n='uploadeddate';e={get-date $_.'uploaded date'}} -auto

Name                                             Size     uploadeddate       
----                                             ----     ------------       
windows/386/docker-1.11.0-dev.exe                9.456 MB 14/04/2016 22:14:38
windows/386/docker-1.11.0-dev.exe.md5            56 B     14/04/2016 22:14:39
windows/386/docker-1.11.0-dev.exe.sha256         88 B     14/04/2016 22:14:39
windows/386/docker-1.11.0-dev.tgz                3.089 MB 01/04/2016 01:28:35
windows/386/docker-1.11.0-dev.tgz.md5            56 B     01/04/2016 01:28:35
windows/386/docker-1.11.0-dev.tgz.sha256         88 B     01/04/2016 01:28:35
windows/386/docker-1.11.0-dev.zip                3.092 MB 14/04/2016 22:14:50
windows/386/docker-1.11.0-dev.zip.md5            56 B     14/04/2016 22:14:51
windows/386/docker-1.11.0-dev.zip.sha256         88 B     14/04/2016 22:14:51
windows/386/docker-1.12.0-dev.exe                12.3 MB  29/07/2016 19:01:36
windows/386/docker-1.12.0-dev.exe.md5            56 B     29/07/2016 19:01:37
windows/386/docker-1.12.0-dev.exe.sha256         88 B     29/07/2016 19:01:37
windows/386/docker-1.12.0-dev.zip                4.041 MB 29/07/2016 19:01:50
windows/386/docker-1.12.0-dev.zip.md5            56 B     29/07/2016 19:01:50
windows/386/docker-1.12.0-dev.zip.sha256         88 B     29/07/2016 19:01:50
windows/386/docker-1.13.0-dev.exe                10.55 MB 20/10/2016 09:13:18
windows/386/docker-1.13.0-dev.exe.md5            56 B     20/10/2016 09:13:19
windows/386/docker-1.13.0-dev.exe.sha256         88 B     20/10/2016 09:13:20
windows/386/docker-1.13.0-dev.zip                3.698 MB 20/10/2016 09:14:14
windows/386/docker-1.13.0-dev.zip.md5            56 B     20/10/2016 09:14:14
windows/386/docker-1.13.0-dev.zip.sha256         88 B     20/10/2016 09:14:14
windows/386/docker.exe                           10.55 MB 20/10/2016 09:13:20
windows/amd64/docker-1.11.0-dev.exe              30.46 MB 14/04/2016 22:14:40
windows/amd64/docker-1.11.0-dev.exe.md5          56 B     14/04/2016 22:14:43
windows/amd64/docker-1.11.0-dev.exe.sha256       88 B     14/04/2016 22:14:43
windows/amd64/docker-1.11.0-dev.tgz              8.567 MB 01/04/2016 01:28:35
windows/amd64/docker-1.11.0-dev.tgz.md5          56 B     01/04/2016 01:28:35
windows/amd64/docker-1.11.0-dev.tgz.sha256       88 B     01/04/2016 01:28:36
windows/amd64/docker-1.11.0-dev.zip              8.591 MB 14/04/2016 22:14:51
windows/amd64/docker-1.11.0-dev.zip.md5          56 B     14/04/2016 22:14:52
windows/amd64/docker-1.11.0-dev.zip.sha256       88 B     14/04/2016 22:14:52
windows/amd64/docker-1.12.0-dev.exe              15.19 MB 29/07/2016 19:01:38
windows/amd64/docker-1.12.0-dev.exe.md5          56 B     29/07/2016 19:01:39
windows/amd64/docker-1.12.0-dev.exe.sha256       88 B     29/07/2016 19:01:39
windows/amd64/docker-1.12.0-dev.zip              16.37 MB 29/07/2016 19:01:51
windows/amd64/docker-1.12.0-dev.zip.md5          56 B     29/07/2016 19:01:51
windows/amd64/docker-1.12.0-dev.zip.sha256       88 B     29/07/2016 19:01:52
windows/amd64/docker-1.13.0-dev.exe              11.67 MB 20/10/2016 09:13:20
windows/amd64/docker-1.13.0-dev.exe.md5          56 B     20/10/2016 09:13:21
windows/amd64/docker-1.13.0-dev.exe.sha256       88 B     20/10/2016 09:13:21
windows/amd64/docker-1.13.0-dev.zip              14.65 MB 20/10/2016 09:14:15
windows/amd64/docker-1.13.0-dev.zip.md5          56 B     20/10/2016 09:14:17
windows/amd64/docker-1.13.0-dev.zip.sha256       88 B     20/10/2016 09:14:17
windows/amd64/docker-proxy-1.12.0-dev.exe        2.936 MB 29/07/2016 19:01:39
windows/amd64/docker-proxy-1.12.0-dev.exe.md5    62 B     29/07/2016 19:01:39
windows/amd64/docker-proxy-1.12.0-dev.exe.sha256 94 B     29/07/2016 19:01:39
windows/amd64/docker-proxy-1.13.0-dev.exe        1.875 MB 20/10/2016 09:13:21
windows/amd64/docker-proxy-1.13.0-dev.exe.md5    62 B     20/10/2016 09:13:22
windows/amd64/docker-proxy-1.13.0-dev.exe.sha256 94 B     20/10/2016 09:13:22
windows/amd64/docker-proxy.exe                   1.875 MB 20/10/2016 09:13:22
windows/amd64/docker.exe                         11.67 MB 20/10/2016 09:13:22
windows/amd64/dockerd-1.12.0-dev.exe             40.28 MB 29/07/2016 19:01:40
windows/amd64/dockerd-1.12.0-dev.exe.md5         57 B     29/07/2016 19:01:42
windows/amd64/dockerd-1.12.0-dev.exe.sha256      89 B     29/07/2016 19:01:42
windows/amd64/dockerd-1.13.0-dev.exe             32.42 MB 20/10/2016 09:13:23
windows/amd64/dockerd-1.13.0-dev.exe.md5         57 B     20/10/2016 09:13:25
windows/amd64/dockerd-1.13.0-dev.exe.sha256      89 B     20/10/2016 09:13:25
windows/amd64/dockerd.exe                        32.42 MB 20/10/2016 09:13:25
Excluding all that's not a zip archive is achieved with Regex:
$o |? name -Match ^windows.*?zip$ | ft name,size,@{n='uploadeddate';e={get-date $_.'uploaded date'}} -auto

Name                                Size     uploadeddate       
----                                ----     ------------       
windows/386/docker-1.11.0-dev.zip   3.092 MB 14/04/2016 22:14:50
windows/386/docker-1.12.0-dev.zip   4.041 MB 29/07/2016 19:01:50
windows/386/docker-1.13.0-dev.zip   3.698 MB 20/10/2016 09:14:14
windows/amd64/docker-1.11.0-dev.zip 8.591 MB 14/04/2016 22:14:51
windows/amd64/docker-1.12.0-dev.zip 16.37 MB 29/07/2016 19:01:51
windows/amd64/docker-1.13.0-dev.zip 14.65 MB 20/10/2016 09:14:15
So cool. We have the list of the latest master builds for each release. Now, getting only the current master build is just one step away:
($o |? name -Match ^windows.*?zip$)[-1] | ft name,size,@{n='uploadeddate';e={get-date $_.'uploaded date'}} -auto

Name                                Size     uploadeddate       
----                                ----     ------------       
windows/amd64/docker-1.13.0-dev.zip 14.65 MB 20/10/2016 09:14:15
Hey, this is newer than the one I currently have, so let me download it:
$l = ($o |? name -Match ^windows.*?zip$)[-1].Name

iwr "$uri$l" -OutFile "$env:TEMP\docker.zip" -UseBasicParsing
I am out testing it. If you have any question on the code or on the aliases that I have used, do not hesitate to ask.

No comments:

Post a Comment

Related Posts Plugin for WordPress, Blogger...