Windows Docker Gitlab CI Runner (2019)
2019-02-11 20:40:51    1883    0    0
andrew@alelec.net

Windows CI Build Node

Install Windows Server Core 2019 or Windows 10 (eval licence or full).

https://uupdump.ml/ Can be used to download latest versions.

 

Hyper V Server 2019 not working as of Feb 2019: https://github.com/MicrosoftDocs/Virtualization-Documentation/issues/945

https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-server-2019

https://medium.com/@RealNetwork/direct-download-links-windows-server-2019-datacenter-standard-essentials-hyper-v-server-2019-5f53357eeeac

https://software-download.microsoft.com/download/pr/17763.1.180914-1434.rs5_release_SERVERHYPERCORE_OEM_x64FRE_en-us.iso

 

 Install on your hardware/vm of choice.

 

System Setup

Once it's installed you'll see a console on the vm screen. Firstly you'll be asked to set a password, do so with something memorable.

There should be a blue menu visible on the console.

Remote Desktop can be enabled with menu '7'

Press '8' to show Network Settings to see the current IP address. We'll need that and the "Computer Name" displayed on the home screen below.


Hyperv server 2019 currently has a bug with RDP, not needed on server core: 

https://social.technet.microsoft.com/Forums/en-US/c9d82adb-5dd5-4e43-878c-2e98e00c2943/hyperv-server-2019-rdp-not-working?forum=winserverhyperv

Enable-WindowsOptionalFeature -Online -FeatureName Remote-Desktop-Services 

# If that fails, reboot. If still no good, try this: 

wget -UseBasicParsing https://github.com/stascorp/rdpwrap/releases/download/v1.6.2/RDPWInst-v1.6.2.msi -Outfile RDPWInst-v1.6.2.msi
msiexec.exe /i RDPWInst-v1.6.2.msi   

The hardware drivers in the VM can now be changed to VirtIO interfaces for best performance.

Remote desktop on your local machine can now be used to connected to the IP address shown with the username "Administrator" and the password set earlier.  This allows copy/paste to work. 

 

oVirt / KVM

If you're using oVirt or other KVM based hypervisor for you host, you'll likely want the console viewer application installed on your local machine: https://virt-manager.org/download/sources/virt-viewer/virt-viewer-x64-6.0.msi

Set up the vm with the disk connected to IDE initially, we can switch it over to faster VirtIO once drivers are installed.

The network adapter will need to be set to "Dual mode rtl8139, VirtIO"

Once installed and initial setup completed, ensure the guest tools are installed:

Import-Module BitsTransfer
$url = "https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe"
$dest = "spice-guest-tools-latest.exe"
Start-BitsTransfer -Source $url -Destination $dest
.\spice-guest-tools-latest.exe 

 The hardware drivers in the VM can now be changed to VirtIO interfaces for best performance.

 

Install FOD

There are a bunch of extra features available, installing this package from ISO will provide explorer and a bunch of other tools, make more apps run.

https://medium.com/@RealNetwork/direct-download-links-windows-server-2019-datacenter-standard-essentials-hyper-v-server-2019-5f53357eeeac

wget https://software-download.microsoft.com/download/pr/17763.1.180914-1434.rs5_release_amd64fre_SERVER-FOD-PACKAGES_OEM_amd64fre_MULTI.iso -Outfile FOD-PACKAGES.iso
Mount-DiskImage -ImagePath $pwd\FOD-PACKAGES.iso
# E drive below may need to be changed
DISM /Online /Add-Capability /CapabilityName:"ServerCore.AppCompatibility~~~~0.0.1.0" /Source:E: /LimitAccess
# Will need to reboot, then can also do this if desired
Mount-DiskImage -ImagePath FOD-PACKAGES.iso
Dism /online /add-package:E:"Microsoft-Windows-InternetExplorer-Optional-Package~31bf3856ad364e35~amd64~~.cab"


 

Install choco, tools, git

Start with choco, then use it to install useful tools and git for windows

Set-ExecutionPolicy AllSigned
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
refreshenv

choco install git -y -params '"/GitOnlyOnPath"'
#choco install -y multicommander  # no longer needed with FOD above
choco install -y notepad2

# Use notepad2 as replacement for notebad - works with gitlab runner config files
choco install -y notepadreplacer 
# will open wizard, when asked for "Text Editor" path, provide:
# C:\Program Files\Notepad2\Notepad2.exe

refreshenv​   


 

WSL

linux subsystem, including bash:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
# Reboots
Import-Module BitsTransfer
$url = "https://aka.ms/wsl-ubuntu-1804"
$dest = "c:\Ubuntu.zip"
Start-BitsTransfer -Source $url -Destination $dest

Expand-Archive c:\Ubuntu.zip C:\Ubuntu
# Add to path
$userenv = [System.Environment]::GetEnvironmentVariable("Path", "User")
[System.Environment]::SetEnvironmentVariable("PATH", $userenv + "C:\Ubuntu", "User")
# Initialise
c:\Ubuntu\ubuntu1804.exe

# Enable WSL2
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
wsl -l -v
wsl --set-default-version 2


 

Desktop

The Hyper-V / server core server desktop is quite sparse, it is easier to use with Cairo Desktop installed: https://cairoshell.com/

wget -UseBasicParsing "https://github.com/cairoshell/cairoshell/releases/download/v0.3.6693/CairoSetup_64bit.exe" -Outfile CairoSetup_64bit.exe
.\CairoSetup_64bit.exe  


During install, when selecting components, you will want to enable "Advanced users only: Replace Explorer"

Once Cairo Desktop is installed, the service console wont be automatically openned at login. It can be accessed by openning a cmd prompt and running

sconfig 


Installing Docker

Until https://gitlab.com/gitlab-org/gitlab-runner/issues/4295 we need to go via a tcp socket

 

Enable-WindowsOptionalFeature -online -FeatureName Microsoft-Hyper-V-Management-Powershell -all
$ip = "192.168.50.1"
New-VMSwitch -Name "Internal vSwitch" -SwitchType "Internal" -Notes "Internal IP used for docker"
New-NetIPAddress -IPAddress ${ip} -PrefixLength 24 -InterfaceIndex (Get-NetAdapter -Name "*Internal vSwitch*").InterfaceIndex

 

Follow: https://www.kauffmann.nl/2019/03/04/how-to-install-docker-on-windows-10-without-hyper-v/

# Install Windows feature containers
$restartNeeded = $false
if (!(Get-WindowsOptionalFeature -FeatureName containers -Online).State -eq 'Enabled') {    
    Enable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform
    Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
    Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V –All
    $restartNeeded = (Enable-WindowsOptionalFeature -FeatureName containers -Online –All).RestartNeeded
}

if (Get-Service docker -ErrorAction SilentlyContinue)
{
    Stop-Service docker
}

# Download the zip file.
$json = Invoke-WebRequest -UseBasicparsing https://download.docker.com/components/engine/windows-server/index.json | ConvertFrom-Json
$version = $version = $json.channels.'18.09'.version
$url = $json.versions.$version.url
$zipfile = Join-Path "$env:USERPROFILE\Downloads\" $json.versions.$version.url.Split('/')[-1]
Import-Module BitsTransfer
Start-BitsTransfer -Source $url -Destination $zipfile

# Extract the archive.
Expand-Archive $zipfile -DestinationPath $Env:ProgramFiles -Force

# Modify PATH to persist across sessions.
$newPath = [Environment]::GetEnvironmentVariable("PATH",[EnvironmentVariableTarget]::Machine) + ";$env:ProgramFiles\docker"
$splittedPath = $newPath -split ';'
$cleanedPath = $splittedPath | Sort-Object -Unique
$newPath = $cleanedPath -join ';'
[Environment]::SetEnvironmentVariable("PATH", $newPath, [EnvironmentVariableTarget]::Machine)
$env:path = $newPath

# Register the Docker daemon as a service.
if (!(Get-Service docker -ErrorAction SilentlyContinue)) {
  # Add new group with access
  New-LocalGroup -Name "docker"
  Add-LocalGroupMember -Group "docker" -Member "SYSTEM"
  #dockerd --exec-opt isolation=process --group docker --register-service

  # Until https://gitlab.com/gitlab-org/gitlab-runner/issues/4295

  $ip=(Get-NetAdapter -Name "*Internal vSwitch*" | Get-NetIPAddress -AddressFamily IPv4).IPAddress
  dockerd --exec-opt isolation=process --group docker -H tcp://${ip}:2375 -H "npipe://" --register-service
  New-NetFirewallRule -DisplayName "Allow inbound TCP port 2375 (Docker)" -Direction inbound -LocalPort 2375 -Protocol TCP -Action Allow

  cmd /c "sc config docker start= delayed-auto"
}

# Start the Docker service.
if ($restartNeeded) {
    Write-Host 'A restart is needed to finish the installation' -ForegroundColor Green
    If ((Read-Host 'Do you want to restart now? [Y/N]') -eq 'Y') {
      Restart-Computer
    }
} else {
    Start-Service docker
}    

 

If you would like your docker stored on a different drive, change this configuration before pulling any images

notepad C:\ProgramData\docker\config\daemon.json

It will ask you to create file, select yes and paste the following content

{
"graph": "D:\\ProgramData\\Docker"
} 

Changing the path as appropriate. Save, exit, then restart the docker service.

 

REF: how to write config file from powershell: https://bcthomas.com/2019/02/getting-started-with-linux-containers-on-windows-server-2019/

 

For reference, my previous docker instructions :

https://docs.docker.com/docker-for-windows

https://docs.microsoft.com/en-us/virtualization/windowscontainers/quick-start/quick-start-windows-server

From the powershell prompt:

# Enable features required (HNS Service)
Enable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform

Enable-WindowsOptionalFeature -Online -FeatureName containers –All
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V –All

# worth a try
Install-WindowsFeature -Name Containers

# Will reboot
Set-Service -Name hns -StartupType Automatic

Uninstall-WindowsFeature Windows-DefenderInstall-Module -Name DockerMsftProvider -Repository PSGallery -Force
# Press Y to continue
Install-Package -Name docker -ProviderName DockerMsftProvider # -Update -Force # -RequiredVersion 18.03 
# Press A to accept installation from DockerDefault source
# This can display an error which can be ignored: "The role, role service, or feature name is not valid: 'containers'. The name was not found."

Restart-Computer -Force         

 

Verify docker

cmd /c "sc config docker start= delayed-auto"
cmd /c "sc config docker depend= hns"
Start-Service docker

docker run -it --rm mcr.microsoft.com/windows/servercore:1809
docker run -it --rm --isolation=hyperv mcr.microsoft.com/windows/servercore:1809
# Will download (big) base image, should eventually dump you in a new shell
dir
# Should look like a different system
exit 

 

Powershell download errors

Powershell has be giving errors below on some sites, ensure it accepts tls

[Net.ServicePointManager]::SecurityProtocol = "tls12, tls11, tls" 

 

Network Certificate

If you're running on a network with custom server certificates (eg. work cert) they can be installed

wget -UseBasicParsing http://internal.localnet/work-root-ca.crt -Outfile work-root-ca.crt
Import-Certificate -FilePath work-root-ca.crt -CertStoreLocation Cert:\LocalMachine\Root


Gitlab CI runner

Install gitlab runner and configure one of the following two setup types:

Gitlab Docker runner

# Beta runner with native docker support: https://gitlab.com/andrewleech/gitlab-runner
Import-Module BitsTransfer
#$url = "https://gitlab.com/andrewleech/gitlab-runner/-/jobs/155104654/artifacts/raw/out/binaries/gitlab-runner-windows-amd64.exe"

# beta has windows docker support
$url = "https://s3.amazonaws.com/gitlab-runner-downloads/master/binaries/gitlab-runner-windows-amd64.exe"


# official now has windows docker support
$url = "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-windows-amd64.exe"
Start-BitsTransfer -Source $url -Destination c:\windows\system32\gitlab-runner.exe


   
# Get registration token from https://gitlab.com/<project settings>/runners
$env:REGISTRATION_TOKEN="abc123"
$env:CI_SERVER_URL="https://gitlab.com"

$env:RUNNER_NAME="WindowsDockerRunner1809"

$env:CONFIG_FILE="C:\gitlab\config.toml"
$env:REGISTER_LOCKED="false"
$env:RUNNER_EXECUTOR="docker-windows"
$env:RUNNER_TAG_LIST="docker,windows,1809"
$env:RUNNER_SHELL="powershell"
$env:RUNNER_BUILDS_DIR="c:\gitlab\builds"
#$env:RUNNER_CACHE_DIR="c:\gitlab\cache"

$env:DOCKER_MEMORY="4g"
$env:DOCKER_TLS_VERIFY="false"
#$env:DOCKER_PRIVILEGED="true"
$env:DOCKER_IMAGE="mcr.microsoft.com/windows/servercore:1809"
$env:DOCKER_VOLUMES="\\.\pipe\docker_engine"
#$env:DOCKER_CACHE_DIR="c:\gitlab\cache"
#$env:DOCKER_PULL_POLICY="if-not-present"
#$env:DOCKER_HELPER_IMAGE="registry.gitlab.com/andrewleech/gitlab-runner/gitlab-runner-helper:win-x86_64-1809-latest"

# TODO Switch back to npipe once it works in gitlab (see note above)
$ip=(Get-NetAdapter -Name "*Internal vSwitch*" | Get-NetIPAddress -AddressFamily IPv4).IPAddress
$env:DOCKER_HOST="tcp://${ip}:2375"
$env:RUNNER_ENV="DOCKER_HOST=tcp://${ip}:2375"

cd \
mkdir \gitlab

gitlab-runner register --non-interactive


# You may want to increase the number of concurrent jobs, if so:
notepad c:\gitlab\config.toml
# Then add/edit the following line to the top of the file (withouth the #)
# concurrent = 4
# Save and close

 
# Install the service
gitlab-runner install -user .\LocalSystem --working-directory="C:\gitlab" --config=$env:CONFIG_FILE
cmd /c "sc.exe config gitlab-runner start= delayed-auto"

 
# edit config.toml, add session_server listen address, then open it in firewall:
New-NetFirewallRule -DisplayName "Allow inbound TCP port 8093 (gitlab runner session server)" -Direction inbound -LocalPort 8093 -Protocol TCP -Action Allow

 

Local powershell runner

This is useful for building docker images, etc. I haven't got docker-in-docker working on windows yet (haven't really tried recently)

# Skip the wget if you've already installed the docker beta one above
Import-Module BitsTransfer
$url = "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-windows-amd64.exe"
$dest = "c:\windows\system32\gitlab-runner.exe"
Start-BitsTransfer -Source $url -Destination $dest  
# To register a shared host shell runner for building docker images etc
$REGISTRATION_TOKEN="<token from https://gitlab.com/admin/runners>"
 
gitlab-runner register --non-interactive --url="https://gitlab.com" --registration-token=$REGISTRATION_TOKEN --description="WindowsHostRunner1809" --executor="shell" --shell="powershell" --tag-list="windows,windows_docker_host,powershell,1809"
  
# Install the service, if not already done above
gitlab-runner install -user .\LocalSystem

Start-Service gitlab-runner 

 

Other Features...

Get-WindowsOptionalFeature -online | ft
Enable-WindowsOptionalFeature -Online -FeatureName 

 

Random Shutdowns

Sometimes the free licence can get out of whack and expires, resulting in the vm starts shutting down every hour.
This should be resolved by running

Slmgr.vbs -rearm



Windows Admin Center


$cert = (New-SelfSignedCertificate -DnsName "SleakWin" -CertStoreLocation "cert:\LocalMachine\My" -NotAfter (Get-Date).AddYears(10)).Thumbprint | Out-String
# TODO these params settings don't appear to work
# choco install -y windows-admin-center --params "/Thumbprint:$cert /Port:443"
$args = """/Thumbprint:" + $cert + " /Port:443"""
echo $args
echo $cert
choco install -y windows-admin-center --params "'/Thumbprint:<copy the thumbnail printed above> /Port:443'"
Restart-Computer -Force 

 

The Hyper-V Server can be remotely administered with MS Remote Server Administration Tools (RSAT): https://www.microsoft.com/en-au/download/details.aspx?id=45520

To enable this, the following firewall ports need to be opened on the hyperv server:

Set-NetFirewallRule -DisplayGroup 'Windows Management Instrumentation (WMI)' -Enabled true -PassThru

Netsh advfirewall firewall set rule group="Remote Volume Management" new enable=yes
Netsh advfirewall firewall set rule group="Windows Firewall Remote Management" new enable=yes
Netsh advfirewall firewall set rule group="Remote Services Management" new enable=yes
Netsh advfirewall firewall set rule group="File and Printer Sharing" new enable=yes
Netsh advfirewall firewall set rule group="Remote Scheduled Tasks Management" new enable=yes
Netsh advfirewall firewall set rule group="Performance Logs and Alerts" new enable=yes
Netsh advfirewall firewall set rule group="Remote Event Log Management" new enable=yes​Netsh advfirewall firewall set rule group="File and Printer Sharing" new enable=yes
Netsh advfirewall firewall set rule group="Remote Scheduled Tasks Management" new enable=yes
Netsh advfirewall firewall set rule group="Performance Logs and Alerts" new enable=yes
Netsh advfirewall firewall set rule group="Remote Event Log Management" new enable=yes

Then install RSAT and start server manager.

Add the server by dns / ip address and provide the Administrator user/password in "Manage as..." context menu item.

You will likely get a "Kerberos authentication error" listed against the server, but most functions should work fine.

"Computer Management" in particular works and gives access to Services, which don't exist on the hyper-v server installation natively.

sconfig

 

WSL

linux subsystem, including bash:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux 



Random Shutdowns

Sometimes the free licence can get out of whack and expires, resulting in the vm starts shutting down every hour.
This should be resolved by running

12Slmgr.vbs -rearm 

.appveyor.yml


Pre: Sunbeam Torino EM8000: Reducing the pressure with a dimmer

Next: Windows Docker Gitlab CI Runner

1883
Table of content