andrew@alelec.net
Windows CI Build Node
Install Windows Server Core 2019 or Windows 10 (eval licence or full).
https://uupdump.ml/ Can be used to download latest versions.
Hyper V Server 2019 not working as of Feb 2019: https://github.com/MicrosoftDocs/Virtualization-Documentation/issues/945
https://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-server-2019
Install on your hardware/vm of choice.
System Setup
Once it's installed you'll see a console on the vm screen. Firstly you'll be asked to set a password, do so with something memorable.
There should be a blue menu visible on the console.
Remote Desktop can be enabled with menu '7'
Press '8' to show Network Settings to see the current IP address. We'll need that and the "Computer Name" displayed on the home screen below.
Hyperv server 2019 currently has a bug with RDP, not needed on server core:
Enable-WindowsOptionalFeature -Online -FeatureName Remote-Desktop-Services # If that fails, reboot. If still no good, try this: wget -UseBasicParsing https://github.com/stascorp/rdpwrap/releases/download/v1.6.2/RDPWInst-v1.6.2.msi -Outfile RDPWInst-v1.6.2.msi msiexec.exe /i RDPWInst-v1.6.2.msi
The hardware drivers in the VM can now be changed to VirtIO interfaces for best performance.
Remote desktop on your local machine can now be used to connected to the IP address shown with the username "Administrator" and the password set earlier. This allows copy/paste to work.
oVirt / KVM
If you're using oVirt or other KVM based hypervisor for you host, you'll likely want the console viewer application installed on your local machine: https://virt-manager.org/download/sources/virt-viewer/virt-viewer-x64-6.0.msi
Set up the vm with the disk connected to IDE initially, we can switch it over to faster VirtIO once drivers are installed.
The network adapter will need to be set to "Dual mode rtl8139, VirtIO"
Once installed and initial setup completed, ensure the guest tools are installed:
Import-Module BitsTransfer $url = "https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe" $dest = "spice-guest-tools-latest.exe" Start-BitsTransfer -Source $url -Destination $dest .\spice-guest-tools-latest.exe
The hardware drivers in the VM can now be changed to VirtIO interfaces for best performance.
Install FOD
There are a bunch of extra features available, installing this package from ISO will provide explorer and a bunch of other tools, make more apps run.
wget https://software-download.microsoft.com/download/pr/17763.1.180914-1434.rs5_release_amd64fre_SERVER-FOD-PACKAGES_OEM_amd64fre_MULTI.iso -Outfile FOD-PACKAGES.iso Mount-DiskImage -ImagePath $pwd\FOD-PACKAGES.iso # E drive below may need to be changed DISM /Online /Add-Capability /CapabilityName:"ServerCore.AppCompatibility~~~~0.0.1.0" /Source:E: /LimitAccess # Will need to reboot, then can also do this if desired Mount-DiskImage -ImagePath FOD-PACKAGES.iso Dism /online /add-package:E:"Microsoft-Windows-InternetExplorer-Optional-Package~31bf3856ad364e35~amd64~~.cab"
Install choco, tools, git
Start with choco, then use it to install useful tools and git for windows
Set-ExecutionPolicy AllSigned
iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
refreshenv
choco install git -y -params '"/GitOnlyOnPath"'
#choco install -y multicommander # no longer needed with FOD above
choco install -y notepad2
# Use notepad2 as replacement for notebad - works with gitlab runner config files
choco install -y notepadreplacer
# will open wizard, when asked for "Text Editor" path, provide:
# C:\Program Files\Notepad2\Notepad2.exe
refreshenv
WSL
linux subsystem, including bash:
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
# Reboots
Import-Module BitsTransfer
$url = "https://aka.ms/wsl-ubuntu-1804"
$dest = "c:\Ubuntu.zip"
Start-BitsTransfer -Source $url -Destination $dest
Expand-Archive c:\Ubuntu.zip C:\Ubuntu
# Add to path
$userenv = [System.Environment]::GetEnvironmentVariable("Path", "User")
[System.Environment]::SetEnvironmentVariable("PATH", $userenv + "C:\Ubuntu", "User")
# Initialise
c:\Ubuntu\ubuntu1804.exe
# Enable WSL2
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
wsl -l -v
wsl --set-default-version 2
Desktop
The Hyper-V / server core server desktop is quite sparse, it is easier to use with Cairo Desktop installed: https://cairoshell.com/
wget -UseBasicParsing "https://github.com/cairoshell/cairoshell/releases/download/v0.3.6693/CairoSetup_64bit.exe" -Outfile CairoSetup_64bit.exe .\CairoSetup_64bit.exe
During install, when selecting components, you will want to enable "Advanced users only: Replace Explorer"
Once Cairo Desktop is installed, the service console wont be automatically openned at login. It can be accessed by openning a cmd prompt and running
sconfig
Installing Docker
Until https://gitlab.com/gitlab-org/gitlab-runner/issues/4295 we need to go via a tcp socket
Enable-WindowsOptionalFeature -online -FeatureName Microsoft-Hyper-V-Management-Powershell -all
$ip = "192.168.50.1"
New-VMSwitch -Name "Internal vSwitch" -SwitchType "Internal" -Notes "Internal IP used for docker"
New-NetIPAddress -IPAddress ${ip} -PrefixLength 24 -InterfaceIndex (Get-NetAdapter -Name "*Internal vSwitch*").InterfaceIndex
Follow: https://www.kauffmann.nl/2019/03/04/how-to-install-docker-on-windows-10-without-hyper-v/
# Install Windows feature containers
$restartNeeded = $false
if (!(Get-WindowsOptionalFeature -FeatureName containers -Online).State -eq 'Enabled') {
Enable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V –All
$restartNeeded = (Enable-WindowsOptionalFeature -FeatureName containers -Online –All).RestartNeeded
}
if (Get-Service docker -ErrorAction SilentlyContinue)
{
Stop-Service docker
}
# Download the zip file.
$json = Invoke-WebRequest -UseBasicparsing https://download.docker.com/components/engine/windows-server/index.json | ConvertFrom-Json
$version = $version = $json.channels.'18.09'.version
$url = $json.versions.$version.url
$zipfile = Join-Path "$env:USERPROFILE\Downloads\" $json.versions.$version.url.Split('/')[-1]
Import-Module BitsTransfer
Start-BitsTransfer -Source $url -Destination $zipfile
# Extract the archive.
Expand-Archive $zipfile -DestinationPath $Env:ProgramFiles -Force
# Modify PATH to persist across sessions.
$newPath = [Environment]::GetEnvironmentVariable("PATH",[EnvironmentVariableTarget]::Machine) + ";$env:ProgramFiles\docker"
$splittedPath = $newPath -split ';'
$cleanedPath = $splittedPath | Sort-Object -Unique
$newPath = $cleanedPath -join ';'
[Environment]::SetEnvironmentVariable("PATH", $newPath, [EnvironmentVariableTarget]::Machine)
$env:path = $newPath
# Register the Docker daemon as a service.
if (!(Get-Service docker -ErrorAction SilentlyContinue)) {
# Add new group with access
New-LocalGroup -Name "docker"
Add-LocalGroupMember -Group "docker" -Member "SYSTEM"
#dockerd --exec-opt isolation=process --group docker --register-service
# Until https://gitlab.com/gitlab-org/gitlab-runner/issues/4295
$ip=(Get-NetAdapter -Name "*Internal vSwitch*" | Get-NetIPAddress -AddressFamily IPv4).IPAddress
dockerd --exec-opt isolation=process --group docker -H tcp://${ip}:2375 -H "npipe://" --register-service
New-NetFirewallRule -DisplayName "Allow inbound TCP port 2375 (Docker)" -Direction inbound -LocalPort 2375 -Protocol TCP -Action Allow
cmd /c "sc config docker start= delayed-auto"
}
# Start the Docker service.
if ($restartNeeded) {
Write-Host 'A restart is needed to finish the installation' -ForegroundColor Green
If ((Read-Host 'Do you want to restart now? [Y/N]') -eq 'Y') {
Restart-Computer
}
} else {
Start-Service docker
}
If you would like your docker stored on a different drive, change this configuration before pulling any images
notepad C:\ProgramData\docker\config\daemon.json
It will ask you to create file, select yes and paste the following content
{
"graph": "D:\\ProgramData\\Docker"
} Changing the path as appropriate. Save, exit, then restart the docker service.
REF: how to write config file from powershell: https://bcthomas.com/2019/02/getting-started-with-linux-containers-on-windows-server-2019/
For reference, my previous docker instructions :
https://docs.docker.com/docker-for-windows
From the powershell prompt:
# Enable features required (HNS Service) Enable-WindowsOptionalFeature -Online -FeatureName HypervisorPlatform Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform Enable-WindowsOptionalFeature -Online -FeatureName containers –All Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V –All # worth a try Install-WindowsFeature -Name Containers # Will reboot Set-Service -Name hns -StartupType Automatic Uninstall-WindowsFeature Windows-DefenderInstall-Module -Name DockerMsftProvider -Repository PSGallery -Force # Press Y to continue Install-Package -Name docker -ProviderName DockerMsftProvider # -Update -Force # -RequiredVersion 18.03 # Press A to accept installation from DockerDefault source # This can display an error which can be ignored: "The role, role service, or feature name is not valid: 'containers'. The name was not found." Restart-Computer -Force
Verify docker
cmd /c "sc config docker start= delayed-auto" cmd /c "sc config docker depend= hns" Start-Service docker docker run -it --rm mcr.microsoft.com/windows/servercore:1809 docker run -it --rm --isolation=hyperv mcr.microsoft.com/windows/servercore:1809 # Will download (big) base image, should eventually dump you in a new shell dir # Should look like a different system exit
Powershell download errors
Powershell has be giving errors below on some sites, ensure it accepts tls
[Net.ServicePointManager]::SecurityProtocol = "tls12, tls11, tls"
Network Certificate
If you're running on a network with custom server certificates (eg. work cert) they can be installed
wget -UseBasicParsing http://internal.localnet/work-root-ca.crt -Outfile work-root-ca.crt Import-Certificate -FilePath work-root-ca.crt -CertStoreLocation Cert:\LocalMachine\Root
Gitlab CI runner
Install gitlab runner and configure one of the following two setup types:
Gitlab Docker runner
# Beta runner with native docker support: https://gitlab.com/andrewleech/gitlab-runner Import-Module BitsTransfer #$url = "https://gitlab.com/andrewleech/gitlab-runner/-/jobs/155104654/artifacts/raw/out/binaries/gitlab-runner-windows-amd64.exe" # beta has windows docker support $url = "https://s3.amazonaws.com/gitlab-runner-downloads/master/binaries/gitlab-runner-windows-amd64.exe" # official now has windows docker support $url = "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-windows-amd64.exe" Start-BitsTransfer -Source $url -Destination c:\windows\system32\gitlab-runner.exe
# Get registration token from https://gitlab.com/<project settings>/runners
$env:REGISTRATION_TOKEN="abc123"
$env:CI_SERVER_URL="https://gitlab.com"
$env:RUNNER_NAME="WindowsDockerRunner1809"
$env:CONFIG_FILE="C:\gitlab\config.toml"
$env:REGISTER_LOCKED="false"
$env:RUNNER_EXECUTOR="docker-windows"
$env:RUNNER_TAG_LIST="docker,windows,1809"
$env:RUNNER_SHELL="powershell"
$env:RUNNER_BUILDS_DIR="c:\gitlab\builds"
#$env:RUNNER_CACHE_DIR="c:\gitlab\cache"
$env:DOCKER_MEMORY="4g"
$env:DOCKER_TLS_VERIFY="false"
#$env:DOCKER_PRIVILEGED="true"
$env:DOCKER_IMAGE="mcr.microsoft.com/windows/servercore:1809"
$env:DOCKER_VOLUMES="\\.\pipe\docker_engine"
#$env:DOCKER_CACHE_DIR="c:\gitlab\cache"
#$env:DOCKER_PULL_POLICY="if-not-present"
#$env:DOCKER_HELPER_IMAGE="registry.gitlab.com/andrewleech/gitlab-runner/gitlab-runner-helper:win-x86_64-1809-latest"
# TODO Switch back to npipe once it works in gitlab (see note above)
$ip=(Get-NetAdapter -Name "*Internal vSwitch*" | Get-NetIPAddress -AddressFamily IPv4).IPAddress
$env:DOCKER_HOST="tcp://${ip}:2375"
$env:RUNNER_ENV="DOCKER_HOST=tcp://${ip}:2375"
cd \
mkdir \gitlab
gitlab-runner register --non-interactive
# You may want to increase the number of concurrent jobs, if so:
notepad c:\gitlab\config.toml
# Then add/edit the following line to the top of the file (withouth the #)
# concurrent = 4
# Save and close
# Install the service
gitlab-runner install -user .\LocalSystem --working-directory="C:\gitlab" --config=$env:CONFIG_FILE
cmd /c "sc.exe config gitlab-runner start= delayed-auto"
# edit config.toml, add session_server listen address, then open it in firewall:
New-NetFirewallRule -DisplayName "Allow inbound TCP port 8093 (gitlab runner session server)" -Direction inbound -LocalPort 8093 -Protocol TCP -Action Allow
Local powershell runner
This is useful for building docker images, etc. I haven't got docker-in-docker working on windows yet (haven't really tried recently)
# Skip the wget if you've already installed the docker beta one above Import-Module BitsTransfer $url = "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-windows-amd64.exe" $dest = "c:\windows\system32\gitlab-runner.exe" Start-BitsTransfer -Source $url -Destination $dest # To register a shared host shell runner for building docker images etc $REGISTRATION_TOKEN="<token from https://gitlab.com/admin/runners>" gitlab-runner register --non-interactive --url="https://gitlab.com" --registration-token=$REGISTRATION_TOKEN --description="WindowsHostRunner1809" --executor="shell" --shell="powershell" --tag-list="windows,windows_docker_host,powershell,1809" # Install the service, if not already done above gitlab-runner install -user .\LocalSystem Start-Service gitlab-runner
Other Features...
Get-WindowsOptionalFeature -online | ft Enable-WindowsOptionalFeature -Online -FeatureName
Random Shutdowns
Sometimes the free licence can get out of whack and expires, resulting in the vm starts shutting down every hour.
This should be resolved by running
Slmgr.vbs -rearm
Windows Admin Center
$cert = (New-SelfSignedCertificate -DnsName "SleakWin" -CertStoreLocation "cert:\LocalMachine\My" -NotAfter (Get-Date).AddYears(10)).Thumbprint | Out-String # TODO these params settings don't appear to work # choco install -y windows-admin-center --params "/Thumbprint:$cert /Port:443" $args = """/Thumbprint:" + $cert + " /Port:443""" echo $args echo $cert choco install -y windows-admin-center --params "'/Thumbprint:<copy the thumbnail printed above> /Port:443'" Restart-Computer -Force
The Hyper-V Server can be remotely administered with MS Remote Server Administration Tools (RSAT): https://www.microsoft.com/en-au/download/details.aspx?id=45520
To enable this, the following firewall ports need to be opened on the hyperv server:
Set-NetFirewallRule -DisplayGroup 'Windows Management Instrumentation (WMI)' -Enabled true -PassThru Netsh advfirewall firewall set rule group="Remote Volume Management" new enable=yes Netsh advfirewall firewall set rule group="Windows Firewall Remote Management" new enable=yes Netsh advfirewall firewall set rule group="Remote Services Management" new enable=yes Netsh advfirewall firewall set rule group="File and Printer Sharing" new enable=yes Netsh advfirewall firewall set rule group="Remote Scheduled Tasks Management" new enable=yes Netsh advfirewall firewall set rule group="Performance Logs and Alerts" new enable=yes Netsh advfirewall firewall set rule group="Remote Event Log Management" new enable=yesNetsh advfirewall firewall set rule group="File and Printer Sharing" new enable=yes Netsh advfirewall firewall set rule group="Remote Scheduled Tasks Management" new enable=yes Netsh advfirewall firewall set rule group="Performance Logs and Alerts" new enable=yes Netsh advfirewall firewall set rule group="Remote Event Log Management" new enable=yes
Then install RSAT and start server manager.
Add the server by dns / ip address and provide the Administrator user/password in "Manage as..." context menu item.
You will likely get a "Kerberos authentication error" listed against the server, but most functions should work fine.
"Computer Management" in particular works and gives access to Services, which don't exist on the hyper-v server installation natively.
sconfig
WSL
linux subsystem, including bash:
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
Random Shutdowns
Sometimes the free licence can get out of whack and expires, resulting in the vm starts shutting down every hour.
This should be resolved by running
12Slmgr.vbs -rearm