Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update MacOS Software #15258

Merged
merged 9 commits into from
Dec 24, 2020
Merged

Update MacOS Software #15258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
908708a
Update vsts agent URL to 2.179.0.
BillyONeal Dec 23, 2020
6544319
Delete no longer used archives sshfs infrastructure.
BillyONeal Dec 23, 2020
5abea87
Add nasm package from https://github.com/microsoft/vcpkg/pull/14545 a…
BillyONeal Dec 23, 2020
bbb386b
Add gperf package from https://github.com/microsoft/vcpkg/pull/14228
BillyONeal Dec 23, 2020
86f12a9
Add meson package from https://github.com/microsoft/vcpkg/pull/12860
BillyONeal Dec 23, 2020
0ecc296
Update virtualization software.
BillyONeal Dec 23, 2020
b5c6185
Fixed a few pitfalls from testing on vcpkgmm-11
BillyONeal Dec 24, 2020
a35d122
Use brew install --cask rather than brew cask install.
BillyONeal Dec 24, 2020
85b8ba2
Change gfortran cask => gfortran non-cask.
BillyONeal Dec 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions scripts/azure-pipelines/osx/Get-InternalBaseBox.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,15 @@ sshfs "fileshare@${FileshareMachine}:/Users/fileshare/share" $mountPoint
if ($LASTEXITCODE -eq 1) {
Write-Error 'sshfs returned 1.
This means that the osxfuse kernel extension was not allowed to load.
Please open System Preferences > Security & Privacy > General,
You may need to force un/reinstall osxfuse and/or sshfs with
brew uninstall osxfuse
brew uninstall sshfs
brew install osxfuse
brew install sshfs
Then, rerun this script.

If you''ve already done that, Please open
System Preferences > Security & Privacy > General,
and allow the kernel extension to load.
Then, rerun this script.

Expand All @@ -58,4 +66,3 @@ if (-not [String]::IsNullOrEmpty($BoxVersion)) {
}

vagrant box add "$mountPoint/vcpkg-boxes/macos-ci.json" @versionArgs

2 changes: 1 addition & 1 deletion scripts/azure-pipelines/osx/Install-Prerequisites.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ $Installables.Brew | ForEach-Object {
brew install $installable.Name
} else {
switch ($installable.Kind) {
'cask' { brew cask install $installable.Name }
'cask' { brew install --cask $installable.Name }
default {
Write-Error "Invalid kind: $_. Expected either empty, or 'cask'."
}
Expand Down
37 changes: 8 additions & 29 deletions scripts/azure-pipelines/osx/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,17 @@ Next, install prerequisites and grab the current base box with:
```sh
$ cd vcpkg/scripts/azure-pipelines/osx
$ ./Install-Prerequisites.ps1 -Force
$ ./Get-InternalBaseBox.ps1 -FileshareMachine vcpkgmm-01.guest.corp.microsoft.com
$ ./Get-InternalBaseBox.ps1 -FileshareMachine vcpkgmm-01.guest.corp.microsoft.com -BoxVersion 2020-09-28
```

Getting the base box will fail due to missing kernel modules for sshfs and VirtualBox. Log in to the
machine, open System Preferences > Security & Privacy > General, and allow the kernel extensions for
VirtualBox and sshfs to load. Then, again:
... where -BoxVersion is the version you want to use.

Getting the base box will fail due to missing kernel modules for osxfuse, sshfs, and/or VirtualBox.
Log in to the machine, open System Preferences > Security & Privacy > General, and allow the kernel
extensions for VirtualBox and sshfs to load. Then, again:

```sh
$ ./Get-InternalBaseBox.ps1 -FileshareMachine vcpkgmm-01.guest.corp.microsoft.com
$ ./Get-InternalBaseBox.ps1 -FileshareMachine vcpkgmm-01.guest.corp.microsoft.com -BoxVersion 2020-09-28
```

Replace `XX` with the number of
Expand All @@ -68,8 +70,7 @@ for the physical machine; i.e., vcpkgmm-04 would use 04.
$ ./Setup-VagrantMachines.ps1 \
-MachineId XX \
-DevopsPat '<get this from azure devops; it needs agent pool read and manage access>' \
-Date <this is the date of the pool; 2020-09-28 at time of writing> \
-ArchivesMachine 'vcpkgmm-01.guest.corp.microsoft.com'
-Date <this is the date of the pool; 2020-09-28 at time of writing>
$ cd ~/vagrant/vcpkg-eg-mac
$ vagrant up
```
Expand Down Expand Up @@ -137,31 +138,9 @@ $ vagrant ssh -c 'sudo installer -pkg "/Volumes/setup-installer/Command Line Too
$ vagrant ssh -c 'hdiutil detach /Volumes/setup-installer'
$ vagrant ssh -c 'rm clt.dmg'
$ vagrant ssh -c '/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"'
$ vagrant ssh -c 'ssh-keygen -t rsa -b 4096 -c "Fileshare Key: <date>" -N "" -f "~/.ssh/id_rsa"'
$ vagrant ssh -c 'cat ~/.ssh/id_rsa.pub'
$ vagrant ssh -c 'brew cask install osxfuse && brew install sshfs'
$ vagrant reload
```

After this, you should add the printed ssh public key to the archives share's `.ssh/authorized_keys` file.
Then, we also now need to make sure that osxfuse is set up correctly;
macOS requires the user to accept that this signed binary is "okay to be loaded" by the kernel.
We can get `sshfs` to try to start the `osxfuse` kernel module by attempting to start it:

```sh
$ vagrant ssh -c 'mkdir testmnt && sshfs <fileshare ssh>:/Users/fileshare/share testmnt'
```

Then, you'll need to open the VM in VirtualBox, go to System Preferences,
go to Security & Privacy, General, unlock the settings,
and allow system extensions from the osxfuse developer to run.

Then, retry the above, and see if it works:

```sh
$ vagrant ssh -c 'sshfs <fileshare ssh>:/Users/fileshare/share testmnt'
```

if that works, you can now package the box:

```sh
Expand Down
25 changes: 0 additions & 25 deletions scripts/azure-pipelines/osx/Setup-VagrantMachines.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,16 +28,6 @@ The agent pool to add the machine to. If -Date is passed, uses "PrOsx-$Date" as
.PARAMETER DevopsUrl
The URL of the ADO instance; defaults to vcpkg's, which is https://dev.azure.com/vcpkg.

.PARAMETER ArchivesMachine
The machine where the archives are located; a URN.

.PARAMETER ArchivesPath
The path to where the archives are located on the machine. If -Date is passed,
uses "/Users/${ArchivesUsername}/share/archives/${Date}".

.PARAMETER ArchivesUsername
The user to log in to on the archives machine. Defaults to 'fileshare'.

.PARAMETER BaseName
The base name for the vagrant VM; the machine name is $BaseName-$MachineId.
Defaults to 'vcpkg-eg-mac'.
Expand Down Expand Up @@ -78,15 +68,6 @@ Param(
[Parameter(Mandatory=$False)]
[String]$DevopsUrl = 'https://dev.azure.com/vcpkg',

[Parameter(Mandatory=$True)]
[String]$ArchivesMachine,

[Parameter(Mandatory=$True, ParameterSetName='DefineVersionAndAgentPool')]
[String]$ArchivesPath,

[Parameter(Mandatory=$False)]
[String]$ArchivesUsername = 'archivesshare',

[Parameter()]
[String]$BaseName = 'vcpkg-eg-mac',

Expand All @@ -109,7 +90,6 @@ if (-not $IsMacOS) {
if (-not [String]::IsNullOrEmpty($Date)) {
$BoxVersion = $Date
$AgentPool = "PrOsx-$Date"
$ArchivesPath = "/Users/${ArchivesUsername}/share/archives/${Date}"
}

if (Test-Path '~/vagrant/vcpkg-eg-mac') {
Expand Down Expand Up @@ -139,11 +119,6 @@ $configuration = @{
box_name = $BoxName;
box_version = $BoxVersion;
disk_size = $DiskSize;
archives = @{
username = $ArchivesUsername;
urn = $ArchivesMachine;
path = $ArchivesPath;
};
}
ConvertTo-Json -InputObject $configuration -Depth 5 `
| Set-Content -Path '~/vagrant/vcpkg-eg-mac/vagrant-configuration.json'
27 changes: 7 additions & 20 deletions scripts/azure-pipelines/osx/configuration/Vagrantfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,22 @@ brew_formulas = [
'autoconf',
'automake',
'bison',
'gfortran',
'gperf',
'libtool',
'meson',
'mono',
'nasm',
'pkg-config',
'yasm' ]

brew_cask_formulas = [
'powershell',
'gfortran' ]
'powershell' ]

azure_agent_url = 'https://vstsagentpackage.azureedge.net/agent/2.171.1/vsts-agent-osx-x64-2.171.1.tar.gz'
azure_agent_url = 'https://vstsagentpackage.azureedge.net/agent/2.179.0/vsts-agent-osx-x64-2.179.0.tar.gz'
devops_url = configuration['devops_url']
agent_pool = configuration['agent_pool']
pat = configuration['pat']
archives = configuration['archives']
archives_username = archives['username']
archives_urn = archives['urn']
archives_path = archives['path']

Vagrant.configure('2') do |config|
# give them extra time to boot up
Expand Down Expand Up @@ -92,19 +91,7 @@ Vagrant.configure('2') do |config|
config.vm.provision 'shell',
run: 'once',
name: 'Install brew applications',
inline: "brew install #{brew_formulas.join(' ')} && brew cask install #{brew_cask_formulas.join(' ')}",
privileged: false

config.vm.provision 'shell',
run: 'once',
name: 'Create archives mountpoint',
inline: 'mkdir ~/Data/archives',
privileged: false

config.vm.provision "shell",
run: 'once',
name: 'Mount archives directory',
inline: "sshfs #{archives_username}@#{archives_urn}:#{archives_path} ~/Data/archives",
inline: "brew install #{brew_formulas.join(' ')} && brew install --cask #{brew_cask_formulas.join(' ')}",
privileged: false

config.vm.provision 'shell',
Expand Down
12 changes: 6 additions & 6 deletions scripts/azure-pipelines/osx/configuration/installables.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@
{
"Name": "VirtualBox",
"TestCommand": "VBoxManage",
"DmgUrl": "https://download.virtualbox.org/virtualbox/6.1.10/VirtualBox-6.1.10-138449-OSX.dmg",
"Sha256": "EF0CA4924922514B6AD71469998821F2CF7C596B4B8B59736C3699759E0F1DF8",
"DmgUrl": "https://download.virtualbox.org/virtualbox/6.1.16/VirtualBox-6.1.16-140961-OSX.dmg",
"Sha256": "D7DF0F05D9A9E7CBA50EA01DA264AC20948B1C9C0E0CCCD2D628085C9F434D45",
"InstallerPath": "VirtualBox.pkg"
},
{
"Name": "vagrant",
"TestCommand": "vagrant",
"DmgUrl": "https://releases.hashicorp.com/vagrant/2.2.9/vagrant_2.2.9_x86_64.dmg",
"Sha256": "529CDE2A78E6DF38EC906B65C70B36A087E2601EAB42E25856E35B20CCB027C0",
"DmgUrl": "https://releases.hashicorp.com/vagrant/2.2.14/vagrant_2.2.14_x86_64.dmg",
"Sha256": "76B849B26E6D6187A7829212B05545D3B424E05F1BCD0F7163DA1E5117084FA6",
"InstallerPath": "vagrant.pkg"
}
],
Expand All @@ -30,8 +30,8 @@
{
"Name": "Extension Pack",
"FullName": "Oracle VM VirtualBox Extension Pack",
"Url": "https://download.virtualbox.org/virtualbox/6.1.10/Oracle_VM_VirtualBox_Extension_Pack-6.1.10.vbox-extpack",
"Sha256": "03067F27F4DA07C5D0FDAFC56D27E3EA23A60682B333B2A1010FB74EF9A40C28"
"Url": "https://download.virtualbox.org/virtualbox/6.1.16/Oracle_VM_VirtualBox_Extension_Pack-6.1.16.vbox-extpack",
"Sha256": "9802482B77B95A954CB5111793DA10D009009A4E9A9C4EAA4BD1AE5DAFE9DB46"
}
]
}
16 changes: 1 addition & 15 deletions scripts/azure-pipelines/osx/configuration/vagrant-configuration.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@
"pat",
"base_name",
"disk_size",
"machine_identifiers",
"archives"
"machine_identifiers"
],

"properties": {
Expand All @@ -32,19 +31,6 @@
},
"disk_size": {
"type": "integer"
},
"archives": {
"type": "object",
"required": [
"username",
"urn",
"path"
],
"properties": {
"username": { "type": "string" },
"urn": { "type": "string" },
"path": { "type": "string" }
}
}
}
}