Skip to content

Commit

Permalink
multistep macos setup (digital-asset#5768)
Browse files Browse the repository at this point in the history 
multistep macos setup

This updates the macOS node setup instructions to avoid repeating
identical work and network traffic across all machines through
initialization by building a "daily" image with all the tools and code
we need.

CHANGELOG_BEGIN
CHANGELOG_END

* Fix 3-running-box to remount nix partition

* updated scripts to use multi-step process

* add copyright notices

Co-authored-by: nycnewman <edward@digitalasset.com>
  • Loading branch information
@garyverhaegen-da @nycnewman
garyverhaegen-da and nycnewman authored Aug 18, 2020
1 parent 4f2bd15 commit 5b2319e
Show file tree
Hide file tree
Showing 16 changed files with 285 additions and 178 deletions.
31 changes: 17 additions & 14 deletions infra/macos/1-create-box/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ This will open a graphical dialog to install the XCode Command-Line Tools.

## Vagrant

To easily manage the guest VM from scripts, we will be using [Vagrant](https://releases.hashicorp.com/vagrant/2.2.7/vagrant_2.2.7_x86_64.dmg).
To easily manage the guest VM from scripts, we will be using [Vagrant](https://releases.hashicorp.com/vagrant/2.2.7/vagrant_2.2.9_x86_64.dmg).

Alternatively, installing with Homebrew:

Expand All @@ -65,20 +65,20 @@ These instructions have been tested with 2.2.9.

## Hypervisor Selection

We provide two options for the hypervisor: VirtualBox (open-source) and VMWare Fusion (commercial license). This
We provide two options for the hypervisor: VirtualBox (open-source) and VMWare Fusion (commercial license). This
resulted from our testing on large Mac Mini nodes (6 Core, 64Gb, 500Gb drives) where we found VirtualBox to be
less stable when attempting to use > 6 virtual cores or 32Gb or more of memory. The Guest OS would hang on boot, or
experience slow processing or network to the extent that it became unusable.
less stable when attempting to use > 6 virtual cores or 32Gb or more of memory. The Guest OS would hang on boot, or
experience slow processing or network to the extent that it became unusable.

## VMWare Fusion

Purchase a license for
Purchase a license for

* [VMware Fusion Pro 11.5.3](http://www.vmware.com/products/fusion.html)
* [Vagrant VMware Desktop Provider 2.0.3](https://www.vagrantup.com/vmware/)

Download the installer packages for this software. Install VMWare Fusion per vendor instructions and accept
the security settings in Catalina.
the security settings in Catalina.

In a Terminal, ensure you have vagrant 2.2.9 and upgrade if necessary or you will receive error from vagrant plugin
on VM creation.
Expand All @@ -104,10 +104,9 @@ vagrant plugin list
## VirtualBox

Download and install
[https://download.virtualbox.org/virtualbox/6.1.6/VirtualBox-6.1.6-137129-OSX.dmg](VirtualBox),
including the
[https://download.virtualbox.org/virtualbox/6.1.6/Oracle_VM_VirtualBox_Extension_Pack-6.1.6.vbox-extpack](extension
pack).
[VirtualBox](https://download.virtualbox.org/virtualbox/6.1.6/VirtualBox-6.1.6-137129-OSX.dmg),
including the [extension
pack](https://download.virtualbox.org/virtualbox/6.1.6/Oracle_VM_VirtualBox_Extension_Pack-6.1.6.vbox-extpack).

Alternatively, it can be installed from Homebrew using:

Expand All @@ -121,10 +120,14 @@ The extension pack has to be manually added after the installation of
VirtualBox; this can be done through the UI, or running:

```bash
sudo VBoxManage extpack install /path/to/download/Oracle_VM_VirtualBox_Extension_Pack-6.1.6.vbox-extpack
V=$(VBoxManage --version | sed 's/r.*//')
sudo VBoxManage extpack install /path/to/download/Oracle_VM_VirtualBox_Extension_Pack-$V.vbox-extpack
```

These instructions have been tested against 6.1.6.
These instructions have been tested against 6.1.4 and 6.1.6. The extension
version must match the version of VirtualBox itself. As explained above,
VirtualBox does not seem to fully support our production configuration, though
it may still be useful for local testing.

## rbenv

Expand Down Expand Up @@ -205,7 +208,7 @@ sudo macinbox --box-format vmware_desktop --disk 250 --memory 57344 --cpu 10 --u
NOTE: Limited use of hardware due to possible bugs in VirtualBox.

```bash
sudo macinbox --box-format virtualbox --disk 250 --memory 32000 --cpu 5 --user-script user-script.sh
sudo macinbox --box-format virtualbox --disk 50 --memory 4096 --cpu 1 --user-script user-script.sh
```

The disk size given here (in GB) will be the disk size used by the individual
Expand All @@ -219,7 +222,7 @@ progressively incrementing the size each time we had a "disk is full" error.
build against various virtual instance types on Linux and Windows (see
[#4520](https://github.com/digital-asset/daml/pull/4520)).

250Gb disk, 56Gb memory and 10 virtual cores was arrived at from testing on Mac Mini 2018 nodes with 6 core
250Gb disk, 56Gb memory and 10 virtual cores was arrived at from testing on Mac Mini 2018 nodes with 6 core
processors.

The provided "user script", which can be inspected in the current directory,
Expand Down
0 infra/macos/2-vagrant-files/.gitignore → infra/macos/2-common-box/.gitignore
View file
File renamed without changes.
71 changes: 71 additions & 0 deletions infra/macos/2-common-box/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Host setup

The host machine for this step needs three things:

1. The "macinbox" [Vagrant base box](../1-create-box/README.md) for macOS.
2. VirtualBox (with the extension) or VMWare.
3. Vagrant (and the VMWare plugin if using VMWare).

As all three are already covered by the `macinbox` step, the easiest approach
is probably to just run this on the same box.

You can refer to the instructions in [this README](../1-create-box/README.md),
or, for a more automated setup, use the following set of commands if using
Virtualbox (which will require a sudoer password at some points):

```
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" < /dev/null
brew cask install virtualbox
brew cask install vagrant
cd $(mktemp -d)
V=$(VBoxManage --version | sed 's/r.*//')
wget https://download.virtualbox.org/virtualbox/$V/Oracle_VM_VirtualBox_Extension_Pack-$V.vbox-extpack
sudo VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-$V.vbox-extpack
cd -
```

This will install homebrew, then use it to install the latest versions of
VirtualBox and Vagrant. The `brew cask` invocations need to be ran from a
sudoer account and will request a password (unless passwordless sudo is
configured, which I would not recommend), so this cannot be completely
scripted. `brew` itself will refuse to run if started as root.

# Creating the daily Vagrant box

This step builds upon the `macinbox` step to take a blank macOS image and add:

- The vsts user that will run the Azure agent.
- The XCode CLI tools.
- A working nix installation.
- All the nix dependencies for the daml project.
- A populated Bazel cache for the daml project.

This results in a machine that is completely initialized, without having needed
any credentials. The only bit missing is the actual Azure agent.

This can be run every day, though it is probably good enough to run it once a
week.

To select a provider, you need to set the `PROVIDER` environment variable. If
you do not set it, `vmware_desktop` will be used. The only other valid option
at the moment is `virtualbox`.

The steps to create the box are:

```
vagrant up
vagrant package --output initialized-$(date +%Y%m%d).box
```

The `vagrant up` command can take a while as it is doing all of the internal
setup for the machine. It also ends with shutting down the machine so there
should be no lingering process.

The output file is all that needs to be moved to the other machines; they do
not need the original `macinbox` files.

Once the output file is created, you can also remove the "active" (though at
this point shut down) Vagrant box for this folder by running
```
vagrant destroy
```
20 changes: 20 additions & 0 deletions infra/macos/2-common-box/Vagrantfile
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
provider = ENV["PROVIDER"] || "vmware_desktop"

providers = { "vmware_desktop" => [57344, 10],
"virtualbox" => [4096, 1] }

raise "invalid provider: #{provider}" unless providers.member? provider

Vagrant.configure("2") do |config|
config.vm.box = "macinbox"
config.vm.boot_timeout = 900 # 15 minutes
config.vm.synced_folder ".", "/vagrant", disabled: true
config.vm.provider provider do |prov|
prov.gui = false
prov.memory = providers[provider][0]
prov.cpus = providers[provider][1]
end
config.vm.provision "shell" do |script|
script.path = "init.sh"
end
end
102 changes: 10 additions & 92 deletions infra/macos/2-vagrant-files/init.sh → infra/macos/2-common-box/init.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,12 @@

set -euo pipefail

if [ "$1" = "unset" ] || [ "$2" = "unset" ]; then
echo "Please set the VSTS_TOKEN and GUEST_NAME env vars before running \`vagrant up\`." >&2
exit 1
fi

LOGFILE=/Users/vagrant/run.log
touch $LOGFILE
chmod a+w $LOGFILE

log () {
echo $(/bin/date -u +%Y-%m-%dT%H:%M:%S%z) [$SECONDS] $1 >> $LOGFILE
}

log "Starting init script."
echo "Starting init script."

# macOS equivalent of useradd & groupadd
dscl . -create /Users/vsts
dscl . -create /Users/vsts UserShell /bin/bash
dscl . -create /Users/vsts RealName "$2"
dscl . -create /Users/vsts RealName "Azure Agent"
# Any number is fine here, but it should not exist already.
USER_ID=3000
if id $USER_ID 2>/dev/null; then
Expand All @@ -43,7 +30,7 @@ mkdir -p /Users/vsts
chown vsts:staff /Users/vsts
# END: macOS equivalent of useradd & groupadd

log "Done creating user vsts."
echo "Done creating user vsts."

# Homebrew must be installed by an admin, and I would rather not make vsts one.
# So we need a little dance here.
Expand All @@ -55,7 +42,7 @@ HOMEBREW_INSTALL
# the subdirs
chown -R vsts /usr/local/*

log "Done installing Homebrew."
echo "Done installing Homebrew."

# Install jq
su -l vsts <<'TOOLS_INSTALL'
Expand All @@ -68,68 +55,13 @@ BASHRC
/usr/local/bin/brew install jq netcat xz
TOOLS_INSTALL

log "Done installing tools through Homebrew."

su -l vsts <<AGENT_SETUP
set -euo pipefail
cat /etc/passwd
echo $SHELL
echo \$SHELL
# somehow the change to ~/.bashrc doesn't get picked up here, though it does
# get picked up when I ssh in and run su.
export PATH="/usr/local/bin:\$PATH"
VSTS_ACCOUNT=digitalasset
VSTS_POOL=macOS-pool
VSTS_TOKEN=$1
mkdir -p ~/agent
cd ~/agent
echo Determining matching VSTS agent...
VSTS_AGENT_RESPONSE=\$(curl -sSfL \
-u "user:\$VSTS_TOKEN" \
-H 'Accept:application/json;api-version=3.0-preview' \
"https://\$VSTS_ACCOUNT.visualstudio.com/_apis/distributedtask/packages/agent?platform=osx-x64")
VSTS_AGENT_URL=\$(echo "\$VSTS_AGENT_RESPONSE" \
| jq -r '.value | map([.version.major,.version.minor,.version.patch,.downloadUrl]) | sort | .[length-1] | .[3]')
if [ -z "\$VSTS_AGENT_URL" -o "\$VSTS_AGENT_URL" == "null" ]; then
echo 1>&2 error: could not determine a matching VSTS agent - check that account \\'\$VSTS_ACCOUNT\\' is correct and the token is valid for that account
exit 1
fi
echo Downloading and installing VSTS agent...
curl -sSfL "\$VSTS_AGENT_URL" | tar -xz --no-same-owner
set +u
source ./env.sh
set -u
./config.sh \
--acceptTeeEula \
--agent "$2" \
--auth PAT \
--pool "\$VSTS_POOL" \
--replace \
--token "\$VSTS_TOKEN" \
--unattended \
--url "https://\$VSTS_ACCOUNT.visualstudio.com"
AGENT_SETUP

## Hardening
chown -R root:wheel /Users/vsts/agent/{*.sh,bin,externals}

log "Done installing VSTS agent."
echo "Done installing tools through Homebrew."

# create /nix partition
hdiutil create -size 20g -fs 'Case-sensitive APFS' -volname Nix -type SPARSE /System/Volumes/Data/Nix.dmg
hdiutil attach /System/Volumes/Data/Nix.dmg.sparseimage -mountpoint /nix

log "Created /nix partition."
echo "Created /nix partition."

# Note: installing Nix in single-user mode with /nix already existing and
# writeable does not require sudoer access
Expand All @@ -140,7 +72,7 @@ bash <(curl -sSfL https://nixos.org/nix/install)
echo "build:darwin --disk_cache=~/.bazel-cache" > ~/.bazelrc
END

log "Done installing nix."
echo "Done installing nix."

# This one is allowed to fail; the goal here is to initialize /nix and
# ~/.bazel-cache. If this fails, we probably still have a useful node, though
Expand All @@ -156,21 +88,7 @@ rm -rf daml
exit 0
END

log "Done initializing nix store."

# run the fake local webserver, taken from the docker image
web-server() {
while true; do
printf 'HTTP/1.1 302 Found\r\nLocation: https://%s.visualstudio.com/_admin/_AgentPool\r\n\r\n' "digitalasset" | /usr/local/bin/nc -l -p 80 > /dev/null
done
}
web-server &
echo "Done initializing nix store & Bazel cache."
echo "Machine setup complete; shutting down."

log "Started web server."

# Start the VSTS agent
log "Starting agent..."
su -l vsts <<END
cd /Users/vsts/agent
./run.sh >> $LOGFILE &
END
shutdown -h now
16 changes: 0 additions & 16 deletions infra/macos/2-vagrant-files/Vagrantfile
View file

This file was deleted.

16 changes: 0 additions & 16 deletions infra/macos/2-vagrant-files/Vagrantfile.virtual-box-sample
View file

This file was deleted.

1 change: 1 addition & 0 deletions infra/macos/3-running-box/.gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.vagrant
Loading

0 comments on commit 5b2319e

Please sign in to comment.