In this blog post, we describe the procedure we followed in order to create a custom Google Compute Engine (GCE) stemcell with a user cunnie whose ~/.ssh/authorized_keys is pre-populated with a specific public key.

0. Download the Stemcell to Customize

We need to make our changes from a Linux machine (macOS can not mount Linux filesystems as a loopback device). [macOS mount]

We repurpose our BOSH Lite Director as a stemcell-building VM (BOSH Lite is already installed on our macOS workstation, it has mounted our macOS’s filesystem under /vagrant (simplifies stemcell transfer and eliminates running-out-of-space issues), and using it to build our stemcell in no way impedes its ability to perform as a BOSH Director).

 # we use the BOSH Lite directory as a staging point
cd ~/workspace/bosh-lite
 # download the stemcell:
curl -L -o tmp/custom_stemcell_3263.3.tgz

1. Use BOSH Lite to Modify stemcell

 # bring up BOSH Lite (if it isn't already running):
vagrant up
vagrant ssh
 # we are now in a Linux box; we will customize our stemcell here:
 # create our mountpoint
sudo mkdir /mnt/stemcell
cd /vagrant/tmp
mkdir stemcell image
cd stemcell
 # unpack the stemcell:
tar xvf ../custom_stemcell_3263.3.tgz
cd ../image
 # unpack the bootable Linux disk image
tar xvf ../stemcell/image
 # connect the bootable Linux disk image to a loopback device
 # the extracted file may be `disk.raw` instead of `root.img`
sudo losetup /dev/loop0 root.img
 # probe for the new partitions
sudo partprobe /dev/loop0
 # mount the disk image
sudo mount /dev/loop0p1 /mnt/stemcell
 # we like to use chroot to avoid accidentally polluting the BOSH Lite filesystem
sudo chroot /mnt/stemcell /bin/bash
 # update the stemcell to 3263.3.1
echo -n 3263.3.1 > /var/vcap/bosh/etc/stemcell_version
 # we create user 'cunnie' in the admin, bosh_sudoers, and bosh_sshers groups
 # (sudo & ssh), passwd `c1oudc0w`.
 # CentOS stemcells should use group wheel instead of group admin
useradd -m -G admin,bosh_sudoers,bosh_sshers -p $(openssl passwd -1 -salt xyz c1oudc0w) cunnie
 # change to the new user's homedir
cd ~cunnie
 # we create the directory
mkdir .ssh
 # install the public key
echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9An3FOF/vUnEA2VkaYHoACjbmk3G4yAHE3lXnGpIhz3EV5k4B5RzEFKZnAIFcX18eBjYQIN9xQO0L9xkhlCyrQHrnXBjCDwt/BuQSiRvp3tlx9g0tGyuuJRI5n656Shc7w/g4UbrQWUBdLKjxTT4kTgAdK+1pgDbhAXdPtMwt4D/sz5OEFdf5O5Cp+0spxC+Ctdb94taZhScqB4xt6dRl7bwI28vZdq6Sjg/hbMBbTXzSJ17+ql8LJtXiUHO5W7MwNtKdZmlglOUy3CEIwDz3FdI9zKEfnfpfosp/hu+07/8Y02+U/fsjQyJy8ZCSsGY2e2XpvNNVj/3mnj8fP5cX' > .ssh/authorized_keys
 # set ownerships and permissions
chmod 700 .ssh
chmod 600 .ssh/authorized_keys
chown -R cunnie:cunnie .ssh
 # we're done, hop back to our "normal" BOSH Lite root filesystem
 # unmount our filesystem
sudo umount /mnt/stemcell
 # delete our loop device
sudo losetup -d /dev/loop0
 # tar up our modified Linux bootable disk image
tar czvf ../stemcell/image *

We find it good practice to modify the stemcell number when customizing a stemcell — it prevents many kinds of errors, and well worth the additional time spent re-compiling BOSH releases. In this example, we bump the stemcell number to 3263.3.1:

 # change back to our stemcell directory
cd ../stemcell
 # update the manifest to use a new stemcell number
vi stemcell.MF
-version: '3263.3'
+version: '3263.3.1'
bosh_protocol: 1
sha1: 6dec63560e5ee516e8495eeb39553e81049e19b8
operating_system: ubuntu-trusty
  name: bosh-google-kvm-ubuntu-trusty-go_agent
-  version: '3263.3'
+  version: '3263.3.1'

We create our new, custom stemcell; we put the new version number in the name:

 # create the stemcell
tar czvf ../custom_stemcell_3263.3.1.tgz *
 # exit the Vagrant ssh session

2. Upload the New Stemcell

We upload our new stemcell to our director, which takes ~6 minutes with our internet connection.

 # we upload the stemcell to our BOSH Director
bosh upload-stemcell ~/workspace/bosh-lite/tmp/custom_stemcell_3263.3.1.tgz

3. Deploy

We use the new BOSH Golang CLI to deploy: [Golang CLI]

bosh deploy -d concourse concourse-ntp-pdns-gce.yml \
  -l <(lpass show --note deployments) \
  -l <(curl -L \

4. Test

We test our custom stemcell by ssh’ing into our newly-deployed VM ( using the special user account we created, “cunnie”, with our private key:

ssh -i ~/.ssh/google

5. Alternate Stemcell Formats: .vmdk

Virtual Machine Disk (VMDK) is a format frequently used by VMware and VirtualBox virtualization.

The above instructions do not work for .vmdk-format disks; however, such disks can be customized with additional steps. The steps require the executable qemu-img, a utility which converts different disk formats.

We need to convert the .vmdk to a raw disk image (root.img); Run the following command after running tar xvf ../stemcell/image in the instructions above:

if [ -f image-disk1.vmdk ]; then
  qemu-img convert image-disk1.vmdk -O raw root.img

Similarly, when we tar up the new stemcell, we need to convert the raw disk image (root.img) back to a .vmdk. Run the following command after running sudo losetup -d /dev/loop0 in the instructions above:

if [ -f image-disk1.vmdk ]; then
  rm image-disk1.vmdk
  qemu-img convert root.img -O vmdk image-disk1.vmdk
  # For Vmware Vsphere
  # qemu-img convert -o subformat=streamOptimized,compat6 root.img -O vmdk image-disk1.vmdk
  rm root.img
  SHASUM=($(shasum image-disk1.vmdk))
  perl -pi -e "s/vmdk\)=\s\S+\$/vmdk\)= $SHASUM/"

Note that we have taken the opportunity to update the Secure Hash Algorithms (SHA) inside the image’s manifest ( If we didn’t update the SHA, the stemcell would fail to upload to the director.


[macOS mount] We would be interested if someone has mounted a Linux filesystem to a macOS machine using a loopback device. If the procedure is not too burdensome, we may include it in our blog post.

[Golang CLI] We are using an experimental Golang-based BOSH command line interface (CLI), and the arguments are slightly different than those of canonical Ruby-based BOSH CLI; however, the arguments are similar enough to be readily adapted to the Ruby CLI (e.g. the Golang CLI’s bosh upload-stemcell equivalent to the Ruby CLI’s bosh upload stemcell (no dash)).

The new CLI also allows variable interpolation, with the value of the variables to interpolate passed in via YAML file on the command line. This feature allows the redaction of sensitive values (e.g. SSL keys) from the manifest. The format is similar to Concourse’s interpolation, except that interpolated values are bracketed by double parentheses “((key))”, whereas Concourse uses double curly braces “{{key}}”.

Similar to Concourse, the experimental BOSH CLI allows the YAML file containing the secrets to be passed via the command line, e.g. -l ~/secrets.yml or -l <(lpass show --note secrets)

The Golang CLI is in alpha and should not be used on production systems.

Corrections & Updates


Added instructions for customizing .vmdk stemcells.


Userid created is a member of the bosh_sshers and bosh_sudoers groups. Newer stemcells have been hardened to only allow ssh access to users who are a member of the bosh_sshers group.