VMware’s vSphere is an Infrastructure as a Service (IaaS) which runs Virtual Machines (VMs). BOSH is a VM orchestrator which automates the creation of VMs. NSX-T is a pluggable Network backend for vSphere (and other hypervisors). NSX-T allows the creation of opaque networks in vSphere, networks whose detail and configuration of the network is unknown to vSphere and which is managed outside vSphere.
With the release of BOSH vSphere CPI v40, users can attach their BOSH-deployed VMs to an NSX-T opaque network.
Opaque networks are treated as ordinary BOSH networks; in other words, writers of BOSH manifests need not concern themselves with the underlying type of network whether it be opaque, distributed switch port group or standard switch port group. The manifest need only contain the name of the network; it can be blissfully ignorant of the underlying implementation.
This blog post describes how to attach a deployed VM to an opaque network. We first deploy a BOSH director attached to an opaque network via the BOSH CLI, then we use the BOSH director to deploy two VMs to the opaque network.
0. Quick Start
Like other vSphere networks, the name of the opaque network as it appears in
vSphere should match the name of the network as it appears under the
cloud_properties section of the manifest.
In the following screenshot from the vSphere web interface, we can see the opaque network, opaque. The unique icon (a network interface sprouting from a cloud) identifies it as an opaque network.
In our corresponding BOSH Cloud Config we specify that VMs placed in the BOSH
western_us network should
be attached to the vSphere opaque network. Note that under
we make sure to use the opaque network name as it appears in vSphere, opaque:
networks: - name: western_us type: manual subnets: - range: 192.168.0.0/24 cloud_properties: name: opaque # must match name of network in vSphere
At this point, the seasoned BOSH manifest writer will have enough information to deploy to NSX-T networks, and may find the remainder of this post uninteresting. The remainder of the blog post is directed towards those interested in detailed examples.
1. Double Deployment: BOSH Director First, VMs Second
We will deploy twice to demonstrate BOSH’s opaque network feature: first to demonstrate its effectiveness with the BOSH CLI, second to demonstrate its effectiveness with the BOSH director:
- Our first deployment is the BOSH Director itself, using the new BOSH Command
Line Interface (CLI) (
bosh2). Due to an artifact of our environment [artifact] , The BOSH director cannot attach solely to the opaque network; to work around this restriction, we will attach it to two networks: a distributed virtual port group, scarlet, and the NSX-T opaque network, opaque.
- Our second deployment is two VMs which reside exclusively on the opaque network. A successful BOSH deploy indicates that the opaque network is functioning properly — a BOSH deploy won’t succeed unless the director is able to communicate with the VMs over the network.
1.0 Deploy BOSH Director
We use the this manifest to deploy our BOSH director.
We type the following commands (note the
have been obscured) to deploy our BOSH director, log into it, and upload a
bosh2 create-env bosh-vsphere.yml -v vcenter_ip=vcenter.XXXX -v vcenter_password=XXXX -l vsphere-creds.yml bosh2 -e 10.85.46.6 --ca-cert <(bosh2 int vsphere-creds.yml --path /director_ssl/ca) alias-env nsx-t
We check our vSphere Web Client to make sure our BOSH director is attached to both networks:
We authenticate against our BOSH director and upload a stemcell:
export BOSH_CLIENT=admin export BOSH_CLIENT_SECRET=`bosh2 int ./vsphere-creds.yml --path /admin_password` bosh2 -e nsx-t upload-stemcell https://s3.amazonaws.com/bosh-core-stemcells/vsphere/bosh-stemcell-3363.15-vsphere-esxi-centos-7-go_agent.tgz
Our director is up and running; we are now ready to use our director to deploy two VMs.
1.1. Use BOSH Director to Deploy Two VMs
We upload our Cloud Config:
bosh2 -e nsx-t -n update-cloud-config cloud-config-vsphere.yml
Now we create a minimal deployment consisting of two VMs (BOSH manifest).
bosh2 -e nsx-t -n deploy -d minimal minimal.yml
We make sure the deploy succeeds — a successful deploy indicates the BOSH director and its two VMs are able to communicate over the opaque network:
... Task 56 done Succeeded
Addendum: Technical Requirements
- BOSH vSphere CPI v40+ (v40 tested)
- vSphere 5.5 (VMware vCenter Server Appliance 22.214.171.12400 tested)
- NSX-T 1.0+ (126.96.36.199.0.4191070 tested)
We use bosh-deployment to generate our BOSH director’s manifest. We use this custom script which uses this custom configuration. Much of the complexity derives from the need to create a dual-homed director.
All hosts in a cluster should have at least one physical interface allocated to NSX-T. If not, VMs deployed to hosts with no physical cards allocated to NSX-T will not be able to communicate.
If the operators of a vSphere environment have made the unfortunate decision to
identically name multiple networks (e.g.
VM Network), the distributed virtual
port group will be selected first, followed by the opaque network, followed by
the standard switch port group.
Technical details: The BOSH vSphere CPI introduces a new code path which examines the type of network to which the VM is being attached, and, if it’s an opaque network, uses the vSphere API to apply an opaque-specific backing to the VM’s network interface card.
The astute reader may ask, “Multi-homed? Is there something about NSX-T’s opaque networks that require a multi-homed BOSH director?”
[A computer which is attached to more than one network is referred to as a “multi-homed” computer. Routers and gateways are the canonical multi-homed computers.]
The short answer is no, opaque networks do not require a multi-homed BOSH director. The reason we’re deploying a multi-homed director is a side-effect of the manner in which the opaque network was created for our environment: it was created without routable IP addresses and without a router. Had it been created with routable IP addresses and had a default gateway attached, a multi-homed director would not have been necessary.