On-premise is Almost Four Times Cheaper * than the Cloud

* If you don’t count the amount of time spent maintaining the on-premise equipment.

Abstract

My 48-VM (virtual machine) homelab configuration costs me approximately $430/month in hardware, electricity, virtualization software, and internet, but an equivalent configuration on AWS (Amazon Web Services) would cost $1,660/month (almost four times as expensive)!

Disclosures:

• I work for VMware, which sells on-premise virtualization software (i.e. vSphere).
• I didn’t put a dollar value on the time spent maintaining on-premise because I had a hard time assigning a dollar value. For one thing, I don’t track how much time I spend maintaining my on-premise equipment. For another, I enjoy maintaining on-premise equipment, so it doesn’t feel like work.

Shortcomings of On-Premise

• Time and Effort: Before you leap into on-premise, you need to ask yourself the following, “Am I interested, and do I have the time, to maintain my own infrastructure?” If you like swapping out broken hard drives, troubleshooting failed power supplies, creating VLANs, building firewalls, configuring backups, and flashing BIOS—if you like getting your hands dirty—then on-premise is for you.
• Only one IPv4 address: This is a big drawback. Who gets the sole IPv4 (73.189.219.4) address’s HTTPS port—the Kubernetes cluster or the Cloud Foundry foundation? In my case, the Cloud Foundry foundation won that battle. On the IPv6 front there’s no scarcity: Xfinity has allocated me 2601:646💯69f0/60 (eight /64 subnets!).
• Poor upload speed: Although my Xfinity download speed at 1.4 Gbps can rival the cloud VMs’, the anemic 40 Mbps upload speed can’t. I don’t host large files on my on-premise home lab. This may not be a problem if your internet connection has symmetric speeds (e.g. fiber).
• Scalability: I can’t easily scale up my home lab. For example, my 15 amp outlet won’t support more than what it already has (2 ESXi hosts, 1 TrueNAS ZFS fileserver, two switches, an access point, a printer). Similarly, my modestly-sized San Francisco apartment’s closet doesn’t have room to accommodate additional hardware.
• Widespread outages: When I upgraded my TrueNAS ZFS fileserver that supports the VMs, I had to power-off every single VM. Only then could I safely upgrade the fileserver.
• Ground-up Rebuilds: One time I made the mistake of not powering down my 48 VMs before rebooting my fileserver, and I spent a significant portion of my winter break recovering corrupted VMs (re-installing vCenter, rebuilding my Unifi console from scratch).

How I Calculated the AWS Costs

First, I pulled a list of my VMs and their hardware configuration (number of CPUs (cores), amount of RAM (Random Access Memory)) I used the following govc command:

export GOVC_USERNAME=administrator@vsphere.local GOVC_PASSWORD='some-password' GOVC_URL=vcenter-80.nono.io
govc ls -json '/*/vm/*' | jq -r '.elements[].Object.Config
    | select(.Name == null | not)
    | {Name, "NumCPU": .Hardware["NumCPU"], "MemoryMB": .Hardware["MemoryMB"] }
    | join("\t")'

The output is a tab-separated file with three columns: VM Name, cores, RAM. Here’s a typical line from my Ubuntu Jammy-based VM with 4 cores and 2GiB of RAM:

jammy.nono.io	4	2048

I imported the data into a Google Sheet, and then found the closest available instance in the AWS region us-east-1 using the following criteria:

• Use the closest instance type, but round down in AWS’s favor (this post isn’t a hit piece). For example, my biggest VM, nsx.nono.io, has 6 vCPUs and 24 GiB RAM, but AWS doesn’t offer that class of instance, so instead we drop down to the closest comparable instance, the t4g.xlarge with 4 vCPUs and 16 GiB of RAM.
• When there are multiple instance types which match the hardware configuration, use the instance type most favorable (cheapest) to AWS. As a result, many of the instances are Graviton-based. Graviton is AWS’s custom implementation of the ARM (Advanced RISC Machines) architecture, and its instances cost less than an equivalent x86-based instance. For example, for a 2-vCPU 4 GiB instance, we use the t4g.medium instance type (Graviton-based, as noted by the “g”) which costs $15.40 per month and which is 19% percent cheaper than an equivalent t3.medium at $19.05 per month.
• Use a 1-year Reserved Instance pricing instead of On-Demand pricing, which again favors AWS. For example, a t4g.medium with a 1-year reservation costs $15.40 per month which is 37% cheaper than an On-Demand which costs $24.54. I don’t use the 3-year Reserved Instance pricing because I used it once and was trapped in an instance type I didn’t want for two years.

How I Calculated the On-Premise Monthly Cost

I added up the hardware I’ve purchased, then divided by the number of months it has been in use to determine its monthly cost. Admittedly this is not a GAAP-approved (generally accepted accounting principles) method of assigning a monthly cost, but I’m an engineer not an accountant.

Why I didn’t Include the Cost of Storage (Disks)

The costs of disk was overshadowed by the cost of cores and RAM, and I was loath to include another metric. For example, a t4g.medium instance costs $15.40/month, and adding a 30 GiB gp3 EBS (Elastic Block Store) adds a paltry $2.40 (15% increase).

This decision to overlook the cost of disk plays in AWS’s favor.

“Why Do You Hate the Cloud?”

I don’t hate the cloud—I love the cloud! I maintain personal servers on AWS, Azure, Digital Ocean, Google Cloud Platform (GCP) (a GKE cluster!), and Hetzner, and I’ve had them for years. And this is a good excuse to list them all: