Tuning HAProxy in a vSphere Environment

Summary

We were able to push through almost 450 MB/sec through HAProxy (which terminated our SSL) by carefully matching our 4-core HAProxy with 2 x 4-core Gorouters (which were on a much slower ESXi host).

Results

0. HAProxy with 1 vCPU

HAProxy had only 1 vCPU during this iteration, and the CPU was maxed to 100% during the test (according to htop). We suspect that TLS was the culprit for much of the traffic: HAProxy terminated TLS traffic inbound, and initiated TLS to the Gorouters.

1. HAProxy with 2 vCPUs

Surprisingly, adding a second vCPU (2 vCPUs, 1 socket) made things worse; bandwidth dropped >30%.

Although the HAProxy never maxed-out its CPU, the Gorouter did, pinned at 100% during the duration of the test (htop).

2. Two Gorouters

Previously we only had 1 Gorouter backing the HAProxy, but now we doubled it to 2 Gorouters. The doubling of VMs notwithstanding, the Gorouters’ CPUs were still pegged at 100%.

2. Two Gorouters with 2 vCPUs

Previously our Gorouters had 1 vCPU. We doubled it to 2 vCPUs. We saw that our HAProxy’s 2 cores were pegged at 100%, and the Gorouters’ 2 cores were almost maxed-out.

Note that the Gorouters are on a Xeon which can be charitably characterized as “the world’s slowest Xeon”. In other words, don’t extrapolate the ratios of HAProxies to Gorouters from this particularly lopsided setup.

2. One HAProxy, Two Gorouters with 4 vCPUs

We were able to push through almost 450 MB/s, but the HAProxy’s CPU was almost, but not quite, 100%. We’ve reached the hardware limits of our homelab—we don’t have enough physical cores to tune any further, so we bring this post to a close.

References

• Test suite: https://github.com/wg/wrk
• Test suite invocation:

./wrk -t8 -c32 -d60s https://10.9.250.10/10mb -H "Host: dora.foundry.fun"

• Our 10MB app: https://github.com/cloudfoundry/cf-acceptance-tests
• Our change to the code (thanks Matthew Kocher!):

--- a/assets/dora/dora.rb
+++ b/assets/dora/dora.rb
@@ -39,6 +39,11 @@ class Dora < Sinatra::Base
     end
   end

+  get '/10mb' do
+    require 'securerandom'
+    $ten_mb ||= SecureRandom.random_bytes(10 * 1024 * 1024)
+  end
+