Network Diagram

Network Diagram. We want to maximize the throughput from the blue box (the client) to the green box (HAProxy)


We were able to push through almost 450 MB/sec through HAProxy (which terminated our SSL) by carefully matching our 4-core HAProxy with 2 x 4-core Gorouters (which were on a much slower ESXi host).


Bandwidth MB/secondConfiguration
201.27MB1 HAProxy: 1 vCPU
136.47MB1 HAProxy: 2 vCPUs
270.56MB2 Gorouters: 1 vCPU
350.48MB2 Gorouters: 2 vCPUs
447.49MB1 HAProxy, 2 Gorouters: 4 vCPUs

0. HAProxy with 1 vCPU

HAProxy had only 1 vCPU during this iteration, and the CPU was maxed to 100% during the test (according to htop). We suspect that TLS was the culprit for much of the traffic: HAProxy terminated TLS traffic inbound, and initiated TLS to the Gorouters.

1. HAProxy with 2 vCPUs

Surprisingly, adding a second vCPU (2 vCPUs, 1 socket) made things worse; bandwidth dropped >30%.

Although the HAProxy never maxed-out its CPU, the Gorouter did, pinned at 100% during the duration of the test (htop).

2. Two Gorouters

Previously we only had 1 Gorouter backing the HAProxy, but now we doubled it to 2 Gorouters. The doubling of VMs notwithstanding, the Gorouters’ CPUs were still pegged at 100%.

2. Two Gorouters with 2 vCPUs

Previously our Gorouters had 1 vCPU. We doubled it to 2 vCPUs. We saw that our HAProxy’s 2 cores were pegged at 100%, and the Gorouters’ 2 cores were almost maxed-out.

Note that the Gorouters are on a Xeon which can be charitably characterized as “the world’s slowest Xeon”. In other words, don’t extrapolate the ratios of HAProxies to Gorouters from this particularly lopsided setup.

2. One HAProxy, Two Gorouters with 4 vCPUs

We were able to push through almost 450 MB/s, but the HAProxy’s CPU was almost, but not quite, 100%. We’ve reached the hardware limits of our homelab—we don’t have enough physical cores to tune any further, so we bring this post to a close.


./wrk -t8 -c32 -d60s -H "Host:"
--- a/assets/dora/dora.rb
+++ b/assets/dora/dora.rb
@@ -39,6 +39,11 @@ class Dora < Sinatra::Base

+  get '/10mb' do
+    require 'securerandom'
+    $ten_mb ||= SecureRandom.random_bytes(10 * 1024 * 1024)
+  end