A 2012 Mac Mini Beats a Xeon Per Core. That Changed How I Schedule Jobs.

I ran the first real load across my whole cluster expecting to confirm something obvious: that the dual-Xeon server would beat a stack of second-hand Mac minis at everything. It did not.

On two cores, a Mac mini from 2012 hashed SHA-256 at 250 MB/s. The 2016 PowerEdge T430, with two Xeon E5-2620 v4, managed 220 MB/s under the same two-core load. The fourteen-year-old desktop computer won by 14%.

Say precisely what was measured, or the number is worthless: this is megabytes per second of SHA-256, on exactly two cores, for 90 seconds, with no disk or network I/O in the loop. It is not a general-purpose score and it predicts nothing about compile times, database latency, or memory bandwidth. It is a clean two-core arithmetic throughput number, run identically on every machine, which is what makes the machines comparable to each other — and that was enough to invert how I place work.

The fleet

The cluster is twelve Nomad worker nodes across two sites: 76 threads and 124 GB of RAM. Ten of those machines live at the remote site — nine Mac minis and one Dell PowerEdge T430 — and two are LXC containers on a Proxmox host at the main site. The control plane is three Nomad + Consul servers with a real Raft quorum.

Here is the actual inventory, collected over SSH with dmidecode, lscpu, lsblk and smartctl rather than from memory:

Three CPU generations among the minis: Sandy Bridge (2011) ×5, Ivy Bridge (2012) ×3, Haswell (2014) ×1. Every one has a single 1 GbE NIC. The T430 on its own carries more RAM than the entire mini fleet combined (62 GB against ~53 GB) and nearly as many threads as all nine of them together, which is exactly why it earned its own node class.

nproc lies to you, and it lies by exactly 2x

Before any of the benchmark numbers mean anything, a correction I had to make to my own notes.

nproc returns 4 on every mini. For weeks my tallies in this repo said "4 cores per mini" and therefore "36 cores of compute". Both wrong. Those are threads:

$ nproc
4

$ lscpu | grep -E 'Socket|Core|Thread'
Thread(s) per core:   2
Core(s) per socket:   2
Socket(s):            1

Two physical cores, Hyper-Threading on, four logical CPUs. The real number for nine minis is 18 physical cores / 36 threads, not 36 cores. If you size a fleet on nproc you overstate it by exactly a factor of two, and every capacity estimate downstream inherits that error.

The benchmark job

I wanted one measurement per node, once, then done. Nomad has a job type for exactly that shape.

batch with a count lets the scheduler stack several instances on the same node — useless for a per-node comparison. system runs forever. sysbatch runs once on every eligible node and then the job completes. That is the type designed for fleet-wide one-shot work: benchmarks, migrations, collection passes.

The second decision, the one that makes the numbers comparable:

resources {
  cores  = 2   # two whole cores, exclusively reserved, everywhere
  memory = 128
}

cpu = 2000 would reserve 2000 MHz of bandwidth. On a T430 with roughly 67,000 MHz aggregate that is 3% of the machine; on a mini with roughly 10,000 MHz it is 20%. The load would not be comparable and it would not heat the big server at all. cores = 2 reserves two entire physical cores on every machine — identical physical load, comparable numbers, and a real thermal rise.

The payload is deliberately dumb: two shell subshells, each looping dd if=/dev/zero bs=1M count=32 | sha256sum until a deadline 90 seconds out, then summing the megabytes hashed. Pure arithmetic, zero I/O, so I am comparing CPUs and not the mixed bag of HDDs and SSDs underneath them. Eleven of twelve allocations completed; the twelfth is the first trap below.

The result

The Xeon lands sixth. All three 2012 minis beat it, and the 2014 one is 45% faster under the same load.

The explanation is not mysterious once you look at the part numbers instead of the badge. The E5-2620 v4 runs at 2.1 GHz base. It is a many-cores/low-frequency server part: it exists to run 32 threads of throughput work in a rack, not to finish one thread quickly. The i5-3210M from 2012 has a 2.5 GHz base and turbos to 3.1 GHz; the i5-4308U turbos to 3.3 GHz. Laptop-class silicon optimised for burst.

A necessary caveat: I did not measure the T430 with all 32 threads loaded — the benchmark deliberately loads two cores everywhere. Its aggregate advantage, sixteen physical cores against a mini's two, is arithmetic rather than something this run measured. What this run measured is per-core speed, and there the server loses.

Why this changed my placement rules

Before this, my mental scheduler was "big machine gets the important work". That is the rule a spec sheet gives you, and it is wrong in a specific, actionable way.

Core count is throughput. Core speed is latency. They are different resources and they live on different machines.

So the rules became:

• Parallelisable batch work goes to node_class = beefy, the T430. Sixteen cores at its rate beats two fast cores by a wide margin, and it is the only machine with the RAM to hold a large working set.
• Single-threaded and latency-sensitive work — request handlers, a leader process, anything where one thread's wall-clock time is the user-visible number — goes to the fastest per-core node available. At the remote site that is a Mac mini from 2014, then the 2012 ones, and the dual-Xeon server is fifth in line.
• Nothing gets placed on core count alone.

The part I keep coming back to: no spec sheet told me this. "2× Xeon E5-2620 v4, 16C/32T, 62 GB" versus "Core i5, 2 cores, 4 GB" reads as a rout on paper. Ninety seconds of hashing on each machine, which cost me one job file, said otherwise. This is only knowable by measuring, on your hardware, under a load you chose.

Thermals: the minis cook, the Xeon does not notice

This was also the first real load these machines had seen since being racked, so I watched the temperature alert while it ran.

Five minis crossed 80 °C and the temperature alert fired, correctly, for the first time since I wrote it. Small aluminium boxes from 2011–2014, two cores pinned flat out — that is the expected shape, and I would rather prove an alert on a benchmark than on a real workload at 3 AM.

The T430 moved one degree. Two loaded cores out of thirty-two is nothing to it, and its thermal paste had just been replaced — a benchmark is also the moment a bad reseat reveals itself, and this one did not.

One detail that cost me an evening earlier: the Mac hardware exposes around sixty pseudo-sensors through applesmc, and many of them are not temperatures at all — you read -127, -48, 0. My first alert rule fired on all nine minis at idle while their CPUs sat at 40 °C. The real CPU sensor is coretemp:

- alert: TemperatureElevee
  expr: node_hwmon_temp_celsius{chip=~".*coretemp.*"} > 80
  for: 5m

An alert that screams at noise teaches you to ignore alerts, which is worse than no alert.

Three traps I hit running this

1. cores does not work on an LXC-hosted client

The missing twelfth allocation. On nc-client-2, a Nomad client running inside an LXC container, the task refused to start:

Requested CPUs are not available - requested 0,1, available: 1-2,8,10

Nomad's cores does not mean "any two cores". It reserves specific core IDs, starting at 0. An LXC container has a restricted cpuset inherited from the host, and that set may not contain core 0 at all. Here it was 1-2,8,10 — four CPUs, none of them the ones Nomad had picked.

The fix is cpu = <MHz> on LXC nodes, which reserves bandwidth rather than named cores. Keep cores for comparing physical machines: it applies identical load everywhere, whereas a fixed MHz figure is a very different fraction of each machine.

2. HCL2 interpolation eats your shell variables

The first submission lost all twelve allocations instantly:

Unknown variable: There is no variable named "DUR"

There is no such variable because DUR is a shell variable, defined on the first line of my inline script. The problem is that HCL2 parses the file before Docker ever sees the string, and a dollar-brace sequence is HCL's interpolation syntax. It grabbed my shell reference and failed to resolve it.

Write $VAR with no braces. Do not reach for $$ either. Command substitution $( ) and arithmetic $(( )) are both fine — HCL only reacts to the dollar-brace sequence, so the inner $(date +%s) in END=$(( $(date +%s) + DUR )) passes through untouched.

3. sysbatch refuses a reschedule block

My reflex was to add a reschedule policy. Nomad rejects it: System or sysbatch jobs should not have a reschedule policy. That is coherent rather than annoying — rescheduling means "place this work on a different node", and for a job whose entire definition is "once on every node" there is no different node to move to. What you want instead is a restart policy that gives up:

restart {
  attempts = 0
  mode     = "fail"
}

A benchmark that fails should stay failed. I want a photograph, not a permanent load.

What the hardware audit turned up

While I had SSH open on every machine I collected a full audit. Three findings worth repeating for anyone buying used hardware:

The old disks are worn but healthy. SMART shows zero reallocated sectors across the entire fleet, on drives with 16,698 / 18,371 / 20,471 power-on hours. Roughly two years of continuous spinning, all PASSED. Worn is not the same as failing, and the used-hardware market prices them as if it were.

One mini was booting from its hard disk with a healthy SSD idle in the second bay. findmnt showed / on the 931 G HDD while a 113 G SSD sat unmounted on the second bay — the stock Fusion Drive split, never separated after the Linux install. Measured with dd iflag=direct, the idle SSD reads at 664 MB/s against 115 MB/s for the disk it was actually booting from. That machine also has the fastest CPU of the nine minis. Free performance, sitting there, wired up backwards.

The firmware under-reports the RAM ceiling. dmidecode -t 16 claims 8 GB maximum on every unit. The 2011 and 2012 models take 16 GB in practice (2×8 GB DDR3 SO-DIMM); the vendor spec understates it. The exception is the 2014 model, where the RAM is soldered and 8 GB is a hard ceiling. So the four 4 GB machines are the ones worth upgrading, and the one I would most like to upgrade is the one I cannot.

Turning measurements into placement

A benchmark you file away is a blog post. A benchmark the scheduler can read is infrastructure. Every node now carries its class and its measured capability:

client {
  enabled    = true
  node_class = "legacy"        # beefy | legacy | general

  meta {
    site      = "cave"
    model     = "Macmini7,1"
    year      = "2014"
    cpu       = "i5-4308U"
    ram_gb    = "8"
    disk_type = "ssd"
    bench_mbs = "319"          # measured, not guessed
  }
}

Three classes: beefy for the T430, legacy for the minis, general for the two containers at the main site. Jobs then express what they actually need, as hard constraints plus a soft preference:

constraint {
  attribute = "${meta.disk_type}"
  value     = "ssd"
}
constraint {
  attribute = "${meta.ram_gb}"
  operator  = ">="
  value     = "8"
}
constraint {
  distinct_hosts = true
}

affinity {
  attribute = "${meta.bench_mbs}"
  operator  = ">"
  value     = "200"
  weight    = 100
}

That is the whole point of the benchmark. bench_mbs is a number the machine earned by doing work, so placement can be based on measured capability rather than on a model name and an assumption. The affinity is a preference, not a filter: if the fast nodes are full, the job still runs, just slower.

Declaring metadata proves nothing on its own, so I ran a throwaway placement-test job with those constraints and a count of 3. The allocations landed on three of the five nodes that are both SSD and 8 GB, one per host, preferring the fast ones. Had they stacked on one machine or landed on a 4 GB node, the labels would have been decoration.

What this hardware cost

Twelve workers, 76 threads, 124 GB of RAM, across two physical sites. Nine of those machines are second-hand Mac minis, plus one used enterprise server. The oldest disks have twenty thousand hours on them and zero bad sectors.

I am not going to pretend this is the economically rational way to get 76 threads. A cloud instance is faster to obtain and someone else replaces the fans. But the failure domains are real, the network partitions are real, the thermal limits are real, and a lab where nothing can actually break teaches nothing about distributed systems.

And the number I did not expect: the oldest machines in this pile land within about a quarter of a 2016 datacenter CPU on this measurement, and the 2012 ones beat it. Hardware from 2011 to 2014 is not slow. It stopped being new, which is a different property entirely, and the two got confused somewhere along the way.

Measure your own fleet. The spec sheet will not tell you which machine is fastest at the thing you actually run.