An Hour of CI to Emulate Two Minutes of Work

We cut the first public release of Toolspoke this week. The image is published for linux/amd64 and linux/arm64, because a self-hosted gateway should run on Graviton and Hetzner ARM as happily as on x86.

The first attempt at that build ran for sixty minutes and we cancelled it, unfinished. The commands it was running take about two minutes.

This is what went wrong, how we measured it, and the fix — which is not a clever flag but a different shape of workflow.

The obvious version is the slow one

Building a multi-architecture image with Docker Buildx looks like one line:

- uses: docker/build-push-action@v6
  with:
    platforms: linux/amd64,linux/arm64
    push: true

On an x86 runner, the arm64 half of that runs under QEMU. Every ARM instruction is translated to x86 at runtime. It works, it needs no extra infrastructure, and it is the shape almost every tutorial shows you.

For a Go binary or a static site, the cost is real but tolerable — emulating straight-line code runs maybe five to fifteen times slower. We assumed that was our situation. It was not.

Node is the worst case

The image builds a Next.js application: npm ci, then npm run build, a full webpack production build.

That is a JIT-heavy workload, and a JIT is pathological under emulation. V8 compiles JavaScript to ARM machine code at runtime. QEMU then translates that freshly generated ARM code to x86 — code it has never seen, cannot cache from a previous run, and must handle as it appears. You are not emulating a program. You are emulating a program that writes programs.

The multiplier stops being five to fifteen and becomes twenty to forty.

Measuring instead of guessing

We had a native baseline available the whole time and did not look at it for an hour. Our ordinary CI runs the same two commands on a normal runner:

About two minutes of real work per architecture. Meanwhile the emulated arm64 leg passed sixty minutes and was still going.

The uncomfortable part is that the emulated build never fails. It gives you no error, no warning, and no progress output — GitHub Actions does not serve logs until a job exits, so a build that is crawling and a build that is wedged look identical from outside. We spent that hour unable to tell which one we had.

An hour of runner time to emulate two minutes of work. Once you write it down like that, the fix is obvious.

The fix: stop emulating

Build each architecture on a machine of that architecture. GitHub offers ubuntu-24.04-arm as a hosted runner, so this needs no self-hosted infrastructure.

The shape is two jobs instead of one. Build jobs push by digest with no tag, and a merge job assembles those digests into one manifest:

jobs:
  build:
    runs-on: ${{ matrix.runner }}
    strategy:
      matrix:
        include:
          - platform: linux/amd64
            runner: ubuntu-latest
          - platform: linux/arm64
            runner: ubuntu-24.04-arm
    steps:
      - uses: docker/build-push-action@v6
        with:
          platforms: ${{ matrix.platform }}
          outputs: type=image,name=${{ env.IMAGE }},push-by-digest=true,name-canonical=true,push=true

The no-tag part matters. If both build jobs pushed the same tag, they would fight over one name and whichever finished last would win — until the merge step corrected it, if it ran at all. Pushing by digest means neither half claims the name; only the manifest does.

The merge job then hangs every tag on the manifest list, and a docker pull resolves to whichever architecture the puller is on.

The result:

The two builds run in parallel, so the wall-clock is the slower of them rather than their sum.

Two guards worth copying

Splitting a build into "produce parts" and "assemble parts" introduces a failure the single-job version cannot have: assembling with a part missing.

If one build job fails and the merge runs anyway, you publish a manifest advertising one architecture under latest. That is worse than publishing nothing. An amd64 host pulls happily and nobody notices, while every arm64 host fails to pull an image that plainly claims to exist. So the merge counts its inputs and refuses:

count="$(find . -type f | wc -l | tr -d ' ')"
if [ "$count" -ne 2 ]; then
  echo "::error::Expected a digest from each build job, found $count."
  exit 1
fi

The second guard is to inspect the manifest afterwards and confirm it really carries both platforms, rather than trusting that the command did what it was asked.

That verification step taught us one more thing. On our first release it failed with not found — six seconds after the push had reported success. A registry read straight after a write is not guaranteed to see it. The publish was fine; the check was racing it. It now retries with a widening gap, because a verification step that fails at random is worse than no verification step: the next person learns to re-run it rather than read it.

What we would tell ourselves an hour earlier

Get the native baseline first. It took one command against an existing CI run, and it turned "this feels slow" into "this is thirty times slower than it should be" — which is a different problem with a different fix.

If you build container images for more than one architecture and your build does real compute, check what you are paying for emulation. You may be buying an hour to avoid configuring a second runner.