Container image vulnerability scanning

govulncheck reads our Go module graph. It says nothing about the Debian packages in python:3.x-slim, and it says nothing about a compiled Go binary that came from somebody else’s build. Those are scanned separately, by Trivy, under the policy on this page.

The second gap is the easier one to misread, so state it precisely — and it is worth stating even though the repository no longer has an instance of it, because it is the trap that reopens the moment someone adds a FROM <vendor>/<tool> to a Dockerfile. leoflow-migrate used to be FROM migrate/migrate. golang-migrate is in our go.mod and govulncheck does see it, resolved against our MVS versions and our toolchain — but what govulncheck could not see was what upstream had vendored into the release binary inside that image: a different dependency set, frozen at their build. That was where 48 of the 50 fixable CRITICAL/HIGH findings in leoflow-migrate lived (usr/local/bin/migrate). A dependency being covered by govulncheck says nothing about a third-party binary of the same name in an image we pull.

deploy/Dockerfile.migrate now compiles that CLI itself, from the version in our go.mod, so there is one migrate binary instead of two and govulncheck covers it by name (see the govulncheck step). The image went from 84 fixable findings — 50 of them CRITICAL/HIGH — to zero, and moved from the reporting lane to the blocking one.

What gets scanned, and what happens when it finds something

“Fails a build” means the job goes red. Whether that blocks a merge is a branch-protection setting: a new job is not a required check by default, so the Trivy (leoflow-server image) and Trivy (leoflow-migrate image) checks have to be added to the protected-branch rules before they can stop anything from merging.

The split is deliberate, and it is about who can fix the finding.

leoflow-server is gcr.io/distroless/static-debian12 plus a Go binary built from this repository. There is no inherited OS package set, so a finding there was introduced by a commit and is closed by a go get. Blocking the PR that introduced it puts the work in front of the person who can do it.

leoflow-migrate is the same shape and is gated on the same criterion: distroless static on Debian 13, a Go binary cross-compiled from our go.mod, and our SQL files. Its Go half is entirely ours, so a finding there is closed by a go get or a toolchain bump. It is scanned by its own job so a red check names which image is red.

Distroless is not zero packages — it carries six (base-files, ca-certificates, media-types, netbase, tzdata, tzdata-legacy), and a fixable finding in one of those is not closeable by a commit here. That residual is handled the same way leoflow-server’s is: a dated entry in .trivyignore.yaml, which requires a statement and an expiry and caps it at 180 days. A reviewed exception that expires on its own is what keeps a blocking gate from turning into a disabled one — which is the failure the report-only lane exists to prevent, and the reason six packages belong in the blocking lane while leoflow-runtime’s full Debian set does not.

That argument only holds if the gate scans the artifact we publish, so it does. deploy/Dockerfile.server — which compiles from source inside a floating golang:1.x-bookworm — is the developer and kind-cluster convenience path; GoReleaser publishes deploy/Dockerfile.server.release, which copies in a binary built by the release workflow’s pinned GO_VERSION. The gate reproduces that recipe and scans the result. This matters more than it looks: Trivy keys stdlib findings to the Go toolchain recorded in the binary’s build info, so a gate compiling with a different toolchain reports a different vulnerability set — and a floating build tag would also mean a stdlib CVE closed by a base rebuild rather than by a go get, reddening the blocking gate for whoever opened the next unrelated PR.

leoflow-migrate needs none of that reproduction: deploy/Dockerfile.migrate is the file the release builds and pushes, so the gate builds the published recipe directly. Its Go toolchain is pinned in the Dockerfile itself, and scripts/check-migrate-cli-build-tags.sh fails CI if that pin, or the golang-migrate build tags, stop matching what govulncheck is given.

The gate still builds HEAD’s binary rather than pulling a published tag, and the daily scan likewise builds the runtime images from the checkout. So a CVE in an already-published image that HEAD no longer produces does not appear in either. Scan the tag directly when that is the question — Trivy reads a remote reference with no local build.

leoflow-runtime inherits a package set we do not author: Debian, underneath python:3.x-slim. A CVE lands in it because a distro security team published an advisory, on a day when nobody here pushed anything. A blocking gate on that fails whoever opens the next unrelated PR, and the person who can actually fix it (by rebasing a base image) is not in that PR. That gate gets switched off within a week — and switched off is worse than never installed, because it still looks like coverage. So it is reported instead, loudly and on a schedule.

Note what decided the lane, because it is not seniority or noisiness: it is whether a finding is closed by a commit here. leoflow-migrate moved from the reporting lane to the blocking one when what the image contains changed, not because the policy was tightened.

The two filters, and why fixability comes first

Both scans apply the same policy:

1. Fixable only (--ignore-unfixed). Debian stable permanently carries CVEs that are not fixed there and never will be. Severity tells you how bad a finding is; fixability tells you whether anyone can do anything about it today. Only the second works as a gate criterion, because a policy that cannot be satisfied is a policy that gets deleted. On leoflow-runtime:py3.11 this filter takes 273 findings down to 14 — and removes all five CRITICALs, every one of which is marked will_not_fix, fix_deferred, or affected upstream.
2. A severity floor of CRITICAL/HIGH, applied after fixability, for the blocking decision only.

Everything fixable is reported at every severity; only fixable CRITICAL/HIGH blocks (on leoflow-server and leoflow-migrate) or is tracked in the issue (on leoflow-runtime).

Unfixed findings are left out of the Security tab as well as the gate. An alert nobody can ever close is not visibility — it is noise that teaches people to stop opening the tab. The complete unfiltered scan is attached to every image-scan run as the trivy-image-reports artifact, kept 30 days.

Where findings appear

• Security tab → Code scanning. Every fixable finding, at every severity, under one category per image: trivy-server-image, trivy-migrate-image, and one trivy-image-runtime-pyXYZ per published Python line. Uploaded from main and from scheduled runs only — a PR upload would overwrite the branch baseline.
• The failing check, for leoflow-server and leoflow-migrate: the job prints the table of fixable CRITICAL/HIGH findings that failed it.
• A single tracking issue, titled Container image CVEs: fixable findings in published images, for the runtime images. The daily scan refreshes its body, comments only when the set of findings actually changes, closes it when the list empties, and reopens that same issue — rather than filing a second one — if the findings come back. One issue, always current. If you reopen it yourself, the scan leaves it alone: it checks who performed the last reopen and will not close a human’s out from under them.
• The job summary of any image-scan run, including on a PR that touches runtime/**, go.mod, or the authoring schema. On a PR it reports and stays green, so you can see what your change did to the image without being blocked by what was already there.

Triaging a finding

Fix it, if the fix is ours. A Go dependency in leoflow-server is a go get. A Python package in the task runtime is a pin in runtime/python/pyproject.toml. A dependency finding in leoflow-migrate is a go get too — it is built from the same go.mod — and a stdlib finding there is a bump of ARG GO_VERSION in deploy/Dockerfile.migrate, kept in step with CI’s by scripts/check-migrate-cli-build-tags.sh. This is the expected outcome for most findings, and Dependabot already opens PRs for the base images weekly.

Wait for it, if the fix is upstream’s. An unfixed Debian CVE needs no action and will not be reported — that is what the fixability filter is for. If it later becomes fixable, it appears on the next daily run.

Accept it, if it genuinely does not apply. Add an entry to .trivyignore.yaml in the PR that argues for it:

vulnerabilities:
  - id: CVE-2023-45853
    statement: >-
      zlib1g in Debian bookworm; upstream marked will_not_fix. Reachable only
      through MiniZip, which nothing in the task runtime links against.
      Re-evaluate when the base moves off bookworm (#NNN).
    expired_at: 2026-12-31

Both fields are mandatory and CI enforces them (scripts/check-trivyignore-entries.sh):

• statement must be a real explanation, not TODO. Say what the finding is, why it is accepted, and what would change that — for someone who finds the line in eighteen months with no other context.
• expired_at must be a future date no more than 180 days out. Trivy enforces it itself: once the date passes it stops honouring the entry and the finding comes back. That makes the result of neglect the finding returns, which is the only safe default for an ignore list. The gate additionally rejects an already-lapsed entry, so dead weight is renewed deliberately or deleted rather than left for the next reader to decode.

The same gate checks the other end of the chain, per scan: every step that runs a trivy scan must pass the file, through --ignorefile, the action’s trivyignores: input, or TRIVY_IGNOREFILE. A renamed ignore file or a flag dropped from one workflow would otherwise leave entries that look accepted and suppress nothing, and asking only whether some workflow still passes it is a question the repository keeps answering “yes” to while the gate that matters has gone unfiltered.

A scan that must apply no suppressions says so where it runs, with a reason the gate length-checks:

# trivy-ignorefile-exempt: this is the unsuppressed copy of record — applying
# .trivyignore.yaml here would make the accepted findings unrecoverable.
- name: Full unfiltered inventory

Two scans are exempt today: that inventory, and the repository trivy fs scan, whose findings come from our own module graph rather than from a base image nobody here authored.

Changing a base image

Not from this policy, and not from a scan result alone. Which base leoflow-runtime ships is an authoring-surface decision — users pin it through base_image in leoflow.yaml, so our choice becomes theirs for years. It is argued separately, on its own evidence. Detection tells you a base has a problem; it does not tell you which base to move to.

Running a scan locally

GO_VERSION is not optional in either recipe below. runtime/Dockerfile defaults it to an older line than CI uses, and the agent binary baked into the image carries the toolchain it was built with — which is exactly what Trivy keys stdlib findings to. Omit the build arg and you scan a binary CI never builds. Take the value from env.GO_VERSION in .github/workflows/image-scan.yaml (1.26.6 at the time of writing).

# Build and scan the task runtime base as CI does.
docker build -f runtime/Dockerfile \
  --build-arg PYTHON_VERSION=3.11 --build-arg GO_VERSION=1.26.6 \
  -t leoflow-runtime:py3.11 .

trivy image --scanners vuln --ignore-unfixed \
  --ignorefile .trivyignore.yaml leoflow-runtime:py3.11

# Just what a blocking gate would fail on.
trivy image --scanners vuln --ignore-unfixed \
  --ignorefile .trivyignore.yaml \
  --severity CRITICAL,HIGH --exit-code 1 leoflow-runtime:py3.11

The control-plane image is the one the release publishes, so reproduce the GoReleaser recipe rather than building deploy/Dockerfile.server:

mkdir -p dist/image-scan-context
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" \
  -o dist/image-scan-context/leoflow-server ./cmd/leoflow-server
docker build -f deploy/Dockerfile.server.release \
  -t leoflow-server:local dist/image-scan-context

trivy image --scanners vuln --ignore-unfixed \
  --ignorefile .trivyignore.yaml \
  --severity CRITICAL,HIGH --exit-code 1 leoflow-server:local

The migration image is published from the same Dockerfile CI gates, so there is nothing to reproduce — build it and scan it:

docker build -f deploy/Dockerfile.migrate -t leoflow-migrate:local .

trivy image --scanners vuln --ignore-unfixed \
  --ignorefile .trivyignore.yaml \
  --severity CRITICAL,HIGH --exit-code 1 leoflow-migrate:local

Do not pass --build-arg GO_VERSION there: unlike runtime/Dockerfile, this one pins the toolchain CI uses as its own default, and overriding it scans a build the release never produces.

Trivy scans a remote reference too, so a published image needs no local build, and this is the only way to see what an already-published tag carries: trivy image ghcr.io/dexadata/leoflow-runtime:py3.11.