A pull request edits base/deployment.yaml to turn on debug logging for staging. The diff shows
one line in a shared file, so the reviewer approves a staging change. Production inherits the same
base, so its pods start logging at debug on the next deploy. Kustomize applies the overlays at
render time, after the review.
What you get
You will end up with a pull-request check that renders every environment from both sides and keeps one comment listing each environment’s changes. This is for you if your overlays share a base.
Short answer
Render every overlay with kustomize build twice, from the pull request’s merge commit and from
the base it merges into. Compare each environment’s pair with dyff between, and post the result
as one pull-request comment. Keep kubectl diff for the deploy step, because an environment not
yet deployed diffs as whole objects, defaulted fields included.
You will need
Kustomize overlays for two or more environments, a CI system that can comment on pull requests,
and Node 22 or later to run the npm ci that fetches the tools. Verified 2026-10-21 against Node
22.22.0, Kustomize 5.8.3, dyff 1.12.0, and kubectl, kube-apiserver, and etcd from envtest
1.37.0. npm ci downloads each tool from its GitHub release into node_modules/.bin and checks
it against a hash pinned in install-tools.sh. The API server and etcd come from the
envtest bundle that controller tests use, and
only the cluster diff needs them. The sample keeps the two checkouts as plain
directories, main/ and pr/, so it runs without a repository.
Approaches compared
| Approach | When it fits | What it costs you | When to pick something else |
|---|---|---|---|
| dyff | Two rendered files and no cluster, for a comment people read | A second binary to pin, and no view of the cluster, so drift stays out of the comment | You need to know what an apply would change in the cluster |
| helm diff plugin | Helm releases, compared with what the cluster has deployed | Helm only, a plugin on every runner, and cluster credentials in the CI job | The manifests come from Kustomize, or CI must not reach the cluster |
| kubectl diff | The deploy step, run against the cluster it is about to change | Cluster credentials in CI, and whole objects with defaulted fields for anything not yet deployed | Pull requests from forks, or a comment a reviewer must read quickly |
kubectl diff answers the deploy step’s question, what this apply changes in this cluster, at the
price of credentials and server-side detail nobody wrote. The helm diff plugin answers the same
question for Helm releases, with the same credentials. dyff compares two files, so it needs
nothing but the renders and cannot see drift in the cluster. Use dyff in the pull-request check
and kubectl diff in the deploy step.
Render both sides of the pull request
The check needs two renders of every environment: the pull request as it would merge, and the
base it merges into. render.sh builds each overlay of one checkout into its own file:
set -euo pipefail
export PATH="$(cd "$(dirname "$0")" && pwd)/node_modules/.bin:$PATH"
checkout=$1
out=$2
mkdir -p "$out"
for overlay in "$checkout"/overlays/*/; do
env=$(basename "$overlay")
kustomize build "$overlay" > "$out/$env.yaml"
donekustomize build prints
the overlay as the cluster would receive it. For a Helm chart, run
helm template with each environment’s values file
instead, and the rest of the check stays the same. In CI, the pull request’s side is the merge
commit GitHub builds for the run, and the base side is the first parent of that commit.1 The
diff then shows what merging would change, not what the branch changed since it forked.
Diff each environment and write the comment
diff-envs.sh renders both checkouts, compares each environment’s pair with dyff between, and
prints a Markdown comment.2 An environment that exists on one side only is compared with an
empty file, so a new overlay shows as documents added. dyff exits 0 for no difference, 1 for
differences, and 255 for an error, so only an exit code greater than 1 stops the script:
envs=$(cd "$work" && ls base head | grep '\.yaml$' | sed 's/\.yaml$//' | sort -u)
for env in $envs; do
touch "$work/base/$env.yaml" "$work/head/$env.yaml"
set +e
dyff between --omit-header --color off --output github --set-exit-code \
"$work/base/$env.yaml" "$work/head/$env.yaml" > "$work/$env.diff"
status=$?
set -e
if [ "$status" -gt 1 ]; then
echo "dyff failed on $env" >&2
exit 1
fi
donedyff matches documents by their Kubernetes identity and list entries by name, so a change reads as a path and two values rather than as shifted lines. Run it on the pull request that turns on debug logging:
bash diff-envs.sh main pr### Rendered manifests: this pull request against its base
| Environment | Changes |
| --- | --- |
| production | 1 |
| staging | 1 |
#### production
~~~diff
@@ spec.template.spec.containers.checkout.env.LOG_LEVEL.value @@
# apps/v1/Deployment/production/checkout
! ± value change
- info
+ debug
~~~
#### staging
~~~diff
@@ spec.template.spec.containers.checkout.env.LOG_LEVEL.value @@
# apps/v1/Deployment/staging/checkout
! ± value change
- info
+ debug
~~~The table is the review. production | 1 says the change meant for staging reaches production,
because the edit sits in the shared base. Moving the variable into the staging overlay leaves the
production render as it was.
Post the comment on every push
The workflow builds the two checkouts from real commits and posts the comment with the GitHub CLI:
# The merge commit GitHub builds for the pull request. Its first parent
# is the commit of the base branch it was merged onto.
- uses: actions/checkout@v6
with:
path: pr
fetch-depth: 2
- name: Check out the base beside it
run: git -C pr worktree add ../main HEAD^1
- name: Fetch the pinned kustomize and dyff
working-directory: pr/ci
env:
SKIP_ENVTEST: 1
run: npm ci --no-audit --no-fund
- name: Render and diff every environment
run: bash pr/ci/diff-envs.sh main pr > comment.md
# One comment per pull request, edited on every push.
- name: Post the comment
env:
GH_TOKEN: ${{ github.token }}
PR: ${{ github.event.pull_request.number }}
run: gh pr comment "$PR" --repo "$GITHUB_REPOSITORY" --edit-last --create-if-none --body-file comment.md--edit-last with --create-if-none, both documented in the
gh pr comment manual, keeps one comment per pull
request instead of one per push. It edits the newest comment by the token’s user, and every
workflow’s GITHUB_TOKEN posts as the same bot, so this holds only while no other workflow
comments that way. Two quick pushes start two runs, and the comment ends up holding whichever
render finished last. A concurrency group at the top of the workflow cancels the older run:
concurrency:
group: manifest-diff-${{ github.event.pull_request.number }}
cancel-in-progress: trueSKIP_ENVTEST=1 leaves out the API server, which only the cluster diff below needs. The job declares
pull-requests: write, and GitHub grants it only to pull requests from branches in the same
repository.
Check it worked
A pull request that changes a production-only value must produce a comment showing that change
and nothing else. check.sh builds that pull request from a copy of main/, with production’s
replicas raised from three to four, then checks two more cases and the workflow’s concurrency
setting:
bash check.shok 1 - a production-only change shows one change in production
ok 2 - and none in staging
ok 3 - and names the field and both values, line for line
ok 4 - an unchanged tree shows no changes anywhere
ok 5 - a base edit shows up in every environment
ok 6 - the workflow cancels the run for an older push
6 of 6 checks passedCheck 3 compares the production section line for line, from @@ spec.replicas @@ to - 3 and
+ 4, so a comment that grows unrelated lines fails it.
When it goes wrong
kubectl diff against a cluster looks like the natural check, but on its first run it buries the
change. cluster-diff.sh starts a throwaway API server, sets dyff as kubectl’s external differ,
and diffs production before anything is deployed:
bash cluster-diff.sh fresh spec:
progressDeadlineSeconds: 600
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: checkout
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
template:
metadata:
labels:
app: checkout
spec:
containers:
- name: checkout
image: "registry.example.com/checkout:1.4.2"
imagePullPolicy: IfNotPresent
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
env:
- name: LOG_LEVEL
value: info
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
memory: 512Mi
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status: {}The namespace holds nothing yet, so kubectl diff prints both objects whole. Sixteen of the
Deployment’s spec lines are fields no manifest set, from progressDeadlineSeconds to
terminationGracePeriodSeconds, filled in by the API server’s defaults.3 Diff renders against
each other in the pull request, and keep the cluster diff for the deploy step, where the objects
exist:
bash cluster-diff.sh deployedservice/checkout created
deployment.apps/checkout created
kubectl diff -f the pull request's production render
metadata.generation (apps/v1/Deployment/production/checkout)
± value change
- 1
+ 2
spec.template.spec.containers.checkout.env.LOG_LEVEL.value (apps/v1/Deployment/production/checkout)
± value change
- info
+ debug
kubectl diff exited 1With production applied, the same pull request diffs as the one real change, plus
metadata.generation, a counter the server raises on every spec change.
The comment step also fails on a pull request from a fork, because GitHub gives a fork’s run a
read-only token.
Upload comment.md as an artifact, and post it from a second workflow triggered by
workflow_run, which never runs the fork’s code, as
GitHub Security Lab describes.
When not to do this
Do not run kubectl diff against production from the pull-request job. It needs production
credentials in a job that runs the branch’s own scripts, and a run from a fork cannot have them.
Do not switch the trigger to pull_request_target to get a write token for forks while checking
out the pull request. That runs the fork’s scripts with the base repository’s permissions.
Do not read a clean render diff as a clean deploy. The renders cannot see drift in the cluster,
such as a manual kubectl edit or replicas that a HorizontalPodAutoscaler owns. The
kubectl diff in the deploy step is the check for that.
Do not build this for one environment with no shared base. The pull request’s own diff already shows every line that reaches the cluster.
Related how-tos
Last verified
Verified 2026-10-21 against Node 22.22.0, Kustomize 5.8.3, dyff 1.12.0, and kubectl,
kube-apiserver, and etcd from envtest 1.37.0. Every output block is what the command preceding
it printed. The cluster is a local API server with no controllers or nodes, started by
cluster-diff.sh and deleted when it exits. The GitHub Actions workflow was not run: main/ and
pr/ stand in for its two checkouts, and the comment it would post is the output of
diff-envs.sh. The helm diff plugin was not run, because the sample renders with Kustomize.
Footnotes
-
GitHub’s documentation sets
GITHUB_REFtorefs/pull/<number>/mergefor apull_requestrun. The page gives the reason in a sentence of its own: tests “run against the merged result” instead of the branch alone. The commit the check renders as the pull request’s side is a trial merge that no branch contains and nobody wrote. ↩︎ Back to text -
The project’s README spells the name with a Greek delta and two florin signs, and gives its pronunciation in the International Phonetic Alphabet as /ˈdʏf/. The line under the title is plainer: “A diff tool for YAML files, and sometimes JSON.” ↩︎ Back to text
-
The two at the top of the list each get a sentence in the Deployment documentation. Ten old
ReplicaSetobjects are kept for rollback by default, and the documentation adds that the ideal number depends on how often and how reliably you deploy. A rollout that stops making progress is reported as stalled onceprogressDeadlineSecondsruns out, and on that field the documentation is brief: “This defaults to 600.” ↩︎ Back to text