ci(ff): support feature freeze automation by fr4nc1sc0-r4m0n · Pull Request #7339 · elastic/fleet-server

fr4nc1sc0-r4m0n · 2026-07-09T09:30:20Z

What is the problem this PR solves?

Fleet Server release automation still relies on Makefile-based scripts and external CLI tools. This PR migrates the feature-freeze and patch release workflows to pure Go Mage targets so release steps can run without depending on hub, gh, sed, yq, or Python.

This continues the work from #6584 and aligns the process with beats (#51831) and elastic-agent (#15433).

How does this PR solve the problem?

Nested Go module under dev-tools/mage/release/ with cmd/fleet-release CLI so go-git / go-github stay out of the root go.mod / NOTICE.txt
Thin mage release:* wrappers invoke the nested CLI
Shared process shape with beats/elastic-agent: version guards, merge-timing labels, idempotent branches/PRs, release issue tracker
Feature freeze opens PR-A and PR-D (no docs/test-env PR)
Removes runNextRelease (next-patch prep is PR-D / patch PR-B)
DRY_RUN=true mode for safe local testing
Docs in RELEASE.md and dev-tools/mage/release/README.md

Workflow alignment

Former (ingest-dev `fleet-server.mak`)	Mage command	PRs produced
`prepare-major-minor-release` + `create-branch-major-minor-release` (+ next-release steps)	`mage release:runMajorMinor`	Push release branch `X.Y`; PR-A (main: mergify + next minor); PR-D (release branch: next patch)
`prepare-patch-release` + `create-prs-patch-release`	`mage release:runPatch`	PR-B (next patch on release branch)

Feature freeze (`CURRENT_RELEASE` already on `main`)

Slot	Branch → base	Changes	Merge label
Release branch	`X.Y` from `main`	pushed	—
PR-A	`ff-prep-main-{CURRENT}` → `main`	Mergify backport + bump to next minor	`merge:1-ff-day`
PR-D	`ff-prep-next-patch-{NEXT}` → `X.Y`	bump to next patch	`merge:4-after-release`

Patch (`CURRENT_RELEASE` already on release branch)

Slot	Branch → base	Changes	Merge label
PR-B	`ff-prep-next-patch-{NEXT}` → `X.Y`	bump to next patch	`merge:4-after-release`

Files updated

version/version.go — DefaultVersion
.mergify.yml — backport rule (PR-A)

Idempotency

Step	Behavior on re-run
`UpdateVersion`	No-op when version already matches
`UpdateMergify`	No-op when backport rule already exists
Branch creation	Reuses existing branch
`CommitAll`	Skips when worktree is clean
`CreatePR`	Returns existing open PR for same head/base

How to test this PR locally

export PROJECT_OWNER="your-user"
export CURRENT_RELEASE="$(grep DefaultVersion version/version.go | sed -E 's/.*"([^"]+)".*/\1/')"
export GITHUB_TOKEN=$(gh auth token)
export DRY_RUN=true

mage release:runMajorMinor
mage release:runPatch

cd dev-tools/mage/release && go test ./... -count=1

Discard local workflow changes after review with git reset --hard HEAD.

Checklist

I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool

Related issue: https://github.com/elastic/observability-robots/issues/3404

Resolve go.mod/go.sum conflicts by keeping main dependency versions and retaining go-git/go-github deps required for release automation.

snyk-io · 2026-07-09T09:30:37Z

⚠️ Snyk checks are incomplete.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
⚠️	Open Source Security	0	0	0	0	See details
⚠️	Licenses	0	0	0	0	See details

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

mergify · 2026-07-09T09:30:56Z

This pull request does not have a backport label. Could you fix it @fr4nc1sc0-r4m0n? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit
backport-active-all is the label that automatically backports to all active branches.
backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

Allow release workflows to be safely retriggered by no-oping when the target version is already applied, reusing existing branches and open PRs, and skipping empty commits.

Move release workflow logic into a dedicated package aligned with elastic-agent and beats, leaving thin mage release:* wrappers in magefile.go.

Enable gomoddirectives replace-local so the dev-tools submodule replace required by mage release imports passes golangci-lint.

The root go.mod replaces dev-tools with a local path. Docker layer caching must include dev-tools/go.mod and go.sum so go mod download can resolve the replaced module.

mergify · 2026-07-09T12:36:07Z

This pull request is now in conflicts. Could you fix it @fr4nc1sc0-r4m0n? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/support-ff-fleet-server upstream/feature/support-ff-fleet-server
git merge upstream/main
git push upstream feature/support-ff-fleet-server

Reorganize dev-tools/mage/release into config, workflows, mergify, git, and github packages matching beats and elastic-agent. Workflows now produce the same branches, PRs, labels, and file updates as the former Makefile-based release process.

mergify · 2026-07-15T12:20:06Z

This pull request is now in conflicts. Could you fix it @fr4nc1sc0-r4m0n? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/support-ff-fleet-server upstream/feature/support-ff-fleet-server
git merge upstream/main
git push upstream feature/support-ff-fleet-server

Adopt the shared nested-module release tooling, merge-timing labels, version guards, and issue tracker. Feature freeze opens PR-A/B/D (omit PR-C; no docs/test-env), remove runNextRelease, and keep fleet-server file updates limited to version.go and mergify.

# Conflicts: # NOTICE-fips.txt # NOTICE.txt

github-actions · 2026-07-22T11:23:48Z

TL;DR

Run check-ci fails because NOTICE.txt and NOTICE-fips.txt contain machine-specific macOS licence-cache paths committed by the PR. CI regenerates them with $GOMODCACHE, so the generated files differ and mage check:ci exits non-zero.

Remediation

Regenerate both NOTICE files from a clean checkout using the repository check target (for example, mage check:notice), then commit the resulting files without /var/folders/... or other host-specific paths.
Re-run .buildkite/scripts/check_ci.sh / mage check:ci and confirm the worktree remains clean.

Investigation details

Root Cause

The PR changes both generated NOTICE files by replacing portable $GOMODCACHE/... licence paths with /var/folders/zq/.../go-mod/... paths. The latest PR commit is a merge that explicitly conflicts in NOTICE.txt and NOTICE-fips.txt, consistent with those generated files being resolved from a macOS environment rather than regenerated in CI.

Evidence

Build: https://buildkite.com/elastic/fleet-server/builds/15648
Job/step: Run check-ci
Key log excerpt: NOTICE-fips.txt: needs update, NOTICE.txt: needs update, followed by Error: git update-index failure ... exit code 1. The diff shows the same path-only substitutions, including /var/folders/.../go-mod/... versus $GOMODCACHE/....
PR diff: NOTICE.txt and NOTICE-fips.txt each contain 81 additions and 81 deletions consisting of these host-specific path substitutions.

Verification

Not run locally against the PR commit because the failed commit is not present in the local checkout; diagnosis is based on the Buildkite log and PR diff.

Follow-up

No matching flaky-test issue was found; this is a deterministic generated-file mismatch, not an infrastructure failure.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

fleet-server publishes version tags but no GitHub Releases, so runMajorMinor failed resolving LatestRelease for minor versions.

fr4nc1sc0-r4m0n · 2026-07-22T11:37:00Z

Test Minor

Click here to expand

DRY_RUN=false CURRENT_RELEASE="9.6.0" mage release:runMajorMinor
=== Starting Major/Minor Release Workflow ===
Resolved LatestRelease from elastic/fleet-server: 9.4.4
Checked out branch: main
Verified CURRENT_RELEASE=9.6.0 matches 9.6.0 on branch main
Creating release branch: 9.6
Already on branch: main
Created branch: 9.6
Checked out branch: 9.6

--- Preparing PR-A: backport rule + version 9.7.0 on main ---
Checked out branch: main
Created branch: ff-prep-main-9.6.0
Checked out branch: ff-prep-main-9.6.0
Added backport rule for 9.6 to .mergify.yml
Updated version to 9.7.0 in version/version.go
Created commit: c0b0a7f275a902c7fc6961504e0468b054d187ba

--- Preparing PR-B: ff-release 9.6.0 on 9.6 ---
Checked out branch: 9.6
Created branch: ff-release-9.6.0
Checked out branch: ff-release-9.6.0
Version already set to 9.6.0 in version/version.go
No changes to commit

--- Preparing PR-D: next patch 9.6.1 on 9.6 ---
Checked out branch: 9.6
Created branch: ff-prep-next-patch-9.6.1
Checked out branch: ff-prep-next-patch-9.6.1
Updated version to 9.6.1 in version/version.go
Created commit: 5ba76c11eeff9dc1c0cf34b6f44925158a5ac19a
Checked out branch: 9.6
Pushed branch 9.6 to remote: origin
Checked out branch: ff-prep-main-9.6.0
Pushed branch ff-prep-main-9.6.0 to remote: origin
Created label "merge:1-ff-day"
Added labels to #2: [release impact:critical backport-9.6 skip-changelog Team:Automation merge:1-ff-day]
Created PR #2: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/2
Checked out branch: ff-release-9.6.0
No new commits on ff-release-9.6.0 compared to 9.6; skipping push and PR creation
Checked out branch: ff-prep-next-patch-9.6.1
Pushed branch ff-prep-next-patch-9.6.1 to remote: origin
Created label "merge:4-after-release"
Added labels to #3: [release Team:Automation skip-changelog merge:4-after-release]
Created PR #3: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/3

=== Major/Minor Release Workflow Complete ===
Release branch created: 9.6
PR 1: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/2 (open)
PR 2: skipped (no related open/merged PR for ff-release-9.6.0 → 9.6)
PR 3: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/3 (open)

Note: PR-C (main docs) is omitted for fleet-server (no docs/test-env updates)

PR Comparison

Former bot flow opened 3 PRs per minor FF (version bump + backport rule + next patch). The new runMajorMinor flow groups bump + mergify into PR-A and opens PR-D for the next patch (PR-C omitted; no docs/test-env; PR-B is a no-op when the release branch already has the correct version):

Current (grouped)	Former (elastic/fleet-server)	Role
fr4nc1sc0-r4m0n#2	#7333 + #7334	PR-A: next minor on `main` + backport rule
fr4nc1sc0-r4m0n#3	#7335	PR-D: next patch on release branch

fr4nc1sc0-r4m0n · 2026-07-22T12:01:25Z

Test Patch

Click here to expand

$ DRY_RUN=false CURRENT_RELEASE="9.4.5" mage release:runPatch
=== Starting Patch Release Workflow ===
Checked out branch: 9.4
Verified CURRENT_RELEASE=9.4.5 matches 9.4.5 on branch 9.4

--- Preparing PR-A: ensure version 9.4.5 on 9.4 ---
Already on branch: 9.4
Checked out branch: patch-release-9.4.5
Version already set to 9.4.5 in version/version.go
No changes to commit

--- Preparing PR-D: next patch 9.4.6 on 9.4 ---
Checked out branch: 9.4
Checked out branch: ff-prep-next-patch-9.4.6
Version already set to 9.4.6 in version/version.go
No changes to commit
Checked out branch: patch-release-9.4.5
No new commits on patch-release-9.4.5 compared to 9.4; skipping push and PR creation
Checked out branch: ff-prep-next-patch-9.4.6
Pushed branch ff-prep-next-patch-9.4.6 to remote: origin
Added labels to #4: [release Team:Automation skip-changelog merge:4-after-release]
Created PR #4: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/4

=== Patch Release Workflow Complete ===
PR 1: skipped (no related open/merged PR for patch-release-9.4.5 → 9.4)
PR 2: https://github.com/fr4nc1sc0-r4m0n/fleet-server/pull/4 (open)

PR Comparison

Former bot flow opened 1 PR per patch (next-patch version bump only; fleet-server has no docs/manifests patch PR). The new runPatch flow opens PR-D for the next patch (PR-A ensure-version is usually a no-op and was skipped in this run):

Current (grouped)	Former (elastic/fleet-server)	Role
fr4nc1sc0-r4m0n#4	#7415	PR-D: next patch on release branch

v1v and others added 8 commits March 12, 2026 17:39

ci(ff): support feature freeze automation phase 1

a1fecf3

ci(ff): support feature freeze automation phase 2

23422b9

update

a666488

fix linting

eff7847

Merge branch 'main' into feature/support-ff-fleet-server

8f0005e

go mod tidy

de623a9

Merge branch 'main' into feature/support-ff-fleet-server

f48d911

Merge upstream/main into feature/support-ff-fleet-server

6d46ae9

Resolve go.mod/go.sum conflicts by keeping main dependency versions and retaining go-git/go-github deps required for release automation.

fr4nc1sc0-r4m0n requested a review from a team as a code owner July 9, 2026 09:30

fr4nc1sc0-r4m0n added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog labels Jul 9, 2026

fr4nc1sc0-r4m0n requested review from blakerouse and macdewee July 9, 2026 09:30

fr4nc1sc0-r4m0n added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team skip-changelog labels Jul 9, 2026

fr4nc1sc0-r4m0n self-assigned this Jul 9, 2026

fr4nc1sc0-r4m0n added backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches backport-8.19 Automated backport to the 8.19 branch labels Jul 9, 2026

fr4nc1sc0-r4m0n mentioned this pull request Jul 9, 2026

ci(ff): support feature freeze automation #6584

Closed

8 tasks

This comment has been minimized.

Sign in to view

fr4nc1sc0-r4m0n added 3 commits July 9, 2026 11:53

fix(ci): update magefile_test.go license header to Elastic License 2.0

bf17d6f

Make release automation idempotent on re-run

c67df45

Allow release workflows to be safely retriggered by no-oping when the target version is already applied, reusing existing branches and open PRs, and skipping empty commits.

refactor(release): extract FF automation to dev-tools/mage/release

64eae34

Move release workflow logic into a dedicated package aligned with elastic-agent and beats, leaving thin mage release:* wrappers in magefile.go.

This comment has been minimized.

Sign in to view

fr4nc1sc0-r4m0n added 2 commits July 9, 2026 13:15

fix(ci): allow local go.mod replace for dev-tools module

815c688

Enable gomoddirectives replace-local so the dev-tools submodule replace required by mage release imports passes golangci-lint.

fix(docker): copy dev-tools go.mod before go mod download

457a0a7

The root go.mod replaces dev-tools with a local path. Docker layer caching must include dev-tools/go.mod and go.sum so go mod download can resolve the replaced module.

mallendem reviewed Jul 10, 2026

View reviewed changes

fr4nc1sc0-r4m0n marked this pull request as draft July 13, 2026 16:57

Merge branch 'main' into feature/support-ff-fleet-server

8b52201

This comment has been minimized.

Sign in to view

fr4nc1sc0-r4m0n added 2 commits July 21, 2026 19:45

Merge upstream/main into feature/support-ff-fleet-server

ddee08c

This comment has been minimized.

Sign in to view

Merge branch 'main' into feature/support-ff-fleet-server

358baf1

# Conflicts: # NOTICE-fips.txt # NOTICE.txt

fix(release): fall back to git tags when GitHub Releases are empty

fb93ee0

fleet-server publishes version tags but no GitHub Releases, so runMajorMinor failed resolving LatestRelease for minor versions.

Uh oh!

Conversation

fr4nc1sc0-r4m0n commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is the problem this PR solves?

How does this PR solve the problem?

Workflow alignment

Feature freeze (CURRENT_RELEASE already on main)

Patch (CURRENT_RELEASE already on release branch)

Files updated

Idempotency

How to test this PR locally

Checklist

Uh oh!

snyk-io Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Snyk checks are incomplete.

Uh oh!

mergify Bot commented Jul 9, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

mergify Bot commented Jul 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

mergify Bot commented Jul 15, 2026

Uh oh!

This comment has been minimized.

github-actions Bot commented Jul 22, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Follow-up

Uh oh!

fr4nc1sc0-r4m0n commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Minor

PR Comparison

Uh oh!

fr4nc1sc0-r4m0n commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Patch

PR Comparison

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fr4nc1sc0-r4m0n commented Jul 9, 2026 •

edited

Loading

Feature freeze (`CURRENT_RELEASE` already on `main`)

Patch (`CURRENT_RELEASE` already on release branch)

snyk-io Bot commented Jul 9, 2026 •

edited

Loading

fr4nc1sc0-r4m0n commented Jul 22, 2026 •

edited

Loading

fr4nc1sc0-r4m0n commented Jul 22, 2026 •

edited

Loading