feat(image): guard against kernel/modules version mismatch - #673
Open
HarryWaschkeit wants to merge 1 commit into
Open
feat(image): guard against kernel/modules version mismatch#673HarryWaschkeit wants to merge 1 commit into
HarryWaschkeit wants to merge 1 commit into
Conversation
Adds an image-postprocess check that compares the kernel version string embedded in the deployed kernel Image/fitImage against the version directory name shipped in /lib/modules, and fails the build if they don't match. This is a build-time consistency guard, not a caching fix: BitBake's per-task sstate/setscene mechanism can independently sstate-restore do_package_write_ipk/do_populate_sysroot from an older cached object while do_compile/do_sign/do_deploy execute fresh in the same build. Combined with the kernel-yocto SCC/KMETA merge tree's non-reproducible LOCALVERSION suffix (untracked by BitBake signatures), this can produce an internally consistent-looking (per BitBake bookkeeping) but actually broken image where the kernel and its modules originate from different kernel builds and modules silently fail to load at runtime. Observed directly in omnect-os-build/tauril2,gateway-devel build 107 (2026-07-10). See PR omnect#670 for the part of the root cause (a spurious DISTRO_FEATURES vardep on do_sign/do_deploy) that could be fixed at the signature level. This check is independent of and complements that fix: it catches this class of mismatch regardless of cause, turning a silent runtime failure into a loud CI build failure. Current implementation extracts the kernel version by scanning the deployed kernel/fitImage blob for the "Linux version ..." banner string, which works for the uncompressed arm64 Image format used by our machines; compressed KERNEL_IMAGETYPEs would need the check extended to decompress first (it currently warns and skips rather than false-failing in that case).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
BitBake's per-task sstate/setscene mechanism decides cache reuse independently, per task, based only on whether a task's own declared taskhash matches an already-cached object. For linux-yocto style kernel recipes this means
do_package_write_ipk/do_populate_sysrootcan be sstate-restored from an older cached object whiledo_compile/do_sign/do_deployexecute fresh in the same build — each decision is individually 'correct' per BitBake's own bookkeeping.This is compounded by the kernel-yocto SCC/KMETA merge tree embedding a non-reproducible
git describehash intoLOCALVERSION/vermagic at build time, which BitBake's signatures never track. The result: a kernel Image and its shipped kernel-modules package can silently originate from two different kernel builds, producing an image whose modules fail to load at runtime (module version magic mismatch) — with no error anywhere in the build.Observed directly in
omnect-os-build/tauril2,gateway-develbuild 107 (2026-07-10): confirmed via the Concourse build log thatdo_package_write_ipkwas a setscene cache-restore from build 104, whiledo_kernel_metadata/do_compile/do_sign/do_compile_kernelmodules/do_deployall executed fresh — and via device evidence (uname -rvs. the/lib/modulesdirectory name showing different non-reproducible hash suffixes).See #670 for the part of this specific incident's root cause that's fixable at the BitBake-signature level (a spurious
DISTRO_FEATURESvardep ondo_sign/do_deploy).Fix
This PR does not attempt to fix the caching behavior itself — that isn't reliably possible within BitBake's pre-execution signature model when the actual divergence stems from unTracked non-determinism in an upstream task's real output. Instead, it adds an
IMAGE_POSTPROCESS_COMMANDcheck (classes/omnect-verify-kernel-modules.bbclass, inherited byomnect-os-image.bb) that:Linux version ...) from the deployed kernel/fitImage blob inDEPLOY_DIR_IMAGE./lib/modules/<version>directory name actually present in the built rootfs.bb.fatal) if they don't match.This is independent of and complements #670: it catches this class of mismatch regardless of cause, turning a silent runtime failure into a loud, immediate CI build failure — so an inconsistent image like build 107's never reaches a test device or the field again.
Limitations / follow-ups
Imageformat used by our current machines (verified againstphygate-tauri-l-imx8mm-2'sKERNEL_IMAGETYPE = "Image"). A compressedKERNEL_IMAGETYPE(zImage/uImage with gzip/lz4) would need the check extended to decompress first — it currently warns and skips rather than false-failing in that case.