Error during upgrade w/ composefs-backend and UKI since 1.16.4: composefs= parameter mismatch

[marcoh00]

Issue

When updating an existing system that makes use of UKIs with bootc < 1.16.4 to a system built with bootc >= 1.16.4, the following error emerges:

error: Upgrading composefs: Performing Upgrade Operation: Setting up UKI boot: Writing {{kernel version}}.efi to ESP: The UKI has the wrong composefs= parameter (is 'sha512:{{ composefs digest embedded in the UKI }}, should be sha512:{{ composefs digest as seen by bootc }})

Reference

This was already identified as part of a discussion and the commit that introduced this bisected by @jnperry : #1984 (comment)

As far as I understand, the behavior with regard to which xattrs that are part of the OCI image are actually considered for digest calucations have changed. Subsequently, older versions of bootc calculate digests that are different from current versions of bootc.

Workaround (that doesn't work)

In order to fix the upgrade short term, my idea was to utilize a git build of bootc to update the system, which I built from cd35048 (that seems to fix another composefs upgrading issue). This should actually calculate the digest correctly, ... right?

$ sudo env RUST_LOG=debug target/release/bootc upgrade
[...]
 INFO Direct pull: fetching docker://{{ target image}} into composefs repository
[...]
DEBUG get_blob: fetching blob self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:90d5bf511ed8d6aed3b07b660b50809270d4b353314f2c8c834be5b3142439ab", split: 6 } size=377465793
[...]
 INFO Pulled image into composefs repository message_id="4c3b2a1f0e9d8c7b6a5f4e3d2c1b0a9f8" bootc.operation="pull" bootc.manifest_digest=sha256:544acedbab7177b24082466c6d9aeada2dab7ee307835347b31d348f4089c6b0 bootc.manifest_verity="3604ee50a9015d37a1630d8fc847aa2e3ffbc170f0ee3dfae9bc592f6ede353cbcfadca01b096d9687dac34c98e9b7630a5512d2afea1ebbe4e08122a04ea0d8" bootc.config_digest=sha256:afc3b05f1087f69c723635009a9fbd7000adb58ff2cf9578d2022d999c6878f0 bootc.config_verity="97d688244ae037de20478ee26ab76934262eb1c9e6ac097a262e4d5d8f076b14c7cc3b9753dadbf0a562f12a052fc8267d424be2e907103ec98e23ceff90d99f" bootc.tag="localhost/bootc-sha256:544acedbab7177b24082466c6d9aeada2dab7ee307835347b31d348f4089c6b0"
[...]
error: Upgrading composefs: Performing Upgrade Operation: Setting up UKI boot: Writing 7.1.4-zen1-1-zen.efi to ESP: The UKI has the wrong composefs= parameter (is 'sha512:f72c5621c43a55c3ada4e4a982a7c4f2fa310cfd05f224aef4a410aaea367fe5d3ff9f5735eb9d1833c1dc34d09ebb0a31bde656e52326efefe9d69b1b1ddf36', should be sha512:cdb5590aa75c3d7900fd3a4f3946940435b5c44f8d3cc5496f09b60f10d98ffde7012a7c4cf145239825f53b7b1dd9ee8f12c0a40e4767fbc7bc15a79f2f6b66)

Surprisingly (for me), I was not successful. Even more surprisingly, the current version does calculate the correct digest using compute-composefs-digest-from-storage (while failing to do so during upgrade):

sudo target/release/bootc container compute-composefs-digest-from-storage --write-dumpfile-to _from-outside_current_bootc.dump {{ target image }}
f72c5621c43a55c3ada4e4a982a7c4f2fa310cfd05f224aef4a410aaea367fe5d3ff9f5735eb9d1833c1dc34d09ebb0a31bde656e52326efefe9d69b1b1ddf36

While the "old", system bootc indeed calculates the wrong digest, both when using upgrade and when using compute-composefs-digest-from-storage:

sudo bootc container compute-composefs-digest-from-storage --write-dumpfile-to _from-outside_old_bootc.dump {{ target image }}
cdb5590aa75c3d7900fd3a4f3946940435b5c44f8d3cc5496f09b60f10d98ffde7012a7c4cf145239825f53b7b1dd9ee8f12c0a40e4767fbc7bc15a79f2f6b66

edit: If it is of interest, all dumpfiles are completely equal:

$ diff -Naur _from-containerbuild.dump _from-outside_current.dump
$ diff -Naur _from-containerbuild.dump _from-outside_old_bootc.dump
$

[jnperry]

my fix was to use the version of bootc that was used to generate the image when i was updating. then after that my system has the new bootc from the update

[marcoh00]

Yes, that was the plan :-) Unfortunately it didn't work out (v1.16.4 shows the same behavior).

I became a bit suspicious because of (1) the matching dumpfiles (the extra attributes like user.overlay and other problematic attributes weren't present in my images anyway, so why should excluding them change anything?), and (2) the differing hashes using the same bootc version.

I was able to find out that this seems to be the repo format. Both compute-composefs-digest-from-storage and ukify will create a temporary repo with default (and thus, FormatVersion::V1) settings. If there's nothing specified (as is the case with my system), format_config from composefs-rs will default to V2. Because the version is passed to compute_image_id (as it influences the output erofs image format, which the digest is the verity value of), the hashes differ.

So to me it looks like a problem of different default values for the repo metadata file vs. default repo config. I have no idea how this could be solved cleanly without saving the format version alongside each image/UKI, or forcing migration to any of the two to a fixed version. In any case, this is super confusing (to quote format_config: right now the choice is between "Legacy V1 repo" and "Old V2 repo", with V1 being the new default - while legacy_erofs_default a few lines above returns V2).

[marcoh00]

Yes, that's the reason. As a workaround it is possible to force bootc to create images using the old/new/legacy V1 format by setting it in meta.json.

So, if the configuration looked like this:

{
  "version": 1,
  "algorithm": "fsverity-sha512-12",
  "features": {
    "compatible": [],
    "read-only-compatible": [],
    "incompatible": []
  }
}

V1 format can be forced like this:

{
  "version": 1,
  "algorithm": "fsverity-sha512-12",
  "features": {
    "compatible": [],
    "read-only-compatible": [],
    "incompatible": []
  },
  "erofs_formats": {
    "default": 1
  }
}

I have tested this on my "I-dont-care-if-it-catches-fire" laptop and it seems to work. As far as I can see this should mainly influence the creation of composefs/erofs images on pull, but I'm not sure. Would be happy if anyone with a deeper understanding could confirm that it's safe.

[cgwalters]

As far as I can see this should mainly influence the creation of composefs/erofs images on pull, but I'm not sure. Would be happy if anyone with a deeper understanding could confirm that it's safe.

That is 100% safe.

I'll look at what we can do to fix this though.

[cgwalters]

composefs/composefs-rs#367 should fix this once we update bootc to use it