(This issue is meant to describe the problem, sketch out requirements, and propose one or more solution(s) in order to be fed into a LLM for initial implementation)
Currently bootc tmt integration tests run inside of a bcvk-provisioned VM. However, this VM has general network access to the outside world. That means the test environment can pull arbitrary data at execution time. We are currently trying to reduce the number of testing flakes in bootc CI. A common source of error is the tests pulling container images from quay.io from inside of the VM. This is problematic because it may be many hours into a CI run before a given test is run and encounters a flake, which invalidates the hours of testing that happened beforehand. It should be noted that although quay.io flakes are the most common, there are other flakes both known and unknown that can also occur from within network-enabled test runners.
The general idea then is to enable support in bcvk to disable as much network as possible. Probably it is sufficient to disable outbound routing from the VM, but other options such as manipulating firewall rules could be considered as well. Care must be taken to ensure connectivity between the host and VM is preserved; for example it should still be possible to bcvk libvirt ssh into the VM from the host, but once inside the VM it should not be possible to connect to arbitrary services on the internet.
A followup to this is to pre-stage any required resources on the test runner host (container images, packages, etc.) and expose those resources into the VM. The network isolation support implementation should take into consideration how those resources are exposed to the VM. I suspect that will likely be handled directly on the filesystem via virtiofs, but it could also be feasible to expose network resources from the host onto the VM. As an example, maybe container images are pulled into the host containers-storage, and then the host re-exports those to the VM via a container registry which is reachable via the local network.
This would make the test execution environment more hermetic and similar to something like mock where the inputs (source, spec, buildroot) are populated at the beginning, and then once the build executes it does not have network access.
This a forcing function to make sure all testing inputs are accounted for beforehand. It should greatly cut down on the number of network failures that happen deep into CI runs. If all needed resources are collected at the beginning of the CI pipeline, this means that any flakes or outages in dependent services will be encountered early and should hopefully cause the pipeline to fail on the order of minutes, instead of hours. It also gives us a common place to implement retry logic in order to tolerate transient network errors.
(This issue is meant to describe the problem, sketch out requirements, and propose one or more solution(s) in order to be fed into a LLM for initial implementation)
Currently bootc tmt integration tests run inside of a bcvk-provisioned VM. However, this VM has general network access to the outside world. That means the test environment can pull arbitrary data at execution time. We are currently trying to reduce the number of testing flakes in bootc CI. A common source of error is the tests pulling container images from quay.io from inside of the VM. This is problematic because it may be many hours into a CI run before a given test is run and encounters a flake, which invalidates the hours of testing that happened beforehand. It should be noted that although quay.io flakes are the most common, there are other flakes both known and unknown that can also occur from within network-enabled test runners.
The general idea then is to enable support in bcvk to disable as much network as possible. Probably it is sufficient to disable outbound routing from the VM, but other options such as manipulating firewall rules could be considered as well. Care must be taken to ensure connectivity between the host and VM is preserved; for example it should still be possible to
bcvk libvirt sshinto the VM from the host, but once inside the VM it should not be possible to connect to arbitrary services on the internet.A followup to this is to pre-stage any required resources on the test runner host (container images, packages, etc.) and expose those resources into the VM. The network isolation support implementation should take into consideration how those resources are exposed to the VM. I suspect that will likely be handled directly on the filesystem via virtiofs, but it could also be feasible to expose network resources from the host onto the VM. As an example, maybe container images are pulled into the host containers-storage, and then the host re-exports those to the VM via a container registry which is reachable via the local network.
This would make the test execution environment more hermetic and similar to something like
mockwhere the inputs (source, spec, buildroot) are populated at the beginning, and then once the build executes it does not have network access.This a forcing function to make sure all testing inputs are accounted for beforehand. It should greatly cut down on the number of network failures that happen deep into CI runs. If all needed resources are collected at the beginning of the CI pipeline, this means that any flakes or outages in dependent services will be encountered early and should hopefully cause the pipeline to fail on the order of minutes, instead of hours. It also gives us a common place to implement retry logic in order to tolerate transient network errors.