Creating a local cluster with `make deploy-bink` fails

[Johan-Liebert1]

Reading the docs and trying out make deploy-bink, I'm greeted with a failure. Which is weird as it did work before

Here's what I ran

$ systemctl --user start podman.socket
$ make teardown-bink
$ make deploy bink

Logs

bink registry start
podman pull ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk
Trying to pull ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk...
Getting image source signatures
Copying blob e94c7e10879b skipped: already exists  
Copying blob 5cdd656c1d1a skipped: already exists  
Copying config f22f6c351d done   | 
Writing manifest to image destination
f22f6c351d46996f6fe42c016f7fb0667290e2b6a83d6ff650d60f952df82674
bootc_img=$(podman inspect --format '{{index .Config.Labels "bink.bootc-image"}}' ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk) && \
        bootc_digest=$(podman inspect --format '{{index .Config.Labels "bink.bootc-image-digest"}}' ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk) && \
        podman pull "$bootc_img@$bootc_digest" && \
        podman tag "$bootc_img@$bootc_digest" localhost:5000/node:latest
Trying to pull ghcr.io/bootc-dev/bink/node@sha256:646a1b971d9bb3f2d3f2d43c63bcb721bcdd803a8db36d1e298dc4b828ba90b2...
Getting image source signatures
Copying blob bd9ddc54bea9 skipped: already exists  
Copying blob e4331430029c skipped: already exists  
Copying blob 5735cdb2999a skipped: already exists  
Copying blob 38bdfd546a3f skipped: already exists  
Copying config cd5d55566e done   | 
Writing manifest to image destination
cd5d55566e9ca3a49ea588e9d44c54f4b48e1387646c273d1f56e8ad33b8ecac
podman push --tls-verify=false localhost:5000/node:latest
Getting image source signatures
Copying blob 38bdfd546a3f skipped: already exists  
Copying blob e4331430029c skipped: already exists  
Copying blob 5735cdb2999a skipped: already exists  
Copying blob bd9ddc54bea9 skipped: already exists  
Copying config cd5d55566e done   | 
Writing manifest to image destination
bink cluster list 2>&1 | grep -qw e2e || { \
        node_digest=$(skopeo inspect --tls-verify=false --format '{{.Digest}}' docker://localhost:5000/node:latest) && \
        bink cluster start --cluster-name e2e --node-name controller --api-port 0 --expose ./kubeconfig-e2e \
        --node-image ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk --target-imgref registry.cluster.local:5000/node@$node_digest; }
INFO[0000] === Creating Kubernetes cluster ===          
INFO[0000]                                              
INFO[0000] Step 1: Creating cluster network...          
WARN Subnet 10.88.0.0/16 already in use, creating network with auto-assigned subnet 
INFO[0000]                                              
INFO[0000] Step 2: Ensuring local registry...           
INFO[0000]                                              
INFO[0000] Step 3: Ensuring DNS container...            
INFO[0000] DNS server running at 10.89.0.2:53           
INFO[0000]                                              
INFO[0000] Step 4: Ensuring required images...          
INFO[0000]                                              
INFO[0000] Step 5: Preparing cluster images volume...   
INFO[0000]                                              
INFO[0000] Step 6: Creating control plane node (controller)... 
INFO[0000] Node image: ghcr.io/bootc-dev/bink/node:v1.35-fedora-44-disk 
INFO[0003] Adding controller DNS entry...               
INFO[0004]                                              
INFO[0004] Step 7: Initializing Kubernetes cluster...   
INFO[0004] === Setting up kubeadm config on controller === 
INFO[0004] SSH endpoint: localhost:2222 (for SSH from container) 
INFO[0004] VM cluster IP: 10.0.0.202 (for Kubernetes communication) 
INFO[0004] Waiting for cloud-init to complete on controller... 
INFO[0004] Waiting for SSH to be ready on localhost...  
INFO[0024] ✓ SSH is ready                               
INFO[0024] Checking cloud-init status...                
INFO[0025] cloud-init status: done (attempt 1/120)      
INFO[0025] ✓ cloud-init completed                       
INFO[0025] Creating kubeadm config file...              
INFO[0025] Copying kubeadm config to VM...              
INFO[0025] ✓ kubeadm config created at /etc/kubernetes/kubeadm-config.yaml 
INFO[0025]                                              
INFO[0025] === Initializing Kubernetes cluster on controller === 
INFO[0025]                                              
[init] Using Kubernetes version: v1.35.7
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [controller kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 10.0
.0.202 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [controller localhost] and IPs [10.0.0.202 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [controller localhost] and IPs [10.0.0.202 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 548.033µs
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://10.0.0.202:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-scheduler is healthy after 3.072259ms
[control-plane-check] kube-controller-manager is healthy after 3.395909ms
[control-plane-check] kube-apiserver is healthy after 2.001775746s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node controller as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node controller as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: c9t2ti.8ixru3yl84tf6xmd
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Configured RBAC rules to allow the API server kubelet client certificate to access the kubelet API
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:

  kubeadm join 10.0.0.202:6443 --token c9t2ti.8ixru3yl84tf6xmd \
        --discovery-token-ca-cert-hash sha256:ddfd2466919521a04bbf335185d41437c3bd25fe54d21b4a8e73d7d3171476ba \
        --control-plane 

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.0.0.202:6443 --token c9t2ti.8ixru3yl84tf6xmd \
        --discovery-token-ca-cert-hash sha256:ddfd2466919521a04bbf335185d41437c3bd25fe54d21b4a8e73d7d3171476ba 
INFO[0042]                                              
INFO[0042] === Setting up kubectl for core user ===     
INFO[0042]                                              
INFO[0045] === Installing Calico CNI plugin ===         
INFO[0049] CNI plugins will be installed to /var/lib/cni/bin 
INFO[0049] Waiting for Calico CNI configuration...      
INFO[0053] ✓ Calico CNI configuration ready             
INFO[0053]                                              
INFO[0053] === Installing CoreDNS ===                   
[addons] Applied essential addon: CoreDNS
INFO[0054]                                              
INFO[0054] === Patching CoreDNS for cluster.local fallthrough === 
INFO[0054]                                              
INFO[0054] === Patching CoreDNS for CRI-O compatibility === 
INFO[0054]                                              
INFO[0054] ✅ Cluster initialized on controller with Calico CNI 
INFO[0054]                                              
INFO[0054] Step 8: Creating HAProxy load balancer...    
INFO[0088]                                              
INFO[0088] ✅ Cluster created successfully!              
INFO[0088]                                              
INFO[0088] Local registry:                              
INFO[0088]   Push:  podman push --tls-verify=false localhost:5000/<image>:<tag> 
INFO[0088]   Pull (in-cluster): registry.cluster.local:5000/<image>:<tag> 
INFO[0088]                                              
INFO[0088] Step 9: Exposing API server...               
INFO[0088] === Exposing API server to localhost:43851 via HAProxy === 
INFO[0088]                                              
INFO[0088] Using control-plane node controller to fetch kubeconfig 
INFO[0088] API server exposed: localhost:43851 -> HAProxy -> control-plane nodes:6443 
INFO[0088]                                              
INFO[0088] Generating kubeconfig at ./kubeconfig-e2e... 
export KUBECONFIG=/home/pragyan/RedHat/bootc-operator/kubeconfig-e2e
INFO[0088] Kubeconfig generated at ./kubeconfig-e2e     
INFO[0088]    Server URL: https://localhost:43851       
INFO[0088]                                              
INFO[0088] Usage:                                       
INFO[0088]   eval "$(bink api expose)"                  
INFO[0088]   kubectl get nodes                          
INFO[0088]                                              
INFO[0088]                                              
INFO[0088] To add worker nodes:                         
INFO[0088]   bink node add node2                        
kubectl --kubeconfig ./kubeconfig-e2e wait --for=condition=Ready node/controller --timeout=5m
node/controller condition met
STEP 1/2: FROM localhost:5000/node:latest
STEP 2/2: RUN touch /usr/share/update-marker
--> Using cache 7d3b1b5483771492fe7c7d86411df778ec06dea6b07e642599a6a9d693c4da0d
COMMIT localhost:5000/node:update
--> 7d3b1b548377
Successfully tagged localhost:5000/node:update
7d3b1b5483771492fe7c7d86411df778ec06dea6b07e642599a6a9d693c4da0d
podman push --tls-verify=false localhost:5000/node:update
Getting image source signatures
Copying blob 38bdfd546a3f skipped: already exists  
Copying blob bd9ddc54bea9 skipped: already exists  
Copying blob c70b8a378ec2 skipped: already exists  
Copying blob 5735cdb2999a skipped: already exists  
Copying blob e4331430029c skipped: already exists  
Copying config 7d3b1b5483 done   | 
Writing manifest to image destination
podman push --tls-verify=false bootc-operator:dev localhost:5000/bootc-operator-e2e:latest
Error: bootc-operator:dev: image not known
make: *** [Makefile:151: deploy-bink] Error 125

[Johan-Liebert1]

So, running make buildimg first did the trick. Maybe buildimg should be a dependency to deploy-bink in the Makefile?

[jlebon]

So, running make buildimg first did the trick. Maybe buildimg should be a dependency to deploy-bink in the Makefile?

Right, this is mentioned as a soft dependency in

bootc-operator/Makefile

Line 150 in da998a0

deploy-bink: start-bink build-update-image kustomize ## Deploy to a bink cluster (requires: buildimg).

.

Not strongly against making it a hard dep, but it also felt weird that e.g. make teardown-bink && make deploy-bink to get a fresh cluster also would trigger a rebuild.