Andrew Mercer
on this page

Check the address pool

kubectl get ipaddresspools.metallb.io -n metallb
NAME           AUTO ASSIGN   AVOID BUGGY IPS   ADDRESSES
default-pool   true          false             ["10.0.0.100-10.0.0.200"]

AUTO ASSIGN true means MetalLB assigns from this pool to any LoadBalancer service that doesn't ask for a specific pool. If a service isn't getting an IP:

kubectl get l2advertisements.metallb.io -n metallb    # must exist, or nothing is announced
kubectl describe svc <service>                        # MetalLB posts allocation events here
kubectl logs -n metallb deploy/metallb-controller

Node reports the wrong IP (Calico tunl0)

Symptom

kubectl get pods -n metallb -o wide
NAME                                  READY   STATUS    IP               NODE
metallb-controller-7bd5848b94-c826n   1/1     Running   10.244.158.207   cluster1
metallb-speaker-22l62                 4/4     Running   10.0.0.10        cluster0
metallb-speaker-4vvsw                 4/4     Running   10.0.0.12        cluster2
metallb-speaker-zq25m                 4/4     Running   10.0.0.11        cluster1

The controller's 10.244.x.x address is expected. The controller is an ordinary pod and gets a pod-network IP. The speakers run with hostNetwork: true, so they should show each node's LAN address, as they do here.

The real thing to check is the node's own reported address:

kubectl get nodes -o wide

If a node's INTERNAL-IP is wrong, or Calico picked the wrong interface for the node, the speaker announces from the wrong place and the service IPs are unreachable. On that node, Calico's IPIP tunnel interface carries an address from the pod range:

ip a s tunl0
4: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 10.244.158.192/32 scope global tunl0

Fix

1. Pin the kubelet's node IP to the LAN address. Use /etc/default/kubelet on Debian/Ubuntu and /etc/sysconfig/kubelet on RHEL/Alma:

KUBELET_EXTRA_ARGS=--node-ip=10.0.0.11

sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl get node cluster1 -o wide     # INTERNAL-IP should now be the LAN address

2. Pin Calico's address autodetection so it doesn't choose the wrong interface on multi-NIC hosts. In the operator's Installation resource:

kubectl edit installation default

spec:
  calicoNetwork:
    nodeAddressAutodetectionV4:
      kubernetes: NodeInternalIP     # follow the kubelet's --node-ip
      # or: interface: "eth0"  /  canReach: 10.0.0.1

For the other known Calico/MetalLB interactions, see https://metallb.universe.tf/configuration/calico/

Restart MetalLB

Use this after a config change that isn't being picked up, or to clear a stuck speaker:

kubectl rollout restart deployment metallb-controller -n metallb
kubectl rollout restart daemonset metallb-speaker -n metallb
kubectl get pods -n metallb -w

Alternatively, delete the pods outright. The Deployment and DaemonSet recreate them:

kubectl delete pods -n metallb -l app.kubernetes.io/name=metallb

If a speaker keeps crash-looping, read the crashed instance's logs before restarting it again:

kubectl logs -n metallb -l app.kubernetes.io/component=speaker --previous