Nightmare started on the Monday, one day before the important demo: All of the sudden, all the services that I deploy on the kubernetes not able to reach

All of the services deployed inside the private cluster, and incoming/outgoing traffics through the Firewall. I tried to ping the firewall public IP, the same happened.

I used kubectl port-forward to test the services, everything seems fine. But the nginx pod in the nginx ingress controller return the error 404, but /healthz path returned 200. I didn’t thought much about it.

Then checking the infrastructure further, I noticed the kubernetes API upgraded from 1.23.x to 1.24.x few days ago, the same time with all the resources were recreated, and nginx pods failed.

After much researching all other aspects, I found this and it saved my day!

• https://serverfault.com/questions/1110280/kubernetes-v1-24-3-upgrade-broke-ingress-nginx

• https://github.com/Azure/AKS/issues/3210

Meaning adding this to the annotations:

service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz

Voila! The services are coming back <3