Disclaimer: This issue is specific for the Azure cloud. With different cloud provider or on-prem, the solution will be different.

Whatever you do, basic command with the kubectl, you all get this error message:

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-11-03T10:34:57+08:00 is after 2022-08-27T08:37:55Z

You can further check the issue with this

$ kubectl get secret remote-certs -o json | jq -r '.data | ."remote.ca.crt"' | base64 -d | openssl x509 -noout -text | grep -A 2 -i validity

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-11-03T10:38:31+08:00 is after 2022-08-27T08:37:55Z
unable to load certificate
140116703741248:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE

What to do?

• First you need to check the API endpoint of your cluster
• Then you run the command to rotate the certificate
• Get the new certificate by reconnecting to your cluster

1. Check the endpoint of your cluster:

You can check by reading the ~/.kube/config and check the part of the server, it should have something similar to this

https://<your-cluster-name-id>.southeastasia.azmk8s.io:443

2. Run command to manually rotate the certificate. Refer from Azure official document page, this source

Use az aks get-credentials to sign in to your AKS cluster. This command also downloads and configures the kubectl client certificate on your local machine.

Azure CLICopy

az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME

Use az aks rotate-certs to rotate all certificates, CAs, and SAs on your cluster.

Azure CLICopy

az aks rotate-certs -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME

 Important

It may take up to 30 minutes for az aks rotate-certs to complete. If the command fails before completing, use az aks show to verify the status of the cluster is Certificate Rotating. If the cluster is in a failed state, rerun az aks rotate-certs to rotate your certificates again.

3. Verify the certificate again

az aks get-credentials -g $RESOURCE_GROUP_NAME -n $CLUSTER_NAME --overwrite-existing

Your application (service principle) expired.

Error or event:

🐬 $ kubectl get event -w --all-namespaces 

NAMESPACE       LAST SEEN   TYPE      REASON                 OBJECT                             MESSAGE
ingress-nginx   3m8s        Normal    EnsuringLoadBalancer   service/ingress-nginx-controller   Ensuring load balancer
ingress-nginx   3m8s        Warning   ListLoadBalancers      service/ingress-nginx-controller   (combined from similar events): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to http://localhost:7788/subscriptions/<subscriptionID>/resourceGroups/mc_kaldi-test_sgdecoding-batch-scaled_southeastasia/providers/Microsoft.Network/loadBalancers?api-version=2019-06-01: StatusCode=401 -- Original Error: adal: Refresh request failed. Status Code = '401'. Response body: {"error":"invalid_client","error_description":"AADSTS7000222: The provided client secret keys for app '<appID>' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds.\r\nTrace ID: e90dea03-fdfc-4823-a4a6-96b7fd330d00\r\nCorrelation ID: 99420313-023b-4c86-acc6-42c32048736b\r\nTimestamp: 2022-11-04 02:30:31Z","error_codes":[7000222],"timestamp":"2022-11-04 02:30:31Z","trace_id":"e90dea03-fdfc-4823-a4a6-96b7fd330d00","correlation_id":"99420313-023b-4c86-acc6-42c32048736b","error_uri":"https://login.microsoftonline.com/error?code=7000222"}

Steps (worked for me) to resolve this issue

1. Follow the guide at https://learn.microsoft.com/en-gb/azure/active-directory/develop/quickstart-register-app#add-a-client-secret to add a client secret =>
   Take note of the client secret value! (SECRET_VALUE)
   Also check the application name! (APPLICATION_NAME)
2. Remove the expired secret
3. Update the cluster secret

az ad sp list --filter "displayname eq APPLICATION_NAME" 
-> Check the APP_ID
export SP_ID=APP_ID
export SP_SECRET=SECRET_VALUE
az aks update-credentials \
    --resource-group <RESOURCE_GROUP> \
    --name <CLUSTER_NAME> \
    --reset-service-principal \
    --service-principal $SP_ID \
    --client-secret $SP_SECRET

After that, you should able to do all the commands and get the correct values

kubectl get pods
kubectl get certificates
kubectl get clusterissuer
kubectl get nodes
kubectl exec -it <pod> -- bash
kubectl get event --all-namespaces

End.