Recently I followed the guideline to create a private cluster on Azure (private AKS) with terraform here, and all kubernetes resources are placed inside this private cluster. Due to some differences, I can’t follow the entire instruction, omitting the part of github actions, agent and Azure DevOps pipeline.

Beside, I have to add few things to fit into our system, parts that terraform can’t automate it right now

• (1) Update the Firewall policy to workaround the routing issue [also documented in the link]
• (2) Using nginx ingress and cert-manager, and your own hosting solution to expose services to the end-user.

Update 1:

The link over here explain why we need to do workaround

• The firewall is stateful (in layman term, it means it will monitor and track the network connection as a whole: if the incoming connection is allowed, the outgoing connection is allowed too. And if there is no incoming connection, the outgoing connection is denied too.)
• The internet connection goes to the private AKS -> go to Public IP of the LoadBalancer -> Private IP of the Firewall but not its Public IP
• The Firewall cannot see the incoming connection of the request, therefore, will drop the outgoing response from private AKS to the end-user @internet.

What do you need to do is adding the DNAT rule to the firewall policy, as demo in the screenshot

We also have to modify the default Route table, to allow connection from public IP of the firewall to the internet (so the Firewall can see both go in and out to the internet).

Update 2:

• When install nginx-ingress without internal LB, it will use another PIP for the ingress controller.
• All the k8s ingress resource should pick up the PIP of the Public LB (ingress controller). However, if it’s not, you can update the helm chart of the ingress controller, to allow them.
• You should also modify the application routing to allow connection to Lets Encrypt and cert-manager to verify the challenge later.

Update the ingress resource to use the public IP as the external IP

INGRESS_NAMESPACE=ingress-nginx
INGRESS_RELEASE=nginx-ingress
helm -n $INGRESS_NAMESPACE upgrade $INGRESS_RELEASE ingress-nginx/ingress-nginx --set controller.publishService.enabled=true

Modify the application routing in the Firewall Policy

In the hosting service, when adding A records, you should put the public IP address of the Firewall instead of the public IP of the LoadBalancer.

You can verify its exactly IP by command

dig <your-domain-name>

I have tried and deploy our services follow the instructions, adding our modifications and can publish our services to the end-user.

Tech stack used: terraform, helm chart, Az cli + plugins, Azure Portal, Ubuntu 20.04, kubernetes, kubctl, nginx ingress, cert-manager with Lets Encrypts.

End.