Kubernetes - Required ports for CertManager and Nginx Ingress controller
A gold mine of ports extracted from undocumented error hell
Published:
Heads up! This content is more than six months old. Take some time to verify everything still works as expected.
It's been a grueling, slightly inebriated, 12 hours. We (read: Java and Typescript developers) had a plan for getting Kubernetes running. Public clusters, wide open, CertManager and Nginx Ingress controllers. Then our real DevOps team got involved and pointed out the very important flaw in our plan, security.
That's the thing. All the best laid plans and 'OMG. I got it working!' from newbies (read: me) should be met with cheers. Kubernetes is hard. Harder and more important, is getting security right. I owe thanks Mark Kalusha and Greg Lettow for working with me on this.
Do the right thing, help your fellow DevOps teams out. Secure your cluster.
Error messages
You may be facing this problem in your own cluster if you're seeing any of the following errors:
- Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": context deadline exceeded
- Error: UPGRADE FAILED: failed to create resource: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": dial tcp 10.0.21.149:8443: i/o timeout
- Reason: Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://test..torch.ai/.well-known/acme-challenge/LtQsipkH5PE4p6NiT0lfur-7ljl4tLEHA9zpWEg6iv4': Get "http://test..torch.ai/.well-known/acme-challenge/LtQsipkH5PE4p6NiT0lfur-7ljl4tLEHA9zpWEg6iv4": context deadline exceeded (Client.Timeout exceeded while awaiting headers), Processing: true, Presented: true
- Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://test..torch.ai/.well-known/acme-challenge/LtQsipkH5PE4p6NiT0lfur-7ljl4tLEHA9zpWEg6iv4': Get "http://test..torch.ai/.well-known/acme-challenge/LtQsipkH5PE4p6NiT0lfur-7ljl4tLEHA9zpWEg6iv4": dial tcp 18.217.101.253:80: i/o timeout (Client.Timeout exceeded while awaiting headers), Processing: true, Presented: true
- State: invalid, Reason: Error accepting authorization: acme: authorization error for test.*.torch.ai: 400 urn:ietf:params:acme:error:dns: DNS problem: SERVFAIL looking up CAA for torch.ai the domain's nameservers may be malfunctioning, Processing: true, Presented: true
Ports required
You'll need to these ports added to your own security groups:
Ingress
- 80 - acme solver
- 443 - Standard communications
- 8443 - Nginx Ingress service internal lookups
- 9443 - K8s Ingress webhook
- 8089 - acme solver
Egress
- 80 - egress solver
- 443 - egress communications