Kubernetes ingress kepps returning 502 server error

Question

I am trying to setup an Ingress in GCE Kubernetes. But when I visit the IP address and path combination defined in the Ingress, I keep getting the following 502 error:

Error:Server error
The server encountered a temporary error that could not complete your request.
Please try again in 30 seconds

Here is what I get when I run: kubectl describe ing --namespace dpl-staging

Name:           dpl-identity
Namespace:      dpl-staging
Address:        35.186.221.153
Default backend:    default-http-backend:80 (10.0.8.5:8080)
TLS:
  dpl-identity terminates
Rules:
  Host  Path    Backends
  ----  ----    --------
  *
        /api/identity/*     dpl-identity:4000 (<none>)
Annotations:
  https-forwarding-rule:    k8s-fws-dpl-staging-dpl-identity--5fc40252fadea594
  https-target-proxy:       k8s-tps-dpl-staging-dpl-identity--5fc40252fadea594
  url-map:          k8s-um-dpl-staging-dpl-identity--5fc40252fadea594
  backends:         {"k8s-be-31962--5fc40252fadea594":"HEALTHY","k8s-be-32396--5fc40252fadea594":"UNHEALTHY"}
Events:
  FirstSeen LastSeen    Count   From                SubObjectPath   Type        Reason  Message
  --------- --------    -----   ----                -------------   --------    ------  -------
  15m       15m     1   {loadbalancer-controller }          Normal      ADD dpl-staging/dpl-identity
  15m       15m     1   {loadbalancer-controller }          Normal      CREATE  ip: 35.186.221.153
  15m       6m      4   {loadbalancer-controller }          Normal      Service no user specified default backend, using system default

Here is my service description: kubectl describe svc --namespace dpl-staging

Name:           dpl-identity
Namespace:      dpl-staging
Labels:         app=dpl-identity
Selector:       app=dpl-identity
Type:           NodePort
IP:             10.3.254.194
Port:           http    4000/TCP
NodePort:       http    32396/TCP
Endpoints:      10.0.2.29:8000,10.0.2.30:8000
Session Affinity:   None
No events.

Also, here is the result of executing: kubectl describe ep -n dpl-staging dpl-identity

Name:       dpl-identity
Namespace:  dpl-staging
Labels:     app=dpl-identity
Subsets:
  Addresses:        10.0.2.29,10.0.2.30
  NotReadyAddresses:    <none>
  Ports:
    Name    Port    Protocol
    ----    ----    --------
    http    8000    TCP

No events.

---

Here is my deployment.yaml:

apiVersion: v1
kind: Secret
metadata:
  namespace: dpl-staging
  name: dpl-identity
type: Opaque
data:
  tls.key: <base64 key>
  tls.crt: <base64 crt>
---
apiVersion: v1
kind: Service
metadata:
  namespace: dpl-staging
  name: dpl-identity
  labels:
    app: dpl-identity
spec:
  type: NodePort
  ports:
    - port: 4000
      targetPort: 8000
      protocol: TCP
      name: http
  selector:
    app: dpl-identity
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: dpl-staging
  name: dpl-identity
  labels:
    app: dpl-identity
  annotations:
    kubernetes.io/ingress.allow-http: "false"
spec:
  tls:
  - secretName: dpl-identity
  rules:
  - http:
      paths:
        - path: /api/identity/*
          backend:
            serviceName: dpl-identity
            servicePort: 4000
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: dpl-staging
  name: dpl-identity
kind: Ingress
metadata:
  namespace: dpl-staging
  name: dpl-identity
  labels:
    app: dpl-identity
  annotations:
    kubernetes.io/ingress.allow-http: "false"
spec:
  tls:
  - secretName: dpl-identity
  rules:
  - http:
      paths:
        - path: /api/identity/*
          backend:
            serviceName: dpl-identity
            servicePort: 4000
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  namespace: dpl-staging
  name: dpl-identity
  labels:
    app: dpl-identity
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: dpl-identity
    spec:
      containers:
      - image: gcr.io/munpat-container-engine/dpl/identity:0.4.9
        name: dpl-identity
        ports:
        - containerPort: 8000
          name: http
        volumeMounts:
        - name: dpl-identity
          mountPath: /data
      volumes:
      - name: dpl-identity
        secret:
          secretName: dpl-identity

Answer 1

Your backend k8s-be-32396--5fc40252fadea594 is showing unhealthy. Thtas exactly when you get the 502 server error.  Ingress doesn not forward the traffic if the backend is unhealthy. Its failing some health checks. 

Check the health check setting and check if all the parameters are suitable for your pod.

You can get the health check setting here: Compute engine -> Health check.

If all the parameters seem fine then there might be something about the traffic that is sent from browser to the container. You can check that using the following command:

kubectl exec -it PODID -- bash

and then curl localhost to check if the container is responding as per expected.

Try all of these and let us know.

For further details, refer to the Kubernetes Training.

Answer 2

Your endpoint does not return Success and that's the issue here. Health checks are failing. HTTP/HTTPS load balancer will not send the request to the cluster node if these health checks keep failing. 

I created an endpoint that always returns 200 OK. Once the health checks start passing, your load balancer also would start without any errors.

Answer 3

I totally agree with @Kalgi's and @Girish's answers. I would like to add something to it. If you do not want to make changes to your current container, you can also create another container and mention the endpoint there. This way you'll have your initial workaround safeguarded.

Answer 4

I had a similar error, wasted a lot of time behind it. In my case, the problem was that I did not have a web server running on my instance to handle the network requests. Nginx solved the problem!

Answer 5

I have added "/" endpoints in each service with 200 response and it did the trick.