Kubernetes pods percona-galera-0 percona-galera-slave-0 MySQL yaml with CrashLoopBackOff status

Question

I’m trying to implement Kubernetes pod and its status is always CrashLoopBackOff  and iam delete the pod and restart ,again  show  CrashLoopBackOff 

how to fix crashloopbackoff kubernetes pod and contine restarting pod 

Please find  below details please tell me the issue 

telpsg@kubems:~$ kubectl get pod -o wide | grep -i percona-galera
percona-galera-0                                  0/1     CrashLoopBackOff   39         3h22m   10.244.4.44   node4   <none>           <none>
percona-galera-slave-0                            0/1     Init:0/1           0          3h30m   10.244.5.25   node5   <none>           <none>


telpsg@kubems:~$ kubectl logs -f  percona-galera-0
+ . /functions.sh
++ awk ' { print $1 } '
++ hostname -i
I AM percona-galera-0 - 10.244.4.41
+ ipaddr=10.244.4.41
++ hostname
+ hostname=percona-galera-0
+ echo 'I AM percona-galera-0 - 10.244.4.41'
+ '[' '' = - ']'
++ resolveip -s percona-galera
+ cluster_join=10.106.155.206
+ [[ -z 10.106.155.206 ]]
+ echo 'I am not the Primary Node'
I am not the Primary Node
++ resolveip -s percona-galera-0.percona-galera.default.svc.cluster.local
+ master_ip=10.244.4.41
+ write_password_file
+ [[ -n root@123 ]]
+ cat
+ exec mysqld --user=mysql --wsrep_cluster_name=percona-galera --wsrep_node_name=percona-galera-0 --wsrep_cluster_address=gcomm://10.244.4.41 --wsrep_sst_method=xtrabackup-v2 --wsrep_sst_auth=xtrabackup: --wsrep_node_address=10.244.4.41
telpsg@kubems:~$ cd /var/tmp
telpsg@kubems:/var/tmp$ cd TMLDeployment_sprint_8d/


This is my yaml file: 

telpsg@kubems:/var/tmp/TMLDeployment_sprint_8d$ cat mysql.yaml
#MySQL Begin

kind: PersistentVolume
apiVersion: v1
metadata:
  name: pvmysql
  labels:
    type: local
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/var/tmp"
  claimRef:
    namespace: default
    name: mysqlclaim
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysqlclaim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Mi
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  labels:
    app: percona-galera
  name: percona-galera
spec:
  replicas: 1
  selector:
    matchLabels:
      app: percona-galera
  serviceName: percona-galera
  template:
    metadata:
      labels:
        app: percona-galera
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "kubernetes.io/hostname"
                operator: In
                values:
                - node4
      initContainers:
      - name: init-percona-galera
        image:  kubems:5000/alpine
        command: ["/bin/sh","-c"]
        args: ["sed -i -e 's/.*safe_to_bootstrap: 0.*/safe_to_bootstrap: 1/' /var/lib/mysql/grastate.dat; exit 0"]
        volumeMounts:
        - mountPath: "/var/lib/mysql"
          name: mysqlstorage
          subPath: mysql/
      containers:
      - name: percona-galera
        image: kubems:5000/percona
        env:
        - name: CLUSTER_NAME
          value: percona-galera
        - name: MYSQL_ROOT_PASSWORD
          value: root@123
        - name: MYSQL_DATABASE
          value: cpi
        - name: MYSQL_USER
          value: cpi
        - name: MYSQL_PASSWORD
          value: root@123
        - name: K8S_SERVICE_NAME
          value: percona-galera
        - name: LOG_TO_STDOUT
          value: "true"
        - name: DEBUG
          value: "true"
        - name: MYSQL_NO_ENFORCE
          value: "true"
        livenessProbe:
          exec:
            command: ["mysqladmin", "-p$(MYSQL_ROOT_PASSWORD)", "ping"]
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command: ["mysql", "-h", "127.0.0.1", "-p$(MYSQL_ROOT_PASSWORD)", "-e", "SELECT 1;","mysql","-h","127.0.0.1", "-p$(MYSQL_ROOT_PASSWORD)","-e","SET GLOBAL max_connections=20000;"]
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
        ports:
          - containerPort: 4567
          - containerPort: 4568
          - containerPort: 4444
          - containerPort: 3306
        volumeMounts:
        - mountPath: /docker-entrypoint-initdb.d/cpi.sql
          name: mysqlinitdb
        - mountPath: /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf
          name: mysqldcnf
        - mountPath: "/var/lib/mysql"
          name: mysqlstorage
          subPath: mysql/
        - name: tz-config
          mountPath: /etc/localtime
      volumes:
      - name: mysqlinitdb
        hostPath:
          path: /var/tmp/cpi.sql
          type: File
      - name: mysqldcnf
        hostPath:
          path: /var/tmp/mysqld.cnf
          type: File
      - name: mysqlstorage
        persistentVolumeClaim:
          claimName: mysqlclaim
      - name: tz-config
        hostPath:
          path: /usr/share/zoneinfo/Asia/Kolkata
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: percona-galera
  name: percona-galera
spec:
  ports:
  - name: galera-replication
    port: 4567
  - name: state-transfer
    port: 4568
  - name: state-snapshot
    port: 4444
  - name: mysql
    port: 3306
  selector:
    app: percona-galera
  type: ClusterIP
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  labels:
    app: percona-galera-slave
  name: percona-galera-slave
spec:
  replicas: 2
  selector:
    matchLabels:
      app: percona-galera-slave
  serviceName: percona-galera-slave
  template:
    metadata:
      labels:
        app: percona-galera-slave
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - percona-galera
                    - percona-galera-slave
              topologyKey: "kubernetes.io/hostname"
      initContainers:
      - name: init-percona-galera
        image:  kubems:5000/alpine
        command: ["sh", "-c", "until nc -zvw1 percona-galera 4567; do sleep 3; done; sleep 60; exit 0"]
      containers:
      - name: percona-galera-slave
        image: kubems:5000/percona
        env:
        - name: CLUSTER_NAME
          value: percona-galera
        - name: MYSQL_ROOT_PASSWORD
          value: root@123
        - name: MYSQL_DATABASE
          value: cpi
        - name: MYSQL_USER
          value: cpi
        - name: MYSQL_PASSWORD
          value: root@123
        - name: K8S_SERVICE_NAME
          value: percona-galera
        - name: LOG_TO_STDOUT
          value: "true"
        - name: DEBUG
          value: "true"
        - name: MYSQL_NO_ENFORCE
          value: "true"
        livenessProbe:
          exec:
            command: ["mysqladmin", "-p$(MYSQL_ROOT_PASSWORD)", "ping"]
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command: ["mysql", "-h", "127.0.0.1", "-p$(MYSQL_ROOT_PASSWORD)", "-e", "SELECT 1;","mysql","-h","127.0.0.1", "-p$(MYSQL_ROOT_PASSWORD)","-e","SET GLOBAL max_connections=20000;"]
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
        ports:
          - containerPort: 4567
          - containerPort: 4568
          - containerPort: 4444
          - containerPort: 3306
        volumeMounts:
        - mountPath: /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf
          name: mysqldcnf
        - name: tz-config
          mountPath: /etc/localtime
      volumes:
      - name: mysqldcnf
        hostPath:
          path: /var/tmp/mysqld.cnf
          type: File
      - name: tz-config
        hostPath:
          path: /usr/share/zoneinfo/Asia/Kolkata
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: percona-galera-slave
  name: percona-galera-slave
spec:
  ports:
  - name: galera-replication
    port: 4567
  - name: state-transfer
    port: 4568
  - name: state-snapshot
    port: 4444
  - name: mysql
    port: 3306
  selector:
    app: percona-galera-slave
  type: ClusterIP
---


Describe pod 


telpsg@kubems:~$ kubectl describe pod percona-galera-0
Name:               percona-galera-0
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               node4/192.168.0.18
Start Time:         Thu, 05 Mar 2020 13:15:26 +0530
Labels:             app=percona-galera
                    controller-revision-hash=percona-galera-7775f48c96
                    statefulset.kubernetes.io/pod-name=percona-galera-0
Annotations:        <none>
Status:             Running
IP:                 10.244.4.44
Controlled By:      StatefulSet/percona-galera
Init Containers:
  init-percona-galera:
    Container ID:  docker://ec0438962fa2edf624818e7220ee8ca4a3e5eeea89279575012d8015e866e3d4
    Image:         kubems:5000/alpine
    Image ID:      docker-pullable://kubems:5000/alpine@sha256:02892826401a9d18f0ea01f8a2f35d328ef039db4e1edcc45c630314a0457d5b
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      sed -i -e 's/.*safe_to_bootstrap: 0.*/safe_to_bootstrap: 1/' /var/lib/mysql/grastate.dat; exit 0
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 05 Mar 2020 13:15:30 +0530
      Finished:     Thu, 05 Mar 2020 13:15:30 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/lib/mysql from mysqlstorage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7wt9z (ro)
Containers:
  percona-galera:
    Container ID:   docker://76780bd35aa650d9fb3d6603d8fcf425c9bf8d7a00d81a4d6aaa2beef8b1a415
    Image:          kubems:5000/percona
    Image ID:       docker-pullable://kubems:5000/percona@sha256:65897eacc03f1d1904e8cef120c56efc4dd3297b813d4b0f17644311cd47221b
    Ports:          4567/TCP, 4568/TCP, 4444/TCP, 3306/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 05 Mar 2020 16:31:07 +0530
      Finished:     Thu, 05 Mar 2020 16:31:45 +0530
    Ready:          False
    Restart Count:  38
    Liveness:       exec [mysqladmin -p$(MYSQL_ROOT_PASSWORD) ping] delay=60s timeout=5s period=10s #success=1 #failure=3
    Readiness:      exec [mysql -h 127.0.0.1 -p$(MYSQL_ROOT_PASSWORD) -e SELECT 1; mysql -h 127.0.0.1 -p$(MYSQL_ROOT_PASSWORD) -e SET GLOBAL max_connections=20000;] delay=60s timeout=5s period=10s #success=1 #failure=3
    Environment:
      CLUSTER_NAME:         percona-galera
      MYSQL_ROOT_PASSWORD:  root@123
      MYSQL_DATABASE:       cpi
      MYSQL_USER:           cpi
      MYSQL_PASSWORD:       root@123
      K8S_SERVICE_NAME:     percona-galera
      LOG_TO_STDOUT:        true
      DEBUG:                true
      MYSQL_NO_ENFORCE:     true
    Mounts:
      /docker-entrypoint-initdb.d/cpi.sql from mysqlinitdb (rw)
      /etc/localtime from tz-config (rw)
      /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf from mysqldcnf (rw)
      /var/lib/mysql from mysqlstorage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7wt9z (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  mysqlinitdb:
    Type:          HostPath (bare host directory volume)
    Path:          /var/tmp/cpi.sql
    HostPathType:  File
  mysqldcnf:
    Type:          HostPath (bare host directory volume)
    Path:          /var/tmp/mysqld.cnf
    HostPathType:  File
  mysqlstorage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysqlclaim
    ReadOnly:   false
  tz-config:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/share/zoneinfo/Asia/Kolkata
    HostPathType:
  default-token-7wt9z:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7wt9z
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                    From            Message
  ----     ------   ----                   ----            -------
  Warning  BackOff  40s (x817 over 3h19m)  kubelet, node4  Back-off restarting failed container

Comments

Have a look at how much memory is allocated to the pod at initialisation and have a look at how much memory your service is swallowing up, the pod should be higher than the service. So that's where I'd start.

---

Try to do a describe on that pod and chck the issue.....you will get a better clarity

Answer 1

Check the description part of pod y its crashing and then logs...This is the only way to figure out issue..

Answer 2

Run kubectl describe podname and check the reason of crash