Kubernetes pods percona-galera-0 percona-galera-slave-0 (MySQL.yaml ) with CrashLoopBackOff status

0 votes

I’m trying to implement Kubernetes pod and its status is always CrashLoopBackOff  and iam delete the pod and restart ,again  show  CrashLoopBackOff 

how to fix crashloopbackoff kubernetes pod and contine restarting pod 

Please find  below details please tell me the issue 

telpsg@kubems:~$ kubectl get pod -o wide | grep -i percona-galera
percona-galera-0                                  0/1     CrashLoopBackOff   39         3h22m   10.244.4.44   node4   <none>           <none>
percona-galera-slave-0                            0/1     Init:0/1           0          3h30m   10.244.5.25   node5   <none>           <none>


telpsg@kubems:~$ kubectl logs -f  percona-galera-0
+ . /functions.sh
++ awk ' { print $1 } '
++ hostname -i
I AM percona-galera-0 - 10.244.4.41
+ ipaddr=10.244.4.41
++ hostname
+ hostname=percona-galera-0
+ echo 'I AM percona-galera-0 - 10.244.4.41'
+ '[' '' = - ']'
++ resolveip -s percona-galera
+ cluster_join=10.106.155.206
+ [[ -z 10.106.155.206 ]]
+ echo 'I am not the Primary Node'
I am not the Primary Node
++ resolveip -s percona-galera-0.percona-galera.default.svc.cluster.local
+ master_ip=10.244.4.41
+ write_password_file
+ [[ -n root@123 ]]
+ cat
+ exec mysqld --user=mysql --wsrep_cluster_name=percona-galera --wsrep_node_name=percona-galera-0 --wsrep_cluster_address=gcomm://10.244.4.41 --wsrep_sst_method=xtrabackup-v2 --wsrep_sst_auth=xtrabackup: --wsrep_node_address=10.244.4.41
telpsg@kubems:~$ cd /var/tmp
telpsg@kubems:/var/tmp$ cd TMLDeployment_sprint_8d/


This is my yaml file: 

telpsg@kubems:/var/tmp/TMLDeployment_sprint_8d$ cat mysql.yaml
#MySQL Begin

kind: PersistentVolume
apiVersion: v1
metadata:
  name: pvmysql
  labels:
    type: local
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/var/tmp"
  claimRef:
    namespace: default
    name: mysqlclaim
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysqlclaim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Mi
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  labels:
    app: percona-galera
  name: percona-galera
spec:
  replicas: 1
  selector:
    matchLabels:
      app: percona-galera
  serviceName: percona-galera
  template:
    metadata:
      labels:
        app: percona-galera
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: "kubernetes.io/hostname"
                operator: In
                values:
                - node4
      initContainers:
      - name: init-percona-galera
        image:  kubems:5000/alpine
        command: ["/bin/sh","-c"]
        args: ["sed -i -e 's/.*safe_to_bootstrap: 0.*/safe_to_bootstrap: 1/' /var/lib/mysql/grastate.dat; exit 0"]
        volumeMounts:
        - mountPath: "/var/lib/mysql"
          name: mysqlstorage
          subPath: mysql/
      containers:
      - name: percona-galera
        image: kubems:5000/percona
        env:
        - name: CLUSTER_NAME
          value: percona-galera
        - name: MYSQL_ROOT_PASSWORD
          value: root@123
        - name: MYSQL_DATABASE
          value: cpi
        - name: MYSQL_USER
          value: cpi
        - name: MYSQL_PASSWORD
          value: root@123
        - name: K8S_SERVICE_NAME
          value: percona-galera
        - name: LOG_TO_STDOUT
          value: "true"
        - name: DEBUG
          value: "true"
        - name: MYSQL_NO_ENFORCE
          value: "true"
        livenessProbe:
          exec:
            command: ["mysqladmin", "-p$(MYSQL_ROOT_PASSWORD)", "ping"]
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command: ["mysql", "-h", "127.0.0.1", "-p$(MYSQL_ROOT_PASSWORD)", "-e", "SELECT 1;","mysql","-h","127.0.0.1", "-p$(MYSQL_ROOT_PASSWORD)","-e","SET GLOBAL max_connections=20000;"]
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
        ports:
          - containerPort: 4567
          - containerPort: 4568
          - containerPort: 4444
          - containerPort: 3306
        volumeMounts:
        - mountPath: /docker-entrypoint-initdb.d/cpi.sql
          name: mysqlinitdb
        - mountPath: /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf
          name: mysqldcnf
        - mountPath: "/var/lib/mysql"
          name: mysqlstorage
          subPath: mysql/
        - name: tz-config
          mountPath: /etc/localtime
      volumes:
      - name: mysqlinitdb
        hostPath:
          path: /var/tmp/cpi.sql
          type: File
      - name: mysqldcnf
        hostPath:
          path: /var/tmp/mysqld.cnf
          type: File
      - name: mysqlstorage
        persistentVolumeClaim:
          claimName: mysqlclaim
      - name: tz-config
        hostPath:
          path: /usr/share/zoneinfo/Asia/Kolkata
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: percona-galera
  name: percona-galera
spec:
  ports:
  - name: galera-replication
    port: 4567
  - name: state-transfer
    port: 4568
  - name: state-snapshot
    port: 4444
  - name: mysql
    port: 3306
  selector:
    app: percona-galera
  type: ClusterIP
---
apiVersion: apps/v1beta2
kind: StatefulSet
metadata:
  labels:
    app: percona-galera-slave
  name: percona-galera-slave
spec:
  replicas: 2
  selector:
    matchLabels:
      app: percona-galera-slave
  serviceName: percona-galera-slave
  template:
    metadata:
      labels:
        app: percona-galera-slave
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - percona-galera
                    - percona-galera-slave
              topologyKey: "kubernetes.io/hostname"
      initContainers:
      - name: init-percona-galera
        image:  kubems:5000/alpine
        command: ["sh", "-c", "until nc -zvw1 percona-galera 4567; do sleep 3; done; sleep 60; exit 0"]
      containers:
      - name: percona-galera-slave
        image: kubems:5000/percona
        env:
        - name: CLUSTER_NAME
          value: percona-galera
        - name: MYSQL_ROOT_PASSWORD
          value: root@123
        - name: MYSQL_DATABASE
          value: cpi
        - name: MYSQL_USER
          value: cpi
        - name: MYSQL_PASSWORD
          value: root@123
        - name: K8S_SERVICE_NAME
          value: percona-galera
        - name: LOG_TO_STDOUT
          value: "true"
        - name: DEBUG
          value: "true"
        - name: MYSQL_NO_ENFORCE
          value: "true"
        livenessProbe:
          exec:
            command: ["mysqladmin", "-p$(MYSQL_ROOT_PASSWORD)", "ping"]
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command: ["mysql", "-h", "127.0.0.1", "-p$(MYSQL_ROOT_PASSWORD)", "-e", "SELECT 1;","mysql","-h","127.0.0.1", "-p$(MYSQL_ROOT_PASSWORD)","-e","SET GLOBAL max_connections=20000;"]
          initialDelaySeconds: 60
          periodSeconds: 10
          timeoutSeconds: 5
        ports:
          - containerPort: 4567
          - containerPort: 4568
          - containerPort: 4444
          - containerPort: 3306
        volumeMounts:
        - mountPath: /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf
          name: mysqldcnf
        - name: tz-config
          mountPath: /etc/localtime
      volumes:
      - name: mysqldcnf
        hostPath:
          path: /var/tmp/mysqld.cnf
          type: File
      - name: tz-config
        hostPath:
          path: /usr/share/zoneinfo/Asia/Kolkata
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: percona-galera-slave
  name: percona-galera-slave
spec:
  ports:
  - name: galera-replication
    port: 4567
  - name: state-transfer
    port: 4568
  - name: state-snapshot
    port: 4444
  - name: mysql
    port: 3306
  selector:
    app: percona-galera-slave
  type: ClusterIP
---


Describe pod 


telpsg@kubems:~$ kubectl describe pod percona-galera-0
Name:               percona-galera-0
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               node4/192.168.0.18
Start Time:         Thu, 05 Mar 2020 13:15:26 +0530
Labels:             app=percona-galera
                    controller-revision-hash=percona-galera-7775f48c96
                    statefulset.kubernetes.io/pod-name=percona-galera-0
Annotations:        <none>
Status:             Running
IP:                 10.244.4.44
Controlled By:      StatefulSet/percona-galera
Init Containers:
  init-percona-galera:
    Container ID:  docker://ec0438962fa2edf624818e7220ee8ca4a3e5eeea89279575012d8015e866e3d4
    Image:         kubems:5000/alpine
    Image ID:      docker-pullable://kubems:5000/alpine@sha256:02892826401a9d18f0ea01f8a2f35d328ef039db4e1edcc45c630314a0457d5b
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      sed -i -e 's/.*safe_to_bootstrap: 0.*/safe_to_bootstrap: 1/' /var/lib/mysql/grastate.dat; exit 0
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 05 Mar 2020 13:15:30 +0530
      Finished:     Thu, 05 Mar 2020 13:15:30 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/lib/mysql from mysqlstorage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7wt9z (ro)
Containers:
  percona-galera:
    Container ID:   docker://76780bd35aa650d9fb3d6603d8fcf425c9bf8d7a00d81a4d6aaa2beef8b1a415
    Image:          kubems:5000/percona
    Image ID:       docker-pullable://kubems:5000/percona@sha256:65897eacc03f1d1904e8cef120c56efc4dd3297b813d4b0f17644311cd47221b
    Ports:          4567/TCP, 4568/TCP, 4444/TCP, 3306/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 05 Mar 2020 16:31:07 +0530
      Finished:     Thu, 05 Mar 2020 16:31:45 +0530
    Ready:          False
    Restart Count:  38
    Liveness:       exec [mysqladmin -p$(MYSQL_ROOT_PASSWORD) ping] delay=60s timeout=5s period=10s #success=1 #failure=3
    Readiness:      exec [mysql -h 127.0.0.1 -p$(MYSQL_ROOT_PASSWORD) -e SELECT 1; mysql -h 127.0.0.1 -p$(MYSQL_ROOT_PASSWORD) -e SET GLOBAL max_connections=20000;] delay=60s timeout=5s period=10s #success=1 #failure=3
    Environment:
      CLUSTER_NAME:         percona-galera
      MYSQL_ROOT_PASSWORD:  root@123
      MYSQL_DATABASE:       cpi
      MYSQL_USER:           cpi
      MYSQL_PASSWORD:       root@123
      K8S_SERVICE_NAME:     percona-galera
      LOG_TO_STDOUT:        true
      DEBUG:                true
      MYSQL_NO_ENFORCE:     true
    Mounts:
      /docker-entrypoint-initdb.d/cpi.sql from mysqlinitdb (rw)
      /etc/localtime from tz-config (rw)
      /etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf from mysqldcnf (rw)
      /var/lib/mysql from mysqlstorage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7wt9z (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  mysqlinitdb:
    Type:          HostPath (bare host directory volume)
    Path:          /var/tmp/cpi.sql
    HostPathType:  File
  mysqldcnf:
    Type:          HostPath (bare host directory volume)
    Path:          /var/tmp/mysqld.cnf
    HostPathType:  File
  mysqlstorage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysqlclaim
    ReadOnly:   false
  tz-config:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/share/zoneinfo/Asia/Kolkata
    HostPathType:
  default-token-7wt9z:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7wt9z
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                    From            Message
  ----     ------   ----                   ----            -------
  Warning  BackOff  40s (x817 over 3h19m)  kubelet, node4  Back-off restarting failed container





Mar 5 in DevOps & Agile by Sarfaraz
• 120 points

recategorized Mar 6 by Gitika 331 views
Have a look at how much memory is allocated to the pod at initialisation and have a look at how much memory your service is swallowing up, the pod should be higher than the service. So that's where I'd start.
Try to do a describe on that pod and chck the issue.....you will get a better clarity

2 answers to this question.

0 votes
Check the description part of pod y its crashing and then logs...This is the only way to figure out issue..
answered Mar 17 by Nitin Nagpal
0 votes

Run kubectl describe podname and check the reason of crash

answered Mar 17 by Manoj Gupta

Related Questions In DevOps & Agile

0 votes
1 answer
+4 votes
2 answers

run docker-daemon with --iptables=false

The docker network model uses iptables to ...READ MORE

answered Aug 2, 2018 in DevOps & Agile by Kalgi
• 51,890 points
1,358 views
0 votes
1 answer

How do I use Devops with Xamarin and hockeyapp

It would be better if you use ...READ MORE

answered Mar 26, 2018 in DevOps & Agile by Atul
• 10,240 points
123 views
+5 votes
3 answers

Error while trying to install postgis extension with greenplum

Try re downloading the package and installing ...READ MORE

answered Mar 27, 2018 in DevOps & Agile by DragonLord999
• 8,420 points
197 views
0 votes
1 answer

Web Service with WebSocket Implementation on AWS Blue-green deployment

Try connection draining. It will send all ...READ MORE

answered Apr 2, 2018 in DevOps & Agile by DareDev
• 6,850 points
229 views
+1 vote
3 answers

How can I connect an app with an aws ec2 instance?

This is how you deploy a web ...READ MORE

answered Oct 18, 2018 in DevOps & Agile by Hannah
• 18,060 points
113 views
0 votes
1 answer
0 votes
3 answers

Error while joining cluster with node

Hi Kalgi after following above steps it ...READ MORE

answered Jan 17, 2019 in Others by anonymous
7,201 views
+4 votes
1 answer

Installing Web UI (Dashboard):kubernetes-dashboard on main Ubuntu 16.04.6 LTS (Xenial Xerus) server

Follow these steps: $ kubeadm reset $ kubeadm init ...READ MORE

answered Apr 11, 2019 in Kubernetes by Kalgi
• 51,890 points

reshown Apr 12, 2019 by Kalgi 2,123 views