Monitoring kubernetes job

0 votes
I have a kubernetes job that takes quite some time to finish the job, almost around 4 minutes in best case and 8 mins in worse case.

Instead of just ideally waiting for the job to be completed, I tried creating a code that alerts me when the job is completed.

It does the following:

Submits the kubernetes job.

Waits for its completion.

Checks whether the job has had the expected affect.

But the problem is, no matter how quickly the job is completed I have to wait for 8 minutes as the worse case input is 8 mins.

Is there a way to make this code more efficient?
Sep 3, 2018 in Kubernetes by Hannah
• 14,090 points
25 views

2 answers to this question.

0 votes
<kube master>/apis/batch/v1/namespaces/default/jobs 

endpoint lists status of the jobs. I have parsed this json and retrieved the name of the latest running job that starts with "deploy...".

Then we can hit

<kube master>/apis/batch/v1/namespaces/default/jobs/<job name retrieved above>

And monitor the status field value which is as below when the job succeeds

"status": {
    "conditions": [
      {
        "type": "Complete",
        "status": "True",
        "lastProbeTime": "2016-09-22T13:59:03Z",
        "lastTransitionTime": "2016-09-22T13:59:03Z"
      }
    ],
    "startTime": "2016-09-22T13:56:42Z",
    "completionTime": "2016-09-22T13:59:03Z",
    "succeeded": 1
  }

So we keep polling this endpoint till it completes.

answered Sep 3, 2018 by Kalgi
• 39,330 points
0 votes

you can do

watch 'kubectl get pods | grep <name of the pod>'

or

kubectl get pods -w

It will not be the full name of course as most of the time the pods get random names if you are running nginx replica or deployment your pods will end up with something like nginx-1696122428-ftjvy so you will want to do

watch 'kubectl get pods | grep nginx'

You can replace the pods with whatever job you are doing i.e (rc,svc,deployments....)

Even better you can use labels: watch 'kubectl get pods -l job=foobar' or kubectl get -w pods -l job=foobar

answered Sep 3, 2018 by Nilesh
• 6,900 points

Related Questions In Kubernetes

0 votes
1 answer

Kubernetes cron job call curl in loop

It's a small silly mistake: dollar sign $ should be ...READ MORE

answered Sep 17, 2018 in Kubernetes by Kalgi
• 39,330 points
873 views
0 votes
1 answer

Kubernetes - Job scheduling API

You will find the CronJobresource in the batch/v1beta1 API group. ...READ MORE

answered Sep 17, 2018 in Kubernetes by Kalgi
• 39,330 points
181 views
0 votes
3 answers

Change the schedule of Kubernetes cron job

kubectl patch <backup-cronjob> -p '{"spec":{"schedule": "0 0 ...READ MORE

answered Jun 20 in Kubernetes by sudhams reddy duba
324 views
0 votes
1 answer

Autodelete persistant volume claim when kubernetes job is completed

@Nilesh your idea seems really nice but ...READ MORE

answered Sep 19, 2018 in Kubernetes by Kalgi
• 39,330 points
99 views
0 votes
1 answer
0 votes
3 answers

Error while joining cluster with node

Hi Kalgi after following above steps it ...READ MORE

answered Jan 17 in Others by anonymous
1,680 views
+3 votes
1 answer
0 votes
1 answer

Spark job using kubernetes instead of yarn

I should be in the format of ...READ MORE

answered Sep 6, 2018 in Kubernetes by Kalgi
• 39,330 points
99 views
0 votes
1 answer

Connect to existing pod, execute script, disconnect - Kubernetes cron job

As far as i know there's no ...READ MORE

answered Sep 17, 2018 in Kubernetes by Kalgi
• 39,330 points
621 views