How to setup error reporting in Stackdriver from kubernetes pods?

0 votes

I'm a bit confused at how to setup error reporting in kubernetes, so errors are visible in Google Cloud Console / Stackdriver "Error Reporting"?

According to documentation https://cloud.google.com/error-reporting/docs/setting-up-on-compute-engine we need to enable fluentd' "forward input plugin" and then send exception data from our apps. I think this approach would have worked if we had setup fluentd ourselves, but it's already pre-installed on every node in a pod that just runs gcr.io/google_containers/fluentd-gcp docker image.

How do we enable forward input on those pods and make sure that http port available to every pod on the nodes? We also need to make sure this config is used by default when we add more nodes to our cluster.

Any help would be appreciated, may be I'm looking at all this from a wrong point?

Nov 13, 2018 in Kubernetes by Atul
• 5,530 points
83 views

1 answer to this question.

0 votes

The basic idea is to start a separate pod that receives structured logs over TCP and forwards it to Cloud Logging, similar to a locally-running fluentd agent. See below for the steps I used.

(Unfortunately, the logging support that is built into Docker and Kubernetes cannot be used - it just forwards individual lines of text from stdout/stderr as separate log entries which prevents Error Reporting from seeing complete stack traces.)

Create a docker image for a fluentd forwarder using a Dockerfile as follows:

FROM gcr.io/google_containers/fluentd-gcp:1.18

COPY fluentd-forwarder.conf /etc/google-fluentd/google-fluentd.conf

Where fluentd-forwarder.conf contains the following:

<source>
  type forward
  port 24224
</source>

<match **>
  type google_cloud
  buffer_chunk_limit 2M
  buffer_queue_limit 24
  flush_interval 5s
  max_retry_wait 30
  disable_retry_limit
</match>

Then build and push the image:

$ docker build -t gcr.io/###your project id###/fluentd-forwarder:v1 .
$ gcloud docker push gcr.io/###your project id###/fluentd-forwarder:v1

You need a replication controller (fluentd-forwarder-controller.yaml):

apiVersion: v1
kind: ReplicationController
metadata:
  name: fluentd-forwarder
spec:
  replicas: 1
  template:
    metadata:
      name: fluentd-forwarder
      labels:
        app: fluentd-forwarder
    spec:
      containers:
      - name: fluentd-forwarder
        image: gcr.io/###your project id###/fluentd-forwarder:v1
        env:
        - name: FLUENTD_ARGS
          value: -qq
        ports:
        - containerPort: 24224

You also need a service (fluentd-forwarder-service.yaml):

apiVersion: v1
kind: Service
metadata:
  name: fluentd-forwarder
spec:
  selector:
    app: fluentd-forwarder
  ports:
  - protocol: TCP
    port: 24224

Then create the replication controller and service:

$ kubectl create -f fluentd-forwarder-controller.yaml
$ kubectl create -f fluentd-forwarder-service.yaml

Finally, in your application, instead of using 'localhost' and 24224 to connect to the fluentd agent as described on https://cloud.google.com/error-reporting/docs/setting-up-on-compute-engine, use the values of evironment variables FLUENTD_FORWARDER_SERVICE_HOST and FLUENTD_FORWARDER_SERVICE_PORT.

answered Nov 13, 2018 by ajs3033
• 7,280 points

Related Questions In Kubernetes

0 votes
1 answer
0 votes
1 answer

Not able to access kubernetes api from a pod in azure

Follow these steps Add --bind-address=0.0.0.0 option to the line https://github.com/kubernetes/kubernetes/blob/v1.2.0/docs/getting-started-guides/coreos/azure/cloud_config_templates/kubernetes-cluster-main-nodes-template.yml#L218  Created ...READ MORE

answered Aug 30, 2018 in Kubernetes by Kalgi
• 40,440 points
57 views
0 votes
1 answer

How do I sign-in to kubernetes dashboard?

Since version 1.7 Dashboard uses more secure ...READ MORE

answered Sep 7, 2018 in Kubernetes by DareDev
• 6,810 points
136 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

permissions related to AWS ECR

if you add allowContainerRegistry: true, kops will add those permissions ...READ MORE

answered Oct 9, 2018 in Kubernetes by Kalgi
• 40,440 points
61 views
0 votes
3 answers

Error while joining cluster with node

Hi Kalgi after following above steps it ...READ MORE

answered Jan 17 in Others by anonymous
2,111 views
0 votes
1 answer

How to use gravitational teleport in a container/kubernetes environment?

You can use teleport to augment kubernetes ...READ MORE

answered Jun 28, 2018 in Kubernetes by ajs3033
• 7,280 points
320 views