How to send application logs from docker to cloudwatch

Question

We are using Kubernetes to deploy our application docker images.

We would like to be able to take the application logs and push it to cloudwatch.

The application logs are generated using log4j or log4js depending on language the microservice was built.

What is the right way to do this?

Answer 1

Build containers with the Cloudwatch Agent. To do this you will need a Dockerfile.

Make sure your base container is either Debian or RHEL based because Amazon docs seem to only support these types of distros with the agent.

Execute this when you build the container

curl https://s3.amazonaws.com//aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O

For your IAM policy concerns, Amazons example policy is below; you will need to make sure that your containers have access.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::myawsbucket/*"
            ]
        }
    ]
}

Someone on GitHub has done this already:

FROM ubuntu:latest
MAINTAINER Ryuta Otaki <otaki.ryuta@classmethod.jp>, Sergey Zhukov <sergey@jetbrains.com>
...

RUN apt-get install -q -y python python-pip wget
RUN cd / ; wget https://s3.amazonaws.com/aw

Answer 2

You can use fluentd for this purpose

You could use a Helm chart to install Fluentd:

$ helm install --name my-release incubator/fluentd-cloudwatch

Answer 3

Follow these steps:

1. Open CloudWatch Logs in the Management Console.
2. Create a log group name docker-logs.
3. Go to IAM and create a role for the use with EC2 named docker-logs and attach the CloudWatchLogsFullAccess policy. Note: do not use the CloudWatchLogsFullAccess policy for production workloads. Restrict access to the specific resource and actions instead.
4. Launch an EC2 Instance based on the Amazon Linux AMI 2017.03.*, select the IAM role ‘docker-logs’, and attach a security group allowing SSH access.
5. Log into the EC2 Instance via SSH.
6. yum install docker to install Docker.
7. service docker start to start Docker.
8. Start a container with docker run --log-driver=awslogs --log-opt awslogs-group=docker-logs alpine echo 'a cloudonaut.io example'
9. Open your CloudWatch Logs group to find your log message

Answer 4

There are 3 main steps involved it to it.

1. Create an IAM role/User
2. Install CloudAgent
3. Modify docker-compose file or docker run command