Elastic BeanStalk logs with Cloudwatch and ElasticSearch – Part 1

Using SSH to watch logs directly on a server is boring, especially if you work with an application in a cluster: you have to check each machine before finding the trace that you were looking for. In an elastic environement (like Elastic BeanStalk), it’s a tough job: it takes time to retrieve the IP/domain name of your instances and they have no guarantees to stay enough time alive to see their logs… Thanks to AWS, by configuring the right tools, you can easily centralize your logs: Amazon CloudWatch Logs (to read each log file, of each EC2 instance) and Amazon ElasticSearch Service (to centralize all the logs file in one place).

Overview of the strategy

Each EC2 managed by Elastic Beanstalk will be configured to send logs to Cloudwatch Logs. Then, each logs file streamed from Amazon Cloudwatch Logs will be transfered to an ElasticSearch database that will index all the logs. The logs will be centralized and searchable, with Kibana.

Sending Application logs to Cloudwatch Logs

Your application deployed with Elastic Beanstalk have to write in files, the logs that you are expecting… a very classic way! A dedicated agent named Cloudwatch Agent is setup on your EC2 instance, and will stream, by default every 5s and based on your files, your logs to CloudWatch Logs.

If you don’t use Elastic Beanstalk, you would have to install it manually (refer to the Amazon documentation about that topic). With Elastic Beanstalk, the Cloudwatch Agent could be installed and setup automatically  with the insertion of scripts in the directory .ebextensions at the root of your Docker or of any file that you are deploying.

You can add AWS Elastic Beanstalk configuration files (.ebextensions) to your web application’s source code to configure your environment and customize the AWS resources that it contains. Configuration files are YAML formatted documents with a .config file extension that you place in a folder named .ebextensions and deploy in your application source bundle.

Setup the deployed application

The following Github repo will show you a demo of a Docker Spring Boot application, configured to have its logs transfered to CloudWatch Logs.

In the .ebextensions directory of this Github repo, you’ll see the 4 following files that will enable the CloudWatch agent and forward your logs to CloudWatch Logs:

  • cwl-setup.config: setup the AWS CloudWatch agent on your EC2 instances
  • eb-logs.config:  setup the Elastic BeanStalk log publication to include the admin logs for Cloudwatch Logs.
  • cwl-springbootlogsapp.config: it contains the configuration to find your application logs file. Modify this file to adapt the path/pattern to your log file.
  • cwl-webrequest-metrics.config: it contains the configuration to produce statistics, based on the HTTP status, logged by the nginx server (default proxy for Docker deployments).

Setup the Elastic Beanstalk rights

The user that will run your EC2 instances must have the IAM rights to manage CloudWatch:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
    ],
      "Resource": [
        "arn:aws:logs:*:*:*"
    ]
  }
 ]
}

Visualize your log files

Once your application, will be deployed with the right .extensions and the right rights in AWS, go in AWS console to CloudWatch Logs, then you will be able to see each file in what is named a Log Group.

If you click on a Log Group, you’ll see Log Streams. Usually, you have a Log Stream per instance of server and of file:

And if you click on a Log Stream, you’ll be able to read your application logs.

Going  Further

Now, you are able to monitor your logs in your AWS Console. As seen, the logs are split in files and servers. For a better analysis, for an application running on a distributed/clustered environement, we will see in a next article how to manage them with ElasticSearch.