AWS CloudWatch Integration for Humio
Beta

Table of Contents

Introduction

Humio’s CloudWatch integration sends your AWS CloudWatch Logs and Metrics to Humio by using AWS Lambda functions to ship the data.

The integration was last updated 11th of June, 2020. If you have installed the integration prior to this date, you are recommended to update your current integration to the newest version. See the section concerned Updating and Deleting the Integration.

Quick Installation

Use the Launch Stack buttons below to install in a region of your choice.

Use a globally unique stack name. The integration uses an S3 bucket and bucket names in S3 needs to be globally unique.

Launch Stack

Region
US East (N. Virginia) - US East 1 Install cloudwatch2humio in US East 1
US East (Ohio) - US East 2 Install cloudwatch2humio in US East 2
US West (Oregon) - US West 2 Install cloudwatch2humio in US West 2
EU (Frankfurt) - EU Central 1 Install cloudwatch2humio in EU Central 1
EU (Ireland) - EU West 1 Install cloudwatch2humio in EU West 1
EU (London) - EU West 2 Install cloudwatch2humio in EU West 2

If your region is missing, contact us - mail - Slack

Launch parameters

Humio installs the integration using a CloudFormation template.

The template supports the following parameters:

  • HumioHost — The host you want to ship your Humio logs to. The default value is cloud.humio.com.
  • HumioProtocol — The transport protocol used for delivering log events to Humio. HTTPS is default and recommended, but HTTP is possible as well.
  • HumioIngestToken — The value of your ingest token for the repository from your Humio account that you want your logs delivered to.
  • HumioCloudWatchLogsAutoSubscription — Enable automatic subscription to new log groups. The default value is true.
  • HumioCloudWatchLogsSubscriptionPrefix — By adding this filter the Humio logs ingester will only subscribe to log groups whose paths start with this prefix.

The integration uses a set of AWS Lambdas, which needs to be manually enabled before use. In Configuring the Integration is explained how logs and metrics can be retrieved.

Manual Installation

The integration is available from GitHub:

https://github.com/humio/cloudwatch2humio

Following is described how to manually install the integration without using the launch buttons.

Prerequisites

  • AWS CLI installed and setup with an AWS account allowed to create a CloudFormation stack.
  • Python 3x installed.

Setup

  1. Clone the git repository: https://github.com/humio/cloudwatch2humio
  2. In the project folder create a zip file with the content of the src folder from the repository.

    • On Linux/MacOS:

      • This is done by using a makefile:

        $ make
        
    • On Windows:

      1. Create a folder named target in the project root folder, and copy all files from src into it.
      2. Install requirements into the target folder using pip:

        pip3 install -r requirements.txt -t target
        
      3. In the target folder, zip all files into cloudwatch_humio.zip.

  3. Create an AWS S3 bucket using the following command:

    aws s3api create-bucket --bucket humio-public-REGION --create-bucket-configuration LocationConstraint=REGION
    

    The name of the AWS S3 bucket must be the same as the one specified in the CloudFormation file, beware of this if you choose another name.

  4. Upload the zip file to the AWS S3 bucket:

    aws s3 cp target/cloudwatch_humio.zip s3://humio-public-REGION/
    
  5. Create a parameters.json file in the project root folder, and specify the CloudFormation parameters. Only the HumioIngestToken is required as the rest have default values, for example:

    [
      { 
        "ParameterKey": "HumioHost", 
        "ParameterValue": "cloud.humio.com" 
      },
      { 
        "ParameterKey": "HumioProtocol", 
        "ParameterValue": "https" 
      },
      { 
        "ParameterKey": "HumioIngestToken", 
        "ParameterValue": "YOUR-SECRET-INGEST-TOKEN" 
      },
      { 
        "ParameterKey": "HumioCloudWatchLogsAutoSubscription", 
        "ParameterValue": "true" 
      },
      { 
        "ParameterKey": "HumioCloudWatchLogsSubscriptionPrefix", 
        "ParameterValue": "true" 
      }
    ]
    
  6. Create the stack using the CloudFormation file and the parameters that you have defined:

    aws cloudformation create-stack --stack-name STACK-NAME --template-body file://cloudformation.json --parameters file://parameters.json --capabilities CAPABILITY_IAM --region REGION
    

Configuring the Integration

The integration needs to be manually activated before logs and metrics are shipped to Humio. This is required both when using the quick and manual installation.

Retrieving CloudWatch Logs

For retrieving CloudWatch Logs, the integration uses three AWS Lambdas:

If you only want specific log groups to be ingested into Humio, you can use the HumioCloudWatchLogsSubscriber as this only subscribes the log ingester to one log group at a time. If you want to subscribe to all log groups available, you can use the HumioCloudWatchBackfiller. Both lambdas need to be enabled using test events.

For the HumioCloudWatchLogsSubscriber lambda, configure your test event like the example below with “EXAMPLE” representing an actual log group, and click Test.

{
  "detail": { "requestParameters": {"logGroupName":"EXAMPLE"}}
}

For the HumioCloudWatchLogsBackfiller lambda, use the default test event and click Test. This might take awhile depending on the number of log groups that you are subscribing to.

Retrieving CloudWatch Metrics

For retrieving CloudWatch Metrics, the integration can use either one of the two AWS Lambdas:

Both lambdas can essentially retrieve the same information, but there are some differences in their limitations and cost.

There are already defined request parameters for both of these lambdas, and by clicking the Test button for either, using the default test parameters, will make a request retrieving metrics regarding the number of lambda invocations made. The default settings of the request only looks at the metrics from the last 15 minutes.

To change the API request parameters, you can edit the conf_metric_ingester.json file for the HumioCloudWatchMetricIngester lambda, and the conf_metric_statistics_ingester.json file for the HumioCloudWatchMetricStatisticsIngester lambda.

These can be found in your Lambda Console for each lambda under the Function code section.

Setting the HumioCloudWatchMetricIngester API Parameters

In the conf_metric_ingester.json, the following code is present:

{
    "MetricDataQueries": [
        {
            "Id": "test_cloudwatch_metrics_lambda_invocations",
            "MetricStat": {
                "Metric": {
                    "Namespace": "AWS/Lambda",
                    "MetricName": "Invocations"
                },
                "Period": 60,
                "Stat": "Sum",
                "Unit": "Count"
            }
        }
    ]
}

This request to the AWS CloudWatch API through a boto3 client retrieves the number of lambda invocations made for the last 15 minutes. The integration will then add a Humio event for each timestamp retrieved.

Change this code accordingly to which metrics you want to retrieve. For more information regarding the different options available, consult the AWS documentation regarding metrics, and for the specifics regarding the request, consult the Boto3 Docs for the get_metric_data(**kwargs) function.

Setting the HumioCloudWatchMetricStatisticsIngester API Parameters

In the conf_metric_statistics_ingester.json, the following code is present:

[
    {
      "Namespace": "AWS/Lambda",
      "MetricName": "Invocations",
      "Period": 60,
      "Statistics": [
        "Sum"
      ]
    },
    {
      "Namespace": "AWS/Lambda",
      "MetricName": "Errors",
      "Period": 60,
      "Statistics": [
        "Sum"
      ]
    }
]

This request to the AWS CloudWatch API through a boto3 client retrieves the number of lambda invocations and errors made for the last 15 minutes. The integration will then add a Humio event for each timestamp retrieved.

Change this code accordingly to which metrics you want to retrieve. For more information regarding the different options available consult, the AWS documentation regarding metrics, and for the specifics regarding the request, consult the Boto3 Docs for the get_metric_statistics(**kwargs) function.

An important thing to notice here, is that the API parameters are wrapped in a list. This is so that more requests can be made with the same function at once.

Updating and Deleting the Integration

When the codebase for the integration is updated, you are recommended to update your current integration setup as well. (See the Introduction section for when the integration was last updated.) Due to how the CloudFormation file is setup, it is not able to automatically detect updates to the lambda functions, and therefore the only way to update the entire stack, is to delete it and launch a new.

Deleting the Integration

To delete the cloudwatch2humio integration, you need to find its stack in Services > CloudFormation, and click its Delete button.

The most commont error that happens when trying to delete the stack concerns the S3 bucket associated with CloudTrail. As this bucket gathers events, it tends not to be empty, and CloudFormation is not allowed to delete a bucket which is not empty.

To fix this error and delete the stack, go to your view of S3 buckets, Services > S3, and mark the offending bucket and click the “Empty” button.

Now you should be able to delete the stack. The bucket is automatically updated with CloudTrail events, so you have to make sure you empty the bucket right before deleting the stack.

How the Integration Works

The integration will install five lambda functions:

  • HumioCloudWatchLogsIngester,
  • HumioCloudWatchLogsSubscriber,
  • HumioCloudWatchLogsBackfiller,
  • HumioCloudWatchMetricIngester, and
  • HumioCloudWatchMetricStatisticsIngester.

The CloudFormation template will also set up CloudTrail and a S3 bucket for your account. This is needed to trigger the HumioCloudWatchLogsSubscriber lambda automatically (if HumioCloudWatchLogsAutoSubscription is set to true) to newly created log groups.

HumioCloudWatchLogsIngester

This lambda handles the delivery of your CloudWatch log events to Humio.

HumioCloudWatchLogsSubscriber

This lambda will subscribe the HumioCloudWatchLogsIngester to a log group every time a new one is created. This is done by filtering CloudTrail events and triggering the HumioCloudWatchLogsSubscriber lambda every time a new log group is created.

HumioCloudWatchLogsBackfiller

This function will paginate through your existing CloudWatch log groups and subscribe the HumioCloudWatchLogsIngester to every single one.

HumioCloudWatchMetricIngester

This lambda handles the delivery of your CloudWatch metrics to Humio. This lambda uses the GetMetricData action from the CloudWatch API reference.

HumioCloudWatchMetricStatisticsIngester

This lambda can also handle the delivery of your CloudWatch metrics to Humio. This lambda uses the GetMetricStatistics action from the CloudWatch API reference.