Amazon CloudWatch

Introduction

Humio’s CloudWatch integration sends your AWS CloudWatch Logs and Metrics to Humio by using AWS Lambda functions to ship the data.

The last major change to the integration was done on the 8th of September 2020 and the version was updated to v1.0.0. If you have installed the integration prior to this date, you are recommended to update your current integration to the newest version. See the section concerning Updating and Deleting the Integration.

Quick Installation

Use the Launch Stack buttons below to install in a region of your choice.

Use a globally unique stack name. The integration uses an S3 bucket and bucket names in S3 needs to be globally unique.

Launch Stack

Region
US East (N. Virginia) - US East 1 Install cloudwatch2humio in US East 1
US East (Ohio) - US East 2 Install cloudwatch2humio in US East 2
US West (Oregon) - US West 2 Install cloudwatch2humio in US West 2
EU (Frankfurt) - EU Central 1 Install cloudwatch2humio in EU Central 1
EU (Ireland) - EU West 1 Install cloudwatch2humio in EU West 1
EU (London) - EU West 2 Install cloudwatch2humio in EU West 2
EU (Stockholm) - EU North 1 Install cloudwatch2humio in EU North 1

If your region is missing, contact us - mail - Slack

Launch Parameters

Humio installs the integration using a CloudFormation template.

The template supports the following parameters:

  • EnableCloudWatchLogsAutoSubscription — Used to enable automatic subscription to new log groups when they are created. The default value is true.
  • EnableCloudWatchLogsBackfillerAutoRun — Used to enable the backfiller to run automatically once created. This will subscribe the log ingester to all present log groups specified with the subscription prefix parameter. The default value is true.
  • EnableVPCForIngesterLambdas — Used to enable an AWS VPC for all three ingester lambdas. If used, the parameters for security groups and subnet ids also need to be specified. The default value is false.
  • HumioCloudWatchLogsSubscriptionPrefix — By adding this filter, the Humio logs ingester will only subscribe to log groups whose paths start with this prefix. The default value is ””, which will subscribe the log ingester to all log groups.
  • HumioHost — The host you want to ship your Humio logs to. The default value is cloud.humio.com.
  • HumioProtocol — The transport protocol used for delivering log events to Humio. HTTPS is default and recommended, but HTTP is possible as well.
  • HumioIngestToken — The value of your ingest token for the repository from your Humio account that you want your logs delivered to. This parameter is required if you want to ingest log/metric events into Humio.
  • SecurityGroups — Security groups used by the VPC to group the ingester lambdas. Only required if you are using a VPC.
  • SubnetIds — Subnet ids used by the VPC to deploy the ingester lambdas into. Only required if you are using a VPC.
  • Version — The version of the integration to be installed. When creating a new stack, the default value is the newest version. Available releases can be found under releases in the GitHub repository.

The integration uses a set of AWS Lambdas, where some of these need to be manually enabled before use. In Configuring the Integration is explained how logs and metrics can be retrieved.

Manual Installation

The integration is available at our GitHub repository.

Following is described how to manually install the integration without using the launch buttons.

Prerequisites

  • AWS CLI installed and setup with an AWS account allowed to create a CloudFormation stack.
  • Python 3x installed.

Setup

  1. Clone the git repository: https://github.com/humio/cloudwatch2humio

  2. In the project folder, create a zip file with the contents of the src folder from the repository.

    • On Linux/MacOS:

    This is done by using the present makefile:

      $ make
    
    • On Windows:

      1. Create a folder named target in the project root folder, and copy all files from src into it.

      2. Install dependencies into the target folder using pip:

        pip3 install -r requirements.txt -t target
        
      3. In the target folder, zip all files into v0.0.0_YOUR-ZIP-FILE.zip. Choose whatever version number is relevant to you.

  3. Create an AWS S3 bucket using the following command:

    aws s3api create-bucket --bucket YOUR-HUMIO-BUCKET --create-bucket-configuration LocationConstraint=YOUR-REGION
    
    • The name of the AWS S3 bucket must be the same as the one specified in the CloudFormation file. The default name is humio-public-YOUR-REGION.
  4. Upload the zip file to the AWS S3 bucket:

    aws s3 cp target/VERSION_YOUR-ZIP-FILE.zip s3://YOUR-HUMIO-BUCKET/
    
  5. Create a parameters.json file in the project root folder, and specify the CloudFormation parameters, for example:

    [
        { 
            "ParameterKey": "EnableCloudWatchLogsAutoSubscription", 
            "ParameterValue": "true" 
        },
        { 
            "ParameterKey": "EnableCloudWatchLogsBackfillerAutoRun", 
            "ParameterValue": "true" 
        },
        { 
            "ParameterKey": "EnableVPCForIngesterLambdas", 
            "ParameterValue": "true" 
        },
        { 
            "ParameterKey": "HumioCloudWatchLogsSubscriptionPrefix", 
            "ParameterValue": "humio" 
        },
        { 
            "ParameterKey": "HumioHost", 
            "ParameterValue": "cloud.humio.com" 
        },
        { 
            "ParameterKey": "HumioProtocol", 
            "ParameterValue": "https" 
        },
        { 
            "ParameterKey": "HumioIngestToken", 
            "ParameterValue": "YOUR-SECRET-INGEST-TOKEN" 
        },
        { 
            "ParameterKey": "SecurityGroups", 
            "ParameterValue": "SECURITY-GROUP-1, SECURITY-GROUP-2" 
        },
        { 
            "ParameterKey": "SubnetIds", 
            "ParameterValue": "SUBNET-ID-1, SUBNET-ID-2, SUBNET-ID-3" 
        },
        {
            "ParameterKey": "Version",
            "ParameterValue": "VERSION"
        }
    
    ]
    
    • Only the HumioIngestToken parameter is required to be specified in the parameters.json file as the rest have default values.
    • The Version must match the version number given in the name of the zip file you have uploaded.
  6. Create the stack using the CloudFormation file and the parameters that you have defined:

    aws cloudformation create-stack --stack-name YOUR-DESIRED-STACK-NAME --template-body file://cloudformation.json --parameters file://parameters.json --capabilities CAPABILITY_IAM --region YOUR-REGION
    
    • CAPABILITY_IAM is the required value here.
  7. Updating the stack.

    • To update the stack, add your changes and use the command:

      aws cloudformation update-stack --stack-name YOUR-DESIRED-STACK-NAME --template-body file://cloudformation.json --parameters file://parameters.json --capabilities CAPABILITY_IAM --region YOUR-REGION
      
    • Note: The stack will only register changes in the CloudFormation file or in the parameters.json file. If you have updated the lambda files, you need to change the name of the version of the zip file, and thus the parameter Version for your changes to be recognized.

  8. Deleting the stack.

    • To delete the stack, use the following command:

      aws cloudformation delete-stack --stack-name YOUR-DESIRED-STACK-NAME
      

Configuring the Integration

The integration needs to be manually activated before logs and metrics are shipped to Humio. This is required both when using the quick and manual installations.

Retrieving CloudWatch Logs

For retrieving CloudWatch Logs, the integration uses three AWS Lambdas:

If EnableCloudWatchLogsBackfillerAutoRun is set to true an additional custom resource will be created — namely the HumioBackfillerAutoRunner.

If you only want specific log groups to be ingested into Humio, you can use the HumioCloudWatchLogsSubscriber as this only subscribes the log ingester to one log group at a time. If you want to subscribe to all log groups available, you can use the HumioCloudWatchBackfiller. If you have set the EnableCloudWatchLogsBackfillerAutoRun parameter to true when creating the stack, then you will not have to manually trigger it as it should already have run on creation and subscribed the log ingester to all available log groups. Otherwise, both lambdas need to be enabled using test events.

For the HumioCloudWatchLogsSubscriber lambda, configure your test event like the example below with “EXAMPLE” representing an actual log group, and click Test.

{
  "detail": { "requestParameters": { "logGroupName":"EXAMPLE" }}
}

For the HumioCloudWatchLogsBackfiller lambda, use the default test event and click Test. This might take awhile depending on the number of log groups that you are subscribing to.

Retrieving CloudWatch Metrics

For retrieving CloudWatch Metrics, the integration can use either one of the two AWS Lambdas:

Both lambdas can essentially retrieve the same information, but there are some differences in their limitations and cost.

There are already defined request parameters for both of these lambdas, and by clicking the Test button for either, using the default test parameters, will make a request retrieving metrics regarding the number of lambda invocations made. The default settings of the request only looks at the metrics from the last 15 minutes.

To change the API request parameters, you can edit the conf_metric_ingester.json file for the HumioCloudWatchMetricIngester lambda, and the conf_metric_statistics_ingester.json file for the HumioCloudWatchMetricStatisticsIngester lambda.

These can be found in your Lambda Console for each lambda under the Function code section.

Setting the HumioCloudWatchMetricIngester API Parameters

In the conf_metric_ingester.json, the following code is present:

{
    "MetricDataQueries": [
        {
            "Id": "test_cloudwatch_metrics_lambda_invocations",
            "MetricStat": {
                "Metric": {
                    "Namespace": "AWS/Lambda",
                    "MetricName": "Invocations"
                },
                "Period": 60,
                "Stat": "Sum",
                "Unit": "Count"
            }
        }
    ]
}

This request to the AWS CloudWatch API through a boto3 client retrieves the number of lambda invocations made for the last 15 minutes. The integration will then add a Humio event for each timestamp retrieved.

Change this code accordingly to which metrics you want to retrieve. For more information regarding the different options available, consult the AWS documentation regarding metrics, and for the specifics regarding the request, consult the Boto3 Docs for the get_metric_data(**kwargs) function.

Setting the HumioCloudWatchMetricStatisticsIngester API Parameters

In the conf_metric_statistics_ingester.json, the following code is present:

[
    {
      "Namespace": "AWS/Lambda",
      "MetricName": "Invocations",
      "Period": 60,
      "Statistics": [
        "Sum"
      ]
    },
    {
      "Namespace": "AWS/Lambda",
      "MetricName": "Errors",
      "Period": 60,
      "Statistics": [
        "Sum"
      ]
    }
]

This request to the AWS CloudWatch API through a boto3 client retrieves the number of lambda invocations and errors made for the last 15 minutes. The integration will then add a Humio event for each timestamp retrieved.

Change this code accordingly to which metrics you want to retrieve. For more information regarding the different options available consult, the AWS documentation regarding metrics, and for the specifics regarding the request, consult the Boto3 Docs for the get_metric_statistics(**kwargs) function.

An important thing to notice here, is that the API parameters are wrapped in a list. This is so that more requests can be made with the same function at once.

Updating the Integration

When the codebase for the integration is updated, you are recommended to update your current integration setup as well. (See the Introduction section for when the integration was last updated.)

To update the cloudwatch2humio integration go to Services > CloudFormation, find the cloudwatch2humio stack and click its Update button. Under Version in the parameter section write the version you want to use with the format vX.X.X although replacing the Xs with the desired version.

For a list of available versions go to the releases section in the GitHub repository.

Deleting the Integration

To delete the cloudwatch2humio integration, you need to find its stack in Services > CloudFormation, and click its Delete button.

The most common error that happens when trying to delete the stack concerns the S3 bucket associated with CloudTrail. (Note: This S3 bucket is only created if the parameter EnableCloudWatchLogsAutoSubscription is set to true.) As this bucket gathers events, it tends not to be empty, and CloudFormation is not allowed to delete a bucket which is not empty.

To fix this error and delete the stack, go to your view of S3 buckets, Services > S3, and mark the offending bucket and click the “Empty” button.

Now you should be able to delete the stack. The bucket is automatically updated with CloudTrail events, so you have to make sure you empty the bucket right before deleting the stack.

How the Integration Works

The integration will install five lambda functions:

  • HumioCloudWatchLogsIngester,
  • HumioCloudWatchLogsSubscriber,
  • HumioCloudWatchLogsBackfiller,
  • HumioCloudWatchMetricIngester,
  • HumioCloudWatchMetricStatisticsIngester.

If EnableCloudWatchLogsAutoSubscription is set to true, then the CloudFormation template will also set up CloudTrail and a S3 bucket for your account. This is needed to trigger the HumioCloudWatchLogsSubscriber lambda automatically to newly created log groups.

If EnableCloudWatchLogsBackfillerAutoRun is set to true, then the CloudFormation template will also create a custom resource named HumioBackfillerAutoRunner. This is needed to trigger the HumioCloudWatchLogsBackfiller lambda automatically when created, which will subscribe the HumioCloudWatchLogsSubscriber to all present log groups in your account.

HumioCloudWatchLogsIngester

This lambda handles the delivery of your CloudWatch log events to Humio.

HumioCloudWatchLogsSubscriber

This lambda will subscribe the HumioCloudWatchLogsIngester to a log group every time a new one is created. This is done by filtering CloudTrail events and triggering the HumioCloudWatchLogsSubscriber lambda every time a new log group is created.

HumioCloudWatchLogsBackfiller

This function will paginate through your existing CloudWatch log groups and subscribe the HumioCloudWatchLogsIngester to every single one.

HumioCloudWatchMetricIngester

This lambda handles the delivery of your CloudWatch metrics to Humio. This lambda uses the GetMetricData action from the CloudWatch API reference.

HumioCloudWatchMetricStatisticsIngester

This lambda can also handle the delivery of your CloudWatch metrics to Humio. This lambda uses the GetMetricStatistics action from the CloudWatch API reference.