Skip to content

Using netdata with AWS Kinesis Data Streams

Prerequisites

To use AWS Kinesis as a backend AWS SDK for C++ should be installed first. libcrypto, libssl, and libcurl are also required to compile netdata with Kinesis support enabled. Next, netdata should be re-installed from the source. The installer will detect that the required libraries are now available.

If AWS SDK for C++ is being installed from sources, it is useful to set -DBUILD_ONLY="kinesis". Otherwise, the building process could take a very long time.

Configuration

To enable data sending to the kinesis backend set the following options in netdata.conf:

[backend]
    enabled = yes
    type = kinesis
    destination = us-east-1
set the destination option to an AWS region.

In the netdata configuration directory run ./edit-config aws_kinesis.conf and set AWS credentials and stream name:

# AWS credentials
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key

# destination stream
stream name = your_stream_name
Alternatively, AWS credentials can be set for the netdata user using AWS SDK for C++ standard methods.

A partition key for every record is computed automatically by the netdata with the purpose to distribute records across available shards evenly.