Setup FluentBit Connector

This tutorial explains how to setup Fluent Bit to send events to Q-Sensei Logs.

Prerequisites

Please verify the following is installed to configure your system correctly:

Component Recommendation
Docker Minimum version 19.03.4
docker-compose Minimum version 1.24.1
Operating System Ubuntu, Linux (Amazon, RHEL)
Available RAM Minimum 4 GB
Available Disk Space 10 to 12 GB for File based buffering mechanism

 

Step 1: Create the Fluent Bit configuration file


Create a Fluent Bit main configuration file qsensei_fluentbit.conf. Paste the following contents to declare some SERVICE parameters:

[SERVICE]
Flush             5
Log_Level     info

Also, paste the following line to include the configuration file to report connector status. We will create this configuration file at STEP 5.

@INCLUDE qsensei_fluentbit_connector_status.conf

 

Step 2: Configure the Input Plugin


The INPUT section defines a way to collect data from source. In this example, we will collect Disk I/O metrics and send it to Q-Sensei Logs deployment. Paste the following contents in your Fluent Bit configuration file:

[INPUT]
Name                   disk
Tag                       disk
Interval_Sec        1
Interval_NSec     0

Name is a mandatory configuration key and it will let Fluent Bit know which input plugin should be loaded. Fluent Bit supports different input plugins. Depending on the input plugin, we may need to specify additional configuration keys. For a complete list of input plugins, check the Fluent Bit input plugin documentation: https://docs.fluentbit.io/manual/pipeline/inputs

 

Step 3: Configure the Filter Plugin


Filtering is a way to transform the event before sending it to destination. Q-Sensei expects the event body in a key called value. The benefit is that the event can be treated as an immutable object and additional metadata can be added at the top level.

In this example, we define a two-stage FILTER pipeline. First to nest all keys under the key value and second is to add a few additional Q-Sensei Logs specific metadata.

[FILTER]
Name nest
Match disk
Operation nest
Wildcard *
Nest_under value

[FILTER]
Name modify
Match disk
Add fuse:type message
Add fuse:action replace_or_create
Add serialization_format JSON/OBJECT
Add topic disk

Note is the second stage of the Filter pipeline, we add a key called topic which matches the tag associated with the events.

 

Step 4: Configure the Output Plugin


The Output plugin helps us define destination for the data. In this example, our destination is Q-Sensei Logs’s HTTPS endpoint. Paste the following contents in your configuration file:

[OUTPUT]
Name http
Port 443
Match disk
Host uxxxxxxxx.execute-api.us-east-1.amazonaws.com
Header x-api-key xxxxxxx
Header x-deployment-id xxxxxx
json_date_key sent-time
json_date_format iso8601
format json
tls On
tls.verify On
uri stack-6-prod/api/upload/events

Note the parameters Host, Header and uri. These parameters are specific to your deployment and can be obtained by downloading the connector configuration from the Manager UI

 

Step 5: Configure connector status reporting


Q-Sensei Logs monitors connector health and reports that on the Manager UI. The following configuration defines Fluent Bit configuration to report FluentBit connector’s health events.

Create a file called qsensei_fluentbit_connector_status.conf and paste the following contents in the file:

[INPUT]
Name proc
Tag fluent_bit_health
Proc_Name fluent-bit
Interval_Sec 60
Interval_NSec 0

[OUTPUT]
Name http
Port 443
Match fluent_bit_health
Host ut7wiror9i.execute-api.us-east-1.amazonaws.com
Header x-api-key xxxx
json_date_key sent-time
json_date_format iso8601
format json
tls On
tls.verify On
uri stack-6-prod/api/upload/connector/status

Note the parameters Host, Header and uri. These parameters are specific to your deployment and can be obtained by downloading the connector configuration from the Manager UI

 

Step 6: Define docker services


Create a file called docker-compose.yaml and paste the following contents in the file. The docker image for Fluent Bit is the official docker image maintained by Treasure Data, Inc.

version: "3"

services:
forwarder:
image: fluent/fluent-bit
volumes:

  • ./qsensei_fluentbit.conf:/fluent-bit/etc/fluent-bit.conf
  • ./qsensei_fluentbit_connector_status.conf:/fluent-bit/etc/ qsensei_fluentbit_connector_status.conf
  • qsensei_fluentbit_data:/fluent-bit

volumes:
qsensei_fluentbit_data: {}

 

Step 7: Start Fluent Bit


Run the following command to start Fluent Bit. This command will pull the latest docker image, create docker volume to persist Fluent Bit data and start sending events to Q-Sensei Logs deployment.

docker-compose up -d

Was this article helpful?
0 out of 0 found this helpful