This tutorial explains how to setup Fluentd to send events to Q-Sensei Logs.
Prerequisites
Please verify the following is installed to configure your system correctly:
| Component | Recommendation |
|---|---|
| Docker | Minimum version 19.03.4 |
| docker-compose | Minimum version 1.24.1 |
| Operating System | Ubuntu, Linux (Amazon, RHEL) |
| Available RAM | Minimum 4 GB |
| Available Disk Space | 10 to 12 GB for File based buffering mechanism |
Step 1: Create Fluentd configuration file
Create a file called qsensei_fluentd.conf
Paste the following configuration in the file to declare system-wide parameters. In this example, we have declared the number of workers as 1 but to achieve higher throughput, this can be increased depending on the number of CPU cores available on the system.
<system>
workers 1
log_level info
</system>
Also, paste the following line to include the configuration file to report connector status. We will learn more about connector status reporting at STEP 5.
@include qsensei_fluentd_connector_status.conf
Step 2: Configure the Input plugin
The input source could be any of the Fluentd supported input plugins. In this example, we will setup an HTTP input source. Paste the following contents in Fluentd configuration file. Notice that, we used the label @QSENSEI_LOGS. Labeling events helps Fluentd in routing events to the correct destination.
<source>
@type http
port 9880
bind 0.0.0.0
@label @QSENSEI_LOGS
</source>
You can forward events in JSON format to this endpoint. Below is an example:
curl -X POST -d 'json={"foo":"bar"}' http://localhost:9880/app.log
Step 3: Configure the Output plugin
The output plugin configuration will define where to forward the events labelled @QSENSEI_LOGS. In this example, the output plugin is an HTTPS API gateway endpoint to your Q-Sensei Logs deployment. To configure the output plugin, paste the following contents in Fluentd configuration file.
<label @QSENSEI_LOGS>
# Enrich and transform the events
<filter **>
@type record_transformer
renew_record true
enable_ruby true
<record>
value ${record}
topic ${tag}
serialization_format JSON/OBJECT
fuse:action replace_or_create
fuse:type message
sent-time ${time}
</record>
</filter>
# Forward the events to Q-Sensei’s HTTPS endpoint
<match **>
@type http
http_method post
endpoint https://uxxx.execute-api.us-east-1.amazonaws.com
/api/upload/events
content_type application/json
json_array true
headers {"x-api-key": "xxxxx","x-deployment-id": "xxxxx"}
<format>
@type json
</format>
</match>
</label>
In the above output plugin configuration, notice the parameters endpoint, x-api-key and x-deployment-id. These parameters are specific to your deployment can be downloaded from the Manager UI.
Step 4: Fine tune the Output plugin configuration
The following parameter are specific to your deployment and can be obtained by downloading the connector configuration from the Manager UI.
| Parameter | Description | Fluentd configuration |
|---|---|---|
| HTTPS Endpoint | The HTTP host name for the API Gateway | endpoint https://uxxx.execute-api.us-east-1.amazonaws.com/api/upload/events |
| API Key | Each request should include the API Key in the request header. If the API key is missing, then HTTP 401 Unauthorized will be returned. | headers {"x-api-key": "xxxxx"} |
| Deployment ID | Each request should include the deployment Id to track usage i.e. size of data ingested | headers {"x-deployment-id": "xxxxx"} |
Optionally, file-based buffering can be enabled for Fluentd output plugins. This provides a persistent buffer implementation and buffer chunks are stored on disk. If file-based buffering is enabled, the final output plugin configuration will look like below.
# Forward the events to Q-Sensei’s HTTPS endpoint
<match **>
@type http
http_method post
endpoint https://uxxx.execute-api.us-east-1.amazonaws.com
/api/upload/events
content_type application/json
json_array true
headers {"x-api-key": "xxxxx","x-deployment-id": "xxxxx"}
<format>
@type json
</format>
<buffer>
@type file
path /fluentd/buffer
total_limit_size 1024MB
chunk_limit_size 16MB
flush_mode interval
flush_interval 10s
flush_thread_count 4
retry_type exponential_backoff
retry_timeout 1h
retry_max_interval 30
overflow_action throw_exception
</buffer>
</match>
For Fluentd buffering documentation refer https://docs.fluentd.org/configuration/buffer-section
Step 5: Configure connector status reporting
Q-Sensei Logs monitors connector health and reports that on the Manager UI. The following configuration is to report Fluentd connector’s health. Create a file qsensei_fluentd_connector_status.conf and paste the contents in the file.
# @FLUENT_LOG is a logical label to handle fluent.* events
<label @FLUENT_LOG>
<filter fluent.info>
@type record_transformer
renew_record true
enable_ruby true
<record>
description ${record}
state UP
</record>
</filter>
<filter fluent.error>
@type record_transformer
renew_record true
enable_ruby true
<record>
description ${record}
state ERROR
</record>
</filter>
<filter fluent.fatal>
@type record_transformer
enable_ruby true
<record>
description ${record}
state ERROR
</record>
</filter>
<match **>
@type http
http_method post
endpoint https://uxxx.execute-api.us-east-1.amazonaws.com/api/
upload/connector/status
content_type application/json
json_array true
headers {"x-api-key" : "xxxxx"}
<format>
@type json
</format>
<buffer>
flush_interval 10s
</buffer>
</match>
</label>
Step 6: Define docker services
Create a file called docker-compose.yaml and paste the following contents in the file. The docker image for Fluentd is the official docker image maintained by Treasure Data, Inc.
version: "3"
services:
aggregator:
image: fluent/fluentd:v1.12
ports:
- 9880:9880
volumes:
- ./qsensei_fluentd.conf:/fluentd/etc/qsensei_fluentd.conf
- ./qsensei_fluentd_connector_status.conf:/fluentd/etc/ qsensei_fluentd_connector_status.conf
- qsensei_fluentd_data:/fluentd
environment:
FLUENTD_CONF: qsensei_fluentd.conf
volumes:
qsensei_fluentd_data: {}
Step 7: Start Fluentd
Run the following command to start Fluentd. This command will pull the latest docker image, create docker volume to persist Fluentd data and start sending events to Q-Sensei Logs deployment.
docker-compose up -d