Fluentd is an open-source data collector that can be used as a unified logging layer between a variety of input data sources and Q-Sensei Logs.
Fluentd Data Pipeline
A Fluentd data pipeline can be configured using 4 plugins: Input, Filter, Buffer and Output. Fluentd’s flexible plugin system allows to easily extend its functionality. Each event in the pipeline is JSON formatted and has a tag, timestamp and record.
- Input plugin
- Input plugins define the input data sources
- Filter plugin
- With filter plugins, you can modify or filter records based on certain fields
- Buffer Plugin
- Buffering events provides reliability. In case of a network failure, events are buffered on the file system.
- Output Plugin
- Define the destination of the records. In case of Q-Sensei Logs, this will be an HTTPS endpoint.