Q-Sensei Logs can be deployed either in cloud or on-prem. Our JSON based REST API endpoint makes it easy to upload events from any Log data source.
Basic Architecture
Log data source
Log data source can be any kind of streaming, machine and historical data source. A few examples of log data sources are:
- Web server logs
- Live application logs
- Network events
- Message Queues; Apache Kafka
Log data source can be on-prem, private cloud or public cloud. Q-Sensei Logs exposes an HTTP endpoint using which connectors can upload events to the deployment.
Connector
Connectors connect to the log data source, monitors for new events and forwards events to Q-Sensei Logs with minimal processing overhead. With an HTTP upload endpoint, there are multiple options:
- Fluentd
- Fluent Bit
- Logstash
- Q-Sensei’s official connectors
Use the following link for a complete list of integration options: https://www.qsensei.com/q-sensei-connectors
HTTP Upload API
A highly available HTTP endpoint to upload events. It implements buffering to handle events generated at high volume.
Indexer
Q-Sensei Fuse instance to handle database updates and index updates.
Search Head
Q-Sensei Fuse instance to handle user query requests.
AWS SaaS Architecture
Q-Sensei Logs’ AWS SaaS architecture extends the basic architecture to provide a fully managed, highly available and secure service. The table below lists all AWS services used.
| Category | List of AWS Services |
|---|---|
| Security, Identity and Compliance | AWS Cognito AWS Identity and Access Management AWS Secrets Manager AWS Key Management Service (KMS) |
| Network & Content delivery | AWS PrivateLink AWS Route53 AWS Virtual Private Cloud (VPC) Elastic Load Balancing |
| Management & Governance | AWS CloudFormation AWS Organizations AWS Systems Manager |
| Compute | AWS Elastic Compute Cloud (EC2) |
| Application Integration | Amazon API Gateway AWS Simple Queue Service AWS Simple Notification Service |
| Customer Engagement | AWS Simple Email Service (SES) |
To deploy Q-Sensei Logs on-prem a mapping of following services should be established in the on-prem environment.
User Identity and Access Management
- AWS Cognito
Authentication flow
- In Q-Sensei Logs, the authentication work is offloaded to the Internet facing Application Load Balancer.
Email Notifications
- AWS Simple Email Service