A Q-Sensei Logs SaaS deployment is created in an AWS Virtual Private Cloud (VPC). Each deployment is created in a logically isolated virtual network. Having a VPC allows us to use multiple layers of security:
- Each EC2 instance is part of a private subnet with no internet access.
- Each EC2 instance is part of a security group that acts as a virtual firewall to control incoming and outgoing traffic.
- The private subnet has an associated network access control list (ACL), an additional layer of security that acts as a firewall for the private subnet in which the deployment is created.
Events Collection Tier
The events collection tier is located in your local environment. It could be either on-prem or in your cloud environment. A connector is deployed in the events collection tier to consume events, batch and securely forward them to your deployment for ingestion. To read more about Q-Sensei Logs Connector read the section on connectors.
Upload API Gateway
The upload API gateway authenticates each incoming request for your deployment and forwards it to the backend events ingestion and indexing tier.
Events Ingestion & Indexing Tier
Upload API Instance: The Upload API instance queries the service registry for healthy indexers and forwards the batch of events to the indexer.
Indexer: The indexer saves the batch of events in a distributed & fully replicated database, updates the indexes and synchronizes the indexes with the Analytical Search Tier.
Analytical Search Tier
This tier is responsible for serving all the analytical query workload. The Search Head keeps the updated index in-memory to serve analytical queries.
Distributed Service Registry
- Q-Sensei uses a highly available, distributed service registry to reduce the configuration effort required by an administrator or DevOps team.
- Each node, as it joins the cluster, registers a service (or services) in the Service Registry.
- Other clients use the Service Registry to discover providers of a given service.
- Clients also provide a number of health checks associated with services. This information is used to route traffic away from unhealthy nodes.
- Please read the section on node roles to understand which services each node registers in the service registry.