1. Introduction
Apache Pinot, developed originally by LinkedIn, is a real-time distributed OLAP (Online Analytical Processing) datastore designed to deliver low latency and high throughput for analytical queries. In this article, we’ll explore Apache Pinot’s key features and architecture, and learn how to interact with it.
2. What Is Apache Pinot?
Apache Pinot is a real-time distributed OLAP (Online Analytical Processing) data store designed to deliver low latency and high throughput for analytical queries. It is optimized for handling large-scale, time-series data and is commonly used for analytics on event streams, logs, and other types of real-time data. Below are some of its key features:
- Real-time and Batch Data Ingestion: Pinot can ingest data in real-time from streams like Kafka and in batches from sources like Hadoop or S3.
- Low Latency Queries: Pinot is designed to handle complex OLAP queries with sub-second response times.
- Scalability: It can scale horizontally by adding more servers to handle increased load.
- Pluggable Indexing: Supports various indexing techniques like inverted index, sorted index, range index, and more to optimize query performance.
- Schema Flexibility: Allows for evolving schemas without downtime.
- Support for SQL-like Query Language: Provides a SQL-like language for querying data, making it accessible to users familiar with SQL.
3. Architecture
Apache Pinot comprises several key components that work together to provide real-time distributed OLAP capabilities. These components include:
- Cluster: It collects the software processes and hardware resources required to ingest, store, and process data. The processes include controller, zookeeper, server, broker, and minion. Pinot uses Apache Zookeeper as a distributed metadata store and Apache Helix for cluster management.
- Controller: The controller manages the cluster and coordinates tasks such as segment creation, routing, and data management. It also handles configuration management and cluster metadata.
- Broker: The broker component is responsible for query routing. It receives queries from clients and routes them to the appropriate servers that hold the relevant data segments. The broker then aggregates and returns results to the client.
- Server: The server stores and manages data segments, processes queries, and returns results to the broker. It’s responsible for the real-time ingestion and indexing of data.
- Minion: The minion component handles background tasks such as data compaction, segment management, and offline segment generation. It offloads these tasks from the servers to ensure efficient resource utilization.
- Tenant: A tenant enables multi-tenancy, allowing users or applications to share the cluster while maintaining data and resource isolation, ensuring fair usage and performance isolation.
- Segment: Pinot stores data in segments, immutable files containing a subset of the dataset. Each segment optimizes for fast reads, utilizing techniques like columnar storage to improve query performance. Pinot replicates segments across multiple nodes to ensure data availability and fault tolerance.
When we submit a query, broker nodes distribute it to the appropriate server nodes containing the relevant data segments. The server nodes process the query and return results to the broker nodes, which aggregate the results and send them back to the client. This distributed query processing ensures efficient and quick execution of queries.
4. Installing Pinot
We can install Pinot using Docker, Kubernetes, or directly on our local machine. The official documentation provides detailed instructions for various installation methods. We’ll follow the installation using Docker.
To install Pinot via Docker, the system needs to pass the following criteria:
- Docker must be installed on the machine.
- The docker memory must be configured with a minimum of 4 CPUs, 16GM Memory, 4GB Swap, and 60 GB disk image size.
After setting up and running Docker, execute the following command in a terminal to fetch the latest image:
docker pull apachepinot/pinot:latest
5. Working with Pinot
6. Conclusion
In this tutorial, we covered the basics of Apache Pinot and explored its architecture. Apache Pinot is a leading datastore for real-time analytics, enabling organizations to process and analyze large data volumes instantly. Its scalable architecture, low-latency queries, and versatility make it a top choice for businesses.
As demand for real-time insights grows, Apache Pinot drives innovation and transformation in the digital landscape.