Apache Pinot is a real-time distributed OLAP data warehouse
Apache Pinot is a real-time, distributed OLAP storage solution designed to deliver scalable, low-latency, real-time analytics.
It can ingest data from batch data sources (such as HDFS, S3, Azure Data Lake, Google Cloud Storage) as well as streaming sources (such as Kafka). Pinot is designed to scale horizontally, so it can scale to larger datasets and higher query rates as needed.
About Apache Pinot
The Pinot project was originally developed by LinkedIn and in 2015 was transferred to the Apache Foundation for further joint development. The storage is designed to operate in an environment where new data is constantly being added and is engineered to provide minimal and predictable latency, enabling its use for real-time query processing.
Like most other data warehouses and OLAP data warehousing solutions, Pinot supports an SQL-like query language that supports selection, aggregation, filtering, grouping, sorting, and other data queries.
Apache Pinot provides horizontal scalability and a means of achieving fault tolerance and survivability against software and hardware failures. Replication and backup processes are integrated directly into the data processing cycle for data added to the data store. On the one hand, this approach significantly simplifies the architecture, but on the other hand, it introduces a delay between data addition and its availability for querying.
The data is stored in tables within a column-oriented database, and various compression schemes and the ability to place multiple values ​​in a field are supported. Pinot provides a pluggable indexing system that can utilize various indexing technologies (ordered index, bitmap index, inverted index, StarTree index, Bloom filter, range index, text search index (Lucence/FST), JSON index, geospatial index).
Of the characteristics that stand out from Apache Pinot:
- Column Oriented– A column-oriented database with various compression schemes such as run length and fixed bit length.
- Pluggable indexing: Pluggable indexing technologies, Sorted Index, Bitmap Index, Inverted Index.
- Query optimization- Ability to optimize the query/execution plan based on query and segment metadata.
- Ingestion of streams and batches: Near real-time ingestion of Hadoop streams and batch ingestion.
- Consultation: SQL-based query execution engine.
- Upsert during ingestion in real time: update data at scale with consistency
- Multiple value fields: support for multi-value fields, allowing you to query fields as comma-separated values.
- Cloud native on Kubernetes: Helm chart provides a horizontally scalable, fault-tolerant clustered deployment that is easy to manage with Kubernetes.
New version of Apache Pinot
It's worth mentioning that version 1.0 of Apache Pinot was recently released , which basically summarized a lot of work to stabilize the codebase and take into account the wishes of the community (more than 300 comments were considered).
In addition, it's worth noting that the new Multi-Stage Query Engine has reached its full potential , enabling support for table merges (JOINs). The engine initially used performed excellently with simple filtering and aggregation operations, but to ensure predictable query execution times, it did not support table join operations.
The new engine includes intermediate stages for processing complex queries, and its SQL semantics are close to ANSI SQL . Furthermore, the new version offers native support for processing JSON data, provides support for the "NULL" value, integrates with Apache Spark 3.x, and improves the implementation of tables in Upsert mode (adding segment compression and providing support for delete operations).
Finally, if you're interested in learning more , you should know that the project's code is written in Java and distributed under the Apache license. You can find details about the new version at the following link.