Polars Primer 03: Descriptive Analysis
Retrieving Information from Polars Data
All code snippets use Rent_Contracts.csv as input data, sourced from lemoninabag/Rentals · Datasets at Hugging Face.
Data analysis is a multi-step process that involves tools and techniques, such as computing descriptive statistics, identifying correlations, trends and anomalies, or applying models (regression, clustering and classification) to uncover relationships. At every stage, Polars is a useful tool to have in our data analysis toolkit.
When it comes to descriptive statistics, our goal generally speaking are:
Sort the dataset by one or more columns
Examine extreme values
Get a sense of the shape of the dataset in a statistical sense
Sorting the Dataset
With a DataFrame on hand, it’s easy in Python to sort the dataset according to a particular column.
import polars as pl
# Eagerly read the CSV into memory
eager_df = pl.read_csv("Rent_Contracts.csv")
sort_df = eager_df.sort("contract_start_date")
print(sort_df)The Rust equivalent in lazy evaluation mode requires us to be more verbose.
use polars::prelude::*;
fn main() -> PolarsResult<()> {
let rental_lazy = LazyCsvReader::new("Rent_Contracts.csv".into())
.with_has_header(true)
.finish()?;
// Wrap the string in an array and use SortMultipleOptions
let sorted_df = rental_lazy
.sort(
["contract_start_date"],
SortMultipleOptions::default()
)
.collect()?;
// Print the resulting sorted DataFrame
println!("{:?}", sorted_df);
Ok(())
}shape: (1_215_770, 30)
┌───────────────┬───────────────────┬───────────────────┬───────────────────┬───┬────────────────┬────────────────┬───────────────────┬──────────┐
│ contract_id ┆ contract_reg_type ┆ contract_reg_type ┆ contract_start_da ┆ … ┆ tenant_type_id ┆ tenant_type_en ┆ project_name ┆ rooms_en │
│ --- ┆ _id ┆ _en ┆ te ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ --- ┆ --- ┆ --- ┆ ┆ f64 ┆ str ┆ str ┆ str │
│ ┆ i64 ┆ str ┆ str ┆ ┆ ┆ ┆ ┆ │
╞═══════════════╪═══════════════════╪═══════════════════╪═══════════════════╪═══╪════════════════╪════════════════╪═══════════════════╪══════════╡
│ CNT152225829 ┆ 2 ┆ Renew ┆ 2013-02-25 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ 1 B/R │
│ CRT1182225816 ┆ 1 ┆ New ┆ 2013-06-20 ┆ … ┆ 1.0 ┆ Person ┆ GRANDEUR ┆ 4 B/R │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ RESIDENCES ┆ │
│ CNT209080015 ┆ 1 ┆ New ┆ 2014-02-01 ┆ … ┆ 2.0 ┆ Authority ┆ null ┆ 9 B/R │
│ CNT345228436 ┆ 1 ┆ New ┆ 2014-02-01 ┆ … ┆ 2.0 ┆ Authority ┆ null ┆ None B/R │
│ CNT208147537 ┆ 1 ┆ New ┆ 2014-02-01 ┆ … ┆ 2.0 ┆ Authority ┆ null ┆ None B/R │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ CNT2126929158 ┆ 2 ┆ Renew ┆ 2025-12-26 ┆ … ┆ 1.0 ┆ Person ┆ DAMAC HEIGHTS ┆ 1 B/R │
│ CNT2128949309 ┆ 2 ┆ Renew ┆ 2025-12-28 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ 2 B/R │
│ CRT2128114436 ┆ 1 ┆ New ┆ 2025-12-31 ┆ … ┆ 1.0 ┆ Person ┆ Azizi Riviera 35 ┆ Studio │
│ CNT1786418853 ┆ 2 ┆ Renew ┆ 2025-12-31 ┆ … ┆ 1.0 ┆ Person ┆ Discovery Gardens ┆ 1 B/R │
│ CNT2126820013 ┆ 2 ┆ Renew ┆ 2025-12-31 ┆ … ┆ 1.0 ┆ Person ┆ Meydan ┆ 2 B/R │
└───────────────┴───────────────────┴───────────────────┴───────────────────┴───┴────────────────┴────────────────┴───────────────────┴──────────┘By default the rows are sorted in increasing order, unless we specify the opposite by means of the descending parameter: sort_df = eager_df.sort("contract_start_date", descending = True). To achieve the same with Rust, we chain the with_order_descending() operation to the sorting option structure.
// Wrap the string in an array and use SortMultipleOptions
let sorted_df = rental_lazy
.sort(
["contract_start_date"],
SortMultipleOptions::default().with_order_descending(true)
)
.collect()?;Can we sort by more than one dataset column? Yes, in Python this simply involves passing the column names to sort().
import polars as pl
eager_df = pl.read_csv("Rent_Contracts.csv")
print(eager_df.sort("contract_start_date", "contract_id"))use polars::prelude::*;
fn main() -> PolarsResult<()> {
let rental_lazy = LazyCsvReader::new("Rent_Contracts.csv".into())
.with_has_header(true)
.finish()?;
let sorted_df = rental_lazy
.sort_by_exprs(
[
col("contract_start_date"),
col("contract_id")
],
SortMultipleOptions::default()
.with_order_descending_multi([false, false])
)
.collect()?;
// Print the resulting DataFrame
println!("{:?}", sorted_df);
Ok(())
}To handle multiple columns, we switch from the Rust sort() method to sort_by_exprs(), passing a collection gof col() expressions specifying the column names.
On that note, Python’s descending is similar to Rust’s with_order_descending_multi() in that both receive a collection of booleans that affect the sorting order of each corresponding column. So these two method calls are similar:
... ,descending = [True, False]... .with_order_descending_multi([true, false])How does Polars sort data that can have null or empty values? For example, a column with names (string data type) can hold empty data. When Polars encounters missing values in a dataset, it maps these to null. By default Polars orders nulls as less-than actual values; therefore, all null data appear at the head of the ascending rows.
import polars as pl
eager_df = pl.read_csv("Rent_Contracts.csv")
sort_df = eager_df.sort("project_name")
print(sort_df)shape: (1_215_770, 30)
┌───────────────┬───────────────────┬───────────────────┬───────────────────┬───┬────────────────┬────────────────┬───────────────────┬──────────┐
│ contract_id ┆ contract_reg_type ┆ contract_reg_type ┆ contract_start_da ┆ … ┆ tenant_type_id ┆ tenant_type_en ┆ project_name ┆ rooms_en │
│ --- ┆ _id ┆ _en ┆ te ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ --- ┆ --- ┆ --- ┆ ┆ f64 ┆ str ┆ str ┆ str │
│ ┆ i64 ┆ str ┆ str ┆ ┆ ┆ ┆ ┆ │
╞═══════════════╪═══════════════════╪═══════════════════╪═══════════════════╪═══╪════════════════╪════════════════╪═══════════════════╪══════════╡
│ CNT2128949309 ┆ 2 ┆ Renew ┆ 2025-12-28 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ 2 B/R │
│ CNT2126854343 ┆ 2 ┆ Renew ┆ 2025-12-25 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ Studio │
│ CNT2126250649 ┆ 2 ┆ Renew ┆ 2025-12-25 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ 2 B/R │
│ CNT2128096830 ┆ 2 ┆ Renew ┆ 2025-12-25 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ 1 B/R │
│ CNT2127546869 ┆ 2 ┆ Renew ┆ 2025-12-20 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ Studio │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ CNT2051701837 ┆ 1 ┆ New ┆ 2023-01-10 ┆ … ┆ 1.0 ┆ Person ┆ joya verde ┆ Studio │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ residences dubai ┆ │
│ CNT2057131469 ┆ 2 ┆ Renew ┆ 2023-01-08 ┆ … ┆ 1.0 ┆ Person ┆ joya verde ┆ 2 B/R │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ residences dubai ┆ │
│ CRT2062946667 ┆ 1 ┆ New ┆ 2023-01-05 ┆ … ┆ 1.0 ┆ Person ┆ joya verde ┆ Studio │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ residences dubai ┆ │
│ CRT2050285077 ┆ 1 ┆ New ┆ 2023-01-05 ┆ … ┆ null ┆ null ┆ joya verde ┆ 1 B/R │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ residences dubai ┆ │
│ CRT1443296356 ┆ 2 ┆ Renew ┆ 2020-03-04 ┆ … ┆ null ┆ null ┆ joya verde ┆ 1 B/R │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ residences dubai ┆ │
└───────────────┴───────────────────┴───────────────────┴───────────────────┴───┴────────────────┴────────────────┴───────────────────┴──────────┘To reverse this ordering, we need to explicitly flags null_last=True (Python) and with_nulls_last(true) as the following examples show.
sorted_df = polars_data.sort("project_name", nulls_last=True)use polars::prelude::*;
fn main() -> PolarsResult<()> {
let rental_lazy = LazyCsvReader::new("Rent_Contracts.csv".into())
.with_has_header(true)
.finish()?;
// Wrap the string in an array and use SortMultipleOptions
let sorted_df = rental_lazy
.sort(
["contract_start_date"],
SortMultipleOptions::default().with_nulls_last(true)
)
.collect()?;
// Print the resulting sorted DataFrame
println!("{:?}", sorted_df);
Ok(())
}shape: (1_215_770, 30)
┌───────────────┬────────────────────┬────────────────────┬───────────────────┬───┬────────────────┬────────────────┬───────────────────┬──────────┐
│ contract_id ┆ contract_reg_type_ ┆ contract_reg_type_ ┆ contract_start_da ┆ … ┆ tenant_type_id ┆ tenant_type_en ┆ project_name ┆ rooms_en │
│ --- ┆ id ┆ en ┆ te ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ --- ┆ --- ┆ --- ┆ ┆ f64 ┆ str ┆ str ┆ str │
│ ┆ i64 ┆ str ┆ str ┆ ┆ ┆ ┆ ┆ │
╞═══════════════╪════════════════════╪════════════════════╪═══════════════════╪═══╪════════════════╪════════════════╪═══════════════════╪══════════╡
│ CNT152225829 ┆ 2 ┆ Renew ┆ 2013-02-25 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ 1 B/R │
│ CRT1182225816 ┆ 1 ┆ New ┆ 2013-06-20 ┆ … ┆ 1.0 ┆ Person ┆ GRANDEUR ┆ 4 B/R │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ RESIDENCES ┆ │
│ CNT209080015 ┆ 1 ┆ New ┆ 2014-02-01 ┆ … ┆ 2.0 ┆ Authority ┆ null ┆ 9 B/R │
│ CNT345228436 ┆ 1 ┆ New ┆ 2014-02-01 ┆ … ┆ 2.0 ┆ Authority ┆ null ┆ None B/R │
│ CNT208147537 ┆ 1 ┆ New ┆ 2014-02-01 ┆ … ┆ 2.0 ┆ Authority ┆ null ┆ None B/R │
│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │
│ CNT2126929158 ┆ 2 ┆ Renew ┆ 2025-12-26 ┆ … ┆ 1.0 ┆ Person ┆ DAMAC HEIGHTS ┆ 1 B/R │
│ CNT2128949309 ┆ 2 ┆ Renew ┆ 2025-12-28 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ 2 B/R │
│ CRT2128114436 ┆ 1 ┆ New ┆ 2025-12-31 ┆ … ┆ 1.0 ┆ Person ┆ Azizi Riviera 35 ┆ Studio │
│ CNT1786418853 ┆ 2 ┆ Renew ┆ 2025-12-31 ┆ … ┆ 1.0 ┆ Person ┆ Discovery Gardens ┆ 1 B/R │
│ CNT2126820013 ┆ 2 ┆ Renew ┆ 2025-12-31 ┆ … ┆ 1.0 ┆ Person ┆ Meydan ┆ 2 B/R │
└───────────────┴────────────────────┴────────────────────┴───────────────────┴───┴────────────────┴────────────────┴───────────────────┴──────────┘Highlighting Extreme values
Oftentimes we want to quickly find only the rows that have the largest or smallest value in a column of interest. Both Polars Python and Rust APIs provide the top_k() and bottom_k() for this.
import polars as pl
eager_df = pl.read_csv("Rentals_Contracts.csv")
top_df = eager_df.top_k(5, by = "contract_start_date") # or polars_data.bottom_k(5, by = "column_name")
print(top_df)use polars::prelude::*;
fn main() -> PolarsResult<()> {
let rental_lazy = LazyCsvReader::new("Rent_Contracts.csv".into())
.with_has_header(true)
.finish()?;
// Fetch the top 5 rows based on "contract_start_date" (newest contracts)
let top_5_df = rental_lazy
.top_k(
5, // k: How many rows to keep
[col("contract_start_date")], // by: The column(s) to evaluate
SortMultipleOptions::default() // options: How to handle ties/nulls
)
.collect()?;
println!("{:?}", top_5_df);
Ok(())
}┌─────────┬─────────┬─────────┬────────┬───┬────────┬────────┬────────┬────────┐
│ contrac ┆ contrac ┆ contrac ┆ contra ┆ … ┆ tenant ┆ tenant ┆ projec ┆ rooms_ │
│ t_id ┆ t_reg_t ┆ t_reg_t ┆ ct_sta ┆ ┆ _type_ ┆ _type_ ┆ t_name ┆ en │
│ --- ┆ ype_id ┆ ype_en ┆ rt_dat ┆ ┆ id ┆ en ┆ --- ┆ --- │
│ str ┆ --- ┆ --- ┆ e ┆ ┆ --- ┆ --- ┆ str ┆ str │
│ ┆ i64 ┆ str ┆ --- ┆ ┆ f64 ┆ str ┆ ┆ │
│ ┆ ┆ ┆ str ┆ ┆ ┆ ┆ ┆ │
╞═════════╪═════════╪═════════╪════════╪═══╪════════╪════════╪════════╪════════╡
│ CRT2128 ┆ 1 ┆ New ┆ 2025-1 ┆ … ┆ 1.0 ┆ Person ┆ Azizi ┆ Studio │
│ 114436 ┆ ┆ ┆ 2-31 ┆ ┆ ┆ ┆ Rivier ┆ │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ a 35 ┆ │
│ CNT2126 ┆ 2 ┆ Renew ┆ 2025-1 ┆ … ┆ 1.0 ┆ Person ┆ Meydan ┆ 2 B/R │
│ 820013 ┆ ┆ ┆ 2-31 ┆ ┆ ┆ ┆ ┆ │
│ CNT1786 ┆ 2 ┆ Renew ┆ 2025-1 ┆ … ┆ 1.0 ┆ Person ┆ Discov ┆ 1 B/R │
│ 418853 ┆ ┆ ┆ 2-31 ┆ ┆ ┆ ┆ ery ┆ │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ Garden ┆ │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ s ┆ │
│ CNT2128 ┆ 2 ┆ Renew ┆ 2025-1 ┆ … ┆ 1.0 ┆ Person ┆ null ┆ 2 B/R │
│ 949309 ┆ ┆ ┆ 2-28 ┆ ┆ ┆ ┆ ┆ │
│ CNT2126 ┆ 2 ┆ Renew ┆ 2025-1 ┆ … ┆ 1.0 ┆ Person ┆ DAMAC ┆ 1 B/R │
│ 929158 ┆ ┆ ┆ 2-26 ┆ ┆ ┆ ┆ HEIGHT ┆ │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ S ┆ │
└─────────┴─────────┴─────────┴────────┴───┴────────┴────────┴────────┴────────┘Behind the scenes Polars maintains a min-heap which is a tree-based data structure that always keeps the smallest value at its root node. At any time, the tree only has K nodes keyed to the contract_start_date value. In our example above, K=5, a very small number compared to reading in all the 1,215,770 rows in Rent_Contracts.csv.
As Polars streams through the dataset, it compares each contract_start_date it scans with the value at the root of the min-heap.
If the date read is earlier (smaller value), it is discarded because it does not belong in the top 5 later (larger value) dates
If the date read is later (larger value), it replaces the date at the top of the heap. The heap then sorts the 5-node tree and places the earliest date at the root node.
The table of rows displayed above results from traversing the min-heap tree, starting from the latest date (the maximum value) to the earliest one to the 5th-latest one. Polars’s heap sorting is a quick, with a running time of O(N log K), much faster than O(N log N) for sorting all the N rows of the dataset.
While the top_k() min-heap is used both lazy and eager execution modes, there is a crucial distinction in the pipeline stage where the algorithm is applied.
Eager Mode Execution. The entire data stream is read into memory first before Polars begins to filter the rows.
Lazy Mode Execution. Polars delegates the min-heap tree algorithm to the I/O stage. Eventually only the K rows of data are read into memory.
Generating Statistical Overview of the Dataset
Python’s describe() method supplies descriptive statistics for any column of the datastream. Not only does it tallies the number of null and non-null rows, the function also returns the mean and standard deviation, the three quartiles, and also the min and the max values of a column. One way of using the describe() in Python is to chain it with the Polars select() method.
import polars as pl
eager_df = pl.read_csv("Rent_Contracts.csv")
summary_df = eager_df.select("tenant_type_id").describe()
print(summary_df)shape: (9, 2)
┌────────────┬────────────────┐
│ statistic ┆ tenant_type_id │
│ --- ┆ --- │
│ str ┆ f64 │
╞════════════╪════════════════╡
│ count ┆ 1.128028e6 │ <-- Total non-NULL rows
│ null_count ┆ 87742.0 │
│ mean ┆ 1.094641 │
│ std ┆ 0.292719 │
│ min ┆ 1.0 │
│ 25% ┆ 1.0 │
│ 50% ┆ 1.0 │
│ 75% ┆ 1.0 │
│ max ┆ 2.0 │
└────────────┴────────────────┘
When we take also do without the select() operation. In this case, the Python code returns information about every single column in the datastream.
shape: (9, 31)
┌───────────┬───────────┬───────────┬───────────┬───┬───────────┬──────────┬──────────┬──────────┐
│ statistic ┆ contract_ ┆ contract_ ┆ contract_ ┆ … ┆ tenant_ty ┆ tenant_t ┆ project_ ┆ rooms_en │
│ --- ┆ id ┆ reg_type_ ┆ reg_type_ ┆ ┆ pe_id ┆ ype_en ┆ name ┆ --- │
│ str ┆ --- ┆ id ┆ en ┆ ┆ --- ┆ --- ┆ --- ┆ str │
│ ┆ str ┆ --- ┆ --- ┆ ┆ f64 ┆ str ┆ str ┆ │
│ ┆ ┆ f64 ┆ str ┆ ┆ ┆ ┆ ┆ │
╞═══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪══════════╪══════════╪══════════╡
│ count ┆ 1215770 ┆ 1.21577e6 ┆ 1215770 ┆ … ┆ 1.128028e ┆ 1128028 ┆ 608093 ┆ 1215770 │
│ ┆ ┆ ┆ ┆ ┆ 6 ┆ ┆ ┆ │
│ null_coun ┆ 0 ┆ 0.0 ┆ 0 ┆ … ┆ 87742.0 ┆ 87742 ┆ 607677 ┆ 0 │
│ t ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ │
│ mean ┆ null ┆ 1.595325 ┆ null ┆ … ┆ 1.094641 ┆ null ┆ null ┆ null │
│ std ┆ null ┆ 0.490829 ┆ null ┆ … ┆ 0.292719 ┆ null ┆ null ┆ null │
│ min ┆ CNT101396 ┆ 1.0 ┆ New ┆ … ┆ 1.0 ┆ Authorit ┆ DAMAC ┆ 1 B/R │
│ ┆ 4269 ┆ ┆ ┆ ┆ ┆ y ┆ HILLS ┆ │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ (2) - ┆ │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ SANCTNAR ┆ │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ Y ┆ │
│ 25% ┆ null ┆ 1.0 ┆ null ┆ … ┆ 1.0 ┆ null ┆ null ┆ null │
│ 50% ┆ null ┆ 2.0 ┆ null ┆ … ┆ 1.0 ┆ null ┆ null ┆ null │
│ 75% ┆ null ┆ 2.0 ┆ null ┆ … ┆ 1.0 ┆ null ┆ null ┆ null │
│ max ┆ CRT997365 ┆ 2.0 ┆ Renew ┆ … ┆ 2.0 ┆ Person ┆ joya ┆ Studio │
│ ┆ 666 ┆ ┆ ┆ ┆ ┆ ┆ verde ┆ │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ residenc ┆ │
│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ es dubai ┆ │
└───────────┴───────────┴───────────┴───────────┴───┴───────────┴──────────┴──────────┴──────────┘There is no direct equivalent describe() method in the Rust Polars API. This is true for both eager and lazy execution modes. That is, as of the Polars crate 0.54.0, there is no decribe feature that we can add to Cargo.toml. The Python describe() is a convenience function that calls into the low-level Rust engine, assembling the data as a bundle.
Going back to the single-column Python example for tenant_type_id, the code below shows the corresponding Rust calculations that, together, summarize the data in roughly the same way for that column.
use polars::prelude::*;
fn main() -> PolarsResult<()> {
let rental_lazy = LazyCsvReader::new("Rent_Contracts.csv".into())
.with_has_header(true)
.finish()?;
let summary_stats = rental_lazy
.select([
col("tenant_type_id").count().alias("count"), // Non-null values
col("tenant_type_id").null_count().alias("null_count"), // Missing values
col("tenant_type_id").min().alias("min"),
col("tenant_type_id").mean().alias("mean"),
col("tenant_type_id").max().alias("max")
])
.collect()?;
println!("{:?}", summary_stats);
Ok(())
}shape: (1, 5)
┌─────────┬────────────┬─────┬──────────┬─────┐
│ count ┆ null_count ┆ min ┆ mean ┆ max │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ u32 ┆ u32 ┆ f64 ┆ f64 ┆ f64 │
╞═════════╪════════════╪═════╪══════════╪═════╡
│ 1128028 ┆ 87742 ┆ 1.0 ┆ 1.094641 ┆ 2.0 │
└─────────┴────────────┴─────┴──────────┴─────┘