Polars Primer 05: Working with Multiple Columns

A look at Polars using Python and Rust

Operations Involving Multiple Columns

All code snippets use Rent_Contracts.csv as input data, sourced from lemoninabag/Rentals · Datasets at Hugging Face.

In earlier chapters we saw how to use pl.col() (or the col() operator in Rust), to apply expressions. The examples we’ve seen so far, whether using select() or with_columns(), manipulate single columns. In this chapter we’ll continue our introduction to Polars expressions, but this time we’ll cover the number of ways to use select(), with_col() and pl.col() with several columns in a Polars statement.

1. Handling Multiple Named Columns

We can take a list of column names, pass that as a parameter to pl.col() and apply an expression to every single one. In the Python example below, we display a DataFrame with two columns. For each one, str.len_chars() calculates the length of each data. Under the hood, Polars transforms the dtype from the original String type to u32.

import polars as pl

eager_df = pl.read_csv("Rent_Contracts.csv")

print(eager_df["contract_id", "contract_reg_type_en"].head(3))
print(eager_df["contract_id", "contract_reg_type_en"].with_columns(pl.col("contract_id", "contract_reg_type_en").str
.len_chars()).head(3))

Here is the Rust version. We need to enable the Polars strings feature in Cargo.toml to compile this program.

use polars::prelude::*;

fn main() -> PolarsResult<()> {
    // 1. Read the CSV lazily (creates a computation graph; does not load into memory yet)
    let lf = LazyCsvReader::new("Rent_Contracts.csv".into())
        .with_has_header(true)
        .finish()?;

    // 2. Select specific columns lazily, limit to 3 rows, and execute with collect()
    // LAZY frames require Expr types, so we wrap them in col()
    let selected_df = lf.clone()
        .select([col("contract_id"), col("contract_reg_type_en")])
        .limit(3) // .limit() is the lazy equivalent of the eager .head()
        .collect()?;

    println!("{:?}", selected_df);

    // 3. Select columns, apply string length mutations, limit, and execute
    let mutated_df = lf
        .select([col("contract_id"), col("contract_reg_type_en")])
        .with_columns([
            col("contract_id").str().len_chars(),
            col("contract_reg_type_en").str().len_chars(),
        ])
        .limit(3)
        .collect()?;

    println!("{:?}", mutated_df);

    Ok(())
}

shape: (3, 2)
┌───────────────┬──────────────────────┐
│ contract_id   ┆ contract_reg_type_en │
│ ---           ┆ ---                  │
│ str           ┆ str                  │
╞═══════════════╪══════════════════════╡
│ CRT2128114436 ┆ New                  │
│ CNT1786418853 ┆ Renew                │
│ CNT2126820013 ┆ Renew                │
└───────────────┴──────────────────────┘
shape: (3, 2)
┌─────────────┬──────────────────────┐
│ contract_id ┆ contract_reg_type_en │
│ ---         ┆ ---                  │
│ u32         ┆ u32                  │
╞═════════════╪══════════════════════╡
│ 13          ┆ 3                    │
│ 13          ┆ 5                    │
│ 13          ┆ 5                    │
└─────────────┴──────────────────────┘

2. Filtering Columns by DTypes

Polars lets us run expressions on several columns selected by dtype. For example, the code below calculates the string length of data that have a Polars String data type. Some of the columns affected include "contract_id", "contract_reg_type_en", "contract_start_date", "tenant_type_en", "project_name" and "rooms_en".

import polars as pl

eager_df = pl.read_csv("Rent_Contracts.csv")

print(eager_df.with_columns(pl.col(pl.String).str.len_chars()).head(5))

use polars::prelude::*;

fn main() -> PolarsResult<()> {

    let mut lf = LazyCsvReader::new("Rent_Contracts.csv".into())
        .with_has_header(true)
        .finish()?;

    let schema = lf.collect_schema()?;
    let string_exprs: Vec<Expr> = schema
        .iter()
        .filter(|(_, dtype)| **dtype == DataType::String)
        .map(|(name, _)| col(name.as_str()).str().len_chars())
        .collect();

    let mutated_df = lf
        .with_columns(string_exprs)
        .limit(5) // The lazy equivalent of .head(5)
        .collect()?;

    println!("{:?}", mutated_df);

    Ok(())
}

The Rust equivalent is interesting.

1. Before processing the data stream, we take a peek at the column headers, revealed to us by collect_schema().

2. In the step where we create string_expr, Polars filters for columns of DataType::String and notifies the Polars engine that the expression as_str()).str().len_chars() will be applied to relevant columns. Once the lazy mode query executes following collect(), Polars constructs a collection of Expr objects associated with an array of u32 values. This is what’s handed to string_exprs variable.

3. When string_exprs is passed to with_columns(), Polars updates the DataFrame, overwrites the original String column with the u32 array and updating the DataFrame metadata accordingly.

shape: (5, 30)
┌────────────┬────────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐
│ contract_i ┆ contract_r ┆ contract_ ┆ contract_ ┆ … ┆ tenant_ty ┆ tenant_ty ┆ project_n ┆ rooms_en │
│ d          ┆ eg_type_id ┆ reg_type_ ┆ start_dat ┆   ┆ pe_id     ┆ pe_en     ┆ ame       ┆ ---      │
│ ---        ┆ ---        ┆ en        ┆ e         ┆   ┆ ---       ┆ ---       ┆ ---       ┆ u32      │
│ u32        ┆ i64        ┆ ---       ┆ ---       ┆   ┆ f64       ┆ u32       ┆ u32       ┆          │
│            ┆            ┆ u32       ┆ u32       ┆   ┆           ┆           ┆           ┆          │
╞════════════╪════════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡
│ 13         ┆ 1          ┆ 3         ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ 16        ┆ 6        │
│ 13         ┆ 2          ┆ 5         ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ 17        ┆ 5        │
│ 13         ┆ 2          ┆ 5         ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ 6         ┆ 5        │
│ 13         ┆ 2          ┆ 5         ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ null      ┆ 5        │
│ 13         ┆ 2          ┆ 5         ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ 13        ┆ 5        │
└────────────┴────────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘

As these two example show, working with the Rust API requires more low-level operations to achieve the same result.

Along with pl.String, there are other dtype selectors we can use, including pl.Int64, plFloat64 or pl.Boolean to name a few.

3. Excluding Named Columns

Sometimes its easier to explicity exclude columns from the expression. Continuing with the prior example, let’s say we want to calculate the string length for of all String type columns, except for the contract_id and the contract_reg_type_en.

import polars as pl

eager_df = pl.read_csv("Rent_Contracts.csv")

print(eager_df.with_columns(pl.col(pl.String).exclude("contract_id", "contract_reg_type_en").str.len_c
hars()).head(5))

Here we use pl.col(pl.String) to limit the str.len_chars() expression only to columns of String dtype, to the exclusion of the two columns we marked with exclude().

Similarly, we extend the previous Rust code. We simply add a boolean exclusion to the filter() closure, precluding the contract_id and contract_reg_type_en columns from the Polars execution plan.

use polars::prelude::*;

fn main() -> PolarsResult<()> {
    let mut lf = LazyCsvReader::new("Rent_Contracts.csv".into())
        .with_has_header(true)
        .finish()?;

    let schema = lf.collect_schema()?;

    // Rust's dynamic equivalent of pl.exclude()
    let string_exprs: Vec<Expr> = schema
        .iter()
        .filter(|(name, dtype)| {
            // Keep the column ONLY if it is a String AND NOT in our exclusion list
            **dtype == DataType::String
                && name.as_str() != "contract_id"
                && name.as_str() != "contract_reg_type_en"
        })
        .map(|(name, _)| col(name.as_str()).str().len_chars())
        .collect();

    let mutated_df = lf
        .with_columns(string_exprs)
        .limit(5) // The lazy equivalent of .head(5)
        .collect()?;

    println!("{:?}", mutated_df);

    Ok(())
}

shape: (5, 30)
┌────────────┬────────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬──────────┐
│ contract_i ┆ contract_r ┆ contract_ ┆ contract_ ┆ … ┆ tenant_ty ┆ tenant_ty ┆ project_n ┆ rooms_en │
│ d          ┆ eg_type_id ┆ reg_type_ ┆ start_dat ┆   ┆ pe_id     ┆ pe_en     ┆ ame       ┆ ---      │
│ ---        ┆ ---        ┆ en        ┆ e         ┆   ┆ ---       ┆ ---       ┆ ---       ┆ u32      │
│ str        ┆ i64        ┆ ---       ┆ ---       ┆   ┆ f64       ┆ u32       ┆ u32       ┆          │
│            ┆            ┆ str       ┆ u32       ┆   ┆           ┆           ┆           ┆          │
╞════════════╪════════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪══════════╡
│ CRT2128114 ┆ 1          ┆ New       ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ 16        ┆ 6        │
│ 436        ┆            ┆           ┆           ┆   ┆           ┆           ┆           ┆          │
│ CNT1786418 ┆ 2          ┆ Renew     ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ 17        ┆ 5        │
│ 853        ┆            ┆           ┆           ┆   ┆           ┆           ┆           ┆          │
│ CNT2126820 ┆ 2          ┆ Renew     ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ 6         ┆ 5        │
│ 013        ┆            ┆           ┆           ┆   ┆           ┆           ┆           ┆          │
│ CNT2128949 ┆ 2          ┆ Renew     ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ null      ┆ 5        │
│ 309        ┆            ┆           ┆           ┆   ┆           ┆           ┆           ┆          │
│ CNT2126929 ┆ 2          ┆ Renew     ┆ 10        ┆ … ┆ 1.0       ┆ 6         ┆ 13        ┆ 5        │
│ 158        ┆            ┆           ┆           ┆   ┆           ┆           ┆           ┆          │
└────────────┴────────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴──────────┘

Polars Selectors

Apart from pl.col(), or col() in Rust, combined with with_columns(), Polars Selectors offer another way to operate on several columns simultaneously.

A Selector applies a search query on columns in a Polars table. Instead of matching by the specific name of a column ("actual_area"), Selectors allow us to apply operations to a group of columns that match a criteria.

Generally, Selectors allow us to filter columns based on three categories:

1. The data type of a column

2. The name pattern of a column

3. The use of set operations

1. Selecting by Column Data Type

In the examples below, the Polars Selector applies the round() operator only to floating point columns.

import polars as pl
import polars.selectors as cs

rental_lazy = pl.scan_csv("Rent_Contracts.csv")

result = (
    rental_lazy
    .select(
        cs.float().round(2)
    )
    .limit(5)
    .collect()
)

print(result)

With the Rust API, we have to specify the float types we’re interested in.

use polars::prelude::*;

fn main() -> PolarsResult<()> {
    let rental_lazy = LazyCsvReader::new("Rent_Contracts.csv".into())
        .with_has_header(true)
        .finish()?;

    let result = rental_lazy
        .select([
            dtype_cols([DataType::Float32, DataType::Float64])
                .as_selector() // 1. Convert DataTypeSelector into a generic Selector
                .as_expr()     // 2. Convert the Selector into an Expression
                .round(2, RoundMode::HalfAwayFromZero)
        ])
        .limit(5)
        .collect()?;

    println!("{:?}", result);
    Ok(())
}

shape: (5, 7)
┌───────────┬───────────┬───────────┬──────────┬─────────┬──────────┬──────────┐
│ is_free_h ┆ ejari_bus ┆ ejari_pro ┆ project_ ┆ area_id ┆ actual_a ┆ tenant_t │
│ old       ┆ _property ┆ perty_typ ┆ number   ┆ ---     ┆ rea      ┆ ype_id   │
│ ---       ┆ _type_id  ┆ e_id      ┆ ---      ┆ f64     ┆ ---      ┆ ---      │
│ f64       ┆ ---       ┆ ---       ┆ f64      ┆         ┆ f64      ┆ f64      │
│           ┆ f64       ┆ f64       ┆          ┆         ┆          ┆          │
╞═══════════╪═══════════╪═══════════╪══════════╪═════════╪══════════╪══════════╡
│ 1.0       ┆ 2.0       ┆ 842.0     ┆ 2041.0   ┆ 412.0   ┆ 30.0     ┆ 1.0      │
│ 1.0       ┆ 2.0       ┆ 842.0     ┆ null     ┆ 445.0   ┆ 89.0     ┆ 1.0      │
│ 1.0       ┆ 2.0       ┆ 842.0     ┆ null     ┆ 335.0   ┆ 123.0    ┆ 1.0      │
│ 0.0       ┆ 2.0       ┆ 842.0     ┆ null     ┆ 244.0   ┆ 68.0     ┆ 1.0      │
│ 1.0       ┆ 2.0       ┆ 842.0     ┆ 430.0    ┆ 330.0   ┆ 88.0     ┆ 1.0      │
└───────────┴───────────┴───────────┴──────────┴─────────┴──────────┴──────────┘

2. Selecting by Column Name Pattern

Polars lets us use prefixes, suffixes or regex patterns to filter columns. This is useful when the columns follow uniform naming conventions.

import polars as pl
import polars.selectors as cs

rental_lazy = pl.scan_csv("Rent_Contracts.csv")

result = (
    rental_lazy
    .select(
        cs.starts_with("contract_")
    )
    .limit(5)
    .collect()
)

print(result)

use polars::prelude::*;

fn main() -> PolarsResult<()> {
    let rental_lazy = LazyCsvReader::new("Rent_Contracts.csv".into())
        .with_has_header(true)
        .finish()?;

    let result = rental_lazy
        .select([
            // Regex pattern matching all columns starting with "contract_"
            col("^contract_.*$")
        ])
        .limit(5)
        .collect()?;

    println!("{:?}", result);
    Ok(())
}

shape: (5, 6)
┌───────────────┬───────────────┬──────────────┬──────────────┬──────────────┬──────────────┐
│ contract_id   ┆ contract_reg_ ┆ contract_reg ┆ contract_sta ┆ contract_end ┆ contract_amo │
│ ---           ┆ type_id       ┆ _type_en     ┆ rt_date      ┆ _date        ┆ unt          │
│ str           ┆ ---           ┆ ---          ┆ ---          ┆ ---          ┆ ---          │
│               ┆ i64           ┆ str          ┆ str          ┆ str          ┆ i64          │
╞═══════════════╪═══════════════╪══════════════╪══════════════╪══════════════╪══════════════╡
│ CRT2128114436 ┆ 1             ┆ New          ┆ 2025-12-31   ┆ 2026-12-30   ┆ 53000        │
│ CNT1786418853 ┆ 2             ┆ Renew        ┆ 2025-12-31   ┆ 2030-12-30   ┆ 100000       │
│ CNT2126820013 ┆ 2             ┆ Renew        ┆ 2025-12-31   ┆ 2026-12-30   ┆ 125000       │
│ CNT2128949309 ┆ 2             ┆ Renew        ┆ 2025-12-28   ┆ 2026-12-27   ┆ 62000        │
│ CNT2126929158 ┆ 2             ┆ Renew        ┆ 2025-12-26   ┆ 2026-12-25   ┆ 140000       │
└───────────────┴───────────────┴──────────────┴──────────────┴──────────────┴──────────────┘

3. Set Operations

The Selector operators we’ve seen so far produce subsets of columns from the upstream DataFrame. Polars Selectors also allow us to apply boolean operations on these derived columns, producing new subsets or combinations of data.

Boolean Logic	Goal
Difference	Select everything in Set A, except what is also in Set B
	Python Syntax: - cs.numeric() - cs.starts_with("id_")
	Rust Syntax: exclude() dtype_cols([DataType::Int32]).exclude(["id_col"])
Union or Combination	Select everything in Set A plus everything in Set B
	Python Syntax: \| cs.float() \| cs.starts_with("target_")
	Rust Syntax: Multiple items in the select array select([ dtype_cols([DataType::Float64]), col("^target_.*$") ])
Intersection or Overlap	Select only columns in both Set A and Set B
	Python Syntax: & cs.starts_with("A_") & cs.ends_with("_B")
	Rust Syntax: Leverage Regex, see example below
Symmetric Difference	Select columns in Set A or Set B, but not in both
	Python Syntax: ^ cs.starts_with("A_") ^ cs.ends_with("_B")
	Rust Syntax: Combination of unions and exclusions, see example below.

The examples below show the use of the Difference Selector.

import polars as pl
import polars.selectors as cs

# 1. Initialize the Lazy computation graph
rental_lazy = pl.scan_csv("Rent_Contracts.csv")

# 2. Apply the Difference logic
result_lazy = (
    rental_lazy
    .select(
        # Set A (All Numbers) MINUS Set B (Specific Names)
        cs.numeric() - cs.by_name("contract_id", "no_of_prop")
    )
    .limit(5)
)

# 3. Materialize the data
print(result_lazy.collect())

use polars::prelude::*;

fn main() -> PolarsResult<()> {
    let rental_lazy = LazyCsvReader::new("Rent_Contracts.csv".into())
        .with_has_header(true)
        .finish()?;

    let result_df = rental_lazy
        .select([
            // 1. Start with the highly specific Type query
            dtype_cols([
                DataType::Float32,
                DataType::Float64
            ])
            // 2. UPCAST to a generic Selector to unlock boolean logic
            .as_selector()

            // 3. Perform the Set Difference using the correct Selector method!
            .exclude_cols(["contract_id", "no_of_prop"])

            // 4. MANDATORY: Cast back to an Expr so .select() will accept it
            .as_expr()
        ])
        .limit(5)
        .collect()?;

    println!("{:?}", result_df);

    Ok(())
}

shape: (5, 7)
┌─────────────┬─────────────┬─────────────┬─────────────┬─────────┬────────────┬────────────┐
│ is_free_hol ┆ ejari_bus_p ┆ ejari_prope ┆ project_num ┆ area_id ┆ actual_are ┆ tenant_typ │
│ d           ┆ roperty_typ ┆ rty_type_id ┆ ber         ┆ ---     ┆ a          ┆ e_id       │
│ ---         ┆ e_id        ┆ ---         ┆ ---         ┆ f64     ┆ ---        ┆ ---        │
│ f64         ┆ ---         ┆ f64         ┆ f64         ┆         ┆ f64        ┆ f64        │
│             ┆ f64         ┆             ┆             ┆         ┆            ┆            │
╞═════════════╪═════════════╪═════════════╪═════════════╪═════════╪════════════╪════════════╡
│ 1.0         ┆ 2.0         ┆ 842.0       ┆ 2041.0      ┆ 412.0   ┆ 30.0       ┆ 1.0        │
│ 1.0         ┆ 2.0         ┆ 842.0       ┆ null        ┆ 445.0   ┆ 89.0       ┆ 1.0        │
│ 1.0         ┆ 2.0         ┆ 842.0       ┆ null        ┆ 335.0   ┆ 123.0      ┆ 1.0        │
│ 0.0         ┆ 2.0         ┆ 842.0       ┆ null        ┆ 244.0   ┆ 68.0       ┆ 1.0        │
│ 1.0         ┆ 2.0         ┆ 842.0       ┆ 430.0       ┆ 330.0   ┆ 88.0       ┆ 1.0        │
└─────────────┴─────────────┴─────────────┴─────────────┴─────────┴────────────┴────────────┘

The next examples show how to use the Intersection Selector. Our "Rent_Contracts.csv" has a number of column names that begin with contract_. In the code below, we select the "contract_id"column together with any other columns whose name begins with "contract_" and ends with"amount". To do so we use the bit (&) operator. As it is only one column satisfies the requirements.

more script.py
import polars as pl
import polars.selectors as cs

rental_df = pl.read_csv("Rent_Contracts.csv")

result_df = (
    rental_df
    .select(
        "contract_id",
        # Set A (Starts with "actual_") AND Set B (Ends with "area")
        cs.starts_with("contract_") & cs.ends_with("amount")
    )
    .head(5) # Eager row truncation
    .sort("contract_amount", descending = True)
)

print(result_df)

Rust doesn’t have a convenient & bit-wise AND operation that Python does. Instead we dip into its Regex module to specify the logic behind the intersection.

use polars::prelude::*;

fn main() -> PolarsResult<()> {
    // 1. Initialize the Lazy computation graph
    let rental_lazy = LazyCsvReader::new("Rent_Contracts.csv".into())
        .with_has_header(true)
        .finish()?;

    // 2. Apply the Intersection logic via Regex
    let result_df = rental_lazy
        .select([
            col("contract_id"),
            // "^" asserts the start: must begin with "actual_"
            // ".*" means any characters can be in the middle
            // "$" asserts the end: must end with "area"
            col("^contract_.*amount$")
        ])
        .limit(5) // Lazy row truncation
        .sort(
            ["contract_amount"],
            SortMultipleOptions::default().with_order_descending(true)
        )
        .collect()?;

    println!("{:?}", result_df);

    Ok(())
}

shape: (5, 2)
┌───────────────┬─────────────────┐
│ contract_id   ┆ contract_amount │
│ ---           ┆ ---             │
│ str           ┆ i64             │
╞═══════════════╪═════════════════╡
│ CNT2126929158 ┆ 140000          │
│ CNT2126820013 ┆ 125000          │
│ CNT1786418853 ┆ 100000          │
│ CNT2128949309 ┆ 62000           │
│ CRT2128114436 ┆ 53000           │
└───────────────┴─────────────────┘

These last examples showcase the Symmetric Difference Selector that implements an XOR logic. Here we’re looking for columns that begin either with "area_" or end with "_name" but not both.

import polars as pl
import polars.selectors as cs

# Eagerly read the CSV into memory
df = pl.read_csv("Rent_Contracts.csv")

# Apply the symmetric difference (^) selector
# Drops 'contract_reg_type_en' because it exists in both sets
selected_df = df.select(
    cs.starts_with("area_") ^ cs.ends_with("_name")
).head(3)

print(selected_df)

Note how ^ is idiomatic to the Python polars.selectors module. Because Rust doesn’t have that sort of syntactic sugar, we need to use vectors, iterators and bitwise XOR operator – its raw tools – to filter the schema.

use polars::prelude::*;

fn main() -> PolarsResult<()> {
    // 1. Setup the Lazy computation graph for the CSV
    let mut lf = LazyCsvReader::new("Rent_Contracts.csv".into())
        .with_has_header(true)
        .finish()?;

    // 2. Resolve the schema cheaply without reading the whole file
    let schema = lf.collect_schema()?;

    // 3. Build our "Selector" using native Rust iterators
    let selected_cols: Vec<Expr> = schema
        .iter_names()
        .filter(|name| {
            let starts_contract = name.starts_with("area");
            let ends_en = name.ends_with("_name");

            // Rust's bitwise XOR operator (^) acts as a logical XOR for booleans
            starts_contract ^ ends_en
        })
        .map(|name| col(name.as_str()))
        .collect();

    // 4. Pass the expressions into the LazyFrame and collect (execute)
    let df = lf.select(selected_cols).limit(3).collect()?;

    println!("{:?}", df);

    Ok(())
}

shape: (3, 3)
┌─────────┬────────────────────┬───────────────────┐
│ area_id ┆ area_name_en       ┆ project_name      │
│ ---     ┆ ---                ┆ ---               │
│ f64     ┆ str                ┆ str               │
╞═════════╪════════════════════╪═══════════════════╡
│ 412.0   ┆ Al Merkadh         ┆ Azizi Riviera 35  │
│ 445.0   ┆ Jabal Ali First    ┆ Discovery Gardens │
│ 335.0   ┆ Nad Al Shiba First ┆ Meydan            │
└─────────┴────────────────────┴───────────────────┘