Slicing and Dicing

In the previous section, we successfully loaded the kloten_summer_2022.txt weather dataset. Let us quickly reload it to your local working directory so we have a fresh DataFrame to work with.

1. Selecting Columns¶

When performing data analysis, you rarely need every variable in your dataset. You can extract specific columns by placing the column name inside square brackets next to your DataFrame variable.

To extract a single column, pass the name as a string. This returns a one dimensional Series.

To extract multiple columns, you must pass a list of column names inside the square brackets. This means you will see double brackets [[ ]].

• The outer brackets tell Pandas you are making a selection: data[ ... ]

• The inner brackets define the Python list of strings: ["YEARMODA", "MAX", "MIN"]

Extracting multiple columns returns a smaller DataFrame, rather than a Series.

“Note: We use the display() function here to explicitly print the DataFrame as a cleanly formatted HTML table in our notebook.”

2. Filtering Rows¶

Selecting columns is easy, but how do we select specific rows? In pure Python, you would use an if statement to check each row one by one. In Pandas, we use a concept called Boolean indexing to ask the entire dataset a True or False question simultaneously.

First, we ask the question. Let us find out which days had a maximum temperature greater than 30 degrees Celsius:

This returns a Series of True and False values, often called a mask.

To actually filter the data, we place this conditional mask inside the selection brackets. Pandas will keep every row that evaluates to True and hide every row that evaluates to False.

For absolute clarity, here is how you do it in two steps:

However, experienced programmers usually combine this into a single line of code by placing the condition directly inside the brackets. Notice how the word data appears twice:

Notice the Index on the far left. It jumped from 15 to 18 to 19, preserving the original row labels of the days that met our criteria.

3. Multiple Conditions¶

Often, you need to filter data using multiple criteria simultaneously. You can combine logical conditions using special operators. Standard Python keywords and and or will not work here, as Pandas needs bitwise operators that perform vectorized logical comparisons on entire Series at once.

Pandas uses these operators:

• & (AND): Both conditions must be True.

• | (OR): At least one condition must be True.

AND Condition Example¶

Let us find “extreme” summer days that were both very hot (MAX > 30) but also had surprisingly cool mornings (MIN < 14).

OR Condition Example¶

Next, let us find days that were either extremely hot (MAX > 33) OR had very cold mornings (MIN < 10). This query will include days meeting either criteria, or both.

4. Sorting Data (sort_values)¶

Once you have isolated your data, you frequently want to order it. The .sort_values() method allows you to sort your DataFrame by one or more columns.

By default, Pandas sorts in ascending order (smallest to largest). To find the absolute hottest days of the summer at the top of our table, we need to set the ascending parameter to False.

5. Copy-on-Write: Why Subsets Behave Safely¶

When you filter a DataFrame, pandas tries to be efficient with your computer’s memory. In older versions of pandas, this created a confusing situation: sometimes a subset behaved like a view into the original data, and sometimes it behaved like a fully independent copy.

Starting with pandas 3.0, pandas uses Copy-on-Write by default. This means that a subset behaves like an independent object when you modify it. Behind the scenes, pandas may still share memory to save resources, but if you write to the subset, pandas protects the original DataFrame by copying the data when needed.

This makes pandas more predictable than older versions. You can create a filtered subset and add a new column to that subset without worrying that the original DataFrame will silently change.

Let us filter the data for early June and add a new column:

The new column is added to early_june. The original data DataFrame remains unchanged.

Do We Still Need .copy()?¶

You can still use .copy() when you want to make your intention extra clear:

This is not wrong. It tells readers of your code: “I want this to be an independent object.”

However, in pandas 3.0, .copy() is no longer needed just to avoid the old SettingWithCopyWarning. The old warning existed because pandas could not always clearly tell whether you were modifying a view or a copy. Copy-on-Write removes this ambiguity for everyday use.

The Important Question: What Are You Trying to Change?¶

When writing pandas code, the most important question is not simply “Do I need a copy?” Instead, ask:

If you create a new object, use a new variable name:

If you want to continue working with this subset, you can modify it:

This changes hot_days, not the original data.

If your actual goal is to change the original data DataFrame, we need a more precise selection tool. That tool is introduced in the next section: .loc[].

Concept Check: Separate object or original data?¶

Imagine you are writing a script to analyze weather patterns.

Scenario A: You filter the dataset to find all days where the maximum temperature exceeded 30°C, just so you can inspect them.

Scenario B: You filter the dataset to create a separate hot_days table and add a new column only to that table.

Scenario C: You want to add a new column to the original data DataFrame marking which days were hot.

Which scenario requires a more precise assignment method introduced in the next section?

A) Scenario A

B) Scenario B

C) Scenario C

6. Index based Selection (loc vs iloc)¶

In the previous section, we saw that filtered subsets can safely be used as separate working objects. But sometimes you do not want to create a separate object. Sometimes you want to select exact rows and columns in the original DataFrame and modify them directly.

For this, pandas provides two powerful indexers: .loc[] and .iloc[].

• .loc[] (Label-based): Selects data based on its exact label (the name of the index row or the name of the column).

• .iloc[] (Integer position-based): Selects data based on its strict numerical position (0, 1, 2...), exactly like a standard Python list.

Let us look at the difference. Imagine we sort our data so the hottest day (originally Index 18) is at the very top of the table.

Notice how .iloc[0] returned the data for the hottest day (labeled 18), while .loc[0] hunted down the specific row labeled 0 (June 1st).

Selecting Rows AND Columns¶

The true power of these indexers is that you can select rows and columns at the same time by separating them with a comma: [rows, columns].

7. Exercise: Isolate the Target Data¶

Let us bring all these skills together using the global cities dataset you downloaded in the previous section.

Imagine you are doing an analysis focused solely on major urban centers in Japan. You need to load the data, extract exactly what you need, and secure it in memory.

Tasks:

1. Load worldcities.csv into a DataFrame.

2. Filter the data to include only rows where the country column is exactly "Japan".

3. Crucial: Make sure to append .copy() to create an independent dataset!

4. Sort this new Japan DataFrame by the population column in descending order (largest to smallest).

5. Display the top 5 rows, but only show the city and population columns.

(Hint for Step 5: Remember that selecting multiple columns requires a list inside the selection brackets, which looks like double brackets [[ ]]).

import pandas as pd

# 1. Load the data
cities_df = pd.read_csv("data/worldcities.csv")

# 2 & 3. Filter for Japan AND create a safe copy
japan_cities = cities_df[cities_df["country"] == "Japan"].copy()

# 4. Sort by population descending
japan_sorted = japan_cities.sort_values(by="population", ascending=False)

# 5. Extract specific columns and show the top 5
top_japan_cities = japan_sorted[["city", "population"]]
display(top_japan_cities.head(5))

8. Summary: Navigating the 2D Grid¶

In this section, you learned how to slice and dice your data to extract exactly the information you need. You now have the tools to surgically navigate large datasets without relying on manual for loops.

Key takeaways¶

• Columns: Extract a single Series using df["col"] or a smaller DataFrame using a list df[["col1", "col2"]].

• Rows: Use Boolean Indexing (df[df["col"] == value]) to act as a filter, keeping only rows that meet logical conditions.

• Conditions: Combine multiple filters using & (AND) or | (OR), always wrapping each individual condition in parentheses ().

• Sorting: Use .sort_values(by="col", ascending=False) to order your DataFrame.

• Safety: With pandas 3.0, filtered subsets behave safely through Copy-on-Write.

• Index Selection: Use .iloc[] to select rows based on their strict numerical position, and .loc[] to select based on exact row/column labels.

What comes next?¶

Now that you can navigate, slice, and filter your data, you might notice a glaring issue: real-world data is rarely perfect.

If you try to do math on a column where numbers are accidentally stored as text, or if a sensor went offline and left blank gaps in your dataset, your code will crash. In the next section, Cleaning the Mess, we will learn how to standardize messy column headers, fix text strings, and handle the infamous NaN (Not a Number) so your data is pristine and ready for analysis!