7 Pandas Tricks for Time-Series Feature Engineering
Image by Editor | ChatGPT
Introduction
Feature engineering is one of the most important steps when it comes to building effective machine learning models, and this is no less important when dealing with time-series data. By being able to create meaningful features from temporal data, you can unlock predictive power that is unavailable when applied to raw timestamps alone.
Fortunately for us all, Pandas offers a powerful and flexible set of operations for manipulating and creating time-series features.
This article will explore 7 practical Pandas tricks that can help transform your time-series data, which can help lead to enhanced models and more powerful prediction. We will use a simple, synthetic dataset to illustrate each technique, allowing you to quickly grasp the concepts and apply them to your own projects.
Setting Up Our Data
First, let’s create a sample time-series DataFrame. This dataset will represent daily sales data over a period of time, which we’ll use for all subsequent examples.
|
import pandas as pd import numpy as np
# Set a random seed for reproducibility np.random.seed(42)
# Create a date range date_range = pd.date_range(start=‘2025-07-01’, end=‘2025-07-30’, freq=‘D’)
# Create a sample DataFrame df = pd.DataFrame(date_range, columns=[‘date’]) df[‘sales’] = np.random.randint(50, 100, size=(len(date_range))) df = df.set_index(‘date’)
print(f“Dataset size: {df.size}”) print(df.head()) |
Output:
|
Dataset size: 30 sales date 2025–07–01 88 2025–07–02 78 2025–07–03 64 2025–07–04 92 2025–07–05 57 |
We have created a small dataset, an entry for each day of July 2025, with a randomly-assigned sales value. Note that your data will look the same as mine above if you use np.random.seed(42).
With our data ready, we can now explore several techniques for creating insightful features.
1. Extracting Datetime Components
One of simplest yet most useful time-series feature engineering techniques is to break down the datetime object into its constituent components. These components can capture seasonality and trends at different granularities (such as day of the week, month of the year, etc.). Pandas makes this really easy with the .dt accessor.
|
# From our earlier section “Extracting Datetime Components” df[‘day_of_week’] = df.index.dayofweek df[‘month’] = df.index.month
# Day of week has a cycle of 7 days df[‘day_of_week_sin’] = np.sin(2 * np.pi * df[‘day_of_week’] / 7) df[‘day_of_week_cos’] = np.cos(2 * np.pi * df[‘day_of_week’] / 7)
# Month has a cycle of 12 months df[‘month_sin’] = np.sin(2 * np.pi * df[‘month’] / 12) df[‘month_cos’] = np.cos(2 * np.pi * df[‘month’] / 12)
print(df.head()) |
Output:
|
sales day_of_week month day_of_week_sin day_of_week_cos month_sin month_cos date 2025–07–01 88 1 7 0.781831 0.623490 –0.5 –0.866025 2025–07–02 78 2 7 0.974928 –0.222521 –0.5 –0.866025 2025–07–03 64 3 7 0.433884 –0.900969 –0.5 –0.866025 2025–07–04 92 4 7 –0.433884 –0.900969 –0.5 –0.866025 2025–07–05 57 5 7 –0.974928 –0.222521 –0.5 –0.866025 |
We now have day of week, day of year, month, quarter, and week of year data points for each of our entries. These new features can help a model learn patterns related to weekly cycles (such as higher sales on weekends) or annual seasonality. A good place to start.
2. Creating Lag Features
Lag features are values from previous time steps. They are essential in time-series forecasting because they represent the state of the system in the past, which is often highly predictive of the future. The shift() method is perfect for this.
|
# Create a lag feature for sales from the previous day df[‘sales_lag_1’] = df[‘sales’].shift(1)
# Create a lag feature for sales from 3 days ago df[‘sales_lag_3’] = df[‘sales’].shift(3)
print(df.head()) |
Output:
|
sales sales_lag_1 sales_lag_3 date 2025–07–01 88 NaN NaN 2025–07–02 78 88.0 NaN 2025–07–03 64 78.0 NaN 2025–07–04 92 64.0 88.0 2025–07–05 57 92.0 78.0 |
Note that our shifting has created a few NaN values at the beginning of the series for obvious reasons, which you’ll need to handle before modeling by either filtering or dropping.
3. Calculating Rolling Window Statistics
Rolling window calculations (also known as moving averages) are helpful for smoothing out short-term fluctuations and highlighting longer-term trends. You can easily calculate various statistics like the mean, median, or standard deviation over a fixed-size window using the rolling() method.
|
# Calculate the 3-day rolling mean of sales df[‘rolling_mean_3’] = df[‘sales’].rolling(window=3).mean()
# Calculate the 3-day rolling standard deviation df[‘rolling_std_3’] = df[‘sales’].rolling(window=3).std()
print(df.head()) |
Output:
|
sales rolling_mean_3 rolling_std_3 date 2025–07–01 88 NaN NaN 2025–07–02 78 NaN NaN 2025–07–03 64 76.666667 12.055428 2025–07–04 92 78.000000 14.000000 2025–07–05 57 71.000000 18.520259 |
These new features can help provide insight into the recent trend and volatility of the series.
4. Generating Expanding Window Statistics
In contrast to a rolling window, an expanding window includes all of the data from the very start of the time series up to the current point in time. This can be useful for capturing statistics which accumulate over time, including running totals and overall averages. This is achieved with the expanding() method.
|
# Calculate the expanding sum of sales df[‘expanding_sum’] = df[‘sales’].expanding().sum()
# Calculate the expanding average of sales df[‘expanding_avg’] = df[‘sales’].expanding().mean()
print(df.head()) |
Output:
|
sales expanding_sum expanding_avg date 2025–07–01 88 88.0 88.000000 2025–07–02 78 166.0 83.000000 2025–07–03 64 230.0 76.666667 2025–07–04 92 322.0 80.500000 2025–07–05 57 379.0 75.800000 |
5. Measuring Time Between Events
Often, the time elapsed since the last event of significance or between consecutive data points can be a desirable feature. You can calculate the difference between consecutive timestamps using diff() on the index.
|
# Our index is daily, so the difference is constant, but this shows the principle df[‘time_since_last’] = df.index.to_series().diff().dt.days
print(df.head()) |
|
sales time_since_last date 2025–07–01 88 NaN 2025–07–02 78 1.0 2025–07–03 64 1.0 2025–07–04 92 1.0 2025–07–05 57 1.0 |
While not exactly useful for our simple regular series, this can become very powerful for irregular time-series data where the time delta varies.
6. Encoding Cyclical Features with Sine/Cosine
Cyclical features like day of the week or month of the year present a problem for machine learning models. This is the case because the end of the cycle (Saturday, day 5, is far from Sunday, day 6, numerically, which can cause confusion). To better handle this, we can transform them into two dimensions using sine and cosine transformations; this preserves the cyclical nature of the relationship.
|
# From our earlier section “Extracting Datetime Components” df[‘day_of_week’] = df.index.dayofweek df[‘month’] = df.index.month
# Day of week has a cycle of 7 days df[‘day_of_week_sin’] = np.sin(2 * np.pi * df[‘day_of_week’] / 7) df[‘day_of_week_cos’] = np.cos(2 * np.pi * df[‘day_of_week’] / 7)
# Month has a cycle of 12 months df[‘month_sin’] = np.sin(2 * np.pi * df[‘month’] / 12) df[‘month_cos’] = np.cos(2 * np.pi * df[‘month’] / 12)
print(df.head()) |
Output:
|
sales day_of_week month day_of_week_sin day_of_week_cos month_sin month_cos date 2025–07–01 88 1 7 0.781831 0.623490 –0.5 –0.866025 2025–07–02 78 2 7 0.974928 –0.222521 –0.5 –0.866025 2025–07–03 64 3 7 0.433884 –0.900969 –0.5 –0.866025 2025–07–04 92 4 7 –0.433884 –0.900969 –0.5 –0.866025 2025–07–05 57 5 7 –0.974928 –0.222521 –0.5 –0.866025 |
This transformation helps models understand that December (month 12) is just as close to January (month 1) as February (month 2) is.
7. Creating Interaction Features
Finally, let’s take a look at how we can create interacting features by combining two or more existing features, which can help capture more complex relationships. For example, a model might benefit from knowing if it’s a “weekday morning” versus a “weekend morning.”
|
# From our earlier section “Calculating Rolling Window Statistics” df[‘rolling_mean_3’] = df[‘sales’].rolling(window=3).mean()
# A feature for the difference between a day’s sales and the 3-day rolling average df[‘sales_vs_rolling_mean’] = df[‘sales’] – df[‘rolling_mean_3’]
print(df.head()) |
Output:
|
sales rolling_mean_3 sales_vs_rolling_mean date 2025–07–01 88 NaN NaN 2025–07–02 78 NaN NaN 2025–07–03 64 76.666667 –12.666667 2025–07–04 92 78.000000 14.000000 2025–07–05 57 71.000000 –14.000000 |
The possibilities for such interacting features are limitless. The greater your domain knowledge and creativity, the more insightful these features can become.
Wrapping Up
Time-series feature engineering is equal parts art and science. Domain expertise is undeniably invaluable, but so is a strong command of tools like Pandas to help provide the foundation for creating features that can help boost model performance and ultimately solve problems.
The seven tricks covered here — from extracting datetime components to creating complex interactions — are powerful building blocks for any time-series analysis or forecasting task. By taking advantage of Pandas and its powerful time-series capabilities, you can more effectively uncover the hidden patterns within your temporal data.