Hey everyone! As I’m working through my ML fundamentals, I wanted to share a deep-dive Exploratory Data Analysis (EDA) I just completed on the classic Ames, Iowa Housing Dataset.
If you are unfamiliar, the Ames dataset is basically the "final boss" version of the Boston Housing dataset. It has 2,930 residential properties and 80+ explanatory variables.
Instead of just running a standard df.describe(), I wanted to focus on Feature Engineering and understanding the actual business logic behind the data. Here are my biggest takeaways and a few counter-intuitive findings!
🛠️ 1. Smarter Feature Engineering
When I first looked at the data, the square footage was split across multiple columns (basement, 1st floor, 2nd floor). Instead of feeding the model raw variables, I created a few composite features:
-
Total_Usable_SF: I combined Above-Grade Living Area and Total Basement square footage. This gave a much more realistic picture of the home's total usable space. -
Price_Per_SqFt: By dividing the Sale Price by my newTotal_Usable_SF, I created a normalized metric. This made comparing different neighborhoods way easier, removing the bias of house size. -
Log Transformation: The
Sale_Pricewas heavily right-skewed (a few multi-million dollar mansions were dragging the tail). Applying a logarithmic transformation (np.log1p) instantly normalized the distribution, which is crucial for linear modeling assumptions.
# Normalizing the target variable
import numpy as np
import seaborn as sns
df['Log_Sale_Price'] = np.log1p(df['Sale_Price'])
sns.histplot(df['Log_Sale_Price'], kde=True)
📊 2. The Power of "Missing" Data
Usually, missing data is annoying. But in housing, "NaN" rarely means the data is missing—it usually means the house structurally lacks that feature.
I converted missing values for things like Pools, Fences, and Garages into binary presence/absence flags.
- Lacking a Basement or Garage carried a massive median price penalty of 65% to 85%+. In a climate like Iowa, homes without these are almost exclusively low-end starter tiers.
🤯 3. Counter-Intuitive Correlations
This was my favorite part of the analysis. Sometimes, features that sound like upgrades actually drag the price down.
For example, having a Fence or Alley access actually correlated with a lower median sale price. Why? It turns out this is a proxy for age and location. Fences and alleys are incredibly common in the older, denser urban tracts of Ames. The expensive, newly built golf-course developments don't have alleys at all!
🤔 My Question for the Community: Extreme Outliers
I ran into one dilemma: Pools.
Pools definitely exhibited a price premium, but they were present in only 13 out of 2,930 homes (0.44%). They are an ultra-luxury edge case.
How do you typically handle features that exist in less than 1% of the dataset?
Do you keep them as binary flags because of their predictive power on high-end outliers, or do you drop them completely due to sparsity before feeding them into a model like XGBoost?
Let me know your thoughts in the comments!
If you want to see the full code, you can check out my project repo on GitHub!
https://github.com/deepakdoriya/ames-housing-eda.git
