When 8GB RAM Isn't Enough for a 2GB CSV
I watched a production data pipeline crash at 3am because pandas tried to load a 2GB CSV into memory and needed 16GB. That 8x memory multiplier isn't a bug—it's pandas parsing strings into Python objects, inferring dtypes, and building indexes. The error message was almost poetic in its simplicity:
MemoryError: Unable to allocate 12.4 GiB for an array with shape (1662382104,) and data type float64
The knee-jerk fix was "just add more RAM" but on a container with 8GB limits, that wasn't happening. Here's what actually works when you need to process CSVs larger than your available memory.
The Real Memory Cost of pd.read_csv()
Before jumping to solutions, let's understand why pandas uses so much memory. A 2GB CSV doesn't become 2GB in memory—it becomes much more because:
- String columns become Python objects (massive overhead per cell)
- Integers default to int64 even if int8 would suffice
- Pandas builds an index even when you don't need it
Continue reading the full article on TildAlice
