Python mmap: Memory-Mapped File I/O for Speed

tags: python, performance, tutorial, advanced

Python Memory Management: How It Actually Works

You’ve probably seen your Python script spike to 2GB of RAM and wondered: Where did all that memory go? Did you create a leak? Is Python just wasteful? The truth is more interesting—and more fixable—than you might think. Python doesn’t just “throw memory away” when you’re done with an object. Instead, it uses a clever, layered system that balances speed, safety, and automatic cleanup. And once you understand how it works, you can write code that’s not only correct but also lean and fast.

Let’s peel back the layers and see what’s really happening under the hood.

The Private Heap: Python’s Memory Playground

Unlike languages like C where you manually call malloc() and free(), Python manages memory in a private heap—a chunk of memory reserved exclusively for your Python process. Every object you create—lists, dictionaries, strings, even functions—lives in this heap [4][11].

This heap is managed by the Python memory manager, which handles three core tasks:

Allocation: Setting aside space for new objects.
Tracking: Knowing which memory is in use and which is free.
Cleanup: Removing data that’s no longer needed [9].

When your program starts, the memory manager requests a block from the operating system and divides it into memory pools based on object size. Objects ≤ 512 bytes use the object allocator; larger ones go to the raw memory allocator [11]. This tiered approach minimizes fragmentation and speeds up allocation.

Reference Counting: The First Line of Defense

The most immediate cleanup mechanism in Python is reference counting. Every Python object internally tracks how many references point to it [1]. When that count hits zero, the object is destroyed immediately.

import sys

a = [1, 2, 3]
print(sys.getrefcount(a))  # Outputs: 2 (one from 'a', one from getrefcount's internal arg)

b = a
print(sys.getrefcount(a))  # Now: 3

del b
print(sys.getrefcount(a))  # Back to 2

This is why del is powerful: it explicitly removes a reference, potentially triggering immediate cleanup [5]. But reference counting has a blind spot: circular references.

Circular References and Generational Garbage Collection

Imagine two objects that reference each other:

class A:
    def __init__(self):
        self.b = None

class B:
    def __init__(self):
        self.a = None

a = A()
b = B()
a.b = b
b.a = a

del a
del b

Even after deleting both a and b, their reference counts aren’t zero—they still point to each other. Reference counting alone can’t clean this up.

That’s where Python’s generational garbage collector steps in. It uses a mark-and-sweep algorithm:

Mark phase: Starts from root objects (like global variables) and traverses the reference graph, marking all reachable objects [1].
Sweep phase: Removes all unmarked (unreachable) objects, including those in cycles [1].

Objects are grouped into generations (0, 1, 2). New objects start in generation 0. If they survive a collection, they move up. This way, the collector focuses on “younger” objects first, where most cleanup happens [1][6].

You can force a collection manually with gc.collect(), but it’s usually better to let Python decide [5].

Practical Tips: What You Can Do TODAY

Understanding the theory is great, but here’s how to apply it right now:

1. Use Generators Instead of Lists

Lists store all elements in memory at once. Generators yield items one-by-one, drastically reducing memory usage for large datasets [5][7].

# Memory-heavy
data = [x * 2 for x in range(10_000_000)]

# Memory-efficient
data = (x * 2 for x in range(10_000_000))

2. Delete Unused Variables Explicitly

If you’re processing huge files or temporary data, use del to remove references early:

large_df = process_huge_file()
result = analyze(large_df)
del large_df  # Frees memory immediately if no other refs exist

3. Avoid Circular References

Design classes to avoid mutual references. If you need weak links, use weakref:

import weakref

class Parent:
    def __init__(self):
        self.children = []

class Child:
    def __init__(self, parent):
        self.parent = weakref.ref(parent)  # Doesn't increase ref count

4. Use `slots` in Custom Classes

Standard Python classes store attributes in a __dict__, which wastes memory. __slots__ replaces this with a fixed structure:

class Point:
    __slots__ = ['x', 'y']
    def __init__(self, x, y):
        self.x = x
        self.y = y

This can save 40–50% memory per instance for large numbers of objects [8].

5. Profile Your Memory Usage

Don’t guess—measure. Use tools like:

sys.getsizeof() for object sizes
tracemalloc for tracking allocations
memory_profiler for line-by-line analysis [5][8][12]

import tracemalloc
tracemalloc.start()
# ... your code ...
tracemalloc.stop()

Why Memory Doesn’t Always Return to the OS

Here’s a subtle but important point: when Python frees an object, that memory often doesn’t go back to the operating system. Instead, it returns to Python’s internal pool for reuse [4]. This means your script’s RAM usage might stay high even after cleanup.

This is normal behavior for CPython. If you need to return memory to the OS, consider splitting work into separate processes—each process’s memory is freed when it exits [12].

Final Thoughts: Write Smarter, Not Harder

Python’s memory management is a dual system: reference counting for immediate cleanup and garbage collection for cycles. It’s automatic, efficient, and mostly invisible—but not magic.

By using generators, deleting unused variables, avoiding circular refs, and profiling your code, you can write Python that’s not just functional but also memory-conscious. And in data-heavy applications, those small optimizations compound into massive gains.

So go ahead—try one of these tips in your next script. Then come back and share what you learned. Did __slots__ surprise you? Did a generator save you 1GB? Let’s talk in the comments.

If you found this helpful, consider buying me a coffee ☕ — it keeps these articles coming!

Also check out my AI tools collection: AI 次元世界 — free AI tools for developers.

Python mmap: Memory-Mapped File I/O for Speed

Python mmap: Memory-Mapped File I/O for Speed

Python Memory Management: How It Actually Works

The Private Heap: Python’s Memory Playground

Reference Counting: The First Line of Defense

Circular References and Generational Garbage Collection

Practical Tips: What You Can Do TODAY

1. Use Generators Instead of Lists

2. Delete Unused Variables Explicitly

3. Avoid Circular References

4. Use `slots` in Custom Classes

5. Profile Your Memory Usage

Why Memory Doesn’t Always Return to the OS

Final Thoughts: Write Smarter, Not Harder

Tags

Author

Stats

Published

You Might Also Like

I Tried to Beat Peter Norvig and Accidentally Became Ryan Gosling

My Cloud Run Multi-Agent Fleet Passed Its Demo. The Architecture Was Still Wrong.

Badger: An E-Ink Badge I Use For Conferences

Why I Decided to Stop Using Claude Code

Dev log #19 WebRTC v2 flows, agentic orchestration, and a perfectly synced vault

Five bugs in my LLM app that never threw an error

Python mmap: Memory-Mapped File I/O for Speed

Python mmap: Memory-Mapped File I/O for Speed

Python Memory Management: How It Actually Works

The Private Heap: Python’s Memory Playground

Reference Counting: The First Line of Defense

Circular References and Generational Garbage Collection

Practical Tips: What You Can Do TODAY

1. Use Generators Instead of Lists

2. Delete Unused Variables Explicitly

3. Avoid Circular References

4. Use __slots__ in Custom Classes

5. Profile Your Memory Usage

Why Memory Doesn’t Always Return to the OS

Final Thoughts: Write Smarter, Not Harder

Tags

Author

Stats

Published

You Might Also Like

I Tried to Beat Peter Norvig and Accidentally Became Ryan Gosling

My Cloud Run Multi-Agent Fleet Passed Its Demo. The Architecture Was Still Wrong.

Badger: An E-Ink Badge I Use For Conferences

Why I Decided to Stop Using Claude Code

Dev log #19 WebRTC v2 flows, agentic orchestration, and a perfectly synced vault

Five bugs in my LLM app that never threw an error

4. Use `slots` in Custom Classes