Memory-mapped files allow programs to treat files on disk as if they were part of the process’s memory. That’s a powerful technique because it removes the need to explicitly read and write file contents; instead, the operating system manages page faults and loads data from disk transparently as you access it. When a portion of the file isn’t currently loaded into memory, accessing it triggers the OS to map the required pages on demand, which can significantly optimize I/O operations.
Under the hood, memory mapping creates a direct byte-for-byte correspondence between virtual memory addresses and the file contents. This means if you have a large file—too big to fit into RAM—your program can still handle it as if it was an array in memory, without reading the entire file up front. The OS uses page tables to map these addresses to disk blocks, swapping them in and out efficiently.
One subtlety to keep in mind is that changes to a memory-mapped region are not necessarily written to disk immediately. The OS may delay syncing changes for efficiency, employing write-back caching. To ensure data consistency, explicit calls to flush changes are sometimes required. Conversely, memory mapping is often safer than traditional buffered I/O because the OS can batch writes and adjust access patterns for best performance.
In Python, this concept is materialized by modules like mmap or higher-level abstractions such as numpy.memmap. When you use numpy.memmap, you get a numpy array interface to the file’s bytes without loading them all at the same time. This enables working with arrays larger than RAM and performing random access seamlessly.
Behind the scenes, the OS-level memory mapping uses platform-specific APIs: on Unix-like systems, mmap(2) syscall, and on Windows, CreateFileMapping and MapViewOfFile. The file descriptor or handle is linked to a virtual memory range, and subsequent accesses behave just like accessing ordinary memory.
Consider this snippet that opens a binary file as a memory-mapped array of integers:
import mmap
import struct
with open('data.bin', 'r+b') as f:
mm = mmap.mmap(f.fileno(), 0) # map entire file
# Read first 4 bytes as an integer
first_int = struct.unpack('i', mm[:4])[0]
print('First integer:', first_int)
# Modify content in place
mm[4:8] = struct.pack('i', 42)
mm.flush() # push changes to disk
mm.close()
Here, you see direct byte manipulation with memory-mapped data, sidestepping the need for explicit seek and read calls. That’s just the beginning—combined with numpy’s indexing and datatype handling, memory mapping becomes even more elegant and potent, especially for handling very large datasets.
Optimizing performance with numpy.memmap
When performance is critical and datasets are huge, numpy.memmap shines by giving you the ability to access and manipulate data directly on disk as if it were a normal array. The key advantage is that it doesn’t load the entire file into memory, drastically reducing your program’s RAM footprint. This is especially advantageous in data science and machine learning workflows where datasets easily reach gigabytes or terabytes in size.
To get the best performance from numpy.memmap, consider how memory access patterns influence page faults and, correspondingly, I/O operations. Sequential accesses trigger the OS to prefetch data efficiently, while random accesses can cause frequent page faults. Arranging your data processing to maximize locality of reference—processing contiguous blocks rather than scattered elements—can yield major speedups.
Here’s a canonical example illustrating how to create, modify, and flush changes back to disk with numpy.memmap:
import numpy as np
# Create a new memmap file with 1 million float64 elements
filename = 'large_array.dat'
shape = (1_000_000,)
# Mode 'w+' creates or overwrites the file
mmap_array = np.memmap(filename, dtype='float64', mode='w+', shape=shape)
# Initialize values without loading entire array into RAM
for i in range(0, shape[0], 100_000):
mmap_array[i:i+100_000] = np.linspace(i, i+100_000-1, 100_000)
mmap_array.flush() # Ensure data is written to disk
# Later, open the file in read-only mode for fast read access
readonly_map = np.memmap(filename, dtype='float64', mode='r', shape=shape)
print(readonly_map[123456]) # Random access with no full memory load
Notice the chunked assignment strategy. Assigning in large slices leverages numpy’s vectorized operations for speed while minimizing memory overhead. Avoid looping over individual elements; it defeats the purpose of memory mapping by causing many smaller I/O operations.
Another optimization strategy: when working with multidimensional arrays, structure your computations to access data along the leading dimension, matching the underlying row-major storage order. Misaligned accesses that jump across rows can lead to inefficient page swapping.
Sometimes, you might want to create an in-memory buffer mirroring a memmapped file, perform computations with full numpy speed, then write back results. A common idiom is:
buffer = np.memmap('file.dat', dtype=np.float32, mode='r+', shape=(1000, 1000))
# Copy a segment to RAM for heavy computation
temp = buffer[100:200, 100:200].copy()
# Perform intricate numpy operations in memory
temp = np.fft.fft2(temp)
# Write back results
buffer[100:200, 100:200] = temp.real
buffer.flush()
This hybrid method avoids repeated overhead of multiple page faults while benefiting from numpy’s optimized routines on fast RAM.
Lastly, keep in mind that numpy.memmap obeys the underlying file permissions and sharing semantics: opening the same file from multiple processes requires careful synchronization to prevent race conditions or data corruption. Coordinating flushing and locking manually or via external tools might be necessary for concurrent access.
