Running operations that constantly allocate new memory can be highly undesirable in machine learning. First, updating hundreds of megabytes of parameters multiple times per second via new allocations creates severe memory overhead, which is why these updates should be performed in-place. Second, if multiple variables point to the same parameters, failing to update the memory in-place requires carefully updating all references to avoid memory leaks or inadvertently referring to stale parameters.

Unnecessary Memory Allocation in Machine Learning

In Python, standard operations like `Y = Y + X` do not update the variable `Y` in-place. Instead, Python first evaluates the expression `Y + X`, allocates new memory to host the resulting values, and then dereferences the tensor that `Y` used to point to, updating `Y` to point to this newly allocated memory location.

Claude

Data types in a program are likely to be allocated into memory when encountered in a program. Python's interpreter includes the Python memory manager, which handles how data types are stored and tracked in memory. The user has no control over this process. Memory allocation for each new variable is performed on demand by the Python memory manager automatically.

Memory Allocation

Dive into Deep Learning

Memory allocation in Python takes place in two parts:
- Stack Allocation
- Heap Allocation

Types of Memory Allocation

PyArena is a structure that is part of the default CPython implementation of Python. It will store pointers to all memory given to it. The functions PyArena_Malloc() and PyArena_Free() are what control memory allocation and deallocation. PyArena also stores a linked-list of allocated memory blocks.

Learn Before

Related

Learn After