Python ошибка memory error

Впервые я столкнулся с Memory Error, когда работал с огромным массивом ключевых слов. Там было около 40 млн. строк, воодушевленный своим гениальным скриптом я нажал Shift + F10 и спустя 20 секунд получил Memory Error.

Memory Error — исключение вызываемое в случае переполнения выделенной ОС памяти, при условии, что ситуация может быть исправлена путем удаления объектов. Оставим ссылку на доку, кому интересно подробнее разобраться с этим исключением и с формулировкой. Ссылка на документацию по Memory Error.

Если вам интересно как вызывать это исключение, то попробуйте исполнить приведенный ниже код.

print('a' * 1000000000000)

Почему возникает MemoryError?

В целом существует всего лишь несколько основных причин, среди которых:

• 32-битная версия Python, так как для 32-битных приложений Windows выделяет лишь 4 гб, то серьезные операции приводят к MemoryError
• Неоптимизированный код
• Чрезмерно большие датасеты и иные инпут файлы
• Ошибки в установке пакетов

Как исправить MemoryError?

Ошибка связана с 32-битной версией

Тут все просто, следуйте данному гайдлайну и уже через 10 минут вы запустите свой код.

Как посмотреть версию Python?

Идем в cmd (Кнопка Windows + R -> cmd) и пишем python. В итоге получим что-то похожее на

Python 3.8.8 (tags/v3.8.8:024d805, Feb 19 2021, 13:18:16) [MSC v.1928 64 bit (AMD64)]

Нас интересует эта часть [MSC v.1928 64 bit (AMD64)], так как вы ловите MemoryError, то скорее всего у вас будет 32 bit.

Как установить 64-битную версию Python?

Идем на официальный сайт Python и качаем установщик 64-битной версии. Ссылка на сайт с официальными релизами. В скобках нужной нам версии видим 64-bit. Удалять или не удалять 32-битную версию — это ваш выбор, я обычно удаляю, чтобы не путаться в IDE. Все что останется сделать, просто поменять интерпретатор.

Идем в PyCharm в File -> Settings -> Project -> Python Interpreter -> Шестеренка -> Add -> New environment -> Base Interpreter и выбираем python.exe из только что установленной директории. У меня это

C:/Users/Core/AppData/LocalPrograms/Python/Python38

Все, запускаем скрипт и видим, что все выполняется как следует.

Оптимизация кода

Пару раз я встречался с ситуацией когда мои костыли приводили к MemoryError. К этому приводили избыточные условия, циклы и буферные переменные, которые не удаляются после потери необходимости в них. Если вы понимаете, что проблема может быть в этом, вероятно стоит закостылить пару del, мануально удаляя ссылки на объекты. Но помните о том, что проблема в архитектуре вашего проекта, и по настоящему решить эту проблему можно лишь правильно проработав структуру проекта.

Явно освобождаем память с помощью сборщика мусора

В целом в 90% случаев проблема решается переустановкой питона, однако, я  просто обязан рассказать вам про библиотеку gc. В целом почитать про Garbage Collector стоит отдельно на авторитетных ресурсах в статьях профессиональных программистов. Вы просто обязаны знать, что происходит под капотом управления памятью. GC — это не только про Python, управление памятью в Java и других языках базируется на технологии сборки мусора. Ну а вот так мы можем мануально освободить память в Python:

Автор оригинала: Team Python Pool.

Что такое Ошибка памяти?

Ошибка памяти Python или на языке непрофессионалов-это именно то, что означает, что у вас закончилась память в вашей оперативной памяти для выполнения вашего кода.

Когда эта ошибка возникает, это, скорее всего, потому, что вы загрузили все данные в память. Для больших наборов данных вы захотите использовать пакетную обработку . Вместо того чтобы загружать весь набор данных в память, вы должны хранить свои данные на жестком диске и получать к ним доступ пакетами.

Ошибка памяти означает, что ваша программа исчерпала память. Это означает, что ваша программа каким-то образом создает слишком много объектов. В вашем примере вы должны искать части вашего алгоритма, которые могут потреблять много памяти.

Если у операции заканчивается память, это называется ошибкой памяти .

Типы ошибок памяти Python

Неожиданная ошибка памяти в Python

Если вы получаете неожиданную ошибку памяти Python и думаете, что у вас должно быть много доступных ОЗУ, это может быть связано с тем, что вы используете 32-битную установку python .

Простое решение для неожиданной ошибки памяти Python

У вашей программы заканчивается виртуальное адресное пространство. Скорее всего, потому, что вы используете 32-битную версию Python. Поскольку Windows (как и большинство других ОС) ограничивает 32-разрядные приложения до 2 ГБ адресного пространства пользовательского режима.

Мы рекомендуем вам установить 64-битную версию Python (если вы можете, я бы рекомендовал обновить ее до Python 3 по другим причинам); она будет использовать больше памяти, но тогда у нее будет доступ к большему объему памяти (и больше физической оперативной памяти).

Проблема в том, что 32-битный python имеет доступ только к ~4 ГБ оперативной памяти. Это может уменьшиться еще больше, если ваша операционная система 32-разрядная, из-за накладных расходов операционной системы.

Например, в Python 2 функция zip принимает несколько итераций и возвращает один итератор кортежей. Во всяком случае, каждый элемент итератора нужен нам один раз для циклирования. Таким образом, нам не нужно хранить все элементы в памяти на протяжении всего цикла. Поэтому было бы лучше использовать izip, который извлекает каждый элемент только на следующих итерациях. Python 3 zip по умолчанию функционирует как izip.

Должен Читать: Python Print Без Новой Строки

Ошибка памяти Python Из-за набора данных

Как и в случае с 32-битной и 64-битной версиями, другой возможностью может быть размер набора данных, если вы работаете с большим набором данных.Загрузка большого набора данных непосредственно в память и выполнение над ним вычислений и сохранение промежуточных результатов этих вычислений могут быстро заполнить вашу память. Функции генератора очень пригодятся, если это ваша проблема. Многие популярные библиотеки python, такие как Keras и TensorFlow, имеют специальные функции и классы для генераторов.

Ошибка памяти Python Из – за неправильной установки Python

Неправильная установка пакетов Python также может привести к ошибке памяти . На самом деле, прежде чем решить проблему, Мы установили на windows вручную python 2.7 и пакеты, которые мне были нужны, после того, как возились почти два дня, пытаясь выяснить, в чем проблема, Мы переустановили все с помощью Conda , и проблема была решена.

Мы предполагаем, что Honda устанавливает лучшие пакеты управления памятью, и это было главной причиной. Таким образом, вы можете попробовать установить пакеты Python с помощью Conda, это может решить проблему ошибки памяти.

Ошибка нехватки памяти в Python

Большинство платформ возвращают “Out of Memory error”, если попытка выделить блок памяти завершается неудачей, но первопричина этой проблемы очень редко имеет какое-либо отношение к тому, что действительно “out of memory”.” Это происходит потому, что почти в каждой современной операционной системе диспетчер памяти с радостью использует доступное место на жестком диске в качестве места для хранения страниц памяти, которые не помещаются в оперативную память; ваш компьютер обычно может выделять память до тех пор, пока диск не заполнится, и это может привести к ошибке Python Out of Memory(или к превышению лимита подкачки; в Windows см. раздел Свойства системы > Параметры производительности > Дополнительно > Виртуальная память).

Что еще хуже, каждое активное выделение в адресном пространстве программы может вызвать “фрагментацию”, которая может предотвратить будущие выделения, разбивая доступную память на куски, которые по отдельности слишком малы, чтобы удовлетворить новое выделение одним непрерывным блоком.

1 Если 32-битное приложение имеет установленный флаг LARGEADDRESSAWARE, оно имеет доступ к полным 4 гб адресного пространства при работе на 64-битной версии Windows.

2 До сих пор четыре читателя написали, чтобы объяснить, что флаг gcAllowVeryLargeObjects устраняет это ограничение .NET. Это не так. Этот флаг позволяет объектам, которые занимают более 2 Гб памяти, но он не позволяет одномерному массиву содержать более 2^31 записей.

Как я могу явно освободить память в Python?

Если вы написали программу Python, которая действует на большой входной файл, чтобы создать несколько миллионов объектов, представляющих собой, и это занимает тонны памяти, и вам нужен лучший способ сказать Python, что вам больше не нужны некоторые данные, и они могут быть освобождены?

Простой ответ на эту проблему:

Принудительно вызовите сборщик мусора для освобождения несвязанной памяти с помощью gc.collect().

Как показано ниже:

импорт gc

gc.collect()

Ошибка памяти в Python, когда 50+ГБ свободны и используют 64-битный python?

В некоторых операционных системах существуют ограничения на объем оперативной памяти, который может обрабатывать один процессор. Таким образом, даже если есть достаточно свободной оперативной памяти, ваш единственный поток на одном ядре) не может взять больше. Но я не знаю, действительно ли это для вашей версии Windows.

Как вы устанавливаете использование памяти для программ python?

Python использует сборку мусора и встроенное управление памятью, чтобы гарантировать, что программа использует только столько оперативной памяти, сколько требуется. Поэтому, если вы специально не пишете свою программу таким образом, чтобы увеличить использование памяти, например, создавая базу данных в оперативной памяти, Python использует только то, что ему нужно.

В связи с этим возникает вопрос: почему вы хотите использовать больше оперативной памяти? Идея большинства программистов сводится к минимизации использования ресурсов.

если вы хотите ограничить использование памяти виртуальной машины python, вы можете попробовать следующее:1、Linux, команда ulimit для ограничения использования памяти на python2、вы можете использовать модуль ресурсов для ограничения использования памяти программы; если вы хотите ускорить программу ur, хотя и дать больше памяти приложению ur, вы можете попробовать следующее:1threading, multiprocessing2pypy3pysco только на python 2.5

Как установить ограничения на использование памяти и процессора

Чтобы ограничить использование памяти или процессора запущенной программой. Так что мы не столкнемся с какой-либо ошибкой памяти. Для этого можно использовать модуль ресурсов , и таким образом обе задачи могут быть выполнены очень хорошо, как показано в приведенном ниже коде:

Код #1: Ограничение процессорного времени

# importing libraries 
import signal 
import resource 
import os 

# checking time limit exceed 
def time_exceeded(signo, frame): 
	print("Time's up !") 
	raise SystemExit(1) 

def set_max_runtime(seconds): 
	# setting up the resource limit 
	soft,.getrlimit(resource.RLIMIT_CPU) 
	resource.setrlimit(resource.RLIMIT_CPU, (seconds, hard)) 
	signal.signal(signal.SIGXCPU, time_exceeded) 

# max run time of 15 millisecond 
if __name__: 
	set_max_runtime(15) 
	while True: 
		pass

Код #2: Чтобы ограничить использование памяти, код устанавливает ограничение на общее адресное пространство

# using resource 
import resource 

def limit_memory(maxsize): 
	soft,.getrlimit(resource.RLIMIT_AS) 
	resource.setrlimit(resource.RLIMIT_AS, (maxsize, hard))

Способы обработки ошибок памяти Python и больших файлов данных

1. Выделите Больше Памяти

Некоторые инструменты или библиотеки Python могут быть ограничены конфигурацией памяти по умолчанию.

Проверьте, можете ли вы перенастроить свой инструмент или библиотеку, чтобы выделить больше памяти.

То есть платформа, предназначенная для обработки очень больших наборов данных, которая позволяет использовать преобразования данных и алгоритмы машинного обучения поверх нее.

Хорошим примером является Weka, где вы можете увеличить объем памяти в качестве параметра при запуске приложения.

2. Работа с меньшим образцом

Вы уверены, что вам нужно работать со всеми данными?

Возьмите случайную выборку данных, например первые 1000 или 100 000 строк. Используйте эту меньшую выборку для проработки вашей проблемы, прежде чем подгонять окончательную модель ко всем вашим данным (используя прогрессивные методы загрузки данных).

Я думаю, что это хорошая практика в целом для машинного обучения, чтобы дать вам быструю выборочную проверку алгоритмов и поворот результатов.

Вы также можете рассмотреть возможность проведения анализа чувствительности объема данных, используемых для подгонки одного алгоритма, по сравнению с навыком модели. Возможно, существует естественная точка уменьшения отдачи, которую вы можете использовать в качестве эвристического размера вашей меньшей выборки.

3. Используйте компьютер с большим объемом памяти

Вам обязательно работать на компьютере?

Возможно, вы сможете получить доступ к гораздо большему компьютеру с на порядок большим объемом памяти.

Например, хорошим вариантом является аренда вычислительного времени на облачном сервисе, таком как Amazon Web Services, который предлагает машины с десятками гигабайт оперативной памяти менее чем за доллар США в час.

4. Используйте реляционную базу данных

Реляционные базы данных обеспечивают стандартный способ хранения и доступа к очень большим наборам данных.

Внутренне данные, хранящиеся на диске, могут быть постепенно загружены пакетами и могут быть запрошены с помощью стандартного языка запросов (SQL).

Бесплатные инструменты базы данных с открытым исходным кодом, такие как href=”https://www.mysql.com/”>MySQL или href=”https://www.postgresql.org/”>Postgres можно использовать, и большинство (все?) языков программирования и многие инструменты машинного обучения могут подключаться непосредственно к реляционным базам данных. Вы также можете использовать легкий подход, например href=”https://www.sqlite.org/”>SQLite. href=”https://www.mysql.com/”>MySQL или href=”https://www.postgresql.org/”>Postgres можно использовать, и большинство (все?) языков программирования и многие инструменты машинного обучения могут подключаться непосредственно к реляционным базам данных. Вы также можете использовать легкий подход, например href=”https://www.sqlite.org/”>SQLite. href=”https://www.postgresql.org/”>Postgres можно использовать, и большинство (все?) языков программирования и многие инструменты машинного обучения могут подключаться непосредственно к реляционным базам данных. Вы также можете использовать легкий подход, например href=”https://www.sqlite.org/”>SQLite. href=”https://www.sqlite.org/”>SQLite.

5. Используйте платформу больших данных

В некоторых случаях вам может потребоваться прибегнуть к платформе больших данных.

Резюме

В этом посте вы обнаружили ряд тактик и способов, которые можно использовать при работе с ошибкой памяти Python.

Есть ли другие методы, о которых вы знаете или пробовали?Поделитесь ими в комментариях ниже.

Вы пробовали какой-нибудь из этих методов?Дайте мне знать в комментариях.

Если ваша проблема все еще не решена и вам нужна помощь относительно Python Memory Error. Прокомментируйте ниже, мы постараемся решить вашу проблему как можно скорее.

What is Memory Error?

Python Memory Error or in layman language is exactly what it means, you have run out of memory in your RAM for your code to execute.

When this error occurs it is likely because you have loaded the entire data into memory. For large datasets, you will want to use batch processing. Instead of loading your entire dataset into memory you should keep your data in your hard drive and access it in batches.

A memory error means that your program has run out of memory. This means that your program somehow creates too many objects. In your example, you have to look for parts of your algorithm that could be consuming a lot of memory.

If an operation runs out of memory it is known as memory error.

Types of Python Memory Error

Unexpected Memory Error in Python

If you get an unexpected Python Memory Error and you think you should have plenty of rams available, it might be because you are using a 32-bit python installation.

The easy solution for Unexpected Python Memory Error

Your program is running out of virtual address space. Most probably because you’re using a 32-bit version of Python. As Windows (and most other OSes as well) limits 32-bit applications to 2 GB of user-mode address space.

We Python Pooler’s recommend you to install a 64-bit version of Python (if you can, I’d recommend upgrading to Python 3 for other reasons); it will use more memory, but then, it will have access to a lot more memory space (and more physical RAM as well).

The issue is that 32-bit python only has access to ~4GB of RAM. This can shrink even further if your operating system is 32-bit, because of the operating system overhead.

For example, in Python 2 zip function takes in multiple iterables and returns a single iterator of tuples. Anyhow, we need each item from the iterator once for looping. So we don’t need to store all items in memory throughout looping. So it’d be better to use izip which retrieves each item only on next iterations. Python 3’s zip functions as izip by default.

Must Read: Python Print Without Newline

Python Memory Error Due to Dataset

Like the point, about 32 bit and 64-bit versions have already been covered, another possibility could be dataset size, if you’re working with a large dataset. Loading a large dataset directly into memory and performing computations on it and saving intermediate results of those computations can quickly fill up your memory. Generator functions come in very handy if this is your problem. Many popular python libraries like Keras and TensorFlow have specific functions and classes for generators.

Python Memory Error Due to Improper Installation of Python

Improper installation of Python packages may also lead to Memory Error. As a matter of fact, before solving the problem, We had installed on windows manually python 2.7 and the packages that I needed, after messing almost two days trying to figure out what was the problem, We reinstalled everything with Conda and the problem was solved.

We guess Conda is installing better memory management packages and that was the main reason. So you can try installing Python Packages using Conda, it may solve the Memory Error issue.

Most platforms return an “Out of Memory error” if an attempt to allocate a block of memory fails, but the root cause of that problem very rarely has anything to do with truly being “out of memory.” That’s because, on almost every modern operating system, the memory manager will happily use your available hard disk space as place to store pages of memory that don’t fit in RAM; your computer can usually allocate memory until the disk fills up and it may lead to Python Out of Memory Error(or a swap limit is hit; in Windows, see System Properties > Performance Options > Advanced > Virtual memory).

Making matters much worse, every active allocation in the program’s address space can cause “fragmentation” that can prevent future allocations by splitting available memory into chunks that are individually too small to satisfy a new allocation with one contiguous block.

¹ If a 32bit application has the LARGEADDRESSAWARE flag set, it has access to s full 4gb of address space when running on a 64bit version of Windows.

² So far, four readers have written to explain that the gcAllowVeryLargeObjects flag removes this .NET limitation. It does not. This flag allows objects which occupy more than 2gb of memory, but it does not permit a single-dimensional array to contain more than 2^31 entries.

How can I explicitly free memory in Python?

If you wrote a Python program that acts on a large input file to create a few million objects representing and it’s taking tons of memory and you need the best way to tell Python that you no longer need some of the data, and it can be freed?

The Simple answer to this problem is:

Force the garbage collector for releasing an unreferenced memory with gc.collect(). 

Like shown below:

import gc

gc.collect()

Memory error in Python when 50+GB is free and using 64bit python?

On some operating systems, there are limits to how much RAM a single CPU can handle. So even if there is enough RAM free, your single thread (=running on one core) cannot take more. But I don’t know if this is valid for your Windows version, though.

How do you set the memory usage for python programs?

Python uses garbage collection and built-in memory management to ensure the program only uses as much RAM as required. So unless you expressly write your program in such a way to bloat the memory usage, e.g. making a database in RAM, Python only uses what it needs.

Which begs the question, why would you want to use more RAM? The idea for most programmers is to minimize resource usage.

How to put limits on Memory and CPU Usage

To put limits on the memory or CPU use of a program running. So that we will not face any memory error. Well to do so, Resource module can be used and thus both the task can be performed very well as shown in the code given below:

Code #1: Restrict CPU time

# importing libraries 
import signal 
import resource 
import os 

# checking time limit exceed 
def time_exceeded(signo, frame): 
	print("Time's up !") 
	raise SystemExit(1) 

def set_max_runtime(seconds): 
	# setting up the resource limit 
	soft, hard = resource.getrlimit(resource.RLIMIT_CPU) 
	resource.setrlimit(resource.RLIMIT_CPU, (seconds, hard)) 
	signal.signal(signal.SIGXCPU, time_exceeded) 

# max run time of 15 millisecond 
if __name__ == '__main__': 
	set_max_runtime(15) 
	while True: 
		pass

Code #2: In order to restrict memory use, the code puts a limit on the total address space

# using resource 
import resource 

def limit_memory(maxsize): 
	soft, hard = resource.getrlimit(resource.RLIMIT_AS) 
	resource.setrlimit(resource.RLIMIT_AS, (maxsize, hard)) 

Ways to Handle Python Memory Error and Large Data Files

1. Allocate More Memory

Some Python tools or libraries may be limited by a default memory configuration.

Check if you can re-configure your tool or library to allocate more memory.

That is, a platform designed for handling very large datasets, that allows you to use data transforms and machine learning algorithms on top of it.

A good example is Weka, where you can increase the memory as a parameter when starting the application.

2. Work with a Smaller Sample

Are you sure you need to work with all of the data?

Take a random sample of your data, such as the first 1,000 or 100,000 rows. Use this smaller sample to work through your problem before fitting a final model on all of your data (using progressive data loading techniques).

I think this is a good practice in general for machine learning to give you quick spot-checks of algorithms and turnaround of results.

You may also consider performing a sensitivity analysis of the amount of data used to fit one algorithm compared to the model skill. Perhaps there is a natural point of diminishing returns that you can use as a heuristic size of your smaller sample.

3. Use a Computer with More Memory

Do you have to work on your computer?

Perhaps you can get access to a much larger computer with an order of magnitude more memory.

For example, a good option is to rent compute time on a cloud service like Amazon Web Services that offers machines with tens of gigabytes of RAM for less than a US dollar per hour.

4. Use a Relational Database

Relational databases provide a standard way of storing and accessing very large datasets.

Internally, the data is stored on disk can be progressively loaded in batches and can be queried using a standard query language (SQL).

Free open-source database tools like MySQL or Postgres can be used and most (all?) programming languages and many machine learning tools can connect directly to relational databases. You can also use a lightweight approach, such as SQLite.

5. Use a Big Data Platform

In some cases, you may need to resort to a big data platform.

Summary

In this post, you discovered a number of tactics and ways that you can use when dealing with Python Memory Error.

Are there other methods that you know about or have tried?
Share them in the comments below.

Have you tried any of these methods?
Let me know in the comments.

If your problem is still not solved and you need help regarding Python Memory Error. Comment Down below, We will try to solve your issue asap.

In Python, errors are exceptions that occur when the interpreter encounters a problem while running a program. These exceptions can be caught and managed using try-except blocks, allowing a program to handle problems gracefully and recover from them. There are many different types of errors in Python, each with its own meaning and origin. Python memory error is one such error. Let’s take a closer look into the python memory error.

What is Python Memory Error?

A Python memory error occurs when a Python program runs out of memory while attempting to allocate space for new objects. This error usually occurs when a program attempts to generate a massive data structure, such as a list or dictionary, or when it creates little objects continuously until the available memory is consumed.

When a memory error occurs, Python throws a MemoryError exception, indicating that the needed memory could not be allocated. This exception can be handled gracefully by implementing a try-except block, allowing the program to handle any necessary cleanup or error reporting.

Here’s an example of a MemoryError in Python:

# Creating a large list
my_list = [i for i in range(1000000000)]

This code creates a list of one billion integers. Depending on the available memory on your system, this can quickly consume all available memory and raise a MemoryError. The error message might look something like this:

Traceback (most recent call last):
  File "", line 1, in 
MemoryError

To avoid this error, you can use a generator expression instead of a list comprehension, as generators create values on-the-fly and don’t store them in memory all at once. Here’s an example:

# Using a generator expression
my_generator = (i for i in range(1000000000))

This code creates a generator expression that generates the integers from 0 to 999,999,999 on-the-fly, rather than creating a list that stores all the integers in memory. This can help avoid the MemoryError.

Types of Python Memory Error

Here are some more typical ways that Python memory errors might be classified:

1. Out of Memory Error

An Out of Memory Error in Python occurs when a program tries to allocate more memory than is available on the system. This can happen when a user is working with large datasets, images, or other memory-consumption tasks. For example, suppose a user tries to create a list of one billion integers on a system with limited memory, it may run out of memory and throws an Out of Memory Error.

2. Stack Overflow Error

A Stack Overflow Error occurs when a program’s call stack becomes too deep, causing a stack overflow. This can happen when using a recursive function. For example, if a program uses a recursive function with no base case, it may run out of stack space and raise a Stack Overflow Error.

3. Memory Leak Error

A Memory Leak Error occurs when a program continuously allocates memory without releasing it, leading to a gradual increase in memory usage over time. This can happen when a program repeatedly allocates memory to new objects or data structures without properly releasing old memory, or when there are circular references between objects that prevent them from being garbage-collected.

Ways to Handle Python Memory Error

These are some ways to handle Python memory errors:

1. Appropriate Python Set-up

This simplest, but perhaps least intuitive, answer to a MemoryError is actually related to a potential problem with your Python setup. If you install the 32-bit version of Python on a 64-bit system, you will have highly limited access to the system’s RAM. This restricted access may result in MemoryErrors on programs that your computer would normally be able to handle.

2. Use Generators or Iterators

Instead of putting all of the data into memory at once, use generators or iterators to process data in smaller chunks. This can help to reduce memory use and avoid memory faults. For example, in a function, use yield to generate data one item at a time.

def generate_data():
    for i in range(1000000):
        yield i

for data in generate_data():
    process_data(data)

3. Use a Database or External Storage

If the data set is too large to fit in memory, consider storing it in a database or an file. This allows the program to access the data without needing to load it all into memory all at once. Here’s an example using the pandas library to read a large CSV file:

# Importing the pandas library
import pandas as pd

# Reading a large CSV file
df = pd.read_csv('large_file.csv')

4. Improve Memory Consumption

Review the code and look for places where memory usage can be improved. For example, instead of copying lists, use slicing, map, or filter built-in functions instead of loops.

5. Catch and Handle Exceptions

Catch MemoryError exceptions and handle them in a way that allows the program to continue functioning. For example, log the error and try again later, or display an error message to the user and politely exit the program. Here’s an code snippet to demonstrate this:

try:
    # Code that might raise a MemoryError
except MemoryError:
    # Handle the error

6. Memory Profiler to Identify Memory Leaks

A memory profiler is a tool that can be used to identify memory leaks in Python programs. Memory leaks occur when a program does not release memory that is no longer needed, causing memory usage to increase over time. Memory profilers help identify the parts of a program that are responsible for memory leaks, allowing developers to address these issues and improve the program’s memory efficiency.

Conclusion
Python memory error occurs when a program tries to use more memory than is available. This can happen due to various reasons, such as not deallocating the memory of unused objects or improper memory usage. To avoid and fix memory errors, it’s important to be aware of the memory usage of our programs and take necessary steps to manage our memory, such as using generators in place of lists, deleting unused objects, and using external storage. By being mindful of memory usage, we can write more efficient Python programs.

FAQs Related to Python Memory Error

Here are some frequently asked questions (FAQs) about Python memory errors:

Q1 – What is a memory error in python?
Ans – A Python memory error occurs when a program tries to allocate more memory than the system can provide. This can happen when a program tries to store too much data in memory or when a program creates too many objects.

Q2 – What causes memory errors in python?
Ans – Python memory errors can be caused by a variety of factors, including a lack of available memory on the system, inefficient code that creates too many objects, or a data set that is too large to fit into memory.

Q3 – How to handle Python memory errors?
Ans – We can handle Python memory errors by catching the MemoryError exception and taking appropriate action, such as freeing up memory or terminating the program. Additionally, we can also use profiling tools to identify memory usage in our code.

Q4 – What should I do if I encounter a Python memory error?
Ans – If you encounter a Python memory error, try to identify the source of the error by using profiling tools or by analyzing your python code. Once you have identified the source, take appropriate action to reduce memory usage, such as using memory-efficient data structures or deleting objects that are no longer needed. If you are working with large data sets, consider using external storage to store them.

Jan 24, 2018 6:00:25 PM |
Python Exception Handling — MemoryError

A deep look at the MemoryError in Python, with code samples illustrating how different Python versions may handle MemoryErrors differently.

Continuing along through our in-depth Python Exception Handling series, today we’ll dig into Python’s MemoryError. As with all programming languages, Python includes a fallback exception for when the interpreter completely runs out of memory and must abort current execution. In these (hopefully rare) instances, Python raises a MemoryError, giving the script a chance to catch itself and break out of the current memory draught and recover. However, since Python uses the C language’s malloc() function for its memory management architecture, it is not guaranteed that all processes will be able to recover — in some cases, a MemoryError will result in an unrecoverable crash.

In today’s article we’ll examine the MemoryError in more detail, starting with where it sits in the larger Python Exception Class Hierarchy. We’ll also examine a simple code sample that illustrates how large memory allocations can occur, how the behavior of using massive objects differs depending on the particular computer architecture and Python version in use, and how MemoryErrors may be raised and handled. Let’s get into it!

The Technical Rundown

All Python exceptions inherit from the BaseException class, or extend from an inherited class therein. The full exception hierarchy of this error is:

• BaseException
  • Exception
    • MemoryError

Full Code Sample

Below is the full code sample we’ll be using in this article. It can be copied and pasted if you’d like to play with the code yourself and see how everything works.

import os

import psutil

import sys

import traceback
PROCESS = psutil.Process(os.getpid())
 MEGA = 10 ** 6
 MEGA_STR = ' ' * MEGA
def main():
 try:
 print_memory_usage()
 alloc_max_str()
 alloc_max_array()
 except MemoryError as error:
 # Output expected MemoryErrors.
 log_exception(error)
 except Exception as exception:
 # Output unexpected Exceptions.
 log_exception(exception, False)
def alloc_max_array():
 """Allocates memory for maximum array.
 See: https://stackoverflow.com/a/15495136
 :return: None
 """
 collection = []
 while True:
 try:
 collection.append(MEGA_STR)
 except MemoryError as error:
 # Output expected MemoryErrors.
 log_exception(error)
 break
 except Exception as exception:
 # Output unexpected Exceptions.
 log_exception(exception, False)
 print('Maximum array size:', len(collection) * 10)
 print_memory_usage()
def alloc_max_str():
 """Allocates memory for maximum string.
 See: https://stackoverflow.com/a/15495136
 :return: None
 """
 i = 0
 while True:
 try:
 a = ' ' * (i * 10 * MEGA)
 del a
 except MemoryError as error:
 # Output expected MemoryErrors.
 log_exception(error)
 break
 except Exception as exception:
 # Output unexpected Exceptions.
 log_exception(exception, False)
 i += 1
 max_i = i - 1
 print('Maximum string size:', (max_i * 10 * MEGA))
 print_memory_usage()
def log_exception(exception: BaseException, expected: bool = True):
 """Prints the passed BaseException to the console, including traceback.
 :param exception: The BaseException to output.
 :param expected: Determines if BaseException was expected.
 """
 output = "[{}] {}: {}".format('EXPECTED' if expected else 'UNEXPECTED', type(exception).__name__, exception)
 print(output)
 exc_type, exc_value, exc_traceback = sys.exc_info()
 traceback.print_tb(exc_traceback)



  


                       
  




def print_memory_usage():
 """Prints current memory usage stats.
 See: https://stackoverflow.com/a/15495136
 :return: None
 """
 total, available, percent, used, free = psutil.virtual_memory()
 total, available, used, free = total / MEGA, available / MEGA, used / MEGA, free / MEGA
 proc = PROCESS.memory_info()[1] / MEGA
 print('process = %s total = %s available = %s used = %s free = %s percent = %s'
 % (proc, total, available, used, free, percent))
if __name__ == "__main__":
 main()
 

When Should You Use It?

In most situations, a MemoryError indicates a major flaw in the current application. For example, an application that accepts files or user data input could run into MemoryErrors if the application has insufficient sanity checks in place. There are tons of scenarios where memory limits can be problematic, but for our code illustration we’ll just stick with a simple allocation in local memory using strings and arrays.

The most important factor in whether your own applications are likely to experience MemoryErrors is actually the computer architecture the executing system is running on. Or, even more specifically, the architecture your version of Python is using. If you’re using a 32-bit Python then the maximum memory allocation given to the Python process is exceptionally low. The specific maximum memory allocation limit varies and depends on your system, but it’s usually around 2 GB and certainly no more than 4 GB.

On the other hand, 64-bit Python versions are more or less limited only by your available system memory. In practical terms, a 64-bit Python interpreter is unlikely to experience memory issues, or if it does, the issue is a much bigger deal since it’s likely impacting the rest of the system anyway.

To test this stuff out we’ll be using the psutil to retrieve information about the active process, and specifically, the psutil.virtual_memory() method, which provides current memory usage stats when invoked. This information is printed within the print_memory_usage() function:

def print_memory_usage():

    """Prints current memory usage stats.

    See: https://stackoverflow.com/a/15495136
 :return: None
 """
 total, available, percent, used, free = psutil.virtual_memory()
 total, available, used, free = total / MEGA, available / MEGA, used / MEGA, free / MEGA
 proc = PROCESS.memory_info()[1] / MEGA
 print('process = %s total = %s available = %s used = %s free = %s percent = %s'
 % (proc, total, available, used, free, percent))
 

We’ll start by using the Python 3.6.4 32-bit version and appending MEGA_STR strings (which contain one million characters each) onto the end of an array until the process catches a MemoryError:

PROCESS = psutil.Process(os.getpid())

MEGA = 10 ** 6

MEGA_STR = ' ' * MEGA
def alloc_max_array():
 """Allocates memory for maximum array.
 See: https://stackoverflow.com/a/15495136
 :return: None
 """
 collection = []
 while True:
 try:
 collection.append(MEGA_STR)
 except MemoryError as error:
 # Output expected MemoryErrors.
 log_exception(error)
 break
 except Exception as exception:
 # Output unexpected Exceptions.
 log_exception(exception, False)
 print('Maximum array size:', len(collection) * 10)
 print_memory_usage()
def log_exception(exception: BaseException, expected: bool = True):
 """Prints the passed BaseException to the console, including traceback.
 :param exception: The BaseException to output.
 :param expected: Determines if BaseException was expected.
 """
 output = "[{}] {}: {}".format('EXPECTED' if expected else 'UNEXPECTED', type(exception).__name__, exception)
 print(output)
 exc_type, exc_value, exc_traceback = sys.exc_info()
 traceback.print_tb(exc_traceback)
 

After we run out of memory and break out of the loop we output the memory usage of the array, along with overall memory usage stats. The result of running this function is the following output:

process = 14.577664 total = 17106.767872 available = 9025.814528 used = 8080.953344 free = 9025.814528 percent = 47.2
[EXPECTED] MemoryError:
 File "D:/work/Airbrake.io/Exceptions/Python/BaseException/Exception/MemoryError/main.py", line 33, in alloc_max_array
 collection.append(MEGA_STR)
Maximum array size: 1074655510
process = 446.603264 total = 17106.767872 available = 8769.028096 used = 8337.739776 free = 8769.028096 percent = 48.7
 

This shows our base memory usage at the top, and the array size we created at the bottom. As expected, after about 15 seconds of execution on my system we experienced a MemoryError. The alloc_max_str() function test creates a large string instead of an array, but we should see similar results:

def alloc_max_str():

    """Allocates memory for maximum string.

    See: https://stackoverflow.com/a/15495136
 :return: None
 """
 i = 0
 while True:
 try:
 a = ' ' * (i * 10 * MEGA)
 del a
 except MemoryError as error:
 # Output expected MemoryErrors.
 log_exception(error)
 break
 except Exception as exception:
 # Output unexpected Exceptions.
 log_exception(exception, False)
 i += 1
 max_i = i - 1
 print('Maximum string size:', (max_i * 10 * MEGA))
 print_memory_usage()
 

Sure enough, executing alloc_max_str() results in a raised MemoryError after a relatively short execution period:

[EXPECTED] MemoryError:

  File "D:/work/Airbrake.io/Exceptions/Python/BaseException/Exception/MemoryError/main.py", line 54, in alloc_max_str

    a = ' ' * (i * 10 * MEGA)
Maximum string size: 1110000000
process = 14.966784 total = 17106.767872 available = 9240.141824 used = 7866.626048 free = 9240.141824 percent = 46.0
 

As mentioned, there is a huge difference between 32- and 64-bit versions of Python. If we swap to Python 3.6.4 64-bit and execute the same code no MemoryError has been thrown after 5+ minutes of iteration. As discussed, this is because 64-bit Python isn’t artificially limited, but can more or less use most of the available system memory!

Airbrake’s robust error monitoring software provides real-time error monitoring and automatic exception reporting for all your development projects. Airbrake’s state of the art web dashboard ensures you receive round-the-clock status updates on your application’s health and error rates. No matter what you’re working on, Airbrake easily integrates with all the most popular languages and frameworks. Plus, Airbrake makes it easy to customize exception parameters, while giving you complete control of the active error filter system, so you only gather the errors that matter most.

Check out Airbrake’s error monitoring software today and see for yourself why so many of the world’s best engineering teams use Airbrake to revolutionize their exception handling practices! Try Airbrake free with a 14-day trial.

What exactly is a Memory Error?

Python Memory Error or, in layman’s terms, you’ve run out of Random access memory (RAM) to sustain the running of your code. This error indicates that you have loaded all of the data into memory. For large datasets, batch processing is advised. Instead of packing your complete dataset into memory, please save it to your hard disk and access it in batches.

Your software has run out of memory, resulting in a memory error. It indicates that your program generates an excessive number of items. You’ll need to check for parts of your algorithm that consume a lot of memory in your case.

A memory error occurs when an operation runs out of memory.

Python has a fallback exception, as do all programming languages, for when the interpreter runs out of memory and must abandon the current execution. Python issues a MemoryError in these (hopefully infrequent) cases, giving the script a chance to catch up and break free from the present memory dearth. However, because Python’s memory management architecture is based on the C language’s malloc() function, It is unlikely that all processes will recover – in some situations, a MemoryError will result in an unrecoverable crash.

A MemoryError usually signals a severe fault in the present application. A program that takes files or user data input, for example, may encounter MemoryErrors if it lacks proper sanity checks. Memory restrictions might cause problems in various situations, but we’ll stick with a simple allocation in local memory utilizing strings and arrays for our code example.

The computer architecture on which the executing system runs is the most crucial element in whether your applications are likely to incur MemoryErrors. Or, to be more particular, the architecture of the Python version you’re using. The maximum memory allocation granted to the Python process is meager if you’re running a 32-bit Python. The maximum memory allocation limit fluctuates and is dependent on your system. However, it is generally around 2 GB and never exceeds 4 GB.

64-bit Python versions, on the other hand, are essentially restricted only by the amount of memory available on your system. Thus, in practice, a 64-bit Python interpreter is unlikely to have memory problems, and if it does, the pain is much more severe because it would most likely affect the rest of the system.

To verify this, we’ll use psutil to get information about the running process, specifically the psutil.virtual memory() method, which returns current memory consumption statistics when called. The print() memory usage method prints the latter information:

Python Memory Errors There are Several Types of Python Memory Errors

In Python, an unexpected memory error occurs

Even if you have enough RAM, you could get an unexpected Python Memory Error, and you may be using a 32-bit Python installation.

Unexpected Python Memory Error: A Simple Solution

Your software has used up all of the virtual address space available to it. It’s most likely because you’re using a 32-bit Python version. Because 32-bit applications are limited to 2 GB of user-mode address space in Windows (and most other operating systems),

We Python Poolers recommend installing a 64-bit version of Python (if possible, update to Python 3 for various reasons); it will use more memory, but it will also have much more memory space available (and more physical RAM as well).

The problem is Python 32-bit only has 4GB of RAM. So it can be reduced due to operating system overhead even more if your operating system is 32-bit.

For example, the zip function in Python 2 accepts many iterables and produces a single tuple iterator. In any case, for looping, we only require each item from the iterator once. As a result, we don’t need to keep all of the things in memory while looping. As a result, it’s preferable to utilize izip, which retrieves each item only on subsequent cycles. Thus, by default, Python 3’s zip routines are called izip.

Memory Error in Python Because of the Dataset

Another choice, if you’re working with a huge dataset, is dataset size. The latter has already been mentioned concerning 32-bit and 64-bit versions. Loading a vast dataset into memory and running computations on it, and preserving intermediate results of such calculations can quickly consume memory. If this is the case, generator functions can be pretty helpful. Many major Python libraries, such as Keras and TensorFlow, include dedicated generator methods and classes.

Memory Error in Python Python was installed incorrectly

Improper Python package installation can also result in a Memory Error. In fact, before resolving the issue, we had manually installed python 2.7 and the programs that I need on Windows. We replaced everything using Conda after spending nearly two days attempting to figure out what was wrong, and the issue was resolved.

Conda is probably installing improved memory management packages, which is the main reason. So you might try installing Python Packages with Conda to see if that fixes the Memory Error.

Conda is a free and open-source package management and environment management system for Windows, Mac OS X, and Linux. Conda is a package manager that installs, runs, and updates packages and their dependencies in a matter of seconds.

Python Out of Memory Error

When an attempt to allocate a block of memory fails, most systems return an “Out of Memory” error, but the core cause of the problem rarely has anything to do with being “out of memory.” That’s because the memory manager on almost every modern operating system will gladly use your available hard disk space for storing memory pages that don’t fit in RAM. In addition, your computer can usually allocate memory until the disk fills up, which may result in a Python Out of Memory Error (or a swap limit is reached; in Windows, see System Properties > Performance Options > Advanced > Virtual memory).

To make matters worse, every current allocation in the program’s address space can result in “fragmentation,” which prevents further allocations by dividing available memory into chunks that are individually too small to satisfy a new allocation with a single contiguous block.

• When operating on a 64bit version of Windows, a 32bit application with the LARGEADDRESSAWARE flag set has access to the entire 4GB of address space.
• Four readers have contacted in to say that the gcAllowVeryLargeObjects setting removes the.NET restriction. No, it doesn’t. This setting permits objects to take up more than 2GB of memory, limiting the number of elements in a single-dimensional array to 231 entries.

In Python, how do I manually free memory?

If you’ve written a Python program that uses a large input file to generate a few million objects, and it’s eating up a lot of memory, what’s the best approach to tell Python that some of the data is no longer needed and may be freed?

This problem has a simple solution:

You can cause the garbage collector to release an unreferenced memory() by using gc.collect.
As illustrated in the example below:

Do you get a memory error when there are more than 50GB of free space in Python, and you’re using 64-bit Python?
On some operating systems, the amount of RAM that a single CPU can handle is limited. So, even if there is adequate RAM available, your single thread (=one core) will not be able to take it anymore. However, we are not certain that this applies to your Windows version.

How can you make python scripts use less memory?

Python uses garbage collection and built-in memory management to ensure that the application consumes as much memory as needed. So, unless you explicitly construct your program to balloon memory utilization, such as creating a RAM database, Python only utilizes what it requires.

Which begs the question of why you’d want to do it in the first place – consume more RAM in the first place. For most programmers, the goal is to use as few resources as possible.

If you wish to keep Python’s memory use low, virtual machine to a minimum, try this:

• On Linux, use the ulimit command to set a memory limit for Python.
• You can use the resource module to limit how much memory the program uses

Consider the following if you wish to speed up your software by giving it more memory: Multiprocessing, threading.
On only python 2.5, use pysco

How can I set memory and CPU usage limits?

To limit the amount of memory or CPU used by an application while it is running. So that we don’t have any memory problems. To accomplish so, the Resource module can be used, and both tasks can be completed successfully, as demonstrated in the code below:

Code 1: Limit CPU usage

# libraries being imported
import signal
import resource
import os

# confirm_exceed_in_time.py 
# confirm if there is an exceed in time limit
def exceeded_time(sig_number, frame):
    print("Time is finally up !")
    raise SystemExit(1)
 
def maximum_runtime(count_seconds):
    # resource limit setup 
    if_soft, if_hard = resource.getrlimit(resource.RLIMIT_CPU)
    resource.setrlimit(resource.RLIMIT_CPU, (count_seconds, if_hard))
    signal.signal(signal.SIGXCPU, exceeded_time)
 
# set a maximum running time of about 25 millisecond
if __name__ == '__main__':
    maximum_runtime(25)
    while True:
        pass

Code #2: To minimize memory usage, the code restricts the total address space available.

# using resource
import resource
 
def limit_memory(maxsize):
    if_soft, if_hard = resource.getrlimit(resource.RLIMIT_AS)
    resource.setrlimit(resource.RLIMIT_AS, (maxsize, if_hard))

How to Deal with Python Memory Errors and Big Data Files

Increase the amount of memory available

A default memory setup may limit some Python tools or modules.

Check to see if your tool or library may be re-configured to allocate more RAM.

That is a platform built to handle massive datasets and allow data transformations. On top of that and machine learning algorithms will be applied.
Weka is a fantastic example of this, as you may increase memory as a parameter when running the app.

Use a Smaller Sample Size

Are you sure you require all of the data?

Take a random sample of your data, such as the first 5,000 or 100,000 rows, before fitting a final model to Use this smaller sample to work through your problem instead of all of your data (using progressive data loading techniques).

It is an excellent practice for machine learning in general, as it allows for quick spot-checks of algorithms and results turnaround.

You may also compare the amount of data utilized to fit one algorithm to the model skill in a sensitivity analysis. Perhaps you can use a natural point of declining returns as a guideline for the size of your smaller sample.

Make use of a computer that has more memory

Is it necessary for you to use your computer? Of course, – it is possible to lay your hand’s on a considerably larger PC with significantly more memory. A good example is renting computing time from a cloud provider like Amazon Web Services, which offers workstations with tens of gigabytes of RAM for less than a dollar per hour.

Make use of a database that is relational

Relational databases are a standard method of storing and retrieving massive datasets.

Internally, data is saved on a disk, loaded in batches, and searched using a standard query language (SQL).

Most (all?) programming languages and many machine learning tools can connect directly to relational databases, as can free open-source database solutions like MySQL or Postgres. You can also use SQLite, which is a lightweight database.

Use a big data platform to your advantage

In some cases, you may need to use a big data platform.

Error Monitoring Software

Airbrake’s powerful error monitoring software delivers real-time error monitoring and automatic exception reporting for all of your development projects. Airbrake’s cutting-edge web dashboard keeps you up to current on your application’s health and error rates at all times. Airbrake effortlessly interacts with all of the most common languages and frameworks, no matter what you’re working on. Furthermore, Airbrake makes it simple to adjust exception parameters while providing you complete control over the active error filter system, ensuring that only the most critical errors are collected.

Check out Airbrake’s error monitoring software for yourself and see why so many of the world’s greatest engineering teams rely on it to transform their exception handling techniques!

Summary

In this article, you learned about various strategies and methods for coping with Python Memory Error.

Would you mind letting us know in the comments section if you have used any of these methods?