I got the following error when running reverse-geocoder. Also, the machine got really hit for a while and I could not even move the mouse.
Traceback (most recent call last):
File «», line 1, in
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 105, in spawn_main
Traceback (most recent call last):
File «», line 1, in
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 105, in spawn_main
exitcode = _main(fd)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 114, in _main
prepare(preparation_data)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 225, in prepare
_fixup_main_from_path(data[‘init_main_from_path’])
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 277, in _fixup_main_from_path
run_name=»mp_main«)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibrunpy.py», line 263, in run_path
exitcode = _main(fd)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 114, in _main
prepare(preparation_data)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 225, in prepare
_fixup_main_from_path(data[‘init_main_from_path’])
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 277, in _fixup_main_from_path
run_name=»mp_main«)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibrunpy.py», line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibrunpy.py», line 96, in _run_module_code
pkg_name=pkg_name, script_name=fname)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibrunpy.py», line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibrunpy.py», line 85, in run_code
mod_name, mod_spec, pkg_name, script_name)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibrunpy.py», line 85, in run_code
exec(code, run_globals)
File «C:UsersMeDocumentsGoogle Location Services Analyzermain.py», line 19, in
exec(code, run_globals)
File «C:UsersMeDocumentsGoogle Location Services Analyzermain.py», line 19, in
result = rg.search(coordinates)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibsite-packagesreverse_geocoder_init.py», line 293, in search
result = rg.search(coordinates)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibsite-packagesreverse_geocoder_init.py», line 293, in search
return rg.query(geo_coords)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibsite-packagesreverse_geocoder_init.py», line 127, in query
_, indices = self.tree.pquery(coordinates, k=1)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibsite-packagesreverse_geocodercKDTree_MP.py», line 97, in pquery
return rg.query(geo_coords)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibsite-packagesreverse_geocoder_init.py», line 127, in query
_, indices = self.tree.pquery(coordinates, k=1)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibsite-packagesreverse_geocodercKDTree_MP.py», line 97, in pquery
for p in pool: p.start()
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingprocess.py», line 112, in start
for p in pool: p.start()
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingprocess.py», line 112, in start
self._popen = self._Popen(self)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingcontext.py», line 223, in _Popen
self._popen = self._Popen(self)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingcontext.py», line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingcontext.py», line 322, in _Popen
return Popen(process_obj)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingpopen_spawn_win32.py», line 46, in init
return _default_context.get_context().Process._Popen(process_obj)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingcontext.py», line 322, in _Popen
return Popen(process_obj)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingpopen_spawn_win32.py», line 46, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 143, in get_preparation_data
_check_not_importing_main()
File «C:UsersMeAppDataLocalContinuumanaconda3envsGoogle Location Services Analyzerlibmultiprocessingspawn.py», line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.»’)
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.
prep_data = spawn.get_preparation_data(process_obj._name)
У меня есть приложение Python с интенсивной памятью (от сотни МБ до нескольких ГБ).
У меня есть несколько ОЧЕНЬ МАЛЕНЬКИХ Linux-исполняемых файлов, которые нужно запустить главному приложению, например.
child = Popen("make html", cwd = r'../../docs', stdout = PIPE, shell = True)
child.wait()
Когда я запускаю эти внешние утилиты (один раз, в конце длинного основного процесса), используя subprocess.Popen, я иногда получаю OSError: [Errno 12] Cannot allocate memory.
Я не понимаю, почему… Запрошенный процесс крошечный!
В системе достаточно памяти для многих других оболочек.
Я использую Linux (Ubuntu 12.10, 64 бит), поэтому я думаю, что подпроцесс вызывает Fork.
И Fork разворачивает мой существующий процесс, тем самым удваивая количество потребляемой памяти и терпя неудачу?
Что случилось с копией на запись?
Можно ли создать новый процесс без вилки (или, по крайней мере, без копирования памяти — начиная с нуля)?
по теме:
Разница между fork(), vfork(), exec() и clone()
fork() и поведение распределения памяти
Подпроцесс Python.Popen с помощью OSError: [Errno 12] Невозможно выделить память через период времени
Ошибка выделения памяти Python с помощью подпроцесса. Открытие
Ответ 1
Не похоже, что появится реальное решение (т.е. альтернативная реализация подпроцесса, использующего vfork). Так как насчет симпатичного взлома? В начале вашего процесса создайте ведомое устройство, которое висит вокруг с небольшим размером памяти, готовым к появлению ваших подпроцессов и поддерживая открытую связь с ним на протяжении всего жизненного цикла основного процесса.
Вот пример использования rfoo (http://code.google.com/p/rfoo/) с именованным сокетом unix, называемым rfoosocket (вы, очевидно, могли бы использовать другие типы соединений, поддерживающие rfoo, или другая библиотека RPC):
Сервер:
import rfoo
import subprocess
class MyHandler(rfoo.BaseHandler):
def RPopen(self, cmd):
c = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
c.wait()
return c.stdout.read()
rfoo.UnixServer(MyHandler).start('rfoosocket')
Клиент:
import rfoo
# Waste a bunch of memory before spawning the child. Swap out the RPC below
# for a straight popen to show it otherwise fails. Tweak to suit your
# available system memory.
mem = [x for x in range(100000000)]
c = rfoo.UnixConnection().connect('rfoosocket')
print rfoo.Proxy(c).RPopen('ls -l')
Если вам требуется взаимодействие в реальном времени с обратным и четвертым взаимодействием с вашими нерешенными подпроцессами, эта модель, вероятно, не сработает, но вы можете ее взломать. Предположительно вы хотите очистить доступные аргументы, которые могут быть переданы Popen на основе ваших конкретных потребностей, но это должно быть относительно простым.
Вы также должны просто запустить сервер в начале работы клиента и управлять файлом сокета (или портом), который будет очищен при выходе.
It doesn’t appear that a real solution will be forthcoming (i.e. an alternate implementation of subprocess that uses vfork). So how about a cute hack? At the beginning of your process, spawn a slave that hangs around with a small memory footprint, ready to spawn your subprocesses, and keep open communication to it throughout the life of the main process.
Here’s an example using rfoo (http://code.google.com/p/rfoo/) with a named unix socket called rfoosocket (you could obviously use other connection types rfoo supports, or another RPC library):
Server:
import rfoo
import subprocess
class MyHandler(rfoo.BaseHandler):
def RPopen(self, cmd):
c = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
c.wait()
return c.stdout.read()
rfoo.UnixServer(MyHandler).start('rfoosocket')
Client:
import rfoo
# Waste a bunch of memory before spawning the child. Swap out the RPC below
# for a straight popen to show it otherwise fails. Tweak to suit your
# available system memory.
mem = [x for x in range(100000000)]
c = rfoo.UnixConnection().connect('rfoosocket')
print rfoo.Proxy(c).RPopen('ls -l')
If you need real-time back and forth coprocess interaction with your spawned subprocesses this model probably won’t work, but you might be able to hack it in. You’ll presumably want to clean up the available args that can be passed to Popen based on your specific needs, but that should all be relatively straightforward.
You should also find it straightforward to launch the server at the start of the client, and to manage the socket file (or port) to be cleaned up on exit.
In python, I am using os.fork() within a for condition. I am doing something like this:
for i in range(1,4):
try:
pid = os.fork()
except OSError:
print ("Error forking process")
continue
if pid == 0:
print "In child process"
os._exit(0)
print "In parent process"
How do I simulate a failure of os.fork() so that I can be sure that «In parent process» is not printed? How to simulate OSError for os.fork() function?
asked Aug 30, 2018 at 12:20
1
You can just replace os.fork() with your own implementation:
import os
def _fork():
raise OSError()
os.fork = _fork
for i in range(1,4):
try:
pid = os.fork()
except OSError:
print ("Error forking process")
continue
if pid == 0:
print "In child process"
os._exit(0)
print "In parent process"
If this is part of a test, have a look at the unittest.mock module.
answered Aug 30, 2018 at 12:26
Erik CederstrandErik Cederstrand
9,5487 gold badges39 silver badges63 bronze badges
1
In this article, we’ll talk about the Python fork() method which allows us to create child processes using the processes through which it was called. Let’s look
What is a system call?
A system call is used to get access to the kernel mode. It is a way with which a program can interact with the operating system and request its services to perform specified tasks.
Hence all the system calls are executed in the kernel (privileged) mode when the execution completes the control is returned to the user mode. Sometimes a system call is also called syscall in abbreviated form.
In modern computer systems, there are generally two modes of operations – User mode and Kernel mode.
All the user programs and processes run in User mode and they cannot access the kernel-mode directly. This is done to protect the operating system from the modifications or changes caused by any user program.
If any user program needs to perform privileged work, it requires the support of the operating system which it can get only through the system calls. And it the system calls which provide the various services of the operating system to the user programs and processes via API (Application Program Interface).
What is Python fork()?
There are multiple system calls for managing different types of services provided by the operating system. Also, these are different for different operating systems.
System calls are broadly classified into five main categories:
- Files related system calls
- Device related system calls
- Process related system calls
- Information related system calls
- Communication-related system calls
So, Python fork() is an example of Process related or Process control system call. It is a method used to create a Child process of the process which calls it just like the clone of the original (calling) process. The process which makes a system call using the fork() statement is called the Parent process.
Importing the OS module
In order to use and implement the fork() system call in Python, we require the os module of Python. This os module in Python allows us to use the various operating system-dependent functionality. It allows the user program to access the operating system functionality on which the Python is running.
We need not install the os module as it comes under Python’s standard utility modules and gets installed when Python is installed in our system.
It is imported inside the program as follows:
Working of Python fork()
Following are the key properties of fork() method/statement:
- It takes no argument/parameter when it is being called inside any program.
- If the child process is created successfully then both the parent process and the child process will execute the next statement/instruction followed by the
fork()statement. - The number of child processes = 2N – 1; where N = Number of
fork()statements used inside the main program.
On execution of the Python fork() statement it returns three types of integer values:
- Zero (0) is returned to the child process if the child process is created successfully.
- A positive (+ve) value is returned to the parent process if the child process is created successfully. This positive value is usually the PID that is the **process ID of the newly created child process.
- A negative (-ve) value is returned to the parent process if some error occurs in the creation of the child process due to any reason.
Process ID usually referred to as PID is a unique identifier associated with every process present inside the computer system. The process ID of any process can be accessed using another system call getpid() which is an example of Information related system calls. The getpid() statement returns the process ID of the process which calls it.
It is called inside the program as follows:
Creating child processes with the fork() method in Python
Example 1:
# Importing os module
import os
# Creating child processes using fork() method
os.fork()
os.fork()
# This will be executed by both parent & child processes
print("AskPython")
Output:
The above python program has produced the correct output as there are two fork() statements used inside the program. Hence the print() statement executed four times (3 + 1) three (22 -1 = 3) times by the three child processes and one time by the parent process.
Example 2:
# Importing os module
import os
# Creating a child process using fork() method
val = os.fork()
# Testing the values returned by fork() method
if val == 0:
pid = os.getpid()
print(f"Hi I am Child Process and my PID is {pid}.")
elif val > 0:
pid = os.getpid()
print(f"Hi I am Parent Process and my PID is {pid} and PID {val} is my Child Process.")
else:
print("Sorry!! Child Process creation has failed...")
Output:
The above python program has produced the correct output and clearly demonstrated the values returned by the fork() statement.
Conclusion
In this tutorial, we have learned what is a system call, what a Python fork() method is, its properties, and how to create child processes using the fork() method in Python.
Note: fork() method/system call is available only for Linux/Unix operating systems. If you try to run any Python program that calls the fork() method on a Windows system then the following error will be produced: