I'm generating a list filled with sublists of randomly generated 0s and 1s, and then trying to compare each list with every other list to determine their similarity, efficiently.

我正在生成一个列表,其中填充了随机生成的0和1的子列表,然后尝试将每个列表与每个其他列表进行比较,以有效地确定它们的相似性。

I know that my code works with a single process (i.e. without involving multiprocessing, but once I start involving multiprocessing.Pool() or multiprocessing.Process() everything starts to break.

我知道我的代码只使用一个进程(即不涉及多处理,但是一旦我开始涉及multiprocessing.Pool()或multiprocessing.Process(),一切都开始破坏。

I want to compare how long a single process would take compared to multiple processes. I've tried this with threading, but a single process actually ended up taking less time, probably due to the Global Interpreter Lock.

我想比较单个流程与多个流程相比需要多长时间。我已尝试使用线程,但单个进程实际上最终花费的时间更少,可能是由于Global Interpreter Lock。

Here's my code:

这是我的代码:

import difflib
import secrets
import timeit
import multiprocessing
import numpy

random_lists = [[secrets.randbelow(2) for _ in range(500)] for _ in range(500)]
random_lists_split = numpy.array_split(numpy.array(random_lists), 5)


def get_similarity_value(lists_to_check, sublists_to_check) -> list:
    ratios = []
    matcher = difflib.SequenceMatcher()
    for sublist_major in sublists_to_check:
        try:
            sublist_major = sublist_major.tolist()
        except AttributeError:
            pass
        for sublist_minor in lists_to_check:
            if sublist_major == sublist_minor or [lists_to_check.index(sublist_major), lists_to_check.index(sublist_minor)] in [ratios[i][1] for i in range(len(ratios))] or [lists_to_check.index(sublist_minor), lists_to_check.index(sublist_major)] in [ratios[i][1] for i in range(len(ratios))]:  # or lists_to_check.index(sublist_major.tolist()) > lists_to_check.index(sublist_minor):
                pass
            else:
                matcher.set_seqs(sublist_major, sublist_minor)
                ratios.append([matcher.ratio(), sorted([lists_to_check.index(sublist_major), lists_to_check.index(sublist_minor)])])
    return ratios


def start():
    test = multiprocessing.Pool(4)
    data = [(random_lists, random_lists_split[i]) for i in range(len(random_lists_split))]
    print(test.map(get_similarity_value, data))


statement = timeit.Timer(start)
print(statement.timeit(1))

statement2 = timeit.Timer(lambda: get_similarity_value(random_lists, random_lists))
print(statement2.timeit(1))

And here's the error:

这是错误:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "timings.py", line 38, in <module>
    print(statement.timeit(1))
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\timeit.py", line 178, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 6, in inner
  File "timings.py", line 32, in start
    test = multiprocessing.Pool(4)
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\pool.py", line 174, in __init__
    self._repopulate_pool()
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\pool.py", line 239, in _repopulate_pool
    w.start()
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "C:\ProgramData\Anaconda3\envs\Computing Coursework\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError: 
    An attempt has been made to start a new process before the
    current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

N.B. I have tried using multiprocessing.freeze_support() but it results in the same error. The code also seems to be attempting to run indefinitely, as the error appears over and over again.

注:我尝试过使用multiprocessing.freeze_support()但它会导致相同的错误。代码似乎也试图无限期地运行,因为错误一再出现。

Thanks!

1 个解决方案

if __name__ == '__main__':
    statement = timeit.Timer(start)
    print(statement.timeit(1))

    statement2 = timeit.Timer(lambda: get_similarity_value(random_lists, random_lists))
    print(statement2.timeit(1))