使用 concurrent.futures 进行并行处理

Question

我试图找到一种方法来使用不同的方法对数据帧进行并行处理，如本教程所示：https ://www.youtube.com/watch?v= fKl2JW_qrso (min >18:26)。但结果显示我出了点问题。代码的想法是在数据框中创建一个新列 ['分母']，其中包含“basalareap”、“basalareas”、“basalaread”列中每个字段的行和。任何建议这里有什么问题，我在打印时得到了这个奇怪的结果？此外，还有其他方法可以最有效地进行并行化吗？

import pandas as pd

import numpy as np

import concurrent.futures

from multiprocessing import cpu_count

np.random.seed(4)

layer = pd.DataFrame(np.random.randint(0,25,size=(10, 3)),

                  columns=list(['basalareap', 'basalareas', 'basalaread']))

def denom():

    layer['denominator'] = layer[["basalareap","basalareas","basalaread"]].sum(axis=1)

data_split = np.array_split(layer,cpu_count())

with concurrent.futures.ProcessPoolExecutor() as executor:

    results = [executor.submit(denom) for i in data_split]

print(results)

>>>print(results)

[<Future at 0x1b45e325108 state=finished raised BrokenProcessPool>, 

<Future at 0x1b45e357708 state=finished raised BrokenProcessPool>, 

<Future at 0x1b45e3577c8 state=finished raised BrokenProcessPool>, 

<Future at 0x1b45e357888 state=finished raised BrokenProcessPool>, 

<Future at 0x1b45e357948 state=finished raised BrokenProcessPool>, 

<Future at 0x1b45e357a48 state=finished raised BrokenProcessPool>, 

<Future at 0x1b45e357b08 state=finished raised BrokenProcessPool>, 

<Future at 0x1b45e357bc8 state=finished raised BrokenProcessPool>]

我的系统：Windows 10 python 3.7.4

Answer 1

这是一种可以使其工作的方法（使用您的示例数据）：import pandas as pdimport numpy as npimport concurrent.futures as cffrom multiprocessing import cpu_countnp.random.seed(4)layer = pd.DataFrame(np.random.randint(0,25,size=(10, 3)),&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; columns=list(['basalareap', 'basalareas', 'basalaread']))def denom(layer):&nbsp; &nbsp; layer['denominator'] = layer[["basalareap","basalareas","basalaread"]].sum(axis=1)&nbsp; &nbsp; return layerif __name__ == '__main__':&nbsp; &nbsp; data_split = np.array_split(layer,cpu_count())&nbsp; &nbsp; # create a function to for process tasks&nbsp; &nbsp; def cpu_tasks(func, *args):&nbsp; &nbsp; &nbsp; &nbsp; with cf.ProcessPoolExecutor() as tp:&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; result = tp.map(func, chunksize=10, *args)&nbsp; &nbsp; &nbsp; &nbsp; return list(result)&nbsp; &nbsp; # get result&nbsp; &nbsp; newdf = cpu_tasks(denom, data_split)&nbsp; &nbsp; # convert list to dataframe&nbsp; &nbsp; newdf = pd.concat(newdf)&nbsp; &nbsp; print(newdf)&nbsp; &nbsp; &nbsp; &nbsp;basalareap&nbsp; basalareas&nbsp; basalaread&nbsp; denominator&nbsp; &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 14&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 23&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;5&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;42&nbsp; &nbsp; 1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;8&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 23&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32&nbsp; &nbsp; 2&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;8&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 18&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;9&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;35&nbsp; &nbsp; 3&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;7&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 13&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 23&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;43&nbsp; &nbsp; 4&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 23&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;8&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;4&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;35&nbsp; &nbsp; 5&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 18&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 12&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;6&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;36&nbsp; &nbsp; 6&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 10&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 20&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;3&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;33&nbsp; &nbsp; 7&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 23&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 21&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;44&nbsp; &nbsp; 8&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 21&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;9&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;6&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;36&nbsp; &nbsp; 9&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;6&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 24&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;2&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;32