我正在尝试计算特征方阵 ( Information_Gains_Matrix
) 和相应的方权重矩阵 ( Weights_Matrix
) 的莫兰指数。对于中的每个特征,Information_Gains_Matrix
我想在固定的情况下计算莫兰指数Weights_Matrix
。
因此,我尝试使用 multiprocessing pool.map 来处理Information_Gains_Matrix
. 我可以让代码在小型测试数据集上以各种方式执行此操作。然而,当我使用实际的大数据集时,代码运行,但随后CPU使用率下降到0%,进程挂起,并且没有任何释放。
import multiprocessing
from multiprocessing import RawArray, Pool, Lock
from functools import partial
import numpy as np
## Set up initial fake data
Information_Gains_Matrix = np.random.uniform(0,1,(22000,22000))
Weights_Matrix = np.random.uniform(0,1,(22000,22000))
## Function I want to parallelise.
def Feature_Moran_Index(Chunks,Wij,N):
Moran_Index_Scores = np.zeros(Chunks.shape[0])
for i in np.arange(Chunks.shape[0]):
print(Chunks[i]) # Print ind to show it's running
Feature = Information_Gains_Matrix[Chunks[i],:]
X_bar = np.mean(Feature)
if X_bar != 0:
Deviance = Feature - X_bar
Outer_Deviance = np.outer(Deviance,Deviance)
Deviance2 = Deviance * Deviance
Denom = np.sum(Deviance2)
Moran_Index_Scores[i] = (N/Wij) * (np.sum((W * np.ndarray.flatten(Outer_Deviance)))/Denom)
return Moran_Index_Scores
## Set up chunks inds for each core.
Use_Cores = (multiprocessing.cpu_count()-2)
Chunk_Size = np.ceil(Information_Gains_Matrix.shape[0] / Use_Cores)
Range = np.arange(Information_Gains_Matrix.shape[0]).astype("i")
Chunk_Range = np.arange(Chunk_Size).astype("i")
Chunks = []
for i in np.arange(Use_Cores-1):
Chunks.append(Range[Chunk_Range])
Range = np.delete(Range,Chunk_Range)
我对这种方法没有忠诚度,如果有人可以帮助我以任何方式跨特征并行计算莫兰指数,我将非常感激,因为我似乎无法让它发挥作用。
收到一只叮咚
相关分类