如何避免PHP / WordPress内存在大文件上传中批量处理数据的致命错误？

Question

我有一个大的CSV文件，我正在上传到WordPress仪表板以导入分类术语。我写了一个小插件，它使用函数来插入每个术语，但是，该函数缓存了大量数据以检查slugs的唯一性和父术语依赖性，尽管将内存分配增加到0.5 Gb，但该进程的内存不足了大约1000个术语。wp_insert_term()

我一直想将文件拆分为可管理的块，以便批量处理数据并运行限制为1000或数据行的会话，这样每个进程都会干净利落地终止。

我一直在寻找这样的解决方案，并发现了这篇有趣的文章，内容涉及批量图像导入面临的类似问题，它概述了开发人员如何使用javascript通过向服务器和可管理块发送ajax请求来控制批处理过程。

它给了我一个想法，即在上传时读取CSV文件，逐行读取它，并向服务器发送ajax请求以处理可管理的行数。

这是实现这一目标的更好方法吗？

Answer 1

我根据问题中的链接和一些额外的修补开发了以下解决方案。在WordPress服务器端，当加载javascript文件时，我根据内存分配来确定服务器可以处理的行数，$limit = ini_get('memory_limit');$limit = wp_convert_hr_to_bytes($limit) / MB_IN_BYTES; //in MBs.switch(true){&nbsp; &nbsp; case $limit >= 512:&nbsp; &nbsp; &nbsp; &nbsp; $limit = 1000;&nbsp; &nbsp; &nbsp; &nbsp; break;&nbsp; &nbsp; default:&nbsp; &nbsp; &nbsp; &nbsp; $limit = 500;&nbsp; &nbsp; &nbsp; &nbsp; break;}wp_enqueue_script( 'my-javascript-file');wp_localize_script( 'my-javascript-file', 'cirData', array(&nbsp; &nbsp; 'limit'=>$limit));您应该根据自己的过程确定并设置自己的限制。在javascript文件中，使用jQuery，var reader,formData, lineMarker=0, csvLines, isEOF=false, $file, $form ;&nbsp; $(document).ready(function(){&nbsp; &nbsp; $file = $(':file'); //file input field&nbsp; &nbsp; $form = $('form');&nbsp; //form&nbsp; &nbsp; //when the file field changes....&nbsp; &nbsp; $file.on('change', function(){&nbsp; &nbsp; &nbsp; //check if the file field has a value.&nbsp; &nbsp; &nbsp; if($file.val()){&nbsp; &nbsp; &nbsp; &nbsp; //setup file reader.&nbsp; &nbsp; &nbsp; &nbsp; reader = new FileReader();&nbsp; &nbsp; &nbsp; &nbsp; //now listen for when the file is ready to be read.&nbsp; &nbsp; &nbsp; &nbsp; reader.addEventListener('load', function (e) {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; csvLines = e.target.result.split("\n");&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; batchProcess(); //launch process.&nbsp; &nbsp; &nbsp; &nbsp; });&nbsp; &nbsp; &nbsp; &nbsp; //when the form is being submitted, start reading the file.&nbsp; &nbsp; &nbsp; &nbsp; $(document).on('click', ':submit', function(e){&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; e.preventDefault(); //disable normal submit.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; //setup data for the ajax.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; formData = new FormData($form.get(0));&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; //read the file and batch request to server.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; reader.readAsBinaryString($file.get(0).files[0]);&nbsp; &nbsp; &nbsp; &nbsp; })&nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; })&nbsp; });&nbsp; // Methods&nbsp; //posting&nbsp; function postCSVdata(csvdata){&nbsp; &nbsp; formData.set('csvlines', csvdata); //set the current datat to send.&nbsp; &nbsp; $.ajax({&nbsp; &nbsp; &nbsp; type: 'POST',&nbsp; &nbsp; &nbsp; url: $form.attr('action'),&nbsp; &nbsp; &nbsp; data: formData,&nbsp; &nbsp; &nbsp; contentType: false,&nbsp; &nbsp; &nbsp; processData: false,&nbsp; &nbsp; &nbsp; cache: false,&nbsp; &nbsp; &nbsp; success: function(data){&nbsp; &nbsp; &nbsp; &nbsp; var msg ="";&nbsp; &nbsp; &nbsp; &nbsp; if(isEOF){ //is this end of the file?&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; console.log("success!");&nbsp; &nbsp; &nbsp; &nbsp; }else{ //continue reading file.&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; console.log("uploaded:"+ Math.round(lineMarker/csvLines.length*100)+"%");&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; batchProcess(); //process the next part of the file.&nbsp; &nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; &nbsp; }&nbsp; &nbsp; })&nbsp; }&nbsp; //batch process.&nbsp; function batchProcess(){&nbsp; &nbsp; //csvlines is the array containing all the lines read from the file.&nbsp; &nbsp; //lineMarker is the index of the last line read.&nbsp; &nbsp; var parsedata='', stop = csvLines.length - lineMarker, line='';&nbsp; &nbsp; for(var i = 0; i < stop; i++) {&nbsp; &nbsp; &nbsp; line = csvLines[i+lineMarker];&nbsp; &nbsp; &nbsp; parsedata +=line+"\n"; //add a new line char for server to process.&nbsp; &nbsp; &nbsp; //check if max limit of lines server can process is reached.&nbsp; &nbsp; &nbsp; if(i>(cirData.limit-2)) break; //batch limit.&nbsp; &nbsp; }&nbsp; &nbsp; lineMarker += i;&nbsp; &nbsp; if(i==stop) isEOF = true;&nbsp; &nbsp; postCSVdata(parsedata); //send to server.&nbsp; }这将以服务器能够处理的行块的顺序方式发送多个 AJAX 请求，而不会出现致命的内存错误。