将 80000 左右的大数据插入到 postgres 数据库中，Java 失败

Question

我有以下file.txt参考这篇文章。

它可能包含多达 400,000 个条目，我需要将它们插入到数据库中，但是对于大约 80000 到最大 100,000 的文件，该过程成功但如果超出则失败。

问题可能出在写入数据库的最后一个循环中，

FILETYPE: STUDENT

FORMATVERSION: 1.2 

ACTION: REPLACEALL

COLUMNS: NAME,SURNAME,AGE,SEX

"raj","jonson",17,"M"

"Rita","Sweety",17,"F"

这是我的示例代码

//入口点

public RestResponse provisionStudent(MultipartFormDataInput input) {

    long processed = 0L;

    HashMap<String, String> columnMaping = new HashMap<String, String>();

    //mapping here as the column name is not same as in the database

    columnMaping.put("NAME","name");

    columnMaping.put("surname","s_name");

    columnMaping.put("AGE","age");

    columnMaping.put("SEX","mf");

    try {

        String lineRead;

        List<Map<String, String>> listheader = new ArrayList<>();

        Map<String, String> obj = new LinkedHashMap<>();

        ArrayList<String> body = new ArrayList<String>();

        InputStream result;

        //checking if file contain data

        if (input == null || input.getParts() == null || input.getParts().isEmpty()) {

            throw new IllegalArgumentException("Multipart request is empty");

        }

        if (input.getParts().size() == 1) {

            InputPart filePart = input.getParts().iterator().next();

            result = filePart.getBody(InputStream.class, null);

        } else {

            result = input.getFormDataPart("file", InputStream.class, null);

        }

        BufferedReader httpResponseReader = new BufferedReader(new InputStreamReader(result));

        // reading the header part into obj and the body part in array list body

        while ((lineRead = httpResponseReader.readLine()) != null) {

            if (lineRead.contains(":")) {

                String[] headervalues = lineRead.replace(" ", "").split(":");

                obj.put(headervalues[0], headervalues[1]);

            } else {

                if (lineRead.length() > 0) body.add(lineRead.replace(" ", ""));

            }

        }

Answer 1

也许您在服务器端超时，请尝试将 statement_timeout 设置得更高。您应该检查许多超时选项。