Java 从CSV文件缓慢读取

Java 从CSV文件缓慢读取,java,csv,Java,Csv,我正在尝试读取csv文件,但速度很慢。下面是大致解释的代码: private static Film[] readMoviesFromCSV() { // Regex to split by comma without splitting in double quotes. // https://regexr.com/3s3me <- example on this data var pattern = Pattern.compile(",(?=(?:[^\\\"]

我正在尝试读取csv文件,但速度很慢。下面是大致解释的代码:

private static Film[] readMoviesFromCSV() {
    // Regex to split by comma without splitting in double quotes.
    // https://regexr.com/3s3me <- example on this data
    var pattern = Pattern.compile(",(?=(?:[^\\\"]*\\\"[^\\\"]*\\\")*[^\\\"]*$)");
    Film[] films = null;
    try (var br = new BufferedReader(new FileReader(FILENAME))) {
        var start = System.currentTimeMillis();
        var temparr = br.lines().skip(1).collect(Collectors.toList());  // skip first line and read into List
        films = temparr.stream().parallel()
                .map(pattern::split)
                .filter(x -> x.length == 24 && x[7].equals("en")) // all fields(total 24) and english speaking movies
                .filter(x -> (x[14].length() > 0)) // check if it has x[14] (date)
                .map(movieData -> new Film(movieData[8], movieData[9], movieData[14], movieData[22], movieData[23], movieData[7]))
                // movieData[8] = String title, movieData[9] = String overview
                // movieData[14] = String date (constructor parses it to LocalDate object)
                // movieData[22] = String avgRating
                .toArray(Film[]::new);
        System.out.println(MessageFormat.format("Execution time: {0}", (System.currentTimeMillis() - start)));
        System.out.println(films.length);
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }
    return films;
}
private static Film[]readMoviesFromCSV(){
//正则表达式以逗号分隔,不以双引号分隔。

// https://regexr.com/3s3me 这里是univocity parsers库的作者。您可以通过如下方式重写您在编辑中发布的代码,从而进一步加快代码的速度:

    //initialize an arraylist with a good size to avoid reallocation
    final ArrayList<Film> films = new ArrayList<Film>(20000);
    CsvParserSettings parserSettings = new CsvParserSettings();
    parserSettings.setLineSeparatorDetectionEnabled(true);
    parserSettings.setHeaderExtractionEnabled(true);

    //don't generate strings for columns you don't want
    parserSettings.selectIndexes(7, 8, 9, 14, 22, 23);

    //keep generating rows with the same number of columns found in the input
    //indexes not selected will have nulls as they are not processed.
    parserSettings.setColumnReorderingEnabled(false); 

    parserSettings.setProcessor(new AbstractRowProcessor(){
        @Override
        public void rowProcessed(String[] row, ParsingContext context) {
            if(row.length == 24 && "en".equals(row[7]) && row[14] != null){
                films.add(new Film(row[8], row[9], row[14], row[22], row[23], row[7]));
            }
        }
    });

    CsvParser parser = new CsvParser(parserSettings);
    long start = System.currentTimeMillis();
    try {
        parser.parse(new File(FILENAME), "UTF-8"); 
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }

    System.out.printf(MessageFormat.format("Time: {0}",(System.currentTimeMillis()-start)));
    return films.toArray(new Film[0]);
//使用合适的大小初始化arraylist以避免重新分配
最终ArrayList影片=新ArrayList(20000);
CsvParserSettings parserSettings=新的CsvParserSettings();
已启用parserSettings.setLineSeparatorDetectionEnabled(true);
parserSettings.setHeaderExtractionEnabled(true);
//不要为不需要的列生成字符串
选择索引(7、8、9、14、22、23);
//继续生成与输入中找到的列数相同的行
//未选择的索引将具有空值,因为它们未被处理。
parserSettings.setColumnReorderingEnabled(false);
setProcessor(新的AbstractRowProcessor(){
@凌驾
public void rowProcessed(字符串[]行,ParsingContext上下文){
if(row.length==24&&“en”.equals(row[7])&&row[14]!=null){
添加(新胶片(第[8]行、第[9]行、第[14]行、第[22]行、第[23]行、第[7]行));
}
}
});
CsvParser parser=新的CsvParser(parserSettings);
长启动=System.currentTimeMillis();
试一试{
parser.parse(新文件(文件名),“UTF-8”);
}catch(filenotfounde异常){
e、 printStackTrace();
}
printf(MessageFormat.format(“Time:{0}”,(System.currentTimeMillis()-start));
返回电影。toArray(新电影[0]);
为了方便起见,如果你必须把东西分成不同的类,你也可以在你的
电影
类中


希望这有帮助。

如果删除
.parallel()
?不确定是否有帮助here@azro同样的,如果没有并行,可能会慢一点,但它的计时不正确,所以我不能确定。我将尝试使用一些库来读取它。要更精确地计算时间,请使用.nanoTime()更准确^^请注意,您不是在测量读取文件所需的时间。您是在测量读取文件和处理文件所需的时间。很可能是读取文件速度快,但处理速度慢。正则表达式的速度不快,逗号的“手动”搜索速度更快(可能是)
    //initialize an arraylist with a good size to avoid reallocation
    final ArrayList<Film> films = new ArrayList<Film>(20000);
    CsvParserSettings parserSettings = new CsvParserSettings();
    parserSettings.setLineSeparatorDetectionEnabled(true);
    parserSettings.setHeaderExtractionEnabled(true);

    //don't generate strings for columns you don't want
    parserSettings.selectIndexes(7, 8, 9, 14, 22, 23);

    //keep generating rows with the same number of columns found in the input
    //indexes not selected will have nulls as they are not processed.
    parserSettings.setColumnReorderingEnabled(false); 

    parserSettings.setProcessor(new AbstractRowProcessor(){
        @Override
        public void rowProcessed(String[] row, ParsingContext context) {
            if(row.length == 24 && "en".equals(row[7]) && row[14] != null){
                films.add(new Film(row[8], row[9], row[14], row[22], row[23], row[7]));
            }
        }
    });

    CsvParser parser = new CsvParser(parserSettings);
    long start = System.currentTimeMillis();
    try {
        parser.parse(new File(FILENAME), "UTF-8"); 
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }

    System.out.printf(MessageFormat.format("Time: {0}",(System.currentTimeMillis()-start)));
    return films.toArray(new Film[0]);