Spring批处理-从S3读取多个文件

就像从s3读取spring批处理中的单个文件一样，我们使用

@Bean
public FlatFileItemReader<Map<String, Object>> itemReader() {
    FlatFileItemReader<Map<String, Object>> reader = new FlatFileItemReader<>();
    reader.setLineMapper(new JsonLineMapper());
    reader.setRecordSeparatorPolicy(new JsonRecordSeparatorPolicy());
    reader.setResource(resourceLoader.getResource("s3://" + amazonS3Bucket + "/" + file));
    return reader;
}

但是，如果我想从某个特定的文件夹/密钥读取所有文件，那么MultiResourceItemReader是否有一些功能，比如下面我们用于本地文件系统的功能

MultiResourceItemReader<UserData> reader = new MultiResourceItemReader<>();
reader.setResources(resources);

---

不，由您创建资源数组并将其传递给MultiResourceItemReader。

不，由您创建资源数组并将其传递给MultiResourceItemReader。

创建这样的MultiResourceItemReader

@Autowired
private AmazonS3 s3;

@Autowired
private ResourceLoader resourceLoader;


public MultiResourceItemReader<String> fileItemReader() throws Exception {

    List<Resource> resourceList = new ArrayList<>();

    String s3ResponseFilePath = "s3://bucket/path/"; //put your s3 path here

    //TODO: warn: this functn can only return max 1000 objects
    s3objects = s3.listObjects("bucket", s3ResponseFilePath).getObjectSummaries();

    for(S3ObjectSummary it:s3objects)
        resourceList.add(resourceLoader.getResource( "s3://" + s3Config.getBucket() + "/" + it.getKey()));

    Resource[] resources = resourceList.toArray(new Resource[resourceList.size()]);

    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(flatFileItemReader());

    return reader;
}

private FlatFileItemReader<String> flatFileItemReader() throws Exception {

    FlatFileItemReader<String> reader = new FlatFileItemReader<>();
    JsonLineMapper lineMapper = new JsonLineMapper();
    reader.setLineMapper(lineMapper);
    reader.afterPropertiesSet();

    return reader;
}


public class JsonLineMapper implements LineMapper<String> {

    private ObjectMapper mapper = new ObjectMapper();

    @Override
    public String mapLine(String s, int i) throws Exception {

        return s;
    }
}

该读取器需要一个委托和线条映射器，您可以这样实现它

@Autowired
private AmazonS3 s3;

@Autowired
private ResourceLoader resourceLoader;


public MultiResourceItemReader<String> fileItemReader() throws Exception {

    List<Resource> resourceList = new ArrayList<>();

    String s3ResponseFilePath = "s3://bucket/path/"; //put your s3 path here

    //TODO: warn: this functn can only return max 1000 objects
    s3objects = s3.listObjects("bucket", s3ResponseFilePath).getObjectSummaries();

    for(S3ObjectSummary it:s3objects)
        resourceList.add(resourceLoader.getResource( "s3://" + s3Config.getBucket() + "/" + it.getKey()));

    Resource[] resources = resourceList.toArray(new Resource[resourceList.size()]);

    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(flatFileItemReader());

    return reader;
}

private FlatFileItemReader<String> flatFileItemReader() throws Exception {

    FlatFileItemReader<String> reader = new FlatFileItemReader<>();
    JsonLineMapper lineMapper = new JsonLineMapper();
    reader.setLineMapper(lineMapper);
    reader.afterPropertiesSet();

    return reader;
}


public class JsonLineMapper implements LineMapper<String> {

    private ObjectMapper mapper = new ObjectMapper();

    @Override
    public String mapLine(String s, int i) throws Exception {

        return s;
    }
}

---

创建这样的MultiResourceItemReader

@Autowired
private AmazonS3 s3;

@Autowired
private ResourceLoader resourceLoader;


public MultiResourceItemReader<String> fileItemReader() throws Exception {

    List<Resource> resourceList = new ArrayList<>();

    String s3ResponseFilePath = "s3://bucket/path/"; //put your s3 path here

    //TODO: warn: this functn can only return max 1000 objects
    s3objects = s3.listObjects("bucket", s3ResponseFilePath).getObjectSummaries();

    for(S3ObjectSummary it:s3objects)
        resourceList.add(resourceLoader.getResource( "s3://" + s3Config.getBucket() + "/" + it.getKey()));

    Resource[] resources = resourceList.toArray(new Resource[resourceList.size()]);

    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(flatFileItemReader());

    return reader;
}

private FlatFileItemReader<String> flatFileItemReader() throws Exception {

    FlatFileItemReader<String> reader = new FlatFileItemReader<>();
    JsonLineMapper lineMapper = new JsonLineMapper();
    reader.setLineMapper(lineMapper);
    reader.afterPropertiesSet();

    return reader;
}


public class JsonLineMapper implements LineMapper<String> {

    private ObjectMapper mapper = new ObjectMapper();

    @Override
    public String mapLine(String s, int i) throws Exception {

        return s;
    }
}

该读取器需要一个委托和线条映射器，您可以这样实现它

@Autowired
private AmazonS3 s3;

@Autowired
private ResourceLoader resourceLoader;


public MultiResourceItemReader<String> fileItemReader() throws Exception {

    List<Resource> resourceList = new ArrayList<>();

    String s3ResponseFilePath = "s3://bucket/path/"; //put your s3 path here

    //TODO: warn: this functn can only return max 1000 objects
    s3objects = s3.listObjects("bucket", s3ResponseFilePath).getObjectSummaries();

    for(S3ObjectSummary it:s3objects)
        resourceList.add(resourceLoader.getResource( "s3://" + s3Config.getBucket() + "/" + it.getKey()));

    Resource[] resources = resourceList.toArray(new Resource[resourceList.size()]);

    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(flatFileItemReader());

    return reader;
}

private FlatFileItemReader<String> flatFileItemReader() throws Exception {

    FlatFileItemReader<String> reader = new FlatFileItemReader<>();
    JsonLineMapper lineMapper = new JsonLineMapper();
    reader.setLineMapper(lineMapper);
    reader.afterPropertiesSet();

    return reader;
}


public class JsonLineMapper implements LineMapper<String> {

    private ObjectMapper mapper = new ObjectMapper();

    @Override
    public String mapLine(String s, int i) throws Exception {

        return s;
    }
}

---

我在问…我可以使用MultiResourceItemReader读取多个s3文件吗？如果可以，那么如何读取？可以。您需要创建一个s3资源数组，并将其传递给MultiResourceItemReader。@GauravRaghav-您能够实现吗？你能出示一些密码吗？@Pra_A-是的，我刚发了我的answer@GauravRaghav如果我的回答有帮助，请接受它：。谢谢。我在问……我可以使用MultiResourceItemReader读取多个s3文件吗？如果可以，那么如何读取？可以。您需要创建一个s3资源数组，并将其传递给MultiResourceItemReader。@GauravRaghav-您能够实现吗？你能出示一些密码吗？@Pra_A-是的，我刚发了我的answer@GauravRaghav如果我的回答有帮助，请接受它：。非常感谢。