S3对象(公共URL)到使用JAVA编程的Google云存储
希望使用Java编程方式将AWS S3公共URL中的文件传输到Google云存储 每个S3文件由第三方自动生成,URL每天提供。为了能够完全访问这些数据,我想将这些文件传输到我们的谷歌云存储中进行进一步分析 无法在计划时间创建传输作业,因为S3 URL是随机的,我们在日常检索时会知道S3对象(公共URL)到使用JAVA编程的Google云存储,java,amazon-s3,google-cloud-storage,file-transfer,gsutil,Java,Amazon S3,Google Cloud Storage,File Transfer,Gsutil,希望使用Java编程方式将AWS S3公共URL中的文件传输到Google云存储 每个S3文件由第三方自动生成,URL每天提供。为了能够完全访问这些数据,我想将这些文件传输到我们的谷歌云存储中进行进一步分析 无法在计划时间创建传输作业,因为S3 URL是随机的,我们在日常检索时会知道 我在java中找不到任何与gsutil相关的代码。您可以像中一样使用google库。为了开始传输数据,您首先需要创建一个传输作业。传输作业管理和协调数据传输。您可以从Google云控制台创建传输作业(如文档所述),
我在java中找不到任何与gsutil相关的代码。您可以像中一样使用google库。为了开始传输数据,您首先需要创建一个传输作业。传输作业管理和协调数据传输。您可以从Google云控制台创建传输作业(如文档所述),也可以按如下编程方式进行:
package com.google.cloud.storage.storagetransfer.samples;
import com.google.api.services.storagetransfer.v1.Storagetransfer;
import com.google.api.services.storagetransfer.v1.model.AwsAccessKey;
import com.google.api.services.storagetransfer.v1.model.AwsS3Data;
import com.google.api.services.storagetransfer.v1.model.Date;
import com.google.api.services.storagetransfer.v1.model.GcsData;
import com.google.api.services.storagetransfer.v1.model.Schedule;
import com.google.api.services.storagetransfer.v1.model.TimeOfDay;
import com.google.api.services.storagetransfer.v1.model.TransferJob;
import com.google.api.services.storagetransfer.v1.model.TransferSpec;
import java.io.IOException;
import java.io.PrintStream;
/**
* Creates a one-off transfer job from Amazon S3 to Google Cloud Storage.
*/
public final class AwsRequester {
/**
* Creates and executes a request for a TransferJob from Amazon S3 to Cloud Storage.
*
* <p>The {@code startDate} and {@code startTime} parameters should be set according to the UTC
* Time Zone. See:
* https://developers.google.com/resources/api-libraries/documentation/storagetransfer/v1/java/latest/com/google/api/services/storagetransfer/v1/model/Schedule.html#getStartTimeOfDay()
*
* @return the response TransferJob if the request is successful
* @throws InstantiationException
* if instantiation fails when building the TransferJob
* @throws IllegalAccessException
* if an illegal access occurs when building the TransferJob
* @throws IOException
* if the client failed to complete the request
*/
public static TransferJob createAwsTransferJob(
String projectId,
String jobDescription,
String awsSourceBucket,
String gcsSinkBucket,
String startDate,
String startTime,
String awsAccessKeyId,
String awsSecretAccessKey)
throws InstantiationException, IllegalAccessException, IOException {
Date date = TransferJobUtils.createDate(startDate);
TimeOfDay time = TransferJobUtils.createTimeOfDay(startTime);
TransferJob transferJob =
new TransferJob()
.setDescription(jobDescription)
.setProjectId(projectId)
.setTransferSpec(
new TransferSpec()
.setAwsS3DataSource(
new AwsS3Data()
.setBucketName(awsSourceBucket)
.setAwsAccessKey(
new AwsAccessKey()
.setAccessKeyId(awsAccessKeyId)
.setSecretAccessKey(awsSecretAccessKey)))
.setGcsDataSink(new GcsData().setBucketName(gcsSinkBucket)))
.setSchedule(
new Schedule()
.setScheduleStartDate(date)
.setScheduleEndDate(date)
.setStartTimeOfDay(time))
.setStatus("ENABLED");
Storagetransfer client = TransferClientCreator.createStorageTransferClient();
return client.transferJobs().create(transferJob).execute();
}
public static void run(PrintStream out)
throws InstantiationException, IllegalAccessException, IOException {
String projectId = TransferJobUtils.getPropertyOrFail("projectId");
String jobDescription = TransferJobUtils.getPropertyOrFail("jobDescription");
String awsSourceBucket = TransferJobUtils.getPropertyOrFail("awsSourceBucket");
String gcsSinkBucket = TransferJobUtils.getPropertyOrFail("gcsSinkBucket");
String startDate = TransferJobUtils.getPropertyOrFail("startDate");
String startTime = TransferJobUtils.getPropertyOrFail("startTime");
String awsAccessKeyId = TransferJobUtils.getEnvOrFail("AWS_ACCESS_KEY_ID");
String awsSecretAccessKey = TransferJobUtils.getEnvOrFail("AWS_SECRET_ACCESS_KEY");
TransferJob responseT =
createAwsTransferJob(
projectId,
jobDescription,
awsSourceBucket,
gcsSinkBucket,
startDate,
startTime,
awsAccessKeyId,
awsSecretAccessKey);
out.println("Return transferJob: " + responseT.toPrettyString());
}
/**
* Output the contents of a successfully created TransferJob.
*/
public static void main(String[] args) {
try {
run(System.out);
} catch (Exception e) {
e.printStackTrace();
}
}
}
请添加到目前为止您已经尝试过的代码。有些人可能会对这样一个事实感到困惑,即在这种情况下,确实需要使用一次性调动作业,因此您可能希望解释为什么这是真的。