Java 从google数据流到google云firestore的输出
日安,在努力掌握这项技术数据流之后,我已经设法使管道100%正常工作 它所做的是将一堆CSV文件从google云存储加载到管道中,将它们转换为域对象,然后以JSON格式保存到文件中 不过,我想做的是将JSON对象直接推送到数据库google cloud firestore 在此阶段,我应用于数据的最终转换是:Java 从google数据流到google云firestore的输出,java,json,firebase,google-cloud-firestore,google-cloud-dataflow,Java,Json,Firebase,Google Cloud Firestore,Google Cloud Dataflow,日安,在努力掌握这项技术数据流之后,我已经设法使管道100%正常工作 它所做的是将一堆CSV文件从google云存储加载到管道中,将它们转换为域对象,然后以JSON格式保存到文件中 不过,我想做的是将JSON对象直接推送到数据库google cloud firestore 在此阶段,我应用于数据的最终转换是: .apply(DatastoreIO.v1().write().withProjectId("____")); 据我所知,该调用需要上一次转换返回一个实体对象,而我无法创建该实体对象 p
.apply(DatastoreIO.v1().write().withProjectId("____"));
据我所知,该调用需要上一次转换返回一个实体对象,而我无法创建该实体对象
public Entity toEntity() {
Datastore datastore = DatastoreOptions.getDefaultInstance().getService();
Key taskKey = datastore.newKeyFactory().setKind("Task").newKey("Test");
Entity e = Entity.newBuilder(taskKey).set("Domain", domain)
.set("LocationOnsite", locOnSite)
.set("Company", company).build();
return e;
}
这将返回com.google.cloud.datastore.Entity,而不是所需的com.google.datastore.v1.Entity
我认为值得注意的是,域对象还包含一些其他对象的ArrayList,例如需要包含在数据库中的电子邮件
下面是我目前拥有的JSON输出示例:
{
"Vertical": "Business And Industrial",
"Zip": "35229",
"Company": "Alabama Association of Nonprofits",
"QuantCast": "229219",
"Twitter": "",
"Vimeo": "",
"LocationOnSite": "",
"LastIndexed": "2018-02-01",
"Pinterest": "",
"Youtube": "",
"TechSpend": "$250+",
"Emails": [
{
"Email": "shannon@alabamanonprofits.org"
},
{
"Email": "support@alabamanonprofits.org"
},
{
"Email": "carla@alabamanonprofits.org"
},
{
"Email": "kellie@alabamanonprofits.org"
},
{
"Email": "ashley@alabamanonprofits.org"
},
{
"Email": "Unknown"
}
],
"Facebook": "",
"Google+": "",
"Alexa": "",
"Github": "",
"FirstIndexed": "2011-01-03",
"People": [
{
"Email": "Unknown",
"Name": "Joshua Cirulnick"
},
{
"Email": "Unknown",
"Position": "Other",
"Name": " Elaine Lin"
},
{
"Email": "Unknown",
"Position": "Other",
"Name": " Terry Burkle"
},
{
"Email": "Unknown",
"Position": "Director",
"Name": " Ashley Gilbert"
},
{
"Email": "Unknown",
"Position": "President",
"Name": " Carol Weisman"
},
{
"Email": "Unknown",
"Position": "Csuite",
"Name": " Shannon Ammons"
},
{
"Email": "Unknown",
"Position": "Founder",
"Name": " Kelly McDonald"
}
],
"City": "Birmingham",
"Telephone#s": [
{
"Telephone#": "+1-205-879-4712"
},
{
"Telephone#": "+1-205-871-7740"
}
],
"FirstDetected": "N/A",
"LinkedIn": "",
"VK": "",
"State": "AL",
"Instagram": "",
"Country": "US",
"Domain": "alabamanonprofits.org",
"LastFound": "N/A"
}
如果有人能为我指明正确的方向,告诉我如何有效地将这些对象导入google cloud firestore数据库,我将非常高兴 您可以将数据写入Cloud Pub/Sub,这可以触发一个函数将数据写入Cloud Firestore。Google I/O 2017就是一个很好的例子,它也做了同样的事情,但是使用了实时数据库
您可以在此处观看:谢谢,这似乎非常可行!我去看看