Pyspark 从cloudant IBM Bluemix NoSQL数据库中提取值
如何从以JSON格式存储的Cloudant IBM Bluemix NoSQL数据库中提取值 我试过这个密码Pyspark 从cloudant IBM Bluemix NoSQL数据库中提取值,pyspark,ibm-cloud,cloudant,pyspark-sql,Pyspark,Ibm Cloud,Cloudant,Pyspark Sql,如何从以JSON格式存储的Cloudant IBM Bluemix NoSQL数据库中提取值 我试过这个密码 def readDataFrameFromCloudant(host,user,pw,database): cloudantdata=spark.read.format("com.cloudant.spark"). \ option("cloudant.host",host). \ option("cloudant.username", user). \
def readDataFrameFromCloudant(host,user,pw,database):
cloudantdata=spark.read.format("com.cloudant.spark"). \
option("cloudant.host",host). \
option("cloudant.username", user). \
option("cloudant.password", pw). \
load(database)
cloudantdata.createOrReplaceTempView("washing")
spark.sql("SELECT * from washing").show()
return cloudantdata
hostname = ""
user = ""
pw = ""
database = "database"
cloudantdata=readDataFrameFromCloudant(hostname, user, pw, database)
它以这种格式存储
{
"_id": "31c24a382f3e4d333421fc89ada5361e",
"_rev": "1-8ba1be454fed5b48fa493e9fe97bedae",
"d": {
"count": 9,
"hardness": 72,
"temperature": 85,
"flowrate": 11,
"fluidlevel": "acceptable",
"ts": 1502677759234
}
}
我想要这个结果
期望
实际结果
创建一个虚拟数据集以再现问题:
cloudantdata = spark.read.json(sc.parallelize(["""
{
"_id": "31c24a382f3e4d333421fc89ada5361e",
"_rev": "1-8ba1be454fed5b48fa493e9fe97bedae",
"d": {
"count": 9,
"hardness": 72,
"temperature": 85,
"flowrate": 11,
"fluidlevel": "acceptable",
"ts": 1502677759234
}
}
"""]))
cloudantdata.take(1)
返回:
[Row(_id='31c24a382f3e4d333421fc89ada5361e', _rev='1-8ba1be454fed5b48fa493e9fe97bedae', d=Row(count=9, flowrate=11, fluidlevel='acceptable', hardness=72, temperature=85, ts=1502677759234))]
[Row(_id='31c24a382f3e4d333421fc89ada5361e', _rev='1-8ba1be454fed5b48fa493e9fe97bedae', count=9, flowrate=11, fluidlevel='acceptable', hardness=72, temperature=85, ts=1502677759234)]
现在展平:
flat_df = cloudantdata.select("_id", "_rev", "d.*")
flat_df.take(1)
返回:
[Row(_id='31c24a382f3e4d333421fc89ada5361e', _rev='1-8ba1be454fed5b48fa493e9fe97bedae', d=Row(count=9, flowrate=11, fluidlevel='acceptable', hardness=72, temperature=85, ts=1502677759234))]
[Row(_id='31c24a382f3e4d333421fc89ada5361e', _rev='1-8ba1be454fed5b48fa493e9fe97bedae', count=9, flowrate=11, fluidlevel='acceptable', hardness=72, temperature=85, ts=1502677759234)]
我使用IBM Data Science Experience笔记本,使用Python3.5(实验版)和Spark 2.0测试了这段代码 这个答案基于: