Python 如何使用pyspark从spark sql获取结果？
目前，我是spark的新手，我正在使用python在spark中编写代码 我能够读取拼花地板文件，并将数据存储在dataframe和temp表中 但它不会打印执行的查询结果。请帮助调试这个 代码：Python 如何使用pyspark从spark sql获取结果？,python,apache-spark,pyspark,apache-spark-sql,Python,Apache Spark,Pyspark,Apache Spark Sql,目前，我是spark的新手，我正在使用python在spark中编写代码 我能够读取拼花地板文件，并将数据存储在dataframe和temp表中 但它不会打印执行的查询结果。请帮助调试这个 代码： import os os.environ['SPARK_HOME']="/opt/apps/spark-2.0.1-bin-hadoop2.7/" from pyspark import SparkContext from pyspark.sql import SQLContext from pysp
import os
os.environ['SPARK_HOME']="/opt/apps/spark-2.0.1-bin-hadoop2.7/"
from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
sc = SparkContext(master='local')
sqlCtx = SQLContext(sc)
df_tract_alpha = sqlCtx.read.parquet("tract_alpha.parquet")
print (df_tract_alpha.columns)
sqlCtx.registerDataFrameAsTable(df_tract_alpha, "table1")
nt = sqlCtx.sql("SELECT COUNT(*) AS pageCount FROM table1 WHERE pp_count>=500").collect()
n1 = nt[0].pageCount
print n1
Column< pageCount['pageCount'] > instead of printing the value
Column而不是打印值
nt = sqlCtx.sql("SELECT COUNT(*) AS pageCount FROM table1 WHERE pp_count>=500") \
.collect()
$> parquet-tools head data.parquet/
a = 1
pp_count = 500
a = 2
pp_count = 750
a = 3
pp_count = 400
a = 4
pp_count = 600
a = 5
pp_count = 700
sc = SparkContext(master='local')
sqlContext = SQLContext(sc)
df = sqlContext.read.parquet("data.parquet")
print("data columns : {} ".format(df.columns))
sqlContext.registerDataFrameAsTable(df, "table1")
results = sqlContext.sql("SELECT COUNT(*) AS pageCount FROM table1 WHERE pp_count>=500").collect()
df.show()
print("initial data count : {}".format(df.count()))
page_count = results[0].pageCount
print("page count : {}".format(page_count))
data columns : ['a', 'pp_count']
+---+--------+
| a|pp_count|
+---+--------+
| 1| 500|
| 2| 750|
| 3| 400|
| 4| 600|
| 5| 700|
+---+--------+
initial data count : 5
page count : 4