Oracle使用python将表分区到Datalake
您好,我正在尝试将Oracle分区表转换为Datalake拼花文件 使用此脚本Oracle使用python将表分区到Datalake,python,oracle,azure-data-lake,Python,Oracle,Azure Data Lake,您好,我正在尝试将Oracle分区表转换为Datalake拼花文件 使用此脚本 # Convert it to Spark SQL table and save it as parquet format df.write \ .format("parquet") \ .option("path","/archive/" + schema_name + "/" + table_name + "
# Convert it to Spark SQL table and save it as parquet format
df.write \
.format("parquet") \
.option("path","/archive/" + schema_name + "/" + table_name + ".parquet") \
.mode("append") \
.saveAsTable(table_name)
此代码获取表的所有数据,而不是分区
spark = SparkSession.builder \
.appName("Load " + schema_name + " " + table_name + " from Oracle into Parquet and creating Table") \
.getOrCreate()
这是从Oracle创建表
我怎样才能得到这个唯一的拼花:)当你从表中选择时,你说你正在从表中获取数据,并且你想要从一个特定的分区中获取数据。您是否尝试使用语法PARTION(partition\u name)放置分区名称
您有多少个分区,如果它们不太多,那么您可以尝试为每个分区创建一个视图,然后从视图中选择数据。在oracle命名检查中创建表 将分区名称添加到表中 在我可以从Spark读取这个分区名称之后
query = '(select partition_name from Schema.checkes c) checkes'
df = spark.read \
.format("jdbc") \
.option("url","jdbc:oracle:thin:@" + db_host + ":" + db_port + "/" + db_service) \
.option("dbtable",query) \
.option("user",db_user) \
.option("password",db_pass) \
.option("driver","oracle.jdbc.OracleDriver") \
.option("encoding","UTF-8") \
.option("fetchSize", 10000) \
.option("numPartitions",40) \
.load()
print("part count: " + str(df.count()))
if df.count() > 0:partition_name = df.select("partition_name").collect()[0]["partition_name"]
df1 = spark.read \
.format("jdbc") \
.option("url","jdbc:oracle:thin:@" + db_host + ":" + db_port + "/" + db_service) \
.option("dbtable",query1) \
.option("user",db_user) \
.option("password",db_pass) \
.option("driver","oracle.jdbc.OracleDriver") \
.option("encoding","UTF-8") \
.option("fetchSize", 10000) \
.option("numPartitions",40) \
.load()
```