如何使用BigQuery连接器从java spark读取BigQuery表
我试图通过spark java代码读取bigquery表,如下所示:如何使用BigQuery连接器从java spark读取BigQuery表,java,apache-spark,google-bigquery,google-cloud-dataproc,Java,Apache Spark,Google Bigquery,Google Cloud Dataproc,我试图通过spark java代码读取bigquery表,如下所示: BigQuerySQLContext bqSqlCtx = new BigQuerySQLContext(sqlContext); bqSqlCtx.setGcpJsonKeyFile("sxxxl-gcp-1x4c0xxxxxxx.json"); bqSqlCtx.setBigQueryProjectId("winged-standard-2xxxx"); bqSqlCtx.setBigQue
BigQuerySQLContext bqSqlCtx = new BigQuerySQLContext(sqlContext);
bqSqlCtx.setGcpJsonKeyFile("sxxxl-gcp-1x4c0xxxxxxx.json");
bqSqlCtx.setBigQueryProjectId("winged-standard-2xxxx");
bqSqlCtx.setBigQueryDatasetLocation("asia-east1");
bqSqlCtx.setBigQueryGcsBucket("dataproc-9cxxxxx39-exxdc-4e73-xx07- 2258xxxx4-asia-east1");
Dataset<Row> testds = bqSqlCtx.bigQuerySelect("select * from bqtestdata.customer_visits limit 100");
响应中的消息
数据集ID必须是字母数字(加下划线).
表示
数据集ID
“spark\u bigquery\u staging\u asia-east1”无效,因为其中有一个连字符,特别是在asia-east1
中,我与samelamin的Scala库有类似的问题。显然,这是因为该库无法处理美国和欧盟以外的位置,因此该库将无法访问来自asia-east1
的数据集
目前,我正在使用BigQuery Spark连接器从BigQuery加载和写入数据
如果您能够找到使用此库的解决方法,请与他人分享
19/01/14 10:52:01 WARN org.apache.spark.sql.SparkSession$Builder: Using an existing SparkSession; some configuration may not take effect.
19/01/14 10:52:01 INFO com.samelamin.spark.bigquery.BigQueryClient: Executing query select * from bqtestdata.customer_visits limit 100
19/01/14 10:52:02 INFO com.samelamin.spark.bigquery.BigQueryClient: Creating staging dataset winged-standard-2xxxxx:spark_bigquery_staging_asia-east1
Exception in thread "main" java.util.concurrent.ExecutionException: com.google.api.client.googleapis.json.GoogleJsonResponseException:
400 Bad Request
{
"code" : 400,
"errors" :
[ {
"domain" : "global",
**"message" : "Invalid dataset ID \"spark_bigquery_staging_asia-east1\". Dataset IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long.",**
"reason" : "invalid"
} ],
"message" : "Invalid dataset ID \"spark_bigquery_staging_asia-east1\". Dataset IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long.",
"status" : "INVALID_ARGUMENT"
}