Apache spark 在Google storage for Apache Spark SQL中访问数据

我想使用Dataproc集群中的ApacheHive查询云存储中大约30Gb的数据。访问这些数据的最佳策略是什么。通过gsutil将数据复制到我的主机并从那里访问数据的最佳方法是，还是我可以直接在云存储中访问它？如果是后者，如何在Spark CLI中指定其位置？我可以具体说明吗

location 'gs://<bucketname>' 

---

？

您应该能够创建一个外部表，直接指向云存储中的数据。这应该适用于配置单元和Spark SQL。在许多情况下，这可能是最好的策略

下面是一个基于云存储中的公共数据集的示例

CREATE EXTERNAL TABLE natality_csv (
  source_year BIGINT, year BIGINT, month BIGINT, day BIGINT, wday BIGINT, 
  state STRING, is_male BOOLEAN, child_race BIGINT, weight_pounds FLOAT, 
  plurality BIGINT, apgar_1min BIGINT, apgar_5min BIGINT, 
  mother_residence_state STRING, mother_race BIGINT, mother_age BIGINT, 
  gestation_weeks BIGINT, lmp STRING, mother_married BOOLEAN, 
  mother_birth_state STRING, cigarette_use BOOLEAN, cigarettes_per_day BIGINT, 
  alcohol_use BOOLEAN, drinks_per_week BIGINT, weight_gain_pounds BIGINT, 
  born_alive_alive BIGINT, born_alive_dead BIGINT, born_dead BIGINT, 
  ever_born BIGINT, father_race BIGINT, father_age BIGINT, 
  record_weight BIGINT 
) 
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' 
LOCATION 'gs://public-datasets/natality/csv'

---

诚然，根据对你问题的评论，我不确定我是否遗漏了你问题的另一部分

您应该能够创建一个外部表，直接指向云存储中的数据。这应该适用于配置单元和Spark SQL。在许多情况下，这可能是最好的策略

下面是一个基于云存储中的公共数据集的示例

CREATE EXTERNAL TABLE natality_csv (
  source_year BIGINT, year BIGINT, month BIGINT, day BIGINT, wday BIGINT, 
  state STRING, is_male BOOLEAN, child_race BIGINT, weight_pounds FLOAT, 
  plurality BIGINT, apgar_1min BIGINT, apgar_5min BIGINT, 
  mother_residence_state STRING, mother_race BIGINT, mother_age BIGINT, 
  gestation_weeks BIGINT, lmp STRING, mother_married BOOLEAN, 
  mother_birth_state STRING, cigarette_use BOOLEAN, cigarettes_per_day BIGINT, 
  alcohol_use BOOLEAN, drinks_per_week BIGINT, weight_gain_pounds BIGINT, 
  born_alive_alive BIGINT, born_alive_dead BIGINT, born_dead BIGINT, 
  ever_born BIGINT, father_race BIGINT, father_age BIGINT, 
  record_weight BIGINT 
) 
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' 
LOCATION 'gs://public-datasets/natality/csv'

---

诚然，根据对你问题的评论，我不确定我是否遗漏了你问题的另一部分

我现在意识到，我可以使用

location'gs://'

直接在云存储中指定位置。尽管如此，我问题的第一部分仍然有效。我现在意识到我可以使用

location'gs://'

直接在云存储中指定位置。尽管如此，我问题的第一部分仍然有效。