如何认证和授权Spark+;Scala应用到云对象存储?

如何认证和授权Spark+;Scala应用到云对象存储?,scala,apache-spark,ibm-cloud,object-storage,Scala,Apache Spark,Ibm Cloud,Object Storage,我正在尝试使用在Bluemix上运行Spark+Scala应用程序。到目前为止,基于和我提出了以下代码片段: val spark: SparkSession = SparkSession .builder .appName("app") .config("spark.hadoop.fs.cos.softlayer.endpoint", "s3-api.us-geo.objectstorage.service.networklayer.com") .config

我正在尝试使用在Bluemix上运行Spark+Scala应用程序。到目前为止,基于和我提出了以下代码片段:

val spark: SparkSession = SparkSession
  .builder
  .appName("app")
  .config("spark.hadoop.fs.cos.softlayer.endpoint",
          "s3-api.us-geo.objectstorage.service.networklayer.com")
  .config("spark.hadoop.fs.cos.softlayer.access.key",
          "auto-generated-apikey-<redacted>")
  .config("spark.hadoop.fs.cos.softlayer.secret.key",
          "<redacted>")
  .getOrCreate()
spark.sparkContext.setLogLevel("TRACE")
spark.sparkContext.textFile("s3d://<bucket>.softlayer/<file>")
由于

DEBUG apache.http.headers: http-outgoing-0 << HTTP/1.1 403 Forbidden

DEBUG apache.http.headers:http-outgoing-0不幸的是,AE测试版的当前文档引用了,它使用AWS样式(HMAC)凭据进行身份验证,而不是IBM Cloud IAM提供的API密钥。今年晚些时候将在启用IAM的COS中支持HMAC凭据

AE文档应该很快更新为使用API密钥连接到COS的示例。同时,请尝试以下配置语法:

.config("spark.hadoop.fs.cos.iamservice.iam.endpoint",
          "https://iam.ng.bluemix.net/oidc/token")   
.config("spark.hadoop.fs.cos.iamservice.endpoint",
          "s3-api.us-geo.objectstorage.service.networklayer.com")
.config("spark.hadoop.fs.cos.iamservice.iam.api.key",
          "<api-key>")
.config("spark.hadoop.fs.cos.iamservice.iam.service.id",
          "<resource-instance-id>")
.config(“spark.hadoop.fs.cos.iamservice.iam.endpoint”,
"https://iam.ng.bluemix.net/oidc/token")   
.config(“spark.hadoop.fs.cos.iamservice.endpoint”,
“s3api.us geo.objectstorage.service.networklayer.com”)
.config(“spark.hadoop.fs.cos.iamservice.iam.api.key”,
"")
.config(“spark.hadoop.fs.cos.iamservice.iam.service.id”,
"")

除了更改配置键外,我还将URL更改为s3d://.iamservice/,并且成功了。谢谢
spark.hadoop.fs.cos.iamservice.iam.service.id
指的是什么服务?IAM服务?如何获取此服务的ID?
.config("spark.hadoop.fs.cos.iamservice.iam.endpoint",
          "https://iam.ng.bluemix.net/oidc/token")   
.config("spark.hadoop.fs.cos.iamservice.endpoint",
          "s3-api.us-geo.objectstorage.service.networklayer.com")
.config("spark.hadoop.fs.cos.iamservice.iam.api.key",
          "<api-key>")
.config("spark.hadoop.fs.cos.iamservice.iam.service.id",
          "<resource-instance-id>")