Apache spark Bluemix Apache Spark：通过Spark提交从python文件访问对象存储_Apache Spark_Ibm Cloud_Object Storage

Apache spark Bluemix Apache Spark：通过Spark提交从python文件访问对象存储

apache-spark ibm-cloud

Apache spark Bluemix Apache Spark：通过Spark提交从python文件访问对象存储,apache-spark,ibm-cloud,object-storage,Apache Spark,Ibm Cloud,Object Storage,我正在使用bluemix的Apache Spark服务。我目前无法通过spark submit访问对象存储我知道该文件存在，可通过jupyter笔记本上的swift访问。我使用以下内容来验证： file_name = "swift://notebooks.spark/small.verbatim" text_file = sc.textFile(file_name) print "number of verbatims", text_file.count() 输出为： number of

我正在使用bluemix的Apache Spark服务。我目前无法通过spark submit访问对象存储

我知道该文件存在，可通过jupyter笔记本上的swift访问。我使用以下内容来验证：

file_name = "swift://notebooks.spark/small.verbatim"
text_file = sc.textFile(file_name)
print "number of verbatims", text_file.count()

输出为：

number of verbatims 100

但是，当我尝试对spark submit执行相同操作时，我得到了一个错误

这是我通过spark submit提交的代码：

import sys, traceback
from pymongo import MongoClient
import time
from datetime import datetime
from pyspark import SparkContext
sc = SparkContext('local', 'Schedule Insight Extractor')

try:
    file_name = "swift://notebooks.spark/small.verbatim"  # small dataset
    text_file = sc.textFile(file_name)
    r = None
    r = "number of verbatims", text_file.count()
except:
    e = sys.exc_info()[0]
    print ("ERROR %s", e)
    traceback.print_exc(file=sys.stdout)

这会引发以下异常：

Traceback (most recent call last):
  File "/gpfs/fs01/user/sf6d-7c3a9c08343577-05540e1c503a/data/workdir/spark-driver-cece5080-17dd-48e4-9036-52788e5a7b77/test_spark_submit.py", line 20, in <module>
    r = "number of verbatims", text_file.count()
  File "/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/pyspark.zip/pyspark/rdd.py", line 1004, in count
    return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
  File "/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/pyspark.zip/pyspark/rdd.py", line 995, in sum
    return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
  File "/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/pyspark.zip/pyspark/rdd.py", line 869, in fold
    vals = self.mapPartitions(func).collect()
  File "/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/pyspark.zip/pyspark/rdd.py", line 771, in collect
    port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
  File "/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/py4j-0.9-src.zip/py4j/protocol.py", line 308, in get_return_value
    format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.lang.NullPointerException
        at org.apache.commons.httpclient.HttpMethodBase.getStatusCode(HttpMethodBase.java:570)
        at org.apache.hadoop.fs.swift.exceptions.SwiftInvalidResponseException.<init>(SwiftInvalidResponseException.java:53)
        at org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException(SwiftRestClient.java:1827)
        at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1728)
        at org.apache.hadoop.fs.swift.http.SwiftRestClient.perform(SwiftRestClient.java:1662)
        at org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate(SwiftRestClient.java:1154)
        at org.apache.hadoop.fs.swift.http.SwiftRestClient.authIfNeeded(SwiftRestClient.java:1618)
        at org.apache.hadoop.fs.swift.http.SwiftRestClient.preRemoteCommand(SwiftRestClient.java:1634)
        at org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest(SwiftRestClient.java:1085)
        at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat(SwiftNativeFileSystemStore.java:258)
        at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:213)
        at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata(SwiftNativeFileSystemStore.java:182)
        at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus(SwiftNativeFileSystem.java:174)
        at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
        at org.apache.hadoop.fs.Globber.glob(Globber.java:252)
        at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1644)
        at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:257)
        at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
        at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
        at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
        at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
        at org.apache.spark.api.python.PythonRDD.getPartitions(PythonRDD.scala:58)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
        at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1934)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:926)
        at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:405)
        at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:507)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
        at py4j.Gateway.invoke(Gateway.java:259)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:209)
        at java.lang.Thread.run(Thread.java:785)

回溯（最近一次呼叫最后一次）：
文件“/gpfs/fs01/user/sf6d-7c3a9c08343577-05540e1c503a/data/workdir/spark-driver-cece5080-17dd-48e4-9036-52788e5a7b77/test_spark_submit.py”，第20行，in
r=“逐字记录的数量”，text_file.count（）
文件“/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/pyspark.zip/pyspark/rdd.py”，第1004行，计数
返回self.mapPartitions（lambda i:[sum（i中的u为1）]）.sum（）
文件“/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/pyspark.zip/pyspark/rdd.py”，第995行，总计
返回self.mapPartitions（lambda x:[求和（x）]）.fold（0，运算符.add）
文件“/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/pyspark.zip/pyspark/rdd.py”，第869行，折叠
vals=self.mapPartitions（func.collect（））
文件“/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/pyspark.zip/pyspark/rdd.py”，第771行，在collect中
port=self.ctx.\u jvm.PythonRDD.collectAndServe（self.\u jrdd.rdd（））
文件“/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py”，第813行，在_u调用中__
回答，self.gateway\u客户端，self.target\u id，self.name）
文件“/usr/local/src/spark160master/spark-1.6.0-bin-2.6.0/python/lib/py4j-0.9-src.zip/py4j/protocol.py”，第308行，在get\u返回值中
格式（目标id，“.”，名称），值）
Py4JJavaError:调用z:org.apache.spark.api.python.PythonRDD.collectAndServe时出错。
：java.lang.NullPointerException
位于org.apache.commons.httpclient.HttpMethodBase.getStatusCode（HttpMethodBase.java:570）
位于org.apache.hadoop.fs.swift.exceptions.SwiftInvalidResponseException.（SwiftInvalidResponseException.java:53）
位于org.apache.hadoop.fs.swift.http.SwiftRestClient.buildException（SwiftRestClient.java:1827）
位于org.apache.hadoop.fs.swift.http.SwiftRestClient.perform（SwiftRestClient.java:1728）
位于org.apache.hadoop.fs.swift.http.SwiftRestClient.perform（SwiftRestClient.java:1662）
位于org.apache.hadoop.fs.swift.http.SwiftRestClient.authenticate（SwiftRestClient.java:1154）
位于org.apache.hadoop.fs.swift.http.SwiftRestClient.authIfNeeded（SwiftRestClient.java:1618）
位于org.apache.hadoop.fs.swift.http.SwiftRestClient.preRemoteCommand（SwiftRestClient.java:1634）
位于org.apache.hadoop.fs.swift.http.SwiftRestClient.headRequest（SwiftRestClient.java:1085）
位于org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.stat（SwiftNativeFileSystemStore.java:258）
位于org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata（SwiftNativeFileSystemStore.java:213）
位于org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObjectMetadata（SwiftNativeFileSystemStore.java:182）
位于org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.getFileStatus（SwiftNativeFileSystem.java:174）
位于org.apache.hadoop.fs.Globber.getFileStatus（Globber.java:57）
位于org.apache.hadoop.fs.Globber.glob（Globber.java:252）
位于org.apache.hadoop.fs.FileSystem.globStatus（FileSystem.java:1644）
位于org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus（FileInputFormat.java:257）
位于org.apache.hadoop.mapred.FileInputFormat.listStatus（FileInputFormat.java:228）
位于org.apache.hadoop.mapred.FileInputFormat.getSplits（FileInputFormat.java:313）
位于org.apache.spark.rdd.HadoopRDD.getPartitions（HadoopRDD.scala:199）
位于org.apache.spark.rdd.rdd$$anonfun$partitions$2.apply（rdd.scala:239）
位于org.apache.spark.rdd.rdd$$anonfun$partitions$2.apply（rdd.scala:237）
在scala.Option.getOrElse（Option.scala:120）
位于org.apache.spark.rdd.rdd.partitions（rdd.scala:237）
位于org.apache.spark.rdd.MapPartitionsRDD.getPartitions（MapPartitionsRDD.scala:35）
位于org.apache.spark.rdd.rdd$$anonfun$partitions$2.apply（rdd.scala:239）
位于org.apache.spark.rdd.rdd$$anonfun$partitions$2.apply（rdd.scala:237）
在scala.Option.getOrElse（Option.scala:120）
位于org.apache.spark.rdd.rdd.partitions（rdd.scala:237）
位于org.apache.spark.api.python.PythonRDD.getPartitions（PythonRDD.scala:58）
位于org.apache.spark.rdd.rdd$$anonfun$partitions$2.apply（rdd.scala:239）
位于org.apache.spark.rdd.rdd$$anonfun$partitions$2.apply（rdd.scala:237）
在scala.Option.getOrElse（Option.scala:120）
位于org.apache.spark.rdd.rdd.partitions（rdd.scala:237）
位于org.apache.spark.SparkContext.runJob（SparkContext.scala:1934）
位于org.apache.spark.rdd.rdd$$anonfun$collect$1.apply（rdd.scala:927）
位于org.apache.spark.rdd.RDDOperationScope$.withScope（RDDOperationScope.scala:150）
位于org.apache.spark.rdd.RDDOperationScope$.withScope（RDDOperationScope.scala:111）
位于org.apache.spark.rdd.rdd.withScope（rdd.scala:316）
位于org.apache.spark.rdd.rdd.collect（rdd.scala:926）
位于org.apache.spark.api.python.PythonRDD$.collectAndServe（PythonRDD.scala:405）
位于org.apache.spark.api.python.PythonRDD.collectAndServe（PythonRDD.scala）
在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）处
位于sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:95）
在sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:55）中
位于java.lang.reflect.Method.invoke（Method.java:507）
在py4j.reflection.MethodInvoker.invoke（MethodInvoker.java:231
prefix = "fs.swift.service." + creds['name']
hconf = sc._jsc.hadoopConfiguration()
hconf.set(prefix + ".auth.url", creds['auth_url'] + '/v2.0/tokens')
hconf.set(prefix + ".auth.endpoint.prefix", "endpoints")
hconf.set(prefix + ".tenant", creds['project_id'])
hconf.set(prefix + ".username", creds['user_id'])
hconf.set(prefix + ".password", creds['password'])
hconf.setInt(prefix + ".http.port", 8080)
hconf.set(prefix + ".region", creds['region'])
hconf.setBoolean(prefix + ".public", True)

'username': 'XXXXXXXXXXX',
'password': 'XXXXXXXXXXX',
'auth_url': 'https://identity.open.softlayer.com',
'project': 'XXXXXXXXXXX',
'project_id': 'XXXXXXXXXXX',
'region': 'dallas',
'user_id': 'XXXXXXXXXXX',
'domain_id': 'XXXXXXXXXXX',
'domain_name': 'XXXXXXXXXXX',
'filename': 'small.verbatim',
'container': 'notebooks',
'tenantId': ‘XXXXXXXXX'

hconf.set(prefix + ".auth.url", creds['auth_url'] + '/v2.0/tokens')

hconf.set(prefix + ".auth.url", creds['auth_url']+'/v3/auth/tokens')