Text 无法在Pyspark中执行用户定义的函数RegexTokenizer

我正在尝试使用Pyspark在数据中使用文本特征执行文本分类。下面是我的文本预处理代码，该代码给出了无法执行用户定义函数RegexOkenizer的代码

    tokenizer = RegexTokenizer(inputCol = "text", outputCol = "words", pattern = "\\W")
    add_stopwords = StopWordsRemover.loadDefaultStopWords("english")
    remover = StopWordsRemover(inputCol = "words", outputCol = "filtered").setStopWords(add_stopwords)
    label_stringIdx = StringIndexer(inputCol = "label", outputCol = "target")
    countVectors = CountVectorizer(inputCol="filtered", outputCol="features", vocabSize=1000, minDF=5)
    #pipleline for text pre-processing
    pipeline = Pipeline(stages=[tokenizer,remover, countVectors, label_stringIdx])

    #fit the dat for the pipeline
    pipelineFit = pipeline.fit(dataset)
    dataset = pipelineFit.transform(dataset)
    dataset.show()

错误是：

/usr/local/lib/python3.6/dist-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o589.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 92.0 failed 1 times, most recent failure: Lost task 2.0 in stage 92.0 (TID 1317, 1e1a151fa0f5, executor driver): org.apache.spark.SparkException: Failed to execute user defined function(RegexTokenizer$$Lambda$3017/0x000000084123e040: (string) => array<string>)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
    at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NullPointerException
    at org.apache.spark.ml.feature.RegexTokenizer.$anonfun$createTransformFunc$2(Tokenizer.scala:146)
    ... 19 more

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2023)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:1972)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:1971)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1971)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:950)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:950)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2203)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2152)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2141)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:752)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2093)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2114)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2133)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2158)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1227)
    at org.apache.spark.ml.feature.CountVectorizer.fit(CountVectorizer.scala:233)
    at org.apache.spark.ml.feature.CountVectorizer.fit(CountVectorizer.scala:149)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.spark.SparkException: Failed to execute user defined function(RegexTokenizer$$Lambda$3017/0x000000084123e040: (string) => array<string>)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:729)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:489)
    at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:192)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:62)
    at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
    at org.apache.spark.scheduler.Task.run(Task.scala:127)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:444)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:447)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more
Caused by: java.lang.NullPointerException
    at org.apache.spark.ml.feature.RegexTokenizer.$anonfun$createTransformFunc$2(Tokenizer.scala:146)
    ... 19 more

获取返回值（应答、网关客户端、目标id、名称）中的

/usr/local/lib/python3.6/dist-packages/py4j/protocol.py
326 raise Py4JJavaError(
327“调用{0}{1}{2}时出错。\n”。
-->328格式（目标id，“.”，名称），值）
329其他：
330升起Py4JError(
Py4JJavaError:调用o589.fit时出错。
：org.apache.SparkException:作业因阶段失败而中止：阶段92.0中的任务2失败1次，最近的失败：阶段92.0中的任务2.0丢失（TID 1317，1e1a151fa0f5，执行器驱动程序）：org.apache.SparkException:未能执行用户定义的函数（RegexTokenizer$$Lambda$3017/0x000000084123e040:（字符串）=>数组）
位于org.apache.spark.sql.catalyst.expressions.GeneratedClass$GenerateEditorForCodeGenStage2.processNext（未知源）
位于org.apache.spark.sql.execution.BufferedRowIterator.hasNext（BufferedRowIterator.java:43）
位于org.apache.spark.sql.execution.whisttagecodegenexec$$anon$1.hasNext（whisttagecodegenexec.scala:729）
位于scala.collection.Iterator$$anon$10.hasNext（Iterator.scala:458）
位于scala.collection.Iterator$$anon$10.hasNext（Iterator.scala:458）
位于scala.collection.Iterator$$anon$10.hasNext（Iterator.scala:458）
位于scala.collection.Iterator$$anon$11.hasNext（Iterator.scala:489）
位于org.apache.spark.util.collection.ExternalSorter.insertAll（ExternalSorter.scala:192）
位于org.apache.spark.shuffle.sort.SortShuffleWriter.write（SortShuffleWriter.scala:62）
位于org.apache.spark.shuffle.shufflewWriteProcessor.write（shufflewWriteProcessor.scala:59）
在org.apache.spark.scheduler.ShuffleMapTask.runTask（ShuffleMapTask.scala:99）上
在org.apache.spark.scheduler.ShuffleMapTask.runTask（ShuffleMapTask.scala:52）上
位于org.apache.spark.scheduler.Task.run（Task.scala:127）
在org.apache.spark.executor.executor$TaskRunner.$anonfun$run$3（executor.scala:444）
位于org.apache.spark.util.Utils$.tryWithSafeFinally（Utils.scala:1377）
位于org.apache.spark.executor.executor$TaskRunner.run（executor.scala:447）
位于java.base/java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1128）
位于java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run（ThreadPoolExecutor.java:628）
位于java.base/java.lang.Thread.run（Thread.java:834）
原因：java.lang.NullPointerException
位于org.apache.spark.ml.feature.RegexTokenizer.$anonfun$createTransformFunc$2（Tokenizer.scala:146）
…还有19个
驱动程序堆栈跟踪：
位于org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages（DAGScheduler.scala:2023）
位于org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2（DAGScheduler.scala:1972）
位于org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted（DAGScheduler.scala:1971）
位于scala.collection.mutable.resizeblearray.foreach（resizeblearray.scala:62）
位于scala.collection.mutable.resizeblearray.foreach$（resizeblearray.scala:55）
位于scala.collection.mutable.ArrayBuffer.foreach（ArrayBuffer.scala:49）
位于org.apache.spark.scheduler.DAGScheduler.abortStage（DAGScheduler.scala:1971）
位于org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1（DAGScheduler.scala:950）
位于org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted（DAGScheduler.scala:950）
位于scala.Option.foreach（Option.scala:407）
位于org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed（DAGScheduler.scala:950）
位于org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive（DAGScheduler.scala:2203）
位于org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive（DAGScheduler.scala:2152）
位于org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive（DAGScheduler.scala:2141）
位于org.apache.spark.util.EventLoop$$anon$1.run（EventLoop.scala:49）
位于org.apache.spark.scheduler.DAGScheduler.runJob（DAGScheduler.scala:752）
位于org.apache.spark.SparkContext.runJob（SparkContext.scala:2093）
位于org.apache.spark.SparkContext.runJob（SparkContext.scala:2114）
位于org.apache.spark.SparkContext.runJob（SparkContext.scala:2133）
位于org.apache.spark.SparkContext.runJob（SparkContext.scala:2158）
位于org.apache.spark.rdd.rdd.count（rdd.scala:1227）
在org.apache.spark.ml.feature.CountVectorizer.fit（CountVectorizer.scala:233）
在org.apache.spark.ml.feature.CountVectorizer.fit（CountVectorizer.scala:149）
位于java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0（本机方法）
位于java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:62）
位于java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
位于java.base/java.lang.reflect.Method.invoke（Method.java:566）
位于py4j.reflection.MethodInvoker.invoke（MethodInvoker.java:244）
位于py4j.reflection.ReflectionEngine.invoke（ReflectionEngine.java:357）
在py4j.Gateway.invoke处（Gateway.java:282）
位于py4j.commands.AbstractCommand.invokeMethod（AbstractCommand.java:132）
在py4j.commands.CallCommand.execute（CallCommand.java:79）
在py4j.GatewayConnection.run处（GatewayConnection.java:238）
位于java.base/java.lang.Thread.run（Thread.java:834）
原因：org.apache.spark.SparkException:未能执行用户定义的函数（RegexTokenizer$$Lambda$3017/0x000000084123e040:（字符串）=>array）
位于org.apache.spark.sql.catalyst.expressions.GeneratedClass$GenerateEditorForCodeGenStage2.processNext（未知源）
位于org.apache.spark.sql.execution.BufferedRowIterator.hasNext（BufferedRowIterator.java:43）
位于org.apache.spark.sql.execution.whistagecodegenexec$$anon$1.0