Warning: file_get_contents(/data/phpspider/zhask/data//catemap/3/apache-spark/5.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Apache spark 如何处理dataframe中的所有列并生成新列_Apache Spark_Row_Rdd_Spark Dataframe - Fatal编程技术网

Apache spark 如何处理dataframe中的所有列并生成新列

Apache spark 如何处理dataframe中的所有列并生成新列,apache-spark,row,rdd,spark-dataframe,Apache Spark,Row,Rdd,Spark Dataframe,这是一个例子 我有一个数据帧 +---+---------+----+------------+-----+ | id| name| age| describle|value| +---+---------+----+------------+-----+ | 0|hansongbo|null| he is great| 11| | 1| marong| 23| null| null| | 2|hansongbo| 34| null| n

这是一个例子

我有一个数据帧

+---+---------+----+------------+-----+
| id|     name| age|   describle|value|
+---+---------+----+------------+-----+
|  0|hansongbo|null| he is great|   11|
|  1|   marong|  23|        null| null|
|  2|hansongbo|  34|        null| null|
|  3|   marong|  34|she love han|   12|
+---+---------+----+------------+-----+
现在我想计算所有字符串列的长度,结果是

 +---+---------+----+------------+-----+------+
| id|     name| age|   describle|value|length|
+---+---------+----+------------+-----+------+
|  0|hansongbo|null| he is great|   11|   20 |
|  1|   marong|  23|        null| null|   6  |
|  2|hansongbo|  34|        null| null|   9  |
|  3|   marong|  34|she love han|   12|   18 |
+---+---------+----+------------+-----+------+

我怎样才能做到,希望你的帮助,谢谢

让我们将数据帧命名为A

A.registerTempTable("tableA")
b = sqlContext.sql("select `id`,`name`,age,describle,value,length(`name`)+length(`describle`) as `length` from tableA")

欢迎来到SO!到目前为止你试过什么?这确实是一个质量很低的问题。试着表现出你做了一些努力,这样我们就愿意帮助你了!