如何在SPark shell中注册Java SPark UDF?
下面是我的java udf代码如何在SPark shell中注册Java SPark UDF?,java,scala,apache-spark,user-defined-functions,Java,Scala,Apache Spark,User Defined Functions,下面是我的java udf代码 package com.udf; import org.apache.spark.sql.api.java.UDF1; public class SparkUDF implements UDF1<String, String> { @Override public String call(String arg) throws Exception { if (validateString(arg))
package com.udf;
import org.apache.spark.sql.api.java.UDF1;
public class SparkUDF implements UDF1<String, String> {
@Override
public String call(String arg) throws Exception {
if (validateString(arg))
return arg;
return "INVALID";
}
public static boolean validateString(String arg) {
if (arg == null | arg.length() != 11)
return false;
else
return true;
}
}
使用以下命令启动火花壳
spark shell--jars SparkUdf-1.0-SNAPSHOT.jar
有人能告诉我,如何在spark shell上注册UDF以在spark sql中使用它吗?经过更多的搜索,我得到了答案 以下是步骤
spark-shell --jars SparkUdf-1.0-SNAPSHOT.jar
scala> import com.udf.SparkUDF;
scala> import com.udf.SparkUDF;
import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType};
scala> spark.udf.register("myfunc", new SparkUDF(),StringType)
scala> val sql1 = """ select myfunc(name) from sample """
scala> spark.sql(sql1).show();
您将得到结果。如果您试图在S3上测试Jupyter笔记本中的UDF和您的UDF jar: 步骤1:将您的UDF JAR加载到Jupyter笔记本中:
%%configure -f
{
"conf": {
"spark.jars": "s3://s3-path/your-udf.jar"
}
}
步骤2:在pySpark中注册基于scala的UDF
spark.udf.registerJavaFunction("myudf", "<udf.package>.<UDFClass>")
也许这里有一些见解。
spark.udf.registerJavaFunction("myudf", "<udf.package>.<UDFClass>")
df = spark.read.parquet("s3://s3-path-to-test-data/ts_date=2021-04-27")
df.createOrReplaceTempView('stable')
spark.sql("select *, myudf(arg1,arg2) as result from stable ").show(5,False)