Emr 为什么在presto的自定义UDAF中找不到返回类型

我所做的自定义函数在EMR上的presto中不起作用

我想创建一个简单的UDAF，它只返回

42

首先，我的自定义函数是我写的一个简单函数，但在presto中不起作用

presto cli中出现错误：

presto> select answer_to_life('the universe');
Query 20180324_120433_00000_7n6s6 failed: answer_to_life(varchar):bigint not found
com.facebook.presto.spi.PrestoException: answer_to_life(varchar):bigint not found
        at com.facebook.presto.metadata.FunctionRegistry.doGetSpecializedFunctionKey(FunctionRegistry.java:972)
        at com.google.common.cache.CacheLoader$FunctionToCacheLoader.load(CacheLoader.java:146)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211)
        at com.google.common.cache.LocalCache.get(LocalCache.java:4154)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147)
        at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:5153)
        at com.facebook.presto.metadata.FunctionRegistry.getSpecializedFunctionKey(FunctionRegistry.java:898)
        at com.facebook.presto.metadata.FunctionRegistry.getAggregateFunctionImplementation(FunctionRegistry.java:875)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.buildAccumulatorFactory(LocalExecutionPlanner.java:1973)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.planGlobalAggregation(LocalExecutionPlanner.java:1984)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:955)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:1919)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitExchange(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:196)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:952)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitAggregation(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.AggregationNode.accept(AggregationNode.java:167)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:638)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner$Visitor.visitOutput(LocalExecutionPlanner.java:596)
        at com.facebook.presto.sql.planner.plan.OutputNode.accept(OutputNode.java:82)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:393)
        at com.facebook.presto.sql.planner.LocalExecutionPlanner.plan(LocalExecutionPlanner.java:324)
        at com.facebook.presto.execution.SqlTaskExecution.<init>(SqlTaskExecution.java:161)
        at com.facebook.presto.execution.SqlTaskExecution.createSqlTaskExecution(SqlTaskExecution.java:121)
        at com.facebook.presto.execution.SqlTaskExecutionFactory.create(SqlTaskExecutionFactory.java:71)
        at com.facebook.presto.execution.SqlTask.updateTask(SqlTask.java:340)
        at com.facebook.presto.execution.SqlTaskManager.updateTask(SqlTaskManager.java:321)
        at com.facebook.presto.server.TaskResource.createOrUpdateTask(TaskResource.java:128)
        at sun.reflect.GeneratedMethodAccessor311.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
        at io.airlift.http.server.TraceTokenFilter.doFilter(TraceTokenFilter.java:63)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at io.airlift.http.server.TimingFilter.doFilter(TimingFilter.java:52)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:454)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
        at org.eclipse.jetty.server.Server.handle(Server.java:564)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
        at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128)
        at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:199)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
        at java.lang.Thread.run(Thread.java:748)

详细代码在这里（）

我部署的结构如下所示：

$ls-1/usr/lib/presto/plugin/my-udaf/
commons-codec-1.4.jar
番石榴-21.0.jar
hive-udf-1.0-SNAPSHOT.jar
presto-array-0.197.jar
stats-0.155.jar
presto-udaf-1.0-SNAPSHOT.jar

我使用
emr-5.12.0
（
Presto 0.188
）

谢谢你抽出时间


Edited1

jar文件中的列表

% jar -tf target/presto-udaf-1.0-SNAPSHOT.jar 
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
META-INF/
META-INF/MANIFEST.MF
META-INF/services/
sample/
META-INF/services/com.facebook.presto.spi.Plugin
sample/AnswerToLife.class
sample/AnswerToLifePlugin.class
sample/NullState.class
META-INF/maven/
META-INF/maven/sample/
META-INF/maven/sample/presto-udaf/
META-INF/maven/sample/presto-udaf/pom.xml
META-INF/maven/sample/presto-udaf/pom.properties

服务器日志

$ grep answer -i  /mnt/var/log/presto/server.log
2018-03-26T06:37:14.213Z        INFO    main    com.facebook.presto.server.PluginManager        Installing sample.AnswerToLifePlugin
2018-03-26T06:37:14.214Z        INFO    main    com.facebook.presto.server.PluginManager        Registering functions from sample.AnswerToLife

并执行
show functions

presto> show functions;
            Function             |                     Return Type                     |                                 Argument Types
---------------------------------+-----------------------------------------------------+------------------------------------------------------------------------------
 ST_Area                         | double                                              | Geometry
 ST_AsText                       | varchar                                             | Geometry
 ...
 answer_to_life                  | bigint                                              | varchar
 ...


Edited2

删除
src/main/resources/META-INF/services/com.facebook.presto.spi.Plugin

向pom.xml添加
presto插件


我在
/usr/lib/presto/plugin/
下部署了Maven制作的zip

但是，错误的内容没有改变


Edited3

我在Mac上从github（分支0.188）获得了源代码，并构建了presto

当我们把上面的UDAF放在它的presto上时，它工作得非常好

也许在EMR上的presto安装过程中有一个错误。
从您所写的内容来看，缺少两件事


确保在pom.xml中使用了maven packaging
presto plugin
还要实现
com.facebook.presto.spi.Plugin
，它在
getFunctions
方法中会返回您的函数

查看
presto geospatial
presto模块，并查看
com.facebook.presto.plugin.geospatial.GeoPlugin
及其pom.xml作为参考。
已解决

自定义插件不起作用，因为它不是插件创建，但在EMR上部署的方式是错误的

由于我没有在执行的过程中部署到所有节点，因此我通过将以下shell注册为引导操作来确认操作

#/bin/bash
aws s3 cp s3://hogehoge/presto-udaf-1.0-SNAPSHOT.zip/tmp/
sudomkdir-p/usr/lib/presto/plugin/
sudo解压-d/usr/lib/presto/plugin//tmp/presto-udaf-1.0-SNAPSHOT.zip




必须在Presto集群中的所有节点上安装插件
（协调员和工人）




您可以使用引导操作将对象从AmazonS3复制到每个
安装应用程序之前，在群集中的节点。AWS CLI
安装在群集的每个节点上，因此可以执行引导操作
调用AWS CLI命令

谢谢你的建议。但是我已经使用了Maven，并实现了
getFunctions
。（github中有所有代码。）
com.facebook.presto.spi.Plugin
文件应该生成。在您的源代码中没有这一点。你重新开始普雷斯托了吗？您是否在Presto日志中看到您的插件在Presto启动期间加载？也可以使用LongState。状态必须是可序列化和可实例化的。添加一个postscript。我已经生成了
com.facebook.presto.spi.Plugin
（）。我已经重新启动了。LongState和NullState之间没有区别，但是LongState是首选的吗？()