Apache flink 阿帕奇·弗林克:什么是;类X不包含字段Y的setter“;什么意思?
我第一次使用flink(1.6,1.7)并使用来自github存档的数据,但将该数据用作流数据源 我的简单示例只是统计每天窗口中每个用户的所有事件,我尝试复制相同的示例,但使用TableEnvironment和SQL支持 但是,我遇到了以下错误: class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit不包含字段修改时间的setter,如下所示:Apache flink 阿帕奇·弗林克:什么是;类X不包含字段Y的setter“;什么意思?,apache-flink,flink-streaming,Apache Flink,Flink Streaming,我第一次使用flink(1.6,1.7)并使用来自github存档的数据,但将该数据用作流数据源 我的简单示例只是统计每天窗口中每个用户的所有事件,我尝试复制相同的示例,但使用TableEnvironment和SQL支持 但是,我遇到了以下错误: class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit不包含字段修改时间的setter,如下所示: 8-12-04 14:17:02:115 I
8-12-04 14:17:02:115 INFO main exploration.StreamingTableApp:32 - Starting Streaming Table Flink App Example...
18-12-04 14:17:02:174 INFO main typeutils.TypeExtractor:1818 - class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does not contain a setter for field modificationTime
18-12-04 14:17:02:176 INFO main typeutils.TypeExtractor:1857 - Class class org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit cannot be used as a POJO type because not all fields are valid POJO fields, and must be processed as GenericType. Please read the Flink documentation on "Data Types & Serialization" for details of the effect on performance.
18-12-04 14:17:02:937 INFO main exploration.StreamingTableApp:74 - Finished...
我将CSV源代码作为数据流读取,并使用Gson解析json行的位并将这些属性映射到元组
有人对此有任何想法/经验吗
主要方法:
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
// Mapped in docker compose file too.
DataStreamSource<String> input = env.readTextFile("/some/path/github/");
// Setup the stream
DataStream<Tuple4<String, Integer, String, Long>> stream = input.map(new GithubTupleConverter())
.assignTimestampsAndWatermarks(new TupleTimestampExtractor());
StreamTableEnvironment tEnv = TableEnvironment.getTableEnvironment(env);
Table tableFromStream = tEnv.fromDataStream(stream, "user_id, kount, basic_date,event_date");
StreamExecutionEnvironment env=StreamExecutionEnvironment.getExecutionEnvironment();
环境setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
//也映射到docker compose文件中。
DataStreamSource input=env.readTextFile(“/some/path/github/”);
//设置流
DataStream stream=input.map(新的gitubtupleconverter())
.assignTimestampsAndWatermarks(新的TupleTimestampExtractor());
StreamTableEnvironment tEnv=TableEnvironment.getTableEnvironment(env);
表tableFromStream=tEnv.fromDataStream(流,“用户id、kount、基本日期、事件日期”);
TupleTimestampExtractor
public class TupleTimestampExtractor
extends BoundedOutOfOrdernessTimestampExtractor<Tuple4<String, Integer, String, Long>> {
private static final long serialVersionUID = 3737675541965835242L;
public TupleTimestampExtractor() {
super(Time.seconds(30L));
}
@Override
public long extractTimestamp(Tuple4<String, Integer, String, Long> element) {
return element.getField(3);
}
}
公共类TupleTimestampExtractor
扩展BoundedAutofordernessTimestampExtractor{
私有静态最终长serialVersionUID=3737675541965835242L;
公共TupleTimestampExtractor(){
超级(时间秒(30L));
}
@凌驾
公共长提取时间戳(Tuple4元素){
返回元素getField(3);
}
}
gitubtupleconverter.java
public class GithubTupleConverter implements MapFunction<String, Tuple4<String, Integer, String, Long>> {
private static final Gson g = new Gson();
@Override
public Tuple4<String, Integer, String, Long> map(String value) throws Exception {
// Take each line as Json.
JsonObject o = g.fromJson(value, JsonObject.class);
// Extract the user id
String userId = o.get("actor").getAsJsonObject().get("login").getAsString();
// Extract the event type (commit, pull request, fork event)
String type = o.get("type").getAsString();
// Get the event date time
String dateTime = o.get("created_at").getAsString();
// Parse date string to Typed type.
LocalDateTime eventTime = LocalDateTime.parse(dateTime, DateTimeFormatter.ISO_DATE_TIME);
// Format the date so it can be used in the output.
DateTimeFormatter formatter = DateTimeFormatter.ISO_DATE;
return Tuple4.of(userId, 1, formatter.format(eventTime), eventTime.toInstant(ZoneOffset.UTC).toEpochMilli());
}
}
公共类GitubTupleConverter实现MapFunction{
私有静态最终Gson g=新Gson();
@凌驾
公共Tuple4映射(字符串值)引发异常{
//将每一行作为Json。
JsonObject o=g.fromJson(值,JsonObject.class);
//提取用户id
字符串userId=o.get(“actor”).getAsJsonObject().get(“login”).getAsString();
//提取事件类型(提交、拉取请求、fork事件)
字符串类型=o.get(“类型”).getAsString();
//获取事件日期和时间
字符串dateTime=o.get(“created_at”).getAsString();
//将日期字符串解析为类型化类型。
LocalDateTime eventTime=LocalDateTime.parse(dateTime,DateTimeFormatter.ISO_DATE_TIME);
//设置日期格式,以便在输出中使用。
DateTimeFormatter formatter=DateTimeFormatter.ISO_日期;
返回Tuple4.of(userId,1,formatter.format(eventTime),eventTime.toInstant(ZoneOffset.UTC.toEpochMilli());
}
}
您共享的日志不会显示错误。日志位于INFO
级别,不会引发异常(至少不会在提供的日志中)
日志条目只是说明类TimestampedFileInputSplit
不能被视为POJO。通常,此消息表示性能不是最佳的,但在这种特殊情况下,这不是问题
您是否收到其他错误消息?好的,这是我的错。读起来像是个错误。但实际上不会抛出任何异常。谢谢@Fabian。我收到了太多这样的日志。它会把我的硬盘炸了。有没有办法将其静音?确实,日志消息在INFO下。但是信息本身(据我所知…)让我觉得有些事情真的错了,我无法理解信息的实际影响。