Java tln(“范围=”+范围); } 公共静态类GroupAssigner实现FlatMapFunction{ @凌驾 公共void平面图(Tuple2输入,收集器输出){ //索引1-5将分配给第1组,索引6-10将分配给第2组,等等。 int n=new Long(input.f0/5).intValue()+1; out.collect(新的Tuple2(n,(Double)input.f1.getField(0)); } }
可以将源连接到多个接收器,源只执行一次,记录被广播到多个接收器。看到这个问题了吗Java tln(“范围=”+范围); } 公共静态类GroupAssigner实现FlatMapFunction{ @凌驾 公共void平面图(Tuple2输入,收集器输出){ //索引1-5将分配给第1组,索引6-10将分配给第2组,等等。 int n=new Long(input.f0/5).intValue()+1; out.collect(新的Tuple2(n,(Double)input.f1.getField(0)); } },java,apache-flink,Java,Apache Flink,可以将源连接到多个接收器,源只执行一次,记录被广播到多个接收器。看到这个问题了吗 getExecutionEnvironment是在运行作业时获取环境的正确方法createCollectionEnvironment是一种很好的游戏和测试方法。见 异常错误消息非常清楚:如果调用print或collect,则会执行数据流。所以你有两个选择: 要么在数据流结束时调用print/collect,然后执行并打印。这对测试东西很好。请记住,每个数据流只能调用collect/print一次,否则它会在
getExecutionEnvironment
是在运行作业时获取环境的正确方法createCollectionEnvironment
是一种很好的游戏和测试方法。见
- 要么在数据流结束时调用print/collect,然后执行并打印。这对测试东西很好。请记住,每个数据流只能调用collect/print一次,否则它会在未完全定义的情况下多次执行
- 要么在数据流末尾添加一个接收器,然后调用env.execute()。这就是你想要做的,一旦你的流量是在一个更成熟的形状
Exception in thread "main" java.lang.RuntimeException: No new data sinks have been defined since the last execution. The last execution refers to the latest call to 'execute()', 'count()', 'collect()', or 'print()'.
at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:940)
at org.apache.flink.api.java.ExecutionEnvironment.createProgramPlan(ExecutionEnvironment.java:922)
at org.apache.flink.api.java.CollectionEnvironment.execute(CollectionEnvironment.java:34)
at org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:816)
at MainClass.main(MainClass.java:114)
//Table is used to calculate the standard deviation as I figured that there is no such calculation in DataSet.
BatchTableEnvironment tableEnvironment = TableEnvironment.getTableEnvironment(env);
//Get Data from a mySql database
DataSet<Row> dbData =
env.createInput(
JDBCInputFormat.buildJDBCInputFormat()
.setDrivername("com.mysql.cj.jdbc.Driver")
.setDBUrl($database_url)
.setQuery("select value from $table_name where id =33")
.setUsername("username")
.setPassword("password")
.setRowTypeInfo(new RowTypeInfo(BasicTypeInfo.DOUBLE_TYPE_INFO))
.finish()
);
// Add index for assigning group (group capacity is 5)
DataSet<Tuple2<Long, Row>> indexedData = DataSetUtils.zipWithIndex(dbData);
// Replace index(long) with group number(int), and convert Row to double at the same time
DataSet<Tuple2<Integer, Double>> rawData = indexedData.flatMap(new GroupAssigner());
//Using groupBy() to combine individual data of each group into a list, while calculating the mean and range in each group
//put them into a POJO named GroupDataClass
DataSet<GroupDataClass> groupDS = rawData.groupBy("f0").combineGroup(new GroupCombineFunction<Tuple2<Integer, Double>, GroupDataClass>() {
@Override
public void combine(Iterable<Tuple2<Integer, Double>> iterable, Collector<GroupDataClass> collector) {
Iterator<Tuple2<Integer, Double>> it = iterable.iterator();
Tuple2<Integer, Double> var1 = it.next();
int groupNum = var1.f0;
// Using max and min to calculate range, using i and sum to calculate mean
double max = var1.f1;
double min = max;
double sum = 0;
int i = 1;
// The list is to store individual value
List<Double> list = new ArrayList<>();
list.add(max);
while (it.hasNext())
{
double next = it.next().f1;
sum += next;
i++;
max = next > max ? next : max;
min = next < min ? next : min;
list.add(next);
}
//Store group number, mean, range, and 5 individual values within the group
collector.collect(new GroupDataClass(groupNum, sum / i, max - min, list));
}
});
//print because if no sink is created, Flink will not even perform the calculation.
groupDS.print();
// Get the max group number and range in each group to calculate average range
// if group number start with 1 then the maximum of group number equals to the number of group
// However, because this is the second sink, data will flow from source again, which will double the group number
DataSet<Tuple2<Integer, Double>> rangeDS = groupDS.map(new MapFunction<GroupDataClass, Tuple2<Integer, Double>>() {
@Override
public Tuple2<Integer, Double> map(GroupDataClass in) {
return new Tuple2<>(in.groupNum, in.range);
}
}).max(0).andSum(1);
// collect and print as if no sink is created, Flink will not even perform the calculation.
Tuple2<Integer, Double> rangeTuple = rangeDS.collect().get(0);
double range = rangeTuple.f1/ rangeTuple.f0;
System.out.println("range = " + range);
}
public static class GroupAssigner implements FlatMapFunction<Tuple2<Long, Row>, Tuple2<Integer, Double>> {
@Override
public void flatMap(Tuple2<Long, Row> input, Collector<Tuple2<Integer, Double>> out) {
// index 1-5 will be assigned to group 1, index 6-10 will be assigned to group 2, etc.
int n = new Long(input.f0 / 5).intValue() + 1;
out.collect(new Tuple2<>(n, (Double) input.f1.getField(0)));
}
}