Apache flink Flink进程函数未将数据返回到Sideoutputstream
我试图用规则集验证JSONObject如果json与规则集匹配,它将返回匹配的规则,而JSONObject如果不匹配,它将返回一个JSONObject到Sideoutput所有这些都是在ProcessFunction中处理的,我得到的是主输出,但无法捕获Sideoutput SideOutput流定义如下Apache flink Flink进程函数未将数据返回到Sideoutputstream,apache-flink,flink-streaming,flink-cep,flink-sql,Apache Flink,Flink Streaming,Flink Cep,Flink Sql,我试图用规则集验证JSONObject如果json与规则集匹配,它将返回匹配的规则,而JSONObject如果不匹配,它将返回一个JSONObject到Sideoutput所有这些都是在ProcessFunction中处理的,我得到的是主输出,但无法捕获Sideoutput SideOutput流定义如下 public final static OutputTag<org.json.JSONObject> unMatchedJSONSideOutput = new OutputTag
public final static OutputTag<org.json.JSONObject> unMatchedJSONSideOutput = new OutputTag<org.json.JSONObject>(
"unmatched-side-output") {};
public class RuleFilter extends ProcessFunction<Tuple2<String,org.json.JSONObject>,Tuple2<String,org.json.JSONObject>> {
@Override
public void processElement(Tuple2<String, org.json.JSONObject> value,
ProcessFunction<Tuple2<String, org.json.JSONObject>, Tuple2<String, org.json.JSONObject>>.Context ctx,
Collector<Tuple2<String, org.json.JSONObject>> out) throws Exception {
if(this.value.matches((value.f1))) {
out.collect(new Tuple2<String, org.json.JSONObject>(value.f0,value.f1));
}else {
ctx.output(RuleMatching.unMatchedJSONSideOutput,value.f1);
}
}
}
public final static OutputTag unMatchedJSONSideOutput=新输出Tag(
“不匹配的边输出”){};
ProcessFunction的定义如下所示
public final static OutputTag<org.json.JSONObject> unMatchedJSONSideOutput = new OutputTag<org.json.JSONObject>(
"unmatched-side-output") {};
public class RuleFilter extends ProcessFunction<Tuple2<String,org.json.JSONObject>,Tuple2<String,org.json.JSONObject>> {
@Override
public void processElement(Tuple2<String, org.json.JSONObject> value,
ProcessFunction<Tuple2<String, org.json.JSONObject>, Tuple2<String, org.json.JSONObject>>.Context ctx,
Collector<Tuple2<String, org.json.JSONObject>> out) throws Exception {
if(this.value.matches((value.f1))) {
out.collect(new Tuple2<String, org.json.JSONObject>(value.f0,value.f1));
}else {
ctx.output(RuleMatching.unMatchedJSONSideOutput,value.f1);
}
}
}
公共类RuleFilter扩展了ProcessFunction{
@凌驾
public void processElement(Tuple2值,
ProcessFunction.Context ctx,
收集器输出)抛出异常{
if(this.value.matches((value.f1))){
collect(新的Tuple2(value.f0,value.f1));
}否则{
ctx.output(RuleMatching.unmatchidJSONSideOutput,value.f1);
}
}
}
我正在打印主数据流输出,如下所示
DataStream<Tuple2<String, org.json.JSONObject>> matchedJSON =
inputSignal.map(new MapFunction<org.json.JSONObject, Tuple2<String, org.json.JSONObject>>() {
@Override
public Tuple2<String, org.json.JSONObject> map(org.json.JSONObject input) throws Exception {
return new Tuple2<>(value, input);
}
}).process(new RuleFilter()).print("MatchedJSON=>");
matchedJSON .print("matchedJSON=>");
DataStream<org.json.JSONObject> unmatchedJSON =
((SingleOutputStreamOperator<org.json.JSONObject>) matchedJSON.map(new MapFunction<Tuple2<String, org.json.JSONObject>, org.json.JSONObject>() {
@Override
public org.json.JSONObject map(Tuple2<String, org.json.JSONObject> value) throws Exception {
return value.f1;
}
})).getSideOutput(unMatchedJSONSideOutput );
unmatchedJSON.print("unmatchedJSON=>");
数据流匹配JSON=
inputSignal.map(新的MapFunction(){
@凌驾
公共Tuple2映射(org.json.JSONObject输入)引发异常{
返回新的Tuple2(值,输入);
}
}).process(新规则过滤器()).print(“MatchedJSON=>”;
print(“matchedJSON=>”;
我正在打印侧输出,如下所示
DataStream<Tuple2<String, org.json.JSONObject>> matchedJSON =
inputSignal.map(new MapFunction<org.json.JSONObject, Tuple2<String, org.json.JSONObject>>() {
@Override
public Tuple2<String, org.json.JSONObject> map(org.json.JSONObject input) throws Exception {
return new Tuple2<>(value, input);
}
}).process(new RuleFilter()).print("MatchedJSON=>");
matchedJSON .print("matchedJSON=>");
DataStream<org.json.JSONObject> unmatchedJSON =
((SingleOutputStreamOperator<org.json.JSONObject>) matchedJSON.map(new MapFunction<Tuple2<String, org.json.JSONObject>, org.json.JSONObject>() {
@Override
public org.json.JSONObject map(Tuple2<String, org.json.JSONObject> value) throws Exception {
return value.f1;
}
})).getSideOutput(unMatchedJSONSideOutput );
unmatchedJSON.print("unmatchedJSON=>");
datastreamunmatchedjson=
((SingleOutputStreamOperator)matchedJSON.map(新的MapFunction(){
@凌驾
public org.json.JSONObject映射(Tuple2值)引发异常{
返回值为0.f1;
}
})).getSideOutput(未匹配的dJSONSideOutput);
unmatchedJSON.print(“unmatchedJSON=>”;
主流正在打印输出,但sideoutput没有打印无效的json。请帮助解决此问题。问题在这里:
datastreamunmatchedjson=
((SingleOutputStreamOperator)matchedJSON.map(…)
.getSideOutput(未匹配的dJSONSideOutput);
您应该直接在matchedJSON
上调用getSideOutput
,而不是对其应用MapFunction
的结果。只有ProcessFunction
可以有一个侧面输出,它需要直接来自ProcessFunction
。通过从映射中强制转换输出流,您诱使编译器接受了这一点,但运行时无法对此执行任何有意义的操作