Google cloud dataflow 光束不发射
我正在构建一个简单的原型,其中我正在从Pubsub读取数据,并使用BeamSQL,代码片段如下Google cloud dataflow 光束不发射,google-cloud-dataflow,apache-beam,spotify-scio,beam-sql,Google Cloud Dataflow,Apache Beam,Spotify Scio,Beam Sql,我正在构建一个简单的原型,其中我正在从Pubsub读取数据,并使用BeamSQL,代码片段如下 val eventStream: SCollection[String] = sc.pubsubSubscription[String]("projects/jayadeep-etl-platform/subscriptions/orders-dataflow") .withFixedWindows(Duration.standardSeconds(10)) val ev
val eventStream: SCollection[String] = sc.pubsubSubscription[String]("projects/jayadeep-etl-platform/subscriptions/orders-dataflow")
.withFixedWindows(Duration.standardSeconds(10))
val events: SCollection[DemoEvents] = eventStream.applyTransform(ParDo.of(new DoFnExample()))
events.map(row=>println("Input Stream:" + row))
val pickup_events = SideOutput[DemoEvents]()
val delivery_events = SideOutput[DemoEvents]()
val (mainOutput: SCollection[DemoEvents], sideOutputs: SideOutputCollections)= events
.withSideOutputs(pickup_events, delivery_events)
.flatMap {
case (evts, ctx) =>
evts.eventType match {
// Send to side outputs via `SideOutputContext`
case "pickup" => ctx.output(pickup_events,evts)
case "delivery" => ctx.output(delivery_events,evts)
}
Some(evts)
}
val pickup: SCollection[DemoEvents] = sideOutputs(pickup_events)
val dropoff = sideOutputs(delivery_events)
pickup.map(row=>println("Pickup:" + row))
dropoff.map(row=>println("Delivery:" + row))
val consolidated_view = tsql"select $pickup.order_id as orderId, $pickup.area as pickup_location, $dropoff.area as dropoff_location , $pickup.restaurant_id as resturantId from $pickup as pickup left outer join $dropoff as dropoff ON $pickup.order_id = $dropoff.order_id ".as[Output]
consolidated_view.map(row => println("Output:" + row))
sc.run().waitUntilFinish()
()
我正在使用Directrunner进行本地测试,在执行beam sql之前,我能够看到结果。beam sql的输出未被打印
Input Stream:DemoEvents(false,pickup,Bangalore,Indiranagar,1566382242,49457442008,1566382242489,7106576,1566382242000,178258,7406545542,,false,null,htr23e22-329a-4b05-99c1-606a3ccf6a48,972)
Pickup:DemoEvents(false,pickup,Bangalore,Indiranagar,1566382242,49457442008,1566382242489,7106576,1566382242000,178258,7406545542,,false,null,htr23e22-329a-4b05-99c1-606a3ccf6a48,972)
Input Stream:DemoEvents(false,delivery,Bangalore,Indiranagar,1566382242,49457442008,2566382242489,7106576,1566382242000,178258,7406545542,,false,null,htr23e22-329a-4b05-99c1-606a3ccf6a48,972)
Delivery:DemoEvents(false,delivery,Bangalore,Indiranagar,1566382242,49457442008,2566382242489,7106576,1566382242000,178258,7406545542,,false,null,htr23e22-329a-4b05-99c1-606a3ccf6a48,972)
这个问题与DirectRunner中的一个bug有关,当我将runner更改为DataflowRunner时,代码按示例运行