Python 数据流-未调用函数-错误-未定义名称
我在Google Dataflow上与Apache Beam合作,通过lambda函数调用函数,得到一个错误,函数名未定义Python 数据流-未调用函数-错误-未定义名称,python,google-cloud-dataflow,apache-beam,dataflow,Python,Google Cloud Dataflow,Apache Beam,Dataflow,我在Google Dataflow上与Apache Beam合作,通过lambda函数调用函数,得到一个错误,函数名未定义 output_tweets = (lines | 'decode' >> beam.Map(lambda x: x.decode('utf-8')) | 'assign window key' >> beam.WindowInto(window.FixedWindow
output_tweets = (lines
| 'decode' >> beam.Map(lambda x: x.decode('utf-8'))
| 'assign window key' >> beam.WindowInto(window.FixedWindows(10))
| 'batch into n batches' >> BatchElements(min_batch_size=49, max_batch_size=50)
| 'sentiment analysis' >> beam.FlatMap(lambda x: sentiment(x))
)
这是我的Apache Beam调用,在最后一行中提到了函数感悟,这给了我一个问题
函数代码如下(我认为这无关紧要):
我得到了下面的回溯
File "stream.py", line 97, in <lambda>
NameError: name 'sentiment' is not defined [while running 'generatedPtransform-441']
文件“stream.py”,第97行,在
NameError:未定义名称“情绪”[在运行“GeneratedPtTransform-441”时]
有什么想法吗?这个问题的发生有两个原因
定义是否与beam管道存在于同一个python文件中
def testing(messages):
return messages.lower()
windowed_lower_word_counts = (windowed_words
| beam.Map(lambda word: testing(word))
| "count" >> beam.combiners.Count.PerElement())
ib.show(windowed_lower_word_counts, include_window_info=True)
0 b'have' 3 2020-04-19 06:04:39.999999+0000 2020-04-19 06:04:30.000000+0000 (10s) Pane 0
1 b'ransom' 1 2020-04-19 06:04:39.999999+0000 2020-04-19 06:04:30.000000+0000 (10s) Pane 0
2 b'let' 1 2020-04-19 06:04:39.999999+0000 2020-04-19 06:04:30.000000+0000 (10s) Pane 0
3 b'me' 1 2020-04-19 06:04:39.999999+0000 2020-04-19 06:04:30.000000+0000 (10s) Pane 0
如果函数是在调用后定义的,那么我们会得到如下所示的错误
windowed_lower_word_counts = (windowed_words
| beam.Map(lambda word: testing_after(word))
| "count" >> beam.combiners.Count.PerElement())
ib.show(windowed_lower_word_counts, include_window_info=True)
ERROR:apache_beam.runners.direct.executor:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7f478f344820>, due to an exception.
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 954, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 552, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/root/apache-beam-custom/packages/beam/sdks/python/apache_beam/transforms/core.py", line 1482, in <lambda>
wrapper = lambda x: [fn(x)]
File "<ipython-input-19-f34e29a17836>", line 2, in <lambda>
| beam.Map(lambda word: testing_after_new(word))
NameError: name 'testing_after' is not defined
def testing_after(messages):
return messages.lower()
windowed\u lower\u words\u counts=(windowed\u words
|beam.Map(lambda词:在(词)之后测试)
|“count”>>beam.combiners.count.PerElement())
ib.show(带窗口的较低单词计数,包括窗口信息=真)
错误:apache_beam.runners.direct.executor:bundle异常函数在同一脚本中,其定义在管道运行之前。我观察到的一件事是,如果我在没有lambda的情况下调用该函数,并且只使用beam.Map,它就到达了该函数。你知道为什么吗?如果你看一下它,就会发现用户定义的函数应该是3种类型之一types.BuiltinFunctionType,types.MethodType,types.FunctionType
。因此,传递函数名是正确的方法,我们的示例中也提到了这一点。在您提到的示例中,您无法在员工中找到情绪,您可能必须在课堂中传递这些情绪。如果这有帮助,请接受答案。嘿@jayadeep jayaraman,这很有效。但是,如果调用仅仅是beam.Map(x)
,beam.Map也可以工作。结果是,它给了我一个discovery.build
错误,即使库已导入。我不确定这是不是该问的地方。
windowed_lower_word_counts = (windowed_words
| beam.Map(lambda word: testing_after(word))
| "count" >> beam.combiners.Count.PerElement())
ib.show(windowed_lower_word_counts, include_window_info=True)
ERROR:apache_beam.runners.direct.executor:Exception at bundle <apache_beam.runners.direct.bundle_factory._Bundle object at 0x7f478f344820>, due to an exception.
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 954, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 552, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "/root/apache-beam-custom/packages/beam/sdks/python/apache_beam/transforms/core.py", line 1482, in <lambda>
wrapper = lambda x: [fn(x)]
File "<ipython-input-19-f34e29a17836>", line 2, in <lambda>
| beam.Map(lambda word: testing_after_new(word))
NameError: name 'testing_after' is not defined
def testing_after(messages):
return messages.lower()