Python 在TextLineDataset#映射中使用标记器

我尝试同时使用TextLineDataset和标记器，但在与#map一起使用时遇到了问题。我得到一个错误，不可能在张量上迭代。我想我理解，文本序列试图在图形表示上运行。对吗？甚至有可能把这两部分结合起来吗？把它们结合起来有意义吗

from keras.preprocessing.text import Tokenizer
import tensorflow as tf

# Example used from
# https://stackoverflow.com/a/51203923/998092

docs = ["A heart that",
         "full up like",
         "a landfill",
        "no surprises",
        "and no alarms"
         "a job that slowly"
         "Bruises that",
         "You look so",
         "tired happy",
         "no alarms",
        "and no surprises"]

T = Tokenizer()
T.fit_on_texts(docs)

def encode(sentence):
  return T.texts_to_sequences(sentence)

data = tf.data.TextLineDataset.from_tensor_slices(docs)
encoded_data = data.map(encode)

print("result for test 1:\n%s" %(data))

导致：

WARNING:tensorflow:Entity <bound method Tokenizer.texts_to_sequences_generator of <keras_preprocessing.text.Tokenizer object at 0x7f9089f5de48>> appears to be a generator function. It will not be converted by AutoGraph.
WARNING: Entity <bound method Tokenizer.texts_to_sequences_generator of <keras_preprocessing.text.Tokenizer object at 0x7f9089f5de48>> appears to be a generator function. It will not be converted by AutoGraph.

---------------------------------------------------------------------------

OperatorNotAllowedInGraphError            Traceback (most recent call last)

<ipython-input-8-46695a877229> in <module>()
     25 
     26 data = tf.data.TextLineDataset.from_tensor_slices(docs)
---> 27 encoded_data = data.map(encode)
     28 
     29 print("result for test 1:\n%s" %(data))

10 frames

/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    263       except Exception as e:  # pylint:disable=broad-except
    264         if hasattr(e, 'ag_error_metadata'):
--> 265           raise e.ag_error_metadata.to_exception(e)
    266         else:
    267           raise

OperatorNotAllowedInGraphError: in user code:

    <ipython-input-8-46695a877229>:24 encode  *
        return T.texts_to_sequences(sentence)
    /usr/local/lib/python3.6/dist-packages/keras_preprocessing/text.py:279 texts_to_sequences  *
        return list(self.texts_to_sequences_generator(texts))
    /usr/local/lib/python3.6/dist-packages/keras_preprocessing/text.py:298 texts_to_sequences_generator  **
        for text in texts:
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:561 __iter__
        self._disallow_iteration()
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:557 _disallow_iteration
        self._disallow_in_graph_mode("iterating over `tf.Tensor`")
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:537 _disallow_in_graph_mode
        " this function with @tf.function.".format(task))

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

警告：tensorflow:实体似乎是生成器函数。它将不会被转换为亲笔签名。
警告：实体似乎是生成器函数。它将不会被转换为亲笔签名。
---------------------------------------------------------------------------
OperatorNotAllowedInGraphError回溯（最近一次呼叫最后一次）
在（）
25
26 data=tf.data.TextLineDataset.from_tensor_切片（文档）
--->27编码的_数据=data.map（编码）
28
29打印（“测试1的结果：\n%s”%（数据））
10帧
/包装器中的usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py（*args，**kwargs）
263例外情况为e:#pylint:disable=broad except
264如果hasattr（即“ag\u错误\u元数据”）：
-->265将e.ag\u错误\u元数据引发到\u异常（e）
266其他：
267提高
OperatorNotAllowedInGraphError:在用户代码中：
：24编码*
将T.text\u返回到\u序列（句子）
/usr/local/lib/python3.6/dist packages/keras_preprocessing/text.py:279 text_to_序列*
返回列表（self.text\u到\u序列\u生成器（text））
/usr/local/lib/python3.6/dist packages/keras_preprocessing/text.py:298 text_to_sequences_生成器**
对于文本中的文本：
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:561__
self.\u不允许\u迭代（）
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:557不允许迭代
self.\u不允许在图模式下（“迭代`tf.Tensor`”）
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:537在图形模式下不允许
“此函数带有@tf.function.”.format（任务））
OperatorNotAllowedInGraphError:在图形执行中不允许迭代'tf.Tensor'。使用渴望执行或用@tf.function修饰此函数。