Python 保存微调的Tensorflow模型时列出索引超出范围
我正在尝试使用Tensorflow从Huggingface微调一个预先训练好的BERT模型。一切运行顺利,模型构建和训练无误。但当我试图保存模型时,它停止了,并出现错误“IndexError:list index out range”。我正在使用谷歌Colab和TPU 任何帮助都将不胜感激 代码: 错误:Python 保存微调的Tensorflow模型时列出索引超出范围,python,tensorflow,google-colaboratory,bert-language-model,huggingface-transformers,Python,Tensorflow,Google Colaboratory,Bert Language Model,Huggingface Transformers,我正在尝试使用Tensorflow从Huggingface微调一个预先训练好的BERT模型。一切运行顺利,模型构建和训练无误。但当我试图保存模型时,它停止了,并出现错误“IndexError:list index out range”。我正在使用谷歌Colab和TPU 任何帮助都将不胜感激 代码: 错误: --------------------------------------------------------------------------- IndexError
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-22-255116b49022> in <module>()
1 SAVE_PATH = 'path/to/save/location'
----> 2 model.save(SAVE_PATH,save_format='tf')
50 frames
/usr/local/lib/python3.7/dist-packages/transformers/modeling_tf_utils.py in input_processing(func, config, input_ids, **kwargs)
372 output[tensor_name] = input
373 else:
--> 374 output[parameter_names[i]] = input
375 elif isinstance(input, allowed_types) or input is None:
376 output[parameter_names[i]] = input
IndexError: list index out of range
---------------------------------------------------------------------------
索引器回溯(最后一次最近调用)
在()
1保存路径='PATH/to/SAVE/location'
---->2.model.save(保存路径,保存格式='tf')
50帧
/输入处理中的usr/local/lib/python3.7/dist-packages/transformers/modeling\u tf\u utils.py(func、config、input\u id、**kwargs)
372输出[张量_名称]=输入
373其他:
-->374输出[参数名称[i]]=输入
375 elif isinstance(输入,允许的类型)或输入为None:
376输出[参数名称[i]]=输入
索引器:列表索引超出范围
使用形状绘制的模型:
问题解决了
移除两个输入层之一(即注意遮罩)解决了问题
工作代码->
import tensorflow as tf
from tensorflow.keras import activations, optimizers, losses
from transformers import TFBertModel
def create_model(max_sequence, model_name, num_labels):
bert_model = TFBertModel.from_pretrained(model_name)
input_ids = tf.keras.layers.Input(shape=(max_sequence,), dtype=tf.int32, name='input_ids')
#attention_mask = tf.keras.layers.Input((max_sequence,), dtype=tf.int32, name='attention_mask')
#output = bert_model([input_ids, attention_mask])[0]
output = bert_model([input_ids])[0]
output = output[:, 0, :]
output = tf.keras.layers.Dense(num_labels, activation='sigmoid')(output)
#model = tf.keras.models.Model(inputs=[input_ids, attention_mask], outputs=output)
model = tf.keras.models.Model(inputs=[input_ids], outputs=output)
return model
with strategy.scope():
model = create_model(20, 'bert-base-uncased', 1)
opt = optimizers.Adam(learning_rate=3e-5)
loss = 'binary_crossentropy'
model.compile(optimizer=opt, loss=loss, metrics=['accuracy'])
model.fit(tfdataset_train, batch_size=32, epochs=2)
SAVE_PATH = 'path/to/save/location'
model.save(SAVE_PATH)
解决办法是改变:
output=bert\u model([input\u id,attention\u mask])[0]
到
output=bert\u model.bert([input\u id,attention\u mask])[0]
参考:
我对你发布的解决方案投了赞成票,但后来我在培训时发现模型存在问题。它的收敛性不好。谢谢!只是想强调一点:这个错误出现在我使用Tf==2.3和transformers==4.3时。这解决了问题,但是模型没有正确训练。
import tensorflow as tf
from tensorflow.keras import activations, optimizers, losses
from transformers import TFBertModel
def create_model(max_sequence, model_name, num_labels):
bert_model = TFBertModel.from_pretrained(model_name)
input_ids = tf.keras.layers.Input(shape=(max_sequence,), dtype=tf.int32, name='input_ids')
#attention_mask = tf.keras.layers.Input((max_sequence,), dtype=tf.int32, name='attention_mask')
#output = bert_model([input_ids, attention_mask])[0]
output = bert_model([input_ids])[0]
output = output[:, 0, :]
output = tf.keras.layers.Dense(num_labels, activation='sigmoid')(output)
#model = tf.keras.models.Model(inputs=[input_ids, attention_mask], outputs=output)
model = tf.keras.models.Model(inputs=[input_ids], outputs=output)
return model
with strategy.scope():
model = create_model(20, 'bert-base-uncased', 1)
opt = optimizers.Adam(learning_rate=3e-5)
loss = 'binary_crossentropy'
model.compile(optimizer=opt, loss=loss, metrics=['accuracy'])
model.fit(tfdataset_train, batch_size=32, epochs=2)
SAVE_PATH = 'path/to/save/location'
model.save(SAVE_PATH)