Warning: file_get_contents(/data/phpspider/zhask/data//catemap/2/python/334.json): failed to open stream: No such file or directory in /data/phpspider/zhask/libs/function.php on line 167

Warning: Invalid argument supplied for foreach() in /data/phpspider/zhask/libs/tag.function.php on line 1116

Notice: Undefined index: in /data/phpspider/zhask/libs/function.php on line 180

Warning: array_chunk() expects parameter 1 to be array, null given in /data/phpspider/zhask/libs/function.php on line 181
Python 遭遇';UnicodeDecodeError';带变压器包的负荷模型_Python_Torch_Transformer - Fatal编程技术网

Python 遭遇';UnicodeDecodeError';带变压器包的负荷模型

Python 遭遇';UnicodeDecodeError';带变压器包的负荷模型,python,torch,transformer,Python,Torch,Transformer,我从谷歌下载了经过预训练的文件,并使用官方纸条将其转换为名为“pytorch_model.bin”的torch verion: 然后我将“bert_config.json”重命名为“config.json”,并将“pytorch_model.bin”、“config.json”和“vocab.txt”放入同一文件夹“torch_bert_model”,以便使用“from_pertained()”方法加载此torch模型: 运行此代码时,出现“UnicodeDecodeError”: Traceb

我从谷歌下载了经过预训练的文件,并使用官方纸条将其转换为名为“pytorch_model.bin”的torch verion:

然后我将“bert_config.json”重命名为“config.json”,并将“pytorch_model.bin”、“config.json”和“vocab.txt”放入同一文件夹“torch_bert_model”,以便使用“from_pertained()”方法加载此torch模型:

运行此代码时,出现“UnicodeDecodeError”:

Traceback (most recent call last):
File "c:/Users/hap/Desktop/Projects/cgt/BERT.py", line 47, in <module>
  model = TFBertModel.from_pretrained(model_file)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site- packages\transformers\modeling_tf_utils.py", line 529, in from_pretrained
  config, model_kwargs = cls.config_class.from_pretrained(
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py", line 311, in from_pretrained
  config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py", line 354, in get_config_dict
  config_dict = cls._dict_from_json_file(resolved_config_file)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py", line 436, in _dict_from_json_file
  text = reader.read()
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\codecs.py", line 322, in decode
  (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte
回溯(最近一次呼叫最后一次):
文件“c:/Users/hap/Desktop/Projects/cgt/BERT.py”,第47行,在
model=TFBertModel.from_pretrained(model_文件)
文件“C:\Users\hap\AppData\Local\Programs\Python\38\lib\site-packages\transformers\modeling\u tf\u utils.py”,第529行,from\u pretrained
config,model_kwargs=cls.config_class.from_pretrained(
文件“C:\Users\hap\AppData\Local\Programs\Python\38\lib\site packages\transformers\configuration\u utils.py”,第311行,from\u pretrained
config\u dict,kwargs=cls.get\u config\u dict(预先训练的模型名称或路径,**kwargs)
文件“C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site packages\transformers\configuration\u utils.py”,第354行,在get\u config\u dict中
config\u dict=cls.\u dict\u来自\u json\u文件(已解析的\u config\u文件)
文件“C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site packages\transformers\configuration\u utils.py”,第436行,位于json文件的目录中
text=reader.read()
文件“C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\codecs.py”,第322行,在decode中
(结果,消耗)=自身缓冲区解码(数据,自身错误,最终)
UnicodeDecodeError:“utf-8”编解码器无法解码位置64中的字节0x80:无效的开始字节
我哪里出了问题?非常感谢您的建议。谢谢

current_dir = os.path.dirname(__file__)
model_folder = os.path.join(current_dir, 'torch_bert_model')
# configer_file = os.path.join(model_folder, 'config.json')
vocab_file = os.path.join(model_folder, 'vocab.txt')
model_file = os.path.join(model_folder, 'pytorch_model.bin')

tokenizer = BertTokenizer.from_pretrained(vocab_file)
# bert_config = BertConfig.from_json_file(configer_file)
model = TFBertModel.from_pretrained(model_file) # Automatically loads the config
Traceback (most recent call last):
File "c:/Users/hap/Desktop/Projects/cgt/BERT.py", line 47, in <module>
  model = TFBertModel.from_pretrained(model_file)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site- packages\transformers\modeling_tf_utils.py", line 529, in from_pretrained
  config, model_kwargs = cls.config_class.from_pretrained(
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py", line 311, in from_pretrained
  config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py", line 354, in get_config_dict
  config_dict = cls._dict_from_json_file(resolved_config_file)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py", line 436, in _dict_from_json_file
  text = reader.read()
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\codecs.py", line 322, in decode
  (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte