Python 解析文本时SpaCy未指定正确的依赖项标签
我从中获取了一些代码,允许您将自定义依赖项标签分配给文本,我想用它来解释用户的意图。它主要工作正常,但例如,当我运行代码时,它会将“delete”标记为“ROOT”,并将其标记为“INTENT”,如Python 解析文本时SpaCy未指定正确的依赖项标签,python,machine-learning,nlp,spacy,Python,Machine Learning,Nlp,Spacy,我从中获取了一些代码,允许您将自定义依赖项标签分配给文本,我想用它来解释用户的意图。它主要工作正常,但例如,当我运行代码时,它会将“delete”标记为“ROOT”,并将其标记为“INTENT”,如deps字典中所示 from __future__ import unicode_literals, print_function import plac import random import spacy from pathlib import Path # training data: t
deps
字典中所示
from __future__ import unicode_literals, print_function
import plac
import random
import spacy
from pathlib import Path
# training data: texts, heads and dependency labels
# for no relation, we simply chose an arbitrary dependency label, e.g. '-'
TRAIN_DATA = [
("How do I delete my account?", {
'heads': [3, 3, 3, 3, 5, 3, 3], # index of token head
'deps': ['ROOT', '-', '-', 'INTENT', '-', 'OBJECT', '-']
}),
("How do I add a balance?", {
'heads': [3, 3, 3, 3, 5, 3, 3],
'deps': ['ROOT', '-', '-', 'INTENT', '-', 'OBJECT', '-']
}),
("How do I deposit my funds into my bank account?", {
'heads': [3, 3, 3, 3, 5, 3, 3, 9, 9, 6, 3],
'deps': ['ROOT', '-', '-', 'INTENT', '-', '-', '-', '-', '-', 'OBJECT', '-']
}),
("How do I fill out feedback forms?", {
'heads': [3, 3, 3, 3, 3, 6, 3, 3],
'deps': ['ROOT', '-', '-', 'INTENT', '-', '-', 'OBJECT', '-']
}),
#("How does my profile impact my score?", {
#'heads': [4, 4, 4, 4, 4, 6, 4, 4],
#'deps': ['ROOT', '-', '-', '-', 'INTENT', '-', 'OBJECT' '-']
#}),
("What are the fees?", {
'heads': [1, 1, 3, 1, 1],
'deps': ['ROOT', '-', '-', 'INTENT', '-']
}),
("How do I update my profile picture?", {
'heads': [3, 3, 3, 3, 6, 6, 3, 3],
'deps': ['ROOT', '-', '-', 'INTENT', '-', 'OBJECT', 'OBJECT', '-']
}),
("How do I add a referral to the marketplace?", {
'heads': [3, 3, 3, 3, 5, 3, 3, 8, 6, 3],
'deps': ['ROOT', '-', '-', 'INTENT', '-', 'OBJECT', '-', '-', 'OBJECT', '-']
}),
]
@plac.annotations(
model=("Model name. Defaults to blank 'en' model.", "option", "m", str),
output_dir=("Optional output directory", "option", "o", Path),
n_iter=("Number of training iterations", "option", "n", int))
def main(model=None, output_dir=None, n_iter=5):
"""Load the model, set up the pipeline and train the parser."""
if model is not None:
nlp = spacy.load(model) # load existing spaCy model
print("Loaded model '%s'" % model)
else:
nlp = spacy.blank('en') # create blank Language class
print("Created blank 'en' model")
# We'll use the built-in dependency parser class, but we want to create a
# fresh instance – just in case.
if 'parser' in nlp.pipe_names:
nlp.remove_pipe('parser')
parser = nlp.create_pipe('parser')
nlp.add_pipe(parser, first=True)
#add new labels to the parser
for text, annotations in TRAIN_DATA:
for dep in annotations.get('deps', []):
parser.add_label(dep)
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'parser']
with nlp.disable_pipes(*other_pipes): # only train parser
optimizer = nlp.begin_training()
for itn in range(n_iter):
random.shuffle(TRAIN_DATA)
losses = {}
for text, annotations in TRAIN_DATA:
nlp.update([text], [annotations], sgd=optimizer, losses=losses)
print(losses)
# test the trained model
test_model(nlp)
# save model to output directory
if output_dir is not None:
output_dir = Path(output_dir)
if not output_dir.exists():
output_dir.mkdir()
nlp.to_disk(output_dir)
print("Saved model to", output_dir)
# test the saved model
print("Loading from", output_dir)
nlp2 = spacy.load(output_dir)
test_model(nlp2)
def test_model(nlp):
texts = ["How do I delete my account?"]
docs = nlp.pipe(texts)
for doc in docs:
print(doc.text)
print([(t.text, t.dep_, t.head.text) for t in doc if t.dep_ != '-'])
if __name__ == '__main__':
plac.call(main)
这是输出:
如何删除我的帐户?
[(u'How',u'ROOT',u'delete'),(u'delete',u'ROOT',u'delete'),(u'account',u'OBJECT',u'delete')]
我认为问题的根源在于依赖关系树的根被自动标记为'ROOT'
,
(并且依赖关系树的根被定义为其头部为自身的标记)
一种可能的解决方法是在训练数据中添加人工根:
("root How do I delete my account?", {
'heads': [0, 4, 4, 4, 0, 6, 4, 4], # index of token head
'deps': ['ROOT', '-', '-', '-', 'INTENT', '-', 'OBJECT', '-']
})
(还要在测试示例中添加符号root
:text=[“root如何删除我的帐户?”]
)
通过这些更改,如果您对模型进行足够长的训练,您将获得:
root How do I delete my account?
[('root', 'ROOT', 'root'), ('delete', 'INTENT', 'root'), ('account', 'OBJECT', 'delete')]
我认为问题的根源在于依赖关系树的根被自动标记为
“根”
,
(并且依赖关系树的根被定义为其头部为自身的标记)
一种可能的解决方法是在训练数据中添加人工根:
("root How do I delete my account?", {
'heads': [0, 4, 4, 4, 0, 6, 4, 4], # index of token head
'deps': ['ROOT', '-', '-', '-', 'INTENT', '-', 'OBJECT', '-']
})
(还要在测试示例中添加符号root
:text=[“root如何删除我的帐户?”]
)
通过这些更改,如果您对模型进行足够长的训练,您将获得:
root How do I delete my account?
[('root', 'ROOT', 'root'), ('delete', 'INTENT', 'root'), ('account', 'OBJECT', 'delete')]
嘿,谢谢你的回复,这似乎确实奏效了。我没有将“root”添加到文本的开头,而是尝试将第一个索引的值更改为“0”,以查看是否也会这样做。但出于某种原因,它不是,为什么我不能让它在本例中识别“How”作为词根?我猜这不是spacy使用的约定:0表示当前标记的头是句子的第一个。(与conllu格式不同,在conllu格式中,0表示根,标记从1索引)。编辑:我误解了,你可以用0来表示根的“How”,但是,你应该将delete的开头改为“How”:
[0,3,3,0,5,3,3]
,否则,你的树将有两个根。是的,我刚刚意识到,我只是运行了一段代码,返回字符串中每个标记的头部。对于“如何删除我的帐户?”“如何”不是它自己的头(不是根),“删除”是它的头。但是对于“root如何删除我的帐户?”“root”是它自己的头,因此它是root。再次感谢你是冠军,干杯谢谢你的回答,这似乎确实奏效了。我没有将“root”添加到文本的开头,而是尝试将第一个索引的值更改为“0”,以查看是否也会这样做。但出于某种原因,它不是,为什么我不能让它在本例中识别“How”作为词根?我猜这不是spacy使用的约定:0表示当前标记的头是句子的第一个。(与conllu格式不同,在conllu格式中,0表示根,标记从1索引)。编辑:我误解了,你可以用0来表示根的“How”,但是,你应该将delete的开头改为“How”:[0,3,3,0,5,3,3]
,否则,你的树将有两个根。是的,我刚刚意识到,我只是运行了一段代码,返回字符串中每个标记的头部。对于“如何删除我的帐户?”“如何”不是它自己的头(不是根),“删除”是它的头。但是对于“root如何删除我的帐户?”“root”是它自己的头,因此它是root。再次感谢你是冠军,干杯