Python 强制spacy不解析标点符号?
有没有办法强迫spacy不要将标点符号解析为单独的标记Python 强制spacy不解析标点符号?,python,tokenize,spacy,punctuation,Python,Tokenize,Spacy,Punctuation,有没有办法强迫spacy不要将标点符号解析为单独的标记 nlp = spacy.load('en') doc = nlp(u'the $O is in $R') [ w for w in doc ] : [the, $, O, is, in, $, R] 我想: : [the, $O, is, in, $R] 是的,有。比如说, import spacy import regex as re from spacy.tokenizer import Tokenizer p
nlp = spacy.load('en')
doc = nlp(u'the $O is in $R')
[ w for w in doc ]
: [the, $, O, is, in, $, R]
我想:
: [the, $O, is, in, $R]
是的,有。比如说,
import spacy
import regex as re
from spacy.tokenizer import Tokenizer
prefix_re = re.compile(r'''^[\[\+\("']''')
suffix_re = re.compile(r'''[\]\)"']$''')
infix_re = re.compile(r'''[\(\-\)\@\.\:\$]''') #you need to change the infix tokenization rules
simple_url_re = re.compile(r'''^https?://''')
def custom_tokenizer(nlp):
return Tokenizer(nlp.vocab, prefix_search=prefix_re.search,
suffix_search=suffix_re.search,
infix_finditer=infix_re.finditer,
token_match=simple_url_re.match)
nlp = spacy.load('en_core_web_sm')
nlp.tokenizer = custom_tokenizer(nlp)
doc = nlp(u'the $O is in $R')
print [w for w in doc] #prints
[the, $O, is, in, $R]
您只需要将“$”字符添加到中缀正则表达式中(显然带有转义字符“\”
旁白:包括前缀和后缀,以展示spaCy标记器的灵活性。在您的情况下,只需中缀正则表达式就足够了。为spaCy的标记器类定制
前缀搜索
函数。参考比如:
import spacy
import re
from spacy.tokenizer import Tokenizer
# use any currency regex match as per your requirement
prefix_re = re.compile('''^\$[a-zA-Z0-9]''')
def custom_tokenizer(nlp):
return Tokenizer(nlp.vocab, prefix_search=prefix_re.search)
nlp = spacy.load("en_core_web_sm")
nlp.tokenizer = custom_tokenizer(nlp)
doc = nlp(u'the $O is in $R')
print([t.text for t in doc])
# ['the', '$O', 'is', 'in', '$R']
使用\$\w+
模式怎么样?