Python 米特纳模型
我一直在探索使用经过训练的MITIE模型进行命名实体提取。我是否可以查看他们的实际ner模型,而不是使用预训练模型?该模型是开源的吗 设置设置: 对于初学者,您可以下载 包含一个名为 总单词特征提取程序.dat 之后,从他们的 官方Git 如果您正在运行Windows O.S,请下载 如果您正在运行基于x64的Windows O.S,请安装Visual Studio 2015社区版的C++编译器。< /P> 下载完以上内容后,将所有内容解压缩到一个文件夹中 从“开始”>“所有应用程序”>“Visual Studio”打开VS 2015的开发者命令提示符,并导航到“工具”文件夹,您将在其中看到5个子文件夹 下一步是通过在VisualStudioDeveloper命令提示符中使用以下Cmake命令来构建ner_conll、ner_stream、train_freebase_relationship_detector和wordrep包 大概是这样的:Python 米特纳模型,python,model,named-entity-recognition,rasa-nlu,Python,Model,Named Entity Recognition,Rasa Nlu,我一直在探索使用经过训练的MITIE模型进行命名实体提取。我是否可以查看他们的实际ner模型,而不是使用预训练模型?该模型是开源的吗 设置设置: 对于初学者,您可以下载 包含一个名为 总单词特征提取程序.dat 之后,从他们的 官方Git 如果您正在运行Windows O.S,请下载 如果您正在运行基于x64的Windows O.S,请安装Visual Studio 2015社区版的C++编译器。< /P> 下载完以上内容后,将所有内容解压缩到一个文件夹中 从“开始”>“所有应用程序”>“Vi
{
"AnnotatedTextList": [
{
"text": "I want to travel from New Delhi to Bangalore tomorrow.",
"entities": [
{
"type": "FromCity",
"startPos": 5,
"length": 2
},
{
"type": "ToCity",
"startPos": 8,
"length": 1
},
{
"type": "TimeOfTravel",
"startPos": 9,
"length": 1
}
]
}
]
}
对于ner_conll:
cd "C:\Users\xyz\Documents\MITIE-master\tools\ner_conll"
i) mkdir构建
ii)cd构建
iii)cmake-G“Visual Studio 14 2015 Win64”。
iv)cmake——构建--配置发布--目标安装
对于ner_流:
cd "C:\Users\xyz\Documents\MITIE-master\tools\ner_stream"
i) mkdir构建
ii)cd构建
iii)cmake-G“Visual Studio 14 2015 Win64”。
iv)cmake——构建--配置发布--目标安装
对于序列自由基关系检测器:
cd "C:\Users\xyz\Documents\MITIE-master\tools\train_freebase_relation_detector"
i) mkdir构建
ii)cd构建
iii)cmake-G“Visual Studio 14 2015 Win64”。
iv)cmake——构建--配置发布--目标安装
对于wordrep:
cd "C:\Users\xyz\Documents\MITIE-master\tools\wordrep"
i) mkdir构建
ii)cd构建
iii)cmake-G“Visual Studio 14 2015 Win64”。
iv)cmake——构建--配置发布--目标安装
构建它们后,您将收到大约150-160条警告,不用担心
现在,导航到“C:\Users\xyz\Documents\MITIE master\examples\cpp\train\u ner”
使用Visual Studio代码创建JSON文件“data.JSON”以手动注释文本,如下所示:
{
"AnnotatedTextList": [
{
"text": "I want to travel from New Delhi to Bangalore tomorrow.",
"entities": [
{
"type": "FromCity",
"startPos": 5,
"length": 2
},
{
"type": "ToCity",
"startPos": 8,
"length": 1
},
{
"type": "TimeOfTravel",
"startPos": 9,
"length": 1
}
]
}
]
}
您可以添加更多的语句并对其进行注释,训练数据越多,预测精度越好
这种带注释的JSON也可以通过jQuery或Angular等前端工具创建。但为了简洁起见,我手工制作了它们
现在,解析带注释的JSON文件并将其传递给ner_training_实例的add_entity方法
<>但是C++不支持反序列化JSON,这就是为什么你可以使用这个库。从他们的Git页面下载包,并将其放在“C:\Users\xyz\Documents\MITIE master\mitielib\include\MITIE”下
现在我们必须自定义train_ner_example.cpp文件,以便解析带注释的自定义实体JSON并将其传递给MITIE进行训练
#include "mitie\rapidjson\document.h"
#include "mitie\ner_trainer.h"
#include <iostream>
#include <vector>
#include <list>
#include <tuple>
#include <string>
#include <map>
#include <sstream>
#include <fstream>
using namespace mitie;
using namespace dlib;
using namespace std;
using namespace rapidjson;
string ReadJSONFile(string FilePath)
{
ifstream file(FilePath);
string test;
cout << "path: " << FilePath;
try
{
std::stringstream buffer;
buffer << file.rdbuf();
test = buffer.str();
cout << test;
return test;
}
catch (exception &e)
{
throw std::exception(e.what());
}
}
//Helper function to tokenize a string based on multiple delimiters such as ,.;:- or whitspace
std::vector<string> SplitStringIntoMultipleParameters(string input, string delimiter)
{
std::stringstream stringStream(input);
std::string line;
std::vector<string> TokenizedStringVector;
while (std::getline(stringStream, line))
{
size_t prev = 0, pos;
while ((pos = line.find_first_of(delimiter, prev)) != string::npos)
{
if (pos > prev)
TokenizedStringVector.push_back(line.substr(prev, pos - prev));
prev = pos + 1;
}
if (prev < line.length())
TokenizedStringVector.push_back(line.substr(prev, string::npos));
}
return TokenizedStringVector;
}
//Parse the JSON and store into appropriate C++ containers to process it.
std::map<string, list<tuple<string, int, int>>> FindUtteranceTuple(string stringifiedJSONFromFile)
{
Document document;
cout << "stringifiedjson : " << stringifiedJSONFromFile;
document.Parse(stringifiedJSONFromFile.c_str());
const Value& a = document["AnnotatedTextList"];
assert(a.IsArray());
std::map<string, list<tuple<string, int, int>>> annotatedUtterancesMap;
for (int outerIndex = 0; outerIndex < a.Size(); outerIndex++)
{
assert(a[outerIndex].IsObject());
assert(a[outerIndex]["entities"].IsArray());
const Value &entitiesArray = a[outerIndex]["entities"];
list<tuple<string, int, int>> entitiesTuple;
for (int innerIndex = 0; innerIndex < entitiesArray.Size(); innerIndex++)
{
entitiesTuple.push_back(make_tuple(entitiesArray[innerIndex]["type"].GetString(), entitiesArray[innerIndex]["startPos"].GetInt(), entitiesArray[innerIndex]["length"].GetInt()));
}
annotatedUtterancesMap.insert(pair<string, list<tuple<string, int, int>>>(a[outerIndex]["text"].GetString(), entitiesTuple));
}
return annotatedUtterancesMap;
}
int main(int argc, char **argv)
{
try {
if (argc != 3)
{
cout << "You must give the path to the MITIE English total_word_feature_extractor.dat file." << endl;
cout << "So run this program with a command like: " << endl;
cout << "./train_ner_example ../../../MITIE-models/english/total_word_feature_extractor.dat" << endl;
return 1;
}
else
{
string filePath = argv[2];
string stringifiedJSONFromFile = ReadJSONFile(filePath);
map<string, list<tuple<string, int, int>>> annotatedUtterancesMap = FindUtteranceTuple(stringifiedJSONFromFile);
std::vector<string> tokenizedUtterances;
ner_trainer trainer(argv[1]);
for each (auto item in annotatedUtterancesMap)
{
tokenizedUtterances = SplitStringIntoMultipleParameters(item.first, " ");
mitie::ner_training_instance *currentInstance = new mitie::ner_training_instance(tokenizedUtterances);
for each (auto entity in item.second)
{
currentInstance -> add_entity(get<1>(entity), get<2>(entity), get<0>(entity).c_str());
}
// trainingInstancesList.push_back(currentInstance);
trainer.add(*currentInstance);
delete currentInstance;
}
trainer.set_num_threads(4);
named_entity_extractor ner = trainer.train();
serialize("new_ner_model.dat") << "mitie::named_entity_extractor" << ner;
const std::vector<std::string> tagstr = ner.get_tag_name_strings();
cout << "The tagger supports " << tagstr.size() << " tags:" << endl;
for (unsigned int i = 0; i < tagstr.size(); ++i)
cout << "\t" << tagstr[i] << endl;
return 0;
}
}
catch (exception &e)
{
cerr << "Failed because: " << e.what();
}
}