elasticsearch 弹性搜索中的LM
我如何提高这种情况下的召回率?有什么建议吗? 我想创建一个包含3900万段的索引,每段至少包含四个英语句子。我的问题是简短的疑问句。我知道,使用Dirichlet平滑、停止字删除和词干分析器的语言模型最适合这种情况。如何使用这些条件编制索引(我已经使用此配置编制了索引,但与默认bm25的结果没有差异) 我的索引:
elasticsearch 弹性搜索中的LM,
elasticsearch,search,indexing,stemming,language-model,
elasticsearch,Search,Indexing,Stemming,Language Model,我如何提高这种情况下的召回率?有什么建议吗? 我想创建一个包含3900万段的索引,每段至少包含四个英语句子。我的问题是简短的疑问句。我知道,使用Dirichlet平滑、停止字删除和词干分析器的语言模型最适合这种情况。如何使用这些条件编制索引(我已经使用此配置编制了索引,但与默认bm25的结果没有差异) 我的索引: { "settings": { "index":{ "similarity" : {
{
"settings": {
"index":{
"similarity" : {
"my_similarity" : {
"type" : "LMDirichlet",
"mu" : 2000
}
},
"analysis":{
"filter":{
"english_stop":{
"type":"stop",
"stopwords":"_english_"
},
"my_stemmer":{
"type":"stemmer",
"name":"english"
}
},
"analyzer":{
"my_custom_analyzer":{
"type":"custom",
"tokenizer":"standard",
"filter":[
"lowercase",
"english_stop",
"my_stemmer"
]
}
}
}
},
"number_of_shards": 1
},
"mappings": {
"properties": {
"content": {
"similarity" : "my_similarity" ,
"analyzer": "my_custom_analyzer",
"type": "text"
}
}
}
}
query = " (" + prevTurn + ")^1 (" + currentTurn + ")^2"
search_param={
"query": {
"query_string": {
"query":query,
"analyzer": "my_stop_analyzer",
"default_field":"doc.content"
}
}
}
Title: The Neolithic Revolution
Description: The neolithic revolution and technology used within it and when it emerged in the british isles. Also, the transition to the bronze age and its significance.
1 What was the neolithic revolution?
2 When did it start and end?
3 Why did it start?
4 What did the neolithic invent?
5 What tools were used?
6 When was it brought to the british isles?
搜索我的python代码的方法是:
{
"settings": {
"index":{
"similarity" : {
"my_similarity" : {
"type" : "LMDirichlet",
"mu" : 2000
}
},
"analysis":{
"filter":{
"english_stop":{
"type":"stop",
"stopwords":"_english_"
},
"my_stemmer":{
"type":"stemmer",
"name":"english"
}
},
"analyzer":{
"my_custom_analyzer":{
"type":"custom",
"tokenizer":"standard",
"filter":[
"lowercase",
"english_stop",
"my_stemmer"
]
}
}
}
},
"number_of_shards": 1
},
"mappings": {
"properties": {
"content": {
"similarity" : "my_similarity" ,
"analyzer": "my_custom_analyzer",
"type": "text"
}
}
}
}
query = " (" + prevTurn + ")^1 (" + currentTurn + ")^2"
search_param={
"query": {
"query_string": {
"query":query,
"analyzer": "my_stop_analyzer",
"default_field":"doc.content"
}
}
}
Title: The Neolithic Revolution
Description: The neolithic revolution and technology used within it and when it emerged in the british isles. Also, the transition to the bronze age and its significance.
1 What was the neolithic revolution?
2 When did it start and end?
3 Why did it start?
4 What did the neolithic invent?
5 What tools were used?
6 When was it brought to the british isles?
一个示例回合:
{
"settings": {
"index":{
"similarity" : {
"my_similarity" : {
"type" : "LMDirichlet",
"mu" : 2000
}
},
"analysis":{
"filter":{
"english_stop":{
"type":"stop",
"stopwords":"_english_"
},
"my_stemmer":{
"type":"stemmer",
"name":"english"
}
},
"analyzer":{
"my_custom_analyzer":{
"type":"custom",
"tokenizer":"standard",
"filter":[
"lowercase",
"english_stop",
"my_stemmer"
]
}
}
}
},
"number_of_shards": 1
},
"mappings": {
"properties": {
"content": {
"similarity" : "my_similarity" ,
"analyzer": "my_custom_analyzer",
"type": "text"
}
}
}
}
query = " (" + prevTurn + ")^1 (" + currentTurn + ")^2"
search_param={
"query": {
"query_string": {
"query":query,
"analyzer": "my_stop_analyzer",
"default_field":"doc.content"
}
}
}
Title: The Neolithic Revolution
Description: The neolithic revolution and technology used within it and when it emerged in the british isles. Also, the transition to the bronze age and its significance.
1 What was the neolithic revolution?
2 When did it start and end?
3 Why did it start?
4 What did the neolithic invent?
5 What tools were used?
6 When was it brought to the british isles?
您可以在查询中尝试相似性