如何删除动态mongodb oid';什么在贮藏室?
Mongodb有两种类型的如何删除动态mongodb oid';什么在贮藏室?,mongodb,
elasticsearch,apache-kafka,logstash,Mongodb,
elasticsearch,Apache Kafka,Logstash,Mongodb有两种类型的$oid参考- 类型1- //MongoDB city_id : "5fe3206428bf745876649fd3" //Kafka Message city_id : { "$oid": "5fe3206428bf745876649fd3" } 类型2- //MongoDB city_ids : ["5fe3206428bf745876649fd3","5fe320
$oid
参考-
类型1-
//MongoDB
city_id : "5fe3206428bf745876649fd3"
//Kafka Message
city_id : {
"$oid": "5fe3206428bf745876649fd3"
}
类型2-
//MongoDB
city_ids : ["5fe3206428bf745876649fd3","5fe3206428bf745876649fd3","5fe3206428bf745876649fd3"]
//Kafka Message
city_ids : [
{
"$oid": "5fe3206428bf745876649fd3"
},
{
"$oid": "5fe3206428bf745876649fd3"
},
{
"$oid": "5fe3206428bf745876649fd3"
}
]
如何在logstash中处理这两种类型,以便在MongoDB中保存elasticsearch的确切数据结构
input {
kafka {
bootstrap_servers => "localhost:9092"
decorate_events => true
topics => ["users","organisations","cities"]
}
}
filter {
json {
source => "message"
target => "json_payload"
}
json {
source => "[json_payload][payload]"
target => "payload"
}
mutate {
rename => { "[payload]" => "document"}
remove_field => ["message","json_payload","payload"]
add_field => {
"[es_index]" => "%{[@metadata][kafka][topic]}"
"[mongo_id]" => "%{[document][_id][$oid]}"
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "%{es_index}"
document_id => "%{mongo_id}"
}
stdout {
codec =>
rubydebug {
metadata => true
}
}
}
这是对上一个字段的后续操作。下面将为包含$oid条目的每个字段动态执行此操作。它对结构做了很多假设——如果它包含$oid条目,那么这就是保留的全部内容
ruby {
code => '
event.to_hash.each { |k, v|
if v.is_a? Hash
if v["$oid"]
event.set(k, v["$oid"])
end
end
if v.is_a? Array
if v[0]["$oid"]
a = []
v.each { |x| a << x["$oid"] }
event.set(k, a)
end
end
}
'
}
ruby{
代码=>'
event.to_hash.each{k,v|
如果v.u是散列
如果v[“$oid”]
事件集(k,v[“$oid”])
结束
结束
如果v.u是一个数组
如果v[0][“$oid”]
a=[]
v、 每个{| x | a可以是动态的吗?或者我必须为所有集合中的每个键编写特定的逻辑吗?这样每次我修改集合并拥有类似的键时,我也需要在logstash中单独添加逻辑。你是指[city_ID]字段名可以更改?是的,这就是我的意思,city_id就是一个例子。但是,$oid
将是每个此类键的常见子项。ruby筛选器会动态更新以执行此操作。