elasticsearch Elastic只保留logstash中的最后一条记录,elasticsearch,logstash,logstash-jdbc,elasticsearch,Logstash,Logstash Jdbc" /> elasticsearch Elastic只保留logstash中的最后一条记录,elasticsearch,logstash,logstash-jdbc,elasticsearch,Logstash,Logstash Jdbc" />

elasticsearch Elastic只保留logstash中的最后一条记录

elasticsearch Elastic只保留logstash中的最后一条记录,elasticsearch,logstash,logstash-jdbc,elasticsearch,Logstash,Logstash Jdbc,下面是select语句,它是一个事务数据,每次将数据插入UserProfile时,旧的索引值都会从中删除 jdbc { statement => "SELECT userId,salesTeam FROM UserProfile with (nolock)" } output { elasticsearch { hosts => ["localhost:9200"] index => "q_d" document_type =>

下面是select语句,它是一个事务数据,每次将数据插入UserProfile时,旧的索引值都会从中删除

  jdbc {
    statement => "SELECT userId,salesTeam FROM UserProfile with (nolock)"
}
output {
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "q_d"
    document_type => "cd"
    document_id => "%{userId}%"
  }
  stdout { codec => rubydebug }
}
如果有任何更改,我希望更新现有文档,否则将为新文档编制索引。
我做错了什么

input {
    jdbc {
        # Postgres jdbc connection string to our database, mydb
        jdbc_connection_string => "jdbc:postgresql://localhost:5432/bhavya"
        # The user we wish to execute our statement as
        jdbc_user => "postgres"
        # The path to our downloaded jdbc driver
        jdbc_driver_library => "/root/postgresql-42.2.2.jar"
        # The name of the driver class for Postgresql
        jdbc_driver_class => "org.postgresql.Driver"
        jdbc_password => "postgres"
        jdbc_validate_connection => true
        #You can schedule input from this plugin,the following uses cron syntax
        schedule => "* * * * *"
        # our query
        statement => "SELECT uid,email,first_name,last_name FROM contacts"
    }
}

output {
    elasticsearch {
    hosts => ["localhost:9200"]
    index => "contacts"
    document_type => "record"
    document_id => "%{uid}"
  }
  stdout { codec => rubydebug }
}
首先,您应该根据您的数据库添加我在input plugin中指定的上述选项。我使用Postgresql作为数据库。因此,您需要为该数据库下载相应的驱动程序库jar,并指定相应的路径

其次,您应该在“jdbc”插件中使用schedule选项,以便它定期从数据库中读取数据

第三,您应该从“输出”插件部分的“文档id”部分中删除额外的“%”

有关从数据库将数据导入日志存储的信息,请参见本页。:->