elasticsearch 将MariaDB中的多个表同步到Elasticsearch 7.5
我试图使用JDBC插件将数据库同步到Elasticsearch,但不知何故,这种配置不适用于多个语句(多个表)
elasticsearch 将MariaDB中的多个表同步到Elasticsearch 7.5,
elasticsearch,mariadb,kibana,elk,
elasticsearch,Mariadb,Kibana,Elk,我试图使用JDBC插件将数据库同步到Elasticsearch,但不知何故,这种配置不适用于多个语句(多个表) 此配置对于单个语句运行良好,但对于多个查询不起作用。我错过了什么?我需要将此数据同步到Elasticsearch中的单个索引。通过创建唯一的文档id解决了此问题 input { jdbc { jdbc_driver_library => "" jdbc_driver_class => "Java::org.mariadb.jdbc.Driver"
此配置对于单个语句运行良好,但对于多个查询不起作用。我错过了什么?我需要将此数据同步到Elasticsearch中的单个索引。通过创建唯一的文档id解决了此问题
input {
jdbc {
jdbc_driver_library => ""
jdbc_driver_class => "Java::org.mariadb.jdbc.Driver"
jdbc_connection_string => "jdbc:mariadb://localhost:3306/DB"
jdbc_user => "username"
jdbc_password => "password"
jdbc_paging_enabled => true
tracking_column => "column1"
use_column_value => true
type => "type1"
tracking_column_type => "numeric"
schedule => "*/5 * * * * *"
statement => "SELECT *, UNIX_TIMESTAMP(date_column) AS unix_ts_in_secs FROM table1 WHERE (UNIX_TIMESTAMP(date_column) >= :sql_last_value) ORDER BY date_column ASC"
}
jdbc {
jdbc_driver_library => ""
jdbc_driver_class => "Java::org.mariadb.jdbc.Driver"
jdbc_connection_string => "jdbc:mariadb://localhost:3306/DB"
jdbc_user => "username"
jdbc_password => "password"
jdbc_paging_enabled => true
tracking_column => "column1"
use_column_value => true
type => "type2"
tracking_column_type => "numeric"
schedule => "*/5 * * * * *"
statement => "SELECT *, UNIX_TIMESTAMP(date_column) AS unix_ts_in_secs FROM table2 WHERE (UNIX_TIMESTAMP(date_column) >= :sql_last_value) ORDER BY date_column ASC"
}
}
filter {
mutate {
copy => { "id" => "[@metadata][_id]"}
remove_field => ["id", "@version", "unix_ts_in_secs"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "test"
document_id => "%{[@metadata][_id]}"
}
}