elasticsearch Logstash输入jdbc正在复制结果
我使用logstash input jdbc插件读取两个(或更多)数据库并将数据发送到elasticsearch,并使用kibana 4将这些数据可视化 这是我的日志存储配置:elasticsearch Logstash输入jdbc正在复制结果,elasticsearch,logstash,kibana-4,elasticsearch,Logstash,Kibana 4,我使用logstash input jdbc插件读取两个(或更多)数据库并将数据发送到elasticsearch,并使用kibana 4将这些数据可视化 这是我的日志存储配置: input { jdbc { type => "A" jdbc_driver_library => "C:\DEV\elasticsearch-1.7.1\plugins\elasticsearch-jdbc-1.7.1.0\lib\jtds-1.3.1.jar" jdbc_dri
input {
jdbc {
type => "A"
jdbc_driver_library => "C:\DEV\elasticsearch-1.7.1\plugins\elasticsearch-jdbc-1.7.1.0\lib\jtds-1.3.1.jar"
jdbc_driver_class => "Java::net.sourceforge.jtds.jdbc.Driver"
jdbc_connection_string => "jdbc:jtds:sqlserver://dev_data_base_server:1433/dbApp1;domain=CORPDOMAIN;useNTLMv2=true"
jdbc_user => "user"
jdbc_password => "pass"
schedule => "5 * * * *"
statement => "SELECT id, date, content, status from test_table"
}
jdbc {
type => "B"
jdbc_driver_library => "C:\DEV\elasticsearch-1.7.1\plugins\elasticsearch-jdbc-1.7.1.0\lib\jtds-1.3.1.jar"
jdbc_driver_class => "Java::net.sourceforge.jtds.jdbc.Driver"
jdbc_connection_string => "jdbc:jtds:sqlserver://dev_data_base_server:1433/dbApp2;domain=CORPDOMAIN;useNTLMv2=true"
jdbc_user => "user"
jdbc_password => "pass"
schedule => "5 * * * *"
statement => "SELECT id, date, content, status from test_table"
}
}
filter {
}
output {
if [type] == "A" {
elasticsearch {
host => "localhost"
protocol => http
index => "logstash-servera-%{+YYYY.MM.dd}"
}
}
if [type] == "B" {
elasticsearch {
host => "localhost"
protocol => http
index => "logstash-serverb-%{+YYYY.MM.dd}"
}
}
stdout { codec => rubydebug }
}
问题是每次运行logstash时,它都会开始保存弹性搜索中已经存在的所有数据
在使用where子句=date>运行“2015-09-10”之后,我停止了日志存储,并使用“特殊参数”(sql\u last\u date)再次运行(使用--debug)。logstash启动后,它开始在日志中显示:
←[36mExecuting JDBC query {:statement=>"SELECT \n\tSUBSTRING(R.RECEBEDOR, 1, 2)
AS 'DDD',\nCASE WHEN R.STATUS <> 'RCON' AND R.COD_RESPOSTA in (428,429,230,425,
430,427,418,422,415,424,214,433,435,207,426) THEN 'REGRA DE NEGÓCIO' \n W
HEN R.STATUS = 'RCON' THEN 'SUCESSO'\n\t ELSE 'ERRO'\n END AS 'TIPO_MENSAGEM
',\nAP.ALIAS as 'CANAL', R.ID_RECARGA, R.VALOR, R.STATUS, R.COD_RESPOSTA, R.DESC
_RESPOSTA, R.DT_RECARGA as '@timestamp', R.ID_CLIENTE, R.ID_DEPENDENTE, R.ID_APL
ICACAO, RECEBEDOR, R.ID_OPERADORA, R.TIPO_PRODUTO \n\nFROM RECARGA R (NOLOCK)\nJ
OIN APLICACAO AP ON R.ID_APLICACAO = AP.ID_APLICACAO \nwhere R.DT_RECARGA > :sql
_last_start\nORDER BY R.DT_RECARGA ASC", :parameters=>{:sql_last_start=>2015-09-
10 18:48:00 UTC}, :level=>:debug, :file=>"/DEV/logstash-1.5.4/vendor/bundle/jrub
y/1.9/gems/logstash-input-jdbc-1.0.0/lib/logstash/plugin_mixins/jdbc.rb", :line=
>"107", :method=>"execute_statement"}←[0m
←[36MExecutingJDBC查询{:statement=>“选择\n\t字符串(R.RECEBEDOR,1,2)
作为“DDD”,当R.STATUS“RCON”和R.COD_RESPOSTA处于(428429230425,
430427418422415424214433435207426)然后是“国家改革”\n W
当R.STATUS='RCON'然后是'successo'\n\t其他'ERRO'\n结束为'TIPO\u mensage'
,\nAP.ALIAS为“CANAL”,R.ID\u RECARGA,R.VALOR,R.STATUS,R.COD\u RESPOSTA,R.DESC
_RESPOSTA,R.DT_RECARGA为“@timestamp”,R.ID_客户,R.ID_家属,R.ID_APL
Icao,接收者,R.ID\u OPERADORA,R.TIPO\u PRODUTO\n\n来自RECARGA R(NOLOCK)\nJ
OIN APLICAO AP ON R.ID\u APLICAO=AP.ID\u APLICAO\n R.DT\u RECARGA>中的位置:sql
_最后一次启动\n由R.DT_RECARGA ASC编写,:参数=>{:sql_最后一次启动=>2015-09-
10 18:48:00 UTC},:level=>:debug,:file=>“/DEV/logstash-1.5.4/vendor/bundle/jrub
y/1.9/gems/logstash-input-jdbc-1.0.0/lib/logstash/plugin_mixins/jdbc.rb“,:行=
>“107”,:方法=>“执行_语句”}←[0m
这一次,我用了“真实”的说法,即:
SELECT
SUBSTRING(R.RECEBEDOR, 1, 2) AS 'DDD',
CASE WHEN R.STATUS <> 'RCON' AND R.COD_RESPOSTA in (428,429,230,425,430,427,418,422,415,424,214,433,435,207,426) THEN 'REGRA DE NEGÓCIO'
WHEN R.STATUS = 'RCON' THEN 'SUCESSO'
ELSE 'ERRO'
END AS 'TIPO_MENSAGEM',
AP.ALIAS as 'CANAL', R.ID_RECARGA, R.VALOR, R.STATUS, R.COD_RESPOSTA, R.DESC_RESPOSTA, R.DT_RECARGA as '@timestamp', R.ID_CLIENTE, R.ID_DEPENDENTE, R.ID_APLICACAO, RECEBEDOR, R.ID_OPERADORA
FROM RECARGA R (NOLOCK)
JOIN APLICACAO AP ON R.ID_APLICACAO = AP.ID_APLICACAO
where R.DT_RECARGA > :sql_last_start
ORDER BY R.DT_RECARGA ASC
选择
子字符串(R.RECEBEDOR,1,2)作为“DDD”,
当R.STATUS“RCON”和R.COD_RESPOSTA处于(428429230425430427418422415424214433435207426)状态时,则为“REGRA DE NEGÓCIO”
当R.STATUS='RCON'时,则为'SUCESSO'
否则“错误”
以“TIPO_MENSAGEM”结尾,
AP别名为“运河”,R.ID\u RECARGA,R.VALOR,R.STATUS,R.COD\u RESPOSTA,R.DESC\u RESPOSTA,R.DT\u RECARGA为“@timestamp”,R.ID\u客户,R.ID\u家属,R.ID\u Aplicaco,接收者,R.ID\u OPERADORA
来自RECARGA R(诺洛克)
在R.ID\u aplicaco=AP.ID\u aplicaco上加入aplicaco AP
其中R.DT_RECARGA>:sql_last_start
R.DT_RECARGA ASC订购
有人知道怎么解决吗
谢谢!默认情况下,
jdbc
输入将执行配置的SQL语句。在您的情况下,您的语句将选择test\u表中的所有内容。您需要指示SQL语句仅加载上次运行jdbc
输入时的数据,方法是在SQL查询中使用预定义参数
input {
jdbc {
type => "A"
jdbc_driver_library => "C:\DEV\elasticsearch-1.7.1\plugins\elasticsearch-jdbc-1.7.1.0\lib\jtds-1.3.1.jar"
jdbc_driver_class => "Java::net.sourceforge.jtds.jdbc.Driver"
jdbc_connection_string => "jdbc:jtds:sqlserver://dev_data_base_server:1433/dbApp1;domain=CORPDOMAIN;useNTLMv2=true"
jdbc_user => "user"
jdbc_password => "pass"
schedule => "5 * * * *"
statement => "SELECT id, date, content, status from test_table WHERE date > :sql_last_start"
}
jdbc {
type => "B"
jdbc_driver_library => "C:\DEV\elasticsearch-1.7.1\plugins\elasticsearch-jdbc-1.7.1.0\lib\jtds-1.3.1.jar"
jdbc_driver_class => "Java::net.sourceforge.jtds.jdbc.Driver"
jdbc_connection_string => "jdbc:jtds:sqlserver://dev_data_base_server:1433/dbApp2;domain=CORPDOMAIN;useNTLMv2=true"
jdbc_user => "user"
jdbc_password => "pass"
schedule => "5 * * * *"
statement => "SELECT id, date, content, status from test_table WHERE date > :sql_last_start"
}
}
此外,如果碰巧从数据库中加载了两次相同的记录,并且您不希望在ES服务器中创建DUP,您还可以指定在elasticsearch
输出中使用记录ID作为文档ID,这样文档将在ES中更新,而不会重复
output {
if [type] == "A" {
elasticsearch {
host => "localhost"
protocol => http
index => "logstash-servera-%{+YYYY.MM.dd}"
document_id => "%{id}" <--- same id as in DB
}
}
if [type] == "B" {
elasticsearch {
host => "localhost"
protocol => http
index => "logstash-serverb-%{+YYYY.MM.dd}"
document_id => "%{id}" <--- same id as in DB
}
}
stdout { codec => rubydebug }
}
输出{
如果[类型]=“A”{
弹性搜索{
主机=>“本地主机”
协议=>http
索引=>“日志存储服务器A-%{+YYYY.MM.dd}”
文档\u id=>“%{id}”“本地主机”
协议=>http
索引=>“日志存储服务器B-%{+YYYY.MM.dd}”
文档\u id=>“%{id}”rubydebug}
}
sql\u last\u start
现在是sql\u last\u值
特殊参数sql\u last\u start
现在被重命名为sql\u last\u value
,以便更清晰地显示,因为它不仅限于datetime,还可能具有其他列类型。
所以现在的解决方案可能是这样的
input {
jdbc {
type => "A"
jdbc_driver_library => "C:\DEV\elasticsearch-1.7.1\plugins\elasticsearch- jdbc-1.7.1.0\lib\jtds-1.3.1.jar"
jdbc_driver_class => "Java::net.sourceforge.jtds.jdbc.Driver"
jdbc_connection_string => "jdbc:jtds:sqlserver://dev_data_base_server:1433/dbApp1;domain=CORPDOMAIN;useNTLMv2=true"
jdbc_user => "user"
jdbc_password => "pass"
schedule => "5 * * * *"
use_column_value => true
tracking_column => date
statement => "SELECT id, date, content, status from test_table WHERE date >:sql_last_value"
#clean_run true means it will reset sql_last_value to zero or initial value if datatype is date(default is also false)
clean_run =>false
}
jdbc{
#for type B....
}
}
我已经用SQLServerDB进行了测试
请使用clean_run=>true首次运行,以避免在开发过程中出现数据类型错误。我们可能在sql_last_value
变量中存储了不同的数据类型值。我尝试过使用此变量。我第一次使用WHERE date>'2015-09-09'。日志从昨天到今天上午11:30获取了所有数据。我在第二次运行时 (11:50),Logstash没有任何数据。是的,数据库中有更多数据。日志中显示的唯一内容是:Logstash启动完成Logstash关闭完成您可以使用--debug
运行Logstash并提供更多输出吗?我将在这个问题上添加此输出。在您的输出中,我看到sql上次启动
是2015-09-10 18:48:00 UTC
。不确定这是否是问题,但这并不是真正的上午11:50。请提供必要的信息在网站上回答这个问题。之前我只是一个评论,因为我没有足够的排名来评论上述答案。现在我写了完整的答案,这样可能会帮助其他人:)