- elasticsearch/
elasticsearch 在logstash中将csv解析为kafka时出错
elasticsearch 在logstash中将csv解析为kafka时出错
我正在尝试使用LogStash将csv数据发送到kafka，并实现我自己的名为test.conf的配置脚本。 我在解析时遇到这个错误
Using JAVA_HOME defined java: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-05-24 19:12:08.565 [main] runner - Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 25.262-b10 on 1.8.0_262-b10 +indy +jit [linux-x86_64]"}
[FATAL] 2021-05-24 19:12:08.616 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:530:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:290:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:201:in `block in validate_all'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:200:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:317:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:273:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in `<main>'"]}
[ERROR] 2021-05-24 19:12:08.623 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
input {
file {
path => "/home/data/*.csv"
start_position =>"beginning"
sincedb_path => "/dev/null"
}
}
filter {
mutate {
add_field => {
"timestamp" => "%{Date} %{Time}"
}
}
date { match => ["timestamp", "dd-MM-YYYY HH:mm:ss"]}
csv {
remove_field => ["Date", "Time"]
}
grok {
match => { "message" => [
"^%{DATE:timestamp},%{NUMBER:ab},%{NUMBER:cd},%{NUMBER:ef},%{NUMBER:gh},%{NUMBER:ij},%{NUMBER:kl},%{NUMBER:mn},%{NUMBER:op},%{NUMBER:qr},%{NUMBER:st},%{NUMBER:uv},%{NUMBER:wx},%{NUMBER:yz}$"
]
}
}
}
output {
stdout { codec => rubydebug }
if "_grokparsefailure" not in [tags] {
kafka {
codec => "json"
topic_id => "abcd1234"
bootstrap_servers => "192.16.12.119:9092"
}
}
}
请帮我解决这个问题。首先，请确保您的服务器ip具有给定端口（192.16.12.119）和“telnet 192.16.12.119 9092”的可用性 在卡夫卡输出部分中忘记一个字段之后，在输出卡夫卡部分中添加group_id字段，例如 卡夫卡{group_id=>“35834” 主题=>[“您的主题名称”] 引导服务器=>“192.16.12.199:9092”
codec=>json}首先，使用“telnet 192.16.12.119 9092”确保您的服务器ip具有给定端口（192.16.12.119）的可用性 在卡夫卡输出部分中忘记一个字段之后，在输出卡夫卡部分中添加group_id字段，例如 卡夫卡{group_id=>“35834” 主题=>[“您的主题名称”] 引导服务器=>“192.16.12.199:9092”
codec=>json}您的配置甚至没有被加载，启动logstash时出现了一个
致命的错误
/usr/share/logstash/bin/logstash -f test.conf
[FATAL] 2021-05-24 19:12:08.616 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>,
您首先需要向运行logstash的用户授予写入path.data
的权限，您可以在logstash.yml
文件中更改path.data
，然后可以在命令行中传递该文件的路径
考虑到您使用包管理器（如yum
或apt
）安装了logstash，您的logstash.yml
文件将位于目录/etc/logstash/
中
因此，您需要以这种方式运行日志存储：
/usr/share/logstash/bin/logstash-f/path/to/your/config.conf--path.settings/etc/logstash/
在logstash.yml
中，需要将path.data
设置为用户有写入权限的目录
path.data: /path/to/writable/directory
您的配置甚至没有被加载，您在启动日志存储时出现了一个致命的错误
/usr/share/logstash/bin/logstash -f test.conf
[FATAL] 2021-05-24 19:12:08.616 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>,
您首先需要向运行logstash的用户授予写入path.data
的权限，您可以在logstash.yml
文件中更改path.data
，然后可以在命令行中传递该文件的路径
考虑到您使用包管理器（如yum
或apt
）安装了logstash，您的logstash.yml
文件将位于目录/etc/logstash/
中
因此，您需要以这种方式运行日志存储：
/usr/share/logstash/bin/logstash-f/path/to/your/config.conf--path.settings/etc/logstash/
在logstash.yml
中，需要将path.data
设置为用户有写入权限的目录
path.data: /path/to/writable/directory
显然，您的LogsAsh配置文件有一些问题，您还需要添加LogsAsh配置文件。@MikaelAmidi我添加了配置文件。这不是您的配置，请查看日志，您的日志中有一个致命的错误，路径/usr/share/Logstash/data”必须是一个可写目录。它也找不到您的logstash.yml
文件，在这种情况下，您需要使用--path.settings
。您正在运行logstash的是哪个用户？此用户需要能够写入path.data
目录。显然，您的LogsAsh配置文件有一些问题，您还需要添加LogsAsh配置文件。@MikaelAmidi我添加了配置文件。它不是您的配置，请查看日志，您的日志中有致命的错误，路径“/usr/share/logstash/data“必须是可写目录
。它也找不到您的logstash.yml
文件，在这种情况下，您需要使用--path.settings
。您正在运行logstash的是哪个用户？此用户需要能够写入路径.data
目录。别担心，我已更改了ip:）。别担心，我已更改了ip:）。