Fatal编程技术网
  • ubuntu/
  • Ubuntu AWS上的fluentd安装

Ubuntu AWS上的fluentd安装

ubuntu amazon-web-services

Ubuntu AWS上的fluentd安装,ubuntu,amazon-web-services,fluentd,Ubuntu,Amazon Web Services,Fluentd,实际上，我们计划在nodes服务器上使用fluentd获取日志，并将这些日志转发到我们的日志收集器（主fluentd服务器），然后我们可以将日志传输到s3存储桶。现在我们可以将日志写入我的s3存储桶，但在调试模式下重新启动td代理时出现错误 fluent节点配置：- 日志聚合器配置：- #输入 @打字前进 绑定0.0.0.0 端口24224 #S3配置 @s3型 aws_key_id Akerpe6JXXXKXSG3GWSA aws第二卷密钥31KNCOJDQRXXXRXEW0DY5RQV9ZE

实际上，我们计划在nodes服务器上使用fluentd获取日志，并将这些日志转发到我们的日志收集器（主fluentd服务器），然后我们可以将日志传输到s3存储桶。现在我们可以将日志写入我的s3存储桶，但在调试模式下重新启动td代理时出现错误

fluent节点配置：- 日志聚合器配置：- 
#输入
@打字前进
绑定0.0.0.0
端口24224
#S3配置
@s3型
aws_key_id Akerpe6JXXXKXSG3GWSA
aws第二卷密钥31KNCOJDQRXXXRXEW0DY5RQV9ZEGTEDFRERE25F1HSVW
沙湖蛙
美国西南部s3_地区2
路径日志/
缓冲区类型文件
缓冲区路径/var/log/td-agent/s3/buffer/td
缓冲区\u块\u限制2m
缓冲区队列限制4098
重试\u等待1s
最大重试次数\u等待5s
禁用\u重试\u限制为真
时间片格式%Y%m%d%H
时间片等待10米
utc
它粘在这东西上了 当我启动td-agent-v时，我遇到了这个错误 2016-06-07 07:11:00+0000[信息]：plugin/in_forward.rb:81:listen:0.0.0:24224上的监听流畅套接字 2016-06-07 07:11:00+0000[错误]：fluent/engine.rb:196:运行中的救援：意外错误\u class=Errno:：eaddrinue错误=# 2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148:开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:in
initialize'
2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148:开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:in
new' 2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148:开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:in
initialize'
2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148：开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_-forward.rb:82:in
new' 2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148：开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_-forward.rb:82:in
listen
2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148：开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_-forward.rb:51:in
start' 2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148：开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root\u-agent.rb:111:in
开始时阻塞'
2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148:开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root\u-agent.rb:110:in
each' 2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148:block-in-start:/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root\u-agent.rb:110:in
start
2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148:阻塞启动：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:225:in
start 2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148：开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:175:in
run
2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148:开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:597:in
运行引擎' 2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148：开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:148:in
开始时阻塞'
2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148:开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:352:in
call' 2016-06-07 07:11:00+0000[错误]：fluent/supervisor.rb:148：开始时阻塞：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:352:in'main_process'

如果您对此有任何解决方案，请告诉我。

我得到了答案，通过这些配置，我们可以将apache2日志转发到log collector，并将其存储到s3 bucket中

谢谢大家。

您没有收听
24224
？-注意你的问题中的格式问题不在配置中。该
24224
端口中可能已经有另一个td代理服务（或另一个）在运行。 
root@ip-172-31-32-115:/var/log/td-agent# cat /etc/td-agent/td-agent.conf
#Apache Access logs
<source>
  @type tail
  format apache2
  path /var/log/apache2/access.log
  pos_file /var/log/td-agent/apache2.access.pos
  tag apache.access
</source>
#Log forwardin
<match apache.access>
  @type forward
  send_timeout 60s
  recover_wait 10s
  heartbeat_interval 1s
  phi_threshold 16
  hard_timeout 60s
#primary host

  <server>
    host xx.xx.xx.xx
    port 24224
  </server>
  flush_interval 60s
</match>

root@ip-172-31-34-201:/home/ubuntu# td-agent -v 
2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:457:read_config: reading config file path="/etc/td-agent/td-agent.conf"
2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:323:supervise: starting fluentd-0.12.20
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-mongo' version '0.7.12'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-s3' version '0.6.5'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-scribe' version '0.10.14'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td' version '0.10.28'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluentd' version '0.12.20'
2016-06-06 18:21:44 +0000 [info]: fluent/agent.rb:125:add_match: adding match pattern="apache2.." type="forward"
2016-06-06 18:21:44 +0000 [info]: plugin/out_forward.rb:141:block in configure: adding forwarding server '172.31.33.196:24224' host="172.31.33.196" port=24224 weight=60 plugin_id="object:11e9668"
2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail"
2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail" plugin/out_forward.rb:232:rebuild_weight_array: rebuilding weight array lost_weight=0
2016-06-06 18:21:44 +0000 [info]: plugin/in_tail.rb:499:initialize: following tail of /var/log/apache2/access.log

#Input
<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>
# S3 config
<match s3.*.*>
    @type s3

    aws_key_id AKIERPE6JxxxKXSG3GWSA
    aws_sec_key 31KnCOJdQrxxxrXEw0dYY5RqV9zegTedfrere25F1Hsvw
    s3_bucket rana.sahu
    s3_region us-west-2
    path logs/
    buffer_type           file
    buffer_path           /var/log/td-agent/s3/buffer/td
    buffer_chunk_limit    2m
    buffer_queue_limit    4098
    retry_wait            1s
    max_retry_wait        5s
    disable_retry_limit   true


    time_slice_format %Y%m%d%H
    time_slice_wait 10m
    utc

</match>