如何在 fluentd csv 格式化程序中禁用包含分隔符的字段值周围的引号？

Question

我使用 fluentd 从 golang 应用程序收集 CSV 格式的日志。

这是流利的 conf 文件

<source>

    @type  forward

    @id    app_logs

    @label @mainstream

    port  24224

</source>

<label @mainstream>

   <match **>

      @type file

      @id   v6_logs

    <format>

      @type csv

      fields log_version,day_time,event_id,request_id,app_id,app_version,account_id,country_id,server_name,remote_ip,process_id,thread_id,item_id,message,parameters

      force_quotes false

    </format>

      path         /fluentd/log/app.log

      append       true

  </match>

</label>

我使用 Fluent golang 客户端从应用程序https://github.com/fluent/fluent-logger-golang写入日志

logger, _ := fluent.New(fluent.Config{FluentPort: 24224, FluentHost: "fluentd"})

defer logger.Close()

tag := "web"

var data = map[string]interface{}{

    "log_version": 6,

    "day_time":    time.Now().UTC().String(),

    "event_id":    1700,

    "request_id":  "54321",

    "account_id":  12345,

    "server_name": hostname,

    "process_id":  os.Getpid(),

    "message":     "Test Message(param1; param2)",

    "parameters":  "value1, value2",

}

error := logger.Post(tag, data)

输出是这样的。

6,2020-09-23 23:48:44.5731073 +0000 UTC,1700,54321,,,123467,,cabf36399a5c,,1,,,Test Message(param1; param2),"value1,value2"

如何在“value1,value2”周围删除引号（使其作为单独的字段出现）。

Answer 1

我按照https://docs.fluentd.org/v/0.12/developer/plugin-development的说明在 ruby 中编写了一个自定义 CSV 格式化程序插件，并将文件放在路径 /etc/fluent/plugin/ 中，从而实现了这一点&nbsp; &nbsp;require 'fluent/plugin/formatter'&nbsp; &nbsp; &nbsp; &nbsp;&nbsp;&nbsp; &nbsp;module Fluent::Plugin&nbsp; &nbsp;class MyCSVFormatter < Formatter&nbsp; &nbsp; &nbsp; &nbsp;# Register MyCSVFormatter as 'my_csv'.&nbsp; &nbsp; &nbsp; &nbsp;Fluent::Plugin.register_formatter('my_csv', self)&nbsp; &nbsp; &nbsp; &nbsp;config_param :csv_fields, :array, value_type: :string&nbsp; &nbsp; &nbsp; &nbsp;# This method does further processing. Configuration parameters can be&nbsp; &nbsp; &nbsp; &nbsp;# accessed either via `conf` hash or member variables.&nbsp; &nbsp; &nbsp; &nbsp;def configure(conf)&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;super&nbsp; &nbsp; &nbsp; &nbsp;end&nbsp; &nbsp; &nbsp; &nbsp;# This is the method that formats the data output.&nbsp; &nbsp; &nbsp; def format(tag, time, record)&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;values = []&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Look up each required field and collect them from the record&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;@csv_fields.each do |field|&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if field == "parameters"&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; parametervalues = []&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; parametervalues = record[field].split(",").map(&:strip)&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; values.push(*parametervalues)&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; next&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; end&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; v = record[field]&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; values << v.to_s&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; end&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;# Output by joining the fields with a comma&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;values.join(',') + "\n"&nbsp; &nbsp; &nbsp; end&nbsp; &nbsp; end&nbsp; &nbsp;end更新了 fluentd conf 文件以使用像这样的自定义格式<label @mainstream>&nbsp; &nbsp;<match **>&nbsp; &nbsp; &nbsp; @type file&nbsp; &nbsp; &nbsp; @id&nbsp; &nbsp;v6_logs&nbsp; &nbsp; <format>&nbsp; &nbsp; &nbsp; @type my_csv&nbsp; &nbsp; &nbsp; csv_fields log_version,day_time,event_id,request_id,app_id,app_version,account_id,country_id,server_name,remote_ip,process_id,thread_id,item_id,message,parameters&nbsp; &nbsp; </format>&nbsp; &nbsp; &nbsp; path&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;/fluentd/log/app.log&nbsp; &nbsp; &nbsp; append&nbsp; &nbsp; &nbsp; &nbsp;true&nbsp; </match></label>这会产生所需的输出6,2020-09-24 06:27:52.1826684 +0000 UTC,1700,54321,,,123467,,hostname,,1,,,Test Message(param1; param2),value1,value2