How an emit count is being calculated

[preeti-16]

.Sample log files
I tried to parse small log file (having 100logs) - 1 emit count got increased
I tried to parse big log file (having 200logs)- suddenly emit got increased with large number

we were trying to find a metrics/ratio/or formula to identify failure of fluentd destinations such as (opensearch,kafka). While browsing we found flush error ratio can be used which is an error count to an emit count. But we found emit count drastically increased whereas an error increased by just 1. and hence ratio was way low to raise an alarm.

Below are the metrics that we used to calculate flush error ratio:

1. fluentd_output_status_emit_count -> To calculate emit count
2. fluentd_output_status_num_errors -> To calculate no of errors

Below are the configuration for output destination

For kafka2 plugin
<match>
@type kafka2
brokers kf-train-headless:9092
default_topic your_topic
output_data_type json
@log_level debug
<buffer>
@type file
chunk_limit_size 8MB
path /var/log/fluent/is-fluentd-buffer/kafka.logging.all
total_limit_size 1024m
flush_interval 3s
retry_max_times 0
</buffer>
<format>
@type json
</format>
</match>

For opensearch
<match>
@type opensearch
@log_level info
include_tag_key true
host indexsearch
port 9200
logstash_format true
logstash_prefix log-${tag[2]}
reload_connections false
reconnect_on_error true
reload_on_failure true
request_timeout 10s
<buffer>
@type file
chunk_limit_size 8MB
path /var/log/fluent/is-fluentd-buffer/os.logging.all
total_limit_size 1024m
retry_max_times 0
</buffer>
</match>

I tried to limit out the no of retries to 0 so to check does an error count reports to an every emit happened since output destination (kafka, opensearch) is unreachable.

So Can you tell how emit count is calculated.

[preeti-16]

Hi, Can you please check the above query asap and tell how emit count is calculated.