elasticsearch集群遭遇error,code 400

[2022-10-31T00:00:05,302][ERROR][logstash.outputs.elasticsearch][main][7de5ed67efbd17713042ae9dceb8f344262c61d98b282c2d962a517b19576719] Encountered a retrya
ble error. Will Retry with exponential backoff  {:code=>400, :url=>"https://30.12.53.149:9200/_bulk"}
[2022-10-31T00:01:12,452][ERROR][logstash.outputs.elasticsearch][main][7de5ed67efbd17713042ae9dceb8f344262c61d98b282c2d962a517b19576719] Encountered a retrya
ble error. Will Retry with exponential backoff  {:code=>400, :url=>"https://30.12.52.215:9200/_bulk"}
[2022-10-31T00:02:19,597][ERROR][logstash.outputs.elasticsearch][main][7de5ed67efbd17713042ae9dceb8f344262c61d98b282c2d962a517b19576719] Encountered a retrya
ble error. Will Retry with exponential backoff  {:code=>400, :url=>"https://30.12.53.107:9200/_bulk"}
[2022-10-31T00:03:26,683][ERROR][logstash.outputs.elasticsearch][main][7de5ed67efbd17713042ae9dceb8f344262c61d98b282c2d962a517b19576719] Encountered a retrya
ble error. Will Retry with exponential backoff  {:code=>400, :url=>"https://30.12.52.215:9200/_bulk"}
[2022-10-31T00:04:33,785][ERROR][logstash.outputs.elasticsearch][main][7de5ed67efbd17713042ae9dceb8f344262c61d98b282c2d962a517b19576719] Encountered a retrya
ble error. Will Retry with exponential backoff  {:code=>400, :url=>"https://30.12.52.215:9200/_bulk"}
[2022-10-31T00:05:40,921][ERROR][logstash.outputs.elasticsearch][main][7de5ed67efbd17713042ae9dceb8f344262c61d98b282c2d962a517b19576719] Encountered a retrya
ble error. Will Retry with exponential backoff  {:code=>400, :url=>"https://30.12.53.107:9200/_bulk"}
[2022-10-31T00:06:48,036][ERROR][logstash.outputs.elasticsearch][main][7de5ed67efbd17713042ae9dceb8f344262c61d98b282c2d962a517b19576719] Encountered a retrya
ble error. Will Retry with exponential backoff  {:code=>400, :url=>"https://30.12.52.215:9200/_bulk"}

[ERROR] 2018-07-02 16:23:20.482 [[main]>worker5] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>“http://ip:9200/_bulk 30”}

logstash 6.0.0, elasticsearch 6.0.0

I increased elasticsearch configuration heap size in jvm.options from 1 gb to 26 gb. It has improved numbers. There are still errors.

now:

– it has stopped loading logs
– logs loaded 44419
– error count 2058, continue to increase as new logs are processed
– index doc.deleted 7701

before:

– it has stopped loading logs
– logs loaded 8412
– error count 12600, continue to increase as new logs are processed
– index doc.deleted 6109

Run logstash job.

input {
file {
path => [
“/path/nifi-app.log”,
“/path/nifi-user.log”,
“/path/nifi-bootstrap.log”
]
type => “nifiapp”
sincedb_path => “/path/nifiapp”
}
}
filter {
if [type] == “nifiapp” {
grok {
match => {
“message” => “%{GREEDYDATA:log_date} %{GREEDYDATA:log_time} %{EMAILLOCALPART:log_level} [%{GREEDYDATA:log_type}] %{GREEDYDATA:log_text}”
}
}
}
}
output {
if [type] == “nifiapp” {
elasticsearch {
hosts => “ip:9200”
index => “nifiapp”
document_id => “%{log_date}%{log_time}%{log_level}%{log_type}%{log_text}”
}
}
}

Run nifi job to copy log files from nifi server to elk server.

image.png1109×150 31.7 KB

At this point, I get this error.

[ERROR] 2018-07-02 16:23:20.482 [[main]>worker5] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>“http://ip:9200/_bulk 30”}

curl -XGET ‘ip:9200/_cat/indices?v&pretty’

index nifiapp, docs.count 44419, doc.deleted 7701

[ERROR] 2018-07-02 16:23:20.482 [[main]>worker5] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>“http://ip:9200/_bulk 30”}

logstash 6.0.0, elasticsearch 6.0.0

I increased elasticsearch configuration heap size in jvm.options from 1 gb to 26 gb. It has improved numbers. There are still errors.

now:

– it has stopped loading logs
– logs loaded 44419
– error count 2058, continue to increase as new logs are processed
– index doc.deleted 7701

before:

– it has stopped loading logs
– logs loaded 8412
– error count 12600, continue to increase as new logs are processed
– index doc.deleted 6109

Run logstash job.

input {
file {
path => [
“/path/nifi-app.log”,
“/path/nifi-user.log”,
“/path/nifi-bootstrap.log”
]
type => “nifiapp”
sincedb_path => “/path/nifiapp”
}
}
filter {
if [type] == “nifiapp” {
grok {
match => {
“message” => “%{GREEDYDATA:log_date} %{GREEDYDATA:log_time} %{EMAILLOCALPART:log_level} [%{GREEDYDATA:log_type}] %{GREEDYDATA:log_text}”
}
}
}
}
output {
if [type] == “nifiapp” {
elasticsearch {
hosts => “ip:9200”
index => “nifiapp”
document_id => “%{log_date}%{log_time}%{log_level}%{log_type}%{log_text}”
}
}
}

Run nifi job to copy log files from nifi server to elk server.

image.png1109×150 31.7 KB

At this point, I get this error.

[ERROR] 2018-07-02 16:23:20.482 [[main]>worker5] elasticsearch - Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>“http://ip:9200/_bulk 30”}

curl -XGET ‘ip:9200/_cat/indices?v&pretty’

index nifiapp, docs.count 44419, doc.deleted 7701