【故障处理】Elasticsearch数据写入不了之ES数据迁移

用户反馈ES写入不了数据，第一反应：磁盘爆满了？进入查看，果然要95%，在系统盘，哭笑不得~

1. 问题分析根源

磁盘剩余2.5G，使用率超过95%；
查看数据和日志文件存放在系统盘，占用大小9.6G（不大），鉴于测试环境，考虑将数据及日志迁移到数据盘存储，缓解系统盘压力

查看日志报错日志如下：
划重点：high disk watermark [90%] ， free: 2.5gb[5.1%]

[2024-06-05T14:32:06,328][WARN ][o.e.c.r.a.DiskThresholdMonitor] [es-node-1] high disk watermark [90%] exceeded on [hE1vR9sATHGeeoep
mSqF1w][es-node-1][/usr/local/elasticsearch-7.10.0/data/nodes/0] free: 2.5gb[5.1%], shards will be relocated away from this node; cu
rrently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these re
locations are complete
[2024-06-05T14:32:23,834][ERROR][o.e.x.i.IndexLifecycleRunner] [es-node-1] policy [kibana-event-log-policy] for index [.kibana-event
-log-7.10.0-000036] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
org.elasticsearch.cluster.block.ClusterBlockException: index [.kibana-event-log-7.10.0-000033] blocked by: [TOO_MANY_REQUESTS/12/dis
k usage exceeded flood-stage watermark, index has read-only-allow-delete block];index [.kibana-event-log-7.10.0-000034] blocked by:
[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];index [.kibana-event-log-7.
10.0-000035] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];in
dex [.kibana-event-log-7.10.0-000036] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-on
ly-allow-delete block];
at org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:223) ~[elasticsearch-7.10.0.jar:
7.10.0]
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:93) ~[ela
sticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.checkBlock(TransportRolloverAction.java:60) ~[ela
sticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.jav
a:138) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:115) [elastics
earch-7.10.0.jar:7.10.0]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:59) [elasticse
arch-7.10.0.jar:7.10.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) [elasticsearch-7.10
.0.jar:7.10.0]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10
.0.jar:7.10.0]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124) [x-pack-security
-7.10.0.jar:7.10.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10
.0.jar:7.10.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:75) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:111) [x-pack-core-7.10.0.jar:7.10.0]
at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:172) [x-pack-core-7.10.0.jar:7.10.0]
at org.elasticsearch.xpack.ilm.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:51) [x-pack-ilm-7.
10.0.jar:7.10.0]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.0.jar:7.10.0]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1292) [elasticsearch-7.10.0.jar:
7.10.0]
at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.rolloverIndex(AbstractClient.java:1804) [elasticsearch-7.10.
0.jar:7.10.0]
at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:141) [x-pack-co
re-7.10.0.jar:7.10.0]
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:174) [x-pack-ilm-7.10.0.jar:7.
10.0]
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:327) [x-pack-ilm-7.10.0.jar:
7.10.0]
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:265) [x-pack-ilm-7.10.0.jar:7.10.0
]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.10.0.jar:
7.10.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:216) [x-pack-core-7.10.0.j
ar:7.10.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
复制

2. 操作ES数据迁移

2.1 关闭ES

ps -ef|grep elasticsearch
kill -15 1701
复制

2.2 迁移数据

为了安全起见，将数据进行复制过去，原数据作为备份，以留后路

cp -rp /usr/local/elasticsearch-7.10.0/data /home/elasticsearch
cp -rp /usr/local/elasticsearch-7.10.0/log /home/elasticsearch
复制

2.3 查看授权是否正确

2.4 修改配置文件 elasticsearch.yml

cat /usr/local/elasticsearch-7.10.0/config/elasticsearch.yml
# 修改数据目录
path.data: /home/elasticsearch/data

# 修改日志目录
path.logs: /home/elasticsearch/logs
复制

2.5 启动es

# 进入ES的bin目录
cd /usr/local/elasticsearch-7.10.0/bin

# 启动
./elasticsearch -d
复制

3. 相关问题处理

3.1 启动报错，提示目录不存在

发现，即使配置了日志目录，且迁移，在安装目录当下的logs目录也还想要存在，且gc.log 默认存放于此

[elasticsearch@localhost bin]$ ./elasticsearch -d
Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
output:
[0.001s][error][logging] Error opening log file 'logs/gc.log': No such file or directory
[0.001s][error][logging] Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.
error:
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
        at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:126)
        at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
        at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
        at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137)
        at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
复制

3.2 启动成功，日志有告警

不影响大局，忽略

[elasticsearch@localhost bin]$ ./elasticsearch -d
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
复制

3.3 启动成功日志，并告知告警红转黄

划重点：Cluster health status changed from [RED] to [YELLOW]

[2024-06-05T16:21:33,173][INFO ][o.e.c.c.Coordinator      ] [es-node-1] cluster UUID [gBm2XYu-SdeolqoQrFjqxA]
[2024-06-05T16:21:33,385][INFO ][o.e.c.s.MasterService    ] [es-node-1] elected-as-master ([1] nodes joined)[{es-node-1}{hE1vR9sATHG
eeoepmSqF1w}{Bg5Kkq6hSG6VQeV3u5L79Q}{10.10.100.139}{10.10.100.139:9300}{cdhilmrstw}{ml.machine_memory=8254849024, xpack.installed=tr
ue, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 19, version: 7245, delta
: master node changed {previous [], current [{es-node-1}{hE1vR9sATHGeeoepmSqF1w}{Bg5Kkq6hSG6VQeV3u5L79Q}{10.10.100.139}{10.10.100.13
9:9300}{cdhilmrstw}{ml.machine_memory=8254849024, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2024-06-05T16:21:33,713][INFO ][o.e.c.s.ClusterApplierService] [es-node-1] master node changed {previous [], current [{es-node-1}{h
E1vR9sATHGeeoepmSqF1w}{Bg5Kkq6hSG6VQeV3u5L79Q}{10.10.100.139}{10.10.100.139:9300}{cdhilmrstw}{ml.machine_memory=8254849024, xpack.in
stalled=true, transform.node=true, ml.max_open_jobs=20}]}, term: 19, version: 7245, reason: Publication{term=19, version=7245}
[2024-06-05T16:21:33,833][INFO ][o.e.h.AbstractHttpServerTransport] [es-node-1] publish_address {10.10.100.139:9200}, bound_addresse
s {10.10.100.139:9200}
[2024-06-05T16:21:33,834][INFO ][o.e.n.Node               ] [es-node-1] started
[2024-06-05T16:21:35,101][INFO ][o.e.l.LicenseService     ] [es-node-1] license [b0d58af0-e26a-42a8-82a6-8dc9dee32d3f] mode [basic] 
- valid
[2024-06-05T16:21:35,103][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [es-node-1] Active license is now [BASIC]; Security is disa
bled
[2024-06-05T16:21:35,112][INFO ][o.e.g.GatewayService     ] [es-node-1] recovered [124] indices into cluster_state
[2024-06-05T16:22:03,881][INFO ][o.e.c.r.a.DiskThresholdMonitor] [es-node-1] releasing read-only-allow-delete block on indices: 【隐私说明：忽略部分index未展示】
[2024-06-05T16:23:32,972][INFO ][o.e.c.r.a.AllocationService] [es-node-1] Cluster health status changed from [RED] to [YELLOW] (reas
on: [shards started [[vir_log_28][0]]]).
复制

到处，问题已经解决

总结：部署安装数据库或者应用的时候，考虑磁盘容量及安全性，未必要不要与系统盘放到一起；且放置数据盘，扩容会更加的方便和快捷。