暂无图片
Mysql Crash:mysqld: invalid fastbin entry
我来答
分享
嘿嘿嘿
2024-03-22
Mysql Crash:mysqld: invalid fastbin entry

MySQL 5.7.28,CentOS6,主机服务器:128c128G

MySQL crash之后被守护进程拉起,现在无法定位原因。Error 日志如下:

*** glibc detected *** /newdata/app/mysql57/bin/mysqld: invalid fastbin entry (free): 0x00002bcb9d48a0b0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3261876166]
/newdata/app/mysql57/bin/mysqld[0xd10509]
/newdata/app/mysql57/bin/mysqld(_Z15mysql_change_dbP3THDRK25st_mysql_const_lex_stringb+0x3ce)[0xd1117e]
/newdata/app/mysql57/bin/mysqld(_Z21mysql_execute_commandP3THDb+0xac4)[0xd48e34]
/newdata/app/mysql57/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3dd)[0xd4d6cd]
/newdata/app/mysql57/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x11d5)[0xd4e975]
/newdata/app/mysql57/bin/mysqld(_Z10do_commandP3THD+0x194)[0xd4f854]
/newdata/app/mysql57/bin/mysqld(handle_connection+0x2ac)[0xe21a8c]
/newdata/app/mysql57/bin/mysqld(pfs_spawn_thread+0x174)[0x10004e4]
/lib64/libpthread.so.0[0x3261c079d1]
/lib64/libc.so.6(clone+0x6d)[0x32618e8b6d]

......

Thread pointer: 0x2bcd21149720
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 2bd54a8dce28 thread_stack 0x40000
/newdata/app/mysql57/bin/mysqld(my_print_stacktrace+0x35)[0xf81bf5]
/newdata/app/mysql57/bin/mysqld(handle_fatal_signal+0x4a4)[0x7fb344]
/lib64/libpthread.so.0[0x3261c0f710]
/lib64/libc.so.6(gsignal+0x35)[0x3261832925]
/lib64/libc.so.6(abort+0x175)[0x3261834105]
/lib64/libc.so.6[0x3261870837]
/lib64/libc.so.6[0x3261876166]
/newdata/app/mysql57/bin/mysqld[0xd10509]
/newdata/app/mysql57/bin/mysqld(_Z15mysql_change_dbP3THDRK25st_mysql_const_lex_stringb+0x3ce)[0xd1117e]
/newdata/app/mysql57/bin/mysqld(_Z21mysql_execute_commandP3THDb+0xac4)[0xd48e34]
/newdata/app/mysql57/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x3dd)[0xd4d6cd]
/newdata/app/mysql57/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x11d5)[0xd4e975]
/newdata/app/mysql57/bin/mysqld(_Z10do_commandP3THD+0x194)[0xd4f854]
/newdata/app/mysql57/bin/mysqld(handle_connection+0x2ac)[0xe21a8c]
/newdata/app/mysql57/bin/mysqld(pfs_spawn_thread+0x174)[0x10004e4]
/lib64/libpthread.so.0[0x3261c079d1]
/lib64/libc.so.6(clone+0x6d)[0x32618e8b6d]

具体其他见附件。

gdb可以定位到

(gdb) b *0xd10509

Breakpoint 1 at 0xd10509: file /export/home/pb2/build/sb_0-36131509-1569573142.98/mysql-5.7.28/sql/sql_class.h, line 4178.
(gdb) b *0xd1117e
Breakpoint 2 at 0xd1117e: file /export/home/pb2/build/sb_0-36131509-1569573142.98/mysql-5.7.28/sql/sql_db.cc, line 1689.

但是没有找到相关bug,崩溃前数据库负载正常,无异常会话,感觉像是内存问题,有没有老师帮看下

我来答
添加附件
收藏
分享
问题补充
5条回答
默认
最新
嘿嘿嘿
升级问题到: 紧急故障
暂无图片 评论
暂无图片 有用 0
打赏 0
嘿嘿嘿
上传附件:error.log
暂无图片 评论
暂无图片 有用 0
打赏 0
zhou

机器:128c128G
申请内存:136G,143492450 K bytes of memory
看看是否OOM了?
key_buffer_size=67108864
read_buffer_size=4194304
max_used_connections=5001
max_threads=5000
thread_count=1469
connection_count=1468
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 143492450 K bytes of memory

暂无图片 评论
暂无图片 有用 0
打赏 0
嘿嘿嘿
题主
2024-03-25
看系统日志并没有OOM,这个是理论情况下达到最大max_thread=5000的时候吧,实际线程数是1469,sort_buffer_size是24M,按这个算也就是41G左右
zhou

1469*(read_buffer_size 2M+sort_buffer_size 24M +2M ?(thread_stack+read_rnd_buffer_size+join_buffer_size+binlog_cache_size))
+innodb_buffer_pool_size?
innodb_buffer_pool_size才是大头吧

暂无图片 评论
暂无图片 有用 0
打赏 0
嘿嘿嘿
题主
2024-03-25
对,innodb_buffer_pool_size 72G,其他几个都是M级别 | bulk_insert_buffer_size | 83886080 | | innodb_log_buffer_size | 536870912 | | innodb_sort_buffer_size | 1048576 | | join_buffer_size | 134217728 | | key_buffer_size | 67108864 | | myisam_sort_buffer_size | 8388608 | | preload_buffer_size | 32768 | | read_buffer_size | 4194304 | | read_rnd_buffer_size | 4194304 | | sort_buffer_size | 25165824 |
zhou

OOM的可能性比较大,join_buffer_size,sort_buffer_size 设置比一般系统大很多,如果确实复杂SQL比较多,控制一下连接数吧(优化慢查询来控制),调低innodb_buffer_pool_size 。按说join_buffer_size 128M太大,在mysql5.7也没用,NL join 不适合返回太多记录,大表关联hash join更合适

暂无图片 评论
暂无图片 有用 0
打赏 0
回答交流
Markdown


请输入正文
提交