三节点的rac，部分sql被hang住，已经做了hang、systemstatedump，awr，在附件中
@盖总，已收集3个节点信息，请查看
231节点1
[root@hbdzzw01 ~]# netstat -s
Ip:
4894820317 total packets received
81 with invalid addresses
0 forwarded
0 incoming packets discarded
3858072622 incoming packets delivered
3601457642 requests sent out
1353 outgoing packets dropped
2465 fragments dropped after timeout
1218952454 reassemblies required
182280690 packets reassembled ok
17385041 packet reassembles failed
203061800 fragments received ok
234 fragments failed
1219264042 fragments created
Icmp:
1643073 ICMP messages received
523 input ICMP message failed.
ICMP input histogram:
destination unreachable: 1567409
timeout in transit: 346
redirects: 2
echo requests: 70738
echo replies: 4518
timestamp request: 30
address mask request: 4
1645192 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 1569426
time exceeded: 456
echo request: 4542
echo replies: 70738
timestamp replies: 30
IcmpMsg:
InType0: 4518
InType3: 1567409
InType5: 2
InType8: 70738
InType11: 346
InType13: 30
InType17: 4
InType37: 26
OutType0: 70738
OutType3: 1569426
OutType8: 4542
OutType11: 456
OutType14: 30
Tcp:
1047465 active connections openings
976661 passive connection openings
10475 failed connection attempts
395069 connection resets received
188 connections established
1315223772 segments received
1865867337 segments send out
14540545 segments retransmited
0 bad segments received.
675619 resets sent
Udp:
2538156309 packets received
1587043 packets to unknown port received.
197258 packet receive errors
2552538087 packets sent
RcvbufErrors: 197258
SndbufErrors: 51
IgnoredMulti: 2488151
UdpLite:
TcpExt:
33 SYN cookies sent
41 SYN cookies received
166 resets received for embryonic SYN_RECV sockets
626755 TCP sockets finished time wait in fast timer
10338 packets rejects in established connections because of timestamp
13243612 delayed acks sent
10002 delayed acks further delayed because of locked socket
Quick ack mode was activated 3780828 times
51 times the listen queue of a socket overflowed
84 SYNs to LISTEN sockets ignored
674396497 packets directly queued to recvmsg prequeue.
2072572590 packets directly received from backlog
181835812533 packets directly received from prequeue
132227316 packets header predicted
662923979 packets header predicted and directly queued to user
58017596 acknowledgments not containing data received
911303953 predicted acknowledgments
1200491 times recovered from packet loss due to SACK data
Detected reordering 140 times using FACK
Detected reordering 2307 times using SACK
Detected reordering 972 times using time stamp
4027 congestion windows fully recovered
2355 congestion windows partially recovered using Hoe heuristic
TCPDSACKUndo: 26438
29449 congestion windows recovered after partial ack
TCPLostRetransmit: 13304
13268 timeouts after SACK recovery
32946 timeouts in loss state
1368903 fast retransmits
32137 forward retransmits
354608 retransmits in slow start
1922623 other TCP timeouts
TCPLossProbes: 10425639
TCPLossProbeRecovery: 6050097
119962 sack retransmits failed
3782001 DSACKs sent for old packets
9 DSACKs sent for out of order packets
1554528 DSACKs received
1838 DSACKs for out of order packets received
145042 connections reset due to unexpected data
33118 connections reset due to early user close
454 connections aborted due to timeout
TCPDSACKIgnoredOld: 468
TCPDSACKIgnoredNoUndo: 1237753
TCPSpuriousRTOs: 173
TCPSackShifted: 131873
TCPSackMerged: 156122
TCPSackShiftFallback: 1573386
IPReversePathFilter: 23267
TCPReqQFullDoCookies: 33
TCPRetransFail: 1
TCPRcvCoalesce: 28023612
TCPOFOQueue: 72366
TCPOFOMerge: 9
TCPChallengeACK: 57271
TCPSpuriousRtxHostQueues: 814
TCPAutoCorking: 34662682
TCPFromZeroWindowAdv: 25974
TCPToZeroWindowAdv: 25974
TCPWantZeroWindowAdv: 346477
TCPSynRetrans: 2664
TCPOrigDataSent: 1713530252
TCPHystartTrainDetect: 38598
TCPHystartTrainCwnd: 754992
TCPHystartDelayDetect: 166
TCPHystartDelayCwnd: 4540
TCPACKSkippedPAWS: 6802
TCPACKSkippedSeq: 13045
IpExt:
InMcastPkts: 271030
OutMcastPkts: 89079
InBcastPkts: 2488543
OutBcastPkts: 193
InOctets: 3393970367372
OutOctets: 4391647611629
InMcastOctets: 42970039
OutMcastOctets: 14266895
InBcastOctets: 1368678924
OutBcastOctets: 66988
InNoECTPkts: 5172633735
InECT0Pkts: 13947
[root@hbdzzw01 ~]# netstat -su
IcmpMsg:
InType0: 4518
InType3: 1567625
InType5: 2
InType8: 70738
InType11: 346
InType13: 30
InType17: 4
InType37: 26
OutType0: 70738
OutType3: 1569642
OutType8: 4542
OutType11: 456
OutType14: 30
Udp:
2538215280 packets received
1587259 packets to unknown port received.
197258 packet receive errors
2552597061 packets sent
RcvbufErrors: 197258
SndbufErrors: 51
IgnoredMulti: 2488172
UdpLite:
IpExt:
InMcastPkts: 271030
OutMcastPkts: 89079
InBcastPkts: 2488564
OutBcastPkts: 193
InOctets: 3394022137115
OutOctets: 4391730270428
InMcastOctets: 42970039
OutMcastOctets: 14266895
InBcastOctets: 1368690365
OutBcastOctets: 66988
InNoECTPkts: 5172746937
InECT0Pkts: 13947
232节点2
[root@hbdzzw02 ~]# netstat -s
Ip:
5024589102 total packets received
220 with invalid addresses
0 forwarded
0 incoming packets discarded
3899238220 incoming packets delivered
3610924308 requests sent out
26198 outgoing packets dropped
103 dropped because of missing route
2495 fragments dropped after timeout
1319329460 reassemblies required
194060698 packets reassembled ok
10373188 packet reassembles failed
180936809 fragments received ok
114 fragments failed
1172497718 fragments created
Icmp:
9255671 ICMP messages received
207 input ICMP message failed.
ICMP input histogram:
destination unreachable: 1615516
timeout in transit: 370
redirects: 2
echo requests: 7637910
echo replies: 1794
timestamp request: 39
address mask request: 6
9258506 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 1617417
time exceeded: 381
echo request: 2759
echo replies: 7637910
timestamp replies: 39
IcmpMsg:
InType0: 1794
InType3: 1615516
InType5: 2
InType8: 7637910
InType11: 370
InType13: 39
InType17: 6
InType37: 34
OutType0: 7637910
OutType3: 1617417
OutType8: 2759
OutType11: 381
OutType14: 39
Tcp:
605789 active connections openings
1118369 passive connection openings
10989 failed connection attempts
394871 connection resets received
204 connections established
1272296371 segments received
2897097092 segments send out
14936523 segments retransmited
1 bad segments received.
722622 resets sent
Udp:
2614696023 packets received
1628048 packets to unknown port received.
122190 packet receive errors
2605985695 packets sent
RcvbufErrors: 122190
IgnoredMulti: 2511106
UdpLite:
TcpExt:
64 SYN cookies sent
110 SYN cookies received
223 resets received for embryonic SYN_RECV sockets
466610 TCP sockets finished time wait in fast timer
8262 packets rejects in established connections because of timestamp
13110427 delayed acks sent
9538 delayed acks further delayed because of locked socket
Quick ack mode was activated 3994553 times
83 times the listen queue of a socket overflowed
147 SYNs to LISTEN sockets ignored
653226771 packets directly queued to recvmsg prequeue.
966591553 packets directly received from backlog
146124467901 packets directly received from prequeue
85911303 packets header predicted
639294122 packets header predicted and directly queued to user
49910190 acknowledgments not containing data received
935248752 predicted acknowledgments
1244778 times recovered from packet loss due to SACK data
Detected reordering 116 times using FACK
Detected reordering 1534 times using SACK
Detected reordering 834 times using time stamp
2743 congestion windows fully recovered
1802 congestion windows partially recovered using Hoe heuristic
TCPDSACKUndo: 26463
25464 congestion windows recovered after partial ack
TCPLostRetransmit: 15501
25909 timeouts after SACK recovery
35745 timeouts in loss state
1467729 fast retransmits
24205 forward retransmits
390111 retransmits in slow start
2015145 other TCP timeouts
TCPLossProbes: 10562580
TCPLossProbeRecovery: 6320488
123820 sack retransmits failed
3996415 DSACKs sent for old packets
3 DSACKs sent for out of order packets
1077485 DSACKs received
300 DSACKs for out of order packets received
141152 connections reset due to unexpected data
38776 connections reset due to early user close
437 connections aborted due to timeout
TCPDSACKIgnoredOld: 390
TCPDSACKIgnoredNoUndo: 799297
TCPSpuriousRTOs: 77
TCPSackShifted: 157769
TCPSackMerged: 211928
TCPSackShiftFallback: 1611333
IPReversePathFilter: 33542
TCPReqQFullDoCookies: 64
TCPRetransFail: 2
TCPRcvCoalesce: 16152337
TCPOFOQueue: 102959
TCPOFOMerge: 3
TCPChallengeACK: 52168
TCPSYNChallenge: 1
TCPSpuriousRtxHostQueues: 460
TCPAutoCorking: 31340082
TCPFromZeroWindowAdv: 9883
TCPToZeroWindowAdv: 9883
TCPWantZeroWindowAdv: 204390
TCPSynRetrans: 6046
TCPOrigDataSent: 2785860744
TCPHystartTrainDetect: 39122
TCPHystartTrainCwnd: 747814
TCPHystartDelayDetect: 104
TCPHystartDelayCwnd: 2677
TCPACKSkippedPAWS: 4466
TCPACKSkippedSeq: 10673
TCPACKSkippedTimeWait: 7
IpExt:
InMcastPkts: 282538
OutMcastPkts: 92679
InBcastPkts: 2512035
OutBcastPkts: 200
InOctets: 3344118865999
OutOctets: 5822450278612
InMcastOctets: 45153198
OutMcastOctets: 14881137
InBcastOctets: 1381466833
OutBcastOctets: 79659
InNoECTPkts: 5275125891
InECT0Pkts: 49984
[root@hbdzzw02 ~]# netstat -su
IcmpMsg:
InType0: 1794
InType3: 1615516
InType5: 2
InType8: 7637942
InType11: 370
InType13: 39
InType17: 6
InType37: 34
OutType0: 7637942
OutType3: 1617417
OutType8: 2759
OutType11: 381
OutType14: 39
Udp:
2614750944 packets received
1628048 packets to unknown port received.
122190 packet receive errors
2606039908 packets sent
RcvbufErrors: 122190
IgnoredMulti: 2511115
UdpLite:
IpExt:
InMcastPkts: 282538
OutMcastPkts: 92679
InBcastPkts: 2512044
OutBcastPkts: 200
InOctets: 3344171553345
OutOctets: 5822493804152
InMcastOctets: 45153198
OutMcastOctets: 14881137
InBcastOctets: 1381472017
OutBcastOctets: 79659
InNoECTPkts: 5275227869
InECT0Pkts: 49987
233节点3
[root@hbdzzw03 ~]# netstat -s
Ip:
4543442754 total packets received
433 with invalid addresses
0 forwarded
0 incoming packets discarded
3726333242 incoming packets delivered
3473880181 requests sent out
103236 outgoing packets dropped
871 dropped because of missing route
2080 fragments dropped after timeout
983541040 reassemblies required
166505513 packets reassembled ok
8049605 packet reassembles failed
176938340 fragments received ok
2559 fragments failed
1131723665 fragments created
Icmp:
1790210 ICMP messages received
275 input ICMP message failed.
ICMP input histogram:
destination unreachable: 1720319
timeout in transit: 426
redirects: 2
echo requests: 67755
echo replies: 1637
timestamp request: 36
address mask request: 4
1792599 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 1722497
time exceeded: 305
echo request: 2006
echo replies: 67755
timestamp replies: 36
IcmpMsg:
InType0: 1637
InType3: 1720319
InType5: 2
InType8: 67755
InType11: 426
InType13: 36
InType17: 4
InType37: 31
OutType0: 67755
OutType3: 1722497
OutType8: 2006
OutType11: 305
OutType14: 36
Tcp:
595237 active connections openings
1005033 passive connection openings
11036 failed connection attempts
395642 connection resets received
199 connections established
1351611580 segments received
2318300633 segments send out
14635043 segments retransmited
1 bad segments received.
704887 resets sent
InCsumErrors: 1
Udp:
2369845769 packets received
1737635 packets to unknown port received.
94782 packet receive errors
2387242605 packets sent
RcvbufErrors: 94782
IgnoredMulti: 2514056
UdpLite:
TcpExt:
3 SYN cookies sent
9 SYN cookies received
214 resets received for embryonic SYN_RECV sockets
346530 TCP sockets finished time wait in fast timer
7602 packets rejects in established connections because of timestamp
17304138 delayed acks sent
12044 delayed acks further delayed because of locked socket
Quick ack mode was activated 3838278 times
43 times the listen queue of a socket overflowed
46 SYNs to LISTEN sockets ignored
712959984 packets directly queued to recvmsg prequeue.
811463201 packets directly received from backlog
183994757024 packets directly received from prequeue
76182224 packets header predicted
697970575 packets header predicted and directly queued to user
71632922 acknowledgments not containing data received
969644040 predicted acknowledgments
1219509 times recovered from packet loss due to SACK data
Detected reordering 105 times using FACK
Detected reordering 1570 times using SACK
Detected reordering 771 times using time stamp
2799 congestion windows fully recovered
1657 congestion windows partially recovered using Hoe heuristic
TCPDSACKUndo: 26259
25175 congestion windows recovered after partial ack
TCPLostRetransmit: 14816
10427 timeouts after SACK recovery
32688 timeouts in loss state
1488142 fast retransmits
38905 forward retransmits
360722 retransmits in slow start
1950668 other TCP timeouts
TCPLossProbes: 10357146
TCPLossProbeRecovery: 6165063
119683 sack retransmits failed
3838570 DSACKs sent for old packets
3 DSACKs sent for out of order packets
1109719 DSACKs received
244 DSACKs for out of order packets received
142362 connections reset due to unexpected data
37525 connections reset due to early user close
595 connections aborted due to timeout
TCPDSACKIgnoredOld: 1093
TCPDSACKIgnoredNoUndo: 814957
TCPSpuriousRTOs: 100
TCPSackShifted: 261207
TCPSackMerged: 473273
TCPSackShiftFallback: 1631568
IPReversePathFilter: 23760
TCPReqQFullDoCookies: 3
TCPRetransFail: 10
TCPRcvCoalesce: 16700961
TCPOFOQueue: 49227
TCPOFOMerge: 3
TCPChallengeACK: 50655
TCPSpuriousRtxHostQueues: 193
TCPAutoCorking: 45827948
TCPFromZeroWindowAdv: 6951
TCPToZeroWindowAdv: 6951
TCPWantZeroWindowAdv: 200586
TCPSynRetrans: 2474
TCPOrigDataSent: 2203149503
TCPHystartTrainDetect: 39101
TCPHystartTrainCwnd: 753168
TCPHystartDelayDetect: 127
TCPHystartDelayCwnd: 3800
TCPACKSkippedPAWS: 4587
TCPACKSkippedSeq: 8519
TCPACKSkippedTimeWait: 1
IpExt:
InMcastPkts: 282244
OutMcastPkts: 97680
InBcastPkts: 2514978
OutBcastPkts: 503
InOctets: 2616688139842
OutOctets: 4849270034208
InMcastOctets: 45154988
OutMcastOctets: 15511205
InBcastOctets: 1383180520
OutBcastOctets: 225154
InNoECTPkts: 4600220754
InECT0Pkts: 24698
[root@hbdzzw03 ~]# netstat -su
IcmpMsg:
InType0: 1637
InType3: 1720319
InType5: 2
InType8: 67755
InType11: 426
InType13: 36
InType17: 4
InType37: 31
OutType0: 67755
OutType3: 1722497
OutType8: 2006
OutType11: 305
OutType14: 36
Udp:
2369870340 packets received
1737635 packets to unknown port received.
94782 packet receive errors
2387268371 packets sent
RcvbufErrors: 94782
IgnoredMulti: 2514065
UdpLite:
IpExt:
InMcastPkts: 282244
OutMcastPkts: 97680
InBcastPkts: 2514987
OutBcastPkts: 503
InOctets: 2616704689734
OutOctets: 4849307243776
InMcastOctets: 45154988
OutMcastOctets: 15511205
InBcastOctets: 1383185704
OutBcastOctets: 225154
InNoECTPkts: 4600265936
InECT0Pkts: 24698
评论
有用 0注意这个信息：
17385041 packet reassembles failed
这是一个已知问题，在我们 3月号的技术通讯中就已经提到过了。你下载来看一下。
在以下链接中可以找到：
https://obs-emcsapp-public.obs.cn-north-4.myhuaweicloud.com:443/docDownload
评论 有用 0
