Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing detection of disk availability when sending heartbeats. #287

Open
cserwen opened this issue Apr 24, 2023 · 2 comments
Open

Missing detection of disk availability when sending heartbeats. #287

cserwen opened this issue Apr 24, 2023 · 2 comments

Comments

@cserwen
Copy link
Contributor

cserwen commented Apr 24, 2023

In RocketMQ DLedger, when a disk which stores data or index fails, a status which No-Master may occur for brokerGroup.
The following is the process of the problem：

# Process

Initial state: term=2, n2 is the Master node

  • 2023-04-13 02:04:53, disk failure, term=3, n0 becomes Master
2023-04-13 02:04:53 INFO QuorumAckChecker-n0 - [n0][LEADER] term=3 ledgerBegin=133702137569 ledgerEnd=137547196422 committed=137547196422 watermarks={3:{"n0":137547196422,"n1":137547196422,"n2":-1}}  
2023-04-13 02:04:56 WARN DLedgerServer-ScheduledExecutor - preferredLeaderId = n2 is not online
  • 2023-04-13 02:05:08, term=3, n0 detected that n2 is online, handing over the Master role.
2023-04-13 02:05:08 INFO DLedgerServer-ScheduledExecutor - preferredLeaderId = n2, which has the smallest fall behind index = 12 and is decided to be transferee.  
2023-04-13 02:05:08 INFO DLedgerServer-ScheduledExecutor - handleLeadershipTransfer: LeadershipTransferRequest{transferId='null', transfereeId='n2', takeLeadershipLedgerIndex=0, group='null', remoteId='null', lo  calId='null', code=200, leaderId='null', term=3}
  • 2023-04-13 02:05:08，term=4, n2 becomes master
2023-04-13 02:05:08 INFO StateMaintainer - [n2] [PARSE_VOTE_RESULT] cost=1 term=4 memberNum=3 allNum=2 acceptedNum=2 notReadyTermNum=0 biggerLedgerNum=0 alreadyHasLeader=false maxTerm=4 result=PASSED
2023-04-13 02:05:08 INFO StateMaintainer - [n2] [VOTE_RESULT] has been elected to be the leader in term 4
2023-04-13 02:05:08 INFO StateMaintainer - TakeLeadershipTask finished. request=LeadershipTransferRequest{transferId='n0', transfereeId='n2', takeLeadershipLedgerIndex=137547318699, group='c4cloudsrv-miot-rocketmq-raft8', remoteId='n2', localId='n0', code=200, leaderId='n0', term=3}, response=LeadershipTransferResponse{group='null', remoteId='null', localId='null', code=200, leaderId='null', term=4}, term=4, role=LEADER
2023-04-13 02:05:08 INFO StateMaintainer - [n2] [ChangeRoleToLeader] from term: 4 and currTerm: 4
  • 2023-04-13 02:05:27, term=4, Master role transparently passed to Broker failed
2023-04-13 02:05:08 INFO DLegerRoleChangeHandler_1 - Begin handling broker role change term=4 role=LEADER currStoreRole=SLAVE
2023-04-13 02:05:27 INFO DLegerRoleChangeHandler_1 - [MONITOR]Failed handling broker role change term=4 role=LEADER currStoreRole=SLAVE cost=19334
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

In the final state, DLedger chose n2 as the Master, but there was no Master for the Broker group, and the election has not been re-initiated since then.

# Question

  • Why did n0 and n1 not initiate an election after n2 became Master in term 4.

If the Master has not sent heartbeats to the follower, the follower will trigger the election; but if the heartbeat has been sent normally, the slave node will not initiate the election.

The memberState object lock is used to detect disk failures. When writing a message, the lock will be held. If the disk fails, the lock will not be released in time, and the heartbeat thread will not acquire the lock, thus detecting the disk failure. It can be seen that writing messages is a trigger to detect disk failures, but if the client no longer writes messages, the heartbeat thread can always acquire the lock, and it keeps sending heartbeats.

dledger-lock drawio

# TODO

If no data is written, the node where the faulty disk is located will also become the Master. Therefore, I think it is necessary to add a task to regularly detect whether the disks are available. to avoid this situation.
@humkum
Copy link

humkum commented Apr 24, 2023

I'd like to follow this issue. Plz assign this issue to me, thanks.

@TheR1sing3un
Copy link
Contributor

I'd like to follow this issue. Plz assign this issue to me, thanks.

Welcome~ You can write a brief improvement proposal~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cserwen @humkum @TheR1sing3un