Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Client occasionally crashes with index out of bounds when getting a node from a partition #367

Open
asecondo opened this issue Feb 2, 2022 · 3 comments
Open

Client occasionally crashes with index out of bounds when getting a node from a partition #367

asecondo opened this issue Feb 2, 2022 · 3 comments

Comments

@asecondo
Copy link

asecondo commented Feb 2, 2022

Occasionally when writing to Aerospike, a service crashes in the Aerospike client when attempting to get a node from the current partition. The length of the array is always 0 when the crash occurs.

The code here could be modified to ensure it's not getting exceeding the bounds of replicas, but I'm trying to understand why this is happening. Any thoughts?

Client version: 5.7.0

Example stack trace (excluding service specific code):

github.com/aerospike/aerospike-client-go/v5.(*Partition).getSequenceNode(0x46d5d3, 0x61f9685e)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/partition.go:195 +0xfc
github.com/aerospike/aerospike-client-go/v5.(*Partition).GetNodeWrite(0xc01b79eef8, 0x48c6d7)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/partition.go:165 +0x53
github.com/aerospike/aerospike-client-go/v5.(*operateCommand).getNode(0xc07677f782a295c3, {0x4e94324576a4, 0x1d0a1c0})
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/operate_command.go:43 +0x2d
github.com/aerospike/aerospike-client-go/v5.(*baseCommand).executeAt(0xc03018ae10, {0x13db778, 0xc03018ae10}, 0xc001fabb90, 0x80, {0xc001fabb90, 0x0, 0x1d0a1c0}, 0x0, 0x0)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/command.go:2058 +0x532
github.com/aerospike/aerospike-client-go/v5.(*baseCommand).execute(0xc001fabb90, {0x13db778, 0xc03018ae10}, 0x4)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/command.go:1991 +0x8a
github.com/aerospike/aerospike-client-go/v5.(*operateCommand).Execute(...)
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/operate_command.go:60
github.com/aerospike/aerospike-client-go/v5.(*Client).Operate(0xc0001fc840, 0x0, 0x0, {0xc01cf073e0, 0xfc4500, 0xc01cf13eb0})
	/go/pkg/mod/github.com/aerospike/aerospike-client-go/v5@v5.7.0/client.go:515 +0x3be
@khaf
Copy link
Collaborator

khaf commented Feb 7, 2022 •
edited
Loading

This is very surprising. Are you monitoring your cluster? Does a partition or major network issue happen at the same time as these errors?

@asecondo
Copy link
Author

@khaf We are monitoring our clusters, but I wasn't able to find any direct indicators that a partition or major network issue happened during the time this error popped up. Do you have any suggestions for anything to specifically look for?

For a bit more context, this has only happened ~75 times in the past 30 days, and 1 time in the past 2 weeks. It's likely that this is related to some partition or network issue as you described since there's no clear pattern that I've been able to identify yet.

@LevOlkha
Copy link

LevOlkha commented Nov 19, 2024 •
edited
Loading

Any updated on this issue?
we are getting the same crash with version 6.

I think it happens if we are closing client from one go-routine, while sending commend from another go-routine

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@khaf @asecondo @LevOlkha