Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cassandra Cluster Example Custom Seed Cores on First Node #5737

Closed
mikeln opened this issue Mar 20, 2015 · 4 comments
Closed

Cassandra Cluster Example Custom Seed Cores on First Node #5737

mikeln opened this issue Mar 20, 2015 · 4 comments
Labels
area/example priority/backlog Higher priority than priority/awaiting-more-evidence. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.

Comments

@mikeln
Copy link
Contributor

mikeln commented Mar 20, 2015

examples/cassandra startup of cassandra pod with the custom Java seed provider code cores on the first pod:

Starting Datastax Agent... stomp:
Starting DataStax Agent datastax-agent
...done.
Starting Cassandra...
INFO 00:51:40,730 Logging initialized
INFO 00:51:40,770 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO 00:51:41,833 Data files directories: /cassandra_data/data
INFO 00:51:41,834 Commit log directory: /cassandra_data/commitlog
INFO 00:51:41,834 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO 00:51:41,834 disk_failure_policy is stop
INFO 00:51:41,834 commit_failure_policy is stop
INFO 00:51:41,840 Global memtable threshold is enabled at 122MB
INFO 00:51:42,358 Not using multi-threaded compaction
ERROR 00:51:43,212 Fatal error during configuration loading
java.lang.NullPointerException
at io.k8s.cassandra.KubernetesSeedProvider.getSeeds(KubernetesSeedProvider.java:116)
at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:500)
at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:111)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:156)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
null
Fatal error during configuration loading; unable to start. See log for stacktrace.

The io.k8s.cassandra.KubernetesSeedProvider.getSeeds pulls the seed addresses from kubernetes. Here is what this container was seeing:

root@opscenter:/# curl -l http://10.100.0.1:80/api/v1beta1/endpoints/cassandra
{
  "kind": "Endpoints",
  "id": "cassandra",
  "uid": "5bdda8b8-c9da-11e4-a97a-08002770ed21",
  "creationTimestamp": "2015-03-13T23:40:40Z",
  "selfLink": "/api/v1beta1/endpoints/cassandra?namespace=default",
  "resourceVersion": 107542,
  "apiVersion": "v1beta1",
  "namespace": "default",
  "protocol": "TCP",
  "endpoints": null
}

Analysis showed the following:

  • This only occurs on the first node of a cluster. When there are no other cassandra endpoints running/registered with kubernetes.
  • The custom java code was not checking for endpoints.endpoints:null. This caused the error.

Prototyped a fix by adding:

if (endpoints != null) {
    // here is a problem point. endpoint.endpoints can be null in first node cases
    if (endpoints.endpoints != null){
        for (String endpoint : endpoints.endpoints) {

Suggest incorporating this modification
@mbforbes
Copy link
Contributor

Thanks for the detailed report! Any chance you'd be willing to open a PR for this?

@mbforbes mbforbes added priority/backlog Higher priority than priority/awaiting-more-evidence. area/example sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels Mar 21, 2015
@mikeln
Copy link
Contributor Author

mikeln commented Mar 21, 2015

I would be willing to open a PR.

mikeln added a commit to mikeln/kubernetes that referenced this issue Mar 21, 2015
@mikeln
Issue kubernetes#5737 Seed Cores on First Node
e2db978
mikeln added a commit to mikeln/kubernetes that referenced this issue Mar 21, 2015
@mikeln
Issue kubernetes#5737 Seed Cores on First Node
6e47e1b
@mikeln mikeln mentioned this issue Mar 21, 2015
Closed
mikeln added a commit to mikeln/kubernetes that referenced this issue Mar 22, 2015
@mikeln
Issue kubernetes#5737 Cassandra Cluster Example Custom Seed Cores on …
d1f64ca 
…First Node
@mikeln mikeln mentioned this issue Mar 22, 2015
Closed
mikeln added a commit to mikeln/kubernetes that referenced this issue Mar 23, 2015
@mikeln
Issue kubernetes#5737 Cassandra Cluster Example Custom Seed Cores on …
bbca721 
…First Node
This was referenced Mar 23, 2015
Merged
Merged
@mikeln
Copy link
Contributor Author

mikeln commented Mar 24, 2015

Update the cassandra example to pick up recent changes. #5857
Incorporates the corrected code, jar, and yaml.
My testing indicates that the issue has been fixed.

@roberthbailey
Copy link
Contributor

Fixed by #5857.

jayunit100 pushed a commit to jayunit100/kubernetes that referenced this issue Mar 26, 2015
@mikeln @jayunit100
Issue kubernetes#5737 Cassandra Cluster Example Custom Seed Cores on …
ae68b48 
…First Node
akram pushed a commit to akram/kubernetes that referenced this issue Apr 7, 2015
@mikeln @akram
Issue kubernetes#5737 Cassandra Cluster Example Custom Seed Cores on …
dbe1f51 
…First Node
@mikeln mikeln mentioned this issue Nov 29, 2018
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/example priority/backlog Higher priority than priority/awaiting-more-evidence. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mbforbes @mikeln @roberthbailey