Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Failing Test] ci-kubernetes-kubemark-gce-scale-scheduler | kubemark-5000-scheduler #123884

Open
pacoxu opened this issue Mar 12, 2024 · 10 comments
Open

[Failing Test] ci-kubernetes-kubemark-gce-scale-scheduler | kubemark-5000-scheduler #123884

pacoxu opened this issue Mar 12, 2024 · 10 comments
Assignees
@alculquicondor
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@pacoxu
Copy link
Member

pacoxu commented Mar 12, 2024

Which jobs are failing?

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale-scheduler/1766766416834334720

Which tests are failing?

  • kubetest.Kubemark
  • kubetest.Kubemark Up

Since when has it been failing?

3 failures in a row since 03-06
lastpass was @ 03-04
before that 3 failures.

Testgrid link

https://testgrid.k8s.io/sig-scalability-kubemark#kubemark-5000-scheduler

Reason for failure (if possible)

WARNING: Could not store access token in cache: database is locked
ERROR: gcloud crashed (OperationalError): database is locked

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics
Traceback (most recent call last):
  File "/google-cloud-sdk/lib/gcloud.py", line 189, in <module>
    main()
  File "/google-cloud-sdk/lib/gcloud.py", line 185, in main
    sys.exit(gcloud_main.main())
             ^^^^^^^^^^^^^^^^^^
  File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/crash_handling.py", line 199, in Wrapper
    target_function(*args, **kwargs)
  File "/google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py", line 186, in main
    gcloud_cli.Execute()
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 1024, in Execute
    self._HandleAllErrors(exc, command_path_string, specified_arg_names)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 1061, in _HandleAllErrors
ERROR: gcloud crashed (OperationalError): database is locked
    exceptions.HandleError(exc, command_path_string, self.__known_error_handler)
  File "/google-cloud-sdk/lib/googlecloudsdk/calliope/exceptions.py", line 557, in HandleError
    core_exceptions.reraise(exc)
  File "/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 146, in reraise
    six.reraise(type(exc_value), exc_value, tb)
  File "/google-cloud-sdk/lib/third_party/six/__init__.py", line 719, in reraise
...
sqlite3.OperationalError: database is locked
WARNING: Could not store access token in cache: database is locked
WARNING: Could not store access token in cache: database is locked

Anything else we need to know?

No response

Relevant SIG(s)

/sig scalibility scheduling
@pacoxu pacoxu added the kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. label Mar 12, 2024
@k8s-ci-robot
Copy link
Contributor

@pacoxu: The label(s) sig/scalibility cannot be applied, because the repository doesn't have them.

In response to this:

Which jobs are failing?

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale-scheduler/1766766416834334720

Which tests are failing?

  • kubetest.Kubemark
  • kubetest.Kubemark Up

Since when has it been failing?

3 failures in a row since 03-06
lastpass was @ 03-04
before that 3 failures.

Testgrid link

https://testgrid.k8s.io/sig-scalability-kubemark#kubemark-5000-scheduler

Reason for failure (if possible)

WARNING: Could not store access token in cache: database is locked
ERROR: gcloud crashed (OperationalError): database is locked

If you would like to report this issue, please run the following command:
 gcloud feedback

To check gcloud for common problems, please run the following command:
 gcloud info --run-diagnostics
Traceback (most recent call last):
 File "/google-cloud-sdk/lib/gcloud.py", line 189, in <module>
   main()
 File "/google-cloud-sdk/lib/gcloud.py", line 185, in main
   sys.exit(gcloud_main.main())
            ^^^^^^^^^^^^^^^^^^
 File "/google-cloud-sdk/lib/googlecloudsdk/command_lib/crash_handling.py", line 199, in Wrapper
   target_function(*args, **kwargs)
 File "/google-cloud-sdk/lib/googlecloudsdk/gcloud_main.py", line 186, in main
   gcloud_cli.Execute()
 File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 1024, in Execute
   self._HandleAllErrors(exc, command_path_string, specified_arg_names)
 File "/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 1061, in _HandleAllErrors
ERROR: gcloud crashed (OperationalError): database is locked
   exceptions.HandleError(exc, command_path_string, self.__known_error_handler)
 File "/google-cloud-sdk/lib/googlecloudsdk/calliope/exceptions.py", line 557, in HandleError
   core_exceptions.reraise(exc)
 File "/google-cloud-sdk/lib/googlecloudsdk/core/exceptions.py", line 146, in reraise
   six.reraise(type(exc_value), exc_value, tb)
 File "/google-cloud-sdk/lib/third_party/six/__init__.py", line 719, in reraise
...
sqlite3.OperationalError: database is locked
WARNING: Could not store access token in cache: database is locked
WARNING: Could not store access token in cache: database is locked

Anything else we need to know?

No response

Relevant SIG(s)

/sig scalibility scheduling

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 12, 2024
@pacoxu
Copy link
Member Author

pacoxu commented Mar 12, 2024

/sig scalability

@k8s-ci-robot k8s-ci-robot added the sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. label Mar 12, 2024
@kerthcet
Copy link
Member

kerthcet commented Apr 6, 2024

Is this related to kube-scheduler, didn't see related logs.

@pacoxu
Copy link
Member Author

pacoxu commented May 14, 2024

/cc @wojtek-t
Do you have any clue about this failure?

@BenTheElder BenTheElder mentioned this issue Jul 25, 2024
Closed
@BenTheElder
Copy link
Member

nudge @wojtek-t [they've been OOO and maybe have a moment now or can reroute]

@wojtek-t
Copy link
Member

@alculquicondor - can you please take a look or delagate?

/assign @alculquicondor

@BenTheElder
Copy link
Member

This seems to be a constant failure for months now? https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-kubernetes-kubemark-gce-scale-scheduler

Should we turn down the job? cc @kubernetes/sig-scalability

@BenTheElder
Copy link
Member

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 12, 2024
@kerthcet
Copy link
Member

kerthcet commented Aug 13, 2024 •
edited
Loading

I'll take a look today. Aldo is on vacation now.

@kerthcet
Copy link
Member

Well, I think the error is obvious, sqlite3.OperationalError: database is locked, I guess gcloud use sqlite to store some configurations and there's a concurrency problem or timeout waiting for the lock to release.

Out of my reach then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alculquicondor alculquicondor

Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. sig/scalability Categorizes an issue or PR as relevant to SIG Scalability. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@BenTheElder @alculquicondor @pacoxu @wojtek-t @kerthcet @k8s-ci-robot