Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prefect stops using cloud after updating to latest version #16665

Open
chiang-yuan opened this issue Jan 9, 2025 · 2 comments
Open

Prefect stops using cloud after updating to latest version #16665

chiang-yuan opened this issue Jan 9, 2025 · 2 comments
Labels
bug Something isn't working

Comments

@chiang-yuan
Copy link

Bug summary

I have been trying to update to the latest version as the old version starts to fail the connection to prefect cloud as well.

After fighting with the same error, I decided to erase all the data under .prefect. It seems solving the authentication problem but now prefect never uses the cloud server anymore, even after I have set PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='False' in prefect.toml

Below is the snippet to use dask Slumcluster to run prefect flow. Any help is very appreciated!

from dask.distributed import Client
from dask_jobqueue import SLURMCluster
from prefect_dask import DaskTaskRunner

nodes_per_alloc = 1
gpus_per_alloc = 4
ntasks = 1

cluster_kwargs = dict(
    cores=1,
    memory="64 GB",
    shebang="#!/bin/bash",
    account="xxxx",
    walltime="00:30:00",
    job_mem="0",
    job_script_prologue=[
        "source ~/.bashrc",
        "module load python",
        "source activate xxxx",
    ],
    job_directives_skip=["-n", "--cpus-per-task", "-J"],
    job_extra_directives=[
        "-J vm",
        "-q debug",
        f"-N {nodes_per_alloc}",
        "-C gpu",
        f"-G {gpus_per_alloc}",
    ],
)

cluster = SLURMCluster(**cluster_kwargs)
print(cluster.job_script())
cluster.adapt(minimum_jobs=2, maximum_jobs=2)
client = Client(cluster)

results = someflow.with_options(
    task_runner=DaskTaskRunner(address=client.scheduler.address),
)()

Version info

Version:             3.1.11
API version:         0.8.4
Python version:      3.11.8
Git commit:          e448bd34
Built:               Thu, Jan 2, 2025 1:11 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         ephemeral
Pydantic version:    2.9.2
Server:
  Database:          sqlite
  SQLite version:    3.45.2
Integrations:
  prefect-dask:      0.3.2

Additional context

PREFECT_PROFILE='default'
PREFECT_API_KEY='********' (from profile)
PREFECT_API_URL='https://api.prefect.cloud/api/accounts/f7d40474-9362-4bfa-8950-ee6a43ec00f3/workspaces/d4bb0913-5f5e-49f7-bfc5-06509088baeb' (from profile)
PREFECT_HOME='/pscratch/sd/c/cyrusyc/.prefect' (from env)
PREFECT_SERVER_ALLOW_EPHEMERAL_MODE='False' (from profile)
@chiang-yuan chiang-yuan added the bug Something isn't working label Jan 9, 2025
@desertaxle
Copy link
Member

Thanks for the bug report @chiang-yuan! I think your settings might not be propagating to your cluster. Can you print the current settings in a task submitted to your cluster with print(prefect.settings.get_current_settings().api.url)? That'll help us see what API URL the tasks running in your cluster use.

@chiang-yuan
Copy link
Author

chiang-yuan commented Jan 10, 2025 •
edited
Loading

hi @desertaxle thanks for the help. To avoid complication from dask cluster, I ran a pure flow with default runner but it also uses the temporary server and all the flow and task prints out None for your command.

08:08:25.591 | INFO    | prefect - Starting temporary server on http://127.0.0.1:8090
See https://docs.prefect.io/3.0/manage/self-host#self-host-a-prefect-server for more information on running a dedicated Prefect server.
08:08:40.734 | INFO    | Flow run 'sandy-starling' - Beginning flow run 'sandy-starling' for flow 'run-fcc'
fcc flow None
Retrieving SummaryDoc documents: 100%
 1/1 [00:00<00:00, 123.99it/s]
vm task None
vm task None

If I run the command directly it also prints out None

import prefect
print(prefect.settings.get_current_settings().api.url)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@desertaxle @chiang-yuan