Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error = failed to retrieve local environment: failed to find local environment #12285

Open
2 tasks done
XBadFellaX opened this issue Oct 3, 2024 · 33 comments · May be fixed by #12336
Open
2 tasks done

Error = failed to retrieve local environment: failed to find local environment #12285

XBadFellaX opened this issue Oct 3, 2024 · 33 comments · May be fixed by #12336
Labels

Comments

@XBadFellaX
Copy link

Before you start please confirm the following.

Problem Description

Getting these errors after update to 2.22.0

2024/10/03 02:31AM WRN github.com/portainer/portainer-ee/api/docker/local.go:79 > failed to clean up updater stack | error="failed to retrieve local environment: failed to find local environment" retry time=2 stack_trace=[{"func":"cleanUpGhostUpdaterStacks.ScanAndCleanUpGhostUpdaterContainers.func1","line":"21","source":"local.go"},{"func":"Retry","line":"74","source":"local.go"},{"func":"cleanUpGhostUpdaterStacks","line":"434","source":"main.go"},{"func":"goexit","line":"1695","source":"asm_amd64.s"}]

2024/10/03 02:32AM WRN github.com/portainer/portainer-ee/api/cmd/portainer/main.go:435 > unable to clean up ghost updater stack | error="failed to retrieve local environment: failed to find local environment" stack_trace=[{"func":"cleanUpGhostUpdaterStacks.ScanAndCleanUpGhostUpdaterContainers.func1","line":"21","source":"local.go"},{"func":"Retry","line":"74","source":"local.go"},{"func":"cleanUpGhostUpdaterStacks","line":"434","source":"main.go"},{"func":"goexit","line":"1695","source":"asm_amd64.s"}]

Expected Behavior

No Error

Actual Behavior

2024/10/03 02:31AM WRN github.com/portainer/portainer-ee/api/docker/local.go:79 > failed to clean up updater stack | error="failed to retrieve local environment: failed to find local environment" retry time=2 stack_trace=[{"func":"cleanUpGhostUpdaterStacks.ScanAndCleanUpGhostUpdaterContainers.func1","line":"21","source":"local.go"},{"func":"Retry","line":"74","source":"local.go"},{"func":"cleanUpGhostUpdaterStacks","line":"434","source":"main.go"},{"func":"goexit","line":"1695","source":"asm_amd64.s"}]

2024/10/03 02:32AM WRN github.com/portainer/portainer-ee/api/cmd/portainer/main.go:435 > unable to clean up ghost updater stack | error="failed to retrieve local environment: failed to find local environment" stack_trace=[{"func":"cleanUpGhostUpdaterStacks.ScanAndCleanUpGhostUpdaterContainers.func1","line":"21","source":"local.go"},{"func":"Retry","line":"74","source":"local.go"},{"func":"cleanUpGhostUpdaterStacks","line":"434","source":"main.go"},{"func":"goexit","line":"1695","source":"asm_amd64.s"}]

Steps to Reproduce

Go to Portainer
Login
Check Logs

Portainer logs or screenshots

2024/10/03 02:31AM WRN github.com/portainer/portainer-ee/api/docker/local.go:79 > failed to clean up updater stack | error="failed to retrieve local environment: failed to find local environment" retry time=2 stack_trace=[{"func":"cleanUpGhostUpdaterStacks.ScanAndCleanUpGhostUpdaterContainers.func1","line":"21","source":"local.go"},{"func":"Retry","line":"74","source":"local.go"},{"func":"cleanUpGhostUpdaterStacks","line":"434","source":"main.go"},{"func":"goexit","line":"1695","source":"asm_amd64.s"}]

2024/10/03 02:32AM WRN github.com/portainer/portainer-ee/api/cmd/portainer/main.go:435 > unable to clean up ghost updater stack | error="failed to retrieve local environment: failed to find local environment" stack_trace=[{"func":"cleanUpGhostUpdaterStacks.ScanAndCleanUpGhostUpdaterContainers.func1","line":"21","source":"local.go"},{"func":"Retry","line":"74","source":"local.go"},{"func":"cleanUpGhostUpdaterStacks","line":"434","source":"main.go"},{"func":"goexit","line":"1695","source":"asm_amd64.s"}]

Portainer version

2.22.0

Portainer Edition

Business Edition (BE/EE) with 5NF / 3NF license

Platform and Version

Docker 27.3.1

OS and Architecture

Ubuntu 24.04

Browser

Chrome 129

What command did you use to deploy Portainer?

No response

Additional Information

No response

@ProfessorCha0s
Copy link

ProfessorCha0s commented Oct 3, 2024

I had the same issue after upgrading to 2.22.0. Not sure if it's related to the method of attaching the environment or something else. I was able to stop getting this error after I removed the environment and added it back (I am using the Portainer Agent method).

Edit:

After a long power outage from Hurricane Milton, I just got my environment back online and it has started giving this error message again. No idea what caused it to go away before, but I have tried all previous steps and the error persists. It doesn't seem to affect anything other than being an annoyance.

@sholdee
Copy link

sholdee commented Oct 4, 2024

Same errors "Failed to find local environment" here after update from 2.21.2 to 2.22.0, although I am still able to connect to and control my agents. Removing and adding back did not resolve for me.

@XBadFellaX
Copy link
Author

I tried removing the environment and adding it back and that did NOT fix it.

@leleobhz
Copy link

leleobhz commented Oct 4, 2024

Same errors "Failed to find local environment" here after update from 2.21.2 to 2.22.0, although I am still able to connect to and control my agents. Removing and adding back did not resolve for me.

Same here, I do have Docker and Kubernetes.

Also, using enterprise edition.

@ProfessorCha0s
Copy link

ProfessorCha0s commented Oct 4, 2024

Maybe I just got lucky, but I can't think of anything else I did other than removing and then re-adding the environment. I think I tried adding via socket after removing the environment the first time, but it only showed the local node (not all swarm members), so I removed that environment and went back to adding the environment via agent.

I know this is a basic thing to check, but just want to be sure for those of you still having this issue: Did you also make sure to upgrade your Portainer agent to 2.22.0 (not Portainer itself, but the actual agent that runs on all nodes)? Sometimes the :latest tag doesn't pull the actual latest release, so I would make sure to specify the 2.22.0 tag when pulling the image.

@pavelpikta
Copy link

@XBadFellaX removing and adding environment is works for me.
Could please share you configuration ?

@michellabbe
Copy link

Same issue here after upgrading portainer-ce to 2.22.0 (standalone docker hosts only here)

The errors popped every couple mins, but they don't appear in console of portainer-ce or agent.

The errors seem to have stopped after I added local Environment using Socket.
(I usually use Agent instead of Socket for local Environment because Volumes Browse button isn't available via Socket)

I didn't need to remove my existing local Environment (tcp://portainer-agent:9001), just add a new one (unix:///var/run/docker.sock)

I waited a couple mins, then removed the Socket Environment (unix:///var/run/docker.sock) and still no error.

Maybe it's some upgrade task (run once after upgrade) that is hardcoded with unix:///var/run/docker.sock, and after it found the local Environment and run once it stops looping.

@XBadFellaX
Copy link
Author

How does one do this with socket-proxy? I am using Socket proxy instead of ///var/run/docker.sock. I have it added via agent with the URL socket-proxy:2375. Do I need to do something different?

@XBadFellaX
Copy link
Author

`services:

Portainer - WebUI for Containers

portainer:
container_name: portainer
image: portainer/portainer-ee:sts # Use portainer-ee if you have a Business Edition license key
security_opt:
- no-new-privileges:true
restart: unless-stopped
# profiles: ["core", "all"]
networks:
- traefik_proxy
- socket_proxy
# command: -H unix:///var/run/docker.sock # # Use Docker Socket Proxy instead for improved security
command: -H tcp://socket-proxy:2375
ports:
- "9000:9000"
volumes:
# - /var/run/docker.sock:/var/run/docker.sock:ro # # Use Docker Socket Proxy instead for improved security
- $DOCKERDIR/appdata/portainer/data:/data # Change to local directory if you want to save/transfer config locally
environment:
TZ: $TZ
UID: $PUID
GID: $PGID`

Here's my compose.

@troykelly
Copy link

troykelly commented Oct 5, 2024

I completely rebuilt portainer from scratch. Everything was fine with multiple environments for a day.
After restarting portainer the "Failed to find local environment" error started every minute or so.

upgrade portainer: issue manifests
clean install portainer: issue manifests

curl 'https://example.invalid/api/system/info' \
  -H 'accept: application/json, text/plain, */*' \
  -H 'accept-language: en-GB,en;q=0.8' \
  -H 'cookie: portainer_api_key=REDACTED' \
  -H 'priority: u=1, i' \
  -H 'referer: https://example.invalid/' \
  -H 'sec-ch-ua: "Brave";v="129", "Not=A?Brand";v="8", "Chromium";v="129"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-gpc: 1' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36' \
  -H 'x-csrf-token: REDACTED'
  
  {"message":"Failed to get platform","details":"Failed to find local environment"}

@XBadFellaX
Copy link
Author

Anyone know how to add using socket when using socket proxy?

@pavelpikta
Copy link

Anyone know how to add using socket when using socket proxy?

  • add volume with socket to portainer
  • manually add environment via portainer UI
image

@sholdee
Copy link

sholdee commented Oct 8, 2024

In my case, I don't even have a local environment. I am running portainer-ce in Kubernetes and only connecting to a portainer agent to control a standalone docker instance. I do not use it to manage the local K8s environment, nor do I want to. This error doesn't seem to break any functionality, but it is annoying.

@fl0wfr
Copy link

fl0wfr commented Oct 9, 2024

I confirm that after removing all environments and adding them back, I still get the error message.
Agent on the environments has been updated to the same version as Portainer, ie 2.22.0.

@ProfessorCha0s
Copy link

I had mine working and posted what worked for me, but now it has returned.

After a long power outage from Hurricane Milton, I just got my environment back online and it has started giving this error message again. No idea what caused it to go away before, but I have tried all previous steps and the error persists. It doesn't seem to affect anything other than being an annoyance.

@XBadFellaX
Copy link
Author

Updated to 2.23.0 but error does NOT stop.

@Vaslo
Copy link

Vaslo commented Oct 17, 2024

Also having the same issue running portainer through docker compose and only a single environment. This started when I upgraded to 2.23.0. I can't figure out why it's doing this suddenly as all containers seem to be ok.

@Nick-Portainer
Copy link
Collaborator

Hi all, I am going to look into replicating this and raising with the team.

@sholdee
Copy link

sholdee commented Oct 28, 2024

Hi all, I am going to look into replicating this and raising with the team.

The error can be replicated by having at least 1 environment configured and no local environment configured. It was introduced in 6f84317 in guessLocalEnvironment():

return nil, "", errors.New("failed to find local endpoint")

The error text was updated in 80f53ed to the current message:

return nil, "", errors.New("failed to find local environment")

This commit also added a condition to suppress any error when no environments are configured:

return nil, "", nil

Since having no local environment is also valid, presumably we just need to do the same thing and nil the error: #12336

In my case, Portainer runs in Kubernetes and only has a remote agent environment.

@Nick-Portainer
Copy link
Collaborator

internal: BE-11353

@hhromic
Copy link
Contributor

hhromic commented Nov 6, 2024

This issue was originally reported in #8806.
I offered a PR to fix it in #11944 but was superseded by #11728, which got reverted in #12236.

@XBadFellaX
Copy link
Author

That's good information hhromic. I would like to point out though for the portainer team that I am not using swarm and just standalone env, similar to most of the users that reported that same issue.

sholdee added a commit to sholdee/home-ops that referenced this issue Nov 8, 2024
Downgrade until this regression is fixed
portainer/portainer#12285
@XBadFellaX
Copy link
Author

Upgraded to 2.24.0 but the error is still NOT resolved.

@Nick-Portainer any success? The constant error is very annoying.

@andrelung
Copy link

Adding a local (socket) environment fixed it for me. Previously there was just an a agent environment connected.

@sholdee
Copy link

sholdee commented Nov 25, 2024

Adding a local (socket) environment fixed it for me. Previously there was just an a agent environment connected.

This is true, but it should not be necessary to mount a local socket if you only want to manage agent(s).

@ProfessorCha0s
Copy link

It absolutely shouldn't be necessary, but it's only temporary. I got it to work by adding it and then removing it. Then the errors stop, even after the socket environment has been removed.

I started getting the error again after another update, but found that the key is to add via socket, connect to the socket environment, then you can remove the socket environment. I kept the socket environment for about a day before removing it because one of my environments didn't "take" if it wasn't connected for a bit. I think the main thing is to switch to the socket at least once after adding it.

Ultimately, this needs to be fixed. But this is a workaround if you're able to do it in your environment.

@sholdee
Copy link

sholdee commented Nov 25, 2024

My solution is to stick with the LTS version. I can't add a socket environment like that with Kubernetes. Even if I could that would then exceed my license count.

@XBadFellaX
Copy link
Author

Updated to 2.24.1 but the error still does NOT go away. My logs are full of these errors now.

@talltechy
Copy link

Adding a local (socket) environment fixed it for me. Previously there was just an a agent environment connected.

This is true, but it should not be necessary to mount a local socket if you only want to manage agent(s).

100% agree, this worked for me but it uses up a license slot to get rid of that annoying message

at least ad an option to disable the message and update the error message to reference it.

@corgan2222
Copy link

same for me. Before updating to 2.24.1 all was fine. Now it's really annoying. :/

@ProfessorCha0s
Copy link

This issue still exists in the latest 2.25.1 version of Portainer.

@DarkSpir
Copy link

Can confirm. Issue still exists in 2.25.1. I also have portainer-ee running on a K3S kubernetes host so there is no local environment I could configure for this to use. And if I do I'd get a license problem in my current environment. So please fix this issue!

@SpeedySH
Copy link

SpeedySH commented Jan 8, 2025

fix please the issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

17 participants