Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubefwd is not properly reconnecting when new pod is created #243

Open
imdbere opened this issue Oct 4, 2022 · 5 comments
Open

Kubefwd is not properly reconnecting when new pod is created #243

imdbere opened this issue Oct 4, 2022 · 5 comments

Comments

@imdbere
Copy link

imdbere commented Oct 4, 2022

Hi, first of all thanks for making this project available :)

One thing that makes kubefwd harder to use is that it doesn't properly reconnect if e.g. a deployment is updated and therefore an old pod is terminated and a new one is created. The error that is created looks like this:

ERRO[17:36:11] Runtime: an error occurred forwarding 80 -> 80: error forwarding port 80 to pod 1b4e23d5467af287d3685ede5f693f04b5a909d475cb31d470381f6c3c956ffe, uid : failed to find sandbox "1b4e23d5467af287d3685ede5f693f04b5a909d475cb31d470381f6c3c956ffe" in store: not found

After that, only restarting the command makes the port-forwarding work again.
Is this intended behaviour ? It would be ideal if kubefwd would properly shift traffic to the new pod like it is done by kubernetes services or at least doing some reconnecting.

This is the command used:
sudo kubefwd svc -n default -d some-name -x some-kubeconfig

Thanks!
@ndj888
Copy link
Contributor

ndj888 commented Oct 6, 2022

@imdbere kubefwd checks the pod list every 5 minutes and will fix this automatically. I don't think it's necessary for kubefwd to monitor pod changes and react in real time, it will consume more resources. It is recommended to use kubefwd for local development environment. If stable access to pods inside the cluster is required, I recommend using a private load balancer for service exposure.

@isurulucky isurulucky mentioned this issue Oct 25, 2022
Open
@kikonen
Copy link

kikonen commented Nov 17, 2022 •
edited
Loading

5 min delay in reconnecting effectively means that user must stop kubefwd and restart it, i.e. waiting 5 mins is long time to wait when trying to do some development work locally.

@gustaff-weldon
Copy link

@ndj888 this is a tool for local development isn't it?

When service gets redeployed in the cluster you're connected to, you want your local app to pick this up. 5 min delay makes reconnect virtually non-functional. For me, 5 seconds would be more appropriate as I want to focus on app I work on, and let kubefwd do its job :)

Idea: why not just make timeout a CLI option and let users decide what is best resource/delay combo?

@gustaff-weldon
Copy link

gustaff-weldon commented Nov 18, 2022 •
edited
Loading

Btw, I'm experiencing very similar problem. The moment pod gets reshuffled or new deployment happens, the port forwarding dies and does not recover (even after 5 minutes), see below:

❯ sudo kubefwd svc -v --namespace postgresql -f metadata.name=postgresql
Password:
INFO[15:32:08]  _          _           __             _
INFO[15:32:08] | | ___   _| |__   ___ / _|_      ____| |
INFO[15:32:08] | |/ / | | | '_ \ / _ \ |_\ \ /\ / / _  |
INFO[15:32:08] |   <| |_| | |_) |  __/  _|\ V  V / (_| |
INFO[15:32:08] |_|\_\\__,_|_.__/ \___|_|   \_/\_/ \__,_|
INFO[15:32:08]
INFO[15:32:08] Version 1.22.4
INFO[15:32:08] https://github.com/txn2/kubefwd
INFO[15:32:08]
INFO[15:32:08] Press [Ctrl-C] to stop forwarding.
INFO[15:32:08] 'cat /etc/hosts' to see all host entries.
INFO[15:32:08] Loaded hosts file /etc/hosts
INFO[15:32:08] HostFile management: Original hosts backup already exists at /Users/kolo/hosts.original
INFO[15:32:08] Successfully connected context: docker-desktop
DEBU[15:32:08] Registry: Start forwarding service postgresql.postgresql.docker-desktop
DEBU[15:32:08] Resolving: postgresql to 127.1.27.1 (postgresql)
INFO[15:32:08] Port-Forward:       127.1.27.1 postgresql:5432 to pod postgresql-0:5432
ERRO[15:34:39] Runtime: an error occurred forwarding 5432 -> 5432: error forwarding port 5432 to pod c9a9c7e84ba15c880fd6ce333089cf1022519ba2ca890a44e7afd9172f1195da, uid : container not running (c9a9c7e84ba15c880fd6ce333089cf1022519ba2ca890a44e7afd9172f1195da)
ERRO[15:34:39] Runtime: lost connection to pod
WARN[15:34:39] Stopped forwarding pod postgresql-0 for postgresql.postgresql.docker-desktop
INFO[15:40:10] update service postgresql/postgresql.

@dobesv
Copy link
Contributor

dobesv commented Aug 31, 2023

I've been having the same problem since I first started using kubefwd years ago, I have to restart it a lot. I even tried putting in a fix that was merged, but the issue resurfaced again later.

e.g. #153

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@dobesv @gustaff-weldon @kikonen @imdbere @ndj888