Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API proxy leads to sandbox 502 and http request time out in self-hosted docker-compose environment #9960

Closed
5 tasks done
Derican opened this issue Oct 28, 2024 · 9 comments
Closed
5 tasks done
Labels
🐞 bug Something isn't working 🤔 cant-reproduce We can not reproduce it or the information is limited

Comments

@Derican
Copy link

Derican commented Oct 28, 2024

Self Checks

  • This is only for bug report, if you would like to ask a question, please head to Discussions.
  • I have searched for existing issues search for existing issues, including closed ones.
  • I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • Please do not modify this template :) and fill in all the required fields.

Dify version

newest from docker image

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

  1. start dify from docker compose as written in docs.

  2. add proxy to api in docker-compose.yaml and restart dify
    image

  3. test code runner in dify
    image

  4. test http request, GET www.google.com
    image

The LLM Module and network on the server is tested ok. And Suggestion in #3079 is already adopted.

✔️ Expected Behavior

for 3, execute code in python script
for 4, got status 200 from response

❌ Actual Behavior

for 3, got the 502 error as the same in #9737
image
for 4, got reached max retries
image

@dosubot dosubot bot added 🐞 bug Something isn't working 🤔 cant-reproduce We can not reproduce it or the information is limited labels Oct 28, 2024
Copy link

dosubot bot commented Oct 28, 2024

Hello, @Derican! I'm here to help you with bug reports, answer questions, and assist you as a contributor. Let's work together to resolve the issue you're facing.

I found some similar issues and discussions that might help you resolve the 502 error and HTTP request timeout in your self-hosted Docker Compose environment:

  1. 502 Error Related to Sandbox Service: A common cause of a 502 error in the Dify self-hosted environment is Nginx forwarding the service to the wrong location. Ensure that the Nginx configuration is updated with the correct IP addresses of the running containers. You can find the IP addresses using the following command:

    docker ps -q | xargs -n 1 docker inspect --format '{{ .Name }}: {{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}'

    Update the Nginx configuration by replacing http://api:5001 and http://web:3000 with the correct IP addresses and ports. After making these changes, restart the Nginx container or reload the configuration [1].

  2. HTTP Request Timeout: The HTTP request utility timing out can be addressed by removing the SSRF_PROXY_HTTP_URL and SSRF_PROXY_HTTPS_URL environment variables from the API container. Additionally, adjusting the GUNICORN_TIMEOUT environment variable to 360 seconds may help prevent timeouts for longer HTTP requests [2].

  3. Proxy Configuration: Ensure that the proxy settings in your Docker Compose file are correctly configured. The default proxy settings are:

    environment:
      HTTP_PROXY: ${SANDBOX_HTTP_PROXY:-http://ssrf_proxy:3128}
      HTTPS_PROXY: ${SANDBOX_HTTPS_PROXY:-http://ssrf_proxy:3128}

    These settings indicate that the default HTTP and HTTPS proxy is http://ssrf_proxy:3128 [3].

By following these steps, you should be able to resolve the 502 error and HTTP request timeout issues. If the problem persists, further investigation into the specific error messages in the logs will be necessary.

To continue talking to Dosu, mention @dosu.

@Derican
Copy link
Author

Derican commented Oct 29, 2024

For anyone who may possibly look into this issue, I provide a PART of solution here. For me, I added http proxy to SSRF in api and worker, and add rules in squid.conf.template to let ssrf access the proxy of the host.
image
image
HOWEVER, while this modification solved the sandbox connection error brought by api proxy, the http request error still persists. I tried replace http_access allow all with rules in the squid config, but still can't make it. Hope anyone can find out the correct way.

@hzhstar
Copy link

hzhstar commented Nov 25, 2024

出现这个 大概率是给容器配置了代理了,要把代理去掉

@Derican
Copy link
Author

Derican commented Nov 27, 2024

出现这个 大概率是给容器配置了代理了,要把代理去掉

但是代理是必须的,无论说的是拉取镜像的代理还是自部署的dify连接OpenAI等服务。所以这个Issue以及相关的大部分Issue讨论都是建立在如何正确地给容器添加代理。可以看到上面我给的让ssrf走代理(收集自CSDN和Discord日本组的相关讨论)一定程度上解决了这个问题(只有http请求节点无法连接)。
But the proxy is necessary, whether it is the proxy that pulls the image or the self-hosted dify to connect to services such as OpenAI. So this issue and most of the related issue discussions are based on how to correctly add a proxy to the container. You can see that the above I gave to let ssrf take the proxy (collected from the relevant discussions of CSDN and Discord Japan group) solves this problem to a certain extent (only the http requesting node cannot connect).

@NiushanDong
Copy link

NiushanDong commented Dec 17, 2024

@Derican
I got succeed by adding upstream proxy server and acl rules in ssrf_proxy/squid.comf.template !!!
image

@Derican
Copy link
Author

Derican commented Dec 17, 2024

@Derican I got succeed by adding upstream proxy server and acl rules in ssrf_proxy/squid.comf.template !!! image

Thanks for the reply and I'll try it out ASAP! :)

@Axaxin
Copy link

Axaxin commented Dec 31, 2024

出现这个 大概率是给容器配置了代理了,要把代理去掉

但是代理是必须的,无论说的是拉取镜像的代理还是自部署的dify连接OpenAI等服务。所以这个Issue以及相关的大部分Issue讨论都是建立在如何正确地给容器添加代理。可以看到上面我给的让ssrf走代理(收集自CSDN和Discord日本组的相关讨论)一定程度上解决了这个问题(只有http请求节点无法连接)。 But the proxy is necessary, whether it is the proxy that pulls the image or the self-hosted dify to connect to services such as OpenAI. So this issue and most of the related issue discussions are based on how to correctly add a proxy to the container. You can see that the above I gave to let ssrf take the proxy (collected from the relevant discussions of CSDN and Discord Japan group) solves this problem to a certain extent (only the http requesting node cannot connect).

  1. LLM平台服务代理问题可以用one-api这类openai compatible凑合解决;
  2. ssrf_proxy配置里面好像有白名单,可以尝试添加sandbox的容器ip到白名单解决;
  3. 我是在docker-compose.yml一开始一大串环境变量里面,注释掉SSRF_PROXY_HTTP_URL就解决本地网络通信问题,HTTPS可以不注释,一些本地http请求有问题也可以在ssrf_proxy配置里加白名单来解决;
  4. docker拉镜像,在终端配置关于代理环境变量就行吧,与容器内的代理无关;

@Derican
Copy link
Author

Derican commented Dec 31, 2024

@Axaxin 感谢回复!

  1. 如果从中转API角度,确实可以解决个人需求,但是作为业务开发不能这么简单处理,况且可能还有像Google等国外服务请求的需求,不能寄希望于都在本地另开服务再调用;
  2. 是的 @NiushanDong 上面给出的配置可以解决这个问题
  3. 我忘记在一开始尝试的时候这个方法是否有用了(当时也是老版本),也是想官方既然给了ssrf的框架,想知道能否在这个框架内一劳永逸地解决;
  4. 嗯这个是我写多余了,抱歉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐞 bug Something isn't working 🤔 cant-reproduce We can not reproduce it or the information is limited
Projects
None yet
Development

No branches or pull requests

5 participants