Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

X-Forwarded-Proto misconfigured error for IPv6-only visitors when running in Docker through Cloudflared tunnel reverse proxy #32994

Open
1 of 6 tasks
pirate opened this issue Jan 10, 2025 · 0 comments
Open
1 of 6 tasks

X-Forwarded-Proto misconfigured error for IPv6-only visitors when running in Docker through Cloudflared tunnel reverse proxy #32994

pirate opened this issue Jan 10, 2025 · 0 comments
Labels
bug

Comments

@pirate
Copy link
Contributor

pirate commented Jan 10, 2025 •
edited
Loading

After upgrading from v7 to v9 we started seeing new errors only when IPv6 visitors try to GET /user_uploads/.* URLs on our Zulip server.

  • we don't see any errors for IPv4 visitors, so it's likely an edge case related to IPv6
  • the errors are reliably triggered anytime an IPv6 visitor tries to GET /user_uploads/.* )(it's possible it happens on other URLs too, but I've only seen errors under that path so far)
  • we are running a standard docker-zulip:9.3 setup on an Ubuntu Digitalocean VPS
  • we are using a Cloudflared ingress tunnel as our reverse proxy / SSL
  • I have read the Zulip Documentation: Reverse Proxies page and confirmed we are doing everything correctly AFAICT, the Cloudflared reverse proxy correctly sets X-Forwarded-For, X-Forwarded-Proto, etc. before sending traffic to the Zulip container
  • our org pays for the basic Zulip self-hosted plan 🥂

https://chat.zulip.org/#narrow/channel/31-production-help/topic/x-forwarded-proto.20in.209.2E3

Config

docker-compose.yml:

services:
    ...
    # stock docker-zulip config for postgres, memcached, rabbitmq, etc.
    ...
    
    zulip:
        image: zulip/docker-zulip:9.3-0
        environment:
            DISABLE_HTTPS: 'True'
            LOADBALANCER_IPS: '192.168.0.0/16,172.0.0.0/8'
            ...
            # let me know if you need to see the rest of our settings

    # cloudflared ingress tunnel, proxied traffic goes to zulip over HTTP w/ X-Forwarded headers set 
    argo:
        image: cloudflare/cloudflared
        command: tunnel --no-autoupdate --protocol http2 --hostname zulip.zervice.io --url http://zulip:80 --name zulip.zervice.io
        # # https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/local/local-management/arguments/
        volumes:
            # download a cert.pem into this folder from https://dash.cloudflare.com/argotunnel
            - ./etc/cloudflared:/etc/cloudflared

etc/zulip/zulip.conf:

[machine]
puppet_classes = zulip::profile::docker
deploy_type = production

[application_server]
http_only = true

[loadbalancer]
ips = 192.168.0.0/16,172.0.0.0/8

Clues

Reverse proxy misconfiguration: No X-Forwarded-Proto header sent from trusted proxy 172.24.0.2

So clearly our LOADBALANCER_IPS setting is correct, yet it's claiming X-Forwarded-Proto header is unset for IPv6 requests, despite cloudflared 100% setting that header before sending traffic to the Zulip Container (verified by swapping out the zulip container with https://github.com/mendhak/docker-http-https-echo and seeing what headers come in).

Here are the only clues I've found in the error output:

# error only seems to happen when client's real IP is an IPv6 IP
HTTP_X_FORWARDED_FOR = '2400:cb00:70:1024::a29e:7dc9'
HTTP_X_REAL_IP = '2400:cb00:70:1024::a29e:7dc9'
REMOTE_ADDR = '2400:cb00:70:1024::a29e:7dc9'

# when client has a IPv6 real ip, HTTP_X_FORWARDED_PROTO ends up unset in django
HTTP_X_FORWARDED_PROTO = ''

REQUEST_SCHEME = 'http'
SERVER_ADDR = '172.24.0.7'
SERVER_PORT = '80'
SERVER_PROTOCOL = 'HTTP/1.1'
wsgi.url_scheme = 'http'

Full Error Output

Here's a sample of one of the errors (let me know if you want the full text / more error logs from similarly failing requests):

Internal Server Error: /user_uploads/thumbnail/4/c5/IrFBJZkeTdbnIXI4svWpXJQB/image.png/840x560.webp

ProxyMisconfigurationError at /user_uploads/thumbnail/4/c5/IrFBJZkeTdbnIXI4svWpXJQB/image.png/840x560.webp
Reverse proxy misconfiguration: No X-Forwarded-Proto header sent from trusted proxy 172.24.0.2; see example configurations in https://zulip.readthedocs.io/en/latest/production/reverse-proxies.html

Request Method: GET
Request URL: http://zulip.monadical.com/user_uploads/thumbnail/4/c5/IrFBJZkeTdbnIXI4svWpXJQB/image.png/840x560.webp
Django Version: 5.0.9
Python Executable: /srv/zulip-venv-cache/d93224eaa47f8f8ab4875fbe3433392459211c73/zulip-py3-venv/bin/uwsgi
Python Version: 3.12.3


Traceback (most recent call last):
 File "/srv/zulip-venv-cache/d93224eaa47f8f8ab4875fbe3433392459211c73/zulip-py3-venv/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
   response = get_response(request)
              ^^^^^^^^^^^^^^^^^^^^^
 File "/srv/zulip-venv-cache/d93224eaa47f8f8ab4875fbe3433392459211c73/zulip-py3-venv/lib/python3.12/site-packages/django/core/handlers/base.py", line 185, in _get_response
   response = middleware_method(

 File "/home/zulip/deployments/2024-11-24-03-10-03/zerver/middleware.py", line 670, in process_view
   raise ProxyMisconfigurationError(proxy_state_header)
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Exception Type: ProxyMisconfigurationError at /user_uploads/thumbnail/4/c5/IrFBJZkeTdbnIXI4svWpXJQB/image.png/840x560.webp
Exception Value: Reverse proxy misconfiguration: No X-Forwarded-Proto header sent from trusted proxy 172.24.0.2; see example configurations in https://zulip.readthedocs.io/en/latest/production/reverse-proxies.html
Raised during: zerver.lib.rest.rest_dispatch
Request information:
USER: redacted@example.com <Realm: redacted 4>

GET: No GET data

POST: No POST data

FILES: No FILES data

django_language = 'en'

META:
CONTENT_LENGTH = ''
CONTENT_TYPE = ''
DOCUMENT_ROOT = '/usr/share/nginx/html'
HTTP_ACCEPT_ENCODING = 'gzip, br'
HTTP_ACCEPT_LANGUAGE = 'en-CA,en-US;q=0.9,en;q=0.8'
HTTP_CDN_LOOP = 'cloudflare; loops=1'
HTTP_CF_RAY = '8ff65986c8dfa308-EWR'
HTTP_CF_WARP_TAG_ID = '3673e589-be4b-459b-b2f8-e97103803123'
HTTP_CONNECTION = 'keep-alive'
HTTP_COOKIE = '********************'
HTTP_HOST = 'zulip.monadical.com'
HTTP_REFERER = 'https://zulip.monadical.com/'
HTTP_SEC_FETCH_DEST = 'image'
HTTP_SEC_FETCH_MODE = 'no-cors'
HTTP_SEC_FETCH_SITE = 'same-origin'
HTTP_USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Safari/605.1.15'
HTTP_X_FORWARDED_FOR = '2400:cb00:70:1024::a29e:7dc9'
HTTP_X_FORWARDED_PROTO = ''
HTTP_X_PROXY_MISCONFIGURATION = 'No X-Forwarded-Proto header sent from trusted proxy 172.24.0.2; see example configurations in https://zulip.readthedocs.io/en/latest/production/reverse-proxies.html'
HTTP_X_REAL_IP = '2400:cb00:70:1024::a29e:7dc9'
PATH_INFO = '/user_uploads/thumbnail/4/c5/IrFBJZkeTdbnIXI4svWpXJQB/image.png/840x560.webp'
QUERY_STRING = ''
REMOTE_ADDR = '2400:cb00:70:1024::a29e:7dc9'
REMOTE_PORT = ''
REQUEST_METHOD = 'GET'
REQUEST_SCHEME = 'http'
REQUEST_URI = '/user_uploads/thumbnail/4/c5/IrFBJZkeTdbnIXI4svWpXJQB/image.png/840x560.webp'
SCRIPT_NAME = ''
SERVER_ADDR = '172.24.0.7'
SERVER_NAME = ''
SERVER_PORT = '80'
SERVER_PROTOCOL = 'HTTP/1.1'
uwsgi.node = b'6567525b066e'
uwsgi.version = b'2.0.28'
wsgi.errors = <_io.TextIOWrapper name=2 mode='w' encoding='UTF-8'>
wsgi.file_wrapper = <built-in function uwsgi_sendfile>
wsgi.input = <uwsgi._Input object at 0x7f64b7c2fc90>
wsgi.multiprocess = True
wsgi.multithread = False
wsgi.run_once = False
wsgi.url_scheme = 'http'
wsgi.version = '(1, 0)'
...

Related Links

Zulip Server and web app version:

  • Zulip Cloud (*.zulipchat.com)
  • Zulip Server 9.x (docker-zulip running behind Cloudflared)
  • Zulip Server 8.x
  • Zulip Server 7.x
  • Zulip Server 6.x or older
  • Other or not sure
@pirate pirate added the bug label Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pirate