Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Funnel works only within tailnet #11849

Open
marcinjahn opened this issue Apr 23, 2024 · 16 comments
Open

Funnel works only within tailnet #11849

marcinjahn opened this issue Apr 23, 2024 · 16 comments
Labels
bug Bug funnel Relating to Tailscale Funnel https://tailscale.com/blog/introducing-tailscale-funnel/ L2 Few Likelihood P2 Aggravating Priority level T5 Usability Issue type

Comments

@marcinjahn
Copy link

What is the issue?

It seems to me that funnel and serve do the same thing.
Both allow me to serve content, which is then available via HTTPS only within my tailnet. As I understand, funnel should allow me to publish content publicly, on internet. However, my funneled resource works only when I'm connected to my tailnet.

Steps to reproduce

  1. On Device A, Run some web app locally on one of the nodes, e.g. via podman run -d -p 8080:80 tutum/hello-world
  2. On Device A, Enable funnel: tailscale funnel 8080
  3. On another Device B (not connected to your tailnet), try to navigate to Device A's DNS address.
  4. Page will not be found

Are there any recent changes that introduced the issue?

Nope, it never worked for me

OS

Linux, iOS

OS version

Fedora 39, iOS 17.4.1

Tailscale version

1.64.0

Other software

No response

Bug report

BUG-e75df7e9a917df61bd3b4c6244e8924e06db465d45115a81747489e4fae37b8c-20240423153805Z-00750e64e97bc4fe

@kelivel kelivel added L2 Few Likelihood P2 Aggravating Priority level T5 Usability Issue type funnel Relating to Tailscale Funnel https://tailscale.com/blog/introducing-tailscale-funnel/ and removed needs-triage labels Apr 26, 2024
@jimmybrancaccio
Copy link

jimmybrancaccio commented May 7, 2024

EDIT: This may have something to do with userspace networking. I setup a basic funnel for a directory using sudo tailscale funnel /home/jimmy/tmp/ and that works fine. I can visit that in my web browser from any device, connected to my tailnet or not. I can see the 3-4 files I have in that directory.

--

I was able to replicate this today due to an incoming support ticket (TSS-37882). I setup a basic docker-compose.yml:

version: "3.7"
services:
  ts-dillinger-webserver:
    image: tailscale/tailscale:latest
    container_name: ts-dillinger
    hostname: dillinger
    environment:
      - TS_AUTHKEY=tskey-client-kMq9ax11CNTRL-gGBoK58SWxnGj
      - TS_EXTRA_ARGS=--advertise-tags=tag:container
      - TS_SERVE_CONFIG=/config/dillinger.json
      - TS_STATE_DIR=/var/lib/tailscale
    volumes:
      - /home/jimmy/.docker/data/ts-dillinger-webserver/state:/var/lib/tailscale
      - /home/jimmy/.docker/data/ts-dillinger-webserver/config:/config
      - /dev/net/tun:/dev/net/tun
    cap_add:
      - net_admin
      - sys_module
    restart: unless-stopped
  ts-dillinger:
    image: lscr.io/linuxserver/dillinger:latest
    container_name: dillinger
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
    volumes:
      - /home/jimmy/.docker/data/dillinger/config:/config
    network_mode: service:ts-dillinger-webserver
    depends_on:
      - ts-dillinger-webserver
    restart: unless-stopped

In my SERVE file I have:

{
  "TCP": {
    "443": {
      "HTTPS": true
    }
  },
  "Web": {
    "${TS_CERT_DOMAIN}:443": {
      "Handlers": {
        "/": {
          "Proxy": "http://127.0.0.1:8080"
        }
      }
    }
  },
  "AllowFunnel": {
    "${TS_CERT_DOMAIN}:443": true
  }
}

I ran docker compose up -d --remove-orphans and everything looks to have started up fine. I observed these couple of lines which seems to indicate the funnel is running:

ts-dillinger  | 2024/05/07 20:34:15 serve: closing idle connections to http://127.0.0.1:8080
ts-dillinger  | boot: 2024/05/07 20:34:15 Applying serve config
ts-dillinger  | 2024/05/07 20:34:15 serve: creating a new proxy handler for http://127.0.0.1:8080

I see my container show up in my admin console, however I observed in the admin console, clicking on the machine to go to the machine page, under the TLS Configuration section the domain shows properly, but the Status shows as No certificate found. I observed that when trying to visit https://dillinger.taildfbdd.ts.net failed and I also couldn't resolve the hostname.

I then swapped from a different tailnet I was on to the tailnet where I had deployed the container node and I tried bringing up https://dillinger.taildfbdd.ts.net. At this point I observed in the Docker container logs:

ts-dillinger  | 2024/05/07 20:43:39 cert("dillinger.taildfbdd.ts.net"): registered ACME account.
ts-dillinger  | 2024/05/07 20:43:39 cert("dillinger.taildfbdd.ts.net"): starting SetDNS call...
ts-dillinger  | 2024/05/07 20:43:50 cert("dillinger.taildfbdd.ts.net"): did SetDNS
ts-dillinger  | 2024/05/07 20:43:51 cert("dillinger.taildfbdd.ts.net"): requesting cert...
ts-dillinger  | 2024/05/07 20:43:52 cert("dillinger.taildfbdd.ts.net"): got cert
ts-dillinger  | 2024/05/07 20:43:52 Accept: TCP{100.65.211.27:65338 > 100.80.14.102:443} 52 tcp non-syn
ts-dillinger  | 2024/05/07 20:43:52 http: TLS handshake error from 100.65.211.27:65338: EOF
dillinger     | 100.65.211.27 - GET / HTTP/1.1 200 - - 329.889 ms
dillinger     | 100.65.211.27 - GET /assets/app-9a52b028cbd397404e2c56df7c2f8e87.css HTTP/1.1 200 72955 - 7.123 ms
dillinger     | 100.65.211.27 - GET /img/icons/file.svg HTTP/1.1 200 - - 12.122 ms
dillinger     | 100.65.211.27 - GET /js/main.bundle.js HTTP/1.1 200 - - 20.965 ms
dillinger     | 100.65.211.27 - GET /theme-github.js HTTP/1.1 404 154 - 27.085 ms
dillinger     | 100.65.211.27 - GET /img/icons/code.svg HTTP/1.1 200 902 - 11.722 ms
dillinger     | 100.65.211.27 - GET /img/icons/enter-zen.svg HTTP/1.1 200 - - 13.855 ms
ts-dillinger  | 2024/05/07 20:44:02 Accept: TCP{100.65.211.27:65339 > 100.80.14.102:443} 107 tcp non-syn

so it seems clear that just trying to visit the service triggered a request for a TLS certificate. I had waited about 15 minutes before trying this as our documentation indicates it should happen in about 10 minutes but didn't for me.

In summary, I experienced the same issue as described in the original post as well as is the customer who submitted a recent support ticket. Funnel is working more like Serve in that you cannot reach the funneled service if you're not connected to the tailnet where the funnel is running.

@tommyboylab
Copy link

tommyboylab commented May 8, 2024

EDIT: Took 2 days to populate the DNS but seems to be working now, removing comment

@jimmybrancaccio
Copy link

jimmybrancaccio commented May 23, 2024

I seem to have hit this again but when trying to expose a TCP port.

  1. Setup netcat to listen on a port - netcat -l 7070
  2. Setup a funnel to expose that port - sudo tailscale funnel --bg --tcp 10000 tcp://127.0.0.1:7070
  3. Attempt to use OpenSSL to connect to that port - openssl s_client -4 -connect machine-name.tailnet-name.ts.net:10000 and observe it's unable too.
openssl s_client -4 -connect machine-name.tailnet-name.ts.net:10000

Connecting to 209.177.145.137
CONNECTED(00000003)
008C450202000000:error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_layer_s3.c:692:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 349 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

I ran an strace and it just sits there trying to connect:

rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
connect(3, {sa_family=AF_INET, sin_port=htons(7071), sin_addr=inet_addr("209.177.145.137")}, 16) = -1 ETIMEDOUT (Connection timed out)
rt_sigaction(SIGALRM, {sa_handler=SIG_IGN, sa_mask=[ALRM], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f4c9e6bda70}, {sa_handler=SIG_IGN, sa_mask=[ALRM], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f4c9e6bda70}, 8) = 0
alarm(0)                                = 0
close(3)                                = 0
close(-1)                               = -1 EBADF (Bad file descriptor)
exit_group(1)                           = ?
+++ exited with 1 +++

I can however connect to the endpoint if I am connected to the same tailnet as where the funnel is running:

openssl s_client -4 -connect machine-name.tailnet-name.ts.net:10000

Connecting to 100.127.189.125
CONNECTED(00000003)

Of note, you can see it's using the Tailscale IP address of the device, and not the public IP address.

@brian316
Copy link

i followed this video here from tailscale https://www.youtube.com/watch?v=tqvvZhGrciQ
and used the following docker compose file

---
version: "3.7"
services:
  ts-homeassistant:
    image: tailscale/tailscale:latest
    container_name: ts-homeassistant
    hostname: homeassistant
    environment:
      - TS_AUTHKEY=tskey-auth-removed-for-obvious-reasons
      - "TS_EXTRA_ARGS=--advertise-tags=tag:container --reset"
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_SERVE_CONFIG=/config/serve-config.json
      - TS_USERSPACE=false
    volumes:
      - ${PWD}/tailscale-nginx/state:/var/lib/tailscale
      - /dev/net/tun:/dev/net/tun
      - ${PWD}/serveconfig:/config
    cap_add:
      - net_admin
      - sys_module
      - sys_admin
    restart: unless-stopped
    privileged: true
    network_mode: host
  homeassistant:
    image: ghcr.io/home-assistant/home-assistant:latest
    container_name: homeassistant
    environment:
      - TZ=America/Los_Angeles
    volumes:
      - /home/homeassistant:/config
      - /run/dbus:/run/dbus:ro
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    privileged: true
    depends_on:
      - ts-homeassistant
    network_mode: service:ts-homeassistant

volumes:
  ts-homeassistant:
    driver: local

and this is my serve-config.json

{
    "TCP": {
      "443": {
        "HTTPS": true
      }
    },
    "Web": {
      "${TS_CERT_DOMAIN}:443": {
        "Handlers": {
          "/": {
            "Proxy": "http://127.0.0.1:8123"
          }
        }
      }
    },
    "AllowFunnel": {
      "${TS_CERT_DOMAIN}:443": false
    }
  }

im not even using the funnel yet and i get an error for acme request.

acme: ARI check failed: failed to fetch renewal info from ACME server: fetching renewal info: 400 urn:ietf:params:acme:error:malformed: While parsing ARI CertID an error occurred :: Invalid path; falling back to expiry-based check

@gglanzani
Copy link

For me the issue was that I had tailnet also installed on the host. Once removed, things started working.

Weird, though, that everything seemed to be running along happy while fundamentally broken.

@JanickGers85
Copy link

I've followed the guide https://www.youtube.com/watch?v=tqvvZhGrciQ too, but I can't still reach the public exposed https service without being connected to tailnet (get a DNS_PROBE_FINISHED_NXDOMAIN from Chrome).

I have funnel attribute setted up and HTTPS Certificates enabled.

This is a part of my docker compose

  mealie:
    image: ghcr.io/mealie-recipes/mealie:v1.9.0
    container_name: mealie
    restart: unless-stopped
    network_mode: service:mealie-tailscale
    deploy:
      resources:
        limits:
          memory: 1000M
    volumes:
      - ./docker/mealie:/app/data/
    environment:
      ALLOW_SIGNUP: true
      PUID: 1000
      PGID: 1000
      TZ: Europe/Rome
      MAX_WORKERS: 1
      WEB_CONCURRENCY: 1
      BASE_URL: https://***.ts.net
    depends_on:
      - mealie-tailscale
  mealie-tailscale:
    image: tailscale/tailscale:latest
    container_name: mealie-tailscale
    hostname: mealie-tailscale
    environment:
      - TS_AUTHKEY=tskey-client-***?ephemeral=false
      - "TS_EXTRA_ARGS=--advertise-tags=tag:container --reset"
      - TS_SERVE_CONFIG=/config/mealie.json
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_USERSPACE=false
    volumes:
      - ./docker/mealie-tailscale/state:/var/lib/tailscale
      - ./docker/mealie-tailscale/config:/config
      - /dev/net/tun:/dev/net/tun
    cap_add:
      - net_admin
      - sys_module
    restart: always
    ports:
      - 9001:9000

And this is the config file content

{
  "TCP": {
    "443": {
      "HTTPS": true
    }
  },
  "Web": {
    "${TS_CERT_DOMAIN}:443": {
      "Handlers": {
        "/": {
          "Proxy": "http://127.0.0.1:9000"
        }
      }
    }
  },
  "AllowFunnel": {
    "${TS_CERT_DOMAIN}:443": true
  }
}

tailscale serve status output (ran on tailscale container)

/ # tailscale serve status

# Funnel on:
#     - https://***.ts.net

https://***.ts.net (Funnel on)
|-- / proxy http://127.0.0.1:9000

tailscale netcheck output (ran on tailscale container)

/ # tailscale netcheck

Report:
        * UDP: true
        * IPv4: yes, ***.***.***.***:******
        * IPv6: no, but OS has support
        * MappingVariesByDestIP: false
        * PortMapping:
        * Nearest DERP: Frankfurt
        * DERP latency:
                - fra: 36.7ms  (Frankfurt)
                - lhr: 38.5ms  (London)
                - ams: 44.3ms  (Amsterdam)
                - par: 46.6ms  (Paris)
                - mad: 55.5ms  (Madrid)
                - waw: 56.4ms  (Warsaw)
                - nyc: 113.3ms (New York City)
                - tor: 127.1ms (Toronto)
                - ord: 130.9ms (Chicago)
                - dbi: 136.6ms (Dubai)
                - mia: 142ms   (Miami)
                - dfw: 142.3ms (Dallas)
                - den: 167.6ms (Denver)
                - sfo: 168ms   (San Francisco)
                - blr: 169.3ms (Bangalore)
                - lax: 171ms   (Los Angeles)
                - jnb: 179.6ms (Johannesburg)
                - sea: 190.8ms (Seattle)
                - hnl: 226.2ms (Honolulu)
                - sao: 242ms   (São Paulo)
                - sin:         (Singapore)
                - syd:         (Sydney)
                - tok:         (Tokyo)
                - hkg:         (Hong Kong)
                - nai:         (Nairobi)

@jimmybrancaccio
Copy link

Can confirm the Mealie section in our Docker blog article isn't working due to the fact that you can't access the hostname which is generated. I've followed the article to a T again today and cannot access the Mealie instance while not being on the tailnet.

From docker exec -it ts-mealie tailscale status:

# Funnel on:
#     - https://mealie.taildfbdd.ts.net

Bringing that URL up in Chrome fails with ERR_NAME_NOT_RESOLVED.

@adriancaruana
Copy link

I also seem to be having the same issue where I can access the container-running funnels on other devices that are connected to my tailnet, but not from the internet. DNS lookups for my MagicDNS name fail, yet the "Machines" tab in my tailscale web admin console, and the command tailscale funnel status when run inside my funnel-running container show that the funnel is supposedly working just fine. I'm also getting ERR_NAME_NOT_RESOLVED in Chrome when trying to access the funnel from the internet.

@nkbooth
Copy link

nkbooth commented Jul 6, 2024

I spent several hours with this annoying me today, and I think I finally solved it for myself.

I made lots of changes, but I think that there are 2 things that fully control outside access via funnel:

In the documentation, there is a config.json file to set up funnel. The last blob is:

"AllowFunnel": {
    "${TS_CERT_DOMAIN}:443": false
  }

The key value should be true, not false.

Secondly, ACLs control publishing funnel data to DNS. Without correct ACL setup, the DNS records will not be updated/set for the funnel, which is what is driving server not found errors - they're correct. In the ACL, we need to edit the nodeAttrs section to add permission for the tag used on containers to use funnel. In my case, I'm using the (very original) tag containers, so my ACL looks like this:

"nodeAttrs": [
		{"target": ["tag:containers"], "attr": ["funnel"]}
	],

Hopefully that helps someone else down the road.

@adriancaruana
Copy link

adriancaruana commented Jul 6, 2024

Oh, wow, thank you @nkbooth, your second point did it for me.

Here's my nodeAttrs before (not working) and after (working):

	"nodeAttrs": [
		{
-                       "target": ["autogroup:member"],
+                       "target": ["autogroup:member", "tag:container"],
			"attr":   ["funnel"],
		},
	],

@jimmybrancaccio
Copy link

Welp, I feel like a dummy! You're totally right @nkbooth! My nodeAttr was missing the tag for my containers. Once I added that a DNS record and the certificate were properly fetched and setup. 🤦🏼

@hrstoyanov
Copy link

hrstoyanov commented Jul 9, 2024

Is it possible to get rid of the hassle of using/maintaining of a json config file, and enable funnel via command line args only, in your compose file or image build file?

My AI friend thinks this will work (I doubt):

Dockerfile:

FROM tailscale/tailscale:latest

# Install necessary packages
RUN apk add --no-cache iptables ip6tables

# Set up environment variables
ENV TS_AUTHKEY=your-tailscale-auth-key
ENV TS_HOSTNAME=your-funnel-hostname
ENV TS_EXTRA_ARGS="--advertise-exit-node"

# Expose the port you want to funnel (e.g., 80 for HTTP)
EXPOSE 80

# Start Tailscale and enable Funnel
CMD ["sh", "-c", "tailscaled --tun=userspace-networking --socks5-server=localhost:1055 & \
    until tailscale up --authkey=$TS_AUTHKEY --hostname=$TS_HOSTNAME $TS_EXTRA_ARGS; do sleep 0.1; done; \
    tailscale funnel 80 on; \
    sleep infinity"]

Docker compose:

version: '3'
services:
  tailscale-funnel:
    build: .
    cap_add:
      - NET_ADMIN
      - NET_RAW
    environment:
      - TS_AUTHKEY=your-tailscale-auth-key
      - TS_HOSTNAME=your-funnel-hostname
    volumes:
      - /dev/net/tun:/dev/net/tun
    ports:
      - "80:80"
    restart: unless-stopped

@derrickrc
Copy link

derrickrc commented Oct 2, 2024

I am currently having the same issue with a Tailscale Funnel I just created on a Linux machine yesterday. I can access the funnel on my Mac which is on the same Tailnet, but only using the http address, e.g:

http://<my tailnet url>.ts.net:<port>/endpoint

But the https full domain just results in a <url> is currently unable to handle this request. HTTP ERROR 502

I have an identical Linux machine in my Tailnet located offsite where I am able to access the https funnel, as well as other machines.

Edit: TS support got back to me, I had specified the wrong port when creating the tunnel. My mistake.

@EDIflyer
Copy link

EDIT: Took 2 days to populate the DNS but seems to be working now, removing comment

Mine thankfully didn't take that long but it did take about 15 min to become active - your post was very helpful though in recognising that might be the issue, I was getting confused if I'd done something wrong as it was showing on the tailnet OK but not via the public internet.

@Marerc
Copy link

Marerc commented Nov 17, 2024

Oh, wow, thank you @nkbooth, your second point did it for me.

Here's my nodeAttrs before (not working) and after (working):

	"nodeAttrs": [
		{
-                       "target": ["autogroup:member"],
+                       "target": ["autogroup:member", "tag:container"],
			"attr":   ["funnel"],
		},
	],

This is a soooo wicked parameter.

@jeremymeyers
Copy link

I am currently having the same issue with a Tailscale Funnel I just created on a Linux machine yesterday. I can access the funnel on my Mac which is on the same Tailnet, but only using the http address, e.g:

http://<my tailnet url>.ts.net:<port>/endpoint

But the https full domain just results in a <url> is currently unable to handle this request. HTTP ERROR 502

I have an identical Linux machine in my Tailnet located offsite where I am able to access the https funnel, as well as other machines.

Edit: TS support got back to me, I had specified the wrong port when creating the tunnel. My mistake.

Could you detail what you did wrong and what fixed it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bug funnel Relating to Tailscale Funnel https://tailscale.com/blog/introducing-tailscale-funnel/ L2 Few Likelihood P2 Aggravating Priority level T5 Usability Issue type
Projects
None yet
Development

No branches or pull requests