Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support HTTP/2 #416

Open
4 tasks done
NghiaTranUIT opened this issue Feb 26, 2020 · 65 comments
Open
4 tasks done

Support HTTP/2 #416

NghiaTranUIT opened this issue Feb 26, 2020 · 65 comments
Assignees
Labels
feature It's new feature

Comments

@NghiaTranUIT
Copy link
Member

NghiaTranUIT commented Feb 26, 2020

🐶 Brief

At the moment, all intercepted HTTP/HTTPS through Proxyman will be downgraded to HTTP/1.1. We should support HTTP/2 too and implement the HTTP/2 parser correctly.

👑 Criteria

  • Support h2 ALPN in CocoaAsyncSocket
  • Determine whether or not the request is HTTP/2
  • Write HTTP/2 Parser. We could inspired from Apple-Server source
  • Work with current architecture. HTTP/1.x and HTTP/2

Update 1 (5 Feb 2023): Beta build for HTTP/2 is ready ✅

Comment: #416 (comment)

Update 2 (14 Feb 2023): New Update

Comment: #416 (comment)

Update 3 (22 Feb 2023):

#416 (comment)

@NghiaTranUIT NghiaTranUIT added the feature It's new feature label Feb 26, 2020
@NghiaTranUIT NghiaTranUIT self-assigned this Feb 26, 2020
@Saklad5
Copy link

Saklad5 commented Apr 13, 2020

Any progress on this?

@NghiaTranUIT
Copy link
Member Author

@Saklad5 Sorry, but at the moment, I'm working on Protobuf feature.

When it's done, I will kick start the HTTP/2 since it's crucial to support gRPC feature too 🙌

@Saklad5
Copy link

Saklad5 commented Apr 29, 2020

Glad to see the Protobuf feature is done: it’s not something I’m interested in, but it means you might work on this soon.
Once Proxyman can run without downgrading traffic, I think I’ll purchase it. Until then, I can’t really justify running it for long periods of time, which limits its utility.

@NghiaTranUIT
Copy link
Member Author

I would like to say that HTTP/2 and HTTP/3 are must-have features of Proxyman, and the progress is really good at the moment 🙌

Protobuf, the first blocker is done, it's time for HTTP/2.

Stay tuned 🌮

@Saklad5
Copy link

Saklad5 commented May 2, 2020

HTTP/3 isn’t formally accepted yet, so I’d say that’s less of a must-have than HTTP/2. I generally don’t expect things to support draft standards, and I understand it is pretty different.

By the way, does Proxyman do any other downgrading at the moment? Besides forcing HTTP/1, does simply running it affect outgoing traffic at all?

@NghiaTranUIT
Copy link
Member Author

At the moment, only HTTPS domains, which enable SSL-Proxying (To see the HTTP Response Content) is downgraded to HTTP/1.

Otherwise, HTTPS (no SSL-Proxying) and HTTP are not, since it's just a proxy server, no interception. Thus, it would keep HTTP/2 or other protocols without changing it.

As there are no reliable HTTP/2 servers on macOS (Swift/Objc) yet, so it's the reason why Proxyman doesn't support HTTP/2 at the first implementation. However, Swift-NIO from Apple is fully supported recently, so I'm planning to rewrite the Proxyman Core with NIO stack.

When 1.23.0 build is out this weekend, I will push HTTP/2 forward 👍

@wasder
Copy link

wasder commented Nov 6, 2020

Looking forward for gRPC support.
Could be a killer feature, as nothing yet so handy support it.👍

@bbuckland
Copy link

@NghiaTranUIT sounds like you've made a ton of progress towards this. Could you share where you're at with HTTP/2, specifically to enable gRPC?

@NghiaTranUIT
Copy link
Member Author

@bbuckland We're migrating the codebase to HTTP/2 but it hasn't finished yet. I will push it forward soon after the Multiple Filters and Diffing is done 🙌

To workaround and see the gRPC data, you might consider using Atlantis from Proxyman (https://github.com/ProxymanApp/atlantis#2-my-app-use-grpc)

@inickt
Copy link

inickt commented Jun 30, 2021

Would love to know if there are any updates on this, our backend microservices only talk over HTTP/2 and I would love to use Proxyman + the beta of the reverse proxy feature to improve my backend workflow.

That being said I switched over from Charles this week and it has been amazing for testing iOS things!

@NghiaTranUIT
Copy link
Member Author

Thanks for the upvote @inickt. Since the Reverse Proxy is almost done (we're going to release it in this week), so our team is working on the HTTP/2 👍

We will send a Beta build here when it's available 😄

@inickt
Copy link

inickt commented Sep 30, 2021

Hey, I was curious if there have been any updates on this? Still been loving the app so thanks for all of your work and the great updates!

@NghiaTranUIT
Copy link
Member Author

Hey @inickt Thanks for your shoutout.

To be honest, I tried to support HTTP/2 months ago but it doesn't work well since it's a breaking change in our codebase 😿 . The problem is not just supporting HTTP/2, it also includes HTTP/1.1 as backward compatible.

At the moment, I'm focusing on the Folder Rule (where you can organize the tool rules in folders). When it's done, I will back to HTTP/2 👍

@NicolasCombe5555
Copy link

Hello @NghiaTranUIT 👍🏻; could we expect http2 support any time soon or not likely to happen soon? Thanks 🤝

@NghiaTranUIT
Copy link
Member Author

Sorry @NicolasCombe5555 This ticket is implemented but it's delayed due to technical problems. Basically, in order to support HTTP/2, we have to redesign the ProxymanCore, which is primarily designed for HTTP/1.1. Hence, we decide to postpone this ticket until we found a better solution.

Can you elaborate on why you need HTTP/2? Does it important for your work? @NicolasCombe5555 🤔

@NicolasCombe5555
Copy link

NicolasCombe5555 commented Jul 3, 2022

@NghiaTranUIT Well part of my project has http/2 in our tech stack; so I cannot see that traffic when using Proxyman. But I understand if we are having issues with supporting it here; thanks for the quick response tho 👍🏻

@eseay
Copy link

eseay commented Jul 15, 2022

I've read this thread and understand the complexity surrounding adding HTTP/2 support, but I just wanted to more or less throw a "+1" onto this issue as someone who would like to see it supported.

In the meantime, I am just using mitmproxy to view any traffic from my application where HTTP/2 is required.

@NghiaTranUIT to give you some context about the HTTP/2 need - we're starting to use more gRPC across my teams, and since that's HTTP/2, we're not able to proxy those requests using Proxyman. In fact, having Proxyman open when those requests go out essentially breaks them. (I work on the iOS apps, and am debugging that app's traffic to our servers)

Thanks for your work on such a great tool!

@NghiaTranUIT
Copy link
Member Author

Even Proxyman can support HTTP/2 and Protobuf, but there is no guarantee that it can capture and properly decrypt the gRPC package because we don't support it yet.

To workaround, @eseay you can use Atlantis, which allows you to capture gPRC traffic 👍 https://github.com/ProxymanApp/atlantis#2-my-app-use-grpc

@eseay
Copy link

eseay commented Jul 16, 2022

@NghiaTranUIT well I'll clarify that the back-end is serving a gRPC hybrid called Connect, so we're actually just using POST requests with normal request bodies, but those requests require HTTP/2.

In either case - I am curious what I may be doing wrong in Proxyman to not be seeing HTTP/2 traffic.

When I run mitmproxy, I see that all of my requests (both gRPC-Connect and classic REST) are being conducted via HTTP/2; I see the same thing when proxying through Charles.

In Proxyman, when I go to inspect my requests, they are all showing as using HTTP/1.1. Is there something I may have configured incorrectly in Proxyman that's preventing the HTTP/2 transmissions?

@j-j-m
Copy link

j-j-m commented Oct 12, 2022

I see the same thing. Is there any way to prevent Proxyman from breaking my gRPC services in app? I have tried Atlantis with no luck.

@NghiaTranUIT
Copy link
Member Author

@j-j-m can you use this Atlantis code to capture gRPC traffic? https://github.com/ProxymanApp/atlantis#2-my-app-use-grpc

and open Tools Menu -> SSL Proxying List -> Remove the gRPC domain in the Include List. It will fix the SSL Error, and allows the Atlantis to work properly 👍

@NghiaTranUIT
Copy link
Member Author

NghiaTranUIT commented Feb 5, 2023

Good news 🎉 🎉

I've supported the HTTP/2 protocol (Beta) for the Proxyman app 🎉

Beta build: https://proxyman.s3.us-east-2.amazonaws.com/beta/Proxyman_4.1.0_Support_HTTP_2.dmg

What's new

  • Support HTTP/2 and HTTP/1.1 (can negotiate during the SSL Handshake)
  • Tested and HTTP/2 work with the following debugging tools:
  1. Breakpoint
  2. Map Local
  3. Scripting
  4. Map Remote
  5. Compose & Edit
  6. Repeat
  7. Block / Allow List
  8. External Proxy
  9. A lot of Unit Tests to cover all cases

Notes

  • In the HTTP Message (Breakpoint or Map Local), you can use HTTP/1.1 or HTTP/2. It doesn't matter since Proxyman will use the correct HTTP Protocol behind the scenes => It means, you can use your current Map Local Rule/ Breakpoint Raw Message without any problems ✅

Screenshots

Proxyman HTTP/2

How to enable HTTP/2

  • Open Preference -> Check "Use HTTP/2" -> Restart the app to take effect.

Screenshot_2023-02-05_at_20_57_11

Known issues:

  • WSS doesn't work over HTTP/2 (WS still works as usual)
  • ???

If you guys find any bugs, please let me know, I'm all ears to fix it 👍


Just a friendly tag: @j-j-m @eseay @NicolasCombe5555 @inickt @bbuckland @Saklad5, @Ashraf-Ali-aa, @julasamer, @wasder, @ildar-gilfanov, @NSMyself, @heyzooi, @inickt, @nxtSwitch, @basecde, water-a

@derekmurawsky
Copy link

Tested it out, and it looks like it works to me! I was able to request an image where, previously, the connection was downgraded to 1.1. Thank you!

@brr53
Copy link

brr53 commented Feb 5, 2023

Thank you. There is a bug though. HTTP2 does not have the host header. So this RAW is incorrect. HTTP2 heading should look something like this.

:method: GET
:authority: www.google.com
:scheme: https
:path: /

Screen Shot 2023-02-05 at 1 48 18 PM

@NghiaTranUIT

@NghiaTranUIT
Copy link
Member Author

Thanks @brr53. It's just a UI bug. Behind the sense, Proxyman automatically translates the HTTP/1.1 Raw format into the HTTP/2 format, so it's still working.

I'm going to fix the Raw Tab for HTTP/2 now 👍

@NghiaTranUIT
Copy link
Member Author

Update 9:

@djtarazona
Copy link

djtarazona commented Jun 11, 2023

Any updates or ETA on HTTP/2 support @NghiaTranUIT? 🙏

@djtarazona
Copy link

Bump @NghiaTranUIT 🙏

@NghiaTranUIT
Copy link
Member Author

NghiaTranUIT commented Jul 30, 2023

@djtarazona Sorry for the late reply. It's still in BETA.

Here is the v10 Update: https://download.proxyman.io/beta/Proxyman_4.9.0_HTTP_2_v10.dmg

Changelog

  • Map Local & Breakpoint for HTTP/2
  • Update the latest Proxyman version 4.9.0

@lukeadamson
Copy link

@NghiaTranUIT Hello! Any update or ETA on HTTP/2 support? I'm still seeing connections be downgraded to HTTP/1.1 when proxied through Proxyman.

The update above indicates that it's still in beta as of Jul 30, but I can't find any mention of it after. Is there a way to access Beta builds of Proxyman?

@ekscrypto
Copy link

Same here, please make HTTP/2 available to the public

@NghiaTranUIT
Copy link
Member Author

Update v11:


Sorry guys @ekscrypto @lukeadamson , HTTP/2 is still in Beta because there is some technical issue that I haven't found a solution yet. It doesn't satisfy my standards so I decided not to bring it to the public until all issues are resolved.

For example:

  • Client -> Proxyman: Use WS/WSS with HTTP/2, but Proxyman -> Server uses HTTP/1.1 and vice versa
  • WS/WSS doesn't work HTTP/2

@djtarazona
Copy link

Thanks for the context. Appreciate you holding a quality bar.

I don't need WS/WSS, so these betas have been working good for me. I'd appreciate an option to download a build with this HTTP/2 beta for each stable Proxyman release (not sure if the beta build generation is automatic). Or perhaps consider including this feature in stable releases but only opt-in from some preferences pane for beta features.

Thanks a bunch for your work @NghiaTranUIT!

@seidnerj
Copy link

@NghiaTranUIT any chance to get an up-to-date beta with HTTP2 support? thanks in advance!

@seidnerj
Copy link

@NghiaTranUIT I'm a bit confused by the HTTP2 support in the builds provided here - does HTTP2 support in Proxyman mean that Proxyman itself will use HTTP2 when connecting to the target server or does it mean that support HTTP2 when receiving incoming traffic, i.e. listening on localhost:9090? or both?

@NghiaTranUIT
Copy link
Member Author

does HTTP2 support in Proxyman mean that Proxyman itself will use HTTP2 when connecting to the target server or does it mean that support HTTP2 when receiving incoming traffic

@Saklad5 yes, it's. Proxyman Beta prefers using HTTP/2, and fallback to HTTP/1 if your server doesn't talk HTTP2.

@seidnerj
Copy link

Okay, but from the client's perspective, the client connecting to proxyman that is, is the connection http2 or http1.1?

@NghiaTranUIT
Copy link
Member Author

NghiaTranUIT commented Jan 16, 2024

@Saklad5 Yes, it supports both HTTP/2 and HTTP/1 too, depending on what your client would like to use what protocol for the communication.

@NghiaTranUIT
Copy link
Member Author

Update HTTP/2 (BETA) v12 🎉

@leberwurstsaft
Copy link

Is HTTP2 support also in the 5.0 release?

@NghiaTranUIT
Copy link
Member Author

@leberwurstsaft HTTP/2 is still in Beta because the Websocket with HTTP/2 is not completed yet.

Here is the new update with Proxyman v5.0.0:

Screenshot

  • Enable HTTP/2 in the Proxy Setting (Proxyman Menu -> Setting -> General Tab -> Proxy Setting)

Screenshot 2024-02-19 at 10 53 18

@shrimpwtf
Copy link

Is this present in 5.5.0? can only find it in the above version

@NicolasCombe5555
Copy link

Sorry I got lost in time!
@NghiaTranUIT amazing progress! ❤️Thank you for your passion on this.
Not trying to rush it but I am just catching up, I saw in the comments you said it was 80% done some time ago, are we near close now? Will test your latest build!

@NghiaTranUIT
Copy link
Member Author

@shrimpyuk @NicolasCombe5555 Sorry, the HTTP/2 is not out of Beta yet because I could not solve some HTTP/2 issue with Websocket.

You can try this beta HTTP/2: https://download.proxyman.io/beta/Proxyman_5.5.0_Support_HTTP_2_v15.dmg

Changelog

  • Support HTTP/2 with all debugging tools (except HTTP/2 with WS)
  • All features/improvements/bugs fixes from Proxyman 5.5.0 => You can see the changelog at Github Release

@elijahdou
Copy link

@shrimpyuk @NicolasCombe5555 Sorry, the HTTP/2 is not out of Beta yet because I could not solve some HTTP/2 issue with Websocket.

You can try this beta HTTP/2: https://download.proxyman.io/beta/Proxyman_5.5.0_Support_HTTP_2_v15.dmg

Changelog

  • Support HTTP/2 with all debugging tools (except HTTP/2 with WS)
  • All features/improvements/bugs fixes from Proxyman 5.5.0 => You can see the changelog at Github Release

Is there a schedule for when http2.0 will be supported?

@NghiaTranUIT
Copy link
Member Author

NghiaTranUIT commented Jul 17, 2024

HTTP/2 v16

HTTP/2 v17

@zedd3v
Copy link

zedd3v commented Sep 26, 2024

Updates ? @NghiaTranUIT

@NghiaTranUIT
Copy link
Member Author

@heyzooi
Copy link

heyzooi commented Sep 27, 2024

any plans to support HTTP/3?

@bru02
Copy link

bru02 commented Oct 7, 2024

is http/2 supported when proxying over socks?

@felipe-linares
Copy link

@zedd3v it's still in Beta.

Update HTTP/2 with Proxyman 5.9.0

Any beta HTTP/2 windows builds by any chance?

@NghiaTranUIT
Copy link
Member Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature It's new feature
Projects
None yet
Development

No branches or pull requests