Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIP002 - Optional extension configurations as query strings in ss URLs #27

Closed
Mygod opened this issue Dec 29, 2016 · 118 comments
Closed

SIP002 - Optional extension configurations as query strings in ss URLs #27

Mygod opened this issue Dec 29, 2016 · 118 comments

Comments

@Mygod
Copy link
Contributor

Mygod commented Dec 29, 2016

Shadowsocks Improvement Proposal 002 (actually why don't we just use issue number to refer to them instead)

Optional configurations as query strings in ss URLs

Since #26, there are at least two optional extension for shadowsocks as far as I can tell:

  • Kcptun
  • HTTP/TLS obfuscation

There may be more extensions in the future. So what about adding them to ss URLs? For example, we can have ss://...?kcpport=8839&kcpcli=--crypt+none......#a+name for easier configuration. Clients that don't support the extensions can safely ignore them.

What should be included:

  • Values that should be consistent with server configuration.

What shouldn't:

  • Arbitrary values that can be configured by users, like whether the extension is enabled.
  • Values that could need changing when copying to other device (except that configuration for client and server can be different), like per-app proxy settings in Android client.

Problems:

  • Should client enable the extension that's been configured when importing?
  • Should client export the extension configuration even if it's not enabled?

Final version:

SIP002 purposed a new URL schema, following RFC3986:

SS-URI = "ss://" userinfo "@" hostname ":" port [ "/" ] [ "?" query ] [ "#" fragment ]
userinfo = websafe-base64-encode-utf8(method  ":" password)

The last / should be appended if query or fragment is present. Example: ss://YmYtY2ZiOnRlc3Q@192.168.100.1:8888/?plugin=url-encoded-plugin-argument-value&unsupported-arguments=should-be-ignored#Dummy+profile+name. This kind of URIs can be parsed by standard libraries provided by most languages.

For plugin argument, we use the similar format as TOR_PT_SERVER_TRANSPORT_OPTIONS, which have the format like simple-obfs;obfs=http;obfs-host=www.baidu.com where colons, semicolons, equal signs and backslashes MUST be escaped with a backslash.

@madeye
Copy link
Contributor

madeye commented Dec 30, 2016

Should client enable the extension that's been configured when importing?

I prefer disabled by default, even the config is available, especially for the experimental obfuscating.

Should client export the extension configuration even if it's not enabled?

I think if the extension is configured, we may also export the settings.

@lqhuang
Copy link

lqhuang commented Jan 1, 2017

Hi, guys. Firstly, thanks for your discussions @Mygod @madeye .

I’m not a expert on TCP/IP protocol. I'm interested in computer science, but I major in physics and write codes for science researches. Maybe, I could provide some views from a pure user rather than a developer.

When I heard @madeye want to deprecate obfuscation in the next release of shadowsocks-libev, I forked a 2.6.0 version branch into into my repositories immediately. I definitely understand why you think it's a dirty hack for given ISPs and breaks KISS principle. But as a user, I don't like the design of seperating obfuscation into simple-obfs. I'm a zealot of minimalist lifestyle and a practitioner for KISS principle, which means I will choose to enable TCP-BBR algorithm to speed up connections instead of choosing kcptun. Using kcptun will involve another thing, so it's not elegant for me.

The same problem is happen to obfuscation. But obfuscation is a little different with kcptun protocol,

On the server side, when the obfuscation is enabled, it still can handle normal protocol without obfuscating. So,

| Client-Obfs |   Server-Obfs  |  Working |
| No          |   Yes          |  Yes     |

If there is no necessary, clients don't need to make too many changes. For now, I'm glad to try this new feature just like enable UDP reply, Fast-open and One-time authentication. Yeah, I like obfuscation as a option --obfs <http|tls> much more than a new project called simple-obfs.

Imaging someone wants to use both kcptun and simple-obfs for multi-user configuration now, I think it's a terrible, trivial and dirty work to configure during different tools. It is not KISS at all.

Hence, please could you reconsider carefully whether to remove this new feature in the next release of shadowsocks-libev? Or just keep a fundanmental and recommended implementation of obfuscation in shadowsocks-libev, but also provide plugin supports to another obfuscations (like obfs4proxy). A much better solution also is welcome.

Anyway, I think your discussions are meaningful! It's a good try for further developments of shadowsocks protocol. Just keep it and make community of shadowsocks better.

These are my humble opinions. And I appreciate your time to read these. Thanks for your efforts.


PS: the following is not a technical content, just a little personal opinion. I'm sorry for that I'm too lazy to open a new email and send to @madeye.

I have some foreign friends who live and work in China. I volunteer to provide shadowsocks service to them for free. Some of them wanted to donate to me and I rufused all the time. Of course, I always got lots of thanks or became respected. "For the freedom of internet" someone said. They don't know who is the hero in the background. But, I know. I think all these honors should belong to you(@madeye) and @clowwindy. What's more, to everyone(shadowsocks.org members and others) who contributes to shadowsocks protocol and makes it better. That's how open source works!

Finally, Happy New Year! 🎉

Cheers!

@madeye
Copy link
Contributor

madeye commented Jan 1, 2017

@lqhuang Wow! It's really the longest feedback we have received in this issue tracker.

Thanks for your feedback and suggestion! I think one misunderstanding here is that we would remove simple-obfs from shadowsocks. Instead, we will keep providing simple-obfs as a plugin service of shadowsocks.

In other words, for the end users, nothing will change in the next release, although the command line may change to --plugin simple-obfs --plugin-args "--obfs http --obfs-host www.example.com".

The biggest advantage from this design is that we can have more and more plugin services designed for shadowsocks without modifying the upstream, e.g. shadowsocks-libev.

In the multi-user scenario, if you're using shadowsocks-manager, everything will work as usual. The server will automatically start the plugin service defined in the config or the command line. However, if you're with any third-party panel, changes may be required.

At last, KCPTUN, simple-obfs and all the future plugins are all optional. If you haven't experienced serious QoS, please don't enable them. Especially, as simple-obfs is actually a "dirty hack for given ISPs", it may introduce additional characteristics to shadowsocks protocol.

Thanks again for your feedback. Please keep watching our projects and send us more feedbacks!

@lqhuang
Copy link

lqhuang commented Jan 1, 2017

@madeye Thanks for your illustration! It's easier to understand now! Pherhaps I was misleaded by the manual page of simple-obfs. I admit that --obfs <http|tls> as a option is a simplest and most lazy way to play obfuscating tricks. I tried obfuscation and it seems to improve connections under some tests.

Thanks again for your replay. You're doing a great work:)

@madeye
Copy link
Contributor

madeye commented Jan 9, 2017

What about encoding a plugin's settings into one BASE64 string? Then the URL looks like ss://passwd:method-auth@ip:port/?plugin=BASE64_STRING. It will help to simplify the URL parsing.

To follow #28, we should remove kcp_port or plugin_port in the url.

@Mygod
Copy link
Contributor Author

Mygod commented Jan 9, 2017 via email

@madeye
Copy link
Contributor

madeye commented Jan 9, 2017

Hmm. The executable name should be the same even in a cross platform scenario? The developer of plugins should take care of this, make sure no conflict between different plugins.

For platform specific settings, users may have to modify them after importing, e.g. interface name to bind. I think shadowsocks clients are only required to pass the arguments to the plugin, never parse them.

@Mygod
Copy link
Contributor Author

Mygod commented Jan 9, 2017

Should plugin provide configuration user interface? In that case plugin would need to do the argument parsing.

For example, I may want to use /data/data/com.github.shadowsocks.plugin.kcptun/lib/libkcptun.so as executable path. What about this?

@madeye
Copy link
Contributor

madeye commented Jan 9, 2017

I prefer no specific UI for different plugins.

IMO, a shadowsocks-android plugin package should provide an interface to copy binaries from its data directory to shadowsocks-android's. In addition, the name of the executable should keep consistent on all the platforms. This name can also be passed from plugin package to shadowsocks-android.

@em0minate
Copy link

em0minate commented Jan 11, 2017

Want to pitch about the Matrix URIs scheme for the configuration, e.g.:

                                plugin  key=val   key=val
                                  |        |         |
                                ------ --------- ---------
   ss://passwd:method@host:port/kcptun;mode=fast;crypt=aes/obfs;host=foobar.io
   \_/  \___________/ \_______/ \________________________/ \_________________/
    |        |            |                 |                       |
 protocol  authority    remote            kcptun                simple-obfs

Which makes grouping more easily.

@Mygod
Copy link
Contributor Author

Mygod commented Jan 12, 2017

@em0minate We don't support multiple plugins though. Also:

...as matric URI parsing is not a feature of the web. This is just something which could have been.

@em0minate
Copy link

em0minate commented Jan 12, 2017

  • It's a proposal for configuration URI spec leaves room for flexibility, not limited by the client side implementation tho.
  • Matrix URLs doesn't makes into web standard we currently are, but it's still being valid URI form, only requires few parsing around URI's pathname, which more organized than plain query parameters.
    Angular 2 for one, its router mechanism build on top of matrix parameters. Also JAX-RS has good support too.

@em0minate
Copy link

Quick demo to illustrate the parsing:

const { parse } = require('url');



const uri = 'ss://passwd:method@hostname:7070/kcptun;mode=fast;crypt=aes/obfs;host=foobar.io';


const {      protocol, auth, hostname, port, pathname } = parse(uri);
console.info(protocol, auth, hostname, port);

const plugins = pathname
    .split('/')
    .filter(Boolean)
    .map(s => s.replace(';', '?').replace(/;/g, '&'))
    .map(s => parse(s, true))
    .map(({pathname, query}) => ({[pathname]: query}))
;

const options = Object.assign({}, ...plugins);
console.info(options);

play this around on runkit

@qiuyuzhou
Copy link

qiuyuzhou commented Jan 13, 2017

Hello,

I have added kcptun to ShadowsocksX-ng recently.Add kcptun configures to the qrcode like

ss://aes-256-cfb:Password@139.165.131.73:4000?Remark=kcptun%20test&OTA=false&Kcptun=true&mode=normal&key=secrit&crypt=aes-128&datashard=10&parityshard=3&nocomp=false&mtu=1350

The codes has been push to develop branch on github, but has not released a new version.

Any suggestions?

@madeye
Copy link
Contributor

madeye commented Jan 13, 2017

The porposal of @em0minate looks quite interesting. What do you think? @Mygod

@qiuyuzhou Thanks for the implementation, please wait for @Mygod's final decision.

@Mygod
Copy link
Contributor Author

Mygod commented Jan 13, 2017

What if plugin uses non-standard CLI though? This would imply a change to SIP003.

@madeye
Copy link
Contributor

madeye commented Jan 13, 2017

@Mygod Could you elaborate more on "non-standard CLI"? Is there any example?

@Mygod Mygod changed the title Optional extension configurations as query strings in ss URLs SIP002 - Optional extension configurations as query strings in ss URLs Jan 13, 2017
@Mygod
Copy link
Contributor Author

Mygod commented Jan 13, 2017

@madeye Maybe like:
some-plugin [file1] [file2] [file3]...?

Actually there's no constraint that one must use getopt-style command line options. One could even use inline bash scripts.

@madeye
Copy link
Contributor

madeye commented Jan 13, 2017

@Mygod OK, we should add a restriction in SIP003.

@Mygod
Copy link
Contributor Author

Mygod commented Jan 13, 2017

And even if you do that, every client from now on will be forced to parse CLI to generate QR code. And what about illegal characters? (URL encode/decode?)

@madeye
Copy link
Contributor

madeye commented Jan 13, 2017

@Mygod Hmm.. You're right.

What about this #27 (comment)? We encode the CLI in BASE64?

@Mygod
Copy link
Contributor Author

Mygod commented Jan 13, 2017

Although I have to say this is actually a more cross-platform solution. For example, options in Windows are by convention prepended by '/' while in getopt they are prepended by one or two -s. I suggest to pass PT-like configuration to client but that would require us to bomb SIP003.

@Mygod
Copy link
Contributor Author

Mygod commented Jan 13, 2017

Personally I prefer to URL/base64 encoding CLI if we're not bombing SIP003.

@madeye
Copy link
Contributor

madeye commented Jan 13, 2017

Unless a plugin is implemented as native Win32 application, I don't think we would have issues in CLI.

In addition, to support cross-platform, a SIP003 plugin should support POSIX getopt() instead of Win32 getopt().

@madeye
Copy link
Contributor

madeye commented Jan 13, 2017

@Mygod Agreed.

@madeye
Copy link
Contributor

madeye commented Mar 1, 2017

@Mygod OK, that should not be a problem for us. At least all the official desktop and mobile clients support the legacy BASE64 URL.

@Mygod
Copy link
Contributor Author

Mygod commented Mar 1, 2017

Should we also keep support for generating old format if the user wishes to use it?

@madeye
Copy link
Contributor

madeye commented Mar 1, 2017

@Mygod I think it should be useful today.

@madeye
Copy link
Contributor

madeye commented Mar 1, 2017

@Mygod What about generating legacy URL for configs without plugins? It makes sense and won't break backward compatibility.

@Mygod
Copy link
Contributor Author

Mygod commented Mar 1, 2017

I don't think that's a good idea.

@shinku721
Copy link

shinku721 commented Mar 2, 2017

Well can we set up a standard of URIs? Could someone please tell me what currently used URIs look like?

Edit:
Standardize one, deprecate others

@Mygod
Copy link
Contributor Author

Mygod commented Mar 9, 2017

@madeye Should we settle this now?

@madeye
Copy link
Contributor

madeye commented Mar 9, 2017

Yes, could you also add a wiki page about the new URL?

@Mygod
Copy link
Contributor Author

Mygod commented Mar 9, 2017

No. 🙃

@madeye
Copy link
Contributor

madeye commented Mar 9, 2017

😓

@madeye madeye closed this as completed Mar 9, 2017
@Mygod
Copy link
Contributor Author

Mygod commented Mar 9, 2017

@madeye I'll let you decide which one we would use. 😛

@Mygod
Copy link
Contributor Author

Mygod commented Mar 13, 2017

So what's the conclusion?

@madeye
Copy link
Contributor

madeye commented Mar 13, 2017

@Mygod I think it's settled as #27 (comment).

@jibon57
Copy link

jibon57 commented Sep 10, 2017

@madeye Will you please give an example of KCPTun? Is that like this:

Plugin: client_windows_amd64.exe
Plugin Options: ss://passwd@host:port/kcptun;mode=fast;crypt=aes-128

@zonyitoo
Copy link
Contributor

zonyitoo commented May 12, 2022

Since SIP022 have already make "password" to become a base64 of the PSK, we may not need to base64 encode the userinfo anymore.

Futher more, go2-shadowsocks2 assumes userinfo is not encoded: https://github.com/shadowsocks/go-shadowsocks2/blob/71a35d3383128957db1327991445fca643feb4e2/main.go#L189-L201

So here I suggest SIP002 should allow userinfo to not being encoded, and implementations should handle both 2 forms. For AEAD-2022 ciphers, userinfo in SIP002 MUST NOT be encoded (just for unifying all implementations' behaviors).

Suggestions have been added to https://github.com/shadowsocks/shadowsocks-org/wiki/SIP002-URI-Scheme

cc @madeye @Mygod @riobard

@madeye
Copy link
Contributor

madeye commented May 15, 2022

I think it's too late to change SIP002. A alternative change is adding ss2:// scheme in SIP022.

@zonyitoo
Copy link
Contributor

I think it is Ok to just change SIP002's spec about allowing plain text user-info, because you can always use the old SIP002 URL to represent SIP022 methods. So plain text user-info is just an extension of SIP002.

@madeye
Copy link
Contributor

madeye commented May 15, 2022

So, some clients have to implement fallback logic for the initial scheme and previous SIP002 scheme, which seems too complicated.

@database64128
Copy link
Contributor

So, some clients have to implement fallback logic for the initial scheme and previous SIP002 scheme, which seems too complicated.

It's actually quite simple: just look for :. All clients I know, including shadowsocks-android, already support it.

@database64128
Copy link
Contributor

A alternative change is adding ss2:// scheme in SIP022.

That's basically what I proposed initially (except I went with ss-2022://). But @zonyitoo had me convinced that a new scheme is not necessary. So I removed the proposal from SIP022.

@madeye
Copy link
Contributor

madeye commented May 17, 2022

Seems no further objections to the change. Let's finalize SIP002 to allow non-encoded user-info.

@cloverzrg
Copy link

ss://v2;xxx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests