-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Gitlab project #630
Gitlab project #630
Conversation
The codeclimate advice is weird. |
providers/gitlab_test.go
Outdated
@@ -108,6 +127,46 @@ func TestGitLabProviderUsername(t *testing.T) { | |||
assert.Equal(t, "FooBar", username) | |||
} | |||
|
|||
func TestGitlabProviderProject(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm taking a look at these units tests for the first time myself since I haven't used Gitlab -- I see alot of redundant test code (historically before you).
Do you have the bandwidth to try to to get these setup using table driven tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@NickMeves Hi, is something like this https://github.com/factorysh/oauth2-proxy/commit/f2480880919359a32d6a9137ff7987e868e1cf92, the way you want it ? If so, i'll finish and clean up. Thanks for your reply. (i searched briefly in the repo for reference and found nothing using table tests)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everything is updated using table tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This has some samples: https://github.com/oauth2-proxy/oauth2-proxy/blob/master/pkg/apis/sessions/session_state_test.go
Key point is using the t.Run
method to take the test case name and make a subtest out of it.
We are also trying wherever possible to migrate new tests to using ginkgo more (you can see an example of adding that to legacy test tables here:
g := NewWithT(t) |
If you went full Ginkgo BDD, here is a sample of a Test Table with that framework we are using for alot of new tests: https://github.com/oauth2-proxy/oauth2-proxy/blob/master/pkg/requests/result_test.go
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for reply, never heard of this framework but it was quite fun to use, full migration done in fecd956
providers/gitlab_test.go
Outdated
b := testGitLabBackend() | ||
defer b.Close() | ||
|
||
bURL, _ := url.Parse(b.URL) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see the older tests do this too -- but lets not ignore the errors.
Can we add a assert.NoError(t, err)
after this?
providers/gitlab_test.go
Outdated
p.Project = "my_group/my_bad_project" | ||
session := &sessions.SessionState{AccessToken: "gitlab_access_token"} | ||
_, err := p.GetEmailAddress(context.Background(), session) | ||
assert.NotNil(t, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
assert.NotNil(t, err) | |
assert.Error(t, err) |
providers/gitlab_test.go
Outdated
p.Project = "my_group/my_project" | ||
session := &sessions.SessionState{AccessToken: "gitlab_access_token"} | ||
email, err := p.GetEmailAddress(context.Background(), session) | ||
assert.Equal(t, nil, err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
assert.Equal(t, nil, err) | |
assert.NoError(t, err) |
providers/gitlab.go
Outdated
} | ||
var body []byte | ||
body, err = ioutil.ReadAll(resp.Body) | ||
resp.Body.Close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm on a warpath for unhandled closes today to eventually get to a clean Gosec 😄 -- can we handle the error response from this?
providers/gitlab.go
Outdated
@@ -250,6 +295,17 @@ func (p *GitLabProvider) GetEmailAddress(ctx context.Context, s *sessions.Sessio | |||
return "", fmt.Errorf("group membership check failed: %v", err) | |||
} | |||
|
|||
// Check project membership |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we make this a verifyProjectMembership
function to mirror the organization of the other verifiers that are similar to this above?
docs/configuration/configuration.md
Outdated
@@ -58,6 +58,7 @@ An example [oauth2-proxy.cfg]({{ site.gitweb }}/contrib/oauth2-proxy.cfg.example | |||
| `--github-token` | string | the token to use when verifying repository collaborators (must have push access to the repository) | | | |||
| `--github-user` | string \| list | To allow users to login by username even if they do not belong to the specified org and team or collaborators | | | |||
| `--gitlab-group` | string | restrict logins to members of any of these groups (slug), separated by a comma | | | |||
| `--gitlab-project` | string | restrict logins to members of any of these project | | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This just merged and fixes an issue with the --gitlab-group
config setup (singular vs plural and StringSlice
flag): #639
I think you likely want to mirror those changes for projects assuming you also want to allow multiples.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I made the changes for multiple groups, not sure about how you want to log failed attempts. Can you give me an advice for this ? Thanks a lot.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR! I'm not fully familiar with the GitLab provider myself, so take my suggestions with a grain of salt if they seem completely out of whack 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Big fan of the ginkgo rewrite of the gitlab provider tests, thanks for that!
pkg/apis/options/options.go
Outdated
@@ -51,6 +51,7 @@ type Options struct { | |||
GitHubToken string `flag:"github-token" cfg:"github_token"` | |||
GitHubUsers []string `flag:"github-user" cfg:"github_users"` | |||
GitLabGroup string `flag:"gitlab-group" cfg:"gitlab_group"` | |||
GitlabProject []string `flag:"gitlab-project" cfg:"gitlab_projects"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't this be named GitlabProjects
since it's a list?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For me yes but I found this in upstream :
GitLabGroup []string `flag:"gitlab-group" cfg:"gitlab_groups"
in https://github.com/oauth2-proxy/oauth2-proxy/blob/master/pkg/apis/options/options.go#L51
And done it the same way
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rather it was with the trailing s
, this is what the other should be as well IMO
pkg/apis/options/options.go
Outdated
@@ -232,6 +233,7 @@ func NewFlagSet() *pflag.FlagSet { | |||
flagSet.String("github-token", "", "the token to use when verifying repository collaborators (must have push access to the repository)") | |||
flagSet.StringSlice("github-user", []string{}, "allow users with these usernames to login even if they do not belong to the specified org and team or collaborators (may be given multiple times)") | |||
flagSet.String("gitlab-group", "", "restrict logins to members of this group") | |||
flagSet.StringSlice("gitlab-project", []string{}, "restrict logins to members of this project (may be given multiple times)") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should include the formatting instructions in this
providers/gitlab.go
Outdated
@@ -32,7 +35,7 @@ func NewGitLabProvider(p *ProviderData) *GitLabProvider { | |||
p.ProviderName = "GitLab" | |||
|
|||
if p.Scope == "" { | |||
p.Scope = "openid email" | |||
p.Scope = "openid email read_api" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps only make this default change if projects are specified? Otherwise we may break existing users applications
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the best approach for this ? I can't setup a condition inside New, because there is no info about projects at this time in the execution. In the github provider, there is method to set group https://github.com/oauth2-proxy/oauth2-proxy/blob/master/providers/github.go#L95, I can duplicate this behavior in the gitlab provider and ensure in this method that "read_api" will be added to scope only once. Thanks for reply.
providers/gitlab.go
Outdated
Path: "/api/v4/projects/", | ||
} | ||
endpoint := endpointURL.String() + url.QueryEscape(project) | ||
req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use the request builder from github.com/oauth2-proxy/oauth2-proxy/pkg/requests
as can be seen in other providers
oauth2-proxy/providers/github.go
Lines 131 to 138 in 37026b6
err := requests.New(endpoint.String()). | |
WithContext(ctx). | |
WithHeaders(makeGitHubHeader(accessToken)). | |
Do(). | |
UnmarshalInto(&op) | |
if err != nil { | |
return false, err | |
} |
I think this should work
err := requests.New(endpoint).
WithContext(ctx).
SetHeader("Authorization", "Bearer "+s.AccessToken).
Do().
UnmarshalInto(&projectInfo)
if err != nil {
return false, err
}
providers/gitlab.go
Outdated
|
||
for _, project := range p.Projects { | ||
project, err := p.getProjectInfo(ctx, s, project) | ||
if err == nil && !project.Archived { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm assuming no error means that the user is a member of the project? This could probably do with a comment to explain if that's the case
providers/gitlab_test.go
Outdated
authHeader := "Bearer gitlab_access_token" | ||
|
||
return httptest.NewServer(http.HandlerFunc( | ||
func(w http.ResponseWriter, r *http.Request) { | ||
if r.URL.Path == "/oauth/userinfo" { | ||
if r.URL.String() == "/oauth/userinfo" { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure I understand why this change was made? Can you remember?
I wonder if it's worth making this a switch statement rather than a series of if
statements 🤔
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure about why my team mate did this thing 🤷. Ok, it was about URL encoding and how to interpret /
, I just fixed it.
assert.Equal(t, nil, err) | ||
assert.Equal(t, "foo@bar.com", email) | ||
} | ||
var _ = Describe("Gitlab Provider Tests", func() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice rewrite of the tests, thanks!!! 😄
providers/gitlab_test.go
Outdated
if in.projects != nil { | ||
if len(in.projects) >= 1 { | ||
p.Projects = in.projects | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this can be simplified
if in.projects != nil { | |
if len(in.projects) >= 1 { | |
p.Projects = in.projects | |
} | |
} | |
if len(in.projects) > 0 { | |
p.Projects = in.projects | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh yes, that's dumb, I forgot that a nil slice have a length of 0. 😹
There are a lot of conflicts on this branch, can you rebase it and resolve the conflicts? |
Sure, i'll clean up the mess to give you a clean PR, thanks for all the feedback. |
This branch is now rebased against master, there is still some real world tests to do to ensure we do not hit some corner cases. In this implementation, a user can not access an archived repo, this behavior can be discuss if needed since archived can not mean the same thing for each users. I just need to migrate to the requests package as requested by @JoelSpeed. Apart from that, I think all the requested changes are updated with all the advices and insights we discuss in this conversation. |
I've test it against our production server, users member of target project can access resources, users not member of target project get a 403. Do you have any rules/guide to update Docs and Changelog ? Do you want me to squash everything before merge ? Thanks. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look good to me!
Please add a CHANGELOG entry and we can then merge 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, just when I approved I eyeballed a change I think needs to be discussed.
With that fix + a CHANGELOG entry, this will be good to go.
providers/gitlab.go
Outdated
// SetProjects adds Projects to a GitLabProvider struct and extends OAuth2 needed scope | ||
func (p *GitLabProvider) SetProjects(projects []string) { | ||
p.Projects = projects | ||
p.Scope += " read_api" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you need to be aware of when the enduser consciously updates the scope
option and not clobber it when they put in custom values.
We just had to think about something similar with the OIDC provider adding group claim AuthZ:
I think similar logic might be warranted here -- only add read_api
if p.Scope
isn't explicitly set by the user. If I'm following this logic correctly, you are always adding this to the Gitlab scope, so should you just add read_api
to gitlabDefaultScope
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just to clarify, this is dynamically added here (if projects options are specified) https://github.com/oauth2-proxy/oauth2-proxy/pull/630/files#diff-80edce26da4509c90a73a1669a5bbdc7R180. This won't break old installation already in place without this extended scope (as requested by @JoelSpeed). If something needs to be change in this behavior, tell me and i'll do it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you only add read_api
in SetProjects
(I agree with @JoelSpeed on his conditionality point) if the user didn't set something explicitly themselves for the scope
🤔
E.g. If I explicitly set the --scope
to openid email read_api
this code turns that into openid email read_api read_api
. I feel like if a user explicitly changes the default scope
for whatever reason (even if it results in bugs for them because they forgot read_api
after wanting this feature), at that point our code can't attempt to mutate it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I think I get it ! While trying to figure it out, i came across this : https://github.com/oauth2-proxy/oauth2-proxy/blob/master/pkg/validation/options.go#L153. This is what I get as default scope when creating the Gitlab Provider struct (without scope option).
I did not find a way to know if the user requested a specific scope or if it's the default one. Doing a diff is way to tricky to be maintainable.
The approach I take is : avoid duplicates read_api
in the scope (ie: check if absent before adding it). I think it's the most viable solution. (See 3514af8)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still have concerns this is always called even when project authorization isn't configured: https://github.com/oauth2-proxy/oauth2-proxy/pull/630/files#diff-2690264192c9633892c7bda108d4a58dR290
The link you sent above about the conditional SetProject
call that has this logic was in a unit test.
As far as the read_api
duplicate concern -- the new commit with your proposed solution looks good to me 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, i just add a check before calling SetProjects c0e9d1c. If everything is fine, I can squash and rebase before you merge. Thanks a lot for your feedback !
pkg/validation/options.go
Outdated
@@ -287,6 +287,9 @@ func parseProviderInfo(o *options.Options, msgs []string) []string { | |||
case *providers.GitLabProvider: | |||
p.AllowUnverifiedEmail = o.InsecureOIDCAllowUnverifiedEmail | |||
p.Groups = o.GitLabGroup | |||
if len(o.GitlabProjects) > 0 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NIT: I think this logic belongs in SetProjects
around the scope manipulation. Just to keep things together, it is detached from the area driving the reason for this logic when over here.
Avoids the double standard since we are perfectly fine setting the parallel functionality p.Groups
to an empty array right above this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I clearly understand what you say, here is the fix : 542056e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately, I think that commit brings us back to the original problem (scope is manipulated even when we don't need to).
This is what I was thinking:
if len(o.GitlabProjects) > 0 && !strings.Contains(p.Scope, "read_api")
Just to keep the logic together and only manipulate scope
if needed.
Tangentially related to this PR, but likely too much scope creep, better in a different PR: How do we want to support gitlab group or project AuthZ in the new I'm curious your thoughts as a Gitlab provider user (I don't use this provider) -- I want to get all our Providers that have some form of Group AuthZ logic to pass the groups data to the backend via the |
No sure about the semantic of the For now the matching project is not returned on success but it's quite easy to add this variable in the early return (https://github.com/oauth2-proxy/oauth2-proxy/pull/630/files#diff-83fab261d52ee47763d81d669a7b7b74R295).
One way could be add a dedicated header for projects, since Github also have this feature. The other way could be passing the projects in the group header since a project is, at the end, a group of users. |
FYI - I have 2 PRs very related to this: #796 & #797 Once they merge I will circle back and we can get this finalized and merged in as well. I really want to clean up the post token redeem session enrichment process (where a lot of providers have hacked in authorization logic into GetEmailAddress) and get a centralized method for authorization under a global In my refactor in #796 I noted the area where Gitlab had hacky AuthZ in an improper spot and added a TODO note. |
Indeed, it looks like there is a lot of plumbing stuff to do to harmonize all the things and remove all the accumlated layers of hacks over hacks. For now, the impact on this PR looks minimal and I hope it will not invalidate the work and efforts I put in. |
Sorry for the delay. If you still want to get this over the finish line: #797 merged. The refactor to |
Hi @NickMeves, thanks for your message and status update. I will work on this next week if I find the bandwith. 👍 |
Thanks again for taking this on! I think the only change is to add any projects (or whatever else) into the session.Groups list in I think for things like this with multiple group-link entities, @JoelSpeed and I were leaning toward a prefix to deconflict between the possible options. (e.g. |
Here is a starting point using I squashed everything in order to make things more readable. I also made a copy of the old branch, just for sanity (see gitlab_project-before-enrich). I tested it on our test project and it seems to work (needs a recheck with a fresh brain) Also a question/opinion : For now, we loop over each project in the config file, impersonating the user to see if project is accessible (see gitlab.go) and adding it to the session state group. In this case : 1 project in the config file means 1 request to Gitlab. I think I will try to go reverse and fetch all the projects of the user and then compare it to the ones listed in the config file. What do you think ? Thanks for reading/feedback. |
Co-authored-by: Joel Speed <Joel.speed@hotmail.co.uk>
This commit fixes a bug caused by an override of the Verifier value from *ProviderData inside GitlabProvider struct
Co-authored-by: Nick Meves <nick.meves@greenhouse.io>
Rebased against master 👍 |
It was a pleasure working with you on this. Thanks again! |
* Add support for gitlab projets * Add group membership in state * Use prefixed allowed groups everywhere * Fix: remove unused function * Fix: rename func that add data to session * Simplify projects and groups session funcs * Add project access level for gitlab projects * Fix: default access level * Add per project access level * Add user email when missing access level * Fix: harmonize errors * Update docs and flags description for gitlab project * Add test with both projects and groups * Fix: log error message Co-authored-by: Joel Speed <Joel.speed@hotmail.co.uk> * Fix: make doc a markdown link * Add notes about read_api scope for projects * Fix: Verifier override in Gitlab Provider This commit fixes a bug caused by an override of the Verifier value from *ProviderData inside GitlabProvider struct * Fix: ensure data in session before using it * Update providers/gitlab.go Co-authored-by: Nick Meves <nick.meves@greenhouse.io> * Rename gitlab project initializer * Improve return value readbility * Use splitN * Handle space delimiters in set project scope * Reword comment for AddProjects * Fix: typo * Rework error handling in addProjectsToSession * Reduce branching complexity in addProjectsToSession * Fix: line returns * Better comment for addProjectsToSession * Fix: enrich session comment * Fix: email domains is handled before provider mechanism * Add archived project unit test * Fix: emails handling in gitlab provider Co-authored-by: Wilfried OLLIVIER <wollivier@bearstech.com> Co-authored-by: Joel Speed <Joel.speed@hotmail.co.uk> Co-authored-by: Nick Meves <nick.meves@greenhouse.io>
Gitlab project
--github-repo
is already a setting, here is the--gitlab-project
setting.Motivation and Context
Filtering by group is an oidc option, but filtering by project is easier.
How Has This Been Tested?
There is unit test.
Functional test are done with this command :
Checklist: