Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce salt Python linting #3066

Merged
merged 63 commits into from
Feb 8, 2021
Merged

Conversation

TeddyAndrieux
Copy link
Collaborator

Component:

'salt', 'lint', 'python'

Context:

#2649

Summary:

  • Fix python linting issue in all Salt modules
  • Enable salt python linting in tox and doit, enforce it in CI build

Fixes: #2649

@TeddyAndrieux TeddyAndrieux added kind:debt Technical debt complexity:medium Something that requires one or few days to fix python Pull requests that update Python code topic:salt Everything related to SaltStack in our product labels Jan 25, 2021
@TeddyAndrieux TeddyAndrieux requested a review from a team January 25, 2021 16:10
@bert-e
Copy link
Contributor

bert-e commented Jan 25, 2021

Hello teddyandrieux,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

@bert-e
Copy link
Contributor

bert-e commented Jan 25, 2021

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • one peer

Peer approvals must include at least 1 approval from the following list:

@TeddyAndrieux TeddyAndrieux force-pushed the improvement/add-salt-python-lint branch from f470a8e to 4e25cd3 Compare January 26, 2021 12:56
salt/_modules/metalk8s_drain.py Outdated Show resolved Hide resolved
salt/_modules/metalk8s_drain.py Outdated Show resolved Hide resolved
salt/_modules/metalk8s_monitoring.py Outdated Show resolved Hide resolved
Copy link
Contributor

@gdemonet gdemonet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking care of this, will be a good protection against simple mistakes! I just answered on some conversations, nothing really blocking though.

salt/_modules/metalk8s_drain.py Outdated Show resolved Hide resolved
salt/_modules/metalk8s_drain.py Outdated Show resolved Hide resolved
@@ -5,6 +5,7 @@
import traceback
import re

from salt.exceptions import CommandExecutionError
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 (undefined vars are a really useful check ❤️)

salt/_renderers/metalk8s_kubernetes.py Outdated Show resolved Hide resolved
salt/.pylintrc Outdated
Comment on lines 72 to 112
locally-disabled,
file-ignored,
unexpected-special-method-signature,
import-error,
no-member,
unsubscriptable-object,
un-indexed-curly-braces-error,
whitespace-before-colon,
indentation-is-not-a-multiple-of-four-comment,
blacklisted-name,
invalid-name,
missing-docstring,
empty-docstring,
misplaced-comparison-constant,
unidiomatic-typecheck,
wrong-import-order,
ungrouped-imports,
wrong-import-position,
bad-mcs-method-argument,
bad-mcs-classmethod-argument,
line-too-long,
too-many-lines,
bad-continuation,
exec-used,
attribute-defined-outside-init,
protected-access,
reimported,
fixme,
global-statement,
unused-variable,
unused-argument,
redefined-outer-name,
redefined-builtin,
undefined-loop-variable,
logging-format-interpolation,
invalid-format-index,
line-too-long,
unexpected-indentation-comment,
continuation-line-indentation-is-not-a-multiple-of-four,
continuation-line-missing-indentation-or-outdented,
closing-bracket-does-not-match-indentation-of-opening-brackets-line,
closing-bracket-does-not-match-visual-indentation,
continuation-line-does-not-distinguish-itself-from-next-logical-line,
continuation-line-over-indented-for-hanging-indent,
continuation-line-over-indented-for-visual-indent,
continuation-line-under-indented-for-visual-indent,
visually-indented-line-with-same-indent-as-next-logical-line,
unaligned-for-hanging-indent,
block-comment-should-start-with-cardinal-space,
too-many-leading-hastag-for-block-comment,
module-level-import-not-at-top-of-file,
do-not-assign-a-lambda-expression-use-a-def,
3rd-party-local-module-not-gated,
pep8-reserved-keywords,
str-format-in-logging,
import-outside-toplevel,
deprecated-method,
repr-flag-used-in-string,
keyword-arg-before-vararg,
incompatible-py3-code,
multiple-spaces-before-operator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this a lot to disable? Recommended by Salt? Some checks like unused-variable/undefined-loop-variable may be useful, don't you think?

Copy link
Collaborator Author

@TeddyAndrieux TeddyAndrieux Feb 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes maybe, btw this one just get copied from pylintrc from salt repo, but indeed we may want to tune it a bit (I remember I already did some minor changes, do not remember the exact ones)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK let's keep it like this for now, and add some debt ticket to track that we'll want to include more rules :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will remove from ignore list (and fix our code if needed) for at least those

whitespace-before-colon,
indentation-is-not-a-multiple-of-four-comment,
whitespace-before-colon,
indentation-is-not-a-multiple-of-four-comment,
undefined-loop-variable,
unexpected-indentation-comment,
continuation-line-indentation-is-not-a-multiple-of-four,
continuation-line-missing-indentation-or-outdented,
closing-bracket-does-not-match-indentation-of-opening-brackets-line,
closing-bracket-does-not-match-visual-indentation,
continuation-line-does-not-distinguish-itself-from-next-logical-line,
continuation-line-over-indented-for-hanging-indent,
continuation-line-over-indented-for-visual-indent,
continuation-line-under-indented-for-visual-indent,
visually-indented-line-with-same-indent-as-next-logical-line,
unaligned-for-hanging-indent,
block-comment-should-start-with-cardinal-space,
too-many-leading-hastag-for-block-comment,
pep8-reserved-keywords,
str-format-in-logging,

Tell me if you have others that you think we should not ignore (anyway we could remove then from ignore list in the future if needed)

@bert-e
Copy link
Contributor

bert-e commented Feb 4, 2021

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • one peer

Peer approvals must include at least 1 approval from the following list:

The following reviewers are expecting changes from the author, or must review again:

gdemonet
gdemonet previously approved these changes Feb 5, 2021
@bert-e
Copy link
Contributor

bert-e commented Feb 5, 2021

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • one peer

Peer approvals must include at least 1 approval from the following list:

gdemonet
gdemonet previously approved these changes Feb 5, 2021
Copy link
Contributor

@gdemonet gdemonet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a style question on some function signatures, where we decide not to use some arguments.
Would you prefer keeping the original signature, as is done today, or make it explicit that you discard some arguments with (..., **_kwargs):?
Anyhow, let's not block the merge for this 😉

salt/_auth/kubernetes_rbac.py Outdated Show resolved Hide resolved
```
salt/_modules/cri.py:174: [W0120(useless-else-on-loop), wait_container] Else clause on loop without a break statement
```
```
salt/_modules/metalk8s_cordon.py:84: [E8303(too-many-blank-lines), ] PEP8 E303: too many blank lines (2)
```
Disable some "raise-missing-from" as it's known exception and we do not
want to have upper exceptions raised
```
salt/_modules/metalk8s_drain.py:388: [W0707(raise-missing-from), Drain.run_drain] Consider explicitly re-raising using the 'from' keyword
salt/_modules/metalk8s_drain.py:394: [W0707(raise-missing-from), Drain.run_drain] Consider explicitly re-raising using the 'from' keyword
```

Use `from` in exception instead of formatting with the exception
message
```
salt/_modules/metalk8s_drain.py:360: [W0707(raise-missing-from), Drain.run_drain] Consider explicitly re-raising using the 'from' keyword
salt/_modules/metalk8s_drain.py:547: [W0707(raise-missing-from), evict_pod] Consider explicitly re-raising using the 'from' keyword
```
Remove "unecessary-pass"
```
salt/_modules/metalk8s_drain.py:59: [W0107(unnecessary-pass), DrainTimeoutException] Unnecessary pass statement
```
Remove the unused variable
```
salt/_modules/metalk8s_drain.py:516: [W0612(unused-variable), evict_pod] Unused variable 'result'
```
Use six urllib parse module from salt instead of six directly
```
salt/_modules/metalk8s_etcd.py:6: [E9402(blacklisted-module), ] Uses of a blacklisted module 'six.moves.urllib.parse': Please report this error to SaltStack so we can fix it: Trying to import urlparse from six.moves.urllib.parse
```
Fix "bare-except" and disable "broad-except" as we want to catch all
exceptions
```
salt/_modules/metalk8s_etcd.py:198: [W0702(bare-except), get_etcd_member_list] No exception type(s) specified
```
Disable "broad-except" as we want to catch all exceptions
```
salt/_modules/metalk8s_kubeconfig.py:43: [W0703(broad-except), validate] Catching too general exception Exception
```
Re-format to not use `backslash` inside brackets
```
salt/_utils/volume_utils.py:280: [E8502(the-backslash-is-redundant-between-brackets), ] PEP8 E502: the backslash is redundant between brackets
```
Disable "unused-argument" for `ext_pillar` as we need a specific
function signature
```
salt/_pillar/metalk8s_endpoints.py:20: [W0613(unused-argument), ext_pillar] Unused argument 'minion_id'
salt/_pillar/metalk8s_endpoints.py:20: [W0613(unused-argument), ext_pillar] Unused argument 'pillar'
```
Disable "broad-except" as we want to catch all exceptions
```
salt/_pillar/metalk8s_etcd.py:18: [W0703(broad-except), _load_members] Catching too general exception Exception
```
Disable "unused-argument" from `ext_pillar` as we need a specific
function signature
```
salt/_pillar/metalk8s_etcd.py:32: [W0613(unused-argument), ext_pillar] Unused argument 'minion_id'
```
Use `fopen` from salt instead of classic `open`
```
salt/_pillar/metalk8s_private.py:19: [W8470(resource-leakage), _read_private_key] Resource leakage detected. Please use 'with salt.utils.files.fopen()' instead of 'with open()'. It assures salt does not leak file handles.
```
Remove uneeded comma
```
salt/_pillar/metalk8s.py:187: [E8231(missing-whitespace-after-comma), ] PEP8 E231: missing whitespace after ','
```
Disable "broad-except" as we want to catch all exceptions
```
salt/_pillar/metalk8s.py:28: [W0703(broad-except), _load_config] Catching too general exception Exception
```
Disable "unused-argument" for `ext_pillar` as we need a specific
function signature
```
salt/_pillar/metalk8s.py:156: [W0613(unused-argument), ext_pillar] Unused argument 'minion_id'
salt/_pillar/metalk8s.py:156: [W0613(unused-argument), ext_pillar] Unused argument 'pillar'
```
Disable "broad-except" as we want to catch all exceptions
```
salt/_pillar/metalk8s_solutions.py:43: [W0703(broad-except), _load_solutions] Catching too general exception Exception
salt/_pillar/metalk8s_solutions.py:50: [W0703(broad-except), _load_solutions] Catching too general exception Exception
salt/_pillar/metalk8s_solutions.py:70: [W0703(broad-except), _load_solutions] Catching too general exception Exception
```
Disable "unused-argument" for `ext_pillar` as we need a specific
function signature
```
salt/_pillar/metalk8s_solutions.py:81: [W0613(unused-argument), ext_pillar] Unused argument 'minion_id'
```
Add missing import for `SaltInvocationError` exception
```
salt/_runners/metalk8s_saltutil.py:136: [E0602(undefined-variable), orchestrate_show_sls] Undefined variable 'SaltInvocationError'
```
Add one more blank line
```
salt/_auth/kubernetes_rbac.py:24: [E8302(expected-2-blank-lines-found-0), ] PEP8 E302: expected 2 blank lines, found 1
```
Remove unused import
```
salt/_auth/kubernetes_rbac.py:1: [W0611(unused-import), ] Unused import base64
salt/_auth/kubernetes_rbac.py:10: [W0611(unused-import), ] Unused ApiException imported from kubernetes.client.rest
```

Disable "unused-import" as we import to check for virtual
```
salt/_auth/kubernetes_rbac.py:15: [W0611(unused-import), ] Unused import requests
```
Add missing import for `CommandExecutionError` exception
```
salt/_auth/kubernetes_rbac.py:57: [E0602(undefined-variable), _check_auth_args.wrapped] Undefined variable 'CommandExecutionError'
```
Disable "unused-argument" for `auth` and `groups` as we need a specific
function signature
```
salt/_auth/kubernetes_rbac.py:198: [W0613(unused-argument), groups] Unused argument 'password'
```

Discard `kwargs` argument as it does not change the function signature
```
salt/_auth/kubernetes_rbac.py:180: [W0613(unused-argument), auth] Unused argument 'kwargs'
salt/_auth/kubernetes_rbac.py:198: [W0613(unused-argument), groups] Unused argument 'kwargs'
```
Use six module from salt instead of six directly
```
salt/_beacons/metalk8s_kubeconfig_info.py:4: [E9405(blacklisted-external-import), ] Uses of an external blacklisted import 'six': Please use 'import salt.ext.six as six'
```
Use `from` in exception instead of formatting with the exception message
```
salt/_renderers/metalk8s_kubernetes.py:41: [W0707(raise-missing-from), _step_name] Consider explicitly re-raising using the 'from' keyword
```
Disable "unused-argument" for `render` as we need a specific function
signature
```
salt/_renderers/metalk8s_kubernetes.py:73: [W0613(unused-argument), render] Unused argument 'saltenv'
salt/_renderers/metalk8s_kubernetes.py:73: [W0613(unused-argument), render] Unused argument 'sls'
```

Discard `kwargs` argument as it does not change the function signature
```
salt/_renderers/metalk8s_kubernetes.py:73: [W0613(unused-argument), render] Unused argument 'kwargs'
```
Discard `kwargs` argument as it does not change the function signature
```
salt/_roster/kubernetes_nodes.py:14: [W0613(unused-argument), targets] Unused argument 'kwargs'
```
Add a dedicated `pylintrc` file for all Salt Python modules and run a
pylint on all thoses files with tox `lint-python` as well as
`./doit.sh lint:python` so that we do some linting check in the CI

Fixes: #2649
@TeddyAndrieux
Copy link
Collaborator Author

/approve

@bert-e
Copy link
Contributor

bert-e commented Feb 5, 2021

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • one peer

Peer approvals must include at least 1 approval from the following list:

The following options are set: approve

@bert-e
Copy link
Contributor

bert-e commented Feb 8, 2021

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

  • ✔️ development/2.8

The following branches will NOT be impacted:

  • development/1.0
  • development/1.1
  • development/1.2
  • development/1.3
  • development/2.0
  • development/2.1
  • development/2.2
  • development/2.3
  • development/2.4
  • development/2.5
  • development/2.6
  • development/2.7

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

  • Any commit you add on the source branch will trigger a new cycle after the
    current queue is merged.
  • Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

@bert-e
Copy link
Contributor

bert-e commented Feb 8, 2021

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/2.8

The following branches have NOT changed:

  • development/1.0
  • development/1.1
  • development/1.2
  • development/1.3
  • development/2.0
  • development/2.1
  • development/2.2
  • development/2.3
  • development/2.4
  • development/2.5
  • development/2.6
  • development/2.7

Please check the status of the associated issue None.

Goodbye teddyandrieux.

@bert-e bert-e merged commit 885810a into development/2.8 Feb 8, 2021
@bert-e bert-e deleted the improvement/add-salt-python-lint branch February 8, 2021 08:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
complexity:medium Something that requires one or few days to fix kind:debt Technical debt python Pull requests that update Python code topic:salt Everything related to SaltStack in our product
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add linting for all Python in Salt directory in CI
4 participants