Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert bash scripts to Python #7465

Merged
merged 156 commits into from
Sep 8, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
156 commits
Select commit Hold shift + click to select a range
6ac0724
initial
Aug 17, 2019
13c37f9
fix lint
Aug 18, 2019
9ae6f2e
fix
Aug 18, 2019
2eca67e
fix
Aug 18, 2019
c9b6f98
conver
Aug 19, 2019
7218092
fix
Aug 19, 2019
9250dcb
fix
Aug 19, 2019
9d33246
convert
Aug 19, 2019
41113da
fix
Aug 19, 2019
4fe2b2d
Merge branch 'develop' into convert_bash_1
Aug 20, 2019
406129e
Merge branch 'develop' into convert_bash_1
Aug 21, 2019
beb3c52
add
Aug 22, 2019
77e21d7
convert
Aug 22, 2019
c5e3c97
fix lint
Aug 22, 2019
a5eebeb
convert
Aug 22, 2019
f9a8586
convert
Aug 22, 2019
afb7068
Merge branch 'develop' into convert_bash_1
Aug 23, 2019
dc1845c
convert
Aug 23, 2019
eee27b9
convert
Aug 24, 2019
a6dc5bb
convert
Aug 24, 2019
e88fd13
address
Aug 24, 2019
c373919
fix path
Aug 24, 2019
06dc081
convert
Aug 24, 2019
6b57f85
Merge branch 'develop' into convert_bash_1
Aug 25, 2019
845a94c
remove files
Aug 25, 2019
4df7247
convert
Aug 25, 2019
d3a2464
fix comment
Aug 25, 2019
275d886
convert
Aug 25, 2019
3482864
convert
Aug 25, 2019
0ab00ec
fix
Aug 25, 2019
cbb09a7
address comments
Aug 25, 2019
d2ed087
convert
Aug 25, 2019
1ced71c
convert
Aug 25, 2019
60510ae
fix lint
Aug 25, 2019
f83f917
update circleci
Aug 25, 2019
fb26396
fix
Aug 25, 2019
d0a9f56
convert
Aug 25, 2019
df0c880
fix
Aug 25, 2019
af66d95
fix
Aug 25, 2019
48c8ce8
fix
Aug 25, 2019
fcbbbb0
fix
Aug 25, 2019
0af2fc4
Merge branch 'develop' into convert_bash_1
Aug 25, 2019
d3f80a3
sync with develop
Aug 25, 2019
740eb7a
fix
Aug 25, 2019
c6ae74b
fix conflicts
Aug 26, 2019
2834e82
use parse known args
Aug 26, 2019
8bbda96
remove user
Aug 26, 2019
cd84f0b
fix pip
Aug 27, 2019
2c03e24
try subprocess on pip install
Aug 27, 2019
cd4a824
fix
Aug 27, 2019
ae51254
fix
Aug 27, 2019
a7e21c5
fix
Aug 27, 2019
e9fd69d
fix
Aug 27, 2019
b26d7dc
fix
Aug 27, 2019
33a10ce
add node path
Aug 27, 2019
8ba23a5
fix
Aug 27, 2019
db7a003
fix lint
Aug 27, 2019
f521c7a
add args
Aug 27, 2019
b6d5878
fix lint
Aug 27, 2019
3086329
merge develo
Aug 27, 2019
b912d6d
add path
Aug 27, 2019
efabda0
address comments
Aug 27, 2019
899228a
address comments
Aug 27, 2019
6044c23
fix
Aug 27, 2019
c28656b
fix
Aug 27, 2019
2d62316
fix
Aug 27, 2019
1e7bcdb
fix
Aug 27, 2019
3a97331
fix frontend
Aug 28, 2019
cfad6b3
fix
Aug 28, 2019
11ae549
fix start
Aug 28, 2019
46eb948
fix lint
Aug 28, 2019
95b6ca9
clear cache
Aug 28, 2019
7130eb7
revert
Aug 28, 2019
13155a5
remove pylint
Aug 28, 2019
86647a8
remove pylint path
Aug 28, 2019
7c47253
add path
Aug 28, 2019
efc69b0
add pylint path
Aug 28, 2019
834bffa
fix
Aug 28, 2019
0186a8a
fix
Aug 28, 2019
326eda8
use vurtual env
Aug 28, 2019
4f4b3c2
fix
Aug 28, 2019
ad9fbae
fix
Aug 28, 2019
34efc74
add url in travis yaml
Aug 28, 2019
2d3d642
test
Aug 28, 2019
dcf6ec3
test
Aug 28, 2019
b532ba0
fix
Aug 28, 2019
f58f27a
test
Aug 28, 2019
93b7cf4
test
Aug 28, 2019
01970a5
test
Aug 28, 2019
eda2856
test
Aug 28, 2019
92d39d5
test
Aug 28, 2019
98b324b
test
Aug 28, 2019
4cc089c
test
Aug 28, 2019
2f2871c
tes
Aug 28, 2019
14d2659
fi
Aug 28, 2019
862490c
fix
Aug 28, 2019
1ec120f
test
Aug 28, 2019
9386c0c
fix conflicts
Aug 29, 2019
11d891d
fix
Aug 29, 2019
07eaf2b
Merge branch 'develop' into convert_bash_1
Aug 30, 2019
ce7d114
fix
Aug 30, 2019
6f04c1b
fix test
Aug 30, 2019
767d97e
fix
Aug 30, 2019
82804cb
fix conflicts
Aug 31, 2019
8e43243
fix
Aug 31, 2019
fabf517
address comments
Aug 31, 2019
8b7bb5a
fix
Aug 31, 2019
057f2e6
fix node
Aug 31, 2019
8003bfe
fix
Aug 31, 2019
25b64ad
remove vagrant
Aug 31, 2019
0492abe
fix
Aug 31, 2019
fd168ab
address comments
Aug 31, 2019
77ddcbc
fix conflicts
Aug 31, 2019
d89c3b0
fix lint
Aug 31, 2019
251669a
fix
Aug 31, 2019
b07b0d9
fix
Aug 31, 2019
03e56b7
address comments
Aug 31, 2019
770cd4f
fix
Aug 31, 2019
fafbde2
fix
Sep 1, 2019
9ec9238
fix
Sep 1, 2019
33f38a7
fix
Sep 1, 2019
35cfe60
fix
Sep 1, 2019
af6ad16
merge develop
Sep 2, 2019
55a1a99
address comments
Sep 2, 2019
c604eae
fix
Sep 2, 2019
90b50d8
fix
Sep 2, 2019
5bdc8b1
fix
Sep 2, 2019
891d088
only one backend_tests.py
Sep 2, 2019
a6e6b28
address comments
Sep 2, 2019
a42a17c
Merge branch 'develop' into convert_bash_1
Sep 2, 2019
28eaabe
fix
Sep 2, 2019
6e9b993
fix
Sep 2, 2019
7bf028d
fix
Sep 2, 2019
f0ebb60
fix
Sep 2, 2019
8fe8d20
fix errors
Sep 2, 2019
b73afbf
address comments
Sep 2, 2019
1b4638b
fix
Sep 2, 2019
64fc6c5
use None and args
Sep 2, 2019
320b65e
fix permission
Sep 2, 2019
127cead
fix
Sep 2, 2019
595be3c
fix conflicts
Sep 2, 2019
6939582
address comments
Sep 2, 2019
80a9599
fix e2e
Sep 3, 2019
5ac03b0
address comments
Sep 3, 2019
9a84e6f
fix
Sep 3, 2019
770c6f3
fix
Sep 3, 2019
d713208
skip e2e
Sep 3, 2019
c3006d2
remove yaml
Sep 3, 2019
2459c84
fix lint
Sep 3, 2019
89f61e6
use single quotes
Sep 4, 2019
07814eb
fix conflicts
Sep 7, 2019
89afe82
fix conflicts
Sep 7, 2019
6967564
test
Sep 7, 2019
6ba9902
fix linr
Sep 7, 2019
943759b
add tests
Sep 8, 2019
911d84a
use triple strings
Sep 8, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use None and args
  • Loading branch information
Rishav Chakraborty committed Sep 2, 2019
commit 64fc6c504de65e671490e84f2cbc47cc6adc5e42
7 changes: 3 additions & 4 deletions scripts/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@
import re
import shutil
import subprocess
import sys
import threading

import python_utils
Expand Down Expand Up @@ -1359,14 +1358,14 @@ def compile_typescript_files_continuously(project_dir):
return


def main(argv=None):
def main(args=None):
"""The main method of this script.

Creates a third-party directory where all the JS and CSS dependencies are
brianrodri marked this conversation as resolved.
Show resolved Hide resolved
built and stored. Depending on the options passed to the script, might also
minify third-party libraries and/or generate a build directory.
"""
options = _PARSER.parse_args(args=argv)
options = _PARSER.parse_args(args=args)
# Regenerate /third_party/generated from scratch.
safe_delete_directory_tree(THIRD_PARTY_GENERATED_DEV_DIR)
build_third_party_libs(THIRD_PARTY_GENERATED_DEV_DIR)
Expand Down Expand Up @@ -1395,4 +1394,4 @@ def main(argv=None):
# The 'no coverage' pragma is used as this line is un-testable. This is because
# it will only be called when build.py is used as a script.
if __name__ == '__main__': # pragma: no cover
main(argv=sys.argv[1:])
main(args=None)
21 changes: 10 additions & 11 deletions scripts/install_third_party_libs.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
import os
import shutil
import subprocess
import sys

# These libraries need to be installed before running or importing any script.
TOOLS_DIR = os.path.join('..', 'oppia_tools')
Expand Down Expand Up @@ -107,7 +106,7 @@ def pip_install(package, version, install_path):
install_path])


def install_skulpt(argv):
def install_skulpt(args):
"""Download and install Skulpt. Skulpt is built using a Python script
brianrodri marked this conversation as resolved.
Show resolved Hide resolved
included within the Skulpt repository (skulpt.py). This script normally
requires GitPython, however the patches to it below
Expand All @@ -117,7 +116,7 @@ def install_skulpt(argv):
warning saying its dist command will not work properly without GitPython,
but it does actually work due to the patches.
"""
parsed_args = _PARSER.parse_args(args=argv)
parsed_args = _PARSER.parse_args(args=args)
brianrodri marked this conversation as resolved.
Show resolved Hide resolved
no_skulpt = parsed_args.nojsrepl or parsed_args.noskulpt

python_utils.PRINT('Checking whether Skulpt is installed in third_party')
Expand Down Expand Up @@ -196,16 +195,16 @@ def maybe_install_dependencies(
"""Parse additional command line arguments."""
if skip_installing_third_party_libs is False:
# Install third party dependencies.
main(argv=[])
main(args=[])
# Ensure that generated JS and CSS files are in place before running the
# tests.
python_utils.PRINT('Running build task with concatenation only')
build.main(argv=[])
build.main(args=[])

if run_minified_tests is True:
python_utils.PRINT(
'Running build task with concatenation and minification')
build.main(argv=['--prod_env'])
build.main(args=['--prod_env'])


def ensure_pip_library_is_installed(package, version, path):
Expand All @@ -225,7 +224,7 @@ def ensure_pip_library_is_installed(package, version, path):
pip_install(package, version, exact_lib_path)


def main(argv=None):
def main(args=None):
"""Install third-party libraries for Oppia."""
setup.main()
setup_gae.main()
brianrodri marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -256,16 +255,16 @@ def main(argv=None):
# 374076889.
subprocess.call([common.NPM_PATH, 'dedupe'])

install_skulpt(argv)
install_skulpt(args)
brianrodri marked this conversation as resolved.
Show resolved Hide resolved

# Install pre-commit script.
brianrodri marked this conversation as resolved.
Show resolved Hide resolved
python_utils.PRINT('Installing pre-commit hook for git')
pre_commit_hook.main(argv=['--install'])
pre_commit_hook.main(args=['--install'])

# Install pre-push script.
python_utils.PRINT('Installing pre-push hook for git')
pre_push_hook.main(argv=['--install'])
pre_push_hook.main(args=['--install'])


if __name__ == '__main__':
main(argv=sys.argv[1:])
main(args=None)
6 changes: 3 additions & 3 deletions scripts/pre_commit_hook.py
Original file line number Diff line number Diff line change
Expand Up @@ -126,14 +126,14 @@ def _revert_changes_in_package_lock_file():
raise ValueError(err_unstage_cmd)


def main(argv=None):
def main(args=None):
"""Main method for pre-commit hook that checks files added/modified
in a commit.
"""
parser = argparse.ArgumentParser()
parser.add_argument('--install', action='store_true', default=False,
help='Install pre_commit_hook to the .git/hooks dir')
args = parser.parse_args(args=argv)
args = parser.parse_args(args=args)
if args.install:
_install_hook()
return
Expand All @@ -150,4 +150,4 @@ def main(argv=None):


if __name__ == '__main__':
main(argv=sys.argv[1:])
main(args=None)
6 changes: 3 additions & 3 deletions scripts/pre_commit_linter.py
Original file line number Diff line number Diff line change
Expand Up @@ -3147,11 +3147,11 @@ def _print_complete_summary_of_errors():
python_utils.PRINT(error_messages)


def main(argv=None):
def main(args=None):
"""Main method for pre commit linter script that lints Python, JavaScript,
HTML, and CSS files.
"""
parsed_args = _PARSER.parse_args(args=argv)
parsed_args = _PARSER.parse_args(args=args)
# Default mode is non-verbose mode, if arguments contains --verbose flag it
# will be made True, which will represent verbose mode.
verbose_mode_enabled = bool(parsed_args.verbose)
Expand Down Expand Up @@ -3198,4 +3198,4 @@ def main(argv=None):


if __name__ == '__main__':
main(argv=sys.argv[1:])
main(args=None)
6 changes: 3 additions & 3 deletions scripts/pre_push_hook.py
Original file line number Diff line number Diff line change
Expand Up @@ -356,7 +356,7 @@ def _does_diff_include_package_json(files_to_lint):
return False


def main(argv=None):
def main(args=None):
"""Main method for pre-push hook that executes the Python/JS linters on all
files that deviate from develop.
"""
Expand All @@ -365,7 +365,7 @@ def main(argv=None):
parser.add_argument('url', nargs='?', help='provided by git before push')
parser.add_argument('--install', action='store_true', default=False,
help='Install pre_push_hook to the .git/hooks dir')
args = parser.parse_args(args=argv)
args = parser.parse_args(args=args)
if args.install:
_install_hook()
return
Expand Down Expand Up @@ -408,4 +408,4 @@ def main(argv=None):


if __name__ == '__main__':
main(argv=sys.argv[1:])
main(args=None)
20 changes: 12 additions & 8 deletions scripts/pylint_extensions.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,18 +24,22 @@
import re
import sys

import python_utils
from . import docstrings_checker

_PARENT_DIR = os.path.abspath(os.path.join(os.getcwd(), os.pardir))
_PYLINT_PATH = os.path.join(_PARENT_DIR, 'oppia_tools', 'pylint-1.9.4')
sys.path.insert(0, _PYLINT_PATH)

import astroid
from pylint import checkers
from pylint import interfaces
from pylint.checkers import typecheck
from pylint.checkers import utils as checker_utils

import python_utils # isort:skip
from . import docstrings_checker # isort:skip
# pylint: disable=wrong-import-order
# pylint: disable=wrong-import-position
import astroid # isort:skip
brianrodri marked this conversation as resolved.
Show resolved Hide resolved
from pylint import checkers # isort:skip
from pylint import interfaces # isort:skip
from pylint.checkers import typecheck # isort:skip
from pylint.checkers import utils as checker_utils # isort:skip
# pylint: enable=wrong-import-position
# pylint: enable=wrong-import-order


def read_from_node(node):
Expand Down
8 changes: 4 additions & 4 deletions scripts/run_backend_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -308,7 +308,7 @@ def _get_test_target_classes(path):
return result


def main(argv=None):
def main(args=None):
"""Run the tests."""
setup.main()
setup_gae.main()
brianrodri marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -321,7 +321,7 @@ def main(argv=None):
import dev_appserver
dev_appserver.fix_sys_path()

parsed_args = _PARSER.parse_args(args=argv)
parsed_args = _PARSER.parse_args(args=args)
if parsed_args.generate_coverage_report:
python_utils.PRINT(
'Checking whether coverage is installed in %s'
Expand All @@ -333,7 +333,7 @@ def main(argv=None):
'coverage', '4.5.4',
os.path.join(common.OPPIA_TOOLS_DIR, 'coverage-4.5.4'))

build.main(argv=[])
build.main(args=[])

# Compile typescript files.
python_utils.PRINT('Compiling typescript...')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't @ankita240796 make this optional? Please don't forget to respect that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expand Down Expand Up @@ -499,4 +499,4 @@ def main(argv=None):


if __name__ == '__main__':
main(argv=sys.argv[1:])
main(args=None)
11 changes: 5 additions & 6 deletions scripts/run_e2e_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
import re
import shutil
import subprocess
import sys
import time

import python_utils
Expand Down Expand Up @@ -102,14 +101,14 @@ def cleanup():
python_utils.PRINT('Done!')


def main(argv=None):
def main(args=None):
"""Runs the end to end tests."""
setup.main()
setup_gae.main()
if os.environ.get('TRAVIS'):
install_chrome_on_travis.main()

parsed_args = _PARSER.parse_args(args=argv)
parsed_args = _PARSER.parse_args(args=args)
install_third_party_libs.maybe_install_dependencies(
parsed_args.skip_install, parsed_args.run_minified_tests)

Expand Down Expand Up @@ -145,7 +144,7 @@ def main(argv=None):
python_utils.PRINT(
re.sub(r'\'DEV_MODE\': .*', constants_env_variable, line),
end='')
build.main(argv=['--prod_env'])
build.main(args=['--prod_env'])
app_yaml_filepath = 'app.yaml'
else:
dev_mode = 'true'
Expand All @@ -157,7 +156,7 @@ def main(argv=None):
python_utils.PRINT(
re.sub(r'\'DEV_MODE\': .*', constants_env_variable, line),
end='')
build.main(argv=[])
build.main(args=[])
app_yaml_filepath = 'app_dev.yaml'

# Start a selenium server using chromedriver 2.41.
Expand Down Expand Up @@ -249,4 +248,4 @@ def main(argv=None):


if __name__ == '__main__':
main(argv=sys.argv[1:])
main(args=None)
11 changes: 5 additions & 6 deletions scripts/run_frontend_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
import argparse
import os
import subprocess
import sys

import python_utils

Expand Down Expand Up @@ -50,12 +49,12 @@
action='store_true')


def main(argv=None):
def main(args=None):
"""Runs the frontend tests."""
setup.main()
setup_gae.main()

parsed_args = _PARSER.parse_args(args=argv)
parsed_args = _PARSER.parse_args(args=args)
install_third_party_libs.maybe_install_dependencies(
parsed_args.skip_install, parsed_args.run_minified_tests)
common.print_each_string_after_two_new_lines([
Expand All @@ -64,7 +63,7 @@ def main(argv=None):
'on your filesystem.',
'Running test in development environment'])

build.main(argv=[])
build.main(args=[])

subprocess.call([
os.path.join(common.NODE_MODULES_PATH, 'karma', 'bin', 'karma'),
Expand All @@ -73,7 +72,7 @@ def main(argv=None):
if parsed_args.run_minified_tests is True:
python_utils.PRINT('Running test in production environment')

build.main(argv=['--prod_env', '--minify_third_party_libs_only'])
build.main(args=['--prod_env', '--minify_third_party_libs_only'])

subprocess.call([
os.path.join(common.NODE_MODULES_PATH, 'karma', 'bin', 'karma'),
Expand All @@ -84,4 +83,4 @@ def main(argv=None):


if __name__ == '__main__':
main(argv=sys.argv[1:])
main(args=None)
7 changes: 3 additions & 4 deletions scripts/run_performance_tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
import atexit
import os
import subprocess
import sys
import time

import python_utils
Expand Down Expand Up @@ -90,12 +89,12 @@ def run_performance_test(test_name, xvfb_prefix):
'--test_target=core.tests.performance_tests.%s' % test_name])


def main(argv=None):
def main(args=None):
"""Main function to run the performance tests."""
setup.main()
setup_gae.main()

parsed_args = _PARSER.parse_args(args=argv)
parsed_args = _PARSER.parse_args(args=args)
install_third_party_libs.maybe_install_dependencies(
parsed_args.skip_install, parsed_args.run_minified_tests)

Expand Down Expand Up @@ -165,4 +164,4 @@ def main(argv=None):


if __name__ == '__main__':
main(argv=sys.argv[1:])
main(args=None)
Loading