[reboot] Turning OWTF into a Python package (#875)

This PR is the first of many steps to refactoring OWTF and make it installable as a Python package. While most of the changes are self-explanatory, here are a couple of notes on the work: - The new web interface was moved to its separate directory (this was done in an earlier commit). We need to deprecate the old method of rendering templates using Tornado. - Completely removed Zest, PlugnHack, WafBypasser and Proxy miner support. We need addons support in OWTF so that optional features can be easily plugged in. - Renamed configuration to conf to separate it from the other config folder. - The OWTF current install runs a post installation step in python setup.py install. The PR removes the virtualenv setup completely, since now it is the user's job to run python setup.py install in a separate virtualenv for maximum compatibility. - Added Sphinx docstrings to almost every function and module in OWTF - Convert all function names to snake case. - All code is now compatible with Python3 and Python2 - Fixed tests - Refactor installation method to install everything to ~/.owtf. - Add Debian packaging scripts - Better Makefile - Create a new virtualenv, virtualenv <env> and activate the environment. - Go into OWTF directory and run python setup.py install which install OWTF as a package and starts the postsetup install script. NOTE: if the user wants to run OWTF in developer mode, they need to set an environment variable, export OWTF_DEV=1 To run OWTF, make a new folder for your target engagement, and run OWTF as python -m owtf. After this, OWTF should no longer be responsible for - running Postgresql on startup (user's job!) - virtualenv management (users should use it by default for separate projects)
owtf · Aug 29, 2017 · 520cdc0 · 520cdc0
1 parent e749af8
commit 520cdc0
Show file tree

Hide file tree

Showing 1,021 changed files with 25,089 additions and 11,789 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,12 @@
+*.pyc
+*.pyc
+*.pyo
+*.tmp
+.eggs/
+.git/
+.tox/
+.ropeproject/
+.cache/
+build/
+htmlcov/
+tmp/
diff --git a/.gitignore b/.gitignore
@@ -1,16 +1,65 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+eggs/
+.eggs/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
 # Compiled source and other garbage #
 #####################################
 *.pyc
 *.swp
 
 # Directories with potential license restrictions that prevent re-distribution #
 ################################################################################
-dictionaries/restricted/*
-tools/restricted/*
+owtf/dictionaries/restricted/*
+owtf/tools/restricted/*
 
 # Dictionaries pulled from Github #
 ###################################
-dictionaries/fuzzdb
+owtf/dictionaries/fuzzdb
 
 # Tools with license restrictions that prevent re-distribution #
 ################################################################
@@ -22,8 +71,8 @@ tests/restricted/*
 
 # Directory with the output of OWTF
 ###################################
-owtf_review/*
-scans/*
+*owtf_review/*
+*scans/*
 
 # Unit Testing Framework files #
 ################################
@@ -33,12 +82,12 @@ tests/tests.log.html
 
 # Zest jars #
 #############
-zest/zest_lib/*
-zest/zest_runner_lib/*
-zest/zest.jar
-zest/zest_runner.jar
-zest/release.hash
-zest/README.md
+owtf/zest/zest_lib/*
+owtf/zest/zest_runner_lib/*
+owtf/zest/zest.jar
+owtf/zest/zest_runner.jar
+owtf/zest/release.hash
+owtf/zest/README.md
 
 # Pycharm (Jetbrains) IDE Specific files #
 ##########################################
@@ -47,7 +96,6 @@ zest/README.md
 # PyDev (Eclipse) IDE Specific Files #
 ######################################
 .metadata
-bin/
 tmp/
 *.tmp
 *.bak
@@ -67,11 +115,17 @@ local.properties
 
 # Front End(ReactJS) development Dependencies#
 #############################################
-webui/src/Report/templates/*
-webui/yarn.lock
-webui/node_modules/*
-webui/public/build/*
+owtf/webui/node_modules/*
+owtf/webui/public/build/*
+owtf/webui/src/Report/templates/*
 
 # Mac/OS specific files
 #############################################
 *.DS_Store
+
+# Deb packaging
+#############################################
+debian/owtf/
+debian/.debhelper/
+*.deb
+*.tar.gz
diff --git a/.project b/.project
diff --git a/.pydevproject b/.pydevproject
diff --git a/.travis.yml b/.travis.yml
@@ -5,6 +5,7 @@ branches:
   only:
   - master
   - develop
+  - viyatb/break
 dist:
 - trusty
 sudo:
@@ -14,48 +15,24 @@ before_install:
 - sudo apt-get install libffi-dev
 - sudo pip install pyopenssl
 install:
-- echo '\n\n' | python ./install/install.py
-- rm -rf ./tests
-- git clone https://github.com/owtf/owtf_testing
-- source /usr/local/bin/virtualenvwrapper.sh
-- workon owtf
-- pip install --upgrade -r ./owtf_testing/owtf_testing.pip
+- echo '\n\n' | python setup.py install
 before_script:
-- cp -f ./owtf_testing/profiles/plugin_web/groups.cfg ~/.owtf/configuration/profiles/plugin_web/groups.cfg
-- cp -f ./owtf_testing/profiles/plugin_net/groups.cfg ~/.owtf/configuration/profiles/plugin_net/groups.cfg
-- cp -f ./owtf_testing/profiles/plugin_aux/groups.cfg ~/.owtf/configuration/profiles/plugin_aux/groups.cfg
+- pip install mock pyHamcrest
+- cp -f ./tests/profiles/plugin_web/groups.cfg ~/.owtf/conf/profiles/plugin_web/groups.cfg
+- cp -f ./tests/profiles/plugin_net/groups.cfg ~/.owtf/conf/profiles/plugin_net/groups.cfg
+- cp -f ./tests/profiles/plugin_aux/groups.cfg ~/.owtf/conf/profiles/plugin_aux/groups.cfg
 - git config --global user.email "tasty@mac.test"
 - git config --global user.name "Tasty Test"
-- git am < ./owtf_testing/patch/0001-unittest-Patch-to-reuse-proxy-address.patch
-- "./owtf_testing/patch/skipfish.sh"
+- "./tests/patch/skipfish.sh"
 script:
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_simulation.OWTFCliSimulationTest.test_cli_simulation
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_simulation.OWTFCliSimulationTest.test_cli_no_simulation
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_nowebui.OWTFCliNoWebUITest.test_cli_no_webui
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_empty_run.OWTFCliEmptyRunTest.test_cli_empty_run
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_list_plugins.OWTFCliListPluginsTest.test_cli_list_plugins_aux
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_list_plugins.OWTFCliListPluginsTest.test_cli_list_plugins_net
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_list_plugins.OWTFCliListPluginsTest.test_cli_list_plugins_web
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_only.OWTFCliOnlyPluginsTest.test_only_one_plugin
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_only.OWTFCliOnlyPluginsTest.test_only_one_plugin_one_type
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_except.OWTFCliExceptTest.test_except
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_scope.OWTFCliScopeTest.test_cli_target_is_valid_ip
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_scope.OWTFCliScopeTest.test_cli_target_is_invalid
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_scope.OWTFCliScopeTest.test_cli_target_is_valid_http
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_scope.OWTFCliScopeTest.test_cli_target_are_mixed
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_scope.OWTFCliScopeTest.test_cli_target_are_mixed_but_web_specified
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_scope.OWTFCliScopeTest.test_cli_target_are_mixed_but_net_specified
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_type.OWTFCliTypeTest.test_cli_type_no_group_and_type_when_http_host
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_type.OWTFCliTypeTest.test_cli_type_no_group_and_type_when_host
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_type.OWTFCliTypeTest.test_cli_type_no_group_and_type_when_http_ip
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.cli.test_type.OWTFCliTypeTest.test_cli_type_no_group_and_type_when_ip
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.plugins.web.test_web.OWTFCliWebPluginTest.test_web_active
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.plugins.web.test_web.OWTFCliWebPluginTest.test_web_passive
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.plugins.web.test_web.OWTFCliWebPluginTest.test_web_semi_passive
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.plugins.web.test_web.OWTFCliWebPluginTest.test_web_external
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.plugins.web.test_web.OWTFCliWebPluginTest.test_web_grep
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.plugins.web.active.test_web_active.OWTFCliWebActivePluginTest.test_web_active_wvs_001
-- sudo $WORKON_HOME/owtf/bin/python -m unittest owtf_testing.tests_functional.plugins.web.active.test_web_active.OWTFCliWebActivePluginTest.test_web_active_wvs_006
+  - python -m unittest tests.tests_functional.cli.test_empty_run
+  - python -m unittest tests.tests_functional.cli.test_except
+  - python -m unittest tests.tests_functional.cli.test_list_plugins
+  - python -m unittest tests.tests_functional.cli.test_nowebui
+  - python -m unittest tests.tests_functional.cli.test_only
+  - python -m unittest tests.tests_functional.cli.test_scope
+  - python -m unittest tests.tests_functional.cli.test_type
+  - python -m unittest tests.tests_functional.plugins.web
 notifications:
   email: false
   irc:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -818,7 +818,7 @@
  + Implemented new CLI option for algorithm: breadth (default): runs each plugin against all targets first. And depth runs all plugins against each target first <=> Abraham Aranguren (@7a_) 
  + Massive refactoring/rewrite of significant portions of the framework to support: algorithm, multi-host targets, future net plugins, etc <=> Abraham Aranguren (@7a_) 
  + Changed Config to transparently handle multi-target configurations (i.e. self.Core.Config.Get('x') --> 'x' is relevant for the current target) <=> Abraham Aranguren (@7a_) 
- + Changed Config to transparently handle multi-target replacements (i.e. self.Core.Config.GetReplacementDict() --> Returns PLACE_HOLDER => VALUE pairs relevant for target) <=> Abraham Aranguren (@7a_) 
+ + Changed Config to transparently handle multi-target replacements (i.e. self.Core.Config.get_replacement_dict() --> Returns PLACE_HOLDER => VALUE pairs relevant for target) <=> Abraham Aranguren (@7a_) 
  + Serious refactoring to dynamically load plugins and plugin types from the file system along with a description of what they do <=> Abraham Aranguren (@7a_) 
  + Significant DB refactoring to accomodate to the new multi-host/multi-db problem, created plugin register db and improved DB simulation mode without touching disk <=> Abraham Aranguren (@7a_) 
  + Bug fix: New Plugin highlighting upon screen refresh is now working again <=> Abraham Aranguren (@7a_) 

diff --git a/agents/imap/error_log.log → CONTRIBUTING.md b/agents/imap/error_log.log → CONTRIBUTING.md
diff --git a/Dockerfile b/Dockerfile
diff --git a/Dockerfile.dev b/Dockerfile.dev
@@ -0,0 +1,43 @@
+FROM kalilinux/kali-linux-docker
+ENV DEBIAN_FRONTEND noninteractive
+RUN apt-get -y update && apt-get -y dist-upgrade && apt-get clean
+
+# Install certificates to ensure https links in wget work
+RUN apt-get -y install ca-certificates
+
+RUN apt-get -y install  xvfb \
+                        xserver-xephyr \
+                        libxml2-dev \
+                        libxslt-dev \
+                        libssl-dev \
+                        zlib1g-dev \
+                        gcc \
+                        python-all-dev \
+                        python-pip
+
+# Needed for installation of pycurl using pip in kali
+ENV PYCURL_SSL_LIBRARY=gnutls
+
+# psycopg2, pycurl dependency
+RUN apt-get -y install  postgresql-server-dev-all \
+                        postgresql-client \
+                        postgresql-client-common \
+                        postgresql  \
+                        libcurl4-openssl-dev \
+                        proxychains \
+                        tor
+
+# Install optional tools (LBD, arachni, gnutls-bin, o-saft and metagoofil)
+RUN apt-get -y install  lbd \
+                        gnutls-bin \
+                        arachni \
+                        o-saft \
+                        metagoofil
+
+
+# Install sudo, python, and Java for Zest functionality
+RUN apt-get -y install sudo git python openjdk-8-jre openjdk-8-jdk
+
+# Fix for exporting a SHELL variable in the environment
+ENV SHELL /bin/bash
+
diff --git a/MANIFEST.in b/MANIFEST.in
@@ -0,0 +1,7 @@
+recursive-include owtf/conf *
+recursive-include owtf/dictionaries *
+recursive-include owtf/plugins *
+recursive-include owtf/scripts *
+recursive-include owtf/tools *
+recursive-include owtf/webui *
+recursive-include owtf/interface/templates *