Turning OWTF into a Python package #875

viyatb · 2017-08-19T14:32:57Z

This PR is the first of many steps to refactoring OWTF and make it installable as a Python package.
While most of the changes are self-explanatory, here are a couple of notes on the work:

Including non-Python files like templates, plugin files, or any directory which is not a Python module is a PITA. I had to add the correct paths to the MANIFEST.in file to correct that.
The new web interface was moved to its separate directory (this was done in an earlier commit). We need to deprecate the old method of rendering templates using Tornado.
I completely removed Zest, PlugnHack, WafBypasser and Proxy miner support. We need addons support in OWTF so that optional features can be easily plugged in.
I have renamed configuration to conf to separate it from the other config folder.
The OWTF current install runs a post installation step in python setup.py install. I removed the virtualenv setup completely, since now it is the user's job to run python setup.py install in a separate virtualenv for maximum compatibility.
Added Sphinx docstrings to almost every function and module in OWTF
Convert all function names to snake case.
All code is now compatible with Python3 and Python2
Fix tests
Refactor installation method to install everything to ~/.owtf.
Add Debian packaging scripts
Better Makefile

Steps for installation

(I have not yet uploaded the package to PyPi, yet!)

Create a new virtualenv, virtualenv <env> and activate the environment.
Go into OWTF directory and run python setup.py install which install OWTF as a package and starts the postsetup install script.
To run OWTF, make a new folder for your target engagement, and run OWTF as python -m owtf.

After this, OWTF should no longer be responsible for

running Postgresql on startup (user's job!)
virtualenv management (users should use it by default for separate projects)

Let me know if there are any questions.

…config

DePierre · 2017-08-19T15:37:41Z

@viyatb Very happy to see such changes :)

I won't be able to review anything until starting next week though. I hope you won't mind :s Even then it might take time to review so many changes, although most are simply moving things around.

I'm surprised regarding your comment about the PITA of including non-python files. I thought that it was fairly easy but I'm surely remembering wrong. I'll have to check next week see what I can find.

viyatb · 2017-08-19T15:51:14Z

@DePierre no problem, I'll merge it after fixing the tests. This goes on develop branch, so it is not a big deal (the branch is by default unstable and should have breaking changes).
re: adding non-Python files, I had not done this before, so I had to look it up and read the docs lol.

saganshul · 2017-08-19T15:56:53Z

@viyatb .gitsubmodule will also need changes.
Also one question, can we move owtf/webui folder out of owtf because I don't think python module will require this?

viyatb · 2017-08-19T15:59:52Z

@saganshul re gitmodule: We discussed this on Slack, we'll have to move it to the installation part (download the templates into ~/.owtf)
re webui: IMO the web interface is the one of the main parts of the tool, so it should come bundled with the package.

…tall

saganshul · 2017-08-19T16:18:54Z

@viyatb re gitmodule: yeah! we can do that.

So basically, if we have plans to render everything using ReactJS so there are two ways of working in webpack environment. ( I will remove the gulp completely because webpack will be able to handle development things.

Development: That will need to install node_modules, webpack etc. These things are only required if we are developing UI. Webpack comes if integrated development environment and working with react-redux is a ++. Only requirement is to render everything using React. https://webpack.github.io/docs/webpack-dev-server.html. Hence using this things development becomes very easier, webpack automatically detects changes using socket and only build for changes.
Also using https://webpack.github.io/docs/hot-module-replacement.html makes ReactJS development a charm.
Production: In production environment we actually don't need any of JS from /owtf/webui/src/. We will just need bundle.js, bundle.css to work with. If some user is not developing UI, he/she doesn't need to install any node_modules. Also if we are providing owtf as python package, I think we should just ship latest bundle.js and bundle.css(after we move to scss).

So moving webui/src/ directory actually make sense. IMO if user is installing/building python module, installer should not build any of JS module. Let me know your thoughts on the same.

viyatb · 2017-08-19T16:22:18Z

@saganshul You are right. The only problem there is we would have to add a hack to get the path before the owtf package is installed and add that to the UI config so that it builds the bundle.js and adds it to the appropriate path. It would be messy.

saganshul · 2017-08-19T16:44:38Z

@viyatb tell me one thing, after this PR how will user launch the UI?

viyatb · 2017-08-19T16:46:39Z

@saganshul a normal run is sufficient, python -m owtf

This is the first step of many. I would like to have different commands to run OWTF as the user wishes, say for example, to only run CLI, the user would have to run python -m owtf start cli or for web interface, python -m owtf start web

saganshul · 2017-08-19T16:54:58Z

@viyatb One thing we can do we can host bundle.js and bundle.css somewhere and wget them at the time of installation.

DePierre

Just a couple of remarks.

DePierre · 2017-08-27T20:07:59Z

owtf/db/transaction_manager.py

@@ -170,6 +170,7 @@ def get_first(self, criteria, target_id=None):
        :return:
        :rtype:
        """
+        print criteria, target_id


Seems like debug output to me.

DePierre · 2017-08-27T20:08:20Z

owtf/db/transaction_manager.py

@@ -195,7 +196,7 @@ def get_transaction(self, trans):
        :return:
        :rtype:
        """
-        if trans:
+        if (trans and len(trans) > 0):


Parenthesis are superfluous.

DePierre · 2017-08-27T20:08:33Z

owtf/db/transaction_manager.py

@@ -217,7 +218,9 @@ def get_transactions(self, transactions):
        """
        owtf_tlist = []
        for transaction_obj in transactions:
-            owtf_tlist.append(self.get_transaction(transaction_obj))
+            print self.get_transaction(transaction_obj)


Seems like debug output to me.

DePierre · 2017-08-27T20:11:08Z

owtf/db/transaction_manager.py

@@ -217,7 +218,9 @@ def get_transactions(self, transactions):
        """
        owtf_tlist = []


Wouldn't it be possible to shorten those lines with:

return [self.get_transaction(transaction) for transaction in transactions if self.get_transaction(transaction) is not None]

DePierre · 2017-08-27T20:11:43Z

owtf/db/url_manager.py

+
+    def add_urls_end(self):
+        num_urls_after = self.get_num_urls()
+        msg = str(num_urls_after - self.num_urls_before)+" URLs have been added and classified"


Missing space surrounding the + operator.

Removing the function.

DePierre · 2017-08-27T20:11:51Z

owtf/db/url_manager.py

+    def add_urls_end(self):
+        num_urls_after = self.get_num_urls()
+        msg = str(num_urls_after - self.num_urls_before)+" URLs have been added and classified"
+        logging.info(msg)


Is this debug output?

This is not, but the function is not used anyway. I am removing it.

DePierre · 2017-08-27T20:13:07Z

owtf/db/url_manager.py

+        count = self.db.session.query(models.Url(target_id=target_id)).count()
+        return count
+
+    def add_urls_start(self):


Is the name of the function accurate? I am not sure to understand its purpose when I read add_urls_start and I see that it stores the number of URLs to a class attribute.

I ported it from a commit while back because the plugin helper had a method which was calling it. I removed it from there.

DePierre · 2017-08-27T20:13:38Z

owtf/http/requester.py

        except HTTPError as Error:  # page NOT found.
            # Error is really a response for anything other than 200 OK in urllib2 :)
            self.http_transaction.set_transaction(False, raw_request[0], Error)
-        except urlError as Error:  # Connection refused?
+        except URLError as Error:  # Connection refused?


Error should be snake case and not camel case.

DePierre · 2017-08-27T20:14:08Z

owtf/http/transaction.py

@@ -17,6 +17,7 @@
 except ImportError:
    from httplib import responses as response_messages

+from owtf.lib.general import *


Wildcard import should be avoided at all cost to ease debugging and prevent scope override.

DePierre · 2017-08-27T20:15:36Z

owtf/plugin/plugin_handler.py

@@ -552,7 +552,7 @@ def process_plugins_for_target_list(self, plugin_group, status, target_list):
                                {'SomeAborted': False, 'SomeSuccessful': False, 'AllSkipped': True},
                                {target})
        else:
-            pass
+            exit(0)


I am not sure about the context. Is the user expecting OWTF to exit in such case? Is it an actual success?

Not needed, was just for debugging. Removing this.

viyatb · 2017-08-27T22:19:56Z

@DePierre @saganshul @7a ready to merge? :)

viyatb added 16 commits August 18, 2017 14:45

Updated .gitignore with OS specific files

714a14b

Initial breaking UI directory change

e5938dc

Moved the web interface to webui/ and working

4a80fc4

Fixed web interface setup

015f656

Remove zest directory and from installation, rename configuration to …

a90b276

…config

Remove ununsed agents directory

bd2341d

Rename Dockerfile to Dockerfile.dev

1165568

Remove editor specific files

cb16d27

Rename framework_config.cfg to framework.cfg

d522a20

Updated .gitignore with new paths

64d7479

Packaging OWTF into a python package, imports still broken

a0f347f

Remove plugnhack remaining files and commands

7ab4e8b

Expanded gitignore to add python packaging garbage

e9db67f

New setup.py and MANIFEST files for correct packaging

747bed3

Updated to correct tool and dictionaries path

d7408e2

Remove restricted content

ce4f9a1

viyatb requested a review from DePierre August 19, 2017 14:32

Fix merge conflicts

95413b1

Fix bug where requirements were not being install during setup.py ins…

22efb4b

…tall

Install cffi before the install script runs

fb4c74d

Fix typo

33a4c54

viyatb force-pushed the viyatb/break branch from dedc79a to 2902a51 Compare August 25, 2017 22:38

Finally fixed all tests

6bc200f

viyatb force-pushed the viyatb/break branch from b528069 to 6bc200f Compare August 26, 2017 02:17

Use makefile

8f29715

viyatb force-pushed the viyatb/break branch from 3951e40 to 8f29715 Compare August 26, 2017 02:47

viyatb added 5 commits August 26, 2017 12:58

Don't use makefile

3fc0b4a

Fix Db cleaning

8f83217

Fix spiders plugin

c44e025

Finally fix spiders passive plugin

4caf6a4

Fix more tests

173a52f

viyatb force-pushed the viyatb/break branch from aa5b524 to 173a52f Compare August 27, 2017 19:19

Add correct syntax

446ebb9

DePierre reviewed Aug 27, 2017

View reviewed changes

Address Tao's comments

f7c81bb

viyatb force-pushed the viyatb/break branch from 6e14446 to f7c81bb Compare August 27, 2017 20:49

viyatb added 3 commits August 27, 2017 16:49

Fix travis again

200223d

More refined makefile commands

e5861c3

Misc files

5b9eb8f

viyatb added 9 commits August 27, 2017 18:42

Update requirements

b551d01

Initial files for debian packaging

02e4342

Ignore deb files

5bf3a9b

Add final debian packaging files

7b05b82

Add better readme and setup.py

2df7738

Add libffi to the package list

63e08fd

Manage OWTF versions through __init__ file and setup.py

0de8c65

Some doc changes

030e0e2

Still install everything in owtf root folder

2c2eb73

viyatb merged commit 520cdc0 into develop Aug 29, 2017

viyatb deleted the viyatb/break branch September 3, 2017 20:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Turning OWTF into a Python package #875

Turning OWTF into a Python package #875

viyatb commented Aug 19, 2017 •

edited

Loading

DePierre commented Aug 19, 2017

viyatb commented Aug 19, 2017

saganshul commented Aug 19, 2017 •

edited

Loading

viyatb commented Aug 19, 2017

saganshul commented Aug 19, 2017 •

edited

Loading

viyatb commented Aug 19, 2017

saganshul commented Aug 19, 2017

viyatb commented Aug 19, 2017 •

edited

Loading

saganshul commented Aug 19, 2017

DePierre left a comment

DePierre Aug 27, 2017

DePierre Aug 27, 2017

DePierre Aug 27, 2017

viyatb Aug 27, 2017

DePierre Aug 27, 2017

viyatb Aug 27, 2017

DePierre Aug 27, 2017

viyatb Aug 27, 2017

DePierre Aug 27, 2017

viyatb Aug 27, 2017

DePierre Aug 27, 2017

viyatb Aug 27, 2017

DePierre Aug 27, 2017

DePierre Aug 27, 2017

viyatb Aug 27, 2017

DePierre Aug 27, 2017

viyatb Aug 27, 2017

viyatb commented Aug 27, 2017 •

edited

Loading

		@@ -217,7 +218,9 @@ def get_transactions(self, transactions):
		"""
		owtf_tlist = []

Turning OWTF into a Python package #875

Turning OWTF into a Python package #875

Conversation

viyatb commented Aug 19, 2017 • edited Loading

Steps for installation

DePierre commented Aug 19, 2017

viyatb commented Aug 19, 2017

saganshul commented Aug 19, 2017 • edited Loading

viyatb commented Aug 19, 2017

saganshul commented Aug 19, 2017 • edited Loading

viyatb commented Aug 19, 2017

saganshul commented Aug 19, 2017

viyatb commented Aug 19, 2017 • edited Loading

saganshul commented Aug 19, 2017

DePierre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viyatb commented Aug 27, 2017 • edited Loading

viyatb commented Aug 19, 2017 •

edited

Loading

saganshul commented Aug 19, 2017 •

edited

Loading

saganshul commented Aug 19, 2017 •

edited

Loading

viyatb commented Aug 19, 2017 •

edited

Loading

viyatb commented Aug 27, 2017 •

edited

Loading