Creeper is a simple web crawler.
./creeper.py -u [seed url] [options]
-u Seed URL to start with.
-h This help output.
-e Elastic search host.
-p Sticky to URL path.
-d Enable dynamic page processing.
-f Make creeper follow outside links.
-c Clear out visited and unvisited domains lists.