Commits

Breadcrumbs

History for

on9bd3d7d

User selector

All users

Datepicker

All time

Commit History

Commits on Mar 31, 2015

• Added new IsIgnoreRobotsDotTextIfRootDisallowedEnabled config value which completely ignores the robots.txt file if it blocks the root. Ie.. act as if there is no robots.txt.

  sjdirect

  committed
  9bd3d7d

Commits on Jan 30, 2015

• Updated urls to localhost.fiddler:1111 so all traffic goes through fiddler. This allows the unit and integration tests to run without the need to run the SiteSimulator website.

  sjdirect

  committed
  94b8491

Commits on Jan 28, 2015

• readded crawl delay to demo

  sjdirect

  committed
  dba93a1

Commits on Jan 23, 2015

• fixed issue sjdirect#56, fixes issue with querystring on root in robots.txt blocking the entire site.

  sjdirect

  committed
  5d89622

Commits on Jan 21, 2015

• Removed logger var in log4net layout

  sjdirect

  committed
  f79aa3e

Commits on Jan 11, 2015

• Comment out all async code.

  sjdirect

  committed
  77d8d48

Commits on Jan 4, 2015

• Ran the memory profiler and my conclusion was that the gcServer config does not help with the overall used memory. It actually seemed to add more fragmentation than not having any entry in the web.…

  Show description for 6adfb7c
  sjdirect

  committed
  6adfb7c

Commits on Dec 31, 2014

• Changed useragent to match ie11's useragent.

  sjdirect

  committed
  f209280

• readded min crawl delay on demo

  sjdirect

  committed
  144f7e0

Commits on Dec 29, 2014

• Removed some project references with resharper, NOT Test.Unit and SiteSimulator since those caused issues.

  sjdirect

  committed
  5acc9bd

Commits on Dec 28, 2014

• Added basic retries, fixes issue sjdirect#47

  sjdirect

  committed
  a798626

Commits on Nov 19, 2014

• Removed logger from end of each log statement since everything is now "AbotLogger"

  sjdirect

  committed
  bf2a48d

• based off of pull request 42. Also added a _config.IsSendingCookiesEnabled to turn this on/off from config file.

  sjdirect

  committed
  d7872da

Commits on Nov 10, 2014

• Made memory management changes based on the pull request sjdirect#39

  sjdirect

  committed
  4822ee6

Commits on Oct 10, 2014

• Added <preserveLogFileNameExtension value="true" />

  sjdirect

  committed
  74afb41

Commits on Aug 17, 2014

• Upgraded to vs 2013 and .net 4.5

  sjdirect

  committed
  a6dac69

• Changing assembly versions to 1.5.0.0

  sjdirect

  committed
  c8d418c

Commits on Aug 15, 2014

• Removed abot from root logger. this fixes sjdirect#13

  sjdirect

  committed
  1c37514

Commits on Jan 29, 2014

• Updated nuget package versions

  sjdirect

  committed
  92fa1c6

• Changed to v1.2.3.1

  sjdirect

  committed
  b1f7627

• Update nuget tranformations

  sjdirect

  committed
  4901831

Commits on Jan 25, 2014

• Changed all abot assemblyinfo files to version 1.2.3

  sjdirect

  committed
  2bd3227

Commits on Nov 6, 2013

• Revert "Added IsDeepCrawlEnabled, not used by crawled yet"

  Show description for 8665023
  sjdirect

  committed
  8665023

Commits on Oct 23, 2013

• Added IsDeepCrawlEnabled, not used by crawled yet

  sjdirect

  committed
  9e2a11c

Commits on Sep 3, 2013

• Sync up with some of Ilyas changes. Added IsRespectRobotsNoFollowEnabled and IsRespectAnchorRelNoFollowEnabled. Left out (DeepCrawlScheduler and Http async client, etc..)

  sjdirect

  committed
  85e090a

Commits on Aug 8, 2013

• Manually Merged 1.2 branch (still under construction) into 1.3

  sjdirect

  committed
  18d3a65

• Initial checkin from svn/trunk google code rev 470, stable 1.1.1

  ssjonesgd

  committed
  1749206

Loading