This library is a php web crawler which takes collection of URLs and DOM selects to crawl through the webpages and executing customized analyzers on each page.
Install this library using composer :
composer require mehrabx/web-crawler
In current version use xpath expressions to select element
//set list of URLs and selects DOM elements of each URL page
$urls = [
'https://test.exp/?page=1' => ["//img[@class='type1']","//a[@class='type1']"],
'https://test.exp/?page=2' => ["//img[@class='type2'"],
'https://test.exp/?page=3' => "//img[@class='type3']",
];
//return array of results
return \Crawler\Facades\CrawlFacade::make($urls)->start() ;
To avoid being blocked by the target url you can set sleep time between crawling each url :
$urls = [
'https://test.exp/?page=1' => ["//img[@class='type1']","//a[@class='type1']"],
'https://test.exp/?page=2' => ["//img[@class='type2'"],
];
//set 5 seconds sleep time
return \Crawler\Facades\CrawlFacade::make($urls)->sleep(10)->start() ;
You can set default select. URLs that have no selects can use it :
$urls = [
'https://test.exp/?page=1', //this url has not select
'https://test.exp/?page=2' => ["//img[@class='type2'"],
];
return \Crawler\Facades\CrawlFacade::make($urls)
->defaultSelect("//img[@class='type1']")
->start() ;