Crawled more than 70 e-commence and 17 ebank domains using Python Scrapy. scraped 10 million records. Can provide extraction sample in chat.
Hello sir,
I have gone through the requirements. I am well versed with crawling whole sites using Sitemaps, Country/Language selectors using Scrapy. I like to craft scalable and stealthy spiders which are designed with keeping the anti bot measures of modern site in mind.
Little bit about my experience, I have maintained a web crawler that have scrapped 7 million records for a single semiconductor marketplace. I have crawled more than 70 e-commence domains using Scrapy. If sites are ajax rendered, I like to simulate XHR requests whenever possible. I have experience with web scrapping using Scrapy, Scrapy-Splash, BeautifulSoup, and NodeJS for one and a half year.
I am akin to tackle and work on challenging problems. I have exposure with task management tools like Jira and versioning tools such as git/bitbucket. Can provide live extraction as skill reference.
Looking forward to hear from you, discuss problem and collaborate. Kindly feel free to chat.
Thanks for reading