基于scrapy的大规模定向爬虫
最大特色是使用代理ip,避免了网站的反爬虫策略。
其中ipport_spider实现了爬取公网代理ip,存储在redis中
soufang_spider实现了定向爬取搜房网信息和图片，从redis中获取代理ip
爬取的数据存储在mysql和mongo中
-
Notifications
You must be signed in to change notification settings - Fork 0
gangly/webcrawler
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
基于scrapy的大规模定向爬虫
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published