1cbyc Web Scraper is a Python-based tool designed to collect data from websites. It uses the requests and BeautifulSoup libraries to retrieve and parse web pages, and stores the extracted log in an SQLite db.

• can scrape from multiple web pages
• can handle pagination
• can store scraped data in an SQLite db (adding support for more soon)
• can mimic a web browser by setting custom headers
• (not to be a brag but) i added a way to classify the scraped data as individual texts
• (since i have lazy pals) i added a way to read the sqlite file without stressing about sqlite on your machine
• adding more features soon

1. just clone my repository.

git clone https://github.com/1cbyc/1cbyc-web-scraper.git
cd 1cbyc-web-scraper

2. then download the required packages.

pip install -r requirements.txt

1. simply update the base_url and num_pages variables in main.py to match the target website and the number of pages you want to scrape.

base_url = 'http://nsisong.com/page/'  # you can replace with the actual base URL
num_pages = 5  # adjust the number of pages to scrape based on the target website

2. then, run the scraper.

python main.py

3. make sure to check the console output to see the progress and results of the scraping.

4. also, to view the scraped data, you can use the provided function in database.py to print all data:

from scraper.database import print_all_data
print_all_data()

like just download and install the db browser for SQLite:

1. go to the DB Browser for SQLite website.
2. then, download and install the version suitable for your pc.

1. open the DB browser.
2. click "Open Database" and go to the data dir.
3. then, select the desired .db file you wanna check and click "open".

1. you can use the "browse data" tab to view the contents of the data table.
2. you can also run SQL queries using the "execute SQL" tab.

1. since i have pushed the project for general use. you can now visit it by webscraper.nsisong.com to get started. however, to do that, since i did not push the "data" folder in my local machine to github, i did this in the command: mkdir -p data && python app.py also, i wrote a deployment shell file you should check for deploy.sh in this repo.
2. I am warning you guys to use SQLite DB browser to read the file, else, you're on your own.

to be honest, i want you guys to fork this repository, make improvements, and submit pull requests. probably we'd get a v2.1 release faster. suggest new features too, and i promise to work on it (if it makes sense).