AliExpress Web Scraper Development

Project Title: AliExpress Product Data Scraper Project Overview: This project involves developing a Python script to automatically extract product data from AliExpress.com. The script will read product URLs and associated data from a MySQL database, scrape relevant information, and then update the database with the scraped results. The scraping process will be managed based on the `scraping_active` status and the `last_scrap_date` in the database. Project Goals: Develop a Python script that can efficiently and reliably scrape product data from AliExpress.com. The script should interact with a specified MySQL database to fetch product URLs and update the scraped data. The script should be able to handle potential errors during scraping (e.g., website changes, network issues). The script should be modular and maintainable for future updates. Specific Requirements: Data to Extract: Product SKU/URL: Product SKU/URL (will be retrieved from database). Sale Price: The standard price of the product. Offer Price: The current promotional price of the product (if available). Overall Rating: The average rating of the product (e.g., 4.5 out of 5). Number of Reviews: The total number of reviews for the product. Total Sales: The total number of sales for the product. MySQL Database Interaction: The script must connect to a specified MySQL database. The database credentials will be provided to the developer. The script should query the alisync table to retrieve product data. Only products where scraping_active = 'yes' and last_scrap_date is older than 5 days from the current date should be processed. After scraping the data, the script must update the following fields in the alisync table: Columns: `sale_price` `offer_price` `overall_rating` `number_reviews` `total_sales` `last_scrap_date` (should be updated with the current date and time) `scrap_status` should be set to 1 after successful scraping and 2 if fails Error Handling: The script should implement robust error handling to manage issues such as: Network connection problems. Website structure changes on AliExpress. Missing data fields. Database connection issues. Log detailed error messages to a file for debugging purposes. The script should not stop execution completely if it encounters an error with a single product; it should log the error and proceed to the next product. Scraping Logic: The script should be able to handle dynamic content and JavaScript rendered elements. The scraper needs to be robust against changes to the AliExpress website structure. The script must comply with AliExpress's terms of service and [login to view URL] rules. Use of appropriate waiting time between requests to avoid overloading the website server. The script should use a user agent to mimic a web browser to prevent blocking by AliExpress. The script must implement robust measures to avoid IP blocking or account bans from AliExpress. This includes using appropriate delays between requests, implementing IP rotation (if necessary and feasible), respecting [login to view URL] rules, and employing browser-like user agents. The developer must prioritize ethical scraping practices to ensure long-term reliability and avoid any actions that could lead to service disruption. Script Execution: The script should be executable from the command line (terminal). It should be designed to be run periodically (e.g., using cron jobs). All configurable data should be handled via config file (database credentials, log file path, ...). Python Libraries: Utilize appropriate Python libraries such as requests, BeautifulSoup4, selenium, or scrapy for web scraping, and [login to view URL] or SQLAlchemy for MySQL interaction. Technical Specifications: Programming Language: Python (Version 3.7 or later) Database: MySQL Web Scraping Library: requests, BeautifulSoup4, selenium, or scrapy (Developer's choice based on proficiency, must handle dynamic content) MySQL Library: [login to view URL] or SQLAlchemy Operating System: Should be compatible with Linux. Environment: A virtual environment is needed to be used for the installation of the libraries. Technical Requirements: The developer will be granted terminal access to the VPS to check all requirements and confirm that the VPS can run the Python Script. Missing libraries can be installed by the developer using sudo if required. The developer is required to document the libraries and the steps for setting up the virtual environment and running the script. It is expected that the virtual environment will be properly set up in a folder inside the user directory that will have permission to execute the script. Deliverables: Complete Python script with clear and well-documented code. Setup instructions to configure and execute the script. A brief explanation of the scraping logic and implementation choices. A log file of the scraping process. The developer provides the full source code and grants all rights to use the code without any restrictions, including modification and redistribution. The developer provides 30 days of post-delivery assistance, without any additional charges, to address any issues, bugs, or queries related to the script and its implementation. Additional Considerations: Scalability: The script should be designed to handle a large number of product URLs in the future. The maximum number of products to be scraped in one run, must be configurable via a config file. Maintainability: The code should be easy to modify or update in the future. Ethical Scraping: The script should be designed to not overload the server and respect AliExpress's terms of service and [login to view URL] file. Timeline: This will depend on the developer's availability and the complexity of the script. Please provide an estimated timeline when applying. Payment: A fixed payment will be negotiated after the developer confirms that the VPS can run the Python Script. Payment milestones may be considered. Communication: We expect regular updates on the progress of the project and a clear channel for communication. Communication could be handled via: WhatsApp, Skype, TeamViewer, or other commonly used channels. Note on Bidding and Pricing: Please provide a realistic and well-considered quote for this project. Bids that appear significantly lower than the expected effort and complexity of the project will be viewed as an attempt to secure the project with the intention of renegotiating the price later. This practice is unacceptable and will result in the disqualification of the developer and a report to the platform. The price submitted is expected to be a firm and final quote for the scope of work detailed in this project description. Only bids that demonstrate a clear understanding of the project requirements, a commitment to fair and transparent pricing, and evidence of relevant experience will be considered. Due to my own extensive software development background, I will carefully evaluate the proposed approach and the ability of each developer to deliver a high-quality solution within the given budget and timeline.