Web Listings Database Development

This project aims to develop a robust and automated database system that collects, stores, and manages listings from public portals related to funding and tenders programmes. The system must include automated daily web scrapers to pull new announcements and update the database efficiently. This solution will enable seamless data retrieval, analysis, and management for stakeholders. Scope of Work 1. Database Development • Design and implement a scalable, secure relational database (e.g., MySQL or similar) for storing funding and tender listings. • Database structure should accommodate fields such as: • Title of the funding or tender. • Description and eligibility criteria. • Submission deadlines. • Funding amount or programme budget. • Contact details or application process links. • Relevant documents (if applicable). 2. Web Scraping Setup • Develop web scrapers to automatically extract data from: • European Commission Funding Portal. • UK Research and Innovation Opportunities. • Configure scrapers to: • Run daily to fetch and add new announcements. • Identify and handle changes in webpage structure. • Implement robust error-handling and notification mechanisms for failed scrapes. 3. Data Processing and Input • Automate data cleaning and formatting before storage. • Ensure daily updates are seamlessly appended to the database without overwriting existing records. • Implement deduplication processes to avoid redundant data entries. • Maintain timestamps to indicate when data was last updated. 4. Optimization and User Access • Optimize the database for high-speed querying using indexing, normalization, and caching. • Include features for filtering, sorting, and exporting data (e.g., CSV or Excel formats). • Develop a user-friendly interface (if required) for accessing and analyzing data. • Ensure secure authentication and access control mechanisms. 5. Testing and Maintenance • Conduct thorough testing of the database and web scrapers under multiple scenarios to ensure reliability. • Establish a maintenance plan to handle: • Changes in portal structures or data formats. • Security updates. • Regular monitoring of daily scraper operations to ensure consistent updates. Key Responsibilities • Database System Development: • Develop and deploy a robust, scalable relational database. • Optimize for performance and scalability. • Web Scraping Implementation: • Build and deploy daily web scrapers for data extraction. • Automate daily updates with scheduling tools (e.g., Cron Jobs, Celery). • Data Integrity and Accuracy: • Ensure accuracy and completeness of collected data. • Implement workflows for deduplication and validation. • System Testing and Handover: • Provide comprehensive testing, training, and documentation. Skills Needed: • Proficiency in MySQL, PHP, or other database management systems. • Expertise in web scraping using tools like BeautifulSoup, Scrapy, Selenium, or PHP scraping libraries. • Familiarity with scheduling tools for automating daily scrapes. • Experience with Hedoku for deployment and hosting. • Knowledge of data cleaning and mining methodologies. Development Workflow 1. Requirement Analysis: • Identify fields to scrape and confirm database architecture. • Define frequency and scope of scraping (daily). 2. Design and Development: • Develop the database schema and web scraping scripts. • Configure scrapers for automated daily operations. 3. Testing: • Validate data accuracy and database performance. • Simulate real-world use cases for troubleshooting. 4. Deployment: • Deploy the database and scrapers on a secure hosting environment. • Provide user training and detailed documentation. 5. Maintenance: • Monitor performance and address issues promptly. • Update scrapers to handle changes in portal structures.