Robots.TXT Checker

We have many web sites that we maintain and enhance and to do that we host both development servers to validate work is completed and the production site, which is what end-users see. We need the dev sites to stay hidden from users (and Google) and the production sites stay visible by users (and Google). We want to build an app that monitors these URLs on three factors and sends an email alert if there is an issue so we can address quickly. The three factors are: 1) That the /[login to view URL] file matches what we think it should be 2) That the HTTP response code is what it should be (usually 401 or 200) 3) That the SSL certificate if valid for the domain and not expiring in the next X days. We would give you a development server on our Azure infrastructure that you could log into to do the work. It would have access to our Active Directory controller (needed for authentication). My goal for the project would be to deliver it in two phases. The first phase would focus on the data model and the scheduled review of the data. We would want this part to be done inside of a MVC app because the second phase would be to extend the app to hvae a front-end that allows you to edit the configuration in the database.