Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add websites from jekyll/wiki/sites #5

Merged
merged 1 commit into from
Aug 6, 2016
Merged

Add websites from jekyll/wiki/sites #5

merged 1 commit into from
Aug 6, 2016

Conversation

alexyorke
Copy link
Contributor

This pull request extracts the websites from https://github.com/jekyll/jekyll/wiki/sites

I also did a quick URL check on each of the links; if one didn't respond for more than 20 seconds (i.e no data sent to client at all within that timeframe) then it was removed from the list.

@abdelhai
Copy link
Owner

abdelhai commented Aug 6, 2016

@decagon Super awesome! Could you please share the script you used to check the "health" of the links?

@abdelhai abdelhai merged commit 3a4da01 into abdelhai:master Aug 6, 2016
@alexyorke alexyorke deleted the patch-1 branch August 6, 2016 11:12
@alexyorke
Copy link
Contributor Author

I just adapted some code from this SO answer: http://stackoverflow.com/questions/12199059

while read line; do; if curl --output /dev/null --silent -m 20 --head --fail "$line"; then echo "$line" else true fi done <~/urls_new.txt > urls_final.txt

However, there are some websites which show "This domain is available" however it doesn't return a 404 error, so this script wouldn't be able to detect that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants