check-my-pages is a scraping script to test large sites. We are using it to test the Spanish soon-to-be archived site and redirects system, with more than 10,000 pages. It checks redirections, http responses, analytics, files hosted in soon-to-die servers, canonical urls and more.
It checks each url in a list and creates report files about what was tested. Each file reports about a specific issue and includes the scanned url together with the result.
Please note that the script was written to check for problems detected when manually inspecting a small sample (about 1%) of the Spanish website.
This script was not built to be used as-it-is in other websites, you are supposed to modify it.
Download and install
Install the lastest version
Go to the releases page and download the last version for your 64bit operating system: Windows, Mac and Linux.
Install from source
If you have the Go compiler installed, you can download and install go with:
go get github.com/greenpeace/gpes-check-my-pages go install github.com/greenpeace/gpes-check-my-pages
Update from source
go get -u github.com/greenpeace/gpes-check-my-pages go install github.com/greenpeace/gpes-check-my-pages
If you downloaded and installed the latest version from the releases page do:
If you installed from source do:
File with list of urls
The urls file, by default
urls.csv must have all the urls you want to check. You can use a text file with 1 url per line or a csv file with the urls on the first column and without headers.
You can use ecounter to create a urls file from a sitemap.xml file.
Http info about a list of urls
If you want to obtain information about http status codes, mime-types, file sizes and redirect urls of any urls, you can use
You must use this check in a separate command like:
./check-my-pages -urls=urls.csv -http -miliseconds=100
because check-my-pages will stop after executing
This check creates a file named
httpResponses.csv with 5 fields:
- initial url
- http status code
- mime type
- file size (adds -1 if the file size is unknown)
- final url
Checking html urls
To do all the checks in
urls.csv (html urls) with all the checks use the command:
./check-my-pages -urls=urls.csv -analytics -canonical -title -linkpattern -cssjspattern -mediapattern
This repository includes a few testing urls in the file
urls.csv. Please replace them by your own.
It will create a couple of files, one per check the script is doing:
analytics.csv- Reports google analytics tracking ID
canonicals.csv- Reports the canonical url for every url
titles.csv- Reports the title for every url
linkpattern.csv- Reports on links that include a regular expression pattern. Useful to track links to specific dead sites. The default pattern can be set by the
cssjspattern.csv- Reports css and js urls that include a regular expression pattern. To detect dead css and js urls in large sites. The pattern can also be defined with the option
mediapattern.csv- Reports media links. Images, videos, audios, iframes and objects. Also use
-patternto define the urls pattern.
Optional command line configurations
-miliseconds=100 - Sets a delay of 100 miliseconds between requests (the default value)
-pattern='https?://(\w|-)+.greenpeace.org/espana/.+' - Changes the search url pattern to the regular expression.
Remove the report files
To remove the files created by check-my-pages:
Rename the report files
Renames the files created by check-my-pages adding date and time to the name.
By renaming the files you prevent the results from future reports being added to the end of each file.
If you don't have a sitemap.xml or another file with the urls, you can crawl the site:
./check-my-pages -crawl -urls=crawledurls.csv -start='https://www.fotografar.net/' -pattern='https://www.fotografar.net/.*' -miliseconds=100
It will save, in crawledurls.csv, all the urls that match the regular expression pattern and that can be obtained by crawling from the start url.
You can make as many crawls as you want to the same crawledurls.csv file. If you get repeated urls, you can use ecounter to obtain unique urls (from one or multiple concatenated files).