-
- GitHub
- GitLab
- Bitbucket
-
By logging in you accept
our terms of service
and privacy policy
Board games data scraping and processing
pip install ludoj-scraper==1.10.18
Scraping data about board games from the web. View the data live at Recommend.Games! Install via
pip install board-game-scraper
bga)bgg)dbpedia)luding)spielen)wikidata)Requires Python 3. Make sure Pipenv is installed and create the virtual environment:
python3 -m pip install --upgrade pipenv
pipenv install --dev
pipenv shell
Run a spider like so:
JOBDIR="jobs/${SPIDER}/$(date --utc +'%Y-%m-%dT%H-%M-%S')"
scrapy crawl "${SPIDER}" \
--output 'feeds/%(name)s/%(time)s/%(class)s.csv' \
--set "JOBDIR=${JOBDIR}"
where $SPIDER is one of the IDs above.
Run all the spiders with the run_all.sh script. Get a list of
the running scrapers' PIDs with the processes.sh script. You
can close all the running scrapers via
./processes.sh stop
and resume them later.
You can run scrapy check to perform contract tests for all spiders, or
scrapy check $SPIDER to test one particular spider. If tests fails,
there most likely has been some change on the website and the spider needs
updating.
If you are interested in using any of the datasets produced by this scraper, take a look at the BoardGameGeek guild. A subset of the data can also be found on Kaggle.
The Tidelift Subscription provides access to a continuously curated stream of human-researched and maintainer-verified data on open source packages and their licenses, releases, vulnerabilities, and development practices.
Learn more →Something wrong with this page? Make a suggestion
Export .ABOUT file for this package
Last synced: 2019-11-05 13:50:08 UTC
Login to resync this project