default-scraper

Web Scraper


License
MIT
Install
pip install default-scraper==1.0.1

Documentation

default-scraper

Python Web Scraper

Features

  • Scrap all search results for a keyword entered as an argument.
  • Can be saved as .csv and .json.
  • Also collect user data who uploaded contents included in search results.

Usage

Install

pip install git+https://github.com/bigpicture-kr/default-scraper.git

It may require authentication before installing since default-scraper is a private repository of bigpicture-kr organization.

Scrap Instagram contents in python script

from default_scraper.instagram.parser import InstagramParser
USERNAME = ""
PASSWORD = ""
KEYWORD = ""
parser = InstagramParser(USERNAME, PASSWORD, KEYWORD, False)
parser.run()

Scrap Instagram contents using bash command

Run following command to scrap contents from Instagram:

python main.py --platform instagram --keyword {KEYWORD} [--output_file OUTPUT_FILE] [--all]

Use --all or -a option to also scrap unstructured fields.

Data description

Instagram

  • Structured fields
    • pk
    • id
    • taken_at
    • media_type
    • code
    • comment_count
    • user
    • like_count
    • caption
    • accessibility_caption
    • original_width
    • original_height
    • images
  • Some fields may be missing depending on Instagram's response data.

Future works

  • Will support scraping from more platform services.