CraigslistScraper

Note: CraigslistScraper is for personal use and data science only.

Updated (12/26/2023): Craigslist made a few updates a couple of months ago that broke the previous version of this library, and I ended up doing a full refactor on this project. That said, the new version 1.1.0 is not backwards compatible with version 1.0.1.

CraigslistScraper is a simple tool for scraping Craigslist. Users define what they would like to search for then CraigslistScraper can fetch and parse data from searches and individual ads.

There are no official docs, but the code-base is sub 200 lines and is documented.

Table of Contents

• Installation
• Usage
• Analyzing
• License

Installation

To install the package just run:

pip install craigslistscraper

The only requirements are Python 3.7+ and the requests and beautifulsoup4 libraries.

Usage

CraigslistScraper is built around 6 functions/classes for flexibility. These functions/classes are listed below.

For general searches:

• Search
• SearchParser
• fetch_search

For single ads/posts:

• Ad
• AdParser
• fetch_ad

SearchParser and AdParser are BeautifulSoup-like abstractions that may be useful to developers for extracting certain fields from html data.

Search and Ad are classes that lazily extract fields from user-defined searches and ads. To define a search you need at least a query and city, and to define an ad you need a url. Examples are provied below and in the examples/ folder.

fetch_search() and fetch_ad() are functional implementations that return a Search and Ad.

Below is a simple example, more examples can be found in the examples/ folder.

import craigslistscraper as cs
import json

# Define the search. Everything is done lazily, and so the 
# html is not fetched at this step.
search = cs.Search(
    query = "bmw e46",
    city = "minneapolis",
    category = "cto"
)

# This is the step that will fetch the html from the server.
search.fetch()

for ad in search.ads:
    # We fetch additional information about each ad.
    ad.fetch()

    # There is a `to_dict()` method for convenience. 
    data = ad.to_dict()

    # json.dumps is merely for pretty printing. 
    print(json.dumps(data, indent = 4))

Analyzing

Data can easily be converted to your json, csv, etc. and used in various downstream data analysis tasks.

This data can then be analyzed, some examples include:

License

Distributed under the MIT License. See LICENSE for more information.