bmaquet-tap-github

This is a Singer tap that produces JSON-formatted data from the GitHub API following the Singer spec.

This tap:

• Pulls raw data from the GitHub REST API
• Extracts the following resources from GitHub for a single repository:
  • Assignees
  • Collaborators
  • Commits
  • Issues
  • Pull Requests
  • Comments
  • Reviews
  • Review Comments
  • Stargazers
• Outputs the schema for each resource
• Incrementally pulls data based on the input state

Quick start

1. Install

   We recommend using a virtualenv:

   > virtualenv -p python3 venv
   > source venv/bin/activate
   > pip install tap-github

2. Create a GitHub access token

   Login to your GitHub account, go to the Personal Access Tokens settings page, and generate a new token with at least the repo scope. Save this access token, you'll need it for the next step.

3. Create the config file

   Create a JSON file containing the access token you just created and specify which repository/organization you want to fetch:

   • If repository is specified in config, the tap will only fetch this specific repo
   • If organization is specified in config, the tap will fetch all repo's from this organization
   • If neither repository nor organization is specified in config, the tap will fetch all repo's from all organizations available

   Each repo path should be space delimited. The repo path is relative to your API endpoint (default is https://github.com/). For example the path for this repository is BenjMaq/bmaquet-tap-github.

   e.g.

   To fetch all repos from all available organizations:

   {"access_token": "your-access-token"}

   To fetch all repos from a specific organization:

    {"access_token": "your-access-token",
     "organization": "BenjMaq"}

   To fetch a specific or a few specific repo's:

    {"access_token": "your-access-token",
     "repository": "BenjMaq/bmaquet-tap-github BenjMaq/another-repo"}

4. Run the tap in discovery mode to get properties.json file

   tap-github --config config.json --discover > properties.json

5. In the properties.json file, select the streams to sync

   Each stream in the properties.json file has a "schema" entry. To select a stream to sync, add "selected": true to that stream's "schema" entry. For example, to sync the pull_requests stream:

   ...
   "tap_stream_id": "pull_requests",
   "schema": {
     "selected": true,
     "properties": {
       "updated_at": {
         "format": "date-time",
         "type": [
           "null",
           "string"
         ]
       }
   ...
   

6. Run the application

   tap-github can be run with:

   tap-github --config config.json --properties properties.json

Copyright © 2018 Stitch