target-bigquery
A Singer target that writes data to Google BigQuery.
How to use it
target-bigquery
works together with any other Singer Tap to move
data from sources like Braintree, Freshdesk and Hubspot to
Google BigQuery.
Step 1: Activate the Google BigQuery API
(originally found in the Google API docs)
-
Use this wizard to create or select a project in the Google Developers Console and activate the BigQuery API. Click Continue, then Go to credentials.
-
On the Add credentials to your project page, click the Cancel button.
-
At the top of the page, select the OAuth consent screen tab. Select an Email address, enter a Product name if not already set, and click the Save button.
-
Select the Credentials tab, click the Create credentials button and select OAuth client ID.
-
Select the application type Other, enter the name "Singer BigQuery Target", and click the Create button.
-
Click OK to dismiss the resulting dialog.
-
Click the Download button to the right of the client ID.
-
Move this file to your working directory and rename it client_secret.json.
Step 2: Configure
Create a file called config.json
in your working directory,
following config.sample.json. The required
parameters are the project name project_id
, the dataset name
dataset_id
, and table name table_id
.
Step 3: Install and Run
First, make sure Python 3 is installed on your system or follow these installation instructions for Mac or Ubuntu.
target-bigquery
can be run with any Singer Tap, but we'll use
tap-fixerio
- which pulls currency exchange rate data
from a public data set - as an example.
These commands will install tap-fixerio
and target-bigquery
with
pip and then run them together, piping the output of tap-fixerio
to
target-bigquery
:
› pip install target-bigquery tap-fixerio
› tap-fixerio | target-bigquery -c config.json
INFO Replicating the latest exchange rate data from fixer.io
INFO Tap exiting normally
If you're using a different Tap, substitute tap-fixerio
in the final
command above to the command used to run your Tap.
target-bigquery
will attempt to open a new window or tab in your
default browser. If this fails, copy the URL from the console and
manually open it in your browser.
If you are not already logged into your Google account, you will be
prompted to log in. If you are logged into multiple Google accounts,
you will be asked to select one account to use for the
authorization. Click the Accept button to allow target-bigquery
to
access your Google BigQuery table. You can close the tab after the signup flow
is complete.
The data will be written to the table specified in your config.json
.
Copyright © 2018 Grayson Williams