pipelinewise-tap-twilio

Singer tap that extracts data from a Twilio API and produces JSON-formatted data following the Singer spec.

This is a PipelineWise compatible tap connector.

This tap:

• Extracts the following resources:

  • accounts
  • addresses
  • dependent_phone_numbers
  • applications
  • available_phone_number_countries
  • available_phone_numbers_local
  • available_phone_numbers_mobile
  • available_phone_numbers_toll_free
  • incoming_phone_numbers
  • keys
  • calls
  • conferences
  • conference_participants
  • outgoing_caller_ids
  • recordings
  • usage_records
  • usage_triggers
  • transcriptions
  • queues
  • message_media
  • alerts

• Extracts TaskRouter resources:

  • workspaces
  • activities
  • events
  • tasks
  • task_channels
  • task_queues
  • cumulative_statistics
  • workers
  • worker_channels
  • workflows

• Extracts Programmable Chat resources (members and chat_messages are FULL_TABLE synced, so take care syncing them, they result in a lot of request/data):

  • services
  • roles
  • chat_channels
  • members
  • chat_messages
  • users

• Outputs the schema for each resource

• Incrementally pulls data based on the input state

Streams

Standard Endpoints:

accounts

• Endpoint: https://api.twilio.com/2010-04-01/Accounts.json
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

addresses

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Addresses.json
• Parent: account
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

dependent_phone_numbers

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Addresses/{ParentId}/DependentPhoneNumbers.json
• Parent: addresses
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

applications

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Applications.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

available_phone_number_countries

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers.json
• Parent: accounts
• Primary key fields: country_code
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

available_phone_numbers_local

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/Local.json
• Parent: available_phone_number_countries
• Primary key fields: iso_country, phone_number
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

available_phone_numbers_mobile

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/Mobile.json
• Parent: available_phone_number_countries
• Primary key fields: iso_country, phone_number
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

available_phone_numbers_toll_free

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/TollFree.json
• Parent: available_phone_number_countries
• Primary key fields: iso_country, phone_number
• Replication strategy: FULL_TABLE
• Transformations: none

incoming_phone_numbers

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/IncomingPhoneNumbers.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

keys

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Keys.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

calls

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Calls.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

conferences

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Conferences.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

conference_participants

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Conferences/{ParentId}/Participants.json
• Parent: conferences
• Primary key fields: uri
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

outgoing_caller_ids

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/OutgoingCallerIds.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

recordings

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Recordings.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

transcriptions

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Transcriptions.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

queues

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Queues.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

message_media

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages/{ParentId}/Media.json
• Parent: messages
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: subresources_to_array

usage_records

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Usage/Records.json
• Parent: accounts
• Primary key fields: account_sid, category, start_date
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

usage_triggers

• Endpoint: https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Usage/Triggers.json
• Parent: accounts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: subresources_to_array

alerts

• Endpoint: https://monitor.twilio.com/v1/Alerts
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

TaskRouter Endpoints:

workspaces

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

activities

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Activities
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

events

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Events
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

tasks

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Tasks
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

task_channels

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskChannels
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

task_queues

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskQueues
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

cumulative_statistics

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskQueues/{ParentId}/CumulativeStatistics
• Parent: task_queues
• Replication strategy: FULL_TABLE
• Transformations: none

workers

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workers
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

channels

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workers/{ParentId}/Channels
• Parent: workers
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: none

workflows

• Endpoint: https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workflows
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

Programmable Chat Endpoints:

services

• Endpoint: https://chat.twilio.com/v2/Services
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

roles

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Roles
• Parent: services
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

chat_channels

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Channels
• Parent: services
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

members

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Channels/{ParentId}/Members
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: none

chat_messages

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Channels/{ParentId}/Messages
• Parent: workspaces
• Primary key fields: sid
• Replication strategy: FULL_TABLE
• Transformations: none

users

• Endpoint: https://chat.twilio.com/v2/Services/{ParentId}/Users
• Parent: services
• Primary key fields: sid
• Replication strategy: INCREMENTAL
• Transformations: none

Authentication

This tap authenticates to the Twilio API using Basic Auth.

To set up authentication simply include your Twilio account_sid and auth_token in the tap config.

Quick Start

1. Install

   python3 -m venv venv
   . venv/bin/activate
   pip install --upgrade pip
   pip install .

2. Create your tap's config.json file. The api_key is available in the twilio Console UI (see Authentication above). The date_window_days is the integer number of days (between the from and to dates) for date-windowing through the date-filtered endpoints (default = 30). The start_date is the absolute beginning date from which incremental loading on the initial load will start.

       {
           "account_sid": "YOUR_ACCOUNT_SID",
           "auth_token": "YOUR_AUTH_TOKEN",
           "start_date": "2019-01-01T00:00:00Z",
           "user_agent": "tap-twilio <api_user_email@your_company.com>",
       }

   Optionally, also create a state.json file. currently_syncing is an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.

   {
       "currently_syncing": "registers",
       "bookmarks": {
           "acounts": "2020-03-23T10:31:14.000000Z",
           "...": "2020-03-23T00:00:00.000000Z"
       }
   }

3. Run the Tap in Discovery Mode This creates a catalog.json for selecting objects/fields to integrate:

   tap-twilio --config config.json --discover > catalog.json

   See the Singer docs on discovery mode here.

4. Run the Tap in Sync Mode (with catalog)

   For Sync mode:

   > tap-twilio --config tap_config.json --catalog catalog.json

   Messages are written to standard output following the Singer specification. The resultant stream of JSON data can be consumed by a Singer target.

To run tests

1. Install python test dependencies in a virtual env and run nose unit and integration tests

  python3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install -e .[test]

2. To run unit tests:

  pytest tests/unit