Airtable Python Library
Airtable library files used by DxE tech. This is a pip installable package you can use like:
pip install dxe-airtable
then:
from dxe_airtable.airtable import get_all_records
You can also run the scripts directly, like so:
python2 -m dxe_airtable.backup_airtable /opt/dxe/airtable/backups
Note that you will need to define the following environment variables to use this:
AIRTABLE_API_KEY
AIRTABLE_BASE_ID
How To Inspect The Airtable Backups
Access
The backups are stored in Amazon's S3 storage service. The access credentials are in config/airtable.sh. Install the AWS CLI, probably just by running pip install awscli
. Then run aws configure
to set the access credentials (do defaults for the other options).
Then you can list all of the airtable backups by running:
aws s3 ls s3://dxe-backup/airtable/
It will output a bunch, all timestamped. Probably grab the one at the bottom, the most recent one. Copy the file name to paste into this command:
mkdir backup
aws s3 cp s3://dxe-backup/airtable/<pasted file name> backup/
Parsing it
Yay, you downloaded a backup. Unzip it:
cd backup/
unzip <pasted file name>
Now you'll see a few json files, each of which represent a data dump of a different table in the base. The contents of each are structured like:
[ {
"createdTime": "2015-11-12T01:39:30.000Z",
"fields": {
"column1": "row1col1 value",
"column2": 2,
...
},
"id": "recvTwBkAt1ThX0ZL"
},
...
]
The only thing interesting is that sometimes values will be these weird strings like recvTwBkAt1ThX0Z
. These are references (by id) to other records. For example, each row in "All Members" has a value in the chapter_id
column that points to a record in the "Chapters" table.
We haven't built anything to help code handle those references because we haven't needed to yet. If you're reading this any about to do that, have fun mate, that doesn't sound like fun.
Airtable To Mailing List Sync
Syncs the Google Groups mailing lists members one-way from Airtable. As each member is added to the Google Group for the first time, the chapter whose mailing list they were added to is recorded in the Airtable. The member will not be added again, even if they are removed from the Google Group. This is so that the members can unsubscribe themselves from the Google Group without us having to use heuristics to figure out whether they unsubscribed or just weren't added.
Run
To sync the Google Groups Mailing Lists from the data in Airtable, run:
python2 -m dxe_airtable.airtable_to_mailing_list
Setup
You need a client_secret.json to authenticate with Google user with the following scopes: "https://www.googleapis.com/auth/admin.directory.group", "https://www.googleapis.com/auth/admin.directory.group.member".
You also need AIRTABLE_API_KEY
set in your env.
Test
python2 dxe_airtable.test_airtable_to_mailing_list
License
dxe-airtable is licensed under GNU GPL version 3.0. For the full license see the LICENSE file.