datacatalog-tag-manager

A Python package to manage Google Cloud Data Catalog tags, loading metadata from external sources. Currently supports the CSV file format.

Table of Contents

• 1. Environment setup
  • 1.1. Python + virtualenv
    • 1.1.1. Install Python 3.6+
    • 1.1.2. Create a folder
    • 1.1.3. Create and activate an isolated Python environment
    • 1.1.4. Install the package
  • 1.2. Docker
    • 1.2.1. Get the source code
  • 1.3. Auth credentials
    • 1.3.1. Create a service account and grant it below roles
    • 1.3.2. Download a JSON key and save it as
    • 1.3.3. Set the environment variables
• 2. Manage Tags
  • 2.1. Create or Update
    • 2.1.1. From a CSV file
  • 2.2. Delete
    • 2.2.1. From a CSV file

1. Environment setup

1.1. Python + virtualenv

Using virtualenv is optional, but strongly recommended unless you use Docker.

1.1.1. Install Python 3.6+

1.1.2. Create a folder

This is recommended so all related stuff will reside at same place, making it easier to follow below instructions.

mkdir ./datacatalog-tag-manager
cd ./datacatalog-tag-manager

All paths starting with ./ in the next steps are relative to the datacatalog-tag-manager folder.

1.1.3. Create and activate an isolated Python environment

pip install --upgrade virtualenv
python3 -m virtualenv --python python3 env
source ./env/bin/activate

1.1.4. Install the package

pip install --upgrade datacatalog-tag-manager

1.2. Docker

Docker may be used as an alternative to run datacatalog-tag-manager. In this case, please disregard the above virtualenv setup instructions.

1.2.1. Get the source code

git clone https://github.com/ricardolsmendes/datacatalog-tag-manager
cd ./datacatalog-tag-manager

1.3. Auth credentials

1.3.1. Create a service account and grant it below roles

• BigQuery Metadata Viewer
• Data Catalog TagTemplate User
• A custom role with bigquery.datasets.updateTag and bigquery.tables.updateTag permissions

1.3.2. Download a JSON key and save it as

• ./credentials/datacatalog-tag-manager.json

1.3.3. Set the environment variables

This step may be skipped if you're using Docker.

export GOOGLE_APPLICATION_CREDENTIALS=./credentials/datacatalog-tag-manager.json

2. Manage Tags

2.1. Create or Update

2.1.1. From a CSV file

• SCHEMA

The metadata schema to create or update Tags is presented below. Use as many lines as needed to describe all the Tags and Fields you need.

• SAMPLE INPUT

1. sample-input/upsert-tags for reference;
2. Data Catalog Sample Tags (Google Sheets) might help to create/export a CSV file.

• COMMANDS

Python + virtualenv

datacatalog-tags upsert --csv-file <CSV-FILE-PATH>

Docker

docker build --rm --tag datacatalog-tag-manager .
docker run --rm --tty \
  --volume <CREDENTIALS-FILE-FOLDER>:/credentials --volume <CSV-FILE-FOLDER>:/data \
  datacatalog-tag-manager upsert --csv-file /data/<CSV-FILE-PATH>

2.2. Delete

2.2.1. From a CSV file

• SCHEMA

The metadata schema to delete Tags is presented below. Use as many lines as needed to delete all the Tags you want.

• SAMPLE INPUT

1. sample-input/delete-tags for reference;
2. Data Catalog Sample Tags (Google Sheets) might help to create/export a CSV file.

• COMMANDS

Python + virtualenv

datacatalog-tags delete --csv-file <CSV-FILE-PATH>

Docker

docker build --rm --tag datacatalog-tag-manager .
docker run --rm --tty \
  --volume <CREDENTIALS-FILE-FOLDER>:/credentials --volume <CSV-FILE-FOLDER>:/data \
  datacatalog-tag-manager delete --csv-file /data/<CSV-FILE-PATH>