github.com/miquido/conduit-connector-azure-storage

Conduit connector for Azure Blob Storage


Keywords
azure-storage, conduit, go, golang
License
Apache-2.0
Install
go get github.com/miquido/conduit-connector-azure-storage

Documentation

Conduit Connector Azure Storage

General

The Azure Storage plugin is one of Conduit plugins. It currently provides only source connector.

How to build it

Run make.

Source

The Source connector monitors given Azure Blob container for file changes and generates appropriate Record when change is detected. It supports two reading modes and switches them automatically:

  • Snapshot
  • CDC

In Snapshot mode, connector reads the current state of the container, meaning it does not include changes made during this process. When interrupted, after restarted, it iterates all over container again.

After Snapshot reading is finished, connector switches to CDC mode. In this mode, connector monitors the container each pollingPeriod period and notifies about changes detected. The iterator stores the last modification timestamp of the files iterated in the previous cycle. When detecting new changes, this timestamp is used to discard files created before it. When interrupted, after restarted, it iterates using the timestamp stored in opencdc.Position, passed to source's Open method.

Both iterators paginate over the container via List Blobs query, with up to maxResults items per page, to read the list of available items and their metadata (Last-Modified and Content-Type). When creating the opencdc.Record, the contents of the file is additionally requested via Get Blob query.

Supported storage changes

Changes regarding adding new files to the storage or updating the existing ones are always detected. However, soft delete for blobs needs to be enabled to detect deleted files.

Configuration Options

name description required default
connectionString Azure Storage connection string as described here: https://docs.microsoft.com/azure/storage/common/storage-configure-connection-string true
containerName The name of the container to monitor. true
pollingPeriod The polling period for the CDC mode, formatted as a time.Duration string. Must be greater then 0. false "1s"
maxResults The maximum number of items, per page, when reading container's items. The minimum value is 1, maximum value is 5000. false "5000"

Testing

Run make test to run all the unit and integration tests, which require Docker to be installed and running. The command will handle starting and stopping docker containers for you.

Known Issues & Limitations

  • The connector is able to send record's Key and Payload as sdk.RawData only.

References