Telemetry-Aware Addon Recommender

• Taar
  • How does it work?
    • Supported models
  • Build and run tests
  • Pinning dependencies
  • Instructions for releasing updates to production
  • Collaborative Recommender
  • Ensemble Recommender
  • Locale Recommender
  • Similarity Recommender
  • Google Cloud Platform resources
    • Google Cloud BigQuery
    • Google Cloud Storage
    • Google Cloud BigTable
  • Production Configuration Settings
  • Deleting individual user data from all TAAR resources
  • Airflow environment configuration
  • Staging Environment
  • A note on cdist optimization.

The recommendation strategy is implemented through the RecommendationManager. Once a recommendation is requested for a specific client id, the recommender iterates through all the registered models (e.g. CollaborativeRecommender) linearly in their registered order. Results are returned from the first module that can perform a recommendation.

Each module specifies its own sets of rules and requirements and thus can decide if it can perform a recommendation independently from the other modules.

This is the ordered list of the currently supported models:

All jobs are scheduled in Mozilla's instance of Airflow. The Collaborative, Similarity and Locale jobs are executed on a daily schedule, while the ensemble job is scheduled on a weekly schedule.

You should be able to build taar using Python 3.5 or 3.7. To run the testsuite, execute ::

$ python setup.py develop
$ python setup.py test

Alternately, if you've got GNUMake installed, a Makefile is included with build and test-container targets.

You can just run make build; make test-container which will build a complete Docker container and run the test suite inside the container.

TAAR uses miniconda and a environment.yml file to manage versioning.

To update versions, edit the environment.yml with the new dependency you need then run make conda_update.

If you are unfamiliar with using conda, see the official documentation for reference.

Building a new release of TAAR is fairly involved. Documentation to create a new release has been split out into separate instructions.

The final TAAR models are stored in:

gs://moz-fx-data-taar-pr-prod-e0f7-prod-models

The TAAR production model bucket is defined in Airflow under the variable taar_etl_model_storage_bucket

Temporary models that the Airflow ETL jobs require are stored in a temporary bucket defined in the Airflow variable taar_etl_storage_bucket

Recommendation engines load models from GCS.

The following table is a complete list of all resources per recommendation engine.

Cloud BigQuery uses the GCP project defined in Airflow in the variable taar_gcp_project_id.

Dataset

• taar_tmp

Table ID

• taar_tmp_profile

Note that this table only exists for the duration of the taar_weekly job, so there should be no need to manually manage this table.

The taar user profile extraction puts Avro format files into a GCS bucket defined by the following two variables in Airflow:

• taar_gcp_project_id
• taar_etl_storage_bucket

The bucket is automatically cleared at the start and end of the TAAR weekly ETL job.

The final TAAR user profile data is stored in a Cloud BigTable instance defined by the following two variables in Airflow:

• taar_gcp_project_id
• taar_bigtable_instance_id

The table ID for user profile information is taar_profile.

Production environment settings are stored in a private repository.

Deletion of records in TAAR is fairly straight forward. Once a user disables telemetry from Firefox, all that is required is to delete records from TAAR.

Deletion of records from the TAAR BigTable instance will remove the client's list of addons from TAAR. No further work is required.

Removal of the records from BigTable will cause JSON model updates to no longer take the deleted record into account. JSON models are updated on a daily basis via the taar_daily

Updates in the weekly Airflow job in taar_weekly only update the ensemble weights and the user profile information.

If the user profile information in clients_last_seen continues to have data for the user's telemetry-id, TAAR will repopulate the user profile data.

Users who wish to remove their data from TAAR need to:

1. Disable telemetry in Firefox
2. Have user telemetry data removed from all telemetry storage systems in GCP. Primarily this means the clients_last_seen table in BigQuery.
3. Have user data removed from BigTable.

TAAR requires some configuration to be stored in Airflow variables for the ETL jobs to run to completion correctly.

The staging environment of the TAAR service in GCP can be reached using curl.

curl https://user@pass:stage.taar.nonprod.dataops.mozgcp.net/v1/api/recommendations/<hashed_telemetry_id>

Requests for a TAAR-lite recommendation can be made using curl as well:

curl https://stage.taar.nonprod.dataops.mozgcp.net/taarlite/api/v1/addon_recommendations/<addon_guid>/

There is a taarlite-redis tool to manage the taarlit redis cache.

The cache needs to be populated using the --load command or TAARlite will return no results.

It is safe to reload new data while TAARlite is running - no performance degradation is expected.

The cache contains a 'hot' buffer for reads and a 'cold' buffer to write updated data to.

Subsequent invocations to --load will update the cache in the cold buffer. After data is successfully loaded, the hot and cold buffers are swapped.

Running the the taarlite-redis tool inside the container:

$ docker run -it taar:latest bin/run python /opt/conda/bin/taarlite-redis.py --help

Usage: taarlite-redis.py [OPTIONS]

  Manage the TAARLite redis cache.

  This expecte that the following environment variables are set:

  REDIS_HOST REDIS_PORT

Options:
  --reset  Reset the redis cache to an empty state
  --load   Load data into redis
  --info   Display information about the cache state
  --help   Show this message and exit.

TAARLite will respond with suggestions given an addon GUID.

A sample URL path may look like this:

/taarlite/api/v1/addon_recommendations/uBlock0%40raymondhill.net/

TAAR will treat any client ID with only repeating digits (ie: 0000) as a test client ID and will return a dummy response.

A URL with the path : /v1/api/recommendations/0000000000/ will return a valid JSON result

cdist can speed up distance computation by a factor of 10 for the computations we're doing. We can use it without problems on the canberra distance calculation.

Unfortunately there are multiple problems with it accepting a string array. There are different problems in 0.18.1 (which is what is available on EMR), and on later versions. In both cases cdist attempts to convert a string to a double, which fails. For versions of scipy later than 0.18.1 this could be worked around with:

distance.cdist(v1, v2, lambda x, y: distance.hamming(x, y))

However, when you manually provide a callable to cdist, cdist can not do it's baked in optimizations (https://github.com/scipy/scipy/blob/v1.0.0/scipy/spatial/distance.py#L2408) so we can just apply the function distance.hamming to our array manually and get the same performance.