Figures

Reporting and data retrieval app for Open edX.

Figures is on PyPI

Oct 13, 2022 - Figures release 0.4.4

This release includes a performance fix for the site-monthly-metric API.

Aug 15, 2022 - Figures release 0.4.3

This release includes bug fixes. Of note:

Fix tasks for standalone and fix average progress validation error
Enable daily metrics models backfill of historical data in platform, but not progress
Update SDM historic course count calculation
Enrollment data handling fixes
Fix broken devsite data seeding
Front end package dependency updates

Also includes a foundational update for MAU 2:

MAU 2 - Add Monthly Active Enrollment Model

Mar 3, 2022 - Figures release 0.4.2

This release adds an optionan new pipeline workflow

For details, please read here

To enable this feature, you need to update the LMS settings (aka 'server-vars.yml') as follows:

server-vars.yml:

FIGURES:
  DAILY_TASK: 'figures.tasks.populate_daily_metrics_next'

In Django settigs, you would check the var here:

from django.conf import settings
settings.ENV_TOKENS['FIGURES'].get('DAILY_TASK')

PR about the workflow update:

Pipeline improvement prerequisites
- #427
Figures pipeline performance improvement
- #429
Fix enrollment data backfill Django management command
- #432

Other PRs

Revert devsite 'courseware' app namepace back to originial
- #434
Bump url-parse from 1.5.1 to 1.5.10 in /frontend
- #431
Bump urijs from 1.19.6 to 1.19.8 in /frontend
- #430

Feb 4, 2022 - Main Branch

main is the new default branch

Jan 28, 2022 - Figures release 0.4.1

Figures 0.4.1 is finally here. After several development releases, we realized it was time to just move to production releases.

Figures 0.4.x release series supports Open edX Juniper, Hawthorn, and Ginkgo

Please visit Figures releases page for details on specific releases.

30 Oct 2020 - Figures release 0.3.19

Fix the comparison showing N/A when comparing to zero
- #277
Fix Figures devsite settings
- #278

26 Oct 2020 - Figures release 0.3.18

FIX - Removed dependency on 'packaging.versions'
- #272
- NOTE: This PR updates a previous commit that required the packaging package

16 Oct 2020 - Figures release 0.3.17

Reworked SiteMonthlyMetrics registered users metric. This was causing the /figures/api/site-monthly-metrics/registered_users endpoint to timeout with a 500 error
- #268
Fixed Ginkgo (Django Filter 0.11.0) Backward compatibility issues
- #266
- #269
UI Bug fix: Add success feedback to csv export dialog
- #265
Bump http-proxy from 1.18.0 to 1.18.1 in /frontend
- #254

28 Sep 2020 - Figures release 0.3.16

Add Learners Progress Overview to main menu
- #256
Performance and test improvement for LearnerMetricsViewSet
- #260
Fix code that doesn't work on Ginkgo (Django 1.8)
- #261

11 Sep 2020 - Figures release 0.3.15

Learner progress overview UI improvements
- #255

24 Aug 2020 - Figures release 0.3.14

Added multi-course filtering to the learner-metrics API endpoint
- #248
Small cosmetic issues in new Learners Progress Overview page
- #247

14 Aug 2020 - Figures release 0.3.13

Learner metrics Prerelease API and UI
- #239
- #240
Improve logging for monthly metrics pipeline and set default to run the monthly metrics pipeline task
- #242
Bug fix: Site level certificate metrics
- #244

15 Jul 2020 - Figures release 0.3.12

Adds enrollment metrics API endpoint
- #233
Site monthly metrics API performance improvement
- #234
Initial implementation of Celery support for Figures devsite
- #215

29 Jun 2020 - Figures release 0.3.11

Fixes incorrect site monthly metrics course completion data
- #219
Fixes CourseDailyMetricsSerializer when average_progress is 1.00
- #230
Updates pipeline enrollment metrics queries to improve performance
- #226
Added site pipeline progress indicator to logging
- #228
Bump devsite Django 1.11 to version 1.11.29
- #227
Bump websocket-extensions from 0.1.3 to 0.1.4 in /frontend
- #222

21 May 2020 - Figures release 0.3.10

Improved daily metrics pipeline performance
- #214
Bug fixes
- #213

24 Apr 2020 - Figures release 0.3.9

Updated UI, MAU fix, style fixes, label changes
Added site monthly metrics scheduled tasks to fill last month's MAU
Added Django Debug Toolbar to devsite

10 Apr 2020 - Figures release 0.3.8

Performance improvement to "Site Monthly Metrics" active users endpoint

8 Apr 2020 - Figures release 0.3.7

Updated UI to improve performance and usability
Added "Course Monthly Metrics" set of API endpoints

16 Feb 2020 - Figures release 0.3.6

Updated UI to address performance issues
Added missing organizations to devsite settings INSTALLED_APPS

20 Feb 2020 - Figures release 0.3.5

Client (UI and API) facing updates

Site Monthly Metrics pipeline and new API endpoints
Added Course MAU metrics API endpoint and pipeline
Added user email address to general user data
Bug fix - URL pattern fix for figures/

Developer facing updates

Updated Django micro version to Figures devsite
Added missing .env file for Makefile support
Added Pylint to testing
Refactored permissions module

29 Jan 2020 - Figures release 0.3.4

Hawthorn support since release 0.3.0
This release includes bug fixes, UI improvements, and backport support for Ginkgo
Includes a standalone development mode. See the developer quickstart guide

Overview

Figures is a reporting and data retrieval app. It plugs into the edx-platform LMS app server. Its goal is to provide site-wide and cross-course analytics that compliment Open edX's traditional course-centric analytics.

To evolve Figures to meet community needs, we are keeping in mind as principles the following features, which Jill Vogel outlined in her lightweight analytics post on ed Xchange:

Real time (or near real time) updates
Lightweight deployment
Flexible reporting
Simpler contributions

Please refer to the Figures design document for more details on goals and architecture.

Requirements

For all Open edX releases:

Python (2.7)

For Hawthorn:

Django (1.11)

For Ginkgo:

Django (1.8)

Installation

NOTICE: Installation instructions are out of date and backlogged for update

Devstack

Go here for instructions to install and run Figures in devstack.

Production

Go here for instructions to install Figures in production.

Project Architecture

Front-end

The Figures user interface is a JavaScript Single Page Application (SPA) built with React and uses the create-react-app build scaffolding generator.

Back-end

The Figures back-end is a reusable Django app. It contains a set of REST API endpoints that serve a dual purpose of providing data to the front-end and to remote clients.

Testing

TODO: Improve the testing instructions

The unit tests should be able to run on any OS that supports Python 2.7.x

Clone the repo:

git@github.com:appsembler/figures.git

Go to the project directory:

cd figures

Create a virtualenv for Python 2.7.x.

Install required Python packages:

pip install -r devsite/requirements.txt

From the figures repository root directory:

py.test

If all goes well, the Figures unit tests will all complete succesfully

Configuration

Figures can be configured via Django settings' FIGURES key. Open edX reads configuration from the /edx/etc/lms.yml file both in devstack and production servers. In releases before Juniper it was the lms.env.json file.

A Figures configuration may look like the following:

FEATURES:  # The standard Open edX feature flags
        # ... other features goes here ...
        FIGURES_IS_MULTISITE: True
        # ... more features goes there ...

FIGURES:  # Other Figures configurations
        SITES_BACKEND: 'openedx.core.djangoapps.appsembler.sites.utils:get_active_sites'
        REQUESTED_SITE_BACKEND: 'tahoe_figures_plugins.sites:get_current_site_or_by_uuid'
        FIGURES_PIPELINE_TASKS_ROUTING_KEY: 'edx.lms.core.high'
        DAILY_METRICS_IMPORT_HOUR: 13
        DAILY_METRICS_IMPORT_MINUTE: 0

Settings like SITES_BACKEND require a path to a Python function or class. The path is consists of two parts: a Python module e.g. my_plugin_package.helpers and an object e.g my_helper separated by a colon e.g. my_plugin_package.helpers:my_helper.

This object would be imported by the import_from_path helper in the figures/helpers.py module.

Configuration options

FEATURES.FIGURES_IS_MULTISITE (default False): Boolean feature flag to run Figures in a single-site mode by default (when set to False) most popular Open edX installation option. The multisite mode requires a custom edx-organizations fork that is used for Appsembler Tahoe clusters.
FIGURES.SITES_BACKEND (default None): A Python path to function to list figures sites. For example, this is useful to customize which sites are processed and which are considered inactive. By default (when None is used) all sites are listed in the multi-site mode.
REQUESTED_SITE_BACKEND (default None): Python path to a function that gets the current site. For example it can be used for API purposes to pass a Site ID to get a different site. By default (when None is used) the Django's get_current_site() helper is used.

TBD: Document FIGURES_PIPELINE_TASKS_ROUTING_KEY, DAILY_METRICS_IMPORT_HOUR and DAILY_METRICS_IMPORT_MINUTE.

Future

Open edX "Hawthorn" will provide a plug-in architecture. This will hopefully simplify Figures installation even more
Downloadable report files
Plugin architecture to extend Figures for custom data sources