Mend SCA Cleanup Tool


License
Other
Install
pip install ws-cleanup-tool==24.1.1

Documentation

⚠️ This tool has moved to a new home, please see the following repository in the Mend Toolkit for the latest version: SCA Cleanup Tool

Logo License Mend projects cleanup Python 3.6 PyPI

Mend SCA Projects Cleanup CLI Tool

  • Current version v23.7.1
  • The self-hosted CLI tool features cleaning up projects and generating reports before deletion in 2 modes:
    • By stating OperationMode=FilterProjectsByUpdateTime and how many days to keep (-r/ DaysToKeep=)
    • By stating OperationMode=FilterProjectsByLastCreatedCopies and how many copies to keep (-r/ DaysToKeep=)
  • The reports are saved in the designated location as follows: [Output_DIR]/[PRODUCT NAME]/[PROJECT NAME]/[REPORT NAME]
    • The default location is the [WORKING DIRECTORY]/Mend/Reports/[PRODUCT NAME]/[PROJECT NAME]/[REPORT NAME]
  • To review the outcome before actual deletion use -y true / DryRun=True flag. It will NOT delete any project nor create reports
  • By default, the tool generates all possible project-level reports. By specifying ((-t / Reports=/) it is possible to select specific reports
    • The full list of available reports is below
  • The full parameters list is available below
  • There are two ways to configure the tool:
    • By configuring params.config on the executed dir or passing a path to the file in the same format
    • By setting command line parameters as specified in the usage below

Supported Operating Systems

  • Linux (Bash): CentOS, Debian, Ubuntu, RedHat
  • Windows (PowerShell): 10, 2012, 2016

Pre-requisites

  • Python 3.8+

Permissions

  • The user used to execute the tool has to have "Organization Administrator" or "Product Administrator" on all the maintained products and "Organization Auditor" permissions.
  • It is recommended to use a service user.

Installation and Execution from PyPi (recommended):

  1. Install by executing: pip install mend-sca-cleanup-tool
  2. Configure the appropriate parameters either by using the command line or in params.config.
  3. Execute the tool (mend_sca_cleanup_tool ...).
  4. In order to update the tool please run pip install mend-sca-cleanup-tool --upgrade

Installation and Execution from GitHub:

  1. Download and unzip mend-sca-cleanup-tool.zip from the most recent tagged release.
  2. Install requirements: pip install -r requirements.txt
  3. Configure the appropriate parameters either by using the command line or params.config.
  4. Execute: python sca_cleanup_tool.py <CONFIG_FILE>

Examples:

Perform dry run check-in to get to know which projects would have been deleted:
mend_sca_cleanup_tool -r 30 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -y true


Keep the last 60 days on each product, omitting a product token <PRODUCT_1> from analyzing:
mend_sca_cleanup_tool -r 60 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -e <PRODUCT_TOKEN_1>


Keep only two of the newest projects in each product token PRODUCT_1 and PRODUCT_2:
mend_sca_cleanup_tool -r 2 -m FilterProjectsByLastCreatedCopies -u <USER_KEY> -k <ORG_TOKEN> -i <PRODUCT_TOKEN_1>,<PRODUCT_TOKEN_2>


Analyze only the projects that have the specified Mend tag and keep the newest project in each product:
mend_sca_cleanup_tool -r 1 -m FilterProjectsByLastCreatedCopies -u <USER_KEY> -k <ORG_TOKEN> -g <KEY>:<VALUE>


Keep the last 2 weeks and analyze only the projects whose match their tag key and the tag value contains the specified value:
mend_sca_cleanup_tool -r 14 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -v <KEY>:<VALUE>


Keep the last 100 days for both PRODUCT_1 and PRODUCT_2, but do not delete the project PROJECT_1 (which is a project in one of the included products):
mend_sca_cleanup_tool -r 100 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -i <PRODUCT_TOKEN_1>,<PRODUCT_TOKEN_2> -x <PROJECT_TOKEN_1>


Keep the last month for both PRODUCT_1 and PRODUCT_2, but do not delete projects that contain provided strings in their names:
mend_sca_cleanup_tool -r 31 -m FilterProjectsByUpdateTime -u <USER_KEY> -k <ORG_TOKEN> -i <PRODUCT_TOKEN_1>,<PRODUCT_TOKEN_2> -n CI_,-test


Full Usage flags:

usage: ws_cleanup_tool [-h] -u MEND_USER_KEY -k MEND_TOKEN [-a MEND_URL] [-t REPORT_TYPES] [-m {FilterProjectsByUpdateTime,FilterProjectsByLastCreatedCopies}] [-o OUTPUT_DIR] [-e EXCLUDED_PRODUCT_TOKENS] [-i INCLUDED_PRODUCT_TOKENS]
                    [-g ANALYZED_PROJECT_TAG] [-r DAYS_TO_KEEP] [-p PROJECT_PARALLELISM_LEVEL] [-y DRY_RUN]

Mend Cleanup Tool

optional arguments:
  -h, --help            show this help message and exit
  -u MEND_USER_KEY, --userKey 
                    Mend User Key
  -k MEND_API_TOKEN, --apiToken, --orgToken
                    Mend Organization Key (API Key)
  -a MEND_URL, --mendUrl, --wsURL
                    Mend URL. This value defaults to saas.whitesourcesoftware.com.
  -t REPORT_TYPES, --reportTypes
                    Report Types to generate (comma seperated list)
  -m OPERATION_MODE, --operationMode {FilterProjectsByUpdateTime,FilterProjectsByLastCreatedCopies}
                    Cleanup operation mode
  -o OUTPUT_DIR, --outputDir
                    Output directory
  -e EXCLUDED_PRODUCT_TOKENS, --excludedProductTokens
                    List of excluded products
  -i INCLUDED_PRODUCT_TOKENS, --includedProductTokens
                    List of included products
  -g ANALYZED_PROJECT_TAG, --AnalyzedProjectTag
                    Analyze only the projects whose contain the specific Mend tag (key:value). Case sensitive.
  -v ANALYZED_PROJECT_TAG_REGEX_IN_VALUE, --AnalyzedProjectTagRegexInValue
                    Analyze only the projects whose match their tag key and the tag value contains the specified value (key:value). Case sensitive.
                    Note: This was originally broken in the original ws-cleanup-tool. The functionality was adjusted to work as originally written. The naming convention is a misnomer but was kept to avoid breaking existing integrations.
  -r DAYS_TO_KEEP, --DaysToKeep
                    Number of days to keep in FilterProjectsByUpdateTime or number of copies in FilterProjectsByLastCreatedCopies
  -p PROJECT_PARALLELISM_LEVEL, --ProjectParallelismLevel
                    Project parallelism level
                    Note: This is currently not used in this version of the mend-sca-cleanup-tool. Was kept to prevent breaking existing integrations.
  -y DRY_RUN, --DryRun
                    Logging the projects that are supposed to be deleted without deleting and creating reports
                    default False
  -s SKIP_REPORT_GENERATION, --SkipReportGeneration
                    Skip report generation step
                    default False
  -j SKIP_PROJECT_DELETION, --SkipProjectDeletion
                    Skip project deletion step
                    default False                                        
  -x EXCLUDED_PROJECT_TOKENS, --excludedProjectTokens
                    List of excluded projects
  -n EXCLUDED_PROJECT_NAME_PATTERNS, --excludedProjectNamePatterns
                    List of excluded project name patterns (comma seperated list). Case sensitive.            

Available reports

The following Mend project reports are available through the clean-up tool. These values can be specified with the -t flag to generate specific reports.

  • alerts
  • alerts_rejected_by_policy
  • attribution
  • bugs
  • due_diligence
  • ignored_alerts
  • in_house_libraries
  • inventory
  • license_compatibility
  • resolved_alerts
  • request_history
  • source_files
  • source_file_inventory
  • vulnerability

SAST Clean up

If you need to run a clean up script for your SAST environment, please refer to the Mend SAST clean up kit in the Mend Toolkit

note: The optimal cleanup scope is derived from the size of the environment, Mend scope size (memory and CPU) allocated for the server, and runtime time constraints.