Various Python utility functions

pip install bx_py_utils

Here only a simple list about existing utilities. Please take a look into the sources and tests for deeper informations.

• anonymize() - Anonymize the given string with special handling for eMail addresses and the possibility to truncate the output.
• anonymize_dict() - Returns a new dict with anonymized values for keys containing one of the given keywords.

• FnmatchExclude() - Helper for auto doc exclude_func that exclude files via fnmatch pattern.
• assert_readme() - Check and update README file with generate_modules_doc()
• assert_readme_block() - Check and update README file: Asset that "text_block" is present between the markers.
• generate_modules_doc() - Generate a list of function/class information via pdoc.
• get_code_location() - Return start and end line number for an object via inspect.

• ClientSideCertManager() - Helper to manage client-side TLS certificate via AWS Secrets Manager by

• SecretsManager() - Access AWS Secrets Manager values

• removeprefix() - Backport of removeprefix from PEP-616 (Python 3.9+)
• removesuffix() - Backport of removesuffix from PEP-616 (Python 3.9+)

• compare_dict_values() - Compare two dictionaries if values of the same keys are present and equal.
• dict_get() - nested dict get()
• dict_list2markdown() - Convert a list of dictionaries into a markdown table.
• pluck() - Extract values from a dict, if they are present

Doc-Write, see: https://github.com/boxine/bx_py_utils/blob/master/bx_py_utils/doc_write/README.md

• OverrideEnviron() - Context manager to change 'os.environ' temporarily.
• cgroup_memory_usage() - Returns the memory usage of the cgroup the Python interpreter is running in.

• print_exc_plus() - Print traceback information with a listing of all the local variables in each frame.

• EmptyFileError() - Will be raised from get_and_assert_file_size() if a 0-bytes file was found.
• FileError() - Base error class for all 'file_utils' exceptions.
• FileHasher() - Context Manager for generate different hashes from file content while processing a file.
• FileSizeError() - File size is not the same as the expected size.
• NamedTemporaryFile2() - Generates a temp file with the given filename without any random name sequence.
• OverlongFilenameError() - cut_filename() error: The file name can not be shortened, because sterm is to short.
• TempFileHasher() - File like context manager that combines NamedTemporaryFile2 and FileHasher.
• cut_filename() - Short the file name (and keep the last suffix). Raise OverlongFilenameError if it can't fit.
• get_and_assert_file_size() - Check file size of given file object. Raise EmptyFileError for empty files or return size
• safe_filename() - Makes an arbitrary input suitable to be used as a filename.

• filename_matcher() - Enhance fnmatch that accept a list of patterns.

• introspection_query() - Generate GraphQL introspection query with variable nested depth.

• collect_hashes() - Get all hash values from a dictionary. Use hashlib.algorithms_available for key names.
• compare_hashes() - Compare hashes from two dictionaries. Return DictCompareResult with the results.
• url_safe_encode() - Encode bytes into a URL safe string.
• url_safe_hash() - Generate a URL safe hash with max_size from given string/bytes.

• ElementsNotFoundError() - Happens if requested HTML elements cannot be found
• InvalidHtml() - XMLSyntaxError with better error messages: used in validate_html()
• get_html_elements() - Returns the selected HTML elements as string
• pretty_format_html() - Pretty format given HTML document via BeautifulSoup (Needs 'beautifulsoup4' package)
• validate_html() - Validate a HTML document via XMLParser (Needs 'lxml' package)

• pformat() - Format given object: Try JSON fist and fallback to pformat()

• human_timedelta() - Converts a time duration into a friendly text representation.

• import_string() - Import a dotted module path and return the attribute/class designated by the last name in the path.

• chunk_iterable() - Returns a generator that yields slices of iterable of the given chunk_size.

• ChangeCurrentWorkDir() - Context Manager change the "CWD" to an other directory.
• MockCurrentWorkDir() - Context Manager to move the "CWD" to a temp directory.
• assert_is_dir() - Check if given path is a directory
• assert_is_file() - Check if given path is a file

• processify() - Decorator to run a function as a process.

• get_pyproject_config() - Get a config section from "pyproject.toml". The path can be optional specify.

• rison_dumps() - Encode as RISON, a URL-safe encoding format.

• FrameNotFound() - Base class for lookup errors.
• last_frame_outside_path() - Returns the stack frame that is the direct successor of given "file_path".

• compare_sentences() - Calculates the Levenshtein distance between text1 and text2. With filter functionality.
• ensure_lf() - Replace line endings to unix-style.
• get_words() - Extract words from a text. With filter functionality.
• is_uuid() - Returns True if text is a valid UUID (https://www.rfc-editor.org/rfc/rfc9562#name-uuid-format).
• levenshtein_distance() - Calculates the Levenshtein distance between two strings.
• startswith_prefixes() - >>> startswith_prefixes('foobar', prefixes=('foo','bar'))
• truncate() - Truncates the given string to the given length
• uuid_from_text() - Generate a UUID instance from the given text in a determinism may via SHA224 hash.

• assert_equal() - Check if the two objects are the same. Display a nice diff, using pformat()
• assert_text_equal() - Check if the two text strings are the same. Display an error message with a diff.
• pformat_ndiff() - Generate a ndiff from two objects, using pformat()
• pformat_unified_diff() - Generate a unified diff from two objects, using pformat()
• text_ndiff() - Generate a ndiff between two text strings.
• text_unified_diff() - Generate a unified diff between two text strings.

• MassContextManager() - A context manager / decorator that enter/exit a list of mocks.
• MassContextManagerExceptions() - Common base class for all non-exit exceptions.

• parse_dt() - Helper for easy generate a datetime instance via string.

• DenyAnyRealRequestContextManager() - Context manager that denies any request via docket/urllib3. Will raise DenyCallError.
• deny_any_real_request() - Deny any request via docket/urllib3. Useful for tests, because they should mock all requests.

• FileWatcher() - Helper to record which new files have been created.

• NoLogs() - Context manager to Suppress all logger outputs
• RaiseLogUsage() - A log handler, that raise an error on every log output.

• SecretsManagerMock() - Mock for bx_py_utils.aws.secret_manager.SecretsManager()

• MockedBoto3Session() - Mock for boto3.session.Session()

• MockUUIDGenerator() - Helper to mock uuid.uuid4() with reproducible results (e.g. for snapshot tests)

A simple mock for Boto3's S3 modules.

• PseudoS3Client() - Simulates a boto3 S3 client object in tests

• RedirectOut() - Redirect stdout + stderr into a buffer (with optional strip the output)

• assert_json_requests_mock() - Check the requests mock history. In this case all requests must be JSON.
• assert_json_requests_mock_snapshot() - Check requests mock history via snapshot. Accepts only JSON requests.
• assert_requests_mock() - Check the requests mock history. Accept mixed "text" and "JSON".
• assert_requests_mock_snapshot() - Check requests mock history via snapshot. Accept mixed "text" and "JSON".

Assert complex output via auto updated snapshot files with nice diff error messages.

• SnapshotChanged() - Assertion failed.
• assert_binary_snapshot() - Assert binary data via snapshot file
• assert_html_snapshot() - Assert "html" string via snapshot file with validate and pretty format
• assert_py_snapshot() - Assert complex python objects vio PrettyPrinter() snapshot file.
• assert_snapshot() - Assert given data serialized to JSON snapshot file.
• assert_text_snapshot() - Assert "text" string via snapshot file
• get_snapshot_file() - Generate a file path use stack information to fill not provided path components.

• MockTimeMonotonicGenerator() - Helper to mock time.monotonic() in tests.

• BaseDocTests() - Helper to include all doctests in unittests, without change unittest setup. Just add a normal TestCase.
• assert_no_flat_tests_functions() - Check if there exists normal test functions (That will not be executed by normal unittests)

• FreezeXlsxTimes() - Context manager / decorator intended to freeze timestamps of xlsx files creation by e.g.: openpyxl.
• generate_xlsx_md_snapshot() - Generate a markdown snapshot of a XLSX: Display ZIP info + Sheets content as Markdown.
• xlsx2dict() - Convert a XLSX file content into a dictionary: Every sheet is a key, and the value is a list of dictionaries.
• xlsx2markdown() - Convert all Sheets of a XLSX into markdown tables.

• FreezeZipFileDatetime() - Context manager / decorator to freezes the modification time of files written to a zip file.
• zip_info() - Generates similar information than unzip -v: Yields ZipFileInfo for each file in the zip file.
• zip_info_markdown() - Generates a markdown representation of the zip file content. Similar to unzip -v output.

• cutout() - Mark a point in a long text by line no + column with context lines around.

Quick hint about snapshot. If you have many snapshots in your project and you need to change many with a code change, then you can run the tests without a snapshot change leading to an error, by set RAISE_SNAPSHOT_ERRORS=0 in your environment.

e.g.:

RAISE_SNAPSHOT_ERRORS=0 python3 -m unittest

Renew all snapshot files with:

make update-test-snapshot-files

We split bx_py_utils and moved all Django related utilities into the separated project:

• https://github.com/boxine/bx_django_utils

So, bx_py_utils is better usable in non-Django projects, because Django will not installed as decency of "bx_py_utils"

To start developing, just run make install to create a .venv and install all needed packages. The minimal requirements are only python3-venv and python3-pip (uv will be installed via pip in .venv)

e.g.:

~$ git clone https://github.com/boxine/bx_py_utils.git
~$ cd bx_py_utils
~/bx_py_utils$ make
help                       List all commands
install-base-req           Install needed base packages via apt
install                    Install the project in a Python virtualenv
update-requirements        Update requirements
lint                       Run code formatters and linter
fix-code-style             Fix code formatting
tox-listenvs               List all tox test environments
tox                        Run tests via tox with all environments
test                       Run tests
coverage                   Run tests with coverage
update-test-snapshot-files Update all snapshot files (by remove and recreate all snapshot files)
mypy                       Run mypy
pip-audit                  Run https://github.com/pypa/pip-audit
publish                    Release new version to PyPi
clean                      Remove created files from the test project

• Increase verion number in bx_py_utils/__init__.py
• Create pull request
• After merge, call: make publish

MIT. Patches welcome!

We’ve been rethinking the listening experience for kids and have created an ecosystem where haptic and listening experience are combined via smart technology - the Toniebox.

We are constantly looking for engineers to join our team in different areas. If you’d be interested in contributing to our platform, have a look at: https://tonies.com/jobs/

• https://pypi.org/project/bx-py-utils/