This repository is a pkg-ized version of Danny Yoo's python-tokenizer repository, which is a fairly close translation of the tokenize.py library from Python.

Install it using

raco pkg install python-tokenizer