Load data about Python packages from PyPI into SQLite
Install this tool using pip:
pip install pypi-to-sqlite
To create a SQLite database with details of one or more packages, run:
pypi-to-sqlite pypi.db datasette sqlite-utils
You can also process JSON that you have previously saved to disk like so:
curl -o datasette.json https://pypi.org/pypi/datasette/json
pypi-to-sqlite pypi.db -f datasette.json
The tool will create three tables: packages, versions and releases. The full table schema is shown below.
To create the tables with a prefix, use --prefix prefix. For example:
pypi-to-sqlite pypi.db datasette --prefix pypi_
This will create tables called pypi_packages, pypi_versions and pypi_releases.
You can see examples of tables created using this tool running in Datasette here:
CREATE TABLE [packages] (
[name] TEXT PRIMARY KEY,
[summary] TEXT,
[classifiers] TEXT,
[description] TEXT,
[author] TEXT,
[author_email] TEXT,
[description_content_type] TEXT,
[home_page] TEXT,
[keywords] TEXT,
[license] TEXT,
[maintainer] TEXT,
[maintainer_email] TEXT,
[package_url] TEXT,
[platform] TEXT,
[project_url] TEXT,
[project_urls] TEXT,
[release_url] TEXT,
[requires_dist] TEXT,
[requires_python] TEXT,
[version] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);
CREATE TABLE [versions] (
[id] TEXT PRIMARY KEY,
[package] TEXT REFERENCES [packages]([name]),
[name] TEXT
);
CREATE TABLE [releases] (
[md5_digest] TEXT PRIMARY KEY,
[package] TEXT REFERENCES [packages]([name]),
[version] TEXT REFERENCES [versions]([id]),
[packagetype] TEXT,
[filename] TEXT,
[comment_text] TEXT,
[digests] TEXT,
[has_sig] INTEGER,
[python_version] TEXT,
[requires_python] TEXT,
[size] INTEGER,
[upload_time] TEXT,
[upload_time_iso_8601] TEXT,
[url] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...
Load data about Python packages from PyPI into SQLite
Usage example:
pypi-to-sqlite pypy.db datasette sqlite-utils
Use -f to load data from a JSON file instead:
pypi-to-sqlite pypy.db -f datasette.json
Created tables will be packages, versions and releases
To create tables called pypi_packages, pypi_versions, pypi_releases use
--prefix pypi_:
pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_
Options:
--version Show the version and exit.
-f, --file FILENAME Import JSON from this file
-d, --delay FLOAT Wait this many seconds between requests
--prefix TEXT Prefix to use for the created database tables
--help Show this message and exit.
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd pypi-to-sqlite
python -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
