This package bundles some of the best Python serialization libraries into one standalone package, with a high-level API that makes it easy to write code that's correct across platforms and Pythons. This allows us to provide all the serialization utilities we need in a single binary wheel. Currently supports JSON, JSONL, MessagePack, Pickle and YAML.

Serialization is hard, especially across Python versions and multiple platforms. After dealing with many subtle bugs over the years (encodings, locales, large files) our libraries like spaCy and Prodigy had steadily grown a number of utility functions to wrap the multiple serialization formats we need to support (especially json, msgpack and pickle). These wrapping functions ended up duplicated across our codebases, so we wanted to put them in one place.

At the same time, we noticed that having a lot of small dependencies was making maintenance harder, and making installation slower. To solve this, we've made srsly standalone, by including the component packages directly within it. This way we can provide all the serialization utilities we need in a single binary wheel.

srsly currently includes forks of the following packages:

• ujson
• msgpack
• msgpack-numpy
• cloudpickle
• ruamel.yaml (without unsafe implementations!)

⚠️ Note that v2.x is only compatible with Python 3.6+. For 2.7+ compatibility, use v1.x.

srsly can be installed from pip. Before installing, make sure that your pip, setuptools and wheel are up to date.

python -m pip install -U pip setuptools wheel
python -m pip install srsly

Or from conda via conda-forge:

conda install -c conda-forge srsly

Alternatively, you can also compile the library from source. You'll need to make sure that you have a development environment with a Python distribution including header files, a compiler (XCode command-line tools on macOS / OS X or Visual C++ build tools on Windows), pip and git installed.

Install from source:

# clone the repo
git clone https://github.com/explosion/srsly
cd srsly

# create a virtual environment
python -m venv .env
source .env/bin/activate

# update pip
python -m pip install -U pip setuptools wheel

# compile and install from source
python -m pip install .

For developers, install requirements separately and then install in editable mode without build isolation:

# install in editable mode
python -m pip install -r requirements.txt
python -m pip install --no-build-isolation --editable .

# run test suite
python -m pytest --pyargs srsly

📦 The underlying module is exposed via srsly.ujson. However, we normally interact with it via the utility functions only.

Serialize an object to a JSON string. Falls back to json if sort_keys=True is used (until it's fixed in ujson).

data = {"foo": "bar", "baz": 123}
json_string = srsly.json_dumps(data)

Deserialize unicode or bytes to a Python object.

data = '{"foo": "bar", "baz": 123}'
obj = srsly.json_loads(data)

Create a JSON file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_json("/path/to/file.json", data)

Load JSON from a file or standard input.

data = srsly.read_json("/path/to/file.json")

Create a gzipped JSON file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_gzip_json("/path/to/file.json.gz", data)

Create a gzipped JSONL file and dump contents.

data = [{"foo": "bar"}, {"baz": 123}]
srsly.write_gzip_json("/path/to/file.jsonl.gz", data)

Load gzipped JSON from a file.

data = srsly.read_gzip_json("/path/to/file.json.gz")

Load gzipped JSONL from a file.

data = srsly.read_gzip_jsonl("/path/to/file.jsonl.gz")

Create a JSONL file (newline-delimited JSON) and dump contents line by line, or write to standard output.

data = [{"foo": "bar"}, {"baz": 123}]
srsly.write_jsonl("/path/to/file.jsonl", data)

Read a JSONL file (newline-delimited JSON) or from JSONL data from standard input and yield contents line by line. Blank lines will always be skipped.

data = srsly.read_jsonl("/path/to/file.jsonl")

Check if a Python object is JSON-serializable.

assert srsly.is_json_serializable({"hello": "world"}) is True
assert srsly.is_json_serializable(lambda x: x) is False

📦 The underlying module is exposed via srsly.msgpack. However, we normally interact with it via the utility functions only.

Serialize an object to a msgpack byte string.

data = {"foo": "bar", "baz": 123}
msg = srsly.msgpack_dumps(data)

Deserialize msgpack bytes to a Python object.

msg = b"\x82\xa3foo\xa3bar\xa3baz{"
data = srsly.msgpack_loads(msg)

Create a msgpack file and dump contents.

data = {"foo": "bar", "baz": 123}
srsly.write_msgpack("/path/to/file.msg", data)

Load a msgpack file.

data = srsly.read_msgpack("/path/to/file.msg")

📦 The underlying module is exposed via srsly.cloudpickle. However, we normally interact with it via the utility functions only.

Serialize a Python object with pickle.

data = {"foo": "bar", "baz": 123}
pickled_data = srsly.pickle_dumps(data)

Deserialize bytes with pickle.

pickled_data = b"\x80\x04\x95\x19\x00\x00\x00\x00\x00\x00\x00}\x94(\x8c\x03foo\x94\x8c\x03bar\x94\x8c\x03baz\x94K{u."
data = srsly.pickle_loads(pickled_data)

📦 The underlying module is exposed via srsly.ruamel_yaml. However, we normally interact with it via the utility functions only.

Serialize an object to a YAML string. See the ruamel.yaml docs for details on the indentation format.

data = {"foo": "bar", "baz": 123}
yaml_string = srsly.yaml_dumps(data)

Deserialize unicode or a file object to a Python object.

data = 'foo: bar\nbaz: 123'
obj = srsly.yaml_loads(data)

Create a YAML file and dump contents or write to standard output.

data = {"foo": "bar", "baz": 123}
srsly.write_yaml("/path/to/file.yml", data)

Load YAML from a file or standard input.

data = srsly.read_yaml("/path/to/file.yml")

Check if a Python object is YAML-serializable.

assert srsly.is_yaml_serializable({"hello": "world"}) is True
assert srsly.is_yaml_serializable(lambda x: x) is False