posixfs
posixfs provides context managers and functions to manipulate files on a POSIX file system with atomicity and durability. The module is intended to be simple and straightforward to use.
The module is written in Python 3 with types annotated and using pathlib.Path
.
Related Projects
There are many modules and projects which already provide this functionality. We give below a non-exhaustive list:
- https://github.com/abarnert/fatomic
- https://github.com/untitaker/python-atomicwrites
- https://github.com/mitsuhiko/python-atomicfile
- https://github.com/sashka/atomicfile
- https://boltons.readthedocs.io/en/latest/fileutils.html#boltons.fileutils.atomic_save
Unfortunately, none of them provides durable writes (only atomic ones), they lack type annotations and they all use
str
instead of pathlib.Path
to deal with file paths.
We particularly found type annotations to be crucial for a
module which is widely used, such as for writing and reading files, in a large code base that requires static checks and
good readability. Moreover, we do not want to give away the manipulation capabilities of pathlib
and be forced to
sprinkle .as_posix()
all over the code.
For more information on atomic and durable writes on POSIX file systems, see http://blog.httrack.com/blog/2013/11/15/everything-you-always-wanted-to-know-about-fsync
Usage
import pathlib
import posixfs
# write bytes to a file atomically and durably
pth = pathlib.Path("/some/file.txt")
posixfs.atomic_write_bytes(path=pth, data=b"hello", durable=True)
# write text to a file atomically and durably
posixfs.atomic_write_bytes(path=pth, text="hello", durable=True)
# use context manager
with posixfs.AtomicWritingText(path=pth, durable=True) as file:
file.write('hello\n')
file.write('how do you do?\n')
Installation
- Create a virtual environment:
python3 -m venv venv3
- Activate it:
source venv3/bin/activate
- Install posixfs with pip:
pip3 install posixfs
Development
- Check out the repository.
- In the repository root, create the virtual environment:
python3 -m venv venv3
- Activate the virtual environment:
source venv3/bin/activate
- Install the development dependencies:
pip3 install -e .[dev]
- We use tox for testing and packaging the distribution. Assuming that the virtual environment has been activated and the development dependencies have been installed, run:
tox
- We also provide a set of pre-commit checks that lint and check code for formatting. Run them locally from an activated virtual environment with development dependencies:
./precommit.py
- The pre-commit script can also automatically format the code:
./precommit.py --overwrite
Versioning
We follow Semantic Versioning. The version X.Y.Z indicates:
- X is the major version (backward-incompatible),
- Y is the minor version (backward-compatible), and
- Z is the patch version (backward-compatible bug fix).