pandas_path - Path style access for pandas
Love pathlib.Path*? Love pandas? Wish it were easy to use pathlib methods on pandas Series?
This package is for you. Just one import adds a .path accessor to any pandas Series or Index so that you can use all of the methods on a Path object.
* If not, you should.
Here's an example:
from pathlib import Path
import pandas as pd
# This is the only line you need to register `.path` as an accessor
# on any Series or Index in pandas.
import pandas_path
# we'll make an example series from the py files in this repo;
# note that every element here is just a string--no need to make Path objects yourself
file_paths = pd.Series(str(s) for s in Path().glob('**/*.py'))
# 0 setup.py
# 1 pandas_path/accessor.py
# 2 pandas_path/test.py
# dtype: objectUse the .path accessor to get just the filename rather than the full path:
file_paths.path.name
# 0 setup.py
# 1 accessor.py
# 2 test.py
# dtype: objectUse the .path accessor to get just the parent folder of each file:
file_paths.path.parent
# 0 .
# 1 pandas_path
# 2 pandas_path
# dtype: objectUse calculated methods like exists to filter for what exists on the filesystem:
file_paths.loc[3] = 'fake_file.txt'
# 0 setup.py
# 1 pandas_path/accessor.py
# 2 pandas_path/test.py
# 3 fake_file.txt
# dtype: object
file_paths.path.exists()
# 0 True
# 1 True
# 2 True
# 3 False
# dtype: boolUse path methods like with_suffix to dynamically create new filenames:
file_paths.path.with_suffix('.png')
# 0 setup.png
# 1 pandas_path/accessor.png
# 2 pandas_path/test.png
# 3 fake_file.png
# dtype: objectUse the / operators just as you would in pathlib (with the .path accessor on either side of the operator.)
"different_root_folder" / file_paths.path
# 0 different_root_folder/setup.py
# 1 different_root_folder/pandas_path/accessor.py
# 2 different_root_folder/pandas_path/test.py
# dtype: objectWe'll even do element wise operations with lists/arrays/series of the same length.
file_paths.path.parent.path / ["other_file1.txt", "other_file2.txt", "other_file3.txt"]
# 0 other_file1.txt
# 1 pandas_path/other_file2.txt
# 2 pandas_path/other_file3.txt
# dtype: objectCustom path accessors
Some libraries (such as cloudpathlib, which support path operations for AWS S3,
Azure Blobs, and Google Cloud Storage) implement the Path interface in other contexts. You can use pandas-path to
register and use any class that implements Path. For example:
import pandas as pd
from pandas_path import register_path_accessor
from cloudpathlib import S3Path
# creates an accessor ".s3" that creates s3 paths
register_path_accessor("s3", S3Path)
test = pd.Series(
S3Path("s3://ladi/Images/FEMA_CAP/2020/70349").iterdir()
)
test.s3.bucket
#> 0 ladi
#> 1 ladi
#> ...
#> 577 ladi
#> 578 ladi
#> Length: 579, dtype: objectIf you need to pass specific args or kwargs to the path instantiation, you can pass those at registration time. For example,
S3Path can be passed an S3Client with explicit credentials.
import pandas as pd
from pandas_path import register_path_accessor
from cloudpathlib import S3Path, S3Client
# creates an accessor ".s3" that creates s3 paths using `S3Path(*, client=S3Client(...))`
register_path_accessor("s3", S3Path, client=S3Client(profile_name='other_aws_profile'))
test = pd.Series(
S3Path("s3://ladi/Images/FEMA_CAP/2020/70349").iterdir()
)
test.s3.bucket
#> 0 ladi
#> 1 ladi
#> ...
#> 577 ladi
#> 578 ladi
#> Length: 579, dtype: objectAnother example is if you want to use Windows paths on a Posix machine. You can explicitly indicate you want
to work with PureWindowsPath to do this on any operating system:
import pandas as pd
from pandas_path import register_path_accessor
from pathlib import PureWindowsPath
register_path_accessor("win", PureWindowsPath)
test = pd.Series([
r"c:\test\f1.txt",
r"c:\test2\f2.txt",
])
test.win.parent
#> 0 c:\test
#> 1 c:\test2
#> dtype: objectLimitations
- While most operations work out of the box, operator chaining with
/will not work as expected since we always return the series itself, not the accessor.
file_paths.path.parent.path / "subfolder" / "other_file1.txt"
# ----> 1 file_paths.path.parent.path / "subfolder" / "other_file1.txt"
# ...
# TypeError: unsupported operand type(s) for /: 'str' and 'str'Instead, either use the .path accessor on the result or re-write without chaining:
(file_paths.path.parent.path / "subfolder").path / "other_file1.txt"
# 0 subfolder/other_file1.txt
# 1 pandas_path/subfolder/other_file1.txt
# 2 pandas_path/subfolder/other_file1.txt
# dtype: object
file_paths.path.parent.path / "subfolder/other_file1.txt"
# 0 subfolder/other_file1.txt
# 1 pandas_path/subfolder/other_file1.txt
# 2 pandas_path/subfolder/other_file1.txt
# dtype: object- A numpy array or pandas series on the left hand side of
/will not work properly.
pd.Series(['a', 'b', 'c']) / pd.Series(['1', '2', '3']).path
## IMPROPERLY BROADCASTS :'(
# 0 0 a/1
# 1 a/2
# 2 a/3
# dtype: object
# 1 0 b/1
# 1 b/2
# 2 b/3
# dtype: object
# 2 0 c/1
# 1 c/2
# 2 c/3
# dtype: object
# dtype: objectInstead, use the path accessor on the right-hand side as well.
pd.Series(['a', 'b', 'c']).path / pd.Series(['1', '2', '3']).path
# 0 a/1
# 1 b/2
# 2 c/3
# dtype: object-
Pathobject on the left-hand side of a join (Python < 3.8)
Due to a bug in Python, this never gets handed off to us.
Path("dir") / pd.Series(['a', 'b', 'c']).path
# TypeError: expected str, bytes or os.PathLike object, not PathAccessorWorkaround is to use a str on the left-hand side:
str(Path("dir")) / pd.Series(['a', 'b', 'c']).path
# 0 dir/a
# 1 dir/b
# 2 dir/c
# dtype: objectThat's all folks, enjoy!
Developed and maintained by your friends at DrivenData! ml competitions | ai consulting
Some examples created with reprexlite v0.4.2 to ensure reproducibility.
