Easy environment : easy-to-use Python environment management toolkit
Easy Environment is a Python tool that provides easy-to-use functionality for managing files and data in different environments. It offers a class that simplifies file operations on the local disk and cloud services such as Google Cloud (Google Cloud Storage and Big Query) or SharePoint.
Features
-
Multi-format loading and saving: Load and save files in various formats with one command line
- Default supported formats: csv, docx, jpg, json, md, parquet, pdf, pickle, png, pptx, sql, toml, txt, xlsx, xml, yaml, yml
- Unsupported formats: Customisable. See Customise supported formats.
-
Multi-environment management:
- Local disk: Loading/saving and management.
- Google Cloud Storage: Loading/saving and management.
- Big Query: Append, write, and run queries on Big Query tables.
- SharePoint: Download, upload, and manage files on SharePoint.
Start by installing easyenvi
:
pip install easyenvi==1.0.6
Multi-format loading and saving
Load or save a large variety of format : csv, docx, jpg, json, md, parquet, pdf, pickle, png, pptx, sql, toml, txt, xlsx, xml, yaml, yml
from easyenvi import file
secrets = file.load('my_path/secrets.toml')
config = file.load('my_path/config.json')
query = file.load('my_path/titanic.sql')
file.save(df, 'my_path/titanic.csv')
file.save(df, 'my_path/my_dict.parquet')
file.save(my_dict, 'my_path/my_dict.pickle')
Multi-environment management
To use Easy Environment, create an instance of the EasyEnvironment
class. All the parameters in the EasyEnvironment
class are optional: it depends on which environment you need to access.
from easyenvi import EasyEnvironment
envi = EasyEnvironment(
local_path="", # Optional
gcloud_project_id="your-project-id", # Optional
gcloud_credential_path="path/to/credentials.json", # Optional
GCS_path="gs://your-bucket-name/", # Optional
sharepoint_site_url="https://{tenant}.sharepoint.com/sites/{site}", # Optional
sharepoint_username="your-username", # Optional
sharepoint_user_password="your-password" # Optional
)
Specifying certain parameters means certain dependencies:
- For using local operation,
local_path
is the path from which local operations should be executed - specify an empty string if you want to use the current directory. - For using Google Cloud, it is necessary to specify the project ID, the path to a credential .json file, and, in case of interaction with Google Cloud Storage, the path to the GCS folder (see Google Cloud Initialisation).
- For using SharePoint, it is necessary to specify the SharePoint site to interact with, as well as authentication credentials: either the client_id/client_secret pair or the username/user_password pair (see SharePoint Initialisation).
Examples of use
Local features
# Load any file format
my_dict = envi.local.load(path='inputs/my_dictionnary.pickle')
my_logo = envi.local.load(path='inputs/my_logo.png')
dataset = envi.local.load(path='inputs/dataset.csv')
# Save any file format
envi.local.save(obj=my_dict, path='outputs/my_dictionnary.pickle')
envi.local.save(obj=my_logo, path='outputs/my_logo.png')
envi.local.save(obj=dataset, path='outputs/dataset.csv')
Google Cloud Storage features
# Load any file format
my_dict = envi.gcloud.GCS.load(path='inputs/my_dictionnary.pickle')
my_logo = envi.gcloud.GCS.load(path='inputs/my_logo.png')
dataset = envi.gcloud.GCS.load(path='inputs/dataset.csv')
# Save any file format
envi.gcloud.GCS.save(obj=my_dict, path='outputs/my_dictionnary.pickle')
envi.gcloud.GCS.save(obj=my_logo, path='outputs/my_logo.png')
envi.gcloud.GCS.save(obj=dataset, path='outputs/dataset.csv')
Big Query features
df = pd.DataFrame(data={'age': [21, 52, 30], 'wage': [12, 17, 11]})
# Create a new table
envi.gcloud.BQ.write(dataset, 'mydata.mytable')
# Append an existing table
envi.gcloud.BQ.append(dataset, 'mydata.mytable')
# Run queries
query = """
SELECT *
FROM mydata.mytable
WHERE age < 40
"""
new_dataset = envi.gcloud.BQ.query(query).to_dataframe()
SharePoint features
# Download a file
envi.sharepoint.download(input_path="/Document partages/sharepoint_folder/my_file.txt",
output_path="local_folder/my_file.txt")
# Upload a file
envi.sharepoint.upload(input_path="local_folder/my_file.txt",
output_path="Document partages/folder/my_file.txt")
# List files
envi.sharepoint.list_files(folder="local_folder")
Documentation
The documentation is available here : Easy Environment - Documentation
Credits
- Thanks to Herve Mignot for his advice on using
fsspec
. - Thanks to Nizar Fawal for encouraging me to deploy this package as a Pypi library.
- Thanks to Julien Lambert for the code snippets and feedback.
Future Improvements
Future releases of Easy Environment will include support for additional cloud storage providers, including Amazon Web Services (AWS) and Microsoft Azure.