LouisTools -- Tools for data/operation/testing engineers
About Louis
Louis is a data engineer working in financial area. A Python programmer, Linux operationer, loving learning and sharing.
About Tools
The tools are designed for the following technicians:
- data engineers (ETL, especially for financial data and time-series data)
- operation engineers (mainly Linux)
- testing engineers (unnittest, logging, HTML, email)
- data analysts (data I/O and data-washing, especially for financial data)
- application developers (regex, logging, config I/O, useful decorators)
A HTMLTestRunner in Python 3 version is contained in the package (Thanks to Wai Yip Tung for making such a good-to-use Python 2 package).
Examples
1: You can parse a path string including date formatters:
import LouisTools as LT
file_path = "/data/certain_data/%Y/%m/%Y%m%d.csv"
today_str = LT.TODAY
today_file_path = LT.parse_date_in_str(file_path, today_str)
print(today_file_path) # /data/certain_data/2020/06/20200616.csv2: You can make a (parent) directory safely:
import LouisTools as LT
file_to_save = "/home/usr/new_project/data.hdf"
LT.make_parent_dir(file_to_save) # /home/usr/new_project will be created if not exist
dir_to_make = "/home/usr/new_project"
LT.make_dir(file_to_save) # /home/usr/new_project will be created if not exist3: You can create a logger very simply and fastly:
import LouisTools as LT
LOGGER = LT.single_lvl_logger("LouisTools") # Only StreamHandler, LouisTools is the log name
LOGGER = LT.single_lvl_logger("LouisTools", "/project/log/a_log_file.log") # FileHandler is created saving in "/project/log/a_log_file.log"4: You can interact with you config files safely and quickly (both json and ini supported):
import LouisTools as LT
json_path = "/home/usr/projectL/projectL.json" # A typical json file path in Linux
config_content = LT.read_json(json_path, encoding_="utf-8") # You can change encoding_ arg as what you want!
assert isinstance(config_content, dict)
ini_path = r"C:\Users\A_USER\projectX\projectX.ini" # A typical ini file path in Windows
config_content = LT.read_ini(ini_path, encoding_="utf-8-sig") # Windows' gift: a BOM added for freee!
assert isinstance(config_content, dict)5: You can dump the result without overwrite possibly existing files (safe, safe, and safe! even the directory will be checked and created if not exist):
import LouisTools as LT
@LT.dump_without_overwrite(sep="\t", encoding="utf-8", header=True, index=False)
def generate_df(df, df_path):
# You can do something on df
return df, df_path
