Keywords
Machine, Learning, nlp, cv, cli, python
License
MIT
Install
pip install sparrow-tool==0.8.5

Documentation

sparrow-python

image image image image


Install

pip install sparrow-python
# Or dev version
pip install sparrow-python[dev]
# Or
pip install -e .
# Or
pip install -e .[dev]

Usage

Multiprocessing SyncManager

Open server first:

$ spr start-server

The defualt port 50001.

(Process1) productor:

from sparrow.multiprocess.client import Client

client = Client(port=50001)
client.update_dict({'a': 1, 'b': 2})

(Process2) consumer:

from sparrow.multiprocess.client import Client

client = Client(port=50001)
print(client.get_dict_data())

>> > {'a': 1, 'b': 2}

Common tools

  • Kill process by port
$ spr kill {port}
  • pack & unpack
    support archive format: "zip", "tar", "gztar", "bztar", or "xztar".
$ spr pack pack_dir
$ spr unpack filename extract_dir
  • Scaffold
$ spr create awosome-project

Some useful functions

sparrow.relp
Relative path, which is used to read or save files more easily.

sparrow.performance.MeasureTime
For measuring time (including gpu time)

sparrow.performance.get_process_memory
Get the memory size occupied by the process

sparrow.performance.get_virtual_memory
Get virtual machine memory information

sparrow.add_env_path
Add python environment variable (use relative file path)

Safe logger in multiprocessing

from sparrow.log import Logger
import numpy as np

logger = Logger(name='train-log', log_dir='./logs', )
logger.info("hello", "numpy:", np.arange(10))

logger2 = Logger.get_logger('train-log')
print(id(logger2) == id(logger))
>> > True