os-dbnetget

Library and command line tool not just for qdb.

This project implement a python qdb toolkit as dbnetget at first. With the progress of development, it is separated into multiple sub-projects as os-qdb-protocal, os-m3-engine, this project become a framework for similar works not just for qdb.

The main components are client libraries and command line tool.

Client is used for network processing(connect, close, send, receive, etc.) . It is protocol-independent and transparent to user. By now, there are sync/tornado-async clients, thread-safe client pool and tornado-async client pool are also convenient.

Command line tool's features can be extended by installing extra packages. See Install.

Greatly appreciate Bear Tian and his dbnetget!

Install

• install package

  pip install os-dbnetget
  

• install extra packages

  subpackage	install command	enables
  m3	pip install os-dbnetget[m3]	Install m3 for command line tool support m3(multi-thread) engine
  tornado	pip install os-dbnetget[tornado]	Install Tornado for async client and command line tool support tornado engine
  rotate	pip install os-dbnetget[rotate]	Enable write data to rotate file

Client API

There are sync/async clients, generally speaking, you should not use them directly. The pool may be your first choice.

SyncClientPool

• native multi-thread, do not need extra packages
• thread safe
• retry when network error
• support multi connections with one endpoint

Example:

from os_qdb_protocal import create_protocal
from os_dbnetget.commands.qdb import qdb_key
from os_dbnetget.clients.sync_client import SyncClientPool

endpoints = ['host%02d:8012' % i for i in range(1, 10)]
pool = SyncClientPool(endpoints)

proto = create_protocal('test', qdb_key(b'test-key'))
result = pool.execute(proto)

pool.close()

TornadoClientPool

• support tornado async
• retry when network error
• support multi connections with one endpoint

Example:

from tornado import gen
from tornado.ioLoop import IOLoop

from os_qdb_protocal import create_protocal
from os_dbnetget.commands.qdb import qdb_key
from os_dbnetget.clients.tonado_client import TornadoClientPool

@gen.coroutine
def main():
     
    endpoints = ['host%02d:8012' % i for i in range(1, 10)]
    pool = TornadoClientPool(endpoints)

    proto = create_protocal('test', qdb_key(b'test-key'))
    result = yield pool.execute(proto)

    yield pool.close()

IOLoop.current().run_sync(main)

Command line

• command line tool is on progress, the supported sub-commands:

  os-dbnetget -h
  

• each sub-command has is own features

  os-dbnetget [sub-command] -h
  

• some extra packages can be installed for enhancement, see Install

  for example, you can install m3 engine to improve processing capacity

  pip install os-dbnetget[m3]
  

  cat data.txt | os-dbnetget test --engine m3 --thread-num 50 -L endpoints.lst
  

Unit Tests

$ tox

License

MIT licensed.